Make image decoding more fault resistant, less verbose.

This change address what happens when a jpeg is partially downloaded
before failing.  Many browsers will render it anyway: we want Skia to
do the same.  The JpegTest takes a perfectly cromulent jpeg file and
only passes into the ImageDecoder the first half of the image.  We
then verify that the image decoder returns a valid bitmap of the
correct dimensions.

We also fixed some png library errors, including issue 1691.

Also, suppressed the majority of warnings from using libpng and
libjpeg.  By default, most warnings are *not* suppressed in debug mode.
If you have a debug binary and wish to suppress warnings, set the
following environment variables to true
    skia_images_png_suppressDecoderWarnings
    skia_images_jpeg_suppressDecoderWarnings
or from within a program that links to Skia:
    #if defined(SK_DEBUG)
    #include "SkRTConf.h"
    SK_CONF_SET("images.jpeg.suppressDecoderWarnings", true);
    SK_CONF_SET("images.png.suppressDecoderWarnings", true);
    #endif

I tested this, before (control) and after these changes (test), on
364,295 skps from the cluster telemetry.
-   number of errors+warnings in control = 2804
-   number of errors+warnings fixed = 2283
-   number of PNG verbosity fixed =  2152
-   number of PNG error fixed = 4
-   number of PNG segfault fixed = 3
-   number of PNG errors changed to warnings = 62
-   number of JPG verbosity fixed =  26
-   number of JPG error fixed = 91
Not all errors and warning have been fixed.

These numbers were generated using the find_bad_images_in_skps.py
program.  This program may be useful going forward for testing
image-decoding libraries on skp files from the cluster telemetry.
find_bad_images_in_skps.py depends on the test_image_decoder program,
which simply executes the SkImageDecoder::DecodeFile function and uses
its exit status to report success or failure.

BUG=skia:1649
BUG=skia:1691
BUG=skia:1680
R=scroggo@google.com

Review URL: https://codereview.chromium.org/24449003

git-svn-id: http://skia.googlecode.com/svn/trunk@11597 2bbb7eff-a529-9590-31e7-b0007b416f81

2 files changed, 229 insertions, 0 deletions

diff --git a/tools/find_bad_images_in_skps.py b/tools/find_bad_images_in_skps.py
new file mode 100755
index 0000000000..5dcf6a477e
--- /dev/null
+++ b/tools/find_bad_images_in_skps.py
@@ -0,0 +1,197 @@
+#!/usr/bin/env python
+
+# Copyright 2013 Google Inc. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""
+This script will take as an argument either a list of skp files or a
+set of directories that contains skp files.  It will then test each
+skp file with the `render_pictures` program. If that program either
+spits out any unexpected output or doesn't return 0, I will flag that
+skp file as problematic.  We then extract all of the embedded images
+inside the skp and test each one of them against the
+SkImageDecoder::DecodeFile function.  Again, we consider any
+extraneous output or a bad return value an error.  In the event of an
+error, we retain the image and print out information about the error.
+The output (on stdout) is formatted as a csv document.
+
+A copy of each bad image is left in a directory created by
+tempfile.mkdtemp().
+"""
+
+import glob
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import threading
+
+import test_rendering  # skia/trunk/tools.  reuse FindPathToProgram()
+
+USAGE = """
+Usage:
+    {command} SKP_FILE [SKP_FILES]
+    {command} SKP_DIR [SKP_DIRS]\n
+Environment variables:
+    To run multiple worker threads, set NUM_THREADS.
+    To use a different temporary storage location, set TMPDIR.
+
+"""
+
+def execute_program(args, ignores=None):
+    """
+    Execute a process and waits for it to complete.  Returns all
+    output (stderr and stdout) after (optional) filtering.
+
+    @param args is passed into subprocess.Popen().
+
+    @param ignores (optional) is a list of regular expression strings
+    that will be ignored in the output.
+
+    @returns a tuple (returncode, output)
+    """
+    if ignores is None:
+        ignores = []
+    else:
+        ignores = [re.compile(ignore) for ignore in ignores]
+    proc = subprocess.Popen(
+        args,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT)
+    output = ''.join(
+        line for line in proc.stdout
+        if not any(bool(ignore.match(line)) for ignore in ignores))
+    returncode = proc.wait()
+    return (returncode, output)
+
+
+def list_files(paths):
+    """
+    Accepts a list of directories or filenames on the command line.
+    We do not choose to recurse into directories beyond one level.
+    """
+    class NotAFileException(Exception):
+        pass
+    for path in paths:
+        for globbedpath in glob.iglob(path): # useful on win32
+            if os.path.isdir(globbedpath):
+                for filename in os.listdir(globbedpath):
+                    newpath = os.path.join(globbedpath, filename)
+                    if os.path.isfile(newpath):
+                        yield newpath
+            elif os.path.isfile(globbedpath):
+                yield globbedpath
+            else:
+                raise NotAFileException('{} is not a file'.format(globbedpath))
+
+
+class BadImageFinder(object):
+
+    def __init__(self, directory=None):
+        self.render_pictures = test_rendering.FindPathToProgram(
+            'render_pictures')
+        self.test_image_decoder = test_rendering.FindPathToProgram(
+            'test_image_decoder')
+        assert os.path.isfile(self.render_pictures)
+        assert os.path.isfile(self.test_image_decoder)
+        if directory is None:
+            self.saved_image_dir = tempfile.mkdtemp(prefix='skia_skp_test_')
+        else:
+            assert os.path.isdir(directory)
+            self.saved_image_dir = directory
+        self.bad_image_count = 0
+
+    def process_files(self, skp_files):
+        for path in skp_files:
+            self.process_file(path)
+
+    def process_file(self, skp_file):
+        assert self.saved_image_dir is not None
+        assert os.path.isfile(skp_file)
+        args = [self.render_pictures, '--readPath', skp_file]
+        ignores = ['^process_in', '^deserializ', '^drawing...', '^Non-defaul']
+        returncode, output = execute_program(args, ignores)
+        if (returncode == 0) and not output:
+            return
+        temp_image_dir = tempfile.mkdtemp(prefix='skia_skp_test___')
+        args = [ self.render_pictures, '--readPath', skp_file,
+                 '--writePath', temp_image_dir, '--writeEncodedImages']
+        subprocess.call(args, stderr=open(os.devnull,'w'),
+                        stdout=open(os.devnull,'w'))
+        for image_name in os.listdir(temp_image_dir):
+            image_path = os.path.join(temp_image_dir, image_name)
+            assert(os.path.isfile(image_path))
+            args = [self.test_image_decoder, image_path]
+            returncode, output = execute_program(args, [])
+            if (returncode == 0) and not output:
+                os.remove(image_path)
+                continue
+            try:
+                shutil.move(image_path, self.saved_image_dir)
+            except (shutil.Error,):
+                # If this happens, don't stop the entire process,
+                # just warn the user.
+                os.remove(image_path)
+                sys.stderr.write('{0} is a repeat.\n'.format(image_name))
+            self.bad_image_count += 1
+            if returncode == 2:
+                returncode = 'SkImageDecoder::DecodeFile returns false'
+            elif returncode == 0:
+                returncode = 'extra verbosity'
+                assert output
+            elif returncode == -11:
+                returncode = 'segmentation violation'
+            else:
+                returncode = 'returncode: {}'.format(returncode)
+            output = output.strip().replace('\n',' ').replace('"','\'')
+            suffix = image_name[-3:]
+            output_line = '"{0}","{1}","{2}","{3}","{4}"\n'.format(
+                returncode, suffix, skp_file, image_name, output)
+            sys.stdout.write(output_line)
+            sys.stdout.flush()
+        os.rmdir(temp_image_dir)
+        return
+
+def main(main_argv):
+    if not main_argv or main_argv[0] in ['-h', '-?', '-help', '--help']:
+        sys.stderr.write(USAGE.format(command=__file__))
+        return 1
+    if 'NUM_THREADS' in os.environ:
+        number_of_threads = int(os.environ['NUM_THREADS'])
+        if number_of_threads < 1:
+            number_of_threads = 1
+    else:
+        number_of_threads = 1
+    os.environ['skia_images_png_suppressDecoderWarnings'] = 'true'
+    os.environ['skia_images_jpeg_suppressDecoderWarnings'] = 'true'
+
+    temp_dir = tempfile.mkdtemp(prefix='skia_skp_test_')
+    sys.stderr.write('Directory for bad images: {}\n'.format(temp_dir))
+    sys.stdout.write('"Error","Filetype","SKP File","Image File","Output"\n')
+    sys.stdout.flush()
+
+    finders = [
+        BadImageFinder(temp_dir) for index in xrange(number_of_threads)]
+    arguments = [[] for index in xrange(number_of_threads)]
+    for index, item in enumerate(list_files(main_argv)):
+        ## split up the given targets among the worker threads
+        arguments[index % number_of_threads].append(item)
+    threads = [
+        threading.Thread(
+            target=BadImageFinder.process_files, args=(finder,argument))
+        for finder, argument in zip(finders, arguments)]
+    for thread in threads:
+        thread.start()
+    for thread in threads:
+        thread.join()
+    number  = sum(finder.bad_image_count for finder in finders)
+    sys.stderr.write('Number of bad images found: {}\n'.format(number))
+    return 0
+
+if __name__ == '__main__':
+    exit(main(sys.argv[1:]))
+
+#  LocalWords:  skp stdout csv
diff --git a/tools/test_image_decoder.cpp b/tools/test_image_decoder.cpp
new file mode 100644
index 0000000000..2ba89aaee3
--- /dev/null
+++ b/tools/test_image_decoder.cpp
@@ -0,0 +1,32 @@
+/*
+ * Copyright 2013 Google Inc.
+ *
+ * Use of this source code is governed by a BSD-style license that can be
+ * found in the LICENSE file.
+ */
+
+#include "SkBitmap.h"
+#include "SkForceLinking.h"
+#include "SkGraphics.h"
+#include "SkImageDecoder.h"
+
+__SK_FORCE_IMAGE_DECODER_LINKING;
+
+/**
+   Simple program to test Skia's ability to decode images without
+   errors or debug messages. */
+int main(int argc, char ** argv) {
+    if (argc < 2) {
+        SkDebugf("Usage:\n %s imagefile\n\n", argv[0]);
+        return 3;
+    }
+    SkBitmap bitmap;
+    if (!(SkImageDecoder::DecodeFile(argv[1], &bitmap))) {
+        return 2;
+    }
+    if (bitmap.empty()) {
+        return 1;
+    }
+    return 0;
+}
+