Truly ignore failures in skimage.

If the expectation is set to ignore failures, do not return a
negative 1 (indicating a failure) when a successful decode
does not match the expectation.

If the file could not be decoded at all, report this to the
user depending on the input expectations file:

No expectations:
Report failure. The user wanted to know if the file could be
decoded.

Expectations expected a valid result:
Report failure. The decode should have matched the result.

Empty expectations:
Print a message that the expectation was missing, but still
return success from skimage, since this is a newly added file
(i.e. it has no expectation yet).

Ignore failure:
Return success from skimage, since it is a known failure.

Update the self tests to ensure these behaviors.

R=epoger@google.com

Review URL: https://codereview.chromium.org/22293006

git-svn-id: http://skia.googlecode.com/svn/trunk@11790 2bbb7eff-a529-9590-31e7-b0007b416f81

6 files changed, 78 insertions, 0 deletions

diff --git a/tools/tests/skimage/input/bad-images/empty-results.json b/tools/tests/skimage/input/bad-images/empty-results.json
new file mode 100644
index 0000000000..8ae56100de
--- /dev/null
+++ b/tools/tests/skimage/input/bad-images/empty-results.json
@@ -0,0 +1,12 @@
+{
+   "actual-results" : {
+      "failed" : null,
+      "failure-ignored" : null,
+      "no-comparison" : null,
+      "succeeded" : null
+   },
+   "expected-results" : {
+      "invalid.png" : {
+      }
+   }
+}
diff --git a/tools/tests/skimage/input/bad-images/ignore-results.json b/tools/tests/skimage/input/bad-images/ignore-results.json
new file mode 100644
index 0000000000..4a0326e108
--- /dev/null
+++ b/tools/tests/skimage/input/bad-images/ignore-results.json
@@ -0,0 +1,16 @@
+{
+   "actual-results" : {
+      "failed" : null,
+      "failure-ignored" : null,
+      "no-comparison" : null,
+      "succeeded" : null
+   },
+   "expected-results" : {
+      "invalid.png" : {
+         "allowed-digests" : [
+            [ "bitmap-64bitMD5", 8011846386646810712 ]
+         ],
+         "ignore-failure" : true
+      }
+   }
+}
diff --git a/tools/tests/skimage/input/bad-images/incorrect-results.json b/tools/tests/skimage/input/bad-images/incorrect-results.json
new file mode 100644
index 0000000000..67b8bdb0c0
--- /dev/null
+++ b/tools/tests/skimage/input/bad-images/incorrect-results.json
@@ -0,0 +1,16 @@
+{
+   "actual-results" : {
+      "failed" : null,
+      "failure-ignored" : null,
+      "no-comparison" : null,
+      "succeeded" : null
+   },
+   "expected-results" : {
+      "invalid.png" : {
+         "allowed-digests" : [
+            [ "bitmap-64bitMD5", 8011846386646810712 ]
+         ],
+         "ignore-failure" : false
+      }
+   }
+}
diff --git a/tools/tests/skimage/input/bad-images/invalid.png b/tools/tests/skimage/input/bad-images/invalid.png
new file mode 100644
index 0000000000..0dd1608e45
--- /dev/null
+++ b/tools/tests/skimage/input/bad-images/invalid.png
@@ -0,0 +1,2 @@
+‰PNG

+
diff --git a/tools/tests/skimage/input/images-with-known-hashes/ignore-failures.json b/tools/tests/skimage/input/images-with-known-hashes/ignore-failures.json
new file mode 100644
index 0000000000..a99cd3e321
--- /dev/null
+++ b/tools/tests/skimage/input/images-with-known-hashes/ignore-failures.json
@@ -0,0 +1,16 @@
+{
+   "actual-results" : {
+      "failed" : null,
+      "failure-ignored" : null,
+      "no-comparison" : null,
+      "succeeded" : null
+   },
+   "expected-results" : {
+      "1209453360120438698.png" : {
+         "allowed-digests" : [
+            [ "bitmap-64bitMD5", 1111111111111111111 ]
+         ],
+         "ignore-failure" : true
+      }
+   }
+}
diff --git a/tools/tests/skimage/input/images-with-known-hashes/incorrect-results.json b/tools/tests/skimage/input/images-with-known-hashes/incorrect-results.json
new file mode 100644
index 0000000000..9cf72c3cd3
--- /dev/null
+++ b/tools/tests/skimage/input/images-with-known-hashes/incorrect-results.json
@@ -0,0 +1,16 @@
+{
+   "actual-results" : {
+      "failed" : null,
+      "failure-ignored" : null,
+      "no-comparison" : null,
+      "succeeded" : null
+   },
+   "expected-results" : {
+      "1209453360120438698.png" : {
+         "allowed-digests" : [
+            [ "bitmap-64bitMD5", 1111111111111111111 ]
+         ],
+         "ignore-failure" : false
+      }
+   }
+}