Run wget with -nv instead of -q, so it will display HTTP errors.

This adds one extra line of output when a download is successful,
after the progress bar. I don't much like that, but wget does not provide a
way to show HTTP errors without it.

5 files changed, 40 insertions, 5 deletions

diff --git a/Build/Configure.hs b/Build/Configure.hs
index 616dcc57d..7c7c2efe0 100644
--- a/Build/Configure.hs
+++ b/Build/Configure.hs
@@ -34,7 +34,7 @@ tests =
 	, TestCase "rsync" $ requireCmd "rsync" "rsync --version >/dev/null"
 	, TestCase "curl" $ testCmd "curl" "curl --version >/dev/null"
 	, TestCase "wget" $ testCmd "wget" "wget --version >/dev/null"
-	, TestCase "wget supports -q --show-progress" checkWgetQuietProgress
+	, TestCase "wget unclutter options" checkWgetUnclutter
 	, TestCase "bup" $ testCmd "bup" "bup --version >/dev/null"
 	, TestCase "nice" $ testCmd "nice" "nice true >/dev/null"
 	, TestCase "ionice" $ testCmd "ionice" "ionice -c3 true >/dev/null"
@@ -105,8 +105,8 @@ getGitVersion = go =<< getEnv "FORCE_GIT_VERSION"
 			error $ "installed git version " ++ show v ++ " is too old! (Need " ++ show oldestallowed ++ " or newer)"
 		return $ Config "gitversion" $ StringConfig $ show v
 
-checkWgetQuietProgress :: Test
-checkWgetQuietProgress = Config "wgetquietprogress" . BoolConfig
+checkWgetUnclutter :: Test
+checkWgetUnclutter = Config "wgetunclutter" . BoolConfig
 	. maybe False (>= normalize "1.16")
 	<$> getWgetVersion 
 
diff --git a/CHANGELOG b/CHANGELOG
index 10c0c9f43..4e150f340 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -26,6 +26,7 @@ git-annex (6.20170215) UNRELEASED; urgency=medium
     webapp and test suite.
   * adjust: Fix behavior when used in a repository that contains
     submodules.
+  * Run wget with -nv instead of -q, so it will display HTTP errors.
 
  -- Joey Hess <id@joeyh.name>  Tue, 14 Feb 2017 15:54:25 -0400
 
diff --git a/Utility/Url.hs b/Utility/Url.hs
index da1a38dd3..45a34bc35 100644
--- a/Utility/Url.hs
+++ b/Utility/Url.hs
@@ -279,8 +279,8 @@ download' quiet url file uo = do
 	 -}
 #ifndef __ANDROID__
 	wgetparams = concat
-		[ if Build.SysConfig.wgetquietprogress && not quiet
-			then [Param "-q", Param "--show-progress"]
+		[ if Build.SysConfig.wgetunclutter && not quiet
+			then [Param "-nv", Param "--show-progress"]
 			else []
 		, [ Param "--clobber", Param "-c", Param "-O"]
 		]
diff --git a/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_1_0eeb859b57d4dc8a3c9c9c3c4f70bb45._comment b/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_1_0eeb859b57d4dc8a3c9c9c3c4f70bb45._comment
new file mode 100644
index 000000000..80dfe504b
--- /dev/null
+++ b/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_1_0eeb859b57d4dc8a3c9c9c3c4f70bb45._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-02-20T18:50:18Z"
+ content="""
+Probably wget is just failing to download the url sometimes.
+Eg, `git annex addurl http://localhost/dne` fails with the same not useful
+output.
+
+wget is run with -q, which is the only way to turn off all its informational
+messages, but unfortunately that also turns off display of HTTP error messages.
+
+Using -nv instead of -q would display HTTP errors,
+but also 1 extra line of output once the download is complete.
+I suppose that's worth the trade-off.
+"""]]
diff --git a/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_2_82b851629c695084cbf62e2b636bcc91._comment b/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_2_82b851629c695084cbf62e2b636bcc91._comment
new file mode 100644
index 000000000..b7c65c51b
--- /dev/null
+++ b/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_2_82b851629c695084cbf62e2b636bcc91._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2017-02-20T19:15:18Z"
+ content="""
+In general, the --json output does include a "note" with any
+available message about why an operation failed. 
+
+Since wget outputs HTTP errors to stdout, there's no way to capture the actual
+message for json, and so swiching to wget -nv won't improve the json.
+(curl also outputs HTTP errors to stdout too, so it's no better)
+
+Short of using a HTTP library for url downloads in --json mode,
+I don't see a way to fix this. It would not be hard to use a HTTP library
+and propagate the HTTP errors into the json "note", but it might be hard to
+get resumption of partial downloads to work as well with a HTTP library as
+it works with wget.
+""]]