diff options
4 files changed, 26 insertions, 12 deletions
@@ -27,6 +27,11 @@ git-annex (6.20170215) UNRELEASED; urgency=medium * adjust: Fix behavior when used in a repository that contains submodules. * Run wget with -nv instead of -q, so it will display HTTP errors. + * Run curl with -S, so HTTP errors are displayed, even when + it's otherwise silent. + * When downloading in --json or --quiet mode, use curl in preference + to wget, since curl is able to display only errors to stderr, unlike + wget. -- Joey Hess <id@joeyh.name> Tue, 14 Feb 2017 15:54:25 -0400 diff --git a/Utility/Url.hs b/Utility/Url.hs index 166783e09..1b68dce7f 100644 --- a/Utility/Url.hs +++ b/Utility/Url.hs @@ -246,17 +246,16 @@ headRequest r = r (requestHeaders r) } -{- Used to download large files, such as the contents of keys. +{- Download a perhaps large file, with auto-resume of incomplete downloads. - - - Uses wget or curl program for its progress bar. (Wget has a better one, - - so is preferred.) Which program to use is determined at run time; it - - would not be appropriate to test at configure time and build support - - for only one in. + - Uses wget or curl program for its progress bar and resuming support. + - Which program to use is determined at run time depending on which is + - in path and which works best in a particular situation. -} download :: URLString -> FilePath -> UrlOptions -> IO Bool download = download' False -{- No output, even on error. -} +{- No output to stdout. -} downloadQuiet :: URLString -> FilePath -> UrlOptions -> IO Bool downloadQuiet = download' True @@ -265,6 +264,12 @@ download' quiet url file uo = do case parseURIRelaxed url of Just u | uriScheme u == "file:" -> curl + -- curl is preferred in quiet mode, because + -- it displays http errors to stderr, while wget + -- does not display them in quiet mode + | quiet -> ifM (inPath "curl") (curl, wget) + -- wget is preferred mostly because it has a better + -- progress bar | otherwise -> ifM (inPath "wget") (wget , curl) _ -> return False where @@ -275,7 +280,8 @@ download' quiet url file uo = do - support, or need that option. - - When the wget version is new enough, pass options for - - a less cluttered download display. + - a less cluttered download display. Using -nv rather than -q + - avoids most clutter while still displaying http errors. -} #ifndef __ANDROID__ wgetparams = concat @@ -296,7 +302,7 @@ download' quiet url file uo = do -- curl does not create destination file -- if the url happens to be empty, so pre-create. writeFile file "" - go "curl" $ headerparams ++ quietopt "-s" ++ + go "curl" $ headerparams ++ quietopt "-sS" ++ [ Param "-f" , Param "-L" , Param "-C", Param "-" diff --git a/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json.mdwn b/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json.mdwn index b9283ae54..3e9c30970 100644 --- a/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json.mdwn +++ b/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json.mdwn @@ -18,3 +18,5 @@ ATM I am experiencing sporadic failures of the batched git annex addurl call -- besides me blindly trying to re-run it e.g. 3 times and only then declare total failure, I wondered if json output could provide more information (if any known) about the failure... e.g. if a custom remote crashed/errorred (I guess the case here due to "from datalad") -- what was stderr/exit code for that process if crashed/ERROR msg... if wget -- what was stderr there [[!meta name=yoh]] + +> Switched to curl with -sS in --json mode. [[done]] I suppose. --[[Joey]] diff --git a/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_2_82b851629c695084cbf62e2b636bcc91._comment b/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_2_82b851629c695084cbf62e2b636bcc91._comment index 40935530e..0ece7ca77 100644 --- a/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_2_82b851629c695084cbf62e2b636bcc91._comment +++ b/doc/todo/more_of_diagnostic_information_in_case_of_failures_into_returned_json/comment_2_82b851629c695084cbf62e2b636bcc91._comment @@ -3,13 +3,14 @@ subject="""comment 2""" date="2017-02-20T19:15:18Z" content=""" +It seems there is no way to get only errors on +stderr with wget; the choice is between no output, and a mixture of errors +and informational messages on stderr. In --json or --quiet mode, only +errors should be output to stderr. + In general, the --json output does include a "note" with any available message about why an operation failed. -Since wget outputs HTTP errors to stdout, there's no way to capture -the actual message for json, and so swiching to wget -nv won't improve -the json, since in json mode wget is run with -q. - It would not be hard to use a HTTP library and propagate the HTTP errors into the json "note", but it might be hard to get resumption of partial downloads to work as well with a HTTP library as it works with wget/curl. |