aboutsummaryrefslogtreecommitdiff
path: root/Utility/Url.hs
Commit message (Collapse)AuthorAge
* import Data.Default in CommonGravatar Joey Hess2015-01-28
|
* addurl: When a Content-Disposition header suggests a filename to use, addurl ↵Gravatar Joey Hess2015-01-22
| | | | will consider using it, if it's reasonable and doesn't conflict with an existing file. (--file overrides this)
* excess indentGravatar Joey Hess2015-01-22
|
* update my email address and homepage urlGravatar Joey Hess2015-01-21
|
* add getFileSize, which can get the real size of a large file on WindowsGravatar Joey Hess2015-01-20
| | | | | | | | | | | | | | Avoid using fileSize which maxes out at just 2 gb on Windows. Instead, use hFileSize, which doesn't have a bounded size. Fixes support for files > 2 gb on Windows. Note that the InodeCache code only needs to compare a file size, so it doesn't matter it the file size wraps. So it has been left as-is. This was necessary both to avoid invalidating existing inode caches, and because the code passed FileStatus around and would have become more expensive if it called getFileSize. This commit was sponsored by Christian Dietrich.
* Use wget -q --show-progress for less verbose wget output, when built with ↵Gravatar Joey Hess2014-12-16
| | | | wget 1.16.
* add compat cruft for old versions of http-types and http-conduitGravatar Joey Hess2014-08-17
|
* work around default Accept-Encoding in http-clientGravatar Joey Hess2014-08-15
|
* memoize construction of the Request -> Request function to apply the UrlOptionsGravatar Joey Hess2014-08-15
|
* Switched from the old haskell HTTP library to http-conduit.Gravatar Joey Hess2014-08-15
| | | | | | | | | | | | | | | | | | The hoary old HTTP library was only used when checking if an url exists, when curl was not available. It had many problems, including not supporting https at all. Now, this is done using http-conduit for all urls that it supports. Falls back to curl for any url that http-conduit doesn't like (probably ftp etc, but could also be an url that its parser chokes on for whatever reason). This adds a new dependency on http-conduit, but webdav support already indirectly depended on that, and the s3-aws branch also uses it. This opens up the possibility of using http-conduit for large file downloads, but for now I've left it using wget/curl. This commit was sponsored by Paul Tötterman.
* unify exception handling into Utility.ExceptionGravatar Joey Hess2014-08-07
| | | | | | | | | | | | | | | | | | | | Removed old extensible-exceptions, only needed for very old ghc. Made webdav use Utility.Exception, to work after some changes in DAV's exception handling. Removed Annex.Exception. Mostly this was trivial, but note that tryAnnex is replaced with tryNonAsync and catchAnnex replaced with catchNonAsync. In theory that could be a behavior change, since the former caught all exceptions, and the latter don't catch async exceptions. However, in practice, nothing in the Annex monad uses async exceptions. Grepping for throwTo and killThread only find stuff in the assistant, which does not seem related. Command.Add.undo is changed to accept a SomeException, and things that use it for rollback now catch non-async exceptions, rather than only IOExceptions.
* relicense general utility library code to BSDGravatar Joey Hess2014-05-10
| | | | | Omitted a couple of files what have had significant contributions from others.
* avoid exception when curl exits nonzero (due to eg, bad domain name)Gravatar Joey Hess2014-03-27
|
* add UrlOptions sum typeGravatar Joey Hess2014-02-24
|
* Make annex.web-options be used in several places that call curl.Gravatar Joey Hess2014-02-24
|
* Android: Avoid passing --clobber to busybox wget.Gravatar Joey Hess2014-01-13
|
* Added support for quvi 0.9. Slightly suboptimal due to limitations in its ↵Gravatar Joey Hess2013-11-24
| | | | interface compared with the old version.
* url size fixesGravatar Joey Hess2013-10-11
| | | | | | | | | | addurl: Improve message when adding url with wrong size to existing file. Before the message suggested the url didn't exist. Fixed handling of URL keys that have no recorded size. Before, if the key has no size, the url also had to not declare any size, which was unlikely and wrong, or it was taken to not exist. This probably would mostly affect keys that were added to the annex with addurl --relaxed.
* Send a git-annex user-agent when downloading urls.Gravatar Joey Hess2013-09-28
| | | | | | | | | Overridable with --user-agent option. Not yet done for S3 or WebDAV due to limitations of libraries used -- nether allows a user-agent header to be specified. This commit sponsored by Michael Zehrer.
* Set --clobber when running wget to ensure resuming works properly.Gravatar Joey Hess2013-08-21
|
* Revert "Delete empty downloaded file when wget fails, to work around ↵Gravatar Joey Hess2013-08-21
| | | | | | | | reported resume failure." This reverts commit 845322a564fe9ff098596f419d06ec9aa748aa2a. Better fix forthcoming
* Delete empty downloaded file when wget fails, to work around reported resume ↵Gravatar Joey Hess2013-08-21
| | | | | | | | | | | | | | | | | | | | | | | | | failure. <RichiH> i richih@eudyptes (git)-[master] ~git/debconf-share/debconf13/photos/chrysn % rm /home/richih/work/git/debconf-share/.git/annex/tmp/SHA256E-s3044235--693b74fcb12db06b5e79a8b99d03e2418923866506ee62d24a4e9ae8c5236758.JPG <RichiH> richih@eudyptes (git)-[master] ~git/debconf-share/debconf13/photos/chrysn % git annex get P8060008.JPG <RichiH> get P8060008.JPG (from website...) --2013-08-21 21:42:45-- http://annex.debconf.org/debconf-share/.git//annex/objects/1a4/67d/SHA256E-s3044235--693b74fcb12db06b5e79a8b99d03e2418923866506ee62d24a4e9ae8c5236758.JPG/SHA256E-s3044235--693b74fcb12db06b5e79a8b99d03e2418923866506ee62d24a4e9ae8c5236758.JPG <RichiH> Resolving annex.debconf.org (annex.debconf.org)... 5.153.231.227, 2001:41c8:1000:19::227:2 <RichiH> Connecting to annex.debconf.org (annex.debconf.org)|5.153.231.227|:80... connected. <RichiH> HTTP request sent, awaiting response... 404 Not Found <RichiH> 2013-08-21 21:42:45 ERROR 404: Not Found. <RichiH> File `/home/richih/work/git/debconf-share/.git/annex/tmp/SHA256E-s3044235--693b74fcb12db06b5e79a8b99d03e2418923866506ee62d24a4e9ae8c5236758.JPG' already there; not retrieving. <RichiH> Unable to access these remotes: website <RichiH> Try making some of these repositories available: <RichiH> 3e0356ac-0743-11e3-83a5-1be63124a102 -- website (annex.debconf.org) <RichiH> a7495021-9f2d-474e-80c7-34d29d09fec6 -- chrysn@hephaistos:~/data/projects/debconf13/debconf-share <RichiH> eb8990f7-84cd-4e6b-b486-a5e71efbd073 -- joeyh passport usb drive <RichiH> f415f118-f428-4c68-be66-c91501da3a93 -- joeyh laptop <RichiH> failed <RichiH> git-annex: get: 1 failed <RichiH> richih@eudyptes (git)-[master] ~git/debconf-share/debconf13/photos/chrysn % I was not able to reproduce the failure, but I did reproduce that wget -O http://404/ results in an empty file being written.
* importfeed: git-annex becomes a podcatcher in 150 LOCGravatar Joey Hess2013-07-28
|
* Pass -f to curl when downloading a file with it, so it propigates failure.Gravatar Joey Hess2013-07-06
|
* Improve error handling when getting uuid of http remotes to auto-ignore, ↵Gravatar Joey Hess2013-05-25
| | | | like with ssh remotes.
* fix permission damage (thanks, Windows)Gravatar Joey Hess2013-05-11
|
* clean up from windows portingGravatar Joey Hess2013-05-11
|
* git-annex now builds on Windows (doesn't work)Gravatar Joey Hess2013-05-11
|
* Fall back to internal url downloader when built without curl.Gravatar Joey Hess2013-04-16
|
* addurl url escaping fooGravatar Joey Hess2013-03-10
| | | | | | * addurl: Escape invalid characters in urls, rather than failing to use an invalid url. * addurl: Properly handle url-escaped characters in file:// urls.
* avoid using curl for file:// urls since it's buggyGravatar Joey Hess2013-03-10
|
* cleanupGravatar Joey Hess2013-01-28
|
* addurl --fast: Use curl, rather than haskell HTTP library, to support https.Gravatar Joey Hess2013-01-27
|
* finished where indentation changesGravatar Joey Hess2012-12-13
|
* use cabal macro to detect if old version of network is being usedGravatar Joey Hess2012-11-11
|
* switch to new URI version by default, -DWITH_OLD_URI for oldGravatar Joey Hess2012-11-03
|
* file:/// URLs can now be used with the web special remote.Gravatar Joey Hess2012-10-21
|
* deal with incompatable api change in network 2.4.0.1Gravatar Joey Hess2012-10-10
| | | | | On the cabal side, let's just require this new version, and set -DURI_24 to enable the code using it.
* bugfixesGravatar Joey Hess2012-04-22
|
* Add annex.httpheaders and annex.httpheader-command config settingsGravatar Joey Hess2012-04-22
| | | | | | Allow custom headers to be sent with all HTTP requests. (Requested by the Internet Archive)
* noopGravatar Joey Hess2012-04-21
|
* optimize monadic ||Gravatar Joey Hess2012-03-16
| | | | | (||) used applicative style runs both conditions rather than short circuiting. Add an orM that properly short-circuits.
* nukes another 15 lines thanks to ifMGravatar Joey Hess2012-03-15
|
* hlintGravatar Joey Hess2012-02-16
|
* work around Network.Browser bug that converts a HEAD to a GET when following ↵Gravatar Joey Hess2012-02-10
| | | | | | | | | | | | | | | | | | | | | | a redirect The code explicitly switches from HEAD to GET for most redirects. Possibly because someone misread a spec (which does require switching from POST to GET for 303 redirects). Or possibly because the spec really is that bad. Upstream bug: https://github.com/haskell/HTTP/issues/24 Since we absolutely don't want to download entire (large) files from the web when checking that they exist with HEAD, I wrote my own redirect follower, based closely on the one used by Network.Browser, but without this misfeature. Note that Network.Browser checks that the redirect url is a http url and fails if not. I don't, because I want to not need to change this code when it gets https support (related: I'm surprised to see it doesn't support https yet..). The check does not seem security significant; it doesn't support file:// urls for example. If a http url is redirected to https, the Network.Browser will actually make a http connection again. This could loop, but only up to 5 times.
* When checking that an url has a key, verify that the Content-Length, if ↵Gravatar Joey Hess2012-02-10
| | | | | | | | available, matches the size of the key. If there's no Content-Length, or the key has no size, this check is not done, but it should happen most of the time, and protect against web content that has changed.
* Added remote.name.annex-web-options configuration setting, which can be used ↵Gravatar Joey Hess2012-01-02
| | | | to provide parameters to whichever of wget or curl git-annex uses (depends on which is available, but most of their important options suitable for use here are the same).
* updateGravatar Joey Hess2011-12-20
|
* commentGravatar Joey Hess2011-12-20
|
* broke up UtilityGravatar Joey Hess2011-10-16
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-03 10:21:07 +0000