aboutsummaryrefslogtreecommitdiff
path: root/Command/AddUrl.hs
Commit message (Collapse)AuthorAge
* Display progress meter in -J mode when downloading from the web.Gravatar Joey Hess2015-11-16
| | | | | Including in addurl, and get --from web, but also in S3 and External special remotes when a web url is known for content in those remotes.
* -J for add/addurl/importGravatar Joey Hess2015-11-05
|
* other 80% of avoding verification when hard linking to objects in shared repoGravatar Joey Hess2015-10-02
| | | | | | | | | | | | | | | | | | | | In c3b38fb2a075b4250e867ebd910324c65712c747, it actually only handled uploading objects to a shared repository. To avoid verification when downloading objects from a shared repository, was a lot harder. On the plus side, if the process of downloading a file from a remote is able to verify its content on the side, the remote can indicate this now, and avoid the extra post-download verification. As of yet, I don't have any remotes (except Git) using this ability. Some more work would be needed to support it in special remotes. It would make sense for tahoe to implicitly verify things downloaded from it; as long as you trust your tahoe server (which typically runs locally), there's cryptographic integrity. OTOH, despite bup being based on shas, a bup repo under an attacker's control could have the git ref used for an object changed, and so a bup repo shouldn't implicitly verify. Indeed, tahoe seems unique in being trustworthy enough to implicitly verify.
* rename functionGravatar Joey Hess2015-10-01
|
* importfeed --relaxed: Avoid hitting the urls of items in the feed.Gravatar Joey Hess2015-08-19
|
* addurl now accepts --prefix and --suffix options to adjust the filenames usedGravatar Joey Hess2015-07-21
|
* converted ImportFeedGravatar Joey Hess2015-07-13
|
* converted addurlGravatar Joey Hess2015-07-13
|
* convert all commands to work with optparse-applicativeGravatar Joey Hess2015-07-08
| | | | Still no options though.
* started converting to use optparse-applicativeGravatar Joey Hess2015-07-08
| | | | | | | | | | | | This is a work in progress. It compiles and is able to do basic command dispatch, including git autocorrection, while using optparse-applicative for the core commandline parsing. * Many commands are temporarily disabled before conversion. * Options are not wired in yet. * cmdnorepo actions don't work yet. Also, removed the [Command] list, which was only used in one place.
* Improve url parsing to handle some urls containing illegal [] characters in ↵Gravatar Joey Hess2015-06-14
| | | | | | their paths. Ie, "https://archive.org/download/zoom-2/Zoom - Release 2 (1996)(Active Software)[!].iso"
* fromkey, registerurl: Allow urls to be specified instead of keys, and ↵Gravatar Joey Hess2015-05-22
| | | | | | | | generate URL keys. This is especially useful because the caller doesn't need to generate valid url keys, which involves some escaping of characters, and may involve taking a md5sum of the url if it's too long.
* Avoid accumulating transfer failure log files unless the assistant is being ↵Gravatar Joey Hess2015-05-12
| | | | | | | | | | | | used. Only the assistant uses these, and only the assistant cleans them up, so make only git annex transferkeys write them, There is one behavior change from this. If glacier is being used, and a manual git annex get --from glacier fails because the file isn't available yet, the assistant will no longer later see that failed transfer file and retry the get. Hope no-one depended on that old behavior.
* Improve quvi 0.4 output parsing to handle cases wher there is no known ↵Gravatar Joey Hess2015-05-08
| | | | filename extension. This is currently the case when using quvi with youtube. In this case, the extension ".m" will be used.
* remove unused paramGravatar Joey Hess2015-04-01
|
* addurl --file: When used with a special remote that claims urls and checks ↵Gravatar Joey Hess2015-03-31
| | | | their contents, don't override the user's provided filename with filenames that the special remote suggests. Also, don't allow adding the url if the special remote says it contains multiple files.
* addurl: Added --raw option, which bypasses special handling of quvi, ↵Gravatar Joey Hess2015-03-05
| | | | bittorrent etc urls.
* remove unused paramGravatar Joey Hess2015-01-27
|
* addurl: When a Content-Disposition header suggests a filename to use, addurl ↵Gravatar Joey Hess2015-01-22
| | | | will consider using it, if it's reasonable and doesn't conflict with an existing file. (--file overrides this)
* update my email address and homepage urlGravatar Joey Hess2015-01-21
|
* revert parentDir changeGravatar Joey Hess2015-01-09
| | | | | | | | Reverts 2bba5bc22d049272d3328bfa6c452d3e2e50e86c Unfortunately, this caused breakage on Windows, and possibly elsewhere, because parentDir and takeDirectory do not behave the same when there is a trailing directory separator.
* made parentDir return a Maybe FilePath; removed most uses of itGravatar Joey Hess2015-01-06
| | | | | | | | parentDir is less safe than takeDirectory, especially when working with relative FilePaths. It's really only useful in loops that want to terminate at / This commit was sponsored by Audric SCHILTKNECHT.
* fix addurl behavior when location and url logs are inconsistentGravatar Joey Hess2014-12-29
| | | | | | | | | | | | | | | | | The url log could have an url for a key, while the location log thinks it's not present in the web. In this case, addurl --file url would not do anything. Fixed it to re-add the web as a location. I don't know how this situation could arise, but I saw it in the wild in the conference_proceedings repo, affecting key URL-s17806003--http://mirror.linux.org.au/pub/linux.conf.au/2014/Wednesday/53-Building_Effective_Alliances_around_the_Trans-Pacific_Partnershi-c0505b631127ccc67e38e637344d988e Investigating the presence log, it looked like that key was originally listed as present in the web, then in commit 56abf9e9f3e691ed9d83513037d4019313321ca3 someone else's git-annex set it and some other things to not present in the web. It would be interesting to know what that user did, but I doubt I'll be able to find out. All I can tell from this investigation is that the inconsistency was not introduced when originally addurl-ing the url.
* fix temp uriGravatar Joey Hess2014-12-17
| | | | should be the loguri
* move dummy uuids to Annex.UUIDGravatar Joey Hess2014-12-17
|
* use right uuid when not downloading from web remoteGravatar Joey Hess2014-12-16
|
* truncate each compontent of pathGravatar Joey Hess2014-12-11
|
* truncate over-long pathsGravatar Joey Hess2014-12-11
|
* sanitize filepaths provided by checkUrlGravatar Joey Hess2014-12-11
|
* better wordGravatar Joey Hess2014-12-11
|
* importfeed: Support for downloading urls claimed by remotesGravatar Joey Hess2014-12-11
|
* reorgGravatar Joey Hess2014-12-11
|
* use subdir for addurl when it creates multiple filesGravatar Joey Hess2014-12-11
| | | | The --file parameter specifies the subdir in this mode.
* addurl on an existing file that already has the url should succeed, not failGravatar Joey Hess2014-12-11
|
* Expand checkurl to support recommended filename, and multi-file-urlsGravatar Joey Hess2014-12-11
| | | | This commit was sponsored by an anonymous bitcoiner.
* Revert "let url claims optionally include a suggested filename"Gravatar Joey Hess2014-12-11
| | | | | | This reverts commit bc0bf97b20c48e1d1a35d25e2e76a311c102438c. Putting filename in the claim was a bad idea.
* let url claims optionally include a suggested filenameGravatar Joey Hess2014-12-11
|
* Urls can now be claimed by remotes. This will allow creating, for example, a ↵Gravatar Joey Hess2014-12-08
| | | | external special remote that handles magnet: and *.torrent urls.
* doh't use "def" for command definitions, it conflicts with Data.Default.defGravatar Joey Hess2014-10-14
|
* fix some mixed space+tab indentationGravatar Joey Hess2014-10-09
| | | | | | | | | This fixes all instances of " \t" in the code base. Most common case seems to be after a "where" line; probably vim copied the two space layout of that line. Done as a background task while listening to episode 2 of the Type Theory podcast.
* import metadata from feedsGravatar Joey Hess2014-07-03
| | | | | | | | | | | | | | | | | | | | | | When annex.genmetadata is set, metadata from the feed is added to files that are imported from it. Reused the same feedtitle and itemtitle, feedauthor, itemauthor, etc names that are used in --template. Also added title and author, which are the item title/author if available, falling back to the feed title/author. These are more likely to be common metadata fields. (There is a small bit of dupication here, but once git gets around to packing the object, it will compress it away.) The itempubdate field is not included in the metadata as a string; instead it is used to generate year and month fields, same as is done when adding files with annex.genmetadata set. This commit was sponsored by Amitai Schlair, who cooincidentially is responsible for ikiwiki generating nice feed metadata!
* replace (Key, Backend) with KeyGravatar Joey Hess2014-04-17
| | | | | | | | | | Only fsck and reinject and the test suite used the Backend, and they can look it up as needed from the Key. This simplifies the code and also speeds it up. There is a small behavior change here. Before, all commands would warn when acting on an annexed file with an unknown backend. Now, only fsck and reinject show that warning.
* improve addurl --file behavior in a confusing corner case involving a ftp urlGravatar Joey Hess2014-04-02
|
* add desktop notificationsGravatar Joey Hess2014-03-22
| | | | | | | Motivation: Hook scripts for nautilus or other file managers need to provide the user with feedback that a file is being downloaded. This commit was sponsored by THM Schoemaker.
* Probe for quvi version at run time.Gravatar Joey Hess2014-02-28
| | | | | Overhead: git annex addurl runs quvi --version once. And more bloat to Annex state..
* Put non-object tmp files in .git/annex/misctmp, leaving .git/annex/tmp for ↵Gravatar Joey Hess2014-02-26
| | | | | | | | | | | | | | | | | | | | only partially transferred objects. This allows eg, putting .git/annex/tmp on a ram disk, if the disk IO of temp object files is too annoying (and if you don't want to keep partially transferred objects across reboots). .git/annex/misctmp must be on the same filesystem as the git work tree, since files are moved to there in a way that will not work cross-device, as well as symlinked into there. I first wanted to put the tmp objects in .git/annex/objects/tmp, but that would pose transition problems on upgrade when partially transferred objects existed. git annex info does not currently show the size of .git/annex/misctemp, since it should stay small. It would also be ok to make something clean it out, periodically.
* add UrlOptions sum typeGravatar Joey Hess2014-02-24
|
* Make annex.web-options be used in several places that call curl.Gravatar Joey Hess2014-02-24
|
* reorganize some files and importsGravatar Joey Hess2014-01-26
|
* fix inversion of control in CommandSeek (no behavior changes)Gravatar Joey Hess2014-01-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | I've been disliking how the command seek actions were written for some time, with their inversion of control and ugly workarounds. The last straw to fix it was sync --content, which didn't fit the Annex [CommandStart] interface well at all. I have not yet made it take advantage of the changed interface though. The crucial change, and probably why I didn't do it this way from the beginning, is to make each CommandStart action be run with exceptions caught, and if it fails, increment a failure counter in annex state. So I finally remove the very first code I wrote for git-annex, which was before I had exception handling in the Annex monad, and so ran outside that monad, passing state explicitly as it ran each CommandStart action. This was a real slog from 1 to 5 am. Test suite passes. Memory usage is lower than before, sometimes by a couple of megabytes, and remains constant, even when running in a large repo, and even when repeatedly failing and incrementing the error counter. So no accidental laziness space leaks. Wall clock speed is identical, even in large repos. This commit was sponsored by an anonymous bitcoiner.