summaryrefslogtreecommitdiff
path: root/Command/Get.hs
Commit message (Collapse)AuthorAge
* reorganize some files and importsGravatar Joey Hess2014-01-26
|
* refactorGravatar Joey Hess2014-01-26
|
* fix transfers of key with no associated fileGravatar Joey Hess2014-01-23
| | | | | | | | | | | | | | | | | Several places assumed this would not happen, and when the AssociatedFile was Nothing, did nothing. As part of this, preferred content checks pass the Key around. Note that checkMatcher is sometimes now called with Just Key and Just File. It currently constructs a FileMatcher, ignoring the Key. However, if it constructed a FileKeyMatcher, which contained both, then it might be possible to speed up parts of Limit, which currently call the somewhat expensive lookupFileKey to get the Key. I have not made this optimisation yet, because I am not sure if the key is always the same. Will need some significant checking to satisfy myself that's the case..
* reorgGravatar Joey Hess2014-01-21
|
* reorganize numcopies code (no behavior changes)Gravatar Joey Hess2014-01-21
| | | | | | | Move stuff into Logs.NumCopies. Add a NumCopies newtype. Better names for various serialization classes that are specific to one thing or another.
* fix inversion of control in CommandSeek (no behavior changes)Gravatar Joey Hess2014-01-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | I've been disliking how the command seek actions were written for some time, with their inversion of control and ugly workarounds. The last straw to fix it was sync --content, which didn't fit the Annex [CommandStart] interface well at all. I have not yet made it take advantage of the changed interface though. The crucial change, and probably why I didn't do it this way from the beginning, is to make each CommandStart action be run with exceptions caught, and if it fails, increment a failure counter in annex state. So I finally remove the very first code I wrote for git-annex, which was before I had exception handling in the Annex monad, and so ran outside that monad, passing state explicitly as it ran each CommandStart action. This was a real slog from 1 to 5 am. Test suite passes. Memory usage is lower than before, sometimes by a couple of megabytes, and remains constant, even when running in a large repo, and even when repeatedly failing and incrementing the error counter. So no accidental laziness space leaks. Wall clock speed is identical, even in large repos. This commit was sponsored by an anonymous bitcoiner.
* sync --content: New option that makes the content of annexed files be ↵Gravatar Joey Hess2014-01-19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | transferred. Similar to the assistant, this honors any configured preferred content expressions. I am not entirely happpy with the implementation. It would be nicer if the seek function returned a list of actions which included the individual file gets and copies and drops, rather than the current list of calls to syncContent. This would allow getting rid of the somewhat reundant display of "sync file [ok|failed]" after the get/put display. But, do that, withFilesInGit would need to somehow be able to construct such a mixed action list. And it would be less efficient than the current implementation, which is able to reuse several values between eg get and drop. Note that currently this does not try to satisfy numcopies when getting/putting files (numcopies are of course checked when dropping files!) This makes it like the assistant, and unlike get --auto and copy --auto, which do duplicate files when numcopies is not yet satisfied. I don't know if this is the right decision; it only seemed to make sense to have this parallel the assistant as far as possible to start with, since I know the assistant works. This commit was sponsored by Øyvind Andersen Holm.
* hlintGravatar Joey Hess2013-09-25
| | | | test suite still passes
* mirror: New command, makes two repositories contain the same set of files.Gravatar Joey Hess2013-08-20
| | | | | | | | | | | | | | This is a simple approach for setting up a mirroring repository. It will work with any type of remotes. Mirror --from is more expensive than mirror --to in general. OTOH, mirror --from will get the file from any remote that has it, not only the named mirror remote. And if the named mirror remote is not the fastest available remote with a file, that can speed things up. It would be possible to make the assistant or watch command do a more dynamic mirroring, that didn't need to scan every time.
* moved AssociatedFile definitionGravatar Joey Hess2013-07-04
|
* --unused: New switch that makes git-annex operate on all data found by the ↵Gravatar Joey Hess2013-07-03
| | | | last run of git annex unused. Supported by fsck, get, move, copy.
* --all for get, move, and copyGravatar Joey Hess2013-07-03
|
* connect existing meters to the transfer log for downloadsGravatar Joey Hess2013-04-11
| | | | | | | | | | | | | | Most remotes have meters in their implementations of retrieveKeyFile already. Simply hooking these up to the transfer log makes that information available. Easy peasy. This is particularly valuable information for encrypted remotes, which otherwise bypass the assistant's polling of temp files, and so don't have good progress bars yet. Still some work to do here (see progressbars.mdwn changes), but this is entirely an improvement from the lack of progress bars for encrypted downloads.
* add section metadata to all commandsGravatar Joey Hess2013-03-24
| | | | Not yet used .. mindless train work.
* two types of byNameGravatar Joey Hess2013-03-05
| | | | | | | | Clean up from 5123a1a83aa3b954fe67629508bab5ccea0e4148. In some cases, looking up a remote by name even though it has no UUID is desirable. This includes git annex sync, which can operate on remotes without an annex, and XMPP pairing, which runs addRemote (with calls byName) before the UUID of the XMPP remote has been configured in git.
* drop: Suggest using git annex move when numcopies prevents dropping a file.Gravatar Joey Hess2013-01-09
|
* --auto fixesGravatar Joey Hess2012-12-06
| | | | | | | * get/copy --auto: Transfer data even if it would exceed numcopies, when preferred content settings want it. * drop --auto: Fix dropping content when there are no preferred content settings.
* where indentationGravatar Joey Hess2012-11-12
|
* generalized Annex.WantedGravatar Joey Hess2012-10-08
| | | | | this should make it easy to use from inside the assistant, where everything is an AssociatedFile.
* make copy --to check preferred content of the remoteGravatar Joey Hess2012-10-08
|
* make the assistant retry failed transfersGravatar Joey Hess2012-09-23
| | | | | | | When a transfer fails, the progress info can be used to intelligently retry it. If the transfer managed to make some progress, but did not fully complete, then there's a good chance that a retry will finish it (or at least make more progress).
* fix transfer log cleanup crashGravatar Joey Hess2012-08-07
| | | | | | | | | | | | | | | Avoid crashing when "git annex get" fails to download from one location, and falls back to downloading from a second location. The problem is that git annex get calls download recursively from within itself if the first download attempt fails. So the first time through, it writes a transfer info file, which is then overwritten on the second, recursive call. Then on cleanup, it tries to delete the file twice, which of course doesn't work. Fixed both by not crashing if the transfer file is removed, and by changing Get to not run download recursively like that. It's the only thing that did so, and it just seems like a bad idea.
* copy, drop: Avoid checking numcopies attribute unnecessarilyGravatar Joey Hess2012-07-10
|
* record transfer information on local git remotesGravatar Joey Hess2012-07-01
| | | | | | | | | | | | | | | In order to record a semi-useful filename associated with the key, this required plumbing the filename all the way through to the remotes' storeKey and retrieveKeyFile. Note that there is potential for deadlock here, narrowly avoided. Suppose the repos are A and B. A sends file foo to B, and at the same time, B gets file foo from A. So, A locks its upload transfer info file, and then locks B's download transfer info file. At the same time, B is taking the two locks in the opposite order. This is only not a deadlock because the lock code does not wait, and aborts. So one of A or B's transfers will be aborted and the other transfer will continue. Whew!
* get, move, copy: Now refuse to do anything when the requested file transfer ↵Gravatar Joey Hess2012-07-01
| | | | | | | | | | | | | | | is already in progress by another process. Note this is per-remote, so trying to get the same file from multiple remotes can still let duplicate downloads run. (And uploading the same file to multiple remotes is not duplicate at all of course.) get, move, and copy are the only git-annex subcommands that transfer files, but there's still git-annex-shell recvkey and sendkey to deal with too. I considered modifying retrieveKeyFile or getViaTmp, but they are called by other code that does not involve expensive file transfers (migrate) or that does file transfers that should not be checked by this (fsck --from).
* hlintGravatar Joey Hess2012-06-12
|
* added ifM and nuked 11 lines of codeGravatar Joey Hess2012-03-14
| | | | no behavior changes
* hlintGravatar Joey Hess2012-02-16
|
* rework git check-attr interfaceGravatar Joey Hess2012-02-13
| | | | | | | | | | | | | | | Now gitattributes are looked up, efficiently, in only the places that really need them, using the same approach used for cat-file. The old CheckAttr code seemed very fragile, in the way it streamed files through git check-attr. I actually found that cad8824852aa0623dc41eac02a9e2bae47d88ec4 was still deadlocking with ghc 7.4, at the end of adding a lot of files. This should fix that problem, and avoid future ones. The best part is that this removes withAttrFilesInGit and withNumCopies, which were complicated Seek methods, as well as simplfying the types for several other Seek methods that had a Backend tupled in.
* fsck --from remote --fastGravatar Joey Hess2012-01-20
| | | | | | | Avoids expensive file transfers, at the expense of checking file size and/or contents. Required some reworking of the remote code.
* add tmp flag parameter to retrieveKeyFileGravatar Joey Hess2012-01-19
|
* tweakGravatar Joey Hess2012-01-06
|
* look up --to and --from remote names only onceGravatar Joey Hess2012-01-06
| | | | This will speed up commands like move and drop.
* more command-specific optionsGravatar Joey Hess2012-01-06
| | | | | | | | | | | | | | | | | | | | | Made --from and --to command-specific options. Added generic storage for values of command-specific options, which allows removing some of the special case fields in AnnexState. (Also added generic storage for command-specific flags, although there are not yet any.) Note that this storage uses a Map, so repeatedly looking up the same value is slightly more expensive than looking up an AnnexState field. But, the value can be looked up once in the seek stage, transformed as necessary, and passed in a closure to the start stage, and this avoids that overhead. Still, I'm hesitant to use this for things like force or fast flags. It's probably best to reserve it for flags that are only used by a few commands, or options like --from and --to that it's important only be allowed to be used with commands that implement them, to avoid user confusion.
* type alias cleanupGravatar Joey Hess2011-12-31
|
* factor out a stopUnlessGravatar Joey Hess2011-12-09
| | | | code melt for lunch
* avoid error message when doing get --from on file not present on remoteGravatar Joey Hess2011-11-18
|
* better limiting of start actions to only run whenAnnexedGravatar Joey Hess2011-11-10
| | | | | Mostly only refactoring, but this does remove one redundant stat of the symlink by copy.
* clean up check selection codeGravatar Joey Hess2011-10-29
| | | | | | | | | This new approach allows filtering out checks from the default set that are not appropriate for a command, rather than having to list every check that is appropriate. It also reduces some boilerplate. Haskell does not define Eq for functions, so I had to go a long way around with each check having a unique id. Meh.
* Fail if --from or --to is passed to commands that do not support them.Gravatar Joey Hess2011-10-27
|
* refactored and generalized pre-command sanity checkingGravatar Joey Hess2011-10-27
|
* renameGravatar Joey Hess2011-10-05
|
* renameGravatar Joey Hess2011-10-04
|
* factor out common importsGravatar Joey Hess2011-10-03
| | | | no code changes
* move annex.numcopies parsing into withNumCopiesGravatar Joey Hess2011-09-15
|
* commentGravatar Joey Hess2011-09-15
|
* clean up params in usage displayGravatar Joey Hess2011-09-15
|
* remove optimize subcommand; use --auto insteadGravatar Joey Hess2011-09-15
| | | | | | | | | | | | | | | | | | get, drop: Added --auto option, which decides whether to get/drop content as needed to work toward the configured numcopies. The problem with bundling it up in optimize was that I then found I wanted to run an optmize that did not drop files, only got them. Considered adding a --only-get switch to it, but that seemed wrong. Instead, let's make existing subcommands optionally smarter. Note that the only actual difference between drop and drop --auto is that the latter does not even try to drop a file if it knows of not enough copies, and does not print any error messages about files it was unable to drop. It might be nice to make get avoid asking git for attributes when not in auto mode. For now it always asks for attributes.
* unify elipsis handlingGravatar Joey Hess2011-07-19
| | | | | And add a simple dots-based progress display, currently only used in v2 upgrade.
* remove unused backend machineryGravatar Joey Hess2011-07-05
| | | | | | | | | | | | | The only remaining vestiage of backends is different types of keys. These are still called "backends", mostly to avoid needing to change user interface and configuration. But everything to do with storing keys in different backends was gone; instead different types of remotes are used. In the refactoring, lots of code was moved out of odd corners like Backend.File, to closer to where it's used, like Command.Drop and Command.Fsck. Quite a lot of dead code was removed. Several data structures became simpler, which may result in better runtime efficiency. There should be no user-visible changes.