aboutsummaryrefslogtreecommitdiff
path: root/Command/Import.hs
Commit message (Collapse)AuthorAge
* import --reinject-duplicatesGravatar Joey Hess2017-02-09
| | | | | | | | | This is the same as running git annex reinject --known, followed by git-annex import. The advantage to having it in one command is that it only has to hash each file once; the two commands have to hash the imported files a second time. This commit was sponsored by Shane-o on Patreon.
* Make import --deduplicate and --skip-duplicates only hash once, not twiceGravatar Joey Hess2017-02-09
| | | | | | | | | | | | | | | | | | | | import: --deduplicate and --skip-duplicates were implemented inneficiently; they unncessarily hashed each file twice. They have been improved to only hash once. The new approach is to lock down (minimally) and hash files, and then reuse that information when importing them. This was rather tricky, especially in detecting changes to files while they are being imported. The output of import changed slightly. While before it silently skipped over files with eg --skip-duplicates, now it shows each file as it starts to act on it. Since every file is hashed first thing, it would otherwise not be clear what file import is chewing on. (Actually, it wasn't clear before when any of the duplicates switches were used.) This commit was sponsored by Alexander Thompson on Patreon.
* import: Changed how --deduplicate, --skip-duplicates, and --clean-duplicates ↵Gravatar Joey Hess2017-02-07
| | | | | | | | | | | | | | | | | | | | | determine if a file is a duplicate Before, only content known to be present somewhere was considered a duplicate. Now, any content that has been annexed before will be considered a duplicate, even if all annexed copies of the data have been lost. Note that --clean-duplicates and --deduplicate still check numcopies, so won't delete duplicate files unless there's an annexed copy. This makes import use the same method as reinject --known. The man page already said that duplicate meant "its content is either present in the local repository already, or git-annex knows of another repository that contains it, or it was present in the annex before but has been removed now". So, this is really only bringing the implementation into line with the man page. This commit was sponsored by Jochen Bartl on Patreon.
* Avoid backtraces on expected failures when built with ghc 8; only use ↵Gravatar Joey Hess2016-11-15
| | | | | | | | | | | | | backtraces for unexpected errors. ghc 8 added backtraces on uncaught errors. This is great, but git-annex was using error in many places for a error message targeted at the user, in some known problem case. A backtrace only confuses such a message, so omit it. Notably, commands like git annex drop that failed due to eg, numcopies, used to use error, so had a backtrace. This commit was sponsored by Ethan Aubin.
* annex.largefiles can be configured in .gitattributes tooGravatar Joey Hess2016-02-02
| | | | | This is particulary useful for v6 repositories, since the .gitattributes configuration will apply in all clones of the repository.
* remove 163 lines of code without changing anything except importsGravatar Joey Hess2016-01-20
|
* add, import: Support --json output.Gravatar Joey Hess2016-01-19
| | | | Include added key in output.
* addurl, importfeed: Changed to honor annex.largefiles settings, when the ↵Gravatar Joey Hess2015-12-02
| | | | | | | | | content of the url is downloaded. (Not when using --fast or --relaxed.) importfeed just calls addurl functions, so inherits this from it. Note that addurl still generates a temp file, and uses that key to download the file. It just adds it to the work tree at the end when the file is small.
* import: Changed to honor annex.largefiles settings.Gravatar Joey Hess2015-12-02
|
* work around git check-ignore --batch bad exit status bug, and bring back ↵Gravatar Joey Hess2015-11-06
| | | | import -J
* import -J fails at the end, disable util it can be fixedGravatar Joey Hess2015-11-05
|
* import: Avoid very ugly error messages when the directory files are imported ↵Gravatar Joey Hess2015-11-05
| | | | to is not a directort, but perhaps an annexed file.
* -J for add/addurl/importGravatar Joey Hess2015-11-05
|
* fix local dropping to not require extra locking of copies, but only that the ↵Gravatar Joey Hess2015-10-09
| | | | local copy be locked for removal
* rename constructorGravatar Joey Hess2015-10-09
|
* verify local copy of content with lockingGravatar Joey Hess2015-10-09
|
* improve drop proof codeGravatar Joey Hess2015-10-09
|
* support invalidating existing VerifiedCopysGravatar Joey Hess2015-10-08
|
* add VerifiedCopy data typeGravatar Joey Hess2015-10-08
| | | | | | | | | There should be no behavior changes in this commit, it just adds a more expressive data type and adjusts code that had been passing around a [UUID] or sometimes a Maybe Remote to instead use [VerifiedCopy]. Although, since some functions were taking two different [UUID] lists, there's some potential for me to have gotten it horribly wrong.
* convert ImportGravatar Joey Hess2015-07-13
|
* finished converting all the main optionsGravatar Joey Hess2015-07-10
|
* convert all commands to work with optparse-applicativeGravatar Joey Hess2015-07-08
| | | | Still no options though.
* started converting to use optparse-applicativeGravatar Joey Hess2015-07-08
| | | | | | | | | | | | This is a work in progress. It compiles and is able to do basic command dispatch, including git autocorrection, while using optparse-applicative for the core commandline parsing. * Many commands are temporarily disabled before conversion. * Options are not wired in yet. * cmdnorepo actions don't work yet. Also, removed the [Command] list, which was only used in one place.
* import --clean-duplicates: Fix bug that didn't count local or trusted repo's ↵Gravatar Joey Hess2015-06-03
| | | | copy of a file as one of the necessary copies to allow removing it from the import location.
* import: Refuse to import files that are within the work tree, as that does ↵Gravatar Joey Hess2015-05-11
| | | | not make sense and could cause data loss.
* improve messagesGravatar Joey Hess2015-04-30
|
* import: Before removing a duplicate file in --deduplicate or ↵Gravatar Joey Hess2015-04-30
| | | | --clean-duplicates mode, verify that enough copies of its content still exist.
* import: Don't stop entire import when one file fails due to being gitignored ↵Gravatar Joey Hess2015-04-29
| | | | or conflicting with something in the work tree.
* import: Check for gitignored files before moving them into the tree. (Needs ↵Gravatar Joey Hess2015-04-29
| | | | git 1.8.4 or newer.)
* Steer towards deduplicationGravatar Jean Jordaan2015-04-03
|
* import: --deduplicate and --cleanduplicates now output the keys ↵Gravatar Joey Hess2015-03-31
| | | | corresponding to duplicated files they process.
* better option handlingGravatar Joey Hess2015-02-08
| | | | At least it avoids the big truth table lookup
* import: Avoid checksumming file twice when run in the default or --duplicate ↵Gravatar Joey Hess2015-02-08
| | | | | | | | | mode. --deduplicate, --skip-duplicates, and --clean-duplicates still checksum the file twice, the first time to determine if it's a duplicate. This cannot be easily merged with the checksumming done to add the file, since the file needs to be locked down before that second checksum is taken.
* The file matching options are now only accepted by commands that can ↵Gravatar Joey Hess2015-02-06
| | | | actually use them.
* update my email address and homepage urlGravatar Joey Hess2015-01-21
|
* revert parentDir changeGravatar Joey Hess2015-01-09
| | | | | | | | Reverts 2bba5bc22d049272d3328bfa6c452d3e2e50e86c Unfortunately, this caused breakage on Windows, and possibly elsewhere, because parentDir and takeDirectory do not behave the same when there is a trailing directory separator.
* made parentDir return a Maybe FilePath; removed most uses of itGravatar Joey Hess2015-01-06
| | | | | | | | parentDir is less safe than takeDirectory, especially when working with relative FilePaths. It's really only useful in loops that want to terminate at / This commit was sponsored by Audric SCHILTKNECHT.
* doh't use "def" for command definitions, it conflicts with Data.Default.defGravatar Joey Hess2014-10-14
|
* hlintGravatar Joey Hess2014-10-09
|
* fix some mixed space+tab indentationGravatar Joey Hess2014-10-09
| | | | | | | | | This fixes all instances of " \t" in the code base. Most common case seems to be after a "where" line; probably vim copied the two space layout of that line. Done as a background task while listening to episode 2 of the Type Theory podcast.
* Do not preserve permissions and acls when copying files from one local git ↵Gravatar Joey Hess2014-08-26
| | | | | | | | | | | | | | | | | repository to another. Timestamps are still preserved as long as cp --preserve=timestamps is supported. This avoids cp -a overriding the default mode acls that the user might have set in a git repository. With GNU cp, this behavior change should not be a breaking change, because git-anex also uses rsync sometimes in the same situation, and has only ever preserved timestamps when using rsync. Systems without GNU cp will no longer use cp -a, but instead just cp. So, timestamps will no longer be preserved. Preserving timestamps when copying between repos is not guaranteed anyway. Closes: #729757
* Windows: Avoid using unix-compat's rename, which refuses to rename directories.Gravatar Joey Hess2014-01-29
| | | | Opened a bug about this: https://github.com/jystic/unix-compat/issues/10
* reorganize some files and importsGravatar Joey Hess2014-01-26
|
* fix inversion of control in CommandSeek (no behavior changes)Gravatar Joey Hess2014-01-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | I've been disliking how the command seek actions were written for some time, with their inversion of control and ugly workarounds. The last straw to fix it was sync --content, which didn't fit the Annex [CommandStart] interface well at all. I have not yet made it take advantage of the changed interface though. The crucial change, and probably why I didn't do it this way from the beginning, is to make each CommandStart action be run with exceptions caught, and if it fails, increment a failure counter in annex state. So I finally remove the very first code I wrote for git-annex, which was before I had exception handling in the Annex monad, and so ran outside that monad, passing state explicitly as it ran each CommandStart action. This was a real slog from 1 to 5 am. Test suite passes. Memory usage is lower than before, sometimes by a couple of megabytes, and remains constant, even when running in a large repo, and even when repeatedly failing and incrementing the error counter. So no accidental laziness space leaks. Wall clock speed is identical, even in large repos. This commit was sponsored by an anonymous bitcoiner.
* better handling of overwriting an existing file/directory/broken link when ↵Gravatar Joey Hess2013-12-09
| | | | | | | | | importing Previous test did not notice if there is a dangling symlink. Also, if a directory exists with the same name as the imported file, that cannot work, so don't let --force have an effect.
* import: Add --skip-duplicates option.Gravatar Joey Hess2013-12-04
| | | | | | | Note that the hash backends were made to stop printing a (checksum..) message as part of this, since it showed up without a file when deciding whether to act on a file. Should have probably removed that message a while ago anyway, I suppose.
* implement import --deduplicate and import --clean-duplicatesGravatar Joey Hess2013-08-20
| | | | | | | | Note that --deduplicate currently checksums each file twice, once to see if it's a known key, and once when importing it. Perhaps this could be revisited and the extra checksum gotten rid of, at the cost of not locking down the file when adding it.
* implement import --duplicateGravatar Joey Hess2013-08-11
| | | | | The other two options are harder, due to needing to get the key for a file before adding it.
* Support import in direct mode.Gravatar Joey Hess2013-07-22
|
* fix permission damage (thanks, Windows)Gravatar Joey Hess2013-05-11
|