diff options
| author |  Joey Hess <joeyh@joeyh.name> | 2017-09-04 14:33:09 -0400 |
|---|---|---|
| committer | Joey Hess <joeyh@joeyh.name> | 2017-09-04 14:39:32 -0400 |
| commit | 25ed1e54abcc25f729fed016ec77a8cd049142fa (patch) | |
| tree | f6130202fc2f23a6b641f8646705c900b757a012 /doc | |
| parent | 3db6fd7621d3fa57bebfea216019e522f6d8d781 (diff) | |
use export db to correctly handle duplicate files
Removed uncorrect UniqueKey key in db schema; a key can appear multiple
times with different files.
The database has to be flushed after each removal. But when adding files
to the export, lots of changes are able to be queued up w/o flushing.
So it's still fairly efficient.
If large removals of files from exports are too slow, an alternative
would be to make two passes over the diff, one pass queueing deletions
from the database, then a flush and the a second pass updating the
location log. But that would use more memory, and need to look up
exportKey twice per removed file, so I've avoided such optimisation yet.
This commit was supported by the NSF-funded DataLad project.
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/design/exporting_trees_to_special_remotes.mdwn | 3 | ||||
| -rw-r--r-- | doc/todo/export.mdwn | 11 |
2 files changed, 2 insertions, 12 deletions
diff --git a/doc/design/exporting_trees_to_special_remotes.mdwn b/doc/design/exporting_trees_to_special_remotes.mdwn index 118f12978..7ff1df870 100644 --- a/doc/design/exporting_trees_to_special_remotes.mdwn +++ b/doc/design/exporting_trees_to_special_remotes.mdwn @@ -147,7 +147,8 @@ remotes, don't let it be turned off. The same file contents may be in a treeish multiple times under different filenames. That complicates using location tracking. One file may have been exported and the other not, and location tracking says that the content -is present in the export. +is present in the export. A sqlite database is needed to keep track of +this. ## recording exported filenames in git-annex branch diff --git a/doc/todo/export.mdwn b/doc/todo/export.mdwn index 828e1c55b..99877423b 100644 --- a/doc/todo/export.mdwn +++ b/doc/todo/export.mdwn @@ -18,17 +18,6 @@ there need to be a new interface in supported remotes? Work is in progress. Todo list: * Use retrieveExport when getting from export remotes. - (Needs a map from key to ExportLocation) * Efficient handling of renames. -* If the same content is present in two different files, export - location tracking can be messed up. - - When one of the files is deleted and - that tree is exported, the location log for the key will be updated - to say it's not present, even though the other file is still present. - - And, once one of the files is uploaded, the location log will - say the content is present, so the pass over the tree won't try to - upload the other file. (See design for a fix for this.) * Support export to aditional special remotes (S3 etc) * Support export to external special remotes. |