1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
|
The use of `.git-annex` to store logs means that if a repo has branches
and the user switched between them, git-annex will see different logs in
the different branches, and so may miss info about what remotes have which
files (though it can re-learn).
An alternative would be to store the log data directly in the git repo
as `pristine-tar` does. Problem with that approach is that git won't merge
conflicting changes to log files if they are not in the currently checked
out branch.
It would be possible to use a branch with a tree like this, to avoid
conflicts:
key/uuid/time/status
As long as new files are only added, and old timestamped files deleted,
there would be no conflicts.
A related problem though is the size of the tree objects git needs to
commit. Having the logs in a separate branch doesn't help with that.
As more keys are added, the tree object size will increase, and git will
take longer and longer to commit, and use more space. One way to deal with
this is simply by splitting the logs amoung subdirectories. Git then can
reuse trees for most directories. (Check: Does it still have to build
dup trees in memory?)
Another approach would be to have git-annex *delete* old logs. Keep logs
for the currently available files, or something like that. If other log
info is needed, look back through history to find the first occurance of a
log. Maybe even look at other branches -- so if the logs were on master,
a new empty branch could be made and git-annex would still know where to
get keys in that branch.
Would have to be careful about conflicts when deleting and bringing back
files with the same name. And would need to avoid expensive searching thru
all history to try to find an old log file.
## fleshed out proposal
Let's use one branch per uuid, named git-annex/$UUID.
- I came to realize this would be a good idea when thinking about how
to upgrade. Each individual annex will be upgraded independantly,
so each will want to make a branch, and if the branches aren't distinct,
they will merge conflict for sure.
- TODO: What will need to be done to git to make it push/pull these new
branches?
- A given repo only ever writes to its UUID branch. So no conflicts.
- (BTW, UUIDs probably don't compress well, and this reduces the bloat of having
them repeated lots of times in the tree.)
- Per UUID branches mean that if it wants to find a file's location
amoung configured remotes, it can examine only their branches, if
desired.
In the branch, only one file is needed. Call it locationlog. git-annex
can cache location log changes and write them all to locationlog in
a single git operation on shutdown.
- TODO: what if it's ctrl-c'd with changes pending? Perhaps it should
collect them to ,git/annex/locationlog, and inject that file on shutdown?
- This will be less overhead than the current staging of all the log files.
The log is not appended to, so in git we have a series of commits each of
which replaces the log's entire contens.
To find locations of a key, all (or all relevant) branches need to be
examined, looking backward through the history of each until a log
with a indication of the presense/absense of the key is found.
- This will be less expensive for files that have recently been added
or transfered.
- It could get pretty slow when digging deeper.
- Only 3 places in git-annex will be affected by any slowdown: move --from,
get and drop.
|
