1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
|
[[done]] !!!
The use of `.git-annex` to store logs means that if a repo has branches
and the user switched between them, git-annex will see different logs in
the different branches, and so may miss info about what remotes have which
files (though it can re-learn).
An alternative would be to store the log data directly in the git repo
as `pristine-tar` does. Problem with that approach is that git won't merge
conflicting changes to log files if they are not in the currently checked
out branch.
It would be possible to use a branch with a tree like this, to avoid
conflicts:
key/uuid/time/status
As long as new files are only added, and old timestamped files deleted,
there would be no conflicts.
A related problem though is the size of the tree objects git needs to
commit. Having the logs in a separate branch doesn't help with that.
As more keys are added, the tree object size will increase, and git will
take longer and longer to commit, and use more space. One way to deal with
this is simply by splitting the logs amoung subdirectories. Git then can
reuse trees for most directories. (Check: Does it still have to build
dup trees in memory?)
Another approach would be to have git-annex *delete* old logs. Keep logs
for the currently available files, or something like that. If other log
info is needed, look back through history to find the first occurance of a
log. Maybe even look at other branches -- so if the logs were on master,
a new empty branch could be made and git-annex would still know where to
get keys in that branch.
Would have to be careful about conflicts when deleting and bringing back
files with the same name. And would need to avoid expensive searching thru
all history to try to find an old log file.
## fleshed out proposal
Let's use one branch per uuid, named git-annex/$UUID.
- I came to realize this would be a good idea when thinking about how
to upgrade. Each individual annex will be upgraded independantly,
so each will want to make a branch, and if the branches aren't distinct,
they will merge conflict for sure.
- TODO: What will need to be done to git to make it push/pull these new
branches?
- A given repo only ever writes to its UUID branch. So no conflicts.
- **problem**: git annex move needs to update log info for other repos!
(possibly solvable by having git-annex-shell update the log info
when content is moved using it)
- (BTW, UUIDs probably don't compress well, and this reduces the bloat of having
them repeated lots of times in the tree.)
- Per UUID branches mean that if it wants to find a file's location
amoung configured remotes, it can examine only their branches, if
desired.
- It's important that the per-repo branches propigate beyond immediate
remotes. If there is a central bare repo, that means push --all. Without
one, it means that when repo B pulls from A, and then C pulls from B,
C needs to get A's branch -- which means that B should have a tracking
branch for A's branch.
In the branch, only one file is needed. Call it locationlog. git-annex
can cache location log changes and write them all to locationlog in
a single git operation on shutdown.
- TODO: what if it's ctrl-c'd with changes pending? Perhaps it should
collect them to .git/annex/locationlog, and inject that file on shutdown?
- This will be less overhead than the current staging of all the log files.
The log is not appended to, so in git we have a series of commits each of
which replaces the log's entire contens.
To find locations of a key, all (or all relevant) branches need to be
examined, looking backward through the history of each until a log
with a indication of the presense/absense of the key is found.
- This will be less expensive for files that have recently been added
or transfered.
- It could get pretty slow when digging deeper.
- Only 3 places in git-annex will be affected by any slowdown: move --from,
get and drop. (Update: Now also unused, whereis, fsck)
## alternate
As above, but use a single git-annex branch, and keep the per-UUID
info in their own log files. Hope that git can auto-merge as long as
each observing repo only writes to its own files. (Well, it can, but for
non-fast-forward merges, the git-annex branch would need to be checked out,
which is problimatic.)
Use filenames like:
<observing uuid>/<location uuid>
That allows one repo to record another's state when doing a
`move`.
## outside the box approach
If the problem is limited to only that the `.git-annex/` files make
branching difficult (and not to the related problem that commits to them
and having them in the tree are sorta annoying), then a simple approach
would be to have git-annex look in other branches for location log info
too.
The problem would then be that any locationlog lookup would need to look in
all other branches (any branch could have more current info after all),
which could get expensive.
## way outside the box approach
Another approach I have been mulling over is keeping the log file
branch checked out in .git/annex/logs/ -- this would be a checkout of a git
repository inside a git repository, using "git fake bare" techniques. This
would solve the merge problem, since git auto merge could be used. It would
still mean all the log files are on-disk, which annoys some. It would
require some tighter integration with git, so that after a pull, the log
repo is updated with the data pulled. --[[Joey]]
> Seems I can't use git fake bare exactly. Instead, the best option
> seems to be `git clone --shared` to make a clone that uses
> `.git/annex/logs/.git` to hold its index etc, but (mostly) uses
> objects from the main repo. There would be some bloat,
> as commits to the logs made in there would not be shared with the main
> repo. Using `GIT_OBJECT_DIRECTORY` might be a way to avoid that bloat.
## notes
Another approach could be to use git-notes. It supports merging branches
of notes, with union merge strategy (a hook would have to do this after
a pull, it's not done automatically).
Problem: Notes are usually attached to git
objects, and there are no git objects corresponding to git-annex keys.
Problem: Notes are not normally copied when cloning.
------
## elminating the merge problem
Most of the above options are complicated by the problem of how to merge
changes from remotes. It should be possible to deal with the merge
problem generically. Something like this:
* We have a local branch `B`.
* For remotes, there are also `origin/B`, `otherremote/B`, etc.
* To merge two branches `B` and `foo/B`, construct a merge commit that
makes each file have all lines that were in either version of the file,
with duplicates removed (probably). Do this without checking out a tree.
-- now implemented as git-union-merge
* As a `post-merge` hook, merge `*/B` into `B`. This will ensure `B`
is always up-to-date after a pull from a remote.
* When pushing to a remote, nothing need to be done, except ensure
`B` is either successfully pushed, or the push fails (and a pull needs to
be done to get the remote's changes merged into `B`).
|
