1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
|
Consider two use cases:
1. Using a v6 repo with locked files on a crippled filesystem not
supporting symlinks. For the files to be usable, they need to be
unlocked. But, the user may not want to unlock the files everywhere,
just on this one crippled system.
2. [[todo/hide_missing_files]]
Both of these could be met by making `git-annex sync` maintain an adjusted
version of the original branch, eg `adjusted/master`.
There would be a adjustment function. For #1 above it would simply convert all
annex symlinks to annex file pointers. For #2 above it would omit files
whose content is not currently in the annex. Sometimes, both #1 and #2 would
be wanted.
[Alternatively, it could stay on the master branch, and only adjust the
work tree and index. See WORKTREE notes below for how this choice would
play out.]
[[!toc]]
## adjusting
master adjusted/master
A
|--------------->A'
| |
When generating commit A', reuse the date of A and use a standard author,
committer, and message. This means that two users with the adjusted branch
checked out and using the same adjustments will get identical shas for A', and
so can collaborate on them.
## commit
When committing changes, a commit is made as usual to the adjusted branch.
So, the user can `git commit` as usual. This does not touch the
original branch yet.
Then we need to get from that commit to one with the adjustments reversed,
which should be the same as if the adjusted branch had not been used.
This commit gets added onto the original branch.
So, the branches would look like this:
master adjusted/master
A
|--------------->A'
| |
| C (new commit)
B < - - - - - - -
|
|--------------->B'
| |
Note particularly that B does not have A' or C in its history;
the adjusted branch is not evident from outside.
Also note that B gets adjusted and the adjusted branch is rebased on top of
it, so C does not remain in the adjusted branch history either. This will
make other checkouts that are in the same adjusted branch end up with the
same B' commit when they pull B.
There may be multiple commits made to the adjusted branch before any get
applied back to the original branch. This is handled by reverse adjusting
commits one at a time and rebasing the others on top.
master adjusted/master
A
|--------------->A'
| |
| C1
| |
| C2
master adjusted/master
A
|--------------->A'
| |
| C1
B1< - - - - - - -
|
|--------------->B1'
| |
| C2'
B2< - - - - - - -
|
|--------------->B2'
[WORKTREE: A pre-commit hook would be needed to update the staged changes,
reversing the adjustment before the commit is made. All the other complications
above are avoided.]
## merge
This would be done by `git annex merge` and `git annex sync`, with the goal
of merging origin/master into master, and updating adjusted/master.
Note that the adjusted files db needs to be updated to reflect the changes
that are merged in, for object add/remove to work as described below.
When merging, there should never be any commits present on the
adjusted/master branch that have not yet been propigated back to the master
branch. If there are any such commits, just propigate them into master
before beginning the merge. There may be staged changes, or changes in the
work tree.
First, merge origin/master into master. This is done in a temp work
tree and with a temp index, so does not affect the checked out adjusted
branch.
(Note that the reason this is done, rather than adjusting origin/master
and merging it into the work tree, is that merge conflicts would be very
common with the naive approach, because the adjusted branch often changes
files, and origin/master may change the same files.)
origin/master master adjusted/master
A------------->A- - - ->A'
| |
B------------->C
While a fast-forward merge is shown here, other merges work the same way.
There may be merge conflicts; if so they're auto-resolved.
Then, adjust merge commit C, and merge that into adjusted/master.
origin/master master adjusted/master
A------------->A- - - ->A'
| | |
B------------->C- - C'->D'
This merge is done in-worktree, so the work tree gets updated.
There may be more merge conflicts here; they're also auto-resolved.
Now, D' is a merge commit, between A' and C'.
To finish, change that commit so it does not have A' as its parent.
This can be accomplished by propigating the reverse-adjusted D'
back to master, and then adjusting master to yield the final
adjusted/master.
origin/master master adjusted/master
A------------->A
| |
B------------->C
|
D - - -> D'
Notice how similar this is to the commit graph. Indeed, "fast-forward"
merging the same B commit from origin/master will lead to an identical
sha for B' as the original committer got!
Since the adjusted/master branch is not present on the remote, if the user
does a `git pull`, it won't merge in changes from origin/master. Which is
good because the adjustment needs to be applied first.
However, if the user does `git merge origin/master`, they'll get into a
state where the adjustment has not been applied. The post-merge hook could be
used to clean up after that. Or, let the user foot-shoot this way; they can
always reset back once they notice the mistake.
[WORKTREE: `git pull` would update the work tree, and may lead to conflicts
between the adjusted work tree and pulled changes. A post-merge hook would
be needed to re-adjust the work tree, and there would be a window where eg,
not present files would appear in the work tree.]
## annex object add/remove
When objects are added/removed from the annex, the associated file has to
be looked up, and the adjustment applied to it. So, dropping a file with the
missing file adjustment would cause it to be removed from the adjusted branch,
and receiving a file's content would cause it to appear in the adjusted
branch. TODO
These changes would need to be committed to the adjusted branch, otherwise
`git diff` would show them.
[WORKTREE: Simply adjust the work tree (and index) per the adjustment.]
## reverse adjusting commits
A user's commits on the adjusted branch have to be reverse adjusted
to get changes to apply to the master branch.
This reversal of one adjustment can be done as just another adjustment.
Since only files touched by the commit will be reverse adjusted, it doesn't
need to reverse all changes made by the original adjustment.
For example, reversing the unlock adjustment might lock the file. Or, it might
do nothing, which would make all committed files remain unlocked.
## push
The new master branch can then be pushed out to remotes. The
adjusted/master branch is not pushed to remotes. `git-annex sync` should
automatically push master when adjusted/master is checked out.
When push.default is "simple" (the new default), running `git push` when in
adjusted/master won't push anything. It would with "matching". Pity. (I
continue to feel git picked the wrong default here.) Users may find that
surprising. Users of `git-annex sync` won't need to worry about it though.
[WORKTREE: push works as usual]
## acting on filtered-out files
If a file is filtered out due to not existing, there should be a way
for `git annex get` to get it. Since the filtered out file is not in the
index, that would not normally work. What to do?
Maybe instead of making a branch where the file is deleted, it would be
better to delete it from the work tree, but keep the branch as-is. Then
`git annex get` would see the file, as it's in the index.
But, not maintaining an adjusted branch complicates other things. See
WORKTREE notes throughout this page. Overall, the WORKTREE approach seems
too problimatic.
Ah, but we know that when adjustment #2 is in place, any file that `git annex
get` could act on is not in the index. So, it could look at the master branch
instead. (Same for `git annex move --from` and `git annex copy --from`)
OTOH, if adjustment #1 is in place and not #2, a file might be renamed in the
index, and `git annex get $newname` should work. So, it should look at the
index in that case.
## problems
Using `git checkout` when in an adjusted branch is problimatic, because a
non-adjusted branch would then be checked out. But, we can just say, if
you want to get into an adjusted branch, you have to run git annex adjust
Or, could make a post-checkout hook. This is would mostly be confusing when
git-annex init switched into the adjusted branch due to lack of symlink
support.
After a commit to an adjusted branch, `git push` won't do anything. The
user has to know to git-annex sync. (Even if a pre-commit hook propigated
the commit back to the master branch, `git push` wouldn't push it with the
default "matching" push strategy.)
Tags are bit of a problem. If the user tags an ajusted branch, the tag
includes the local adjustments.
[WORKTREE: not a problem]
If the user refers to commit shas (in, eg commit messages), those won't be
visible to anyone else.
[WORKTREE: not a problem]
When a pull modifies a file, its content won't be available, and so it
would be hidden temporarily by adjustment #2. So the file would seem to vanish,
and come back later, which could be confusing. Could be fixed as discussed
in [[todo/deferred_update_mode]]. Arguably, it's just as confusing for the
file to remain visible but have its content temporarily replaced with a
annex pointer.
## integration with view branches
Entering a view from an adjusted branch should probably carry the adjusting
over into the creation/updating of the view branch.
Could go a step further, and implement view branches as another branch
adjustment, albeit an extreme one. This might improve view branches.
For example, it's not currently possible to update a view branch with
changes fetched from a remote, and this could get us there.
This would need the reverse adjust to be able to change metadata,
so that a commit that moved files in the view updates their metadata.
[WORKTREE: Wouldn't be able to integrate, unless view branches are changed
into adjusted view worktrees.]
## TODOs
* Interface in webapp to enable adjustments.
* Honor annex.thin when entering an adjusted branch. git checkout
will make copies of the content of annexed files, so this would need
to checkout the adjusted branch some other way. Maybe generalize so this
more efficient checkout is available as a git-annex command?
* sync in adjusted branch runs merge in overlay worktree,
but the merge conflict resolution code does not know to use that
worktree.
* sync in adjusted branch can trigger merge conflict detection where
there should be no conflict.
git init a
cd a
git annex init --version=6
echo hi > f
git annex add f
git annex sync
cd ..
git clone a b
cd b
git annex init --version=6
git annex get
git annex adjust --unlock
cd ..
cd a
git mv f f2
git annex sync
cd ..
cd b
git annex sync
To fix, implement "avoiding conflicted merge" above.
* There are potentially races in code that assumes a branch like
master is not being changed by someone else. In particular,
propigateAdjustedCommits rebases the adjusted branch on top of master.
That is called by sync. The assumption is that any changes in master
have already been handled by updateAdjustedBranch. But, if another remote
pushed a new master at just the right time, the adjusted branch could be
rebased on top of a master that it doesn't incorporate, which is wrong.
|