summaryrefslogtreecommitdiff
path: root/doc/design/assistant/blog/day_64__syncing_robustly.mdwn
blob: ab0090b92d0f9b05586e5be937055d9f02641b35 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Working toward getting the data syncing to happen robustly,
so a bunch of improvements.

* Got unmount events to be noticed, so unplugging and replugging
  a removable drive will resume the syncing to it. There's really no
  good unmount event available on dbus in kde, so it uses a heuristic
  there.
* Avoid requeuing a download from a remote that no longer has a key.
* Run a full scan on startup, for multiple reasons, including dealing with
  crashes.

Ran into a strange issue: Occasionally the assistant will run `git-annex
copy` and it will not transfer the requested file. It seems that
when the copy command runs `git ls-files`, it does not see the file
it's supposed to act on in its output.

Eventually I figured out what's going on: When updating the git-annex
branch, it sets `GIT_INDEX_FILE`, and of course environment settings are
not thread-safe! So there's a race between threads that access
the git-annex branch, and the Transferrer thread, or any other thread
that might expect to look at the normal git index.

Unfortunatly, I don't have a fix for this yet.. Git's only interface for
using a different index file is `GIT_INDEX_FILE`. It seems I have a lot of
code to tear apart, to push back the setenv until after forking every git
command. :(

Before I figured out the root problem, I developed a workaround for the
symptom I was seeing. I added a `git-annex transferkey`, which is
optimised to be run by the assistant, and avoids running `git ls-files`, so
avoids the problem. While I plan to fix this environment variable problem
properly, `transferkey` turns out to be so much faster than how it was
using `copy` that I'm going to keep it.