diff options
author | Joey Hess <joey@kitenet.net> | 2012-08-24 17:40:38 -0400 |
---|---|---|
committer | Joey Hess <joey@kitenet.net> | 2012-08-24 17:40:38 -0400 |
commit | dda61b1b80c42ab91a846537c0cab93a497562a8 (patch) | |
tree | d071de2c8a3c46fb431f74e54c1e876a717e2feb | |
parent | 47875e9b94ed84d4f6ac358e929d7e541c8fcb38 (diff) |
blog for the day
-rw-r--r-- | doc/design/assistant/blog/day_64__syncing_robustly.mdwn | 33 | ||||
-rw-r--r-- | doc/design/assistant/syncing.mdwn | 8 |
2 files changed, 40 insertions, 1 deletions
diff --git a/doc/design/assistant/blog/day_64__syncing_robustly.mdwn b/doc/design/assistant/blog/day_64__syncing_robustly.mdwn new file mode 100644 index 000000000..ab0090b92 --- /dev/null +++ b/doc/design/assistant/blog/day_64__syncing_robustly.mdwn @@ -0,0 +1,33 @@ +Working toward getting the data syncing to happen robustly, +so a bunch of improvements. + +* Got unmount events to be noticed, so unplugging and replugging + a removable drive will resume the syncing to it. There's really no + good unmount event available on dbus in kde, so it uses a heuristic + there. +* Avoid requeuing a download from a remote that no longer has a key. +* Run a full scan on startup, for multiple reasons, including dealing with + crashes. + +Ran into a strange issue: Occasionally the assistant will run `git-annex +copy` and it will not transfer the requested file. It seems that +when the copy command runs `git ls-files`, it does not see the file +it's supposed to act on in its output. + +Eventually I figured out what's going on: When updating the git-annex +branch, it sets `GIT_INDEX_FILE`, and of course environment settings are +not thread-safe! So there's a race between threads that access +the git-annex branch, and the Transferrer thread, or any other thread +that might expect to look at the normal git index. + +Unfortunatly, I don't have a fix for this yet.. Git's only interface for +using a different index file is `GIT_INDEX_FILE`. It seems I have a lot of +code to tear apart, to push back the setenv until after forking every git +command. :( + +Before I figured out the root problem, I developed a workaround for the +symptom I was seeing. I added a `git-annex transferkey`, which is +optimised to be run by the assistant, and avoids running `git ls-files`, so +avoids the problem. While I plan to fix this environment variable problem +properly, `transferkey` turns out to be so much faster than how it was +using `copy` that I'm going to keep it. diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn index 0225a5d39..c3bf3823b 100644 --- a/doc/design/assistant/syncing.mdwn +++ b/doc/design/assistant/syncing.mdwn @@ -3,11 +3,16 @@ all the other git clones, at both the git level and the key/value level. ## immediate action items -* Run transfer scan on startup. * The syncing code currently doesn't run for special remotes. While transfering the git info about special remotes could be a complication, if we assume that's synced between existing git remotes, it should be possible for them to do file transfers to/from special remotes. +* Often several remotes will be queued for full TransferScanner scans, + and the scan does the same thing for each .. so it would be better to + combine them into one scan in such a case. +* Sometimes a Download gets queued from a slow remote, and then a fast + remote becomes available, and a Download is queued from it. Would be + good to sort the transfer queue to run fast Downloads (and Uploads) first. * Ensure that when a remote receives content, and updates its location log, it syncs that update back out. Prerequisite for: * After git sync, identify new content that we don't have that is now available @@ -200,3 +205,4 @@ redone to check it. may get content we were uploading from elsewhere, so check the location log when queuing a failed Upload too.) **done** * Fix MountWatcher to notice umounts and remounts of drives. **done** +* Run transfer scan on startup. **done** |