diff options
| -rw-r--r-- | doc/design/assistant/blog/day_64__syncing_robustly.mdwn | 33 | ||||
| -rw-r--r-- | doc/design/assistant/syncing.mdwn | 23 |
2 files changed, 50 insertions, 6 deletions
diff --git a/doc/design/assistant/blog/day_64__syncing_robustly.mdwn b/doc/design/assistant/blog/day_64__syncing_robustly.mdwn new file mode 100644 index 000000000..ab0090b92 --- /dev/null +++ b/doc/design/assistant/blog/day_64__syncing_robustly.mdwn @@ -0,0 +1,33 @@ +Working toward getting the data syncing to happen robustly, +so a bunch of improvements. + +* Got unmount events to be noticed, so unplugging and replugging + a removable drive will resume the syncing to it. There's really no + good unmount event available on dbus in kde, so it uses a heuristic + there. +* Avoid requeuing a download from a remote that no longer has a key. +* Run a full scan on startup, for multiple reasons, including dealing with + crashes. + +Ran into a strange issue: Occasionally the assistant will run `git-annex +copy` and it will not transfer the requested file. It seems that +when the copy command runs `git ls-files`, it does not see the file +it's supposed to act on in its output. + +Eventually I figured out what's going on: When updating the git-annex +branch, it sets `GIT_INDEX_FILE`, and of course environment settings are +not thread-safe! So there's a race between threads that access +the git-annex branch, and the Transferrer thread, or any other thread +that might expect to look at the normal git index. + +Unfortunatly, I don't have a fix for this yet.. Git's only interface for +using a different index file is `GIT_INDEX_FILE`. It seems I have a lot of +code to tear apart, to push back the setenv until after forking every git +command. :( + +Before I figured out the root problem, I developed a workaround for the +symptom I was seeing. I added a `git-annex transferkey`, which is +optimised to be run by the assistant, and avoids running `git ls-files`, so +avoids the problem. While I plan to fix this environment variable problem +properly, `transferkey` turns out to be so much faster than how it was +using `copy` that I'm going to keep it. diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn index 071ea2730..c3bf3823b 100644 --- a/doc/design/assistant/syncing.mdwn +++ b/doc/design/assistant/syncing.mdwn @@ -3,12 +3,16 @@ all the other git clones, at both the git level and the key/value level. ## immediate action items -* Fix MountWatcher to notice umounts and remounts of drives. -* A remote may lose content it had before, so when requeuing - a failed download, check the location log to see if the remote still has - the content, and if not, queue a download from elsewhere. (And, a remote - may get content we were uploading from elsewhere, so check the location - log when queuing a failed Upload too.) +* The syncing code currently doesn't run for special remotes. While + transfering the git info about special remotes could be a complication, + if we assume that's synced between existing git remotes, it should be + possible for them to do file transfers to/from special remotes. +* Often several remotes will be queued for full TransferScanner scans, + and the scan does the same thing for each .. so it would be better to + combine them into one scan in such a case. +* Sometimes a Download gets queued from a slow remote, and then a fast + remote becomes available, and a Download is queued from it. Would be + good to sort the transfer queue to run fast Downloads (and Uploads) first. * Ensure that when a remote receives content, and updates its location log, it syncs that update back out. Prerequisite for: * After git sync, identify new content that we don't have that is now available @@ -195,3 +199,10 @@ redone to check it. order to retry them later, without the scan. I'm thinking about a directory like `.git/annex/transfer/failed/{upload,download}/uuid/`, which failed transfer log files could be moved to. +* A remote may lose content it had before, so when requeuing + a failed download, check the location log to see if the remote still has + the content, and if not, queue a download from elsewhere. (And, a remote + may get content we were uploading from elsewhere, so check the location + log when queuing a failed Upload too.) **done** +* Fix MountWatcher to notice umounts and remounts of drives. **done** +* Run transfer scan on startup. **done** |