From 601ee470af277ad3d0041b6ce425d808075dc997 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Sun, 16 Sep 2012 17:54:12 -0400 Subject: sync: Pushes the git-annex branch to remote/synced/git-annex, rather than directly to remote/git-annex. This fixes a problem I was seeing in the assistant where two remotes would attempt to sync with one another at the same time, and both failed pushing the diverged git-annex branch. Then when both tried to resolve the failed push, they each modified their git-annex branch, which again each blocked the other from pushing into it. The result was that the git-annex branches were perpetually diverged (despite having the same content!) and once the assistant fell into this trap, it couldn't get out and always had to do the slow push/fail/pull/merge/push/fail cycle. --- doc/design/assistant/syncing.mdwn | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) (limited to 'doc/design/assistant/syncing.mdwn') diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn index 85667301d..f874b9932 100644 --- a/doc/design/assistant/syncing.mdwn +++ b/doc/design/assistant/syncing.mdwn @@ -9,8 +9,37 @@ all the other git clones, at both the git level and the key/value level. (in a fresh clone each time) several times in a row, but then stops happening, which has prevented me from debugging it. This could possibly have been caused by the bug fixed in 750c4ac6c282d14d19f79e0711f858367da145e4. -* The git repository syncing sometimes fails due to the remote having updated. - While syncing retries, this sometimes doesn't work. + +* The transfer code doesn't always manage to transfer file contents. + + Besides reconnection events, there are two places where transfers get queued: + + 1. When the committer commits a file, it queues uploads. + 2. When the watcher sees a broken symlink be created, it queues downloads. + + Consider a doubly-linked chain of three repositories, A B and C. + (C and A do not directly communicate.) + + * File is added to A. + * A uploads its content to B. + * At the same time, A git syncs to B. + * Once B gets the git sync, it git syncs to C. + * When C's watcher sees the file appear, it tries to download it. But if + B had not finished receiving the file from A, C doesn't know B has it, + and cannot download it from anywhere. + + Possible solution: After B receives content, it could queue uploads of it + to all remotes that it doesn't know have it yet, which would include C. + + In practice, this has the problem that when C receives the content, + it will queue uploads of it, which can send back to B (or to some other repo + that already has the content) and loop, until the git-annex branches catch + up and break the cycle. + + Possible solution: C could record a download intent. (Similar to a failed + download, but with an unknown source.) When C next receives a git-annex + branch push, it could try to requeue downloads that it has such intents + registered for. ## TODO -- cgit v1.2.3