diff options
author | Joey Hess <joey@kitenet.net> | 2012-08-23 16:24:22 -0400 |
---|---|---|
committer | Joey Hess <joey@kitenet.net> | 2012-08-23 16:24:22 -0400 |
commit | d5d4b8db345b3e4a81b2ad2d7a4f2e5f1e30d519 (patch) | |
tree | 104066175eeb67de1409ae5798fa0c4f9567d1fd /doc/design | |
parent | dfb67090644af0024abcefe885744df9a46d094d (diff) |
update
Diffstat (limited to 'doc/design')
-rw-r--r-- | doc/design/assistant/syncing.mdwn | 79 |
1 files changed, 39 insertions, 40 deletions
diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn index 83c5e9d22..071ea2730 100644 --- a/doc/design/assistant/syncing.mdwn +++ b/doc/design/assistant/syncing.mdwn @@ -3,42 +3,12 @@ all the other git clones, at both the git level and the key/value level. ## immediate action items -* Optimisations in 5c3e14649ee7c404f86a1b82b648d896762cbbc2 temporarily - broke content syncing in some situations, which need to be added back. - - Now syncing a disconnected remote only starts a transfer scan if the - remote's git-annex branch has diverged, which indicates it probably has - new files. But that leaves open the cases where the local repo has - new files; and where the two repos git branches are in sync, but the - content transfers are lagging behind; and where the transfer scan has - never been run. - - Need to track locally whether we're believed to be in sync with a remote. - This includes: - * All local content has been transferred to it successfully. - * The remote has been scanned once for data to transfer from it, and all - transfers initiated by that scan succeeded. - - Note the complication that, if it's initiated a transfer, our queued - transfer will be thrown out as unnecessary. But if its transfer then - fails, that needs to be noticed. - - If we're going to track failed transfers, we could just set a flag, - and use that flag later to initiate a new transfer scan. We need a flag - in any case, to ensure that a transfer scan is run for each new remote. - The flag could be `.git/annex/transfer/scanned/uuid`. - - But, if failed transfers are tracked, we could also record them, in - order to retry them later, without the scan. I'm thinking about a - directory like `.git/annex/transfer/failed/{upload,download}/uuid/`, - which failed transfer log files could be moved to. - - Note that a remote may lose content it had before, so when requeuing - a failed download, should check the location log to see if it still has +* Fix MountWatcher to notice umounts and remounts of drives. +* A remote may lose content it had before, so when requeuing + a failed download, check the location log to see if the remote still has the content, and if not, queue a download from elsewhere. (And, a remote may get content we were uploading from elsewhere, so check the location log when queuing a failed Upload too.) - * Ensure that when a remote receives content, and updates its location log, it syncs that update back out. Prerequisite for: * After git sync, identify new content that we don't have that is now available @@ -67,18 +37,17 @@ all the other git clones, at both the git level and the key/value level. files in some directories and not others. See for use cases: [[forum/Wishlist:_options_for_syncing_meta-data_and_data]] * speed up git syncing by using the cached ssh connection for it too - (will need to use `GIT_SSH`, which needs to point to a command to run, - not a shell command line) + Will need to use `GIT_SSH`, which needs to point to a command to run, + not a shell command line. Beware that the network connection may have + bounced and the cached ssh connection not be usable. * Map the network of git repos, and use that map to calculate optimal transfers to keep the data in sync. Currently a naive flood fill is done instead. * Find a more efficient way for the TransferScanner to find the transfers that need to be done to sync with a remote. Currently it walks the git - working copy and checks each file. - -## misc todo - -* --debug will show often unnecessary work being done. Optimise. + working copy and checks each file. That probably needs to be done once, + but further calls to the TransferScanner could eg, look at the delta + between the last scan and the current one in the git-annex branch. ## data syncing @@ -196,3 +165,33 @@ redone to check it. drives are mounted. **done** * It would be nice if, when a USB drive is connected, syncing starts automatically. Use dbus on Linux? **done** +* Optimisations in 5c3e14649ee7c404f86a1b82b648d896762cbbc2 temporarily + broke content syncing in some situations, which need to be added back. + **done** + + Now syncing a disconnected remote only starts a transfer scan if the + remote's git-annex branch has diverged, which indicates it probably has + new files. But that leaves open the cases where the local repo has + new files; and where the two repos git branches are in sync, but the + content transfers are lagging behind; and where the transfer scan has + never been run. + + Need to track locally whether we're believed to be in sync with a remote. + This includes: + * All local content has been transferred to it successfully. + * The remote has been scanned once for data to transfer from it, and all + transfers initiated by that scan succeeded. + + Note the complication that, if it's initiated a transfer, our queued + transfer will be thrown out as unnecessary. But if its transfer then + fails, that needs to be noticed. + + If we're going to track failed transfers, we could just set a flag, + and use that flag later to initiate a new transfer scan. We need a flag + in any case, to ensure that a transfer scan is run for each new remote. + The flag could be `.git/annex/transfer/scanned/uuid`. + + But, if failed transfers are tracked, we could also record them, in + order to retry them later, without the scan. I'm thinking about a + directory like `.git/annex/transfer/failed/{upload,download}/uuid/`, + which failed transfer log files could be moved to. |