summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGravatar Joey Hess <joey@kitenet.net>2012-06-29 11:59:25 -0400
committerGravatar Joey Hess <joey@kitenet.net>2012-06-29 11:59:25 -0400
commite7182ad1191b42d3431f14ced24d0a87ab91495e (patch)
treed03cf2c8638b2725d74b9ee54da01147874bf18c
parent29335bf32685ee665f9ec5acbcfe7f8edabd1b96 (diff)
further design
-rw-r--r--doc/design/assistant/syncing.mdwn40
1 files changed, 40 insertions, 0 deletions
diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn
index 8b681ac10..99474928c 100644
--- a/doc/design/assistant/syncing.mdwn
+++ b/doc/design/assistant/syncing.mdwn
@@ -39,6 +39,46 @@ and with appropriate rate limiting and control facilities.
This probably will need lots of refinements to get working well.
+### first pass: flood syncing
+
+Before mapping the network, the best we can do is flood all files out to every
+reachable remote. This is worth doing first, since it's the simplest way to
+get the basic functionality of the assistant to work. And we'll need this
+anyway.
+
+ data ToTransfer = ToUpload Key | ToDownload Key
+ type ToTransferChan = TChan [ToTransfer]
+
+* ToUpload added by the watcher thread when it adds content.
+* ToDownload added by the watcher thread when it seens new symlinks
+ that lack content.
+
+Transfer threads started/stopped as necessary to move data.
+May sometimes want multiple threads downloading, or uploading, or even both.
+
+ data TransferID = TransferThread ThreadID | TransferProcess Pid
+ data Direction = Uploading | Downloading
+ data Transfer = Transfer Direction Key TransferID EpochTime Integer
+ -- add [Transfer] to DaemonStatus
+
+The assistant needs to find out when `git-annex-shell` is receiving or
+sending (triggered by another remote), so it can add data for those too.
+This is important to avoid uploading content to a remote that is already
+downloading it from us, or vice versa, as well as to in future let the web
+app manage transfers as user desires.
+
+For files being received, it can see the temp file, but other than lsof
+there's no good way to find the pid (and I'd rather not kill blindly).
+
+For files being sent, there's no filesystem indication. So git-annex-shell
+(and other git-annex transfer processes) should write a status file to disk.
+
+Can use file locking on these status files to claim upload/download rights,
+which will avoid races.
+
+This status file can also be updated periodically to show amount of transfer
+complete (necessary for tracking uploads).
+
## other considerations
It would be nice if, when a USB drive is connected,