summaryrefslogtreecommitdiff
path: root/doc/design/assistant/syncing.mdwn
diff options
context:
space:
mode:
authorGravatar Joey Hess <joey@kitenet.net>2012-07-01 20:55:20 -0400
committerGravatar Joey Hess <joey@kitenet.net>2012-07-01 20:55:20 -0400
commit2d2bfe9809f8d8d5862bc12fbe40c2e25b2405a3 (patch)
treed0f8e6d258a93d716f6330b6aacb4229021933a0 /doc/design/assistant/syncing.mdwn
parentc53da2b04a2e802253bfbbfd4e00e02807d6de77 (diff)
reorg
Diffstat (limited to 'doc/design/assistant/syncing.mdwn')
-rw-r--r--doc/design/assistant/syncing.mdwn37
1 files changed, 31 insertions, 6 deletions
diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn
index ce7f9673b..c18badb53 100644
--- a/doc/design/assistant/syncing.mdwn
+++ b/doc/design/assistant/syncing.mdwn
@@ -1,6 +1,37 @@
Once files are added (or removed or moved), need to send those changes to
all the other git clones, at both the git level and the key/value level.
+## action items
+
+* on-disk transfers in progress information files (read/write/enumerate)
+ **done**
+* locking for the files, so redundant transfer races can be detected,
+ and failed transfers noticed **done**
+* transfer info for git-annex-shell (problem: how to add a switch
+ with the necessary info w/o breaking backwards compatability?)
+* update files as transfers proceed. See [[progressbars]]
+ (updating for downloads is easy; for uploads is hard)
+* add Transfer queue TChan
+* enqueue Transfers (Uploads) as new files are added to the annex by
+ Watcher.
+* enqueue Tranferrs (Downloads) as new dangling symlinks are noticed by
+ Watcher.
+* add TransferInfo Map to DaemonStatus for tracking transfers in progress.
+* Poll transfer in progress info files for changes (use inotify again!
+ wow! hammer, meet nail..), and update the TransferInfo Map
+* Write basic Transfer handling thread. Multiple such threads need to be
+ able to be run at once. Each will need its own independant copy of the
+ Annex state monad.
+* Write transfer control thread, which decides when to launch transfers.
+* At startup, and possibly periodically, look for files we have that
+ location tracking indicates remotes do not, and enqueue Uploads for
+ them. Also, enqueue Downloads for any files we're missing.
+* Find a way to probe available outgoing bandwidth, to throttle so
+ we don't bufferbloat the network to death.
+* git-annex needs a simple speed control knob, which can be plumbed
+ through to, at least, rsync. A good job for an hour in an
+ airport somewhere.
+
## git syncing
1. Can use `git annex sync`, which already handles bidirectional syncing.
@@ -55,12 +86,6 @@ anyway.
(May sometimes want multiple threads downloading, or uploading, or even both.)
type TransferQueue = TChan [Transfer]
- data Transfer = Upload Key Remote | Download Key Remote
-
- data TransferID = TransferThread ThreadID | TransferProcess Pid
- type BytesComplete = Integer
- type StartedTime = EpochTime
- data TransferInfo = TransferInfo TransferID StartedTime BytesComplete
-- add (M.Map Transfer TransferInfo) to DaemonStatus
startTransfer :: Transfer -> Annex TransferID