blog for the day

major milestone today!

2 files changed, 45 insertions, 30 deletions

diff --git a/doc/design/assistant/blog/day_25__transfer_queueing.mdwn b/doc/design/assistant/blog/day_25__transfer_queueing.mdwn
new file mode 100644
index 000000000..35922c0d1
--- /dev/null
+++ b/doc/design/assistant/blog/day_25__transfer_queueing.mdwn
@@ -0,0 +1,41 @@
+So as not to bury the lead, I've been hard at work on my first day in
+Nicaragua, and ** the git-annex assistant fully syncs files (including
+their contents) between remotes now !! **
+
+Details follow..
+
+Made the committer thread queue Upload Transfers when new files
+are added to the annex. Currently it tries to transfer the new content
+to *every* remote; this innefficiency needs to be addressed later.
+
+Made the watcher thread queue Download Transfers when new symlinks
+appear that point to content we don't have. Typically, that will happen
+after an automatic merge from a remote. This needs to be improved as it
+currently adds Transfers from every remote, not just those that have the
+content.
+
+This was the second place that needed an ordered list of remotes
+to talk to. So I cached such a list in the DaemonStatus state info.
+This will also be handy later on, when the webapp is used to add new
+remotes, so the assistant can know about them immediately.
+
+Added YAT (Yet Another Thread), number 15 or so, the transferrer thread
+that waits for transfers to be queued and runs them. Currently a naive
+implementation, it runs one transfer at a time, and does not do anything
+to recover when a transfer fails.
+
+Actually transferring content requires YAT, so that the transfer
+action can run in a copy of the Annex monad, without blocking
+all the assistant's other threads from entering that monad while a transfer
+is running. This is also necessary to allow multiple concurrent transfers
+to run in the future.
+
+This is a very tricky peice of code, because that thread will modify the
+git-annex branch, and its parent thread has to invalidate its cache in
+order to see any changes the child thread made. Hopefully that's the extent
+of the complication of doing this. The only reason this was possible at all
+is that git-annex already support multiple concurrent processes running
+and all making independant changes to the git-annex branch, etc.
+
+After all my groundwork this week, file content transferring is now
+fully working!
diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn
index d4d89d03b..fc2ac3e5e 100644
--- a/doc/design/assistant/syncing.mdwn
+++ b/doc/design/assistant/syncing.mdwn
@@ -21,8 +21,11 @@ all the other git clones, at both the git level and the key/value level.
   Watcher. **done**
 * Write basic Transfer handling thread. Multiple such threads need to be
   able to be run at once. Each will need its own independant copy of the 
-  Annex state monad.
+  Annex state monad. **done**
 * Write transfer control thread, which decides when to launch transfers.
+  **done**
+* Check that download transfer triggering code works (when a symlink appears
+  and the remote does *not* upload to us.
 * At startup, and possibly periodically, look for files we have that
   location tracking indicates remotes do not, and enqueue Uploads for
   them. Also, enqueue Downloads for any files we're missing.
@@ -80,35 +83,6 @@ reachable remote. This is worth doing first, since it's the simplest way to
 get the basic functionality of the assistant to work. And we'll need this
 anyway.
 
-### transfer tracking
-
-Transfer threads started/stopped as necessary to move data.
-(May sometimes want multiple threads downloading, or uploading, or even both.)
-
-	startTransfer :: TransferQueue -> Transfer -> Annex ()
-	startTransfer q transfer = error "TODO"
-
-	stopTransfer :: TransferQueue -> TransferID -> Annex ()
-	stopTransfer q transfer = error "TODO"
-
-The assistant needs to find out when `git-annex-shell` is receiving or
-sending (triggered by another remote), so it can add data for those too.
-This is important to avoid uploading content to a remote that is already
-downloading it from us, or vice versa, as well as to in future let the web
-app manage transfers as user desires. 
-
-For files being received, it can see the temp file, but other than lsof
-there's no good way to find the pid (and I'd rather not kill blindly).
-
-For files being sent, there's no filesystem indication. So git-annex-shell
-(and other git-annex transfer processes) should write a status file to disk.
-
-Can use file locking on these status files to claim upload/download rights,
-which will avoid races.
-
-This status file can also be updated periodically to show amount of transfer
-complete (necessary for tracking uploads).
-
 ## other considerations
 
 It would be nice if, when a USB drive is connected,