added git-annex remotedaemon

So far, handling connecting to git-annex-shell notifychanges, and
pulling immediately when a change is pushed to a remote.

A little bit buggy (crashes after the first pull), but it already works!

This commit was sponsored by Mark Sheppard.

2 files changed, 68 insertions, 48 deletions

diff --git a/doc/design/git-remote-daemon.mdwn b/doc/design/git-remote-daemon.mdwn
index 0658d4583..db56bd633 100644
--- a/doc/design/git-remote-daemon.mdwn
+++ b/doc/design/git-remote-daemon.mdwn
@@ -37,35 +37,24 @@
 
 # design
 
-Let git-remote-daemon be the name. It runs in a repo and
-either:
+Let git-remote-daemon be the name. Or for git-annex,
+`git annex remotedaemon`.
 
-* forks to background and performs configured actions (ie, `git pull`)
-* with --foreground, communicates over stdio
-  with its caller using a simple protocol (exiting when its caller closes its
-  stdin handle so it will stop when the assistant stops).
+It runs in one of two ways:
 
-It is configured entirely by .git/config.
+1. Forked to background, using a named pipe for the control protocol.
+2. With --foreground, the control protocol goes over stdio.
 
-# encryption & authentication
-
-For simplicity, the network transports have to do their own end-to-end
-encryption. Encryption is not part of this design.
-
-(XMPP does not do end-to-end encryption, but might be supported
-transitionally.)
-
-Ditto for authentication that we're talking to who we indend to talk to.
-Any public key data etc used for authenticion is part of the remote's
-configuration (or hidden away in a secure chmodded file, if neccesary).
-This design does not concern itself with authenticating the remote node,
-it just takes the auth token and uses it.
+Either way, behavior is the same:
 
-For example, in telehash, each node has its own keypair, which is used
-or authentication and encryption, and is all that's needed to route
-messages to that node.
+* Get a list of remotes to act on by looking at .git/config
+* Automatically notices when a remote has changes to branches
+  matching remote.$name.fetch, and pulls them down to the appropriate
+  location.
+* When the control protocol informs it about a new ref that's available,
+  it offers the ref to any interested remotes.
 
-# stdio protocol
+# control protocol
 
 This is an asynchronous protocol. Ie, either side can send any message
 at any time, and the other side does not send a reply.
@@ -82,25 +71,21 @@ the webapp.
 
 * `CONNECTED $remote`
 
-  Send when a connection has been made with a remote.
+  Sent when a connection has been made with a remote.
 
 * `DISCONNECTED $remote`
 
-  Send when connection with a remote has been lost.
-
-* `CHANGED $remote $sha ...`
-
-  This indicates that refs in the named git remote have changed,
-  and indicates the new shas.
+  Sent when connection with a remote has been lost.
 
-* `STATUS $remote $string`
+* `SYNCING $remote`
 
-  A user-visible status message about a named remote.
+  Indicates that a pull or a push with a remote is in progress.
+  Always followed by DONESYNCING.
 
-* `ERROR $remote $string`
+* `DONESYNCING $remote 1|0`
 
-  A user-visible error about a named remote.  
-  (Can continue running past this point, for this or other remotes.)
+  Indicates that syncing with a remote is done, and either succeeded
+  (1) or failed (0).
 
 ## consumed messages
 
@@ -119,21 +104,40 @@ the webapp.
 
   Affects all remotes.
 
-* `PUSH $remote`
+* `CHANGED ref ...`
 
-  Requests that a git push be done with the remote over the network
-  transport when next possible. May be repeated many times before the push
-  finally happens.
+  Indicates that a ref is new or has changed. These can be offered to peers,
+  and peers that are interested in them can pull the content.
 
 * `RELOAD`
 
   Indicates that configs have changed. Daemon should reload .git/config
   and/or restart.
 
-# send-pack and receive-pack
+* `STOP`
 
-Done as the assistant does with XMPP currently. Does not involve
-communication over the above stdio protocol.
+  Shut down git-remote-daemon
+
+  (When using stdio, it also should shutdown when it reaches EOF on 
+  stdin.)
+
+# encryption & authentication
+
+For simplicity, the network transports have to do their own end-to-end
+encryption. Encryption is not part of this design.
+
+(XMPP does not do end-to-end encryption, but might be supported
+transitionally.)
+
+Ditto for authentication that we're talking to who we indend to talk to.
+Any public key data etc used for authenticion is part of the remote's
+configuration (or hidden away in a secure chmodded file, if neccesary).
+This design does not concern itself with authenticating the remote node,
+it just takes the auth token and uses it.
+
+For example, in telehash, each node has its own keypair, which is used
+or authentication and encryption, and is all that's needed to route
+messages to that node.
 
 # network level protocol
 
@@ -143,17 +147,29 @@ This seems to need to be network-layer dependant. Telehash will need
 one design, and git-annex-shell on a central ssh server has a very different
 (and much simpler) design.
 
-## git-annex-shell
+## ssh
+
+`git-annex-shell notifychanges` is run, and speaks a simple protocol
+over stdio to inform when refs on the remote have changed.
+
+No pushing is done for CHANGED, since git handles ssh natively.
 
-Speak a subset of the stdio protocol between git-annex-shell and
-git-remote-daemon, over ssh.
+TODO:
 
-Only thing that seems to be needed is CHANGED, actually!
+* It already detects changes and pulls, but it then dies with a protocol
+  error.
+* Remote system might not be available. Find a smart way to detect it,
+  ideally w/o generating network traffic. One way might be to check
+  if the ssh connection caching control socket exists, for example.
+* Remote system might be available, and connection get lost. Should
+  reconnect, but needs to avoid bad behavior (ie, constant reconnect
+  attempts.)
+* Detect if old system had a too old git-annex-shell and avoid bad behavior
 
 ## telehash 
 
 TODO
 
-## XMPP
+## xmpp
 
 Reuse [[assistant/xmpp]]
diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 987a9ffef..2d43953af 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -922,6 +922,10 @@ subdirectories).
 
   There are several parameters, provided by Haskell's tasty test framework.
 
+* `remotedaemon`
+
+  Detects when remotes have changed and fetches from them.
+
 * `xmppgit`
 
   This command is used internally to perform git pulls over XMPP.