add remote state logs

This allows a remote to store a piece of arbitrary state associated with a
key. This is needed to support Tahoe, where the file-cap is calculated from
the data stored in it, and used to retrieve a key later. Glacier also would
be much improved by using this.

GETSTATE and SETSTATE are added to the external special remote protocol.

Note that the state is left as-is even when a key is removed from a remote.
It's up to the remote to decide when it wants to clear the state.

The remote state log, $KEY.log.rmt, is a UUID-based log. However,
rather than using the old UUID-based log format, I created a new variant
of that format. The new varient is more space efficient (since it lacks the
"timestamp=" hack, and easier to parse (and the parser doesn't mess with
whitespace in the value), and avoids compatability cruft in the old one.

This seemed worth cleaning up for these new files, since there could be a
lot of them, while before UUID-based logs were only used for a few log
files at the top of the git-annex branch. The transition code has also
been updated to handle these new UUID-based logs.

This commit was sponsored by Daniel Hofer.

2 files changed, 27 insertions, 4 deletions

diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 138d9dd18..cac5489d2 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -222,6 +222,17 @@ in control.
   Gets the current preferred content setting of the repository.
   (git-annex replies with VALUE followed by the preferred content
   expression.)
+* `SETSTATE Key Value`  
+  Can be used to store some form of state for a Key. The state stored
+  can be anything this remote needs to store, in any format.
+  It is stored in the git-annex branch. Note that this means that if
+  multiple repositories are using the same special remote, and store
+  different state, whichever one stored the state last will win. Also,
+  it's best to avoid storing much state, since this will bloat the
+  git-annex branch. Most remotes will not need to store any state.
+* `GETSTATE Key`  
+  Gets any state that has been stored for the key.  
+  (git-annex replies with VALUE followed by the state.)
 
 ## general messages
 
diff --git a/doc/internals.mdwn b/doc/internals.mdwn
index 4cc6d3c93..d95ab3f5e 100644
--- a/doc/internals.mdwn
+++ b/doc/internals.mdwn
@@ -39,6 +39,10 @@ are added to git.
 This branch operates on objects exclusively. No file names will ever
 be stored in this branch.
 
+The files stored in this branch are all designed to be auto-merged
+using git's [[union merge driver|git-union-merge]]. So each line
+has a timestamp, to allow the most recent information to be identified.
+
 ### `uuid.log`
 
 Records the UUIDs of known repositories, and associates them with a
@@ -110,7 +114,7 @@ somewhere else.
 ## `aaa/bbb/*.log`
 
 These log files record [[location_tracking]] information
-for file contents. Again these are placed in two levels of subdirectories
+for file contents. These are placed in two levels of subdirectories
 for hashing. See [[hashing]] for details.
 
 The name of the key is the filename, and the content
@@ -122,15 +126,23 @@ Example:
 	1287290776.765152s 1 e605dca6-446a-11e0-8b2a-002170d25c55
 	1287290767.478634s 0 26339d22-446b-11e0-9101-002170d25c55
 
-These files are designed to be auto-merged using git's [[union merge driver|git-union-merge]].
-The timestamps allow the most recent information to be identified.
-
 ## `aaa/bbb/*.log.web`
 
 These log files record urls used by the
 [[web_special_remote|special_remotes/web]]. Their format is similar
 to the location tracking files, but with urls rather than UUIDs.
 
+## `aaa/bbb/*.log.rmt`
+
+These log files are used by remotes that need to record their own state
+about keys. Each remote can store one line of data about a key, in
+its own format.
+
+Example:
+
+	1287290776.765152s e605dca6-446a-11e0-8b2a-002170d25c55 blah blah
+	1287290767.478634s 26339d22-446b-11e0-9101-002170d25c55 foo=bar
+
 ## `schedule.log`
 
 Used to record scheduled events, such as periodic fscks.