add chunk metadata to Key

Added new fields for chunk number, and chunk size. These will not appear
in normal keys ever, but will be used for chunked data stored on special
remotes.

This commit was sponsored by Jouni K Seppanen.

1 files changed, 13 insertions, 9 deletions

diff --git a/doc/design/assistant/chunks.mdwn b/doc/design/assistant/chunks.mdwn
index 42a31bd25..c20bb9aab 100644
--- a/doc/design/assistant/chunks.mdwn
+++ b/doc/design/assistant/chunks.mdwn
@@ -104,7 +104,7 @@ Problem: Does not solve concurrent uploads with different chunk sizes.
 
 When chunking is enabled, always put a chunk number in the Key,
 along with the chunk size.
-So, SHA256-s10000-c1--xxxxxxx for the first chunk of 1 megabyte.
+So, SHA256-1048576-c1--xxxxxxx for the first chunk of 1 megabyte.
 
 Before any chunks are stored, write a chunkcount file, eg
 SHA256-s12345-c0--xxxxxxx. Note that this key is the same as the original
@@ -148,20 +148,24 @@ could lead to data loss. (Same as in design 2.)
 
 # design 4
 
-Use key SHA256-s10000-c1--xxxxxxx for the first chunk of 1 megabyte.
+Use key SHA256-s12345-S1048576-C1--xxxxxxx for the first chunk of 1 megabyte.
+
+Note that keeping the 's'ize field unchanged is necessary because it 
+disambiguates eg, WORM keys. So a 'S'ize field is used to hold the chunk
+size.
 
 Instead of storing the chunk count in the special remote, store it in 
 the git-annex branch. 
 
-Look at git-annex:aaa/bbb/SHA256-s12345--xxxxxxx.log.cnk to get the 
-chunk count and size. File format would be:
+The location log does not record locations of individual chunk keys
+(too space-inneficient).
+Instead, look at git-annex:aaa/bbb/SHA256-s12345--xxxxxxx.log.cnk to get
+the chunk count and size for a key. File format would be:
 
-	ts uuid chunksize chunkcount 0|1
+	ts uuid chunksize chunkcount
 
-Where a trailing 0 means that chunk size is no longer present on the
-remote, and a trailing 1 means it is. For future expansion, any other
-value /= "0" is also accepted, meaning the chunk is present. For example,
-this could be used for [[deltas]], storing the checksums of the chunks.
+Where a chunkcount of 0 means that the object is not longer present in the
+remote using the specified chunk size.
 
 Note that a given remote uuid might have multiple lines, if a key was
 stored on it twice using different chunk sizes. Also note that even when