summaryrefslogtreecommitdiff
path: root/doc/design
diff options
context:
space:
mode:
authorGravatar Joey Hess <joey@kitenet.net>2014-07-27 12:23:28 -0400
committerGravatar Joey Hess <joey@kitenet.net>2014-07-27 12:23:28 -0400
commitf795ef0bece363d00e50e556d14d75f4731d7fe4 (patch)
tree3acbbb15d6117fa596ac9d5aa19cf5459d6790ac /doc/design
parentb1104f97bdb2decee3ae15e1fc940538e4afb238 (diff)
update
Diffstat (limited to 'doc/design')
-rw-r--r--doc/design/assistant/chunks.mdwn33
1 files changed, 27 insertions, 6 deletions
diff --git a/doc/design/assistant/chunks.mdwn b/doc/design/assistant/chunks.mdwn
index d7517243c..51fd72177 100644
--- a/doc/design/assistant/chunks.mdwn
+++ b/doc/design/assistant/chunks.mdwn
@@ -6,15 +6,17 @@ May be a useful starting point for [[deltas]].
May also allow for downloading different chunks of a file concurrently from
multiple remotes.
-# currently
+Also, can allow resuming of interrupted uploads and downloads.
-Currently, only the webdav and directory special remotes support chunking.
+# legacy chunking
+
+Supported by only the webdav and directory special remotes.
Filenames are used for the chunks that make it easy to see which chunks
belong together, even when encryption is used. There is also a chunkcount
file, that similarly leaks information.
-It is not currently possible to enable chunking on a non-chunked remote.
+It is not possible to enable chunking on a non-chunked remote.
Problem: Two uploads of the same key from repos with different chunk sizes
could lead to data loss. For example, suppose A is 10 mb chunksize, and B
@@ -39,9 +41,9 @@ on in the webapp when configuring an existing remote).
Two concurrent uploaders of the same object to a remote should be safe,
even if they're using different chunk sizes.
-The old chunk method needs to be supported for back-compat, so
-keep the chunksize= setting to enable that mode, and add a new setting
-for the new mode.
+The legacy chunk method needs to be supported for back-compat, so
+keep the chunksize= setting to enable that mode, and add a new chunk=
+setting for the new mode.
# obscuring file sizes
@@ -209,3 +211,22 @@ cannot check exact file sizes.
If padding is enabled, gpg compression should be disabled, to not leak
clues about how well the files compress and so what kind of file it is.
+
+## resuming interupted transfers
+
+Resuming interrupted downloads, and uploads are both possible.
+
+Downloads: If the tmp file for a key exists, round it to the chunk size,
+and skip forward to the next needed chunk. Easy.
+
+Uploads: Check if the 1st chunk is present. If so, check the second chunk,
+etc. Once the first missing chunk is found, start uploading from there.
+
+That adds one extra hasKey call per upload. Probably a win in most cases.
+Can be improved by making special remotes open a persistent
+connection that is used for transferring all chunks, as well as for
+checking hasKey.
+
+Note that this is safe to do only as long as the Key being transferred
+cannot possibly have 2 different contents in different repos. Notably not
+necessarily the case for the URL keys generated for quvi.