roadmap for next year's work

8 files changed, 135 insertions, 25 deletions

diff --git a/doc/design/assistant.mdwn b/doc/design/assistant.mdwn
index ced6b5300..bc5c1b9e5 100644
--- a/doc/design/assistant.mdwn
+++ b/doc/design/assistant.mdwn
@@ -5,6 +5,8 @@ and use cases to add. Feel free to chip in with comments! --[[Joey]]
 
 ## roadmap
 
+kickstarter year
+
 * Month 1 "like dropbox": [[!traillink inotify]] [[!traillink syncing]]
 * Month 2 "shiny webapp": [[!traillink webapp]] [[!traillink progressbars]]
 * Month 3 "easy setup": [[!traillink configurators]] [[!traillink pairing]]
@@ -19,7 +21,19 @@ and use cases to add. Feel free to chip in with comments! --[[Joey]]
 * Month 11: [[!traillink Windows]] porting, finishing touches
 * Month 12: [presentation at SELF2013](http://www.southeastlinuxfest.org/), finishing touches
 
-... Now what? :)
+[sustaining git-annex development](https://campaign.joeyh.name/) year
+
+* Month 1 [[!traillink encrypted_git_remotes]]
+* Month 2 [[!traillink disaster_recovery]]
+* Month 3 [[!traillink xmpp_security]]
+* Month 5 [[!traillink Windows]] assistant and webapp
+* Month 6 improve special remote interface & git-annex enhancement contest
+* Month 7 user-driven features and polishing
+* Month 8 [[!traillink gpgkeys]] [[!traillink sshpassword]]
+* Month 9 [[!traillink Android]] (get it out of beta)
+* Month 10 user-driven features and polishing
+* Month 11 [[!chunks]] [[!traillink deltas]]
+* Month 12 user-driven features and polishing
 
 ## porting
 
@@ -31,7 +45,6 @@ and use cases to add. Feel free to chip in with comments! --[[Joey]]
 
 * [[rate_limiting]]
 * [[partial_content]]
-* [[encrypted_git_remotes]]
 * [[deltas]]
 * [[leftovers]]
 * [[other todo items|todo]]
diff --git a/doc/design/assistant/android.mdwn b/doc/design/assistant/android.mdwn
index 3325d4397..8beaffabc 100644
--- a/doc/design/assistant/android.mdwn
+++ b/doc/design/assistant/android.mdwn
@@ -6,7 +6,7 @@ to improve it.
 * [[bugs/Android_app_permission_denial_on_startup]]
 * S3 doesn't work (at least to Internet Archive: 
   "connect: does not exist (connection refused)")
-* Get app into Goole Play and/or FDroid
+* Get app into Google Play and/or FDroid
 
 ## TODO
 
diff --git a/doc/design/assistant/chunks.mdwn b/doc/design/assistant/chunks.mdwn
new file mode 100644
index 000000000..267e202e2
--- /dev/null
+++ b/doc/design/assistant/chunks.mdwn
@@ -0,0 +1,4 @@
+To avoid leaking even the size of your encrypted files to cloud storage
+providers, add a mode that stores fixed size chunks.
+
+May be a useful starting point for [[deltas]].
diff --git a/doc/design/assistant/disaster_recovery.mdwn b/doc/design/assistant/disaster_recovery.mdwn
new file mode 100644
index 000000000..28dd41c5a
--- /dev/null
+++ b/doc/design/assistant/disaster_recovery.mdwn
@@ -0,0 +1,53 @@
+The assistant should help the user recover their repository when things go
+wrong.
+
+## dangling lock files
+
+There are a few ways a git repository can get broken that are easily fixed.
+One is left over index.lck files. When a commit to a repository fails,
+check that nothing else is using it, fix the problem, and redo the commit.
+
+This should be done on both the current repository and any local
+repositories. Maybe also make git-annex-shell be able to do it remotely?
+
+## incremental fsck
+
+Add webapp UI to enable incremental fsck, and choose when to start and how
+long to run each day.
+
+When fsck finds a damanged file, queue a download of the file from a
+remote. If no accessible remote has the file, prompt the user to eg, connect
+a drive containing it.
+
+## git-annex-shell remote fsck
+
+Would be nice; otherwise remote fsck is too expensive (downloads
+everything) to have the assistant do. (remote fsck --fast might be worth
+having the assistant do)
+
+## git fsck
+
+Have the sanity checker run git fsck periodically (it's fairly inexpensive,
+but still not too often, and should be ioniced and niced). 
+
+If committing to the repository fails, after resolving any dangling lock
+files (see above), it should git fsck.
+
+If git fsck finds problems, launch git repository repair.
+
+## git repository repair
+
+There are several ways git repositories can get damanged. 
+
+The most common is empty files in .git/annex/objects and commits that refer
+to those objects. When the objects have not yet been pushed anywhere.
+I've several times recovered from this manually by
+removing the bad files and resetting to before the commits that referred to
+them. Then re-staging any divergence in the working tree. This could
+perhaps be automated.
+
+As long as the git repository has at least one remote, another method is to
+clone the remote, sync from all other remotes, move over .git/config and
+.git/annex/objects, and tar up the old broken git repo and `git annex add`
+it. This should be automatable and get the user back on their feet. User
+could just click a button and have this be done.
diff --git a/doc/design/assistant/gpgkeys.mdwn b/doc/design/assistant/gpgkeys.mdwn
new file mode 100644
index 000000000..582082f2e
--- /dev/null
+++ b/doc/design/assistant/gpgkeys.mdwn
@@ -0,0 +1,24 @@
+Currently the assistant sets up a shared encryption key, which is checked
+into git, so anyone who gets the repository can decrypt files that are
+stored encrypted on special remotes.
+
+To support using gpg keys in the assistant, we need two things:
+
+1. Help user set up a gpg key if they don't have one. This could be a
+   special-purpose key dedicated to being used by git-annex. It might be
+   nice to leave the user with a securely set up general purpose key,
+   but that would certianly preclude prompting for its password in the
+   webapp. Indeed, the password prompt is the main problem here.
+   Best solution would be to get gpg agent working on all supported 
+   platforms.
+2. Help user learn the gpg keys of people they want to share their repo
+   with, and give them access. If the public key was recorded in the git-annex
+   branch, this could be easily determined when sharing repositories with
+   friends. Or, use MonkeySphere..
+
+-----
+
+Another gpg key security thing is that currently git-annex stores
+crypto creds in memory while it's running. Should use locked memory. See
+<https://github.com/vincenthz/hs-securemem> and
+<https://github.com/vincenthz/hs-securemem/issues/1>
diff --git a/doc/design/assistant/sshpassword.mdwn b/doc/design/assistant/sshpassword.mdwn
new file mode 100644
index 000000000..e38769867
--- /dev/null
+++ b/doc/design/assistant/sshpassword.mdwn
@@ -0,0 +1,12 @@
+Currently the assistant sets up dedicated ssh keys, that can just use
+git-annex. This is ok. The problem is that the initial 2 connections to the
+ssh server when setting up these keys involve a password prompt, which is
+done at the console unless the system happens to have a working ssh agent
+that can pop up a dialog. That can be confusing.
+
+It would be nice to have the webapp prompt for the password. Can it be done
+securely?
+
+This might come down to a simple change to the webapp to prompt for the
+password, and then rather a lot of pain to make the webapp use HTTPS so we
+can be pretty sure noone is sniffing the (localhost) connection.
diff --git a/doc/design/assistant/xmpp.mdwn b/doc/design/assistant/xmpp.mdwn
index 5e5efbaa5..4bdc58874 100644
--- a/doc/design/assistant/xmpp.mdwn
+++ b/doc/design/assistant/xmpp.mdwn
@@ -13,28 +13,6 @@ who share a repository, that is stored in the [[cloud]].
   I have seen a log where a push's packet 1 and 3 arrived, but 2 did not.
   To deal with this, need at least a 1 packet buffer and ACK or resend
   request implemented over top of XMPP. Essentially, TCP over XMPP. :(
-* Add an encryption layer that does not rely on trusting the XMPP server's
-  security (currently it's encrypted only to the server, not end-to-end).
-  There are a few options for how to generate the key for eg,
-  AES encryption:
-  * Do a simple Diffie-Hellman shared key generation when starting each XMPP
-    push session. Would not protect the users from active MITM by the
-    XMPP server, but would prevent passive data gathering attacks from
-    getting useful data. Since the key is ephemeral, would provide
-    Forward Security.
-  * Do D-H key generation, but at pairing, not push time. Allows for an
-    optional confirmation step, using eg, BubbleBabble to compare the
-    keys out of band. ("I see xebeb-dibyb-gycub-kacyb-modib-pudub-sefab-vifuc-bygoc-daguc-gohec-kuxax .. do you too?")
-  * Prompt both users for a passphrase when XMPP pairing, and 
-    use SPEKE (or similar methods like J-PAKE) to generate a shared key.
-    Avoids active MITM attacks. Makes pairing harder, especially pairing
-    between one's own devices, since the passphrase has to be entered on
-    all devices. Also problimatic when pairing more than 2 devices,
-    especially when adding a device to the set later, since there
-    would then be multiple different keys in use.
-  * Rely on the user's gpg key, and do gpg key verification during XMPP
-    pairing. Problimatic because who wants to put their gpg key on their
-    phone? Also, require the users be in the WOT and be gpg literate.
 
 ## design goals
 
diff --git a/doc/design/assistant/xmpp_security.mdwn b/doc/design/assistant/xmpp_security.mdwn
new file mode 100644
index 000000000..6b2b728f2
--- /dev/null
+++ b/doc/design/assistant/xmpp_security.mdwn
@@ -0,0 +1,26 @@
+Currently [[xmpp]] relies on the SSL connection to the server for security.
+The server can see git repository data pushed through it. (Also, the SSL
+connection is not pinned or really checked well at all.)
+
+Add an encryption layer that does not rely on trusting the XMPP server's
+security. There are a few options for how to generate the key for eg,
+AES encryption:
+
+  * Do a simple Diffie-Hellman shared key generation when starting each XMPP
+    push session. Would not protect the users from active MITM by the
+    XMPP server, but would prevent passive data gathering attacks from
+    getting useful data. Since the key is ephemeral, would provide
+    Forward Security.
+  * Do D-H key generation, but at pairing, not push time. Allows for an
+    optional confirmation step, using eg, BubbleBabble to compare the
+    keys out of band. ("I see xebeb-dibyb-gycub-kacyb-modib-pudub-sefab-vifuc-bygoc-daguc-gohec-kuxax .. do you too?")
+  * Prompt both users for a passphrase when XMPP pairing, and 
+    use SPEKE (or similar methods like J-PAKE) to generate a shared key.
+    Avoids active MITM attacks. Makes pairing harder, especially pairing
+    between one's own devices, since the passphrase has to be entered on
+    all devices. Also problimatic when pairing more than 2 devices,
+    especially when adding a device to the set later, since there
+    would then be multiple different keys in use.
+  * Rely on the user's gpg key, and do gpg key verification during XMPP
+    pairing. Problimatic because who wants to put their gpg key on their
+    phone? Also, require the users be in the WOT and be gpg literate.