Merge branch 'master' of ssh://git-annex.branchable.com

2 files changed, 109 insertions, 1 deletions

diff --git a/doc/todo/Bittorrent-like_features.mdwn b/doc/todo/Bittorrent-like_features.mdwn
index cf67d5a3f..fe6f78f6b 100644
--- a/doc/todo/Bittorrent-like_features.mdwn
+++ b/doc/todo/Bittorrent-like_features.mdwn
@@ -36,7 +36,7 @@ Update: there are multiple projects trying to solve this problem space outside o
  * [Maidsafe](http://maidsafe.net/) is another option, which provides storage and uses crypto-currency incentives
  * [Storj](http://storj.io/) is similar
  * [camlistore](https://camlistore.org/) is yet another option
- * [syncthing](https://syncthing.net/) looks like a btsync replacement, and could also be interesting
+ * [syncthing](https://syncthing.net/) looks like a btsync replacement, and could also be interesting, see the [[todo/syncthing_special_remote]] discussion
  * [gittorrent](http://blog.printf.net/articles/2015/05/29/announcing-gittorrent-a-decentralized-github/) allows for decentralised sharing of the git objects, which could replace pairing between repositories, except it [doesn't support push yet](https://github.com/cjb/GitTorrent/issues/3)
  * [gitocalypse](https://github.com/SeekingFor/gitocalypse) is similar to gittorrent, except it uses Freenet and HG (Mercurial?!) instead of the bittorrent DHT
 
diff --git a/doc/todo/syncthing_special_remote.mdwn b/doc/todo/syncthing_special_remote.mdwn
new file mode 100644
index 000000000..7ab7bd93e
--- /dev/null
+++ b/doc/todo/syncthing_special_remote.mdwn
@@ -0,0 +1,108 @@
+Among all possible [[todo/Bittorrent-like_features]] implementations,
+i think [Syncthing][] is one of the most interesting ones.
+
+First off, it is already [packaged for Debian][] with an [ITP
+underway][]. Second, it seems to use a fairly simple protocol, the
+[Block Exchange Protocol][]. It doesn't try to do everything under the
+sun and keeps things simple: NAT transversal, reuse TLS primitives and
+TCP, etc. It also seems to scale pretty well, if we are to believe the
+[usage statistics][].
+
+ [Syncthing]: https://syncthing.net/
+ [packaged for Debian]: http://apt.syncthing.net/
+ [ITP underway]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749887
+ [Block Exchange Protocol]: https://github.com/syncthing/specs/blob/master/BEPv1.md
+ [usage statistics]: https://data.syncthing.net/
+
+It does require the syncthing daemon to be running in order to
+transfer files so it could have similar problems than the
+[[special_remotes/ipfs]] remote which is that files get locally copied
+between the git-annex repository and the special remote.
+
+Furthermore, one of the main problems with this remote is that [public
+shares are not supported][], that is, in order to share with another
+remote, both remotes need to explicitely add each other, in syncthing!
+That makes pairing a little more difficult that it needs to.
+
+ [public shares are not supported]: https://forum.syncthing.net/t/implementing-public-shares/1186
+
+[[!toc levels=2]]
+
+Possible implementations
+========================
+
+I can think of a few different ways of implementing such a remote:
+
+ 1. share the `.git/annex/objects` directory through syncthing
+ 2. copy objects to the `~/Sync` directory (or elsewhere)
+ 3. interoperate with syncthing through the API
+ 4. reimplement the [Block Exchange Protocol][] natively
+
+Sharing the objects
+-------------------
+
+This is the easiest, but maybe the most dangerous: start syncthing and
+expose the `.git/annex/objects` directory to other peers.
+
+This of course has the downside that syncthing could technically start
+destroying objects without git-annex's knowledge, which is really
+bad. Hopefully, the readonly permissions on files could keep that from
+happening, but it still seems pretty unsafe.
+
+There is a way to mark a folder as "master" which makes it ignore
+changes from other nodes, but then that breaks the peer to peer nature
+of the protocol, which is hardly what we want. Marking the repo as
+untrusted would also be an important requirement here.
+
+Copying objects
+---------------
+
+Copying objects is the safest and easiest way to implement this. Add a
+new key? You just copy it to the sync directory. Remove a key? Just
+remove the file, and syncthing picks up the change.
+
+The main problem with this approach is of course the duplication of
+data, doubling the disk usage of all objects stored in the syncthing
+remote locally.
+
+There's also the problem that we do not reflect the fact that the
+git-annex objects are (potentially) in multiple syncthing remotes, and
+thus changing the number of copies. Even worse, once a file is dropped
+on one syncthing remote, it gets dropped everywhere. The solution for
+this of course is simply treat syncthing as a single copy of the
+objects. Note that this also applies to the shared objects method
+above.
+
+This can be easily implemented with the [[special_remotes/directory]]
+special remote:
+
+    git annex initremote syncthing type=directory directory=$HOME/Sync/ encryption=none
+    git annex describe syncthing "default syncthing directory"
+    git annex untrust syncthing
+
+Note that the last step isn't necessary if the syncthing folder is
+marked as "master".
+
+Communicate with the API
+------------------------
+
+Another way would be to talk directly to the [REST API][] (there's
+also a separate [event API][] for GUIs). Currently, this doesn't seem
+to hold much promise because the APIs are mostly read-only and don't
+allow adding objects at all, for example.
+
+ [REST API]: http://docs.syncthing.net/dev/rest.html
+ [event API]: http://docs.syncthing.net/dev/events.html
+
+Reimplement the protocol
+------------------------
+
+This would involve writing a syncthing client using the
+[Block Exchange Protocol][] specification. This would allow more
+complete control over the distribution of objects and so on,
+respecting git-annex's wanted/required content policies while at the
+same time sharing the data with other syncthing endpoints. It would
+also allow for tracking the number of copies of the objects and so on.
+
+Of course, this is a major undertaking and probably the hardest
+approach, but also the one potentially giving the most benefits.