experimental ipfs special remote, with addurl support

3 files changed, 212 insertions, 0 deletions

diff --git a/doc/special_remotes.mdwn b/doc/special_remotes.mdwn
index e6c1e6da5..079e9d576 100644
--- a/doc/special_remotes.mdwn
+++ b/doc/special_remotes.mdwn
@@ -42,6 +42,7 @@ for using git-annex with various services:
 * [chef-vault](https://github.com/3ofcoins/knife-annex/)
 * [hubiC](https://github.com/Schnouki/git-annex-remote-hubic)
 * [pCloud](https://github.com/tochev/git-annex-remote-pcloud)
+* [[ipfs]]
 
 Want to add support for something else? [[Write your own!|external]]
 
diff --git a/doc/special_remotes/external/git-annex-remote-ipfs b/doc/special_remotes/external/git-annex-remote-ipfs
new file mode 100755
index 000000000..fc91aefc8
--- /dev/null
+++ b/doc/special_remotes/external/git-annex-remote-ipfs
@@ -0,0 +1,125 @@
+#!/bin/sh
+# This is a git-annex external special remote program,
+# which adds experimental ipfs support to git-annex.
+#
+# Install in PATH as git-annex-remote-ipfs
+#
+# Copyright 2015 Joey Hess; licenced under the GNU GPL version 3 or higher.
+
+set -e
+
+# use ipfs: as a prefix to indicate when an "url" is really stored in ipfs
+isipfsurl () {
+	echo "$1" | egrep -q "^ipfs:"
+}
+
+# convert an ipfs: url to an address that the ipfs client understands
+urltoaddress () {
+	echo "$1" | sed -e 's/^ipfs://'
+}
+
+addresstourl () {
+	echo "ipfs:$1"
+}
+
+# Gets a VALUE response and stores it in $RET
+getvalue () {
+	read resp
+	# Tricky POSIX shell code to split first word of the resp,
+	# preserving all other whitespace
+	case "${resp%% *}" in
+		VALUE)
+			RET="$(echo "$resp" | sed 's/^VALUE \?//')"
+		;;
+		*)
+		RET=""
+		;;
+	esac
+}
+
+# Get a list of all known ipfs addresses for a key,
+# storing it in a temp file.
+getaddrs () {
+        key="$1"
+        tmp="$2"
+
+        echo GETURLS "$key"
+        getvalue
+        while [ -n "$RET" ]; do
+                if isipfsurl "$RET"; then
+                        echo "$RET" >> "$tmp"
+                fi
+                getvalue
+        done
+}
+
+# This has to come first, to get the protocol started.
+echo VERSION 1
+
+while read line; do
+	set -- $line
+	case "$1" in
+		INITREMOTE)
+			echo INITREMOTE-SUCCESS
+		;;
+		PREPARE)
+			echo PREPARE-SUCCESS
+		;;
+		CLAIMURL)
+			url="$2"
+			if isipfsurl "$url"; then
+				echo CLAIMURL-SUCCESS
+			else
+				echo CLAIMURL-FAILURE
+			fi
+		;;
+		CHECKURL)
+			url="$2"
+			# TODO if size of file can be quickly determined
+			# (without downloading it) return the size
+			# instead of UNKNOWN
+			echo CHECKURL-CONTENTS UNKNOWN "$(urltoaddress "$url")"
+		;;
+		TRANSFER)
+			key="$3"
+			file="$4"
+			case "$2" in
+				STORE)
+					addr=$(ipfs add -q "$file" </dev/null) || true
+					if [ -z "$addr" ]; then
+						echo TRANSFER-FAILURE STORE "$key" "ipfs add failed"
+					else
+						echo "SETURLPRESENT" "$key" "$(addresstourl "$addr")"
+						echo TRANSFER-SUCCESS STORE "$key"
+					fi
+				;;
+				RETRIEVE)
+					addrtmp=$(mktemp)
+					getaddrs "$key" "$addrtmp"
+					addr="$(urltoaddress "$(head "$addrtmp")")" || true
+					rm -f "$addrtmp"
+					if [ -z "$addr" ]; then
+						echo TRANSFER-FAILURE RETRIEVE "$key" "no known ipfs address for this key"
+					else
+						if ! ipfs get --output="$file" "$addr" >&2 </dev/null; then
+							echo TRANSFER-FAILURE RETRIEVE "$key" "failed downloading ipfs $addr"
+						else
+							echo TRANSFER-SUCCESS RETRIEVE "$key"
+						fi
+					fi
+				;;
+			esac
+		;;
+		CHECKPRESENT)
+			key="$2"
+			echo CHECKPRESENT-FAILURE "$key"
+		;;
+		REMOVE)
+			key="$2"
+			echo REMOVE-FAILURE "$key" "cannot remove content from ipfs (instead, run ipfs gc to clear your local ipfs cache)"
+		;;
+		*)
+			echo UNSUPPORTED-REQUEST
+		;;
+	esac	
+done
diff --git a/doc/special_remotes/ipfs.mdwn b/doc/special_remotes/ipfs.mdwn
new file mode 100644
index 000000000..61f887c8f
--- /dev/null
+++ b/doc/special_remotes/ipfs.mdwn
@@ -0,0 +1,86 @@
+This special remote stores file contents in [ipfs](http://ipfs.io/).
+
+Warning: As this page is being written, ipfs is still considered alpha
+quality code, not suitable for production use. Still, it's fun to play
+with, has some nice features and great potential, and git-annex can
+keep your data safe while you're using ipfs.
+
+## prerequisites
+
+* git-annex version 5.20141219 or newer, which has [[external]] special remote
+  support.
+* Install [[external/git-annex-remote-ipfs]] somewhere in PATH
+  and `chmod +x` the script.
+* Install [go-ipfs](https://github.com/jbenet/go-ipfs) somewhere in PATH.
+* Run `ipfs init` and start the `ipfs daemon`
+
+(Note that this special remote does not use ipfs's FUSE support; it
+communicates with ipfs using the `ipfs` command-line utility.)
+
+## configuration
+
+These parameters can be passed to `git annex initremote` to configure the
+remote:
+
+* `encryption` - One of "none", "hybrid", "shared", or "pubkey".
+  See [[encryption]]. Note that this is git-annex's encryption, not ipfs's
+  encryption.
+
+* `keyid` - Specifies the gpg key to use for [[encryption]].
+
+Setup example:
+
+	# git annex initremote ipfs type=external externaltype=ipfs encryption=none
+
+## content distribution
+
+After `git annex copy --to ipfs`, a file will typically only have
+been copied to your computer's local ipfs object store. It will not reach
+other ipfs nodes on the network until they request the content.
+
+If you set up a clone of your repository on another computer, and install
+ipfs and enable the ipfs remote there, you can proceed with using it to get
+files that have been stored in ipfs:
+
+	# git annex sync
+	# git annex enableremote ipfs
+	# git annex copy --from ipfs
+
+# content removal
+
+Removing content from ipfs requires all nodes that have a copy to decide to
+delete it. This is not something git-annex can arrange to happen, or
+reliably tell has happened, so `git annex drop --from ipfs` will always fail.
+
+## using ipfs addresses
+
+Once a file has been copied to ipfs, you can use `git annex whereis`
+to look up the ipfs address of the file:
+
+	# git annex whereis somefile
+	whereis somefile
+		ed1c811d-fe42-4436-aa75-56566c990aa8 -- ipfs
+	
+	ipfs: QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W
+
+In the example above, the ipfs address for the file is
+`QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W`. You can give this
+address to any other ipfs user and they can use it to download the file!
+
+You can also use ipfs addresses with `git annex addurl`. For example:
+
+	# git annex addurl ipfs:QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W --file somefile
+
+That's a real file; try it!
+
+## future directions
+
+While perhaps useful, this is just a proof of concept. It's particularly
+lacking in that it doesn't integrate well git-annex's [[location_tracking]]
+with ipfs. 
+
+Tracking which ipfs nodes have a copy of an annexed object
+would make this special remote work better. In particular, git-annex does
+not currently trust ipfs to contain a copy of an object, since it has no
+way of keeping track of which which ipfs nodes might contain it. So, eg, 
+`git annex drop` will refuse to trust ipfs.