external special remote documentation and example script

5 files changed, 171 insertions, 71 deletions

diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 8c5c245b0..7dc8182fa 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -1,10 +1,13 @@
-See [[todo/support_for_writing_external_special_remotes]] for motivation.
+Communication between git-annex and a program implementing an external
+special remote uses this protocol.
 
-This is a design for a protocol to be used to communicate between git-annex
-and a program implementing an external special remote.
+[[!toc]]
+
+## starting the program
 
 The external special remote program has a name like
-`git-annex-remote-$bar`. When `git annex initremote foo type=$bar` is run,
+`git-annex-remote-$bar`. When 
+`git annex initremote foo type=external externaltype=$bar` is run,
 git-annex finds the appropriate program in PATH.
 
 The program is started by git-annex when it needs to access the special
@@ -31,7 +34,7 @@ only sends replies to the requests.
 The special remote is responsible for sending the first message, indicating
 the version of the protocol it is using.
 
-	VERSION 0
+	VERSION 1
 
 Once it knows the version, git-annex will send a message telling the
 special remote to start up.
@@ -154,7 +157,7 @@ These messages may be sent by the special remote at any time that it's
 in control.
 
 * `VERSION Int`  
-  Supported protocol version. Current version is 0. Must be sent first
+  Supported protocol version. Current version is 1. Must be sent first
   thing at startup, as until it sees this git-annex does not know how to
   talk with the special remote program!
 * `PROGRESS Int`  
@@ -200,68 +203,6 @@ remote.
   git-annex will not talk to it any further. If the program receives
   an ERROR from git-annex, it can exit with its own ERROR.
 
-## Simple shell example
-
-[[!format sh """
-#!/bin/sh
-set -e
-
-echo VERSION 0
-
-while read line; do
-	set -- $line
-	case "$1" in
-		INITREMOTE)
-			# XXX do anything necessary to create resources
-			# used by the remote. Try to be idempotent.
-			# Use GETCONFIG to get any needed configuration
-			# settings, and SETCONFIG to set any persistent
-			# configuration settings.
-			echo INITREMOTE-SUCCESS
-		;;
-		GETCOST)
-			echo COST-UNKNOWN
-		;;
-		PREPARE)
-			# XXX Use GETCONFIG to get configuration settings,
-			# and do anything needed to start using the
-			# special remote here.
-			echo PREPARE-SUCCESS
-		;;
-		TRANSFER)
-			key="$3"
-			file="$4"
-			case "$2" in
-				STORE)
-					# XXX upload file here
-					# XXX when possible, send PROGRESS
-					echo TRANSFER-SUCCESS STORE "$key"
-				;;
-				RETRIEVE)
-					# XXX download file here
-					echo TRANSFER-SUCCESS RETRIEVE "$key"
-				;;
-				
-			esac
-		;;
-		CHECKPRESENT)
-			key="$2"
-			echo CHECKPRESENT-UNKNOWN "$key" "not implemented"
-		;;
-		REMOVE)
-			key="$2"
-			# XXX remove key here
-			echo REMOVE-SUCCESS "$key"
-		;;
-		*)
-			echo UNKNOWN-REQUEST
-		;;
-	esac	
-done
-
-# XXX anything that needs to be done at shutdown can be done here
-"""]]
-
 ## TODO
 
 * Communicate when the network connection may have changed, so long-running
diff --git a/doc/special_remotes/external.mdwn b/doc/special_remotes/external.mdwn
new file mode 100644
index 000000000..5d150e583
--- /dev/null
+++ b/doc/special_remotes/external.mdwn
@@ -0,0 +1,20 @@
+There are three ways to implement a new special remote:
+
+1. Using the [[hook]] special remote to tell git-annex what commands
+   to run to store and retrieve data. This is the easiest way, and
+   is great for prototyping.
+2. Writing it in Haskell and adding it to git-annex.
+3. Writing a program in any language you like that speaks the
+   [[external_special_remote_protocol]].
+
+This page is all about writing new external special remotes. It's not hard!
+
+* All you need is to make a program with a name like `git-annex-remote-$bar`.
+* Install it in PATH.
+* When the user runs `git annex initremote foo type=external externaltype=$bar`,
+  it will use your program.
+
+Here's a simple shell script example, which can easily be adapted
+to run whatever commands you need. ([[download|example.sh]])
+
+[[!inline raw=yes pages="special_remotes/external/example.sh"]]
diff --git a/doc/special_remotes/external/example.sh b/doc/special_remotes/external/example.sh
new file mode 100755
index 000000000..07dcb2705
--- /dev/null
+++ b/doc/special_remotes/external/example.sh
@@ -0,0 +1,131 @@
+#!/bin/sh
+# git-annex external special remote program
+# 
+# This is basically the same as git-annex's built-in directory special remote.
+# 
+# Install in PATH as git-annex-remote-directorya
+#
+# Copyright 2013 Joey Hess; licenced under the GNU GPL version 3 or higher.
+
+set -e
+
+# This program speaks a line-based protocol on stdin and stdout.
+# When running any commands, their stdout should be redirected to stderr
+# (or /dev/null) to avoid messing up the protocol.
+runcmd () {
+	"$@" >&2
+}
+
+# Gets a value from the remote's configuration, and stores it in RET
+getconfig () {
+	echo GETCONFIG "$1"
+	read resp
+	set -- $resp
+	case "$1" in
+		VALUE)
+			RET="$2"
+		;;
+		*)
+			RET=""
+		;;
+	esac
+}
+
+# Sets LOC to the location to use to store a key.
+mylocation () {
+	echo HASHDIR "$1"
+	read resp
+	set -- $resp
+	case "$1" in
+		VALUE)
+			LOC="$hashdir/$1"
+		;;
+		*)
+			LOC=
+		;;
+	esac
+}
+
+echo VERSION 1
+
+while read line; do
+	set -- $line
+	case "$1" in
+		INITREMOTE)
+			# XXX do anything necessary to create resources
+			# used by the remote. Try to be idempotent.
+			# Use GETCONFIG to get any needed configuration
+			# settings, and SETCONFIG to set any persistent
+			# configuration settings.
+			getconfig directory
+			mydirectory="$RET"
+			if [ -z "$mydirectory" ]; then
+				echo INITREMOTE-FAILURE "You need to set directory="
+			else
+				mkdir -p "$mydirectory"
+				echo INITREMOTE-SUCCESS
+			fi
+		;;
+		GETCOST)
+			echo COST-UNKNOWN
+		;;
+		PREPARE)
+			# XXX Use GETCONFIG to get configuration settings,
+			# and do anything needed to get ready for using the
+			# special remote here.
+			getconfig directory
+			mydirectory="$RET"
+		;;
+		TRANSFER)
+			key="$3"
+			file="$4"
+			case "$2" in
+				STORE)
+					# XXX upload file here
+					# XXX when possible, send PROGRESS
+					calclocation "$key"
+					mkdir -p "$(dirname "$LOC")"
+					runcmd cp -v "$file" "$LOC"
+					echo TRANSFER-SUCCESS STORE "$key"
+				;;
+				RETRIEVE)
+					# XXX download file here
+					calclocation "$key"
+					runcmd cp -v "$LOC" "$file"
+					echo TRANSFER-SUCCESS RETRIEVE "$key"
+				;;
+			esac
+		;;
+		CHECKPRESENT)
+			key="$2"
+			calclocation "$key"
+			if [ -e "$LOC" ]; then
+				echo CHECKPRESENT-SUCCESS "$key"
+			else
+				if [ -d "$mydirectory" ]; then
+					echo CHECKPRESENT-FAILURE "$key"
+				else
+					# If the directory does not exist,
+					# the remote is not available.
+					# (A network remote would similarly
+					# fail with CHECKPRESENT-UNKNOWN
+					# if it couldn't be contacted).
+					echo CHECKPRESENT-UNKNOWN "$key" "this remote is not currently available"
+				fi
+			fi
+		;;
+		REMOVE)
+			key="$2"
+			calclocation "$key"
+			# Note that it's not a failure to remove a
+			# key that is not present, so -f is used.
+			runcmd rm -f "$LOC"
+			echo REMOVE-SUCCESS "$key"
+		;;
+		*)
+			echo UNKNOWN-REQUEST
+		;;
+	esac	
+done
+
+# XXX anything that needs to be done at shutdown can be done here
diff --git a/doc/special_remotes/hook.mdwn b/doc/special_remotes/hook.mdwn
index eaea940a7..8cf31ed02 100644
--- a/doc/special_remotes/hook.mdwn
+++ b/doc/special_remotes/hook.mdwn
@@ -1,9 +1,12 @@
 This special remote type lets you store content in a remote of your own
-devising.
+devising, configured via some simple hooks.
 
 It's not recommended to use this remote type when another like [[rsync]]
 or [[directory]] will do. If your hooks are not carefully written, data
-could be lost.
+could be lost. 
+
+If you're building a special remote for others to use,
+instead consider building an [[external_special_remote|external]].
 
 ## example
 
@@ -68,6 +71,9 @@ The settings to use in git config for the hook commands are as follows:
 
 ## combined hook program
 
+This interface is deprecated -- it's better, and not much harder,
+to write an [[external_special_remote|external]]!
+
 Rather than setting all of the above hooks, you can write a single
 program that handles everything, and set a single hook to make it be used.
 
@@ -75,7 +81,7 @@ program that handles everything, and set a single hook to make it be used.
 	# git annex initremote mydemorepo type=hook hooktype=demo encryption=none
 
 The program just needs to look at the `ANNEX_ACTION` environment variable
-to see what it's being asked to do  For example:
+to see what it's being asked to do. For example:
 
 [[!format sh """
 #!/bin/sh
diff --git a/doc/todo/support_for_writing_external_special_remotes.mdwn b/doc/todo/support_for_writing_external_special_remotes.mdwn
index 2d7cd9d15..1732f77ea 100644
--- a/doc/todo/support_for_writing_external_special_remotes.mdwn
+++ b/doc/todo/support_for_writing_external_special_remotes.mdwn
@@ -23,3 +23,5 @@ should look for `git-annex-remote-$bar` in PATH if that's not a built-in
 special remote name.
 
 --[[Joey]]
+
+[[done]]