diff options
| -rw-r--r-- | doc/design/external_special_remote_protocol.mdwn | 77 | ||||
| -rw-r--r-- | doc/special_remotes/external.mdwn | 20 | ||||
| -rwxr-xr-x | doc/special_remotes/external/example.sh | 131 | ||||
| -rw-r--r-- | doc/special_remotes/hook.mdwn | 12 | ||||
| -rw-r--r-- | doc/todo/support_for_writing_external_special_remotes.mdwn | 2 |
5 files changed, 171 insertions, 71 deletions
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn index 8c5c245b0..7dc8182fa 100644 --- a/doc/design/external_special_remote_protocol.mdwn +++ b/doc/design/external_special_remote_protocol.mdwn @@ -1,10 +1,13 @@ -See [[todo/support_for_writing_external_special_remotes]] for motivation. +Communication between git-annex and a program implementing an external +special remote uses this protocol. -This is a design for a protocol to be used to communicate between git-annex -and a program implementing an external special remote. +[[!toc]] + +## starting the program The external special remote program has a name like -`git-annex-remote-$bar`. When `git annex initremote foo type=$bar` is run, +`git-annex-remote-$bar`. When +`git annex initremote foo type=external externaltype=$bar` is run, git-annex finds the appropriate program in PATH. The program is started by git-annex when it needs to access the special @@ -31,7 +34,7 @@ only sends replies to the requests. The special remote is responsible for sending the first message, indicating the version of the protocol it is using. - VERSION 0 + VERSION 1 Once it knows the version, git-annex will send a message telling the special remote to start up. @@ -154,7 +157,7 @@ These messages may be sent by the special remote at any time that it's in control. * `VERSION Int` - Supported protocol version. Current version is 0. Must be sent first + Supported protocol version. Current version is 1. Must be sent first thing at startup, as until it sees this git-annex does not know how to talk with the special remote program! * `PROGRESS Int` @@ -200,68 +203,6 @@ remote. git-annex will not talk to it any further. If the program receives an ERROR from git-annex, it can exit with its own ERROR. -## Simple shell example - -[[!format sh """ -#!/bin/sh -set -e - -echo VERSION 0 - -while read line; do - set -- $line - case "$1" in - INITREMOTE) - # XXX do anything necessary to create resources - # used by the remote. Try to be idempotent. - # Use GETCONFIG to get any needed configuration - # settings, and SETCONFIG to set any persistent - # configuration settings. - echo INITREMOTE-SUCCESS - ;; - GETCOST) - echo COST-UNKNOWN - ;; - PREPARE) - # XXX Use GETCONFIG to get configuration settings, - # and do anything needed to start using the - # special remote here. - echo PREPARE-SUCCESS - ;; - TRANSFER) - key="$3" - file="$4" - case "$2" in - STORE) - # XXX upload file here - # XXX when possible, send PROGRESS - echo TRANSFER-SUCCESS STORE "$key" - ;; - RETRIEVE) - # XXX download file here - echo TRANSFER-SUCCESS RETRIEVE "$key" - ;; - - esac - ;; - CHECKPRESENT) - key="$2" - echo CHECKPRESENT-UNKNOWN "$key" "not implemented" - ;; - REMOVE) - key="$2" - # XXX remove key here - echo REMOVE-SUCCESS "$key" - ;; - *) - echo UNKNOWN-REQUEST - ;; - esac -done - -# XXX anything that needs to be done at shutdown can be done here -"""]] - ## TODO * Communicate when the network connection may have changed, so long-running diff --git a/doc/special_remotes/external.mdwn b/doc/special_remotes/external.mdwn new file mode 100644 index 000000000..5d150e583 --- /dev/null +++ b/doc/special_remotes/external.mdwn @@ -0,0 +1,20 @@ +There are three ways to implement a new special remote: + +1. Using the [[hook]] special remote to tell git-annex what commands + to run to store and retrieve data. This is the easiest way, and + is great for prototyping. +2. Writing it in Haskell and adding it to git-annex. +3. Writing a program in any language you like that speaks the + [[external_special_remote_protocol]]. + +This page is all about writing new external special remotes. It's not hard! + +* All you need is to make a program with a name like `git-annex-remote-$bar`. +* Install it in PATH. +* When the user runs `git annex initremote foo type=external externaltype=$bar`, + it will use your program. + +Here's a simple shell script example, which can easily be adapted +to run whatever commands you need. ([[download|example.sh]]) + +[[!inline raw=yes pages="special_remotes/external/example.sh"]] diff --git a/doc/special_remotes/external/example.sh b/doc/special_remotes/external/example.sh new file mode 100755 index 000000000..07dcb2705 --- /dev/null +++ b/doc/special_remotes/external/example.sh @@ -0,0 +1,131 @@ +#!/bin/sh +# git-annex external special remote program +# +# This is basically the same as git-annex's built-in directory special remote. +# +# Install in PATH as git-annex-remote-directorya +# +# Copyright 2013 Joey Hess; licenced under the GNU GPL version 3 or higher. + +set -e + +# This program speaks a line-based protocol on stdin and stdout. +# When running any commands, their stdout should be redirected to stderr +# (or /dev/null) to avoid messing up the protocol. +runcmd () { + "$@" >&2 +} + +# Gets a value from the remote's configuration, and stores it in RET +getconfig () { + echo GETCONFIG "$1" + read resp + set -- $resp + case "$1" in + VALUE) + RET="$2" + ;; + *) + RET="" + ;; + esac +} + +# Sets LOC to the location to use to store a key. +mylocation () { + echo HASHDIR "$1" + read resp + set -- $resp + case "$1" in + VALUE) + LOC="$hashdir/$1" + ;; + *) + LOC= + ;; + esac +} + +echo VERSION 1 + +while read line; do + set -- $line + case "$1" in + INITREMOTE) + # XXX do anything necessary to create resources + # used by the remote. Try to be idempotent. + # Use GETCONFIG to get any needed configuration + # settings, and SETCONFIG to set any persistent + # configuration settings. + getconfig directory + mydirectory="$RET" + if [ -z "$mydirectory" ]; then + echo INITREMOTE-FAILURE "You need to set directory=" + else + mkdir -p "$mydirectory" + echo INITREMOTE-SUCCESS + fi + ;; + GETCOST) + echo COST-UNKNOWN + ;; + PREPARE) + # XXX Use GETCONFIG to get configuration settings, + # and do anything needed to get ready for using the + # special remote here. + getconfig directory + mydirectory="$RET" + ;; + TRANSFER) + key="$3" + file="$4" + case "$2" in + STORE) + # XXX upload file here + # XXX when possible, send PROGRESS + calclocation "$key" + mkdir -p "$(dirname "$LOC")" + runcmd cp -v "$file" "$LOC" + echo TRANSFER-SUCCESS STORE "$key" + ;; + RETRIEVE) + # XXX download file here + calclocation "$key" + runcmd cp -v "$LOC" "$file" + echo TRANSFER-SUCCESS RETRIEVE "$key" + ;; + esac + ;; + CHECKPRESENT) + key="$2" + calclocation "$key" + if [ -e "$LOC" ]; then + echo CHECKPRESENT-SUCCESS "$key" + else + if [ -d "$mydirectory" ]; then + echo CHECKPRESENT-FAILURE "$key" + else + # If the directory does not exist, + # the remote is not available. + # (A network remote would similarly + # fail with CHECKPRESENT-UNKNOWN + # if it couldn't be contacted). + echo CHECKPRESENT-UNKNOWN "$key" "this remote is not currently available" + fi + fi + ;; + REMOVE) + key="$2" + calclocation "$key" + # Note that it's not a failure to remove a + # key that is not present, so -f is used. + runcmd rm -f "$LOC" + echo REMOVE-SUCCESS "$key" + ;; + *) + echo UNKNOWN-REQUEST + ;; + esac +done + +# XXX anything that needs to be done at shutdown can be done here diff --git a/doc/special_remotes/hook.mdwn b/doc/special_remotes/hook.mdwn index eaea940a7..8cf31ed02 100644 --- a/doc/special_remotes/hook.mdwn +++ b/doc/special_remotes/hook.mdwn @@ -1,9 +1,12 @@ This special remote type lets you store content in a remote of your own -devising. +devising, configured via some simple hooks. It's not recommended to use this remote type when another like [[rsync]] or [[directory]] will do. If your hooks are not carefully written, data -could be lost. +could be lost. + +If you're building a special remote for others to use, +instead consider building an [[external_special_remote|external]]. ## example @@ -68,6 +71,9 @@ The settings to use in git config for the hook commands are as follows: ## combined hook program +This interface is deprecated -- it's better, and not much harder, +to write an [[external_special_remote|external]]! + Rather than setting all of the above hooks, you can write a single program that handles everything, and set a single hook to make it be used. @@ -75,7 +81,7 @@ program that handles everything, and set a single hook to make it be used. # git annex initremote mydemorepo type=hook hooktype=demo encryption=none The program just needs to look at the `ANNEX_ACTION` environment variable -to see what it's being asked to do For example: +to see what it's being asked to do. For example: [[!format sh """ #!/bin/sh diff --git a/doc/todo/support_for_writing_external_special_remotes.mdwn b/doc/todo/support_for_writing_external_special_remotes.mdwn index 2d7cd9d15..1732f77ea 100644 --- a/doc/todo/support_for_writing_external_special_remotes.mdwn +++ b/doc/todo/support_for_writing_external_special_remotes.mdwn @@ -23,3 +23,5 @@ should look for `git-annex-remote-$bar` in PATH if that's not a built-in special remote name. --[[Joey]] + +[[done]] |