pre-commit-annex hook script to automatically extract metadata from lots of types of files

Using the extract(1) program to do the heavy lifting.

Decided to make git-annex run pre-commit-annex when committing. Since
git-annex pre-commit also runs it, it'll be run when git commit is run too,
via the pre-commit hook. This basically gives back the pre-commit hook
that git-annex took away. The implementation avoids repeatedly looking
for the hook script when the assistant is running and committing
repeatedly; only checks if the hook is available once.

To make the script simpler, made git-annex metadata -s field?=value
only set a field when it's not already got a value.

This commit was sponsored by bak.

5 files changed, 95 insertions, 9 deletions

diff --git a/doc/design/metadata.mdwn b/doc/design/metadata.mdwn
index 264505a1c..10e79b9f8 100644
--- a/doc/design/metadata.mdwn
+++ b/doc/design/metadata.mdwn
@@ -56,14 +56,7 @@ once, and can be left alone when refining a view.
 When annex.genmetadata is set, git annex add automatically attaches
 some metadata to a file. Currently year and month fields, from its mtime.
 
-TODO A git hook could be run by git annex add to gather more metadata.
-For example, by examining file permisions or MP3 metadata.
-Alternatively, this could be a
-regular post-commit hook, that examines the files committed, and runs git
-annex metadata to add metadata. No extra git-annex support is needed
-to do that!
-However, in direct mode, or when using the assistant, git-annex does its
-own committing, not using git commit, so bypassing the commit hooks.
+There's also a post-commit-annex hook script.
 
 ## directory hierarchy metadata
 
diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index f0b29f336..53b9c2eab 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -717,6 +717,9 @@ subdirectories).
 
   To remove a value, use -s field-=value.
 
+  To set a value, only if the field does not already have a value,
+  use -s field?=value
+
   To set a tag, use -t tag, and use -u tag to remove a tag.
 
   For example, to set some tags on a file and also its author:
@@ -1678,6 +1681,10 @@ used by git-annex.
 `~/.config/git-annex/autostart` is a list of git repositories
 to start the git-annex assistant in.
 
+`.git/hooks/pre-commit-annex` in your git repsitory will be run whenever
+a commit is made, either by git commit, git-annex sync, or the git-annex
+assistant.
+
 # SEE ALSO
 
 Most of git-annex's documentation is available on its web site,
diff --git a/doc/metadata.mdwn b/doc/metadata.mdwn
index 9966e7d7d..df873c4c1 100644
--- a/doc/metadata.mdwn
+++ b/doc/metadata.mdwn
@@ -30,7 +30,8 @@ Here are some recommended metadata fields to use:
   being.
   
 To make git-annex automatically set the year and month when adding files,
-run `git config annex.genmetadata true`
+run `git config annex.genmetadata true`. Also, see
+[[tips/automatically_adding_metadata]].
 
 git-annex's metadata can be updated in a distributed fashion. For example,
 two users, each with their own clone of a repository, can set and unset
diff --git a/doc/tips/automatically_adding_metadata.mdwn b/doc/tips/automatically_adding_metadata.mdwn
new file mode 100644
index 000000000..bd8fea737
--- /dev/null
+++ b/doc/tips/automatically_adding_metadata.mdwn
@@ -0,0 +1,24 @@
+git-annex's [[metadata]] works best when files have a lot of useful
+metadata attached to them.
+
+To make git-annex automatically set the year and month when adding files,
+run `git config annex.genmetadata true`.
+
+A git commit hook can be set up to extract lots of metadata from files
+like photos, mp3s, etc.
+
+* Install the `extract` utility, from <http://www.gnu.org/software/libextractor/>  
+  `apt-get install extract`
+* Download [[pre-commit-annex]] and install it in your git-annex repository
+  as `.git/hooks/pre-commit-annex`.  
+  Remember to make the script executable!
+* Run: `git config metadata.extract "artist album title camera_make video_dimensions"`
+
+Now any fields you list in metadata.extract to will be extracted and
+stored when files are committed.
+
+To get a list of all possible fields, run: `extract -L | sed ' ' _`
+
+By default, if a git-annex already has a metadata field for a file,
+its value will not be overwritten with metadata taken from files.
+To allow overwriting, run: `git config metadata.overwrite true`
diff --git a/doc/tips/automatically_adding_metadata/pre-commit-annex b/doc/tips/automatically_adding_metadata/pre-commit-annex
new file mode 100755
index 000000000..4897493cf
--- /dev/null
+++ b/doc/tips/automatically_adding_metadata/pre-commit-annex
@@ -0,0 +1,61 @@
+#!/bin/sh
+# This script can be used to add git-annex metadata to files when they're
+# committed.
+#
+# Copyright 2014 Joey Hess <id@joeyh.name>
+# License: GPL-3+
+
+extract="$(git config metadata.extract || true)" 
+want="$(perl -e 'print (join("|", map {s/_/ /g; "^$_ - "} (split " ", shift())))' "$extract")"
+
+if [ -z "$want" ]; then
+	exit 0
+fi
+
+echo "$want"
+
+case "$(git config --bool metadata.overwrite || true)" in
+	true)
+		overwrite=1
+	;;
+	*)
+		overwrite=""
+	;;
+esac
+
+# By default, any existing metadata git-annex has stored is not overwritten
+# with metadata taken from files. Uncomment this to enable.
+#overwrite_metadata=1
+
+addmeta () {
+	file="$1"
+	field="$2"
+	value="$3"
+	afield="$(echo "$field" | tr ' ' _)"
+	if [ "$overwrite" ]; then
+		p="$afield=$value"
+
+	else
+		p="$afield?=$value"
+	fi
+	git -c annex.alwayscommit=false annex metadata "$file" -s "$p" --quiet
+}
+
+if git rev-parse --verify HEAD >/dev/null 2>&1; then
+	against=HEAD
+else
+	# Initial commit: diff against an empty tree object
+	against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
+fi
+
+IFS="
+"
+for f in $(git diff-index --name-only --cached $against); do
+	if [ -e "$f" ]; then
+		for l in $(extract "$f" | egrep "$want"); do
+			field="${l%% - *}"
+			value="${l#* - }"
+			addmeta "$f" "$field" "$value"
+		done
+	fi
+done