path: root/doc/todo/smudge.mdwn

git-annex should use smudge/clean filters.

----

Update: Currently, this does not look likely to work. In particular,
the clean filter needs to consume all stdin from git, which consists of the
entire content of the file. It cannot optimise by directly accessing
the file in the repository, because git may be cleaning a different
version of the file during a merge. 

So every `git status` would need to read the entire content of all
available files, and checksum them, which is too expensive.

> Update from GitTogether: Peff thinks a new interface could be added to
> git to handle this sort of case in an efficient way.. just needs someone
> to do the work. --[[Joey]] 

>> Update 2015: git status only calls the clean filter for files
>> that the index says are modified, so this is no longer a problem.
>> --[[Joey]]

----

The clean filter is run when files are staged for commit. So a user could copy
any file into the annex, git add it, and git-annex's clean filter causes
the file's key to be staged, while its value is added to the annex.

The smudge filter is run when files are checked out. Since git annex
repos have partial content, this would not git annex get the file content.
Instead, if the content is not currently available, it would need to do
something like return empty file content. (Sadly, it cannot create a
symlink, as git still wants to write the file afterwards.)

So the nice current behavior of unavailable files being clearly missing due
to dangling symlinks, would be lost when using smudge/clean filters.
(Contact git developers to get an interface to do this?)

Instead, we get the nice behavior of not having to remeber to `git annex
add` files, and just being able to use `git add` or `git commit -a`,
and have it use git-annex when .gitattributes says to. Also, annexed
files can be directly modified without having to `git annex unlock`.

### configuration

In .gitattributes, the user would put something like "* filter=git-annex".
This way they could control which files are annexed vs added normally.

It would also be good to allow using this without having to specify
the files in .gitattributes. Just use "* filter=git-annex" there, and then
let git-annex decide which files to annex and which to pass through the
smudge and clean filters as-is. The smudge filter can just read a little of
its input to see if it's a pointer to an annexed file. The clean filter
could apply annex.largefiles to decide whether to annex a file's content or
not.

For files not configured this way in .gitattributes, git-annex could
continue to use its symlink method -- this would preserve backwards
compatability, and even allow mixing the two methods in a repo as desired.
(But not switching an existing repo between indirect and direct modes;
the user decides which mode to use when adding files to the repo.)

### clean

The trick is doing it efficiently. Since git a2b665d, v1.7.4.1,
something like this works to provide a filename to the clean script:

	git config --global filter.huge.clean huge-clean %f

This could avoid it needing to read all the current file content from stdin
when doing eg, a git status or git commit. Instead it is passed the
filename that git is operating on, in the working directory.
(Update: No, doesn't work; git may be cleaning a different file content
than is currently on disk, and git requires all stdin be consumed too.)

So, WORM could just look at that file and easily tell if it is one
it already knows (same mtime and size). If so, it can short-circuit and
do nothing, file content is already cached.

SHA1 has a harder job. Would not want to re-sha1 the file every time,
probably. So it'd need a local cache of file stat info, mapped to known
objects.

But: Even with %f, git actually passes the full file content to the clean
filter, and if it fails to consume it all, it will crash (may only happen
if the file is larger than some chunk size; tried with 500 mb file and 
saw a SIGPIPE.) This means unnecessary works needs to be done, 
and it slows down *everything*, from `git status` to `git commit`.
**showstopper** I have sent a patch to the git mailing list to address
this. <http://marc.info/?l=git&m=131465033512157&w=2> (Update: apparently
can't be fixed.)

> Update: I tried this again (2015) and it seems that git status and git
> add avoid re-sending the file content to the clean filter, as long as the
> file stat has not changed. I'm not sure when git started doing that,
> but it seems to avoid this problem.
> --[[Joey]]

### smudge

The smudge script can also be provided a filename with %f, but it
cannot directly write to the file or git gets unhappy.

> Still the case in 2015. Means an unnecesary read and pipe of the file
> even if the content is already locally available on disk. --[[Joey]]

### partial checkouts

.. Are very important, otherwise a repo can't scale past the size of the
smallest client's disk!

It would be nice if the smudge filter could hard link or symlink a work
tree file to the annex object.

But currently, the smudge filter can't modify the work tree file on its own
-- git always modifies the file after getting the output of the smudge
filter, and will stumble over any modifications that the smudge filter
makes. And, it's important that the smudge filter never fail as that will
leave the repo in a bad state.

Seems the best that can be done is for the smudge filter to copy from the
annex object when the object is present. When it's not present, the smudge
filter should provide a pointer to its content.

The clean filter should detect when it's operating on that pointer file.

I've a demo implementation of this technique in the scripts below.

### deduplication

.. Is nice; needing 2 copies of every annexed file is annoying.

Unfortunately, when using smudge/clean, `git merge` does not preserve a
smudged file in the work tree when renaming it. It instead deletes the old
file and asks the smudge filter to smudge the new filename.

So, copies need to be maintained in .git/annex/objects, though it's ok
to use hard links to the work tree files.

Even if hard links are used, smudge needs to output the content of an
annexed file, which will result in duplication when merging in renames of
files.

### design

Goal: Get rid of current direct mode, using smudge/clean filters instead to
cover the same use cases, more flexibly and robustly.

Use case 1:

A user wants to be able to edit files, and git-add, git commit,
without needing to worry about using git-annex to unlock files, add files,
etc.

Use case 2:

Using git-annex on a crippled filesystem that does not support symlinks.

Data:

* An annex pointer file has as its first line the git-annex key
  that it's standing in for. Subsequent lines of the file might
  be a message saying that the file's content is not currently available.
  An annex pointer file is checked into the git repository the same way
  that an annex symlink is checked in.
* file2key maps are maintained by git-annex, to keep track of
  what files are pointers at keys.

Configuration: 

* .gitattributes tells git which files to use git-annex's smudge/clean
  filters with. Typically, all files except for dotfiles:

	* filter=annex
	.* !filter

* annex.largefiles tells git-annex which files should in fact be put in 
  the annex. Other files are passed through the smudge/clean as-is and
  have their contents stored in git.

* annex.direct is repurposed to configure how the assistant adds files.
  When set to true, they're added unlocked.

git-annex clean:

* Run by `git add` (and diff and status, etc), and passed the
  filename, as well as fed the file content on stdin.

  Look at configuration to decide if this file's content belongs in the
  annex. If not, output the file content to stdout.

  Generate annex key from filename and content from stdin.

  Hard link .git/annex/objects to the file, if it doesn't already exist.
  (On platforms not supporting hardlinks, copy the file to
  .git/annex/objects.)

  This is done to prevent losing the only copy of a file when eg
  doing a git checkout of a different branch, or merging a commit that
  renames or deletes a file. But, no attempt is made to 
  protect the object from being modified. If a user wants to
  protect object contents from modification, they should use
  `git annex add`, not `git add`, or they can `git annex lock` after adding,.

  There could be a configuration knob to cause a copy to be made to
  .git/annex/objects -- useful for those crippled filesystems. It might
  also drop that copy once the object gets uploaded to another repo ...
  But that gets complicated quickly.

  Update file2key map.

  Output the pointer file content to stdout.

git-annex smudge:

* Run by eg `git checkout` and passed the filename, as well as fed
  the pointer file content on stdin.

  Updates file2key map.

  When an object is present in the annex, outputs its content to stdout.
  Otherwise, outputs the file pointer content.

git annex direct/indirect:

  Previously these commands switched in and out of direct mode.
  Now they become no-ops.

git annex lock/unlock:

  Makes sense for these to change to switch files between using
  git-annex symlinks and pointers. So, this provides both a way to
  transition repositories to using pointers, and a cleaner unlock/lock
  for repos using symlinks.

  unlock will stage a pointer file, and will copy the content of the object
  out of .git/annex/objects to the work tree file. (Might want a --hardlink
  switch.)
  
  lock will replace the current work tree file with the symlink, and stage it.
  Note that multiple work tree files could point to the same object.
  So, if the link count is > 1, replace the annex object with a copy of
  itself to break such a hard link. Always finish by locking down the
  permissions of the annex object.

All other git-annex commands that look at annex symlinks to get keys will
need fall back to checking if a given work tree file is stored in git as
pointer file. This can be done by checking the file2key map (or by looking
it up in the index).

Note that I have not verified if file2key maps can be maintained
consistently using the smudge/clean filters. Seems likely to work,
based on when I see smudge/clean filters being run. The file2key
optimisation may not be needed though, looking at the index 
might be fast enough.

#### Upgrading

annex.version changes to 6

Upgrade should be handled automatically.

On upgrade, update .gitattributes with a stock configuration, unless
it already mentions "filter=annex".

Upgrading a direct mode repo needs to switch it out of bare mode, and
needs to run `git annex unlock` on all files (or reach the same result).
So will need to stage changes to all annexed files.

When a repo has some clones indirect and some direct, the upgraded repo
will have all files unlocked, necessarily in all clones.

----

### test files

huge-smudge:

<pre>
#!/bin/sh
read f
file="$1"
echo "smudging $f" >&2
if [ -e ~/$f ]; then
	cat ~/$f # possibly expensive copy here
else
	echo "$f not available"
fi
</pre>

huge-clean:

<pre>
#!/bin/sh
file="$1"
cat >/tmp/file
# in real life, this should be done more efficiently, not trying to read
# the whole file content!
if grep -q 'not available' /tmp/file; then
	awk '{print $1}' /tmp/file # provide what we would if the content were avail!
	exit 0
fi
echo "cleaning $file" >&2
# XXX store file content here
echo $file
</pre>

.gitattributes:

<pre>
*.huge filter=huge
</pre>

in .git/config:

<pre>
[filter "huge"]
        clean = huge-clean %f
        smudge = huge-smudge %f
<pre>