gitignore support for the assistant and watcher

Requires git 1.8.4 or newer. When it's installed, a background
git check-ignore process is run, and used to efficiently check ignores
whenever a new file is added.

Thanks to Adam Spiers, for getting the necessary support into git for this.

A complication is what to do about files that are gitignored but have
been checked into git anyway. git commands assume the ignore has been
overridden in this case, and not need any more overriding to commit a
changed version.

However, for the assistant to do the same, it would have to run git ls-files
to check if the ignored file is in git. This is somewhat expensive. Or it
could use the running git-cat-file process to query the file that way,
but that requires transferring the whole file content over a pipe, so it
can be quite expensive too, for files that are not git-annex
symlinks.

Now imagine if the user knows that a file or directory tree will be getting
frequent changes, and doesn't want the assistant to sync it, so gitignores
it. The assistant could overload the system with repeated ls-files checks!

So, I've decided that the assistant will not automatically commit changes
to files that are gitignored. This is a tradeoff. Hopefully it won't be a
problem to adjust .gitignore settings to not ignore files you want the
assistant to autocommit, or to manually git annex add files that are listed
in .gitignore.

(This could be revisited if git-annex gets access to an interface to check
the content of the index w/o forking a git command. This could be libgit2,
or perhaps a separate git cat-file --batch-check process, so it wouldn't
need to ship over the whole file content.)

This commit was sponsored by Francois Marier. Thanks!

7 files changed, 135 insertions, 15 deletions

diff --git a/Annex.hs b/Annex.hs
index e15d31e72..7625fa8b6 100644
--- a/Annex.hs
+++ b/Annex.hs
@@ -43,6 +43,7 @@ import qualified Git
 import qualified Git.Config
 import Git.CatFile
 import Git.CheckAttr
+import Git.CheckIgnore
 import Git.SharedRepository
 import qualified Git.Queue
 import Types.Backend
@@ -91,6 +92,7 @@ data AnnexState = AnnexState
 	, repoqueue :: Maybe Git.Queue.Queue
 	, catfilehandles :: M.Map FilePath CatFileHandle
 	, checkattrhandle :: Maybe CheckAttrHandle
+	, checkignorehandle :: Maybe (Maybe CheckIgnoreHandle)
 	, forcebackend :: Maybe String
 	, forcenumcopies :: Maybe Int
 	, limit :: Matcher (FileInfo -> Annex Bool)
@@ -123,6 +125,7 @@ newState gitrepo = AnnexState
 	, repoqueue = Nothing
 	, catfilehandles = M.empty
 	, checkattrhandle = Nothing
+	, checkignorehandle = Nothing
 	, forcebackend = Nothing
 	, forcenumcopies = Nothing
 	, limit = Left []
diff --git a/Annex/CheckIgnore.hs b/Annex/CheckIgnore.hs
new file mode 100644
index 000000000..e5626557d
--- /dev/null
+++ b/Annex/CheckIgnore.hs
@@ -0,0 +1,32 @@
+{- git check-ignore interface, with handle automatically stored in
+ - the Annex monad
+ -
+ - Copyright 2013 Joey Hess <joey@kitenet.net>
+ -
+ - Licensed under the GNU GPL version 3 or higher.
+ -}
+
+module Annex.CheckIgnore (
+	checkIgnored,
+	checkIgnoreHandle
+) where
+
+import Common.Annex
+import qualified Git.CheckIgnore as Git
+import qualified Annex
+
+checkIgnored :: FilePath -> Annex Bool
+checkIgnored file = go =<< checkIgnoreHandle
+  where
+  	go Nothing = return False
+	go (Just h) = liftIO $ Git.checkIgnored h file
+
+checkIgnoreHandle :: Annex (Maybe Git.CheckIgnoreHandle)
+checkIgnoreHandle = maybe startup return =<< Annex.getState Annex.checkignorehandle
+  where
+	startup = do
+		v <- inRepo $ Git.checkIgnoreStart
+		when (isNothing v) $
+			warning "The installed version of git is too old for .gitignores to be honored by git-annex."
+		Annex.changeState $ \s -> s { Annex.checkignorehandle = Just v }
+		return v
diff --git a/Assistant/Threads/Watcher.hs b/Assistant/Threads/Watcher.hs
index 57719473f..ef8bcd41f 100644
--- a/Assistant/Threads/Watcher.hs
+++ b/Assistant/Threads/Watcher.hs
@@ -1,11 +1,11 @@
 {- git-annex assistant tree watcher
  -
- - Copyright 2012 Joey Hess <joey@kitenet.net>
+ - Copyright 2012-2013 Joey Hess <joey@kitenet.net>
  -
  - Licensed under the GNU GPL version 3 or higher.
  -}
 
-{-# LANGUAGE DeriveDataTypeable, CPP #-}
+{-# LANGUAGE DeriveDataTypeable, BangPatterns, CPP #-}
 
 module Assistant.Threads.Watcher (
 	watchThread,
@@ -33,6 +33,7 @@ import qualified Backend
 import Annex.Direct
 import Annex.Content.Direct
 import Annex.CatFile
+import Annex.CheckIgnore
 import Annex.Link
 import Annex.FileMatcher
 import Annex.ReplaceFile
@@ -141,6 +142,8 @@ startupScan scanner = do
 
 		return (True, r)
 
+{- Hardcoded ignores, passed to the DirWatcher so it can avoid looking
+ - at the entire .git directory. Does not include .gitignores. -}
 ignored :: FilePath -> Bool
 ignored = ig . takeFileName
   where
@@ -152,6 +155,12 @@ ignored = ig . takeFileName
 #endif
 	ig _ = False
 
+unlessIgnored :: FilePath -> Assistant (Maybe Change) -> Assistant (Maybe Change)
+unlessIgnored file a = ifM (liftAnnex $ checkIgnored file)
+	( noChange
+	, a
+	)
+
 type Handler = FilePath -> Maybe FileStatus -> Assistant (Maybe Change)
 
 {- Runs an action handler, and if there was a change, adds it to the ChangeChan.
@@ -186,7 +195,9 @@ add bigfilematcher file = ifM (liftAnnex $ checkFileMatcher bigfilematcher file)
 
 onAdd :: FileMatcher -> Handler
 onAdd matcher file filestatus
-	| maybe False isRegularFile filestatus = add matcher file
+	| maybe False isRegularFile filestatus =
+		unlessIgnored file $
+			add matcher file
 	| otherwise = noChange
 
 {- In direct mode, add events are received for both new files, and
@@ -214,9 +225,10 @@ onAddDirect symlinkssupported matcher file fs = do
 					liftAnnex $ changedDirect key file
 					add matcher file
 				)
-		_ -> guardSymlinkStandin Nothing $ do
-			debug ["add direct", file]
-			add matcher file
+		_ -> unlessIgnored file $
+			guardSymlinkStandin Nothing $ do
+				debug ["add direct", file]
+				add matcher file
   where
  	{- On a filesystem without symlinks, we'll get changes for regular
 	 - files that git uses to stand-in for symlinks. Detect when
@@ -240,7 +252,7 @@ onAddDirect symlinkssupported matcher file fs = do
  - before adding it.
  -}
 onAddSymlink :: Bool -> Handler
-onAddSymlink isdirect file filestatus = do
+onAddSymlink isdirect file filestatus = unlessIgnored file $ do
 	linktarget <- liftIO (catchMaybeIO $ readSymbolicLink file)
 	kv <- liftAnnex (Backend.lookupFile file)
 	onAddSymlink' linktarget (fmap fst kv) isdirect file filestatus
diff --git a/Git/CheckIgnore.hs b/Git/CheckIgnore.hs
new file mode 100644
index 000000000..2ab7cb3dc
--- /dev/null
+++ b/Git/CheckIgnore.hs
@@ -0,0 +1,71 @@
+{- git check-ignore interface
+ -
+ - Copyright 2013 Joey Hess <joey@kitenet.net>
+ -
+ - Licensed under the GNU GPL version 3 or higher.
+ -}
+
+module Git.CheckIgnore (
+	CheckIgnoreHandle,
+	checkIgnoreStart,
+	checkIgnoreStop,
+	checkIgnored
+) where
+
+import Common
+import Git
+import Git.Command
+import qualified Git.Version
+import qualified Utility.CoProcess as CoProcess
+
+import System.IO.Error
+
+type CheckIgnoreHandle = CoProcess.CoProcessHandle
+
+{- Starts git check-ignore running, and returns a handle.
+ -
+ - This relies on git check-ignore --non-matching -v outputting
+ - lines for both matching an non-matching files. Also relies on
+ - GIT_FLUSH behavior flushing the output buffer when git check-ignore
+ - is piping to us.
+ -
+ - The first version of git to support what we need is 1.8.4.
+ - Nothing is returned if an older git is installed.
+ -}
+checkIgnoreStart :: Repo -> IO (Maybe CheckIgnoreHandle)
+checkIgnoreStart repo = ifM supportedGitVersion
+	( Just <$> (CoProcess.rawMode =<< gitCoProcessStart True params repo)
+	, return Nothing
+	)
+  where
+	params =
+		[ Param "check-ignore" 
+		, Params "-z --stdin --verbose --non-matching"
+		]
+
+supportedGitVersion :: IO Bool
+supportedGitVersion = do
+	v <- Git.Version.installed
+	return $ v >= Git.Version.normalize "1.8.4"
+
+checkIgnoreStop :: CheckIgnoreHandle -> IO ()
+checkIgnoreStop = CoProcess.stop
+
+{- Returns True if a file is ignored. -}
+checkIgnored :: CheckIgnoreHandle -> FilePath -> IO Bool
+checkIgnored h file = CoProcess.query h send (receive "")
+  where
+	send to = do
+		hPutStr to $ file ++ "\0"
+		hFlush to
+	receive c from = do
+		s <- hGetSomeString from 1024
+		if null s
+			then eofError
+			else do
+				let v = c ++ s
+				maybe (receive v from) return (parse v)
+	parse s = case segment (== '\0') s of
+		(_source:_line:pattern:_pathname:_eol:[]) -> Just $ not $ null pattern
+		_ -> Nothing
+	eofError = ioError $ mkIOError userErrorType "git cat-file EOF" Nothing Nothing
diff --git a/debian/changelog b/debian/changelog
index 0a461c978..ecc850ab4 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+git-annex (4.20130803) UNRELEASED; urgency=low
+
+  * assistant, watcher: .gitignore files and other git ignores are now
+    honored, when git 1.8.4 or newer is installed.
+    (Thanks, Adam Spiers, for getting the necessary support into git for this.)
+
+ -- Joey Hess <joeyh@debian.org>  Fri, 02 Aug 2013 19:26:20 -0400
+
 git-annex (4.20130802) unstable; urgency=low
 
   * dropunused behavior change: Now refuses to drop the last copy of a
diff --git a/doc/bugs/assistant_ignore_.gitignore.mdwn b/doc/bugs/assistant_ignore_.gitignore.mdwn
index 00cdffd66..63d067e67 100644
--- a/doc/bugs/assistant_ignore_.gitignore.mdwn
+++ b/doc/bugs/assistant_ignore_.gitignore.mdwn
@@ -27,3 +27,5 @@ What version of git-annex are you using? On what operating system?
 > or a gitignore parser. --[[Joey]] 
 
 [[!tag /design/assistant]]
+
+> [[fixed|done]]; with git 1.8.4 the assistant honors .gitignore --[[Joey]]
diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn
index 11220c2ee..d1afe63c5 100644
--- a/doc/design/assistant/inotify.mdwn
+++ b/doc/design/assistant/inotify.mdwn
@@ -14,14 +14,6 @@ available!
 
 ## todo
 
-* Run niced and ioniced? Seems to make sense, this is a background job.
-* configurable option to only annex files meeting certian size or
-  filename criteria
-* option to check files not meeting annex criteria into git directly,
-  automatically
-* honor .gitignore, not adding files it excludes (difficult, probably
-  needs my own .gitignore parser to avoid excessive running of git commands
-  to check for ignored files)
 * There needs to be a way for a new version of git-annex, when installed,
   to restart any running watch or assistant daemons. Or for the daemons
   to somehow detect it's been upgraded and restart themselves. Needed