Add and use numcopiesneeded preferred content expression.

* Add numcopiesneeded preferred content expression.
* Client, transfer, incremental backup, and archive repositories
  now want to get content that does not yet have enough copies.

This means the asssistant will make copies of files that don't yet
meet the configured numcopies, even to places that would not normally want
the file.

For example, if numcopies is 4, and there are 2 client repos and
2 transfer repos, and 2 removable backup drives, the file will be sent
to both transfer repos in order to make 4 copies. Once a removable drive
get a copy of the file, it will be dropped from one transfer repo or the
other (but not both).

Another example, numcopies is 3 and there is a client that has a backup
removable drive and two small archive repos. Normally once one of the small
archives has a file, it will not be put into the other one. But, to satisfy
numcopies, the assistant will duplicate it into the other small archive
too, if the backup repo is not available to receive the file.

I notice that these examples are fairly unlikely setups .. the old behavior
was not too bad, but it's nice to finally have it really correct.

.. Almost. I have skipped checking the annex.numcopies .gitattributes
out of fear it will be too slow.

This commit was sponsored by Florian Schlegel.

1 files changed, 26 insertions, 1 deletions

diff --git a/Limit.hs b/Limit.hs
index fa6fa1f41..c0d32c68e 100644
--- a/Limit.hs
+++ b/Limit.hs
@@ -1,6 +1,6 @@
 {- user-specified limits on files to act on
  -
- - Copyright 2011-2013 Joey Hess <joey@kitenet.net>
+ - Copyright 2011-2014 Joey Hess <joey@kitenet.net>
  -
  - Licensed under the GNU GPL version 3 or higher.
  -}
@@ -23,6 +23,7 @@ import qualified Backend
 import Annex.Content
 import Annex.UUID
 import Logs.Trust
+import Logs.NumCopies
 import Types.TrustLevel
 import Types.Key
 import Types.Group
@@ -177,6 +178,30 @@ limitCopies want = case split ":" want of
 		| "+" `isSuffixOf` s = (>=) <$> readTrustLevel (beginning s)
 		| otherwise = (==) <$> readTrustLevel s
 
+{- Adds a limit to match files that need more copies made.
+ -
+ - Does not look at annex.numcopies .gitattributes, because that
+ - would require querying git check-attr every time a preferred content
+ - expression is checked, which would probably be quite slow.
+ -}
+addNumCopiesNeeded :: String -> Annex ()
+addNumCopiesNeeded = addLimit . limitNumCopiesNeeded
+
+limitNumCopiesNeeded :: MkLimit
+limitNumCopiesNeeded want = case readish want of
+	Just needed -> Right $ \notpresent -> checkKey $
+		handle needed notpresent
+	Nothing -> Left "bad value for numcopiesneeded"
+  where
+	handle needed notpresent key = do
+		gv <- getGlobalNumCopies
+		case gv of
+			Nothing -> return False
+			Just numcopies -> do
+				us <- filter (`S.notMember` notpresent)
+					<$> (trustExclude UnTrusted =<< Remote.keyLocations key)
+				return $ numcopies - length us >= needed
+
 {- Adds a limit to skip files not believed to be present in all
  - repositories in the specified group. -}
 addInAllGroup :: String -> Annex ()