Add and use numcopiesneeded preferred content expression.

* Add numcopiesneeded preferred content expression.
* Client, transfer, incremental backup, and archive repositories
  now want to get content that does not yet have enough copies.

This means the asssistant will make copies of files that don't yet
meet the configured numcopies, even to places that would not normally want
the file.

For example, if numcopies is 4, and there are 2 client repos and
2 transfer repos, and 2 removable backup drives, the file will be sent
to both transfer repos in order to make 4 copies. Once a removable drive
get a copy of the file, it will be dropped from one transfer repo or the
other (but not both).

Another example, numcopies is 3 and there is a client that has a backup
removable drive and two small archive repos. Normally once one of the small
archives has a file, it will not be put into the other one. But, to satisfy
numcopies, the assistant will duplicate it into the other small archive
too, if the backup repo is not available to receive the file.

I notice that these examples are fairly unlikely setups .. the old behavior
was not too bad, but it's nice to finally have it really correct.

.. Almost. I have skipped checking the annex.numcopies .gitattributes
out of fear it will be too slow.

This commit was sponsored by Florian Schlegel.

3 files changed, 18 insertions, 6 deletions

diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 4e7bd2395..8a948d303 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -1020,6 +1020,15 @@ file contents are present at either of two repositories.
   copies, on remotes in the specified group. For example,
   `--copies=archive:2`
 
+* `--numcopiesneeded=number`
+
+  Matches only files that git-annex believes need the specified number or 
+  more additional copies to be made in order to satisfy their numcopies
+  setting, as configured by the global numcopies setting of the repository.
+
+  Note that for various reasons, including speed, this does not look 
+  at the annex.numcopies .gitattributes settings of files.
+
 * `--inbackend=name`
 
   Matches only files whose content is stored using the specified key-value
diff --git a/doc/preferred_content.mdwn b/doc/preferred_content.mdwn
index 9c698c8ba..b18f46c33 100644
--- a/doc/preferred_content.mdwn
+++ b/doc/preferred_content.mdwn
@@ -113,7 +113,7 @@ any repository that can will back it up.)
 All content is preferred, unless it's for a file in a "archive" directory,
 which has reached an archive repository.
 
-`((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) or (not copies=semitrusted+:1)`
+`((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) or numcopiesneeded=1`
 
 ### transfer
 
@@ -147,20 +147,20 @@ All content is preferred.
 Only prefers content that's not already backed up to another backup
 or incremental backup repository.
 
-`(include=* and (not copies=backup:1) and (not copies=incrementalbackup:1)) or (not copies=semitrusted+:1)`
+`(include=* and (not copies=backup:1) and (not copies=incrementalbackup:1)) or numcopiesneeded=1`
 
 ### small archive
 
 Only prefers content that's located in an "archive" directory, and
 only if it's not already been archived somewhere else.
 
-`((include=*/archive/* or include=archive/*) and not (copies=archive:1 or copies=smallarchive:1)) or (not copies=semitrusted+:1)`
+`((include=*/archive/* or include=archive/*) and not (copies=archive:1 or copies=smallarchive:1)) or numcopiesneeded=1`
 
 ### full archive
 
 All content is preferred, unless it's already been archived somewhere else.
 
-`(not (copies=archive:1 or copies=smallarchive:1)) or (not copies=semitrusted+:1)`
+`(not (copies=archive:1 or copies=smallarchive:1)) or numcopiesneeded=1`
 
 Note that if you want to archive multiple copies (not a bad idea!),
 you should instead configure all your archive repositories with a
diff --git a/doc/todo/preferred_content_numcopies_check.mdwn b/doc/todo/preferred_content_numcopies_check.mdwn
index 152afe08c..8aa736a04 100644
--- a/doc/todo/preferred_content_numcopies_check.mdwn
+++ b/doc/todo/preferred_content_numcopies_check.mdwn
@@ -54,9 +54,12 @@ Conclusion:
 * Add "numcopiesneeded=N" preferred content expression using the git-annex
   branch numcopies setting, overridden by any .gitattributes numcopies setting
   for a particular file. It should ignore the other ways to specify
-  numcopies.
+  numcopies, particularly git config annex.numcopies. **done**
 * Make the repo groups that currently end with "or (not copies=semitrusted+:1)"
-  to instead end with "or numcopiesneeded=1"
+  to instead end with "or numcopiesneeded=1" **done**
+* See if "numcopiesneeded=N" can check .gitattributes without getting
+  a lot slower. If now, perhaps add a "numcopiesneededaccurate=N" that
+  checks it.
 
 ## Stability analysis