1 files changed, 441 insertions, 0 deletions

diff --git a/doc/bugs/fails_to_handle_lot_of_files.mdwn b/doc/bugs/fails_to_handle_lot_of_files.mdwn
new file mode 100644
index 000000000..5c5a6c8fd
--- /dev/null
+++ b/doc/bugs/fails_to_handle_lot_of_files.mdwn
@@ -0,0 +1,441 @@
+git-annex version: 3.20111011
+local repository version: 3
+default repository version: 3
+supported repository versions: 3
+upgrade supported from repository versions: 0 1 2
+
+I just created a new remote on a USB drive and wanted to copy my files over. git-annex wasn't too happy about that ;)
+I included a few OK transfers as there was an error before git-annex ran into a wall. As I could easily access that temp file after it aborted, I suspect something went wrong internally before git-annex started to throw those errors.
+
+Please note the "_n TIMES_" comments. It's how often I got the same error message...
+
+
+
+    git annex copy . --to USB --fast
+    
+    copy redacted.JPG (to USB...) 
+    redacted
+         4035668 100%   77.91MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4036374 bytes  received 31 bytes  8072810.00 bytes/sec
+    total size is 4035668  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18002094 100%   74.19MB/s    0:00:00 (xfer#1, to-check=0/1)
+    WARNING: redacted failed verification -- update retained (will try again).
+    redacted
+        18002094 100%   19.60MB/s    0:00:00 (xfer#2, to-check=0/1)
+    rsync: open "copy_target/.git/annex/tmp/redacted_E13" failed: Permission denied (13)
+    
+    sent 36008841 bytes  received 52 bytes  24005928.67 bytes/sec
+    total size is 18002094  speedup is 0.50
+    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.8]
+    
+      rsync failed -- run git annex again to resume file transfer
+    failed
+    copy redacted.JPG (to USB...) 
+    redacted
+         3687111 100%   39.16MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 3687773 bytes  received 31 bytes  2458536.00 bytes/sec
+    total size is 3687111  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17877177 100%   79.15MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17879573 bytes  received 31 bytes  11919736.00 bytes/sec
+    total size is 17877177  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         3694921 100%   40.14MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 3695583 bytes  received 31 bytes  2463742.67 bytes/sec
+    total size is 3694921  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17875448 100%   71.20MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17877844 bytes  received 31 bytes  11918583.33 bytes/sec
+    total size is 17875448  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         3833377 100%   62.49MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 3834055 bytes  received 31 bytes  2556057.33 bytes/sec
+    total size is 3833377  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17938200 100%   65.43MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17940604 bytes  received 31 bytes  11960423.33 bytes/sec
+    total size is 17938200  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4512557 100%   83.77MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4513319 bytes  received 31 bytes  3008900.00 bytes/sec
+    total size is 4512557  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18001641 100%   76.16MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18004053 bytes  received 31 bytes  12002722.67 bytes/sec
+    total size is 18001641  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4394272 100%   50.11MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4395022 bytes  received 31 bytes  8790106.00 bytes/sec
+    total size is 4394272  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18095781 100%   73.30MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18098205 bytes  received 31 bytes  12065490.67 bytes/sec
+    total size is 18095781  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4683795 100%   65.23MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4684577 bytes  received 31 bytes  9369216.00 bytes/sec
+    total size is 4683795  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18172801 100%   74.25MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18175233 bytes  received 31 bytes  36350528.00 bytes/sec
+    total size is 18172801  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4486231 100%   77.22MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4486989 bytes  received 31 bytes  8974040.00 bytes/sec
+    total size is 4486231  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17860427 100%   68.56MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17862823 bytes  received 31 bytes  35725708.00 bytes/sec
+    total size is 17860427  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4499768 100%   36.41MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4500530 bytes  received 31 bytes  9001122.00 bytes/sec
+    total size is 4499768  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17840132 100%   74.48MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17842524 bytes  received 31 bytes  11895036.67 bytes/sec
+    total size is 17840132  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4358032 100%   75.00MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4358774 bytes  received 31 bytes  8717610.00 bytes/sec
+    total size is 4358032  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18084753 100%   61.48MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18087173 bytes  received 31 bytes  12058136.00 bytes/sec
+    total size is 18084753  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4270213 100%   68.49MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4270947 bytes  received 31 bytes  2847318.67 bytes/sec
+    total size is 4270213  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17661246 100%   68.34MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17663614 bytes  received 31 bytes  11775763.33 bytes/sec
+    total size is 17661246  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4538305 100%   63.19MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4539071 bytes  received 31 bytes  9078204.00 bytes/sec
+    total size is 4538305  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18672466 100%   68.90MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18674958 bytes  received 31 bytes  12449992.67 bytes/sec
+    total size is 18672466  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4453445 100%   73.96MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4454199 bytes  received 31 bytes  8908460.00 bytes/sec
+    total size is 4453445  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18495494 100%   59.28MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18497966 bytes  received 31 bytes  12331998.00 bytes/sec
+    total size is 18495494  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4255858 100%   70.66MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4256588 bytes  received 31 bytes  1702647.60 bytes/sec
+    total size is 4255858  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18376531 100%   69.15MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18378987 bytes  received 31 bytes  36758036.00 bytes/sec
+    total size is 18376531  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4013365 100%   48.67MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4014067 bytes  received 31 bytes  8028196.00 bytes/sec
+    total size is 4013365  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17606341 100%   51.73MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17608705 bytes  received 31 bytes  11739157.33 bytes/sec
+    total size is 17606341  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4179869 100%   74.62MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4180591 bytes  received 31 bytes  8361244.00 bytes/sec
+    total size is 4179869  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18382569 100%   67.05MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18385025 bytes  received 31 bytes  12256704.00 bytes/sec
+    total size is 18382569  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4318363 100%   44.91MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4319101 bytes  received 31 bytes  8638264.00 bytes/sec
+    total size is 4318363  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17715958 100%   72.69MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17718334 bytes  received 31 bytes  11812243.33 bytes/sec
+    total size is 17715958  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4241893 100%   65.81MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4242623 bytes  received 31 bytes  8485308.00 bytes/sec
+    total size is 4241893  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17717287 100%   71.77MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17719663 bytes  received 31 bytes  11813129.33 bytes/sec
+    total size is 17717287  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4488380 100%   49.99MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4489138 bytes  received 31 bytes  2992779.33 bytes/sec
+    total size is 4488380  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        17770208 100%   38.80MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 17772592 bytes  received 31 bytes  11848415.33 bytes/sec
+    total size is 17770208  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4603958 100%   76.48MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4604732 bytes  received 31 bytes  9209526.00 bytes/sec
+    total size is 4603958  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18744380 100%   74.66MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18746884 bytes  received 31 bytes  12497943.33 bytes/sec
+    total size is 18744380  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4592098 100%   79.06MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4592872 bytes  received 31 bytes  3061935.33 bytes/sec
+    total size is 4592098  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18746205 100%   43.00MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18748709 bytes  received 31 bytes  12499160.00 bytes/sec
+    total size is 18746205  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         7493353 100%   80.85MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 7494479 bytes  received 31 bytes  14989020.00 bytes/sec
+    total size is 7493353  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        19496768 100%   81.77MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 19499360 bytes  received 31 bytes  12999594.00 bytes/sec
+    total size is 19496768  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         5462482 100%   82.19MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 5463360 bytes  received 31 bytes  10926782.00 bytes/sec
+    total size is 5462482  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        19669815 100%   80.37MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 19672431 bytes  received 31 bytes  13114974.67 bytes/sec
+    total size is 19669815  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         5449487 100%   57.40MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 5450365 bytes  received 31 bytes  3633597.33 bytes/sec
+    total size is 5449487  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        19633259 100%   74.18MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 19635871 bytes  received 31 bytes  13090601.33 bytes/sec
+    total size is 19633259  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         5392184 100%   62.33MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 5393054 bytes  received 31 bytes  3595390.00 bytes/sec
+    total size is 5392184  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        18912104 100%   65.00MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 18914628 bytes  received 31 bytes  12609772.67 bytes/sec
+    total size is 18912104  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4869300 100%   80.92MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4870106 bytes  received 31 bytes  9740274.00 bytes/sec
+    total size is 4869300  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        20178932 100%   68.13MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 20181608 bytes  received 31 bytes  13454426.00 bytes/sec
+    total size is 20178932  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         4995425 100%   86.05MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 4996247 bytes  received 31 bytes  9992556.00 bytes/sec
+    total size is 4995425  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        19970679 100%   76.36MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 19973331 bytes  received 31 bytes  13315574.67 bytes/sec
+    total size is 19970679  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         7905795 100%   66.45MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 7906973 bytes  received 31 bytes  15814008.00 bytes/sec
+    total size is 7905795  speedup is 1.00
+    ok
+    copy redacted.NEF (to USB...) 
+    redacted
+        21234069 100%   78.07MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 21236877 bytes  received 31 bytes  8494763.20 bytes/sec
+    total size is 21234069  speedup is 1.00
+    ok
+    copy redacted.JPG (to USB...) 
+    redacted
+         7963979 100%   62.51MB/s    0:00:00 (xfer#1, to-check=0/1)
+    
+    sent 7965165 bytes  received 31 bytes  5310130.67 bytes/sec
+    total size is 7963979  speedup is 1.00
+    git ["--git-dir=copy_target/.git","--work-tree=copy_target","update-index","-z","--index-info"]: Error in fork: forkProcess: resource exhausted (Resource temporarily unavailable)
+    
+    git-annex: user error (git ["--git-dir=copy_target/.git","--work-tree=copy_target","update-index","-z","--index-info"]: Error in fork: forkProcess: resource exhausted (Resource temporarily unavailable))
+    failed
+    _506 TIMES_ (user error (Error in fork: forkProcess: resource exhausted (Resource temporarily unavailable))) failed
+    _11 TIMES_ copy foo (createPipe: resource exhausted (Too many open files)) failed
+    _2 TIMES_ (user error (Error in fork: forkProcess: resource exhausted (Resource temporarily unavailable))) failed
+    _8574 TIMES_: copy foo (createPipe: resource exhausted (Too many open files)) failed
+    git-annex: createPipe: resource exhausted (Too many open files)
+    failed
+    git-annex: 9101 failed
+    
+    % ls copy_target/.git/annex/tmp/redacted_E13 copy_target/.git/annex/tmp/SHA512E-redacted_E13 # works
+    % find source -type l | wc -l
+    13554
+    % find copy_target -type l | wc -l
+    13554
+    % find copy_target/.git/annex/objects -type f | wc -l
+    4455
+    % find source -type f | wc -l
+    13554