move bug out of forum and close

9 files changed, 96 insertions, 0 deletions

diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_1_e456604b26ed9c72b0a88cfb57f1a475._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_1_e456604b26ed9c72b0a88cfb57f1a475._comment
new file mode 100644
index 000000000..dba07bfd5
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_1_e456604b26ed9c72b0a88cfb57f1a475._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
+ nickname="Justin"
+ subject="comment 1"
+ date="2014-03-05T16:11:27Z"
+ content="""
+I tried git annex sync --content, and it failed to copy some files with
+
+  git: createProcess: resource exhausted (Resource temporarily unavailable)
+
+So this sounds like fork is failing; I'm probably exhausting my poor pi's RAM.  Maybe the same thing is happening for git annex copy.  I'll run strace to see.
+"""]]
diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_2_4823d66bfb569605868af5cefe0d94dc._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_2_4823d66bfb569605868af5cefe0d94dc._comment
new file mode 100644
index 000000000..5bcd0e92e
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_2_4823d66bfb569605868af5cefe0d94dc._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="209.250.56.146"
+ subject="comment 2"
+ date="2014-03-05T20:31:57Z"
+ content="""
+How many files copied are we talking about before it begins to fail?
+
+You can try passing --debug, which will make git-annex show every external command it runs, which includes `cp` for a copy to another repo on the same machine.
+
+Might also check memory usage in top during the run.
+"""]]
diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_3_46305aa2d43da000c1a7cb003c822572._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_3_46305aa2d43da000c1a7cb003c822572._comment
new file mode 100644
index 000000000..297c28a40
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_3_46305aa2d43da000c1a7cb003c822572._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
+ nickname="Justin"
+ subject="comment 3"
+ date="2014-03-06T18:21:53Z"
+ content="""
+> How many files copied are we talking about before it begins to fail?
+
+Tens of thousands of files processed, but many of them were already on the other remote so didn't invoke cp (or anything else).  ~3300 invocations of cp.
+
+I saved a log of ps aux, and, while the memory used by git annex remains relatively constant, I do observe /tons/ of zombie processes.  3300, actually.
+
+I didn't check all of them, but all of the zombie pids I checked appear to have corresponded to this command:
+
+    /home/pi/git-annex.linux/shimmed/git/git --git-dir=/home/pi/hdd/annex/.git --work-tree=/home/pi/hdd/annex cat-file --batch
+
+Perhaps git annex is forgetting to reap this processes?
+"""]]
diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_4_1dbdeded7f587e8fc2d1ac5170ecb928._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_4_1dbdeded7f587e8fc2d1ac5170ecb928._comment
new file mode 100644
index 000000000..537736a72
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_4_1dbdeded7f587e8fc2d1ac5170ecb928._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="209.250.56.146"
+ subject="comment 4"
+ date="2014-03-06T18:32:33Z"
+ content="""
+Old versions of git-annex have known bugs involving zombies. What version?
+"""]]
diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_5_1e0c06a07345d85b3712339e6f0d9a9f._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_5_1e0c06a07345d85b3712339e6f0d9a9f._comment
new file mode 100644
index 000000000..ad2b80d66
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_5_1e0c06a07345d85b3712339e6f0d9a9f._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
+ nickname="Justin"
+ subject="comment 5"
+ date="2014-03-06T18:35:00Z"
+ content="""
+5.20140221-g1a47f5f -- I just downloaded it a week or two ago.
+"""]]
diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_6_41798e92068eb227c5e75cae2edef68a._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_6_41798e92068eb227c5e75cae2edef68a._comment
new file mode 100644
index 000000000..a0554cd8a
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_6_41798e92068eb227c5e75cae2edef68a._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="209.250.56.146"
+ subject="comment 6"
+ date="2014-03-06T18:38:43Z"
+ content="""
+Hmm, that version should only start git cat-file --batch a maximum of 10 times (if it is crashing for some reason), and appears to wait on the process if it does crash. And if not, should only start one.
+
+I think you need to post some git-annex --debug output , to show when it's running this command.
+"""]]
diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_7_1f33d694a08d8dcbf04595e3442b8cd5._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_7_1f33d694a08d8dcbf04595e3442b8cd5._comment
new file mode 100644
index 000000000..f846e290d
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_7_1f33d694a08d8dcbf04595e3442b8cd5._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="209.250.56.146"
+ subject="comment 7"
+ date="2014-03-06T18:40:26Z"
+ content="""
+Actually, NM, I have reproduced the bug.
+"""]]
diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_8_884f31ce917c8e5ce9a32a55da9b42d6._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_8_884f31ce917c8e5ce9a32a55da9b42d6._comment
new file mode 100644
index 000000000..5ba7442dc
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_8_884f31ce917c8e5ce9a32a55da9b42d6._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="209.250.56.146"
+ subject="comment 8"
+ date="2014-03-06T20:21:16Z"
+ content="""
+Analysis: Remote.Git's onLocal calls Annex.new to make a new AnnexState for the local remote. This state is not cached, and is regenerated for each file. Since it runs a Annex.Branch check of the location log on the remote, it needs to start catFile, and since the state is not reused, a new CatFileHandle is allocated each time. I'm not sure, but there may have been a recent-ish change that caused the location log to get checked and so catfile to be run; the general inneficiency of making a new AnnexState each time is not new.
+
+Fixing this by caching the AnnexState will not only fix the resource leak, but should speed up local to local copies significantly!
+"""]]
diff --git a/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_9_ab770dafee3bd9212f553db222adbfe6._comment b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_9_ab770dafee3bd9212f553db222adbfe6._comment
new file mode 100644
index 000000000..e6a23eed5
--- /dev/null
+++ b/doc/bugs/copy_fails_for_some_fails_without_explanation/comment_9_ab770dafee3bd9212f553db222adbfe6._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="209.250.56.146"
+ subject="comment 9"
+ date="2014-03-06T21:17:14Z"
+ content="""
+Fixed in git. Also reduced the non-data-transfer work done by `git-annex copy` by around 8%.
+
+I'm going to move this thread to [[bugs]] so I can close it. ;)
+"""]]