close as dup

3 files changed, 27 insertions, 0 deletions

diff --git a/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository.mdwn b/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository.mdwn
index ae5c0e6e4..568b2bf5f 100644
--- a/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository.mdwn
+++ b/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository.mdwn
@@ -53,3 +53,5 @@ I have no idea why it needs to do that, though.
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 
 I've been using git-annex v5 repositories without any issues, and with smaller files, repository v6 works great!
+
+> dup; [[done]] --[[Joey]]
diff --git a/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository/comment_1_b9154c38406fca40c4c0edb716707c3a._comment b/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository/comment_1_b9154c38406fca40c4c0edb716707c3a._comment
new file mode 100644
index 000000000..9ef59ed71
--- /dev/null
+++ b/doc/bugs/Out_of_memory_error_when_adding_large_files_to_v6_repository/comment_1_b9154c38406fca40c4c0edb716707c3a._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2017-06-09T17:34:40Z"
+ content="""
+Unfortunately, git add tries to load the whole file content
+into memory (or at least allocates a buffer for it all),
+even if it's only going to stream it through the clean filter
+used in v6 mode, and even though the git-annex smudge
+filter reads the file content from disk itself.
+
+I submitted a patch to git over a year ago, that IIRC fixes this kind of
+problem, but it was not accepted. Getting an updated version of that patch
+accepted into git is the main blocker for v6 repositories to not be
+experimental.
+
+[[todo/smudge]] documents this problem. I'm going to close this bug
+as it's a duplicate of stuff discussed there.
+"""]]
diff --git a/doc/todo/smudge.mdwn b/doc/todo/smudge.mdwn
index 611722490..266a5d593 100644
--- a/doc/todo/smudge.mdwn
+++ b/doc/todo/smudge.mdwn
@@ -71,6 +71,12 @@ git-annex should use smudge/clean filters.
   avoid the problem for git checkout, since it would use the new interface
   and not the smudge filter.)
 
+* When `git add` is run with a large file, it allocates memory for
+  the whole file content, even though it's only going
+  to stream it to the clean filter. My proposed smudge/clean
+  interface patch also fixed this problem, since it made git not read
+  the file at all.
+
 * Eventually (but not yet), make v6 the default for new repositories.
   Note that the assistant forces repos into direct mode; that will need to
   be changed then, and it should enable annex.thin instead.