From 164466c987a7607a5f598b36e5b3111a68bd101f Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Tue, 31 Jan 2017 17:43:11 -0400 Subject: fix lack of laziness streaming large diffs A commit last year that made a partial function use Maybe unfortunately caused the whole input to need to be consumed, breaking streaming. So, revert it. This commit was sponsored by Nick Daly on Patreon. --- ...omment_2_1eff4497bf0a3e87dc47a1226c5d4af8._comment | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 doc/todo/more_efficient_memory_usage_with_git-annex_unused/comment_2_1eff4497bf0a3e87dc47a1226c5d4af8._comment (limited to 'doc') diff --git a/doc/todo/more_efficient_memory_usage_with_git-annex_unused/comment_2_1eff4497bf0a3e87dc47a1226c5d4af8._comment b/doc/todo/more_efficient_memory_usage_with_git-annex_unused/comment_2_1eff4497bf0a3e87dc47a1226c5d4af8._comment new file mode 100644 index 000000000..df285478c --- /dev/null +++ b/doc/todo/more_efficient_memory_usage_with_git-annex_unused/comment_2_1eff4497bf0a3e87dc47a1226c5d4af8._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2017-01-31T20:24:04Z" + content=""" +The heap profile has multiple spikes (so not an accumulating memory leak). +The diff parsing code is indeed what's using so much memory. Looks like +data is failing to stream through that code and instead the whole +diff output gets buffered. + +Aha.. Git.DiffTree.parseDiffRaw used to return a list, but changed +in [[!commit 8d124beba8]] +to a Maybe list in order to avoid being a partial function. But +that change destroyed laziness, since the whole input has to be parsed +in order to determine if Nothing should be returned. + +However, fixing that only eliminated part of the spike. There's something +else keeping data from streaming. +"""]] -- cgit v1.2.3