| Commit message (Collapse) | Author | Age |
|---|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| | |
This is the last memory leak that prevents git-annex from running
in constant space, as far as I can see. I can now run git annex find
dummied up to repeatedly find the same file over and over, on millions
olf files, and memory stays entirely constant.
|
| | | |
|
| | |
| |
| |
| | |
(cherry picked from commit c8ec0e233e9d47a7e69b3de2952099c221c79ac1)
|
| |/ |
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
useful when adding hundreds of thousands of files on a system with plenty
of memory.
git add gets quite slow in such a large repository, so if the system has
more than the ~32 mb of memory the queue can use by default, it's a useful
optimisation to increase the queue size, in order to decrease the number
of times git add is run.
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
The list of files had to be retained until the end so it could be deleted.
Also, a list of update-index lines was generated and only then fed into it.
Now everything streams in constant space.
|
| |\ |
|
| | | |
|
| |/
|
|
| |
Avoids ugly (and test suite failing) hack in Command.Version
|
| | |
|
| | |
|
| |
|
|
|
|
| |
When hashing the files, the entire list of shas was read strictly.
That was entirely unnecessary, since there's a cleanup action run
after they're consumed.
|
| | |
|
| |
|
|
| |
This is not the place to commit journal files.
|
| |
|
|
| |
Seemed to work without it, but this is correct.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
Passing absolute paths to Command.Add used to work, but after recent
changes doesn't. All LsFiles should use relative paths anyway, so fix it
there.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now gitattributes are looked up, efficiently, in only the places that
really need them, using the same approach used for cat-file.
The old CheckAttr code seemed very fragile, in the way it streamed files
through git check-attr.
I actually found that cad8824852aa0623dc41eac02a9e2bae47d88ec4
was still deadlocking with ghc 7.4, at the end of adding a lot of files.
This should fix that problem, and avoid future ones.
The best part is that this removes withAttrFilesInGit and withNumCopies,
which were complicated Seek methods, as well as simplfying the types
for several other Seek methods that had a Backend tupled in.
|
| | |
|
| |
|
|
|
| |
I removed the now unnecessary forkProcess, but forgot to change back to
pipeBoth, so there was no writer thread.
|
| |
|
|
|
|
| |
When converting to the strict state monad, I missed this place where
thunks to the state could be built up, possibly. This seems to make
it run in some percentage less memory.
|
| |
|
|
|
|
|
|
|
|
| |
Turns out that commit really made some serious improvements to memory use.
With the lazy state monad, git-annex add in a huge tree grew seemingly
without bound until it overflowed the stack. With the strict monad,
it uses 42 mb max.
It's possible another change since the 3.20120123 release fixed that,
but a964012fc36d22e4554dd12e3594579fb3190501 seems most likely.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
a redirect
The code explicitly switches from HEAD to GET for most redirects.
Possibly because someone misread a spec (which does require switching from
POST to GET for 303 redirects). Or possibly because the spec really is that
bad. Upstream bug: https://github.com/haskell/HTTP/issues/24
Since we absolutely don't want to download entire (large) files from
the web when checking that they exist with HEAD, I wrote my own redirect
follower, based closely on the one used by Network.Browser, but without
this misfeature.
Note that Network.Browser checks that the redirect url is a http url
and fails if not. I don't, because I want to not need to change this
code when it gets https support (related: I'm surprised to see it
doesn't support https yet..). The check does not seem security significant;
it doesn't support file:// urls for example. If a http url is redirected
to https, the Network.Browser will actually make a http connection again.
This could loop, but only up to 5 times.
|
| | |
|
| | |
|
| |
|
|
| |
head), and records the size in the key.
|
| |
|
|
|
|
|
|
| |
available, matches the size of the key.
If there's no Content-Length, or the key has no size, this check is not
done, but it should happen most of the time, and protect against web
content that has changed.
|
| |\ |
|
| | | |
|
| | | |
|
| | | |
|
Joey Hess
http://joey.kitenet.net/
antymat