| Commit message (Collapse) | Author | Age |
|---|
| ... | |
| |
|
|
|
|
| |
Stale and bad files are rare, so it's more efficient to use inAnnex to see
if they can be deleted, rather than keeping the list of all present keys
around for them.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
.. Allowing it to be used by things in constant space!
Random statistics: git annex status has gone from taking 239 mb
of memory and 26 seconds in a repo, to 8 mb and 13 seconds.
The trick here is the unsafeInterleaveIO, and the form of the function's
recursion, which I cribbed heavily from System.IO.HVFS.Utils.recurseDirStat.
The difference is, this one goes to a limited depth and avoids statting
everything.
|
| |
|
|
|
|
|
|
| |
Before, it leaked space due to caching lists of keys. Now all necessary
data about keys is calculated as they stream in.
The "nearly constant" is due to getKeysPresent, which builds up a lot
of [] thunks as it traverses .git/annex/objects/. Will deal with it later.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Much of the memory bloat turned out to be due to getKeysReferenced
containing a mapM, which is strict and buffered the whole list
rather than streaming it.
The other half of the bloat was due to building a temporary Set
in order to call S.difference. While that is more cpu efficient,
I switched to successive S.delete, since with it, I can run a whole
git annex unused in less than 8 mb of memory.
The whole Set of keys with content available is still stored in memory,
so running unused in a repo with a whole lot of file content will still
use more memory. In a repo containing 6000 files, it needed 40 mb.
Note that the status command still uses the bloatful getKeysReferenced.
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| | | |
|
| |/ |
|
| |
|
|
|
|
|
|
|
| |
This has two benefits.
1. When a lot of refs are going to be received, get them via lower cost
connection when possible.
2. Allows ctrl-c of sync after the cheaper remotes have been pulled from
(or pushed to).
|
| |
|
|
| |
hash calculation bug fixed in the last release.
|
| |
|
|
|
|
|
|
|
|
| |
Rather than running make, which runs configure, let Setup.hs just include
the configure code. The standalone configure is retained for use by the
Makefile.
This may work better with cabal-dev, since it avoids the Makefile running
ghc, and lets cabal handle all the compiler running, with whatever
flags it uses to expose dependencies.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix Key directory hash calculation code to behave as it did before version
3.20120227 when a key contains non-ascii.
The hash directories for a given Key are based on its md5sum.
Prior to ghc 7.4, Keys contained raw, undecoded bytes, so the md5sum was
taken of each byte in turn. With the ghc 7.4 filename encoding change,
keys contains decoded unicode characters (possibly with surrigates for
undecodable bytes). This changes the result of the md5sum, since the md5sum
used is pure haskell and supports unicode. And that won't do, as git-annex
will start looking in a different hash directory for the content of a key.
The surrigates are particularly bad, since that's essentially a ghc
implementation detail, so could change again at any time. Also, changing
the locale changes how the bytes are decoded, which can also change
the md5sum.
Symptoms would include things like:
* git annex fsck would complain that no copies existed of a file,
despite its symlink pointing to the content that was locally present
* git annex fix would change the symlink to use the wrong hash
directory.
Only WORM backend is likely to have been affected, since only it tends
to include much filename data (SHA1E could in theory also be affected).
I have not tried to support the hash directories used by git-annex versions
3.20120227 to 3.20120308, so things added with those versions with WORM
will require manual fixups. Sorry for the inconvenience!
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| | |
addurl --fast used to avoid network, but it always uses it now, getting at
least size. Thus not appropriate for test suite without a lot of work.
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
annexed file is in a subdirectory that nearly matched to the .git/annex/object/xx/yy subdirectories.
This is a straight up pure-code stinker. The relative path calculation
looked for common subdirectories in the two paths, but failed to stop
after the paths diverged. When a later pair of subdirectories were the
same, the resulting relative path was wrong.
Added regression test for this.
|
| |
|
|
|
|
| |
Locking is used, so that, if there are multiple git-annex processes
using a remote concurrently, the stop hook is only run by the last
process that uses it.
|
| | |
|
| |
|
|
|
|
| |
Avoids opening every chunk at once, instead streaming them in.
Not done for encrypted file retrieval yet.
|
| | |
|
| |
|
|
|
| |
Seeing some weird failures in the davfs2 or box.com layer with 100 mb
chunks.
|
| | |
|
| | |
|
| |
|
|
|
| |
Allows showing progress bar for this last case of the directory special
remote.
|
| |
|
|
|
|
| |
That was actually really easy. But, when getting a file from an encrypted
directory special remote, no meter can be shown, because the total file
size is not known.
|
| |
|
|
|
|
|
| |
So far I've only written progress bars for sending files, not yet
receiving.
No longer uses external cp at all. ByteString IO is fast enough.
|
| |
|
|
|
| |
I was sucked in by the 50 gb free lifetime storage offer.
Happily, it was pretty easy to get it to work with git-annex.
|
| | |
|
Joey Hess
http://joey.kitenet.net/
http://claimid.com/FooBarWidget
https://www.google.com/accounts/o8/id?id=AItOawk_LOahrm_Cdg7io-_H0CNKkaxsRRQgRFo
http://peter-simons.myopenid.com/