| Commit message (Collapse) | Author | Age |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
useful when adding hundreds of thousands of files on a system with plenty
of memory.
git add gets quite slow in such a large repository, so if the system has
more than the ~32 mb of memory the queue can use by default, it's a useful
optimisation to increase the queue size, in order to decrease the number
of times git add is run.
|
| |
|
|
|
|
| |
The list of files had to be retained until the end so it could be deleted.
Also, a list of update-index lines was generated and only then fed into it.
Now everything streams in constant space.
|
| |
|
|
|
|
| |
When hashing the files, the entire list of shas was read strictly.
That was entirely unnecessary, since there's a cleanup action run
after they're consumed.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Turns out that commit really made some serious improvements to memory use.
With the lazy state monad, git-annex add in a huge tree grew seemingly
without bound until it overflowed the stack. With the strict monad,
it uses 42 mb max.
It's possible another change since the 3.20120123 release fixed that,
but a964012fc36d22e4554dd12e3594579fb3190501 seems most likely.
|
| |
|
|
| |
head), and records the size in the key.
|
| |
|
|
|
|
|
|
| |
available, matches the size of the key.
If there's no Content-Length, or the key has no size, this check is not
done, but it should happen most of the time, and protect against web
content that has changed.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Can be used to specify what file the url is added to. This can be used to
override the default filename that is used when adding an url, which is
based on the url. Or, when the file already exists, the url is recorded as
another location of the file.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Done by adding a oneshot mode, in which location log changes are written to
the journal, but not committed. Taking advantage of git-annex's existing
ability to recover in this situation.
This is used by git-annex-shell and other places where changes are made to
a remote's location log.
|
| |
|
|
|
|
|
| |
This drops the >>! and >>? with the nice low fixity. IfElse does have
undocumented >>=>>! and >>=>>? operators, but I deem that too fishy.
Anyway, using whenM and unlessM is easier; I sometimes mixed the operators
up.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Ssh connection caching is now enabled automatically by git-annex. Only one
ssh connection is made to each host per git-annex run, which can speed some
things up a lot, as well as avoiding repeated password prompts. Concurrent
git-annex processes also share ssh connections. Cached ssh connections are
shut down when git-annex exits.
Note: The rsync special remote does not yet participate in the ssh
connection caching.
|
| |
|
|
|
|
|
| |
Avoids expensive file transfers, at the expense of checking file size
and/or contents.
Required some reworking of the remote code.
|
| |
|
|
|
|
|
|
|
| |
Add news item recommending fscking directory special remotes.
Remote news item about URL backend being removed; it was later added back
to be used by git annex addurl --fast.
Link NEWS into top level.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fscking a remote is now supported. It's done by retrieving
the contents of the specified files from the remote, and checking them,
so can be an expensive operation.
(Several optimisations are possible, to speed it up, of course.. This is
the slow and stupid remote fsck to start with.)
Still, if the remote is a special remote, or a git repository that you
cannot run fsck in locally, it's nice to have the ability to fsck it.
If you have any directory special remotes, now would be a good time to
fsck them, in case you were hit by the data loss bug fixed in the
previous release!
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When moving a file to the remote failed, and partially transferred content
was left behind in the directory, re-running the same move would think it
succeeded and delete the local copy.
I reproduced data loss when moving files to a partition that was almost
full. Interrupting a transfer could have similar results.
Easily fixed by using a temp file which is then moved atomically into place
once the transfer completes.
I've audited other calls to copyFileExternal, and other special remote
file transfer code; everything else seems to use temp files correctly
(rsync, git), or otherwise use atomic transfers (bup, S3).
|
| |
|
|
| |
kills that ugly python message during build
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This way, the build log will indicate whether StatFS can be relied on.
I've tested all the failing architectures now, and on all of them,
the StatFS code now returns Nothing, rather than Just nonsense.
Also, if annex.diskreserve is set on a platform where StatFS is not
working, git-annex will complain.
Also, the Makefile was missing the sources target used when building with
cabal.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
git-annex FTBFS on s390, mips, powerpc, sparc. That StatFS code is failing
on all of them. At least on s390, the failure appears as:
Just (FileSystemStats {fsStatBlockSize = 4096, fsStatBlockCount = 0,
fsStatByteCount = 0, fsStatBytesFree = 0, fsStatBytesAvailable = 0,
fsStatBytesUsed = 0})
While I don't understand why this is happening, or how to fix it,
bandaid over it by checking for obviously bad values and returning Nothing.
That disables disk free space checking, but at least git-annex will work.
Upstream bug: http://code.google.com/p/xmobar/issues/detail?id=70
|
| | |
|
| |
|
|
|
| |
Note that GIT_WORK_TREE cannot influence GIT_DIR; that is necessary for
git-fake-bare and vcsh type things to work.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
the trust level of a remote.
This overrides the trust.log, and is overridden by the command-line trust
parameters.
It would have been nicer to have Logs.Trust.trustMap just look up the
configuration for all remotes, but a dependency loop prevented that
(Remotes depends on Logs.Trust in several ways). So instead, look up
the configuration when building remotes, storing it in the same forcetrust
field used for the command-line trust parameters.
|
| |
|
|
| |
A change to break local cycles made remote repos be dropped entirely.
|
| |
|
|
|
| |
As part of this, I fixed up how log was getting the descriptions of
remotes.
|
| | |
|
| |
|
|
|
|
|
| |
Only set --no-tty when GPG_AGENT_INFO is set and batch mode is used.
In the test suite, set GPG_AGENT_INFO to /dev/null to avoid the test suite
relying on /dev/tty.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
repository they were added to and removed from.
This needs to run git log on the location log files to get at all past
versions of the file, which tends to be a bit slow.
It would be possible to make a version optimised for showing the location
logs for every key. That would only need to run git log once, so would be
faster, but it would need to process an enormous amount of data, so
would not speed up the individual file case.
In the future it would be nice to support log --format. log --json also
doesn't work right yet.
|
| |
|
|
|
| |
Let's keep that in a no-s3 branch, which can be merged into eg,
debian-stable.
|
| | |
|
| |
|
|
|
|
|
|
| |
Turns out that git will accept a .git/config containing an url with eg,
spaces in its name. Handle this by escaping the url if it's not valid.
This also fixes support for urls containing escaped characters like %20
for space. Before, the path from the url was not unescaped properly.
|
| | |
|
| | |
|
| |
|
|
|
| |
This way, when a checksum check fails and the content is moved aside,
the numcopies check also warns if there are not enough copies.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Dotfiles, and files inside dotdirs are not added by "git annex add" unless
the dotfile or directory is explicitly listed. So "git annex add ." will
add all untracked files in the current directory except for those in
dotdirs.
One reason for this is that it will make git-annex more usable with vcsh,
where you don't want "vcsh big annex add" to check in all the dotfiles
that are already versioned in other repositories.
(If you're using vcsh for repos that contain non-dotfiles, this won't help,
and you'll need to .gitignore such things, but this will cover the common
case.)
A more general reason why this seems like a good idea is the same reason ls
ignores dotfiles, just the unix convention that they are cruft that is kept
out of the way most of the time.
All the other git-annex commands still do deal with any dotfiles that do
get into the annex. This seemed right because if I've gone to the trouble
to add a dotfile, I will want "git annex get ." to get it along with
everything else.
|
| | |
|
| |
|
|
| |
to provide parameters to whichever of wget or curl git-annex uses (depends on which is available, but most of their important options suitable for use here are the same).
|
| | |
|
Joey Hess