| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
.. and have to be checked to see if they are a pointed to an annexed file.
Cases where such memory use could occur included, but were not limited to:
- git commit -a of a large unlocked file (in v5 mode)
- git-annex adjust when a large file was checked into git directly
Generally, any use of catKey was a potential problem.
Fix by using git cat-file --batch-check to check size before catting.
This adds another git batch process, which is included in the CatFileHandle
for simplicity.
There could be performance impact, anywhere catKey is used. Particularly
likely to affect adjusted branch generation speed, and operations on
unlocked files in v6 mode. Hopefully since the --batch-check and
--batch read the same data, disk buffering will avoid most overhead.
Leaving only the overhead of talking to the process over the pipe and
whatever computation --batch-check needs to do.
This commit was sponsored by Bruno BEAUFILS on Patreon.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
copies of the environment.
Speeds up commands like "git-annex find --in remote" by over 50%.
Profiling showed that adjustGitEnv was 21% of the time and 37% of the
allocations of that command. It copied the environment each time with
getEnvironment.
The only repeated use of adjustGitEnv is in withIndexFile, which tends to
be run at least once per file. So, it was optimised by keeping a cache of
the environment, which can be reused.
There could be other better ways to optimise this. Maybe get the while
environment once at startup. But, then it would have to be serialized back
out each time running a child process, so I doubt that would be a net win.
It might be better to cache a version of the environment that is
pre-modified to use .git-annex/index. But, profiling doesn't show that
modifying the enviroment is taking any significant time.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
to work on. Sped up by around 18%.
key2file and file2key were top cost centers according to profiling.
The repeated use of replace was not efficient. This new approach is quite a
lot more efficient.
This commit was sponsored by Denis Dzyubenko on Patreon.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* sync: Previously, when run in a branch with a slash in its name,
such as "foo/bar", the sync branch was "synced/bar". That conflicted
with the sync branch used for branch "bar", so has been changed to
"synced/foo/bar".
* adjust: Previously, when adjusting a branch with a slash in its name,
such as "foo/bar", the adjusted branch was "adjusted/bar(unlocked)".
That conflicted with the adjusted branch used for branch "bar",
so has been changed to "adjusted/foo/bar(unlocked)"
* Also, running sync in an adjusted branch did not correctly sync
changes back to the parent branch when it had a slash in its name.
This bug has been fixed.
Eliminate use of Git.Ref.under and Git.Ref.basename; using
Git.Ref.underBase and Git.Ref.base make everything handle deep branches
correctly.
Probably noone was adjusting deep branches, and v6 is still experimental
anyway, so I'm not going to worry about the mess that was left by that bug.
In the case of git-annex sync, using a fixed git-annex with an old unfixed
one will mean they use different sync branches for a deep branch, and so
they may stop syncing until the old one is upgraded. However, that's only
a problem when syncing between repositories without going via a central
bare repository. Added a warning about this to the CHANGELOG, but it's
probably not going to affect many people at all.
This commit was sponsored by Riku Voipio.
|
| |
|
|
|
|
|
|
|
|
| |
This makes -Jn work with --json and --quiet, where before
setting -Jn disabled those options.
Concurrent json output is currently a mess though since threads output
chunks over top of one-another.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
have the same costs.
Only done in -J mode because only if there's concurrency can downloading
from two remotes be faster. Without concurrency, it's likely the case that
sequential downloads from the same remote are faster than switching back
and forth between two remotes.
There is some hairy MVar code here, but basically it just keeps
the activeremotes MVar full except when deciding which remote to assign
to a thread.
Also affects gets by sync --content -J
This commit was sponsored by Jochen Bartl.
|
|
|
|
| |
unused after last commit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This was disabled in commit 7ca8bf3321d1b62ea4e817e28914ed2fa56afe30,
because only the assistant used them, and they were clutter. But, now
--failed also uses them.
Remove the failure log files after successful transfers. Should avoid
most of the clutter problems.
Commit 7ca8bf3321d1b62ea4e817e28914ed2fa56afe30 mentions a subtle behavior
change, which has now been reverted:
There is one behavior change from this. If glacier is being used, and a
manual git annex get --from glacier fails because the file isn't available
yet, the assistant will no longer later see that failed transfer file and
retry the get.
|
|
|
|
|
|
|
|
|
| |
Note that get --from foo --failed will get things that a previous get --from bar
tried and failed to get, etc. I considered making --failed only retry
transfers from the same remote, but it was easier, and seems more useful,
to not have the same remote requirement.
Noisy due to some refactoring into Types/
|
|
|
|
| |
modifying metadata for multiple files/keys.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use nextRandom to generate the random UUID, rather than using randomIO.
This gets fixes for the following two bugs in the uuid library.
However, this did not impact git-annex much, so a hard depedency has
not been added on uuid-1.3.12.
https://github.com/aslatter/uuid/issues/15
"v4 UUIDs are not that random"
This doesn't greatly affect git-annex, because even with only
2^64 possible UUIDs, the chance that two git-annex repositories
that are clones of the same git repo get the same UUID is miniscule.
And, git-annex generates only one UUID per run, so preducting
subsequent UUIDs is not a problem.
https://github.com/aslatter/uuid/issues/16
"Remove Random instance for UUID, or mark it as deprecated"
git-annex was using that instance; let's stop before it gets
deprecated or removed.
|
|
|
|
|
|
|
|
| |
Show branch:file that is being operated on.
I had to make ActionItem a type and not a type class because
withKeyOptions' passed two different types of values when using the type
class, and I could not get the type checker to accept that.
|
|
|
|
| |
supposed to use that database.
|
|
|
|
|
|
|
| |
git-annex branch.
This can speed up git-annex commands by as much as a second, depending on
the number of remotes.
|
|
|
|
|
| |
This new class was added to base a while ago; I don't know what uses it,
but it's intended to be an async exception, so make sure we don't catch it.
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
| |
had not been synced back to master.
This bug caused broken tree objects to get built by a later git annex sync.
This is a somewhat unlikely but not impossible situation, and the test
suite's union_merge_regression test tickled it when it was run on FAT.
|
|
|
|
|
|
|
|
| |
Mostly the username is only used for the git committer or other display
purposes, and we can just fall back to a dummy value in these cases.
The only remaining place where an error is thrown is when starting local
pairing, which needs the username to be known.
|
|
|
|
|
|
|
| |
directory with the same name when in an adjusted branch.
When running in an overlay work tree, all unchanged files show as deleted,
so this code that stages deletions should not run.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The queue could potentially contain changes from before withAltRepo, and
get flushed inside the call, which would apply the changes to the modified
repo.
Or, changes could be queued in withAltRepo that were intended to affect
the modified repo, but don't get flushed until later.
I don't know of any cases where either happens, but better safe than sorry.
Note that this affect withIndexFile, which is used in git-annex branch
updates. So, it potentially makes things slower. Should not be by much;
the overhead consists only of querying the current queue a couple of times,
and potentially flushing changes queued within withAltRepo earlier, that
could have maybe been bundled with other later changes.
Notice in particular that the existing queue is not flushed when calling
withAltRepo. So eg when git annex add needs to stage files in the index,
it will still bundle them together efficiently.
|
|
|
|
| |
checked out.
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
| |
has an adjusted branch checked out.
The clone also has the adjusted branch checked out, so it needs to be
initialized to a version that supports that.
|
|
|
|
|
|
|
|
|
|
|
| |
remote that does not have a UUID. This particularly impacted clones from gcrypt repositories.
Added guard in Annex.Transfer to prevent this problem at a deeper level.
I'm unhappy ith NoUUID, but having Maybe UUID instead wouldn't help either
if nothing checked that there was a UUID. Since there legitimately need to
be Remotes that do not have a UUID, I can't see a way to fix it at the type
level, short making there be two separate types of Remotes.
|
|
|
|
|
| |
problematic
flexibility
|
|
|
|
| |
Does not change behavior, only git annex version output
|
|
|
|
|
|
|
|
|
|
|
|
| |
were not before.
Removed the instance LensGpgEncParams RemoteConfig because it encouraged
code that does not take the RemoteGitConfig into account.
RemoteType's setup was changed to take a RemoteGitConfig,
although the only place that is able to provide a non-empty one is
enableremote, when it's changing an existing remote. This led to several
folow-on changes, and got RemoteGitConfig plumbed through.
|
|
|
|
|
|
| |
git annex adjust --force will overwrite any current adjusted branch.
I didn't document this because for the user, deleting the branch is just as
good.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
always make absolute
This is actually worse than I thought; when git is being run with a
detached work tree, GIT_INDEX_FILE is treated as a path relative to CWD,
instead of the normal behavior of relative the top of the work tree.
This seems to make it basically impossible for any program that wants to
use GIT_INDEX_FILE to use anything other than an absolute path to it; there
are too many configurations to keep straight that can change how git
interprets what should be a simple relative path to a file.
(I have complained to the git developers.)
|
|
|
|
| |
avoid redundant work for repeated ForgetDeadRemotes transitions
|
|
|
|
| |
This is probably a git bug that stuck in its interface.
|
|
|
|
| |
dotfile in its root.
|
| |
|
|
|
|
|
|
|
|
| |
subdirectory of the repository.
This affected git annex view. It turns out that some other places
that use GIT_INDEX_FILE were already working around the bug. I removed the
workaround from Annex.Branch since the new workaround will do.
|
|
|
|
|
|
| |
In commit 07b1aa343b6d82e8f75de606a5647b0a789f76e0 I accidentially
made gitAnnexLocation do more work, checking content locations,
when used in a regular repo.
|
|
|
|
| |
nonstandard place.
|
|
|
|
|
| |
This makes git-annex sync also do it, which makes sure that the keys db
info is fresh when doing a sync --content.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
it might contain changes that have not yet been propigated to the original branch.
Could not think of a foolproof way to detect if the old adjusted branch was
just behind the current branch. It's possible that the user amended the
adjusting commit at the head of the adjusted branch, for example.
I decided to bail in this situation, instead of just entering the old
branch, so that if git annex adjust succeeds the user is always in a
*current* adjusted branch, not some old and out of date one.
What could perhaps be done is enter the old branch and then update it. But
that seems too magical; the user may have rebased master or something or
may not want to propigate the changes from the old branch. Best to error
out.
|
|
|
|
|
|
|
| |
core.symlinks=true, store object contents in mixed case hash directories so that symlinks will point to them.
Contents are searched for in both locations, same as before, so this
does not add any overhead.
|
| |
|
| |
|
| |
|
|
|
|
|
| |
normalize filepaths in the map because it may be constructed with
windows-style paths and then queried for git-style
|
| |
|