| Commit message (Collapse) | Author | Age |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
This allows things like Command.Find to use noMessages and generate their
own complete json objects. Previouly, Command.Find managed that only via a
hack, which wasn't compatable with batch mode.
Only Command.Find, Command.Smudge, and Commange.Status use noMessages
currently, and none except for Command.Find are impacted by this change.
Fixes find --json --batch output
|
| | |
|
| |
|
|
|
| |
All three of these are using batch mode to drive their processing, so
there is no automatic output, and noMessages is no longer needed.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
(unless the file was added directly to git due to annex.largefiles configuration.)
(Also done by add --json and import --json)
|
| |
|
|
| |
Include added key in output.
|
| | |
|
| | |
|
| |
|
|
| |
rather than all together at the end.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This breaks any existing keys database!
IKey serializes more efficiently than SKey, although this limits the
use of its Read/Show instances.
This makes the keys database use less disk space, and so should be a win.
Updated benchmark:
benchmarking keys database/getAssociatedFiles from 1000 (hit)
time 64.04 μs (63.95 μs .. 64.13 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 64.02 μs (63.96 μs .. 64.08 μs)
std dev 218.2 ns (172.5 ns .. 299.3 ns)
benchmarking keys database/getAssociatedFiles from 1000 (miss)
time 52.53 μs (52.18 μs .. 53.21 μs)
0.999 R² (0.998 R² .. 1.000 R²)
mean 52.31 μs (52.18 μs .. 52.91 μs)
std dev 734.6 ns (206.2 ns .. 1.623 μs)
benchmarking keys database/getAssociatedKey from 1000 (hit)
time 64.60 μs (64.46 μs .. 64.77 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 64.74 μs (64.57 μs .. 65.20 μs)
std dev 900.2 ns (389.7 ns .. 1.733 μs)
benchmarking keys database/getAssociatedKey from 1000 (miss)
time 52.46 μs (52.29 μs .. 52.68 μs)
1.000 R² (0.999 R² .. 1.000 R²)
mean 52.63 μs (52.35 μs .. 53.37 μs)
std dev 1.362 μs (562.7 ns .. 2.608 μs)
variance introduced by outliers: 24% (moderately inflated)
benchmarking keys database/addAssociatedFile to 1000 (old)
time 487.3 μs (484.7 μs .. 490.1 μs)
1.000 R² (0.999 R² .. 1.000 R²)
mean 490.9 μs (487.8 μs .. 496.5 μs)
std dev 13.95 μs (6.841 μs .. 22.03 μs)
variance introduced by outliers: 20% (moderately inflated)
benchmarking keys database/addAssociatedFile to 1000 (new)
time 6.633 ms (5.741 ms .. 7.751 ms)
0.905 R² (0.850 R² .. 0.965 R²)
mean 8.252 ms (7.803 ms .. 8.602 ms)
std dev 1.126 ms (900.3 μs .. 1.430 ms)
variance introduced by outliers: 72% (severely inflated)
benchmarking keys database/getAssociatedFiles from 10000 (hit)
time 65.36 μs (64.71 μs .. 66.37 μs)
0.998 R² (0.995 R² .. 1.000 R²)
mean 65.28 μs (64.72 μs .. 66.45 μs)
std dev 2.576 μs (920.8 ns .. 4.122 μs)
variance introduced by outliers: 42% (moderately inflated)
benchmarking keys database/getAssociatedFiles from 10000 (miss)
time 52.34 μs (52.25 μs .. 52.45 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 52.49 μs (52.42 μs .. 52.59 μs)
std dev 255.4 ns (205.8 ns .. 312.9 ns)
benchmarking keys database/getAssociatedKey from 10000 (hit)
time 64.76 μs (64.67 μs .. 64.84 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 64.67 μs (64.62 μs .. 64.72 μs)
std dev 177.3 ns (148.1 ns .. 217.1 ns)
benchmarking keys database/getAssociatedKey from 10000 (miss)
time 52.75 μs (52.66 μs .. 52.82 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 52.69 μs (52.63 μs .. 52.75 μs)
std dev 210.6 ns (173.7 ns .. 265.9 ns)
benchmarking keys database/addAssociatedFile to 10000 (old)
time 489.7 μs (488.7 μs .. 490.7 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 490.4 μs (489.6 μs .. 492.2 μs)
std dev 3.990 μs (2.435 μs .. 7.604 μs)
benchmarking keys database/addAssociatedFile to 10000 (new)
time 9.994 ms (9.186 ms .. 10.74 ms)
0.959 R² (0.928 R² .. 0.979 R²)
mean 9.906 ms (9.343 ms .. 10.40 ms)
std dev 1.384 ms (1.051 ms .. 2.100 ms)
variance introduced by outliers: 69% (severely inflated)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
benchmarking keys database/addAssociatedFile to 1000 (old)
time 516.1 μs (514.7 μs .. 517.4 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 514.0 μs (512.1 μs .. 515.2 μs)
std dev 4.740 μs (2.972 μs .. 7.068 μs)
benchmarking keys database/addAssociatedFile to 1000 (new)
time 5.750 ms (4.857 ms .. 6.885 ms)
0.815 R² (0.698 R² .. 0.904 R²)
mean 7.858 ms (7.311 ms .. 8.421 ms)
std dev 1.684 ms (1.383 ms .. 2.027 ms)
variance introduced by outliers: 88% (severely inflated)
benchmarking keys database/addAssociatedFile to 10000 (old)
time 515.7 μs (514.8 μs .. 516.5 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 515.4 μs (513.7 μs .. 516.6 μs)
std dev 4.824 μs (2.957 μs .. 7.167 μs)
benchmarking keys database/addAssociatedFile to 10000 (new)
time 8.934 ms (7.779 ms .. 10.05 ms)
0.868 R² (0.751 R² .. 0.934 R²)
mean 11.51 ms (10.66 ms .. 12.26 ms)
std dev 2.174 ms (1.816 ms .. 2.747 ms)
variance introduced by outliers: 82% (severely inflated)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The benchmark shows that the database access is quite fast indeed!
And, it scales linearly to the number of keys, with one exception,
getAssociatedKey.
Based on this benchmark, I don't think I need worry about optimising
for cases where all files are locked and the database is mostly empty.
In those cases, database access will be misses, and according to this
benchmark, should add only 50 milliseconds to runtime.
(NB: There may be some overhead to getting the database opened and locking
the handle that this benchmark doesn't see.)
joey@darkstar:~/src/git-annex>./git-annex benchmark
setting up database with 1000
setting up database with 10000
benchmarking keys database/getAssociatedFiles from 1000 (hit)
time 62.77 μs (62.70 μs .. 62.85 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 62.81 μs (62.76 μs .. 62.88 μs)
std dev 201.6 ns (157.5 ns .. 259.5 ns)
benchmarking keys database/getAssociatedFiles from 1000 (miss)
time 50.02 μs (49.97 μs .. 50.07 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 50.09 μs (50.04 μs .. 50.17 μs)
std dev 206.7 ns (133.8 ns .. 295.3 ns)
benchmarking keys database/getAssociatedKey from 1000 (hit)
time 211.2 μs (210.5 μs .. 212.3 μs)
1.000 R² (0.999 R² .. 1.000 R²)
mean 211.0 μs (210.7 μs .. 212.0 μs)
std dev 1.685 μs (334.4 ns .. 3.517 μs)
benchmarking keys database/getAssociatedKey from 1000 (miss)
time 173.5 μs (172.7 μs .. 174.2 μs)
1.000 R² (0.999 R² .. 1.000 R²)
mean 173.7 μs (173.0 μs .. 175.5 μs)
std dev 3.833 μs (1.858 μs .. 6.617 μs)
variance introduced by outliers: 16% (moderately inflated)
benchmarking keys database/getAssociatedFiles from 10000 (hit)
time 64.01 μs (63.84 μs .. 64.18 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 64.85 μs (64.34 μs .. 66.02 μs)
std dev 2.433 μs (547.6 ns .. 4.652 μs)
variance introduced by outliers: 40% (moderately inflated)
benchmarking keys database/getAssociatedFiles from 10000 (miss)
time 50.33 μs (50.28 μs .. 50.39 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 50.32 μs (50.26 μs .. 50.38 μs)
std dev 202.7 ns (167.6 ns .. 252.0 ns)
benchmarking keys database/getAssociatedKey from 10000 (hit)
time 1.142 ms (1.139 ms .. 1.146 ms)
1.000 R² (1.000 R² .. 1.000 R²)
mean 1.142 ms (1.140 ms .. 1.144 ms)
std dev 7.142 μs (4.994 μs .. 10.98 μs)
benchmarking keys database/getAssociatedKey from 10000 (miss)
time 1.094 ms (1.092 ms .. 1.096 ms)
1.000 R² (1.000 R² .. 1.000 R²)
mean 1.095 ms (1.095 ms .. 1.097 ms)
std dev 4.277 μs (2.591 μs .. 7.228 μs)
|
| |
|
|
|
| |
It makes sense for migrate to do that, but not for this low-level (and
little used) plumbing command to.
|
| |
|
|
|
|
|
| |
Linking the file to the tmp dir was not necessary in the clean
filter, and it caused the ctime to change, which caused git to think
the file was changed. This caused git status to get slow as it kept
re-cleaning unchanged files.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
d1ce927d95fe7c331cbff3317797a60aa288738b put a cat-file into the fast
bloomfilter generation path. Instead, add another bloom filter which diffs
from the work tree to the index.
Also, pull the sha of the changed object out of the diffs, and cat that
object directly, rather than indirecting through the filename.
Finally, removed some hacks that are unncessary thanks to the worktree to
index diff.
|
| |
|
|
| |
The newline broke this ad-hoc parser; use the normal one.
|
| |
|
|
|
|
|
|
| |
(but not git rmed). git still has the add staged in this case, so the content should not be unused and was wrongly treated as such.
So, we need to look at both the file on disk to see if it's a annex link,
and the file in the index too. lookupFile doesn't look in the index if the file
is not present on disk.
|
| |
|
|
|
|
|
|
| |
In v5, that was not possible, but it is in v6, and so the test was failing.
Investigating, it turns out that locking was copying the pointer file
content to the annex object despite the content not being present. So,
add a check to prevent that.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes several bugs with updates of pointer files. When eg, running
git annex drop --from localremote
it was updating the pointer file in the local repository, not the remote.
Also, fixes drop ../foo when run in a subdir, and probably lots of other
problems. Test suite drops from ~30 to 11 failures now.
TopFilePath is used to force thinking about what the filepath is relative
to.
The data stored in the sqlite db is still just a plain string, and
TopFilePath is a newtype, so there's no overhead involved in using it in
DataBase.Keys.
|
| |
|
|
| |
fields for each backend instead of the previous weird nested lists. This may break existing parsers of this json output, if there were any.
|
| |
|
|
| |
Let's just count the referenced keys for that, and not present keys at all.
|
| |
|
|
|
| |
Fixes at least one bug, in populating existing worktree files that use the
same key that's ingested.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
The smudge filter does need to be run, because if the key is in the local
annex already (due to renaming, or a copy of a file added, or a new file
added and its content has already arrived), git merge smudges the file and
this should provide its content.
This does probably mean that in merge conflict resolution, git smudges the
existing file, re-copying all its content to it, and then the file is
deleted. So, not efficient.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Several tricky parts:
* When the conflict is just between the same key being locked and unlocked,
the unlocked version wins, and the file is not renamed in this case.
* Need to update associated file map when conflict resolution renames
an unlocked file.
* git merge runs the smudge filter on the conflicting file, and actually
overwrites the file with the same content it had before, and so
invalidates its inode cache. This makes it difficult to know when it's
safe to remove such files as conflict cruft, without going so far as to
compare their entire contents.
Dealt with this by preventing the smudge filter from populating the file
when a merge is run. However, that also prevents the smudge filter being
run for non-conflicting files, so eg moving a file won't put its new
content into place.
* Ideally, if a merge or a merge conflict resolution renames an unlocked
file, the file in the work tree can just be moved, rather than copying
the content to a new worktree file.
This is attempted to be done in merge conflict resolution, but
due to git merge's behavior of running smudge filters, what actually
seems to happen is the old worktree file with the content is deleted and
rewritten as a pointer file, so doesn't get reused.
So, this is probably not as efficient as it optimally could be.
If that becomes a problem, could look into running the merge in a separate
worktree and updating the real worktree more efficiently, similarly to the
direct mode merge. However, the direct mode merge had a lot of bugs, and
I'd rather not use that more error-prone method unless really needed.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Decided it's too scary to make v6 unlocked files have 1 copy by default,
but that should be available to those who need it. This is consistent with
git-annex not dropping unused content without --force, etc.
* Added annex.thin setting, which makes unlocked files in v6 repositories
be hard linked to their content, instead of a copy. This saves disk
space but means any modification of an unlocked file will lose the local
(and possibly only) copy of the old version.
* Enable annex.thin by default on upgrade from direct mode to v6, since
direct mode made the same tradeoff.
* fix: Adjusts unlocked files as configured by annex.thin.
|
| |
|
|
|
|
| |
The direct flag is also set when sending unlocked content, to support old
versions of git-annex-shell. At some point, the direct flag will be
removed, and only the unlocked flag will be used.
|
| |
|
|
| |
needs it.
|
| |
|
|
| |
I forgot to convert this to use Annex.Ingest, todo later.
|
| |\ |
|
| | | |
|
| | |
| |
| |
| | |
Seems to basically work now!
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
This fixes a race where the modified file ended up in annex/objects, and
the InodeCache stored in the database was for the modified version, so
git-annex didn't know it had gotten modified.
The race could occur when the smudge filter was running; now it gets the
InodeCache before generating the Key, which avoids the race.
|
| | |
| |
| |
| | |
no behavior changes
|
| | | |
|
Joey Hess