1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
|
git-annex is designed for scalability. The key points are:
* Arbitrarily large files can be managed. The only constraint
on file size are how large a file your filesystem can hold.
While git-annex does checksum files by default, there
is a [[WORM_backend|backends]] available that avoids the checksumming
overhead, so you can add new, enormous files, very fast. This also
allows it to be used on systems with very slow disk IO.
* Memory usage should be constant. This is a "should", because there
can sometimes be leaks (and this is one of haskell's weak spots),
but git-annex is designed so that it does not need to hold all
the details about your repository in memory.
* Many files can be managed. The limiting factor is git's own
limitations in scaling to repositories with a lot of files, and as git
improves this will improve. Scaling to hundreds of thousands of files
is not a problem, scaling beyond that and git will start to get slow.
To some degree, git-annex works around inefficiencies in git; for
example it batches input sent to certain git commands that are slow
when run in an enormous repository.
* It can use as much, or as little bandwidth as is available. In
particular, any interrupted file transfer can be resumed by git-annex.
## scalability tips
* If the files are so big that checksumming becomes a bottleneck, consider
using the [[WORM_backend|backends]]. You can always `git annex migrate`
files to a checksumming backend later on.
* If you're adding a huge number of files at once (hundreds of thousands),
you'll soon notice that git-annex periodically stops and say
"Recording state in git" while it runs a `git add` command that
becomes increasingly expensive. Consider adjusting the `annex.queuesize`
to a higher value, at the expense of it using more memory.
|
