1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
Since last post, I've worked on speeding up `git annex watch`'s startup time
in a large repository.
The problem was that its initial scan was naively staging every symlink in
the repository, even though most of them are, presumably, staged correctly
already. This was done in case the user copied or moved some symlinks
around while `git annex watch` was not running -- we want to notice and
commit such changes at startup.
Since I already had the `stat` info for the symlink, it can look at the
`ctime` to see if the symlink was made recently, and only stage it if so.
This sped up startup in my big repo from longer than I cared to wait (10+
minutes, or half an hour while profiling) to a minute or so. Of course,
inotify events are already serviced during startup, so making it scan
quickly is really only important so people don't think it's a resource hog.
First impressions are important. :)
But what does "made recently" mean exactly? Well, my answer is possibly
over engineered, but most of it is really groundwork for things I'll need
later anyway. I added a new data structure for tracking the status of the
daemon, which is periodically written to disk by another thread (thread #6!)
to `.git/annex/daemon.status` Currently it looks like this; I anticipate
adding lots more info as I move into the [[syncing]] stage:
lastRunning:1339610482.47928s
scanComplete:True
So, only symlinks made after the daemon was last running need to be
expensively staged on startup. Although, as RichiH pointed out,
this fails if the clock is changed. But I have been planning to have a
cleanup thread anyway, that will handle this, and other
potential problems, so I think that's ok.
Stracing its startup scan, it's fairly tight now. There are some repeated
`getcwd` syscalls that could be optimised out for a minor speedup.
----
Added the sanity check thread. Thread #7! It currently only does one sanity
check per day, but the sanity check is a fairly lightweight job,
so I may make it run more frequently. OTOH, it may never ever find a
problem, so once per day seems a good compromise.
Currently it's only checking that all files in the tree are properly staged
in git. I might make it `git annex fsck` later, but fscking the whole tree
once per day is a bit much. Perhaps it should only fsck a few files per
day? TBD
Currently any problems found in the sanity check are just fixed and logged.
It would be good to do something about getting problems that might indicate
bugs fed back to me, in a privacy-respecting way. TBD
----
I also refactored the code, which was getting far too large to all be in
one module.
I have been thinking about renaming `git annex watch` to `git annex assistant`,
but I think I'll leave the command name as-is. Some users might
want a simple watcher and stager, without the assistant's other features
like syncing and the webapp. So the next stage of the
[[roadmap|design/assistant]] will be a different command that also runs
`watch`.
At this point, I feel I'm done with the first phase of [[inotify]].
It has a couple known bugs, but it's ready for brave beta testers to try.
I trust it enough to be running it on my live data.
|
