summaryrefslogtreecommitdiff
path: root/doc/bugs/assistant_eats_all_CPU.mdwn
blob: 4939bf4567bb9a4ecce4984055060c2d61aad86d (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
[[!meta title="debian system runs assistant in tight loop, rather than using select"]]

### Please describe the problem.

After running for a while, the assistant maxes my CPU. I have evidence
here that it is just sitting there, doing apparently nothing (transfers
are finished), eating precious CPU.

Nowadays, I do not let the assistant run on its own, otherwise it makes
my office noisy because the CPU fan starts.

This makes the assistant and the webapp basically unuseable for me:
if i want to use its feature, I must remember to shut down the daemon
every time I am done with it, which seems contrary to the spirit of
the project. :)

### What steps will reproduce the problem?

It's a little unclear. There was a peculiar crash here (power cable
yanked out) during a `git annex copy`, which *was* eating up all CPU,
but mostly through I/O.

That is the purple block in this graph:

![CPU usage during the last 24 hours][cpu-day.png]

Power was yanked out about right after that purple block was gone:

![Uptime in the last 24 hours][uptime-day.png]

Files were copied to an external drive at the time of the crash,
which explains the high iowait. The last lines in daemon.log are:

[[!format txt """
sent 184184937 bytes  received 31 bytes  13643330.96 bytes/sec
total size is 184162304  speedup is 1.00
[2013-12-31 23:56:47 EST] Transferrer: Uploaded American ..Queen.avi
[2013-12-31 23:56:47 EST] Pusher: Syncing with backup
To /backup/srv/video
   5155be4..7d904d5  git-annex -> synced/git-annex
"""]]

And daemon.status:

[[!format txt """
lastRunning:1388569856.366405s
scanComplete:True
sanityCheckRunning:False
lastSanityCheck:1387169250.801117s
"""]]

When the machine rebooted, I logged in, and resumed the transfer using
`git annex copy`. The assistant was also started automatically during
my session, but I wasn't aware of that until this morning (7AM) when i
came back to the terminal and noticed how noisy it was.

At that point i did `killall git-annex` and CPU usage returned to
normal, as the frequency of the CPU:

![CPU frequency scaling in the last 24 hours][cpuspeed-day.png]

I am trying to see if I can reproduce this problem reliably - I 
just started the assistant again now, and while the CPU usage is
already high, I will forgive it for now because it's doing its 
initial scans... Let's see what happens in the afternoon! --[[anarcat]]

Update: I seem to be able to reproduce. I just started the webapp and
the scans were performed. They are now finished, yet the CPU usage is 
still high:

[[!format txt """
top - 14:10:20 up 1 day,  1:16,  7 users,  load average: 5,51, 3,61, 2,53
Tasks: 292 total,   2 running, 288 sleeping,   0 stopped,   2 zombie
%Cpu(s): 60,9 us,  7,8 sy,  0,6 ni, 29,5 id,  0,8 wa,  0,0 hi,  0,5 si,  0,0 st
KiB Mem:   8162960 total,  6943100 used,  1219860 free,  1065464 buffers
KiB Swap:  4022268 total,    15452 used,  4006816 free,  1983252 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
18364 anarcat   20   0  152m  27m  18m S  53,7  0,3   0:35.61 git-annex
 8388 anarcat   20   0  192m  54m  21m S  47,8  0,7  12:55.19 git-annex
18671 anarcat   20   0  157m  33m  18m S  29,9  0,4   0:22.65 git-annex
"""]]

An strace on 18364 yields:

[[!format txt """
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 537245076}) = 0
[pid 18364] <... futex resumed> )       = ? ERESTARTSYS (To be restarted)
[pid 18364] --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
[pid 18364] rt_sigreturn(0x1a)          = 202
[pid 18364] futex(0x3de703c, FUTEX_WAIT_PRIVATE, 51, NULL <unfinished ...>
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 256339453}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 537670797}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 256748972}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 538079838}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 257202502}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 538533443}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 257611085}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 538941839}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 258018398}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 539349100}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 258433069}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 539763939}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 258863864}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 540195885}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 259295204}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 540626505}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 259729523}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 541060751}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 260131616}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 541463041}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 260539761}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 541871201}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 260970889}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 542302047}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 261378036}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 542709166}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 261780507}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 543111819}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 262194032}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 543525143}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 262610040}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 543941528}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 263042272}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 544374016}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18366] rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
[pid 18366] clock_gettime(0x2 /* CLOCK_??? */, {82, 263532359}) = 0
[pid 18366] clock_gettime(0x3 /* CLOCK_??? */, {79, 544864814}) = 0
[pid 18366] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 18364] <... futex resumed> )       = ? ERESTARTSYS (To be restarted)
[pid 18364] --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
[pid 18364] rt_sigreturn(0x1a)          = 202
[pid 18364] futex(0x3de703c, FUTEX_WAIT_PRIVATE, 51, NULL <unfinished ...>
"""]]

 [cpu-day.png]: http://paste.anarc.at/cpu-day.png
 [uptime-day.png]: http://paste.anarc.at/uptime-day.png
 [cpuspeed-day.png]: http://paste.anarc.at/cpuspeed-day.png

### What version of git-annex are you using? On what operating system?

Debian wheezy.

[[!format sh """
anarcat@marcos:~$ git annex version
git-annex version: 5.20131109-gf2cb5b9
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
"""]]

### Please provide any additional information below.

This *may* be related to [[bugs/webapp_takes_100__37___of_the_cpu/]],
but I didn't observe both problems simultaneously so far - ie. the
webapp doesn't need to run for this bug to happen.

Here's the output of top:

[[!format sh """
top - 04:55:01 up 16:01, 10 users,  load average: 3,96, 3,72, 3,59
Tasks: 293 total,   6 running, 284 sleeping,   0 stopped,   3 zombie
%Cpu(s): 87,4 us, 10,9 sy,  0,6 ni,  0,1 id,  0,3 wa,  0,0 hi,  0,7 si,  0,0 st
KiB Mem:   8162960 total,  6847120 used,  1315840 free,  1084756 buffers
KiB Swap:  4022268 total,     1320 used,  4020948 free,  2041108 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 8099 anarcat   20   0  144m  28m  18m S  34,0  0,4 460:10.96 git-annex
 6525 anarcat   20   0  436m 295m  18m S  28,4  3,7 448:50.71 git-annex
11146 munin     20   0 25376 6540 1896 R  28,4  0,1   0:00.05 munin-update
 8045 anarcat   20   0  165m  41m  18m S  22,7  0,5 455:14.41 git-annex
11142 root      20   0 32316 7356 2016 R  22,7  0,1   0:00.05 ikisite
11144 aegir     20   0  161m  16m 6608 R  22,7  0,2   0:00.05 php
 2649 liquidso  20   0  222m  54m 5516 S   5,7  0,7  40:40.61 liquidsoap
11149 anarcat   20   0 24448 1552 1072 R   5,7  0,0   0:00.01 top
    1 root      20   0 10644  784  652 S   0,0  0,0   0:01.10 init
    2 root      20   0     0    0    0 S   0,0  0,0   0:00.01 kthreadd
    3 root      20   0     0    0    0 S   0,0  0,0   2:10.09 ksoftirqd/0
    6 root      rt   0     0    0    0 S   0,0  0,0   0:00.22 migration/0
    7 root      rt   0     0    0    0 S   0,0  0,0   0:00.59 watchdog/0
    8 root      rt   0     0    0    0 S   0,0  0,0   0:00.25 migration/1
   10 root      20   0     0    0    0 S   0,0  0,0   2:11.10 ksoftirqd/1
   12 root      rt   0     0    0    0 S   0,0  0,0   0:00.17 watchdog/1
   13 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 cpuset
   14 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 khelper
   15 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kdevtmpfs
   16 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 netns
   17 root      20   0     0    0    0 S   0,0  0,0   0:00.17 sync_supers
   18 root      20   0     0    0    0 S   0,0  0,0   0:00.00 bdi-default
   19 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kintegrityd
   20 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kblockd
   22 root      20   0     0    0    0 S   0,0  0,0   0:00.06 khungtaskd
   23 root      20   0     0    0    0 S   0,0  0,0   3:04.23 kswapd0
   24 root      25   5     0    0    0 S   0,0  0,0   0:00.00 ksmd
   25 root      39  19     0    0    0 S   0,0  0,0   0:00.00 khugepaged
   26 root      20   0     0    0    0 S   0,0  0,0   0:00.00 fsnotify_mark
   27 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 crypto
  113 root      20   0     0    0    0 S   0,0  0,0   0:00.01 khubd
  133 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ata_sff
  153 root      20   0     0    0    0 S   0,0  0,0   0:05.68 scsi_eh_0
  158 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_1
  160 root      20   0     0    0    0 S   0,0  0,0   0:00.09 kworker/u:2
  161 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_2
  162 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_3
  163 root      20   0     0    0    0 S   0,0  0,0   0:01.43 kworker/u:3
  294 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  295 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kcryptd_io
  296 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kcryptd
  305 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  307 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  311 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  313 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  341 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  342 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kcryptd_io
  343 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kcryptd
  352 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  354 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  356 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  358 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  361 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kdmflush
  400 root      20   0     0    0    0 S   0,0  0,0   0:00.00 jbd2/dm-1-8
  401 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
  468 root      20   0     0    0    0 S   0,0  0,0   0:02.19 kworker/1:2
  538 root      20   0 21804 1804  796 S   0,0  0,0   0:00.10 udevd
  705 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kpsmoused
  878 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 hd-audio0
  890 postfix   20   0 44216 2460 1912 S   0,0  0,0   0:00.00 pickup
  939 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 kvm-irqfd-clean
 1150 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 firewire
 1719 root      20   0     0    0    0 S   0,0  0,0   0:03.40 flush-254:3
 1786 root      20   0     0    0    0 S   0,0  0,0   0:00.17 jbd2/dm-2-8
 1787 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
 1788 root      20   0     0    0    0 S   0,0  0,0   0:06.60 jbd2/dm-3-8
 1789 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
 1790 root      20   0     0    0    0 S   0,0  0,0   0:02.56 jbd2/dm-4-8
 1791 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
 1792 root      20   0     0    0    0 S   0,0  0,0   0:01.19 jbd2/dm-8-8
 1793 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 ext4-dio-unwrit
 2004 anarcat   20   0     0    0    0 Z   0,0  0,0   0:17.57 chromium
 2379 root      20   0  115m 1796  928 S   0,0  0,0   0:02.16 rsyslogd
 2435 icecast2  20   0  148m 4440 2780 S   0,0  0,1   0:56.15 icecast2
 2488 root      20   0  4112  644  500 S   0,0  0,0   0:00.66 acpid
 2513 daemon    20   0 16668  156    0 S   0,0  0,0   0:00.00 atd
 2536 proxy     20   0  6884  344  196 S   0,0  0,0   0:00.00 polipo
 2573 messageb  20   0 31488 2548  884 S   0,0  0,0   0:00.79 dbus-daemon
 2626 bind      20   0  111m  15m 2276 S   0,0  0,2   0:00.46 named
 2694 root      20   0 82308 3112 2520 S   0,0  0,0   0:00.03 gdm3
 2729 root      20   0 99,3m 4652 3608 S   0,0  0,1   0:00.02 gdm-simple-slav
 2731 root      20   0  178m  27m 9908 S   0,0  0,3  40:30.80 Xorg
 2738 root      20   0  282m  14m 6576 S   0,0  0,2   0:02.98 apache2
 2763 root      20   0     0    0    0 S   0,0  0,0   0:00.92 flush-254:4
 2779 root      20   0 21000 1164  904 S   0,0  0,0   0:00.00 bluetoothd
 2784 avahi     20   0 34236 1876 1460 S   0,0  0,0   0:00.15 avahi-daemon
 2786 avahi     20   0 34032  440  184 S   0,0  0,0   0:00.00 avahi-daemon
 2792 root      10 -10     0    0    0 S   0,0  0,0   0:00.00 krfcommd
 2829 bitlbee   20   0 45216 1136  712 S   0,0  0,0   0:00.00 bitlbee
 2973 root      20   0  136m 3572 2892 S   0,0  0,0   0:00.37 accounts-daemon
 2974 clamav    20   0 45432 2172 1260 S   0,0  0,0   0:26.77 freshclam
 2997 root      20   0  141m 4552 3072 S   0,0  0,1   0:00.21 polkitd
 3021 root      20   0 18876  984  740 S   0,0  0,0   0:00.48 cron
 3062 root      20   0 75948 3436 2304 S   0,0  0,0   0:00.03 cupsd
 3064 colord    20   0  152m 8672 3344 S   0,0  0,1   0:00.10 colord
 3069 root      20   0  126m 4116 2964 S   0,0  0,1   0:00.16 console-kit-dae
 3071 colord    20   0  300m  12m 7932 S   0,0  0,2   0:01.23 colord-sane
 3171 nobody    10 -10 44328 1672 1048 S   0,0  0,0   0:00.88 gpsd
 3178 root      20   0  9204 3508  380 S   0,0  0,0   0:39.47 haveged
 3192 ikiwiki-  20   0 13772  864  728 S   0,0  0,0   0:00.00 git-daemon
 3310 anarcat   20   0 91856  27m 2820 S   0,0  0,3   0:03.30 xterm
 3314 anarcat   20   0 24004 5452 1876 S   0,0  0,1   0:00.54 bash
 3336 root      20   0  152m 4172 3300 S   0,0  0,1   0:00.07 upowerd
 3437 root      20   0     0    0    0 S   0,0  0,0   0:00.00 scsi_eh_6
 3438 root      20   0     0    0    0 S   0,0  0,0   6:52.70 usb-storage
 3470 anarcat   20   0 61648 2516 1912 S   0,0  0,0   0:00.01 gvfs-gphoto2-vo
 3474 anarcat   20   0 78024 2564 2048 S   0,0  0,0   0:02.04 gvfs-afc-volume
 3530 root      20   0 52400  10m 2064 S   0,0  0,1   0:03.34 munin-node
 3599 rtkit     21   1 39632 1344 1112 S   0,0  0,0   0:00.30 rtkit-daemon
 3721 root      20   0  4176  700  560 S   0,0  0,0   0:00.01 mysqld_safe
 3899 anarcat   20   0 85216  21m 2872 S   0,0  0,3   0:00.76 xterm
 3903 anarcat   20   0 23216 4688 1804 S   0,0  0,1   0:00.58 bash
 4108 mysql     30  10  434m 137m 4396 S   0,0  1,7   1:53.14 mysqld
 4109 root      20   0  4084  628  524 S   0,0  0,0   0:00.00 logger
 4209 anarcat   20   0  129m  23m  18m S   0,0  0,3   1:06.16 git-annex
 4226 root      20   0  183m 4888 3504 S   0,0  0,1   0:00.08 gdm-session-wor
 4240 anarcat   20   0 21360 2572 2188 S   0,0  0,0   0:00.34 git
 4431 root      20   0     0    0    0 S   0,0  0,0   0:00.00 kauditd
 4447 anarcat   20   0  4176  664  548 S   0,0  0,0   0:00.03 sh
 4530 ntp       20   0 41052 2284 1644 S   0,0  0,0   0:02.71 ntpd
 4593 postgres  20   0  102m 9008 7680 S   0,0  0,1   0:01.04 postgres
 4667 anarcat   20   0 12384 1072  704 S   0,0  0,0   0:00.11 ssh-agent
 4668 anarcat   20   0 20164  372  156 S   0,0  0,0   0:01.03 gpg-agent
 4671 anarcat   20   0 24184  552  304 S   0,0  0,0   0:00.00 dbus-launch
 4672 anarcat   20   0 30668 1592  736 S   0,0  0,0   0:01.10 dbus-daemon
 4704 anarcat   20   0 49456 2740 2216 S   0,0  0,0   0:00.13 xfconfd
 4711 anarcat   20   0 63740 3124 2312 S   0,0  0,0   0:02.61 xscreensaver
 4713 anarcat   20   0  156m 8232 5856 S   0,0  0,1   0:06.14 xfce4-session
 4714 postgres  20   0  102m 1952  616 S   0,0  0,0   0:06.79 postgres
 4715 postgres  20   0  102m 1704  376 S   0,0  0,0   0:04.95 postgres
 4716 postgres  20   0  102m 3068 1128 S   0,0  0,0   0:01.32 postgres
 4717 postgres  20   0 72884 1800  416 S   0,0  0,0   0:00.99 postgres
 4735 postgrey  20   0 66452  14m 3136 S   0,0  0,2   0:00.40 /usr/sbin/postg
 4753 anarcat   20   0  124m 4060 2916 S   0,0  0,0   0:00.02 xfsettingsd
 4754 anarcat   20   0  166m 6496 5204 S   0,0  0,1   0:00.07 Thunar
 4778 anarcat   20   0 63364 2564 2096 S   0,0  0,0   0:00.03 gvfsd
 4783 anarcat   20   0  305m  14m 9568 S   0,0  0,2   0:52.46 xfce4-panel
 4790 anarcat   20   0  156m 7968 6232 S   0,0  0,1   0:00.23 xfdesktop
 4791 root      20   0 21516  804  496 S   0,0  0,0   0:00.00 dovecot
 4837 anarcat   20   0  148m 4296 2768 S   0,0  0,1   0:11.87 xfce4-settings-
 4856 anarcat   20   0 26620 1248  988 S   0,0  0,0   1:19.60 xcompmgr
 4873 anarcat   20   0  115m 8004 3872 S   0,0  0,1   0:07.42 awesome
 4887 anarcat   20   0  267m  12m 8664 S   0,0  0,2   0:02.51 update-notifier
 4899 anarcat   20   0  183m 5868 4708 S   0,0  0,1   0:00.18 polkit-gnome-au
 4905 anarcat   20   0 56320 4180 2040 S   0,0  0,1   0:00.10 gconfd-2
 4908 anarcat    9 -11  300m 6912 4184 S   0,0  0,1   0:08.00 pulseaudio
 4916 root      20   0  126m 4208 3056 S   0,0  0,1   0:02.46 udisks-daemon
 4917 root      20   0 47408  776  428 S   0,0  0,0   0:06.61 udisks-daemon
 4921 anarcat   20   0  113m 3200 2448 S   0,0  0,0   0:00.00 gconf-helper
 4972 anarcat   20   0  145m 7432 5880 S   0,0  0,1   0:00.09 panel-16-systra
 4974 anarcat   20   0  148m 9328 7348 S   0,0  0,1   2:40.37 xfce4-weather-p
 4975 anarcat   20   0  174m  11m 8312 S   0,0  0,1   0:42.26 xfce4-xkb-plugi
 4978 mpd       20   0  503m  29m 2920 S   0,0  0,4   1:24.97 mpd
 5029 mpd       20   0  256m 6412 4008 S   0,0  0,1   0:11.74 pulseaudio
 5030 dovecot   20   0 13044  768  592 S   0,0  0,0   0:00.00 anvil
 5031 root      20   0 13172  944  628 S   0,0  0,0   0:00.00 log
 5034 root      20   0 23156 2536  768 S   0,0  0,0   0:00.00 config
 5040 mpd       20   0  113m 3192 2448 S   0,0  0,0   0:00.00 gconf-helper
 5234 root      20   0     0    0    0 S   0,0  0,0   0:01.19 kworker/0:1
 5745 root      20   0 37788 2472 1912 S   0,0  0,0   0:00.68 master
 5756 postfix   20   0 44368 2700 2100 S   0,0  0,0   0:00.27 qmgr
 5807 root      20   0 49848 1216  608 S   0,0  0,0   0:00.05 sshd
 5908 debian-t  20   0 56008  21m 2988 S   0,0  0,3   0:21.87 tor
 5928 debian-t  30  10  190m  31m 2560 S   0,0  0,4   7:35.56 transmission-da
 5948 anarcat   20   0 71776 8204 2860 S   0,0  0,1   0:05.76 xterm
 5954 anarcat   20   0 22956 4540 1772 S   0,0  0,1   0:00.10 bash
 6019 spampd    20   0  147m  62m 1844 S   0,0  0,8   0:00.55 spampd
 6049 root       0 -20     0    0    0 S   0,0  0,0   0:00.00 iprt
 6085 root      20   0  3948   84    0 S   0,0  0,0   0:00.00 minissdpd
 6095 root      20   0 17404  908  744 S   0,0  0,0   0:00.00 getty
 6096 root      20   0 17404  908  744 S   0,0  0,0   0:00.00 getty
 6097 root      20   0 17404  904  744 S   0,0  0,0   0:00.00 getty
 6098 root      20   0 17404  896  744 S   0,0  0,0   0:00.00 getty
 6099 root      20   0 17404  900  744 S   0,0  0,0   0:00.00 getty
 6100 root      20   0 17404  904  744 S   0,0  0,0   0:00.00 getty
 6108 anarcat   20   0 24248 1040  816 S   0,0  0,0   0:00.08 screen
 6109 anarcat   20   0 25864 2640  796 S   0,0  0,0   0:05.37 screen
 6110 anarcat   20   0 22868 4220 1624 S   0,0  0,1   0:00.06 bash
 6153 root      20   0     0    0    0 S   0,0  0,0   0:00.00 flush-254:8
 6174 anarcat   20   0 22872 4224 1624 S   0,0  0,1   0:00.07 bash
 6238 anarcat   20   0  103m  12m 5232 S   0,0  0,2   0:20.80 irssi
 6252 anarcat   20   0  212m  40m  13m S   0,0  0,5   1:09.74 emacs23
 6534 anarcat   20   0 22876 4400 1716 S   0,0  0,1   0:00.07 bash
 6598 anarcat   20   0  302m 163m 3864 S   0,0  2,0  17:19.63 offlineimap
 6616 anarcat   20   0 22408 2648 2228 S   0,0  0,0   0:05.04 git
 6662 root      20   0     0    0    0 S   0,0  0,0   0:01.25 kworker/1:1
 6666 root      20   0     0    0    0 S   0,0  0,0   0:00.00 flush-254:2
 6897 anarcat   20   0 1011m 159m  47m S   0,0  2,0   4:54.69 chromium
 6902 postfix   20   0 46744 3396 2400 S   0,0  0,0   0:00.09 tlsmgr
 6903 anarcat   20   0  283m 7952 2788 S   0,0  0,1   0:05.03 chromium
 6904 anarcat   20   0  6240  356  272 S   0,0  0,0   0:00.00 chrome-sandbox
 6905 anarcat   20   0  304m  17m  12m S   0,0  0,2   0:00.02 chromium
 6908 anarcat   20   0  320m 5800  764 S   0,0  0,1   0:00.16 chromium
 6950 anarcat   20   0  955m  31m  16m S   0,0  0,4   0:01.68 chromium
 6955 anarcat   20   0  953m  33m  17m S   0,0  0,4   0:01.25 chromium
 6960 anarcat   20   0  959m  36m  15m S   0,0  0,5   0:08.32 chromium
 6965 anarcat   20   0  957m  34m  14m S   0,0  0,4   0:10.25 chromium
 6970 anarcat   20   0 1029m 114m  17m S   0,0  1,4   0:46.99 chromium
 6975 anarcat   20   0  953m  26m  14m S   0,0  0,3   0:00.53 chromium
 6980 anarcat   20   0  959m  35m  16m S   0,0  0,4   0:14.68 chromium
 6985 anarcat   20   0  963m  45m  17m S   0,0  0,6   0:22.59 chromium
 6990 anarcat   20   0  952m  25m  13m S   0,0  0,3   0:00.70 chromium
 6995 anarcat   20   0  960m  36m  18m S   0,0  0,5   0:27.21 chromium
 7040 anarcat   20   0  979m  69m  22m S   0,0  0,9   0:05.41 chromium
 7055 anarcat   20   0  966m  51m  20m S   0,0  0,6   0:04.88 chromium
 7063 anarcat   20   0  967m  42m  18m S   0,0  0,5   0:04.46 chromium
 7072 anarcat   20   0  967m  49m  19m S   0,0  0,6   0:05.17 chromium
 7078 anarcat   20   0  971m  53m  19m S   0,0  0,7   0:05.68 chromium
 7087 anarcat   20   0  963m  51m  19m S   0,0  0,7   0:04.75 chromium
 7096 anarcat   20   0  961m  48m  18m S   0,0  0,6   0:04.75 chromium
 7107 anarcat   20   0  161m 4440 3252 S   0,0  0,1   0:02.90 gvfs-gdu-volume
 7127 anarcat   20   0  967m  55m  22m S   0,0  0,7   1:07.92 chromium
 7183 anarcat   20   0  912m  55m  20m S   0,0  0,7   0:04.97 chromium
 7213 anarcat   20   0  964m  47m  19m S   0,0  0,6   0:04.69 chromium
 7235 anarcat   20   0  965m  65m  21m S   0,0  0,8   0:05.26 chromium
 7258 anarcat   20   0  961m  57m  20m S   0,0  0,7   0:05.39 chromium
 7285 anarcat   20   0 18652 2484 1000 S   0,0  0,0   0:01.25 git
 7434 anarcat   39  19 18124 1960  964 S   0,0  0,0   0:00.29 git
 7445 www-data  20   0  141m  24m 5620 S   0,0  0,3   0:00.71 munin-cgi-graph
 7452 anarcat   20   0  980m  63m  22m S   0,0  0,8   0:02.50 chromium
 7492 anarcat   20   0  978m  64m  21m S   0,0  0,8   0:03.74 chromium
 7498 anarcat   20   0  978m  61m  21m S   0,0  0,8   0:01.42 chromium
 7518 anarcat   20   0  984m  63m  22m S   0,0  0,8   0:04.38 chromium
 7532 anarcat   20   0  973m  56m  21m S   0,0  0,7   0:02.28 chromium
 7813 root      20   0     0    0    0 S   0,0  0,0   0:01.22 kworker/0:2
 7899 root      20   0     0    0    0 S   0,0  0,0   0:00.96 kworker/1:0
 8058 anarcat   20   0 24488 1460 1104 S   0,0  0,0   0:28.34 git
 8107 anarcat   20   0 17412 1384 1100 S   0,0  0,0   0:00.00 git
 8130 anarcat   20   0 23792 7572 1004 S   0,0  0,1   0:41.63 git
 8236 anarcat   39  19 21152 4940  964 S   0,0  0,1   0:00.71 git
 8405 anarcat   20   0 17368 1364 1108 S   0,0  0,0   0:00.54 git
 8407 anarcat   20   0 17464 1336  964 S   0,0  0,0   0:00.03 git
 8610 anarcat   20   0 78228  14m 2860 S   0,0  0,2   0:00.37 xterm
 8614 anarcat   20   0 23084 4548 1796 S   0,0  0,1   0:00.28 bash
 8689 anarcat   20   0 72992 9224 2860 S   0,0  0,1   0:00.38 xterm
 8693 anarcat   20   0 22876 4380 1696 S   0,0  0,1   0:00.07 bash
 8735 root      20   0 99,3m 4616 3576 S   0,0  0,1   0:00.02 gdm-simple-slav
 8736 root      20   0  142m  15m 6664 S   0,0  0,2   0:00.56 Xorg
 8755 Debian-g  20   0 30196 1184  736 S   0,0  0,0   0:00.01 dbus-daemon
 8756 root      20   0  119m 4548 3432 S   0,0  0,1   0:00.08 gdm-session-wor
 8757 root      20   0 55872 2116 1572 S   0,0  0,0   0:00.00 sudo
 8759 aegir     20   0 20640 2248 1660 S   0,0  0,0   0:00.07 bash
 8769 Debian-g  20   0  266m  10m 7720 S   0,0  0,1   0:00.07 gnome-session
 8773 Debian-g  20   0  502m  24m  10m S   0,0  0,3   0:00.85 gnome-settings-
 8780 Debian-g  20   0 51980 2236 1836 S   0,0  0,0   0:00.00 gvfsd
 8784 Debian-g   9 -11  237m 6044 3644 S   0,0  0,1   0:00.07 pulseaudio
 8785 Debian-g  20   0  113m 3196 2448 S   0,0  0,0   0:00.00 gconf-helper
 8787 Debian-g  20   0 52956 3756 1932 S   0,0  0,0   0:00.06 gconfd-2
 8790 Debian-g  20   0  279m 9124 6632 S   0,0  0,1   0:00.12 metacity
 8795 Debian-g  20   0  171m 9328 7056 S   0,0  0,1   0:00.04 polkit-gnome-au
 8796 Debian-g  20   0  362m  16m  11m S   0,0  0,2   0:00.68 gdm-simple-gree
 8812 root      20   0 72360 3916 2948 S   0,0  0,0   0:00.08 gdm-session-wor
 9129 root      20   0     0    0    0 S   0,0  0,0   0:00.74 kworker/0:0
 9311 www-data  20   0  288m  44m  33m S   0,0  0,6   0:07.55 apache2
10405 www-data  20   0  287m  42m  32m S   0,0  0,5   0:06.75 apache2
10406 www-data  20   0  287m  44m  34m S   0,0  0,6   0:07.02 apache2
10409 www-data  20   0  287m  41m  31m S   0,0  0,5   0:06.51 apache2
10412 www-data  20   0  287m  42m  32m S   0,0  0,5   0:06.82 apache2
10413 www-data  20   0  291m  40m  27m S   0,0  0,5   0:06.52 apache2
10415 www-data  20   0  287m  39m  29m S   0,0  0,5   0:06.67 apache2
10416 www-data  20   0  288m  42m  31m S   0,0  0,5   0:06.28 apache2
10417 www-data  20   0  290m  50m  38m S   0,0  0,6   0:07.43 apache2
10878 root      20   0     0    0    0 S   0,0  0,0   0:00.02 kworker/0:3
11037 anarcat   20   0  4180  584  488 S   0,0  0,0   0:00.00 sh
11038 anarcat   20   0 62760 4964 3244 S   0,0  0,1   0:00.48 notmuch
11040 anarcat   20   0  4180  652  544 S   0,0  0,0   0:00.00 post-new
11133 root      20   0 52576 1468 1052 S   0,0  0,0   0:00.00 cron
11134 root      20   0 52576 1464 1052 S   0,0  0,0   0:00.00 cron
11135 root      20   0 52576 1468 1052 S   0,0  0,0   0:00.00 cron
11138 aegir     20   0  4176  576  484 S   0,0  0,0   0:00.00 sh
11139 root      20   0  4176  580  484 S   0,0  0,0   0:00.00 sh
11141 munin     20   0  4176  576  480 S   0,0  0,0   0:00.00 sh
11143 root      20   0  4240  324  248 S   0,0  0,0   0:00.00 xargs
11145 munin     20   0  4176  576  484 S   0,0  0,0   0:00.00 munin-cron
11166 anarcat   20   0 61364 2988 2468 R   0,0  0,0   0:00.00 notmuch
11167 anarcat   20   0 61364  752  212 R   0,0  0,0   0:00.00 notmuch
12004 www-data  20   0  292m  52m  37m S   0,0  0,7   0:09.05 apache2
12172 anarcat   20   0  4092  580  480 S   0,0  0,0   0:00.00 emacsclient
12465 spampd    20   0  149m  65m 2840 S   0,0  0,8   0:01.41 spampd
12468 spampd    20   0  150m  67m 2840 S   0,0  0,8   0:01.07 spampd
12895 anarcat   20   0  970m  51m  21m S   0,0  0,6   0:01.80 chromium
13761 anarcat   20   0     0    0    0 Z   0,0  0,0   0:00.00 git
16056 root      20   0 21800 1464  452 S   0,0  0,0   0:00.00 udevd
16057 root      20   0 21800 1340  332 S   0,0  0,0   0:00.00 udevd
16655 www-data  20   0  147m 8232  700 S   0,0  0,1   0:00.04 apache2
17006 anarcat   20   0  526m  29m  17m S   0,0  0,4   0:19.95 gmpc
17959 www-data  20   0 70828  24m 2172 S   0,0  0,3   1:22.37 munin-cgi-html
18082 anarcat   20   0 71776 8164 2820 S   0,0  0,1   0:00.12 xterm
18086 anarcat   20   0 23048 4664 1804 S   0,0  0,1   0:00.18 bash
18571 anarcat   20   0  258m  11m 8296 S   0,0  0,1   0:00.11 xfrun4
19692 spampd    20   0  148m  64m 2832 S   0,0  0,8   0:00.63 spampd
25574 anarcat   20   0  952m  31m  15m S   0,0  0,4   0:00.71 chromium
25590 anarcat   20   0     0    0    0 Z   0,0  0,0   0:41.75 chromium
"""]]

And here's the process tree:

[[!format sh """
$ ps -O start xf | grep git-annex
 8610 13:03:21 S ?        00:00:00 xterm -class UXTerm -title uxterm -u8
 8614 13:03:21 S pts/4    00:00:00  \_ bash
12172 13:13:15 S pts/4    00:00:00      \_ emacsclient --alternate-editor=vim -c install.hostmaster.inc
 8099 13:00:20 S ?        07:40:31 /usr/local/bin/git-annex assistant --startdelay=5s
 8107 13:00:20 S ?        00:00:00  \_ git --git-dir=/srv/books/.git --work-tree=/srv/books -c core.bare=false cat-file --batch
 8405 13:00:28 S ?        00:00:00  \_ git --git-dir=/srv/books/.git --work-tree=/srv/books -c core.bare=false cat-file --batch
 8407 13:00:28 S ?        00:00:00  \_ git --git-dir=/srv/books/.git --work-tree=/srv/books -c core.bare=false check-ignore -z --stdin --verbose --non-matching
 8045 13:00:11 S ?        07:35:36 /usr/local/bin/git-annex assistant --startdelay=5s
 8058 13:00:11 S ?        00:00:28  \_ git --git-dir=/srv/mp3/.git --work-tree=/srv/mp3 cat-file --batch
 8130 13:00:21 S ?        00:00:41  \_ git --git-dir=/srv/mp3/.git --work-tree=/srv/mp3 check-ignore -z --stdin --verbose --non-matching
 8236 13:00:24 S ?        00:00:00  \_ git --git-dir=/srv/mp3/.git --work-tree=/srv/mp3 check-attr -z --stdin annex.backend annex.numcopies --
 7107 12:58:40 S ?        00:00:02 /usr/lib/gvfs/gvfs-gdu-volume-monitor
 6525 12:57:25 S ?        07:29:13 /usr/local/bin/git-annex assistant --startdelay=5s
 6616 12:57:34 S ?        00:00:05  \_ git --git-dir=/srv/video/.git --work-tree=/srv/video cat-file --batch
 7285 12:58:45 S ?        00:00:01  \_ git --git-dir=/srv/video/.git --work-tree=/srv/video check-ignore -z --stdin --verbose --non-matching
 7434 12:58:51 S ?        00:00:00  \_ git --git-dir=/srv/video/.git --work-tree=/srv/video check-attr -z --stdin annex.backend annex.numcopies --
 4209 19:30:17 S ?        00:01:06  \_ /usr/local/bin/git-annex transferkeys --readfd 25 --writefd 24
 4240 19:30:17 S ?        00:00:00      \_ git --git-dir=/srv/video/.git --work-tree=/srv/video cat-file --batch
13761 23:56:38 Z ?        00:00:00      \_ [git] <defunct>
 6252 12:56:59 S ?        00:01:09 /usr/bin/emacs23
"""]]