From fba5eba1076199c55b9a836e5cad44311d0b66d3 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Tue, 4 Aug 2015 12:21:22 -0400 Subject: response --- ...ent_3_15c4b941a4af8354fee3209c64e89e33._comment | 56 ++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 doc/todo/speed_up_transfers_over_ssh+rsync_--_directly_reuse_the_same_connection__63__/comment_3_15c4b941a4af8354fee3209c64e89e33._comment diff --git a/doc/todo/speed_up_transfers_over_ssh+rsync_--_directly_reuse_the_same_connection__63__/comment_3_15c4b941a4af8354fee3209c64e89e33._comment b/doc/todo/speed_up_transfers_over_ssh+rsync_--_directly_reuse_the_same_connection__63__/comment_3_15c4b941a4af8354fee3209c64e89e33._comment new file mode 100644 index 000000000..d8a85006c --- /dev/null +++ b/doc/todo/speed_up_transfers_over_ssh+rsync_--_directly_reuse_the_same_connection__63__/comment_3_15c4b941a4af8354fee3209c64e89e33._comment @@ -0,0 +1,56 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2015-08-04T16:07:23Z" + content=""" +I don't know how you used ts to get that output, but it does not appear to +be accurate. Here's `git annex get | ts "%b %d %H:%M:%.s"`: + + Aug 04 12:10:1438704612.238164 get bob (from origin...) + Aug 04 12:10:1438704612.238336 SHA256E-s30--7c3722f359d8cfbc594e5dec6d7f096bf4e88adbfb786548a607708d32ef49bb + 30 100% 29.30kB/s 0:00:00 (xfr#1, to-chk=0/1) + Aug 04 12:10:1438704612.597912 ok + Aug 04 12:10:1438704612.642574 get bob2 (from origin...) + Aug 04 12:10:1438704612.642653 SHA256E-s30--22fdcad0fba3537a7a0bc2b824f805fdd2d2c032b6450352cb8d099d03c8d796 + 30 100% 29.30kB/s 0:00:00 (xfr#1, to-chk=0/1) + Aug 04 12:10:1438704613.001088 ok + Aug 04 12:10:1438704613.043018 get bob3 (from origin...) + Aug 04 12:10:1438704613.043105 SHA256E-s30--9f0e99611fe904e173fb1c81d57ba9f01a111afdf222a7763844c7b893af86ff + 30 100% 29.30kB/s 0:00:00 (xfr#1, to-chk=0/1) + Aug 04 12:10:1438704613.393817 ok + Aug 04 12:10:1438704613.421080 get bob5 (from origin...) + Aug 04 12:10:1438704613.421198 SHA256E-s30--7b851aa7791136f783271c109c287614bf2c0e9014d0fab50b1bf32f4ad4678e + 30 100% 29.30kB/s 0:00:00 (xfr#1, to-chk=0/1) + Aug 04 12:10:1438704613.769410 ok + Aug 04 12:10:1438704613.800044 get bob6 (from origin...) + Aug 04 12:10:1438704613.800133 SHA256E-s33--a165803131075f75132f632a6f295b12910f84a5d1776b60d1d998b96a6f20d5 + 33 100% 32.23kB/s 0:00:00 (xfr#1, to-chk=0/1) + Aug 04 12:10:1438704614.148597 ok + Aug 04 12:10:1438704614.200786 get bob7 (from origin...) + Aug 04 12:10:1438704614.200881 SHA256E-s30--8b5b0b239b465a407ca98c8dc82a0081ee5ced4f7402854dc9afac75b65b5b51 + 30 100% 29.30kB/s 0:00:00 (xfr#1, to-chk=0/1) + Aug 04 12:10:1438704614.560315 ok + Aug 04 12:10:1438704614.602174 get bob8 (from origin...) + Aug 04 12:10:1438704614.602255 SHA256E-s30--70657536d54051bd020f984f866b017d5919b7705bf08ffa2786c0dd14f90280 + 30 100% 29.30kB/s 0:00:00 (xfr#1, to-chk=0/1) + Aug 04 12:10:1438704614.961264 ok + +Note that this shows up to 3 files sent per second. The output you pasted shows not 1 file/s, but 2 or almost 3. + +There are no 1s (or other) sleeps. This is quite likely simply the overhead of +git-annex needing to query git to work out what remote each file is located on, +coupled with the overhead of needing to start a new git-annex-shell and rsync +processes. + +Using -J4 or so will speed this up quite a lot. + +Without -J4: + + 0.22user 0.09system 0:04.84elapsed 6%CPU (0avgtext+0avgdata 33644maxresident)k + 0inputs+832outputs (0major+16191minor)pagefaults 0swaps + +With -J4: + + 0.34user 0.06system 0:01.35elapsed 30%CPU (0avgtext+0avgdata 35996maxresident)k + 0inputs+752outputs (0major+17451minor)pagefaults 0swaps +"""]] -- cgit v1.2.3