Merge branch 'master' of ssh://git-annex.branchable.com

2 files changed, 199 insertions, 0 deletions

diff --git a/doc/bugs/Assistant_has_created_155_semitrusted_repositories.mdwn b/doc/bugs/Assistant_has_created_155_semitrusted_repositories.mdwn
new file mode 100644
index 000000000..b1caf530d
--- /dev/null
+++ b/doc/bugs/Assistant_has_created_155_semitrusted_repositories.mdwn
@@ -0,0 +1,191 @@
+### Please describe the problem.
+git annex status reports 160 semitrusted repositories. Four of them are the ones I created (only via webapp, I think I never edited any git-annex config file). One is 	00000000-0000-0000-0000-000000000001 -- web
+ and although I do not know what it is, it is not something new. The remaining 155 appeared spontaneously after several Gb of data (mostly many small files) were added to an Annex (in an 'archive' directory) operated in direct mode by the assistant.
+
+
+
+### What steps will reproduce the problem?
+Add several Gb of files was enough to trigger this problem, but I did not try to reproduce it. It happened the day I installed the 4.20131106~bpo70+1 version.
+
+### What version of git-annex are you using? On what operating system?
+
+4.20131106~bpo70+1 on debian squeeze (7.2), with git  1.8.4.rc3.
+
+
+### Please provide any additional information below.
+May be related or not: at some point the webapp displayed two warning boxes. One of them held a message that I did not wirte down and proposed to "Restart the thread". This apparently worked since the box disappeared. The other warning box indicated "NetWatcherFallback crashed: unknown response from git cat-file" and proposed to restart the thread. Trying to "restart the thread" via the provided button just did not trigger any response of the webapp which seemed dead at that point.
+
+In spite of the git annex status shown below, the webapp still shows only the expected four repositories.
+
+Output of git annex status (hostname and xmpp account name were edited away):
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+repository mode: direct
+trusted repositories: 
+0
+semitrusted repositories: 160
+	00000000-0000-0000-0000-000000000001 -- web
+ 	0ab193eb-0c76-4559-a93c-2e30ed8630a8 -- someMachineIown_datadir (archive)
+ 	1384784127.91222s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384784164.437824s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384784176.944372s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384784179.254498s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384785147.558938s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785147.717223s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785159.041203s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785159.199504s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785185.79485s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785187.318128s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785215.236504s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785215.389096s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785313.539843s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785313.701305s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785315.596206s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785344.184461s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785348.192805s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785402.70316s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785406.524044s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384785446.074236s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384873605.313126s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384873697.029999s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384873761.687234s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384873774.608376s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384926279.456728s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384926368.736s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384926454.99433s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384926494.152645s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384926504.438232s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384934790.89717s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384934848.757067s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384934899.087168s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384934908.238587s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384948772.14552s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384948805.441196s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384948813.397132s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384948921.45481s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384948924.855852s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384949073.988946s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384949082.298976s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384949399.608138s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384949581.12213s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384949583.9923s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384949700.22807s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384949765.484768s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955202.85962s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955230.953995s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955402.534938s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955457.1885s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955524.603709s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955611.891061s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955677.84592s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955689.293082s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955894.057476s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955910.723021s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955914.732132s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955968.717875s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384955969.634658s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956004.284925s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956029.567195s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956188.628995s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956379.844701s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956381.613833s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956387.923418s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956395.418701s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956408.792928s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956504.019733s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956519.578085s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384956524.419783s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384965891.562742s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384965891.815119s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384965903.355602s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384965905.276128s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384965978.806653s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384965979.393089s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966097.495566s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966097.704474s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966154.97658s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966156.967406s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966233.310488s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966233.522324s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966241.284523s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966241.475381s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966301.688497s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966303.427685s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966392.875983s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966393.38718s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966404.708568s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966406.441164s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966553.557387s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966555.752786s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966653.725847s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966654.23288s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966695.201885s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966695.689398s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966784.556877s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966786.574886s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966791.446852s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966793.218318s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384966884.335685s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384966886.147083s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967054.857465s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967055.158871s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967190.980027s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967193.176584s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967328.93796s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967330.428095s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967526.127311s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967526.588491s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967627.132549s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967627.685201s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967686.283694s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967686.728086s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967768.270887s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967768.58402s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967769.245615s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967771.122238s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967813.8197s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967814.168477s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384967915.243469s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384967917.020051s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968031.757775s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968032.190452s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968035.733635s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968036.03299s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968144.555556s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968144.714535s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968150.090148s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968150.820567s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968304.393177s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968304.613624s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968604.499519s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968604.813256s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968702.566939s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968704.427767s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968725.375289s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968725.939271s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384968798.402904s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384968798.659754s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384969055.285004s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384969055.715448s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384969159.885115s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384969162.382266s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384969184.633052s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384969185.413769s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384969374.791849s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384969377.497842s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384969475.469111s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384969489.697737s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384969492.087023s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384969492.58214s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384969784.725195s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384969786.49773s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	1384969814.984624s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
+ 	1384969815.397676s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
+ 	391b0557-dc68-4e40-b6d0-da3033588753 -- here (client)
+ 	668ef9d8-68c6-484e-89e5-06634d590a11 -- rsync.net_datadir_annex (transfer)
+ 	b0d3c000-0ac9-4a05-aef4-47f826d5c759 -- user.name (client)
+
+
+# End of transcript or log.
+"""]]
diff --git a/doc/forum/s3_vs_ssh_Performance_Problems.mdwn b/doc/forum/s3_vs_ssh_Performance_Problems.mdwn
new file mode 100644
index 000000000..5176e196f
--- /dev/null
+++ b/doc/forum/s3_vs_ssh_Performance_Problems.mdwn
@@ -0,0 +1,8 @@
+I backup/sync my music using annex. I used to have one repo with 3 clones, one full copy on my vps, one full copy and one partial copy on my laptops. I decided to move all data to s3. Declared my vps repo dead, purged history (I do not care about history for this particular repo) pushed the git repo to a different computer (bare repo no data) and data to s3 (gpg encrypted). I've just finished uploading all files (around 200gb) couple files failed during the initial upload so I did a `git annex copy --quiet --to mys3 --fast` this command used to take 15 20 secs on my laptop when sending data to vps using ssh but now it took 2 hours to complete (pushing mem usage to 90%).
+
+I have one other repo (1.5gb in size 42k files using indirect mode, data gpg encrypted on s3) using the same setup except this repos content has always been on s3  i had the same behavior on this repo too. adding a file and running a copy to push content to s3 took couple hours even if I add a single 1kb file. I used to blame my hard drive, damn thing is slow but now I think this is related to s3. is there any workaround for this?
+
+Both machines are using,
+
+    git-annex version: 4.20130913-gd20a4f2
+    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP Feeds Quvi