Merge branch 'master' of ssh://git-annex.branchable.com

10 files changed, 279 insertions, 0 deletions

diff --git a/doc/bugs/assistant_expensive_scan_unnecessarily_queues_files/comment_2_a7e9d8ec500399dd6794e168f85e6a5c._comment b/doc/bugs/assistant_expensive_scan_unnecessarily_queues_files/comment_2_a7e9d8ec500399dd6794e168f85e6a5c._comment
new file mode 100644
index 000000000..589086f6e
--- /dev/null
+++ b/doc/bugs/assistant_expensive_scan_unnecessarily_queues_files/comment_2_a7e9d8ec500399dd6794e168f85e6a5c._comment
@@ -0,0 +1,40 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmXSkgjC_ypUQafVwvHTLsStrkiXH8CfHU"
+ nickname="Matthias"
+ subject="comment 2"
+ date="2014-06-13T18:30:25Z"
+ content="""
+Thanks for your fast reply, Joey. I appended the shortened config below. Wanted and groupwanted were untouched and therefore still commented out. I removed the hash from wanted to explicitly set it standard. Now the assistant behaves as expected. However I’m wondering:
+
+* Shouldn’t ‘standard’ be the default rule to apply for wanted/groupwanted in case both are not explicitly set in the config? I assumed this to be the case.
+* Shouldn’t “find --want-get” list the identical set of files that is also requested by the assistant? Especially this one confused me.
+
+[[!format sh \"\"\"
+# Repository trust configuration
+# (for macbook)
+#trust 3938865a-ce40-4166-a918-2012078846c7 = semitrusted
+
+# Repository groups
+# (for macbook)
+group 3938865a-ce40-4166-a918-2012078846c7 = manual
+
+# Repository preferred contents
+# (for macbook)
+#wanted 3938865a-ce40-4166-a918-2012078846c7 = standard
+
+# Group preferred contents
+# (Used by repositories with \"groupwanted\" in their preferred contents)
+(config defaults - all commented out)
+
+# Standard preferred contents
+(config defaults - all commented out)
+
+# Repository required contents
+# (for macbook)
+#required 3938865a-ce40-4166-a918-2012078846c7 = 
+
+# Scheduled activities
+# (for macbook)
+#schedule 3938865a-ce40-4166-a918-2012078846c7 = 
+\"\"\"]]
+"""]]
diff --git a/doc/bugs/assistant_expensive_scan_unnecessarily_queues_files/comment_3_a0824e00f04f95c39823f29d6c76e7fe._comment b/doc/bugs/assistant_expensive_scan_unnecessarily_queues_files/comment_3_a0824e00f04f95c39823f29d6c76e7fe._comment
new file mode 100644
index 000000000..58f2069c1
--- /dev/null
+++ b/doc/bugs/assistant_expensive_scan_unnecessarily_queues_files/comment_3_a0824e00f04f95c39823f29d6c76e7fe._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 3"
+ date="2014-06-13T18:35:53Z"
+ content="""
+I need to look into why --want-get didn't behave as expected.
+
+It would probably make sense to make standard be the default. There could be upgrade concerns though.
+"""]]
diff --git a/doc/bugs/failed_sync_with_direct_mode_repo/comment_1_fb4026cc81eb0ec1e656e4a81ffacc4f._comment b/doc/bugs/failed_sync_with_direct_mode_repo/comment_1_fb4026cc81eb0ec1e656e4a81ffacc4f._comment
new file mode 100644
index 000000000..8b75fa67c
--- /dev/null
+++ b/doc/bugs/failed_sync_with_direct_mode_repo/comment_1_fb4026cc81eb0ec1e656e4a81ffacc4f._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 1"
+ date="2014-06-16T15:10:42Z"
+ content="""
+Reproduced this as described.
+
+Only happens when using the assistant in the second repository, which is a ssh remote of the first, and when there is not also a remote going the other way (so does not affect pairing).
+
+As I thought, the problem is that the master branch is not updated by the assistant when in direct mode, and while annex/direct/master is pulled by the other assistant, it does not merge it.
+
+"""]]
diff --git a/doc/design/assistant/inotify/comment_7_00809aaad6b68f189a9cc42af810a0a6._comment b/doc/design/assistant/inotify/comment_7_00809aaad6b68f189a9cc42af810a0a6._comment
new file mode 100644
index 000000000..7e1ad7db5
--- /dev/null
+++ b/doc/design/assistant/inotify/comment_7_00809aaad6b68f189a9cc42af810a0a6._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawnwfqF4wL6l_O26RyzoBowUMvQ_955Vpao"
+ nickname="Markus"
+ subject="comment 7"
+ date="2014-06-14T06:29:55Z"
+ content="""
+FWIW: Removing the [raspi-copies-and-fills](https://github.com/simonjhall/copies-and-fills) package ('apt-get purge raspi-copies-and-fills') stops annex from printing the message over and over again. According to the package's description though, you suffer a performance penalty without the package.
+"""]]
diff --git a/doc/forum/View_performance_with_7__44__000_files.mdwn b/doc/forum/View_performance_with_7__44__000_files.mdwn
new file mode 100644
index 000000000..21491c77f
--- /dev/null
+++ b/doc/forum/View_performance_with_7__44__000_files.mdwn
@@ -0,0 +1,26 @@
+I've imported about half of my photos into an annex on an external HDD,
+using metadata extensively for EXIF info, including place names.
+
+Checking out a new view is slower than I expected, at ~6 minutes.
+Is this expected behavior, am I pushing the limits of file count already?
+
+Is there anything I can do to speed things up?
+
+    % ls -1 | wc -l
+        7050
+    
+    % git branch -vv
+      git-annex                      4e590d4 update
+    * master                         985ba54 add jen's phone backups
+      views/Year=_;Month=_;Address=_ 795a58b refs/heads/views/Year=_;Month=_;Address=_
+    
+    % /usr/bin/time -p git annex view "Year=*" "Address=*" 
+    view  (searching...)
+    
+    Checking out files: 100% (12789/12789), done.
+    Switched to branch 'views/Year=_;Address=_'
+    ok
+    real       376.80
+    user        17.98
+    sys          9.66
+
diff --git a/doc/forum/View_performance_with_7__44__000_files/comment_1_e45ea752100d09d29efb6136a722eab3._comment b/doc/forum/View_performance_with_7__44__000_files/comment_1_e45ea752100d09d29efb6136a722eab3._comment
new file mode 100644
index 000000000..cf6f140b2
--- /dev/null
+++ b/doc/forum/View_performance_with_7__44__000_files/comment_1_e45ea752100d09d29efb6136a722eab3._comment
@@ -0,0 +1,34 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmvzzyDA8uXFz8yokeCrepbh8PwWe_WrjE"
+ nickname="Michael"
+ subject="debug output"
+ date="2014-06-16T14:17:06Z"
+ content="""
+So after looking for a debug flag, I see that it's spending all its time in cat-file, which I guess is probably not a surprise:
+
+    % /usr/bin/time -p git annex view \"Year=*\" \"Address=*\" -d
+    view  [2014-06-16 06:29:12 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"symbolic-ref\",\"HEAD\"]
+    [2014-06-16 06:29:12 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"show-ref\",\"refs/heads/master\"]
+    [2014-06-16 06:29:12 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"symbolic-ref\",\"HEAD\"]
+    [2014-06-16 06:29:12 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"show-ref\",\"refs/heads/master\"]
+    (searching...)
+    [2014-06-16 06:29:12 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"/Volumes/Four TB Backup/Photos\"]
+    [2014-06-16 06:29:12 PDT] feed: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"update-index\",\"-z\",\"--index-info\"]
+    [2014-06-16 06:29:12 PDT] chat: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"]
+    [2014-06-16 06:29:12 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"show-ref\",\"git-annex\"]
+    [2014-06-16 06:29:12 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"]
+    [2014-06-16 06:29:12 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"log\",\"refs/heads/git-annex..4e590d433e01886e2823c3316d18b7e3cbafe227\",\"--oneline\",\"-n1\"]
+    [2014-06-16 06:29:12 PDT] chat: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"cat-file\",\"--batch\"]
+    [2014-06-16 06:34:54 PDT] read: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"write-tree\"]
+    [2014-06-16 06:34:55 PDT] chat: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"commit-tree\",\"c0fd1ea9ccae70a7353a2e787089b88115a42b7a\"]
+    [2014-06-16 06:34:55 PDT] call: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"update-ref\",\"refs/heads/views/Year=_;Address=_\",\"249c37d78e45d1fccfcb00f40337e05cb69fb64c\"]
+    
+    [2014-06-16 06:34:55 PDT] call: git [\"--git-dir=/Volumes/Four TB Backup/Photos/.git\",\"--work-tree=/Volumes/Four TB Backup/Photos\",\"checkout\",\"views/Year=_;Address=_\"]
+    Checking out files: 100% (12789/12789), done.
+    Switched to branch 'views/Year=_;Address=_'
+    ok
+    real       362.14
+    user        18.16
+    sys          7.70
+
+"""]]
diff --git a/doc/forum/downloading_from_moodle.mdwn b/doc/forum/downloading_from_moodle.mdwn
new file mode 100644
index 000000000..ad38d7f1f
--- /dev/null
+++ b/doc/forum/downloading_from_moodle.mdwn
@@ -0,0 +1,76 @@
+my school uses moodle for our classes.  We have to sign in and then manually click to download each file, assignment and video uploaded.  I asked the school's tech administrator if there was a direct way I could access the videos through the ssh access they've given us to one of the servers, but he said it wasn't possible.
+
+when I click on the link shown, the location I see is: http://moodle.jct.ac.il/mod/resource/view.php?id=135374
+Inspect element gives more information. the response from the server is:
+
+    Remote Address:147.161.6.59:80
+    Request URL:http://moodle.jct.ac.il/mod/resource/view.php?id=135374
+    Request Method:GET
+    Status Code:303 See Other
+    Request Headersview source
+    Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
+    Accept-Encoding:gzip,deflate,sdch
+    Accept-Language:en-US,en;q=0.8,he;q=0.6
+    Connection:keep-alive
+    Cookie:visid_incap_97364=qJx2WaKqQfGidGf9VfM6QWrnlFIAAAAAQUIPAAAAAAC5EXcbt00vqNu9jdVDwEDN; __utma=98014340.1409421308.1381214363.1381214363.1390387318.2; __utmz=98014340.1381214363.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); MoodleSession5771=7s1cfqfo4ahdtmna5h7vserg97; MOODLEID1_5771=%257F%25D39%2522N%25B4%25AFY
+    DNT:1
+    Host:moodle.jct.ac.il
+    Referer:http://moodle.jct.ac.il/course/view.php?id=20151
+    User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36
+    Query String Parametersview sourceview URL encoded
+    id:135374
+    Response Headersview source
+    Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
+    Connection:Keep-Alive
+    Content-Encoding:gzip
+    Content-Language:he
+    Content-Length:503
+    Content-Type:text/html
+    Date:Mon, 16 Jun 2014 12:31:22 GMT
+    Expires:Thu, 19 Nov 1981 08:52:00 GMT
+    Keep-Alive:timeout=15, max=100
+    Location:http://moodle.jct.ac.il/pluginfile.php/288409/mod_resource/content/0/movie%205773/150151.5773.week1.wmv?forcedownload=1
+    Pragma:no-cache
+    Server:Apache/2.2.14 (Ubuntu)
+    Vary:Accept-Encoding
+    X-Powered-By:PHP/5.3.2-1ubuntu4.24
+
+this then pulls the following: 
+
+    Remote Address:147.161.6.59:80
+    Request URL:http://moodle.jct.ac.il/pluginfile.php/288409/mod_resource/content/0/movie%205773/150151.5773.week1.wmv?forcedownload=1
+    Request Method:GET
+    Status Code:200 OK
+    Request Headersview source
+    Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
+    Accept-Encoding:gzip,deflate,sdch
+    Accept-Language:en-US,en;q=0.8,he;q=0.6
+    Connection:keep-alive
+    Cookie:visid_incap_97364=qJx2WaKqQfGidGf9VfM6QWrnlFIAAAAAQUIPAAAAAAC5EXcbt00vqNu9jdVDwEDN; __utma=98014340.1409421308.1381214363.1381214363.1390387318.2; __utmz=98014340.1381214363.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); MoodleSession5771=7s1cfqfo4ahdtmna5h7vserg97; MOODLEID1_5771=%257F%25D39%2522N%25B4%25AFY
+    DNT:1
+    Host:moodle.jct.ac.il
+    Referer:http://moodle.jct.ac.il/course/view.php?id=20151
+    User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36
+    Query String Parametersview sourceview URL encoded
+    forcedownload:1
+    Response Headersview source
+    Accept-Ranges:bytes
+    Cache-Control:max-age=86400
+    Connection:Keep-Alive
+    Content-Disposition:attachment; filename="150151.5773.week1.wmv"
+    Content-Length:1353673203
+    Content-Type:application/x-forcedownload
+    Date:Mon, 16 Jun 2014 12:31:23 GMT
+    ETag:675e7d2cffd7a79afd8686c59ff2533f9e3508b7
+    Expires:Tue, 17 Jun 2014 12:31:23 GMT
+    Keep-Alive:timeout=15, max=99
+    Last-Modified:Fri, 19 Jul 2013 17:06:54 GMT
+    Pragma:
+    Server:Apache/2.2.14 (Ubuntu)
+    X-Powered-By:PHP/5.3.2-1ubuntu4.24
+
+when I right-click on the second one in the inspect element window, I can select "copy as cURL" i get:
+
+    curl 'http://moodle.jct.ac.il/pluginfile.php/288409/mod_resource/content/0/movie%205773/150151.5773.week1.wmv?forcedownload=1' -H 'DNT: 1' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: en-US,en;q=0.8,he;q=0.6' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Referer: http://moodle.jct.ac.il/course/view.php?id=20151' -H 'Cookie: visid_incap_97364=qJx2WaKqQfGidGf9VfM6QWrnlFIAAAAAQUIPAAAAAAC5EXcbt00vqNu9jdVDwEDN; __utma=98014340.1409421308.1381214363.1381214363.1390387318.2; __utmz=98014340.1381214363.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); MoodleSession5771=7s1cfqfo4ahdtmna5h7vserg97; MOODLEID1_5771=%257F%25D39%2522N%25B4%25AFY' -H 'Connection: keep-alive' --compressed
+
+if I append " > week1.wmv" to the end of that output it downloads fine.  How can I add this file to a git-annex repo?
diff --git a/doc/forum/recovery_from_failed_merge.mdwn b/doc/forum/recovery_from_failed_merge.mdwn
new file mode 100644
index 000000000..802c66e51
--- /dev/null
+++ b/doc/forum/recovery_from_failed_merge.mdwn
@@ -0,0 +1,7 @@
+Starting the assistant version 5.20140613 on my  repository (~60GB), it performed a merge
+with an offline-repository, deleting a large part of the files.
+Since the repo is in direct mode, I cannot do a git revert. The other repo is not available anymore
+
+Any way of getting them back?
+Also, du shows me that the .git/annex/objects folder has approximately the size of my repo before the incident.
+
diff --git a/doc/todo/wishlist__91__webapp__93__:_add_an_option_to_install__SSH_key_on_remote/comment_2_06230669218541ac392d674bedd43176._comment b/doc/todo/wishlist__91__webapp__93__:_add_an_option_to_install__SSH_key_on_remote/comment_2_06230669218541ac392d674bedd43176._comment
new file mode 100644
index 000000000..79ebb5e7c
--- /dev/null
+++ b/doc/todo/wishlist__91__webapp__93__:_add_an_option_to_install__SSH_key_on_remote/comment_2_06230669218541ac392d674bedd43176._comment
@@ -0,0 +1,43 @@
+[[!comment format=mdwn
+ username="http://olivier.mehani.name/"
+ nickname="olivier-mehani"
+ subject="Manual solution"
+ date="2014-06-14T13:59:38Z"
+ content="""
+My problem stems from the fact that I manually git clone the git-annex repo, which prevents the assistant from creating the setup to use passwordless keys. I just reverse-engineered a working setup to work up what I was missing. I jot it down here for reference, but I guess the bottomline is that if you want to use the assistant with a repo, do it from the start.
+
+I assume that the client has a clone of the git(-annex) repo of the server.
+
+    client$ git clone server:annex
+
+Our goal is to let git-annex on the client know that there is a specific key to use when connecting to server that will let it access the git-annex-shell (without a password). We first create the key.
+
+    client:~$ ssh-keygen -t rsa -f ~/.ssh/git-annex/key.git-annex-server-user_annex
+    [enter an empty passphrase]
+
+We can then create a virtual SSH host on the client that will use this key to connect to the server, in client:~/.ssh/config:
+
+    # Added manually for git-annex
+    Host git-annex-server-user_annex
+    	Hostname server
+    	Port 22
+    	IdentityFile ~/.ssh/git-annex/key.git-annex-server-user_annex
+    	IdentitiesOnly yes
+    	StrictHostKeyChecking yes
+
+(git-annex seems to use .2F (%2F) to encode path separators in the filenames.)
+
+The server then needs to know to let the key in, but only for git-annex in the specific folder. This is done in server:.ssh/authorized_keys:
+
+    command=\"GIT_ANNEX_SHELL_DIRECTORY='annex' ~/.ssh/git-annex-shell\",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAA... user@client
+
+The bit starting with ssh-rsa is the public key created in client:.ssh/git-annex/key.git-annex-server-user_annex.pub at the same time as the private key.
+
+Finally, all that remains is to change the remote in the client clone to use the virtual SSH host.
+
+    client:~/annex $ git remote set-url origin ssh://user@git-annex-server-user_annex/~/annex
+    client:~/annex $ git remote set-url origin --push ssh://user@git-annex-server-user_annex/~/annex
+
+If everything worked, a sync from the client should now work without asking for a password, and starting the assistant will not either.
+
+"""]]
diff --git a/doc/todo/wishlist__91__webapp__93__:_add_an_option_to_install__SSH_key_on_remote/comment_3_002afd775b82a0ced609c8305803a6c2._comment b/doc/todo/wishlist__91__webapp__93__:_add_an_option_to_install__SSH_key_on_remote/comment_3_002afd775b82a0ced609c8305803a6c2._comment
new file mode 100644
index 000000000..2515349a6
--- /dev/null
+++ b/doc/todo/wishlist__91__webapp__93__:_add_an_option_to_install__SSH_key_on_remote/comment_3_002afd775b82a0ced609c8305803a6c2._comment
@@ -0,0 +1,22 @@
+[[!comment format=mdwn
+ username="http://olivier.mehani.name/"
+ nickname="olivier-mehani"
+ subject="comment 3"
+ date="2014-06-14T14:15:55Z"
+ content="""
+After having done that on my first test repo, git-annex could sync, but failed to get the files.
+
+    client:~/annex$ git annex get file
+    get file (not available) 
+      Try making some of these repositories available:
+      	12345678-90ab-cdef-1234567890abcdef1 -- user@server:~/annex [origin]
+    
+      (Note that these git remotes have annex-ignore set: origin)
+    failed
+    git-annex: get: 1 failed
+ 
+The note helps: the problem is with the origin remote having annex-ignore set. git-annex therefore ignores it. This is easily fixed by just setting the flag to false.
+
+    client:~/annex$ git config remote.origin.annex-ignore false
+
+"""]]