Merge branch 'master' of ssh://git-annex.branchable.com

13 files changed, 329 insertions, 2 deletions

diff --git a/doc/assistant/local_pairing_walkthrough.mdwn b/doc/assistant/local_pairing_walkthrough.mdwn
index fe61c71d9..f6282ec28 100644
--- a/doc/assistant/local_pairing_walkthrough.mdwn
+++ b/doc/assistant/local_pairing_walkthrough.mdwn
@@ -38,7 +38,29 @@ git-annex folders. Just like that you can share files.
 
 For local pairing to work, you must have sshd (ssh server daemon) installed and working on all machines involved. That means you must allow at least local connections to sshd. On most Linux distributions, sshd is packaged in either openssh (openSUSE) or openssh-server (Debian). 
 
-It is highly recommended that you disable root login and login from outside of your local network. To do this, after installing sshd, edit the sshd config (usually /etc/ssh/sshd_config file). See man sshd_config for details.
+It is highly recommended that you disable root login, disable password login to sshd and just enable key based authentication instead. No one will be able to login without your key.
+
+To disable root, after installing sshd, edit the sshd config (usually /etc/ssh/sshd_config file) and disable root login by adding:
+
+    PermitRootLogin no
+
+Restart sshd. See man sshd_config for details.
+
+To disable password login and enable key based authentication, edit the sshd config (just like above) by uncommenting and changing the following options:
+
+    ChallengeResponseAuthentication no
+    PasswordAuthentication no
+    UsePAM no
+    
+    PubkeyAuthentication yes
+
+Restart sshd. See man sshd_config for details.
+
+You can also restrict login to your local network only (not allow internet users from trying to log into your computer). Edit the hosts.deny file (usually /etc/hosts.deny) by adding the following:
+
+    sshd : ALL EXCEPT LOCAL
+
+Do note that restricting login to your local network may or may not block git-annex. Also note that this will not work on Mac OSX because Apple decided to disable this feature and replace it with a crippled version made by Apple.
 
 ## Tips
 
diff --git a/doc/bugs/Large_unannex_operations_result_in_stale_symlinks_and_data_loss/comment_10_52364dc5b1b43b51748453d1896e35c6._comment b/doc/bugs/Large_unannex_operations_result_in_stale_symlinks_and_data_loss/comment_10_52364dc5b1b43b51748453d1896e35c6._comment
new file mode 100644
index 000000000..c0b286a2b
--- /dev/null
+++ b/doc/bugs/Large_unannex_operations_result_in_stale_symlinks_and_data_loss/comment_10_52364dc5b1b43b51748453d1896e35c6._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmVV_nBwlsyCv53BXoJt8YpCX_wZPfzpyo"
+ nickname="Peter"
+ subject="Progress"
+ date="2013-10-10T01:17:08Z"
+ content="""
+Is there any type of script / tool / patch which does the --fast but with a copy instead of only a hard link?  Can someone point me towards how I'm supposed to do this?  I'm a technical user, however I don't really fancy having try to go learn the source code of git-annex to fix this really bad flaw :-/ 
+"""]]
diff --git a/doc/bugs/Large_unannex_operations_result_in_stale_symlinks_and_data_loss/comment_11_99b4db1841f8630a9c5efd08910e87a3._comment b/doc/bugs/Large_unannex_operations_result_in_stale_symlinks_and_data_loss/comment_11_99b4db1841f8630a9c5efd08910e87a3._comment
new file mode 100644
index 000000000..4e55bd020
--- /dev/null
+++ b/doc/bugs/Large_unannex_operations_result_in_stale_symlinks_and_data_loss/comment_11_99b4db1841f8630a9c5efd08910e87a3._comment
@@ -0,0 +1,104 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmVV_nBwlsyCv53BXoJt8YpCX_wZPfzpyo"
+ nickname="Peter"
+ subject="Productive Annoyance"
+ date="2013-10-10T04:30:47Z"
+ content="""
+Ok, so I'm annoyed by this enough (and desperate enough to want to get my data back) that I wrote up a few scripts to help with this.  I make no claims regarding how well these will work, but they seem to work with some minimal testing on a Fedora 17 machine.
+
+READ THROUGH THESE SCRIPTS BEFORE RUNNING THEM TO MAKE SURE YOU ARE OK WITH WHAT THEY ARE DOING!!!
+
+First, a script to create a bad git-annex: one with missing files (with a few corner case names) after a git unannex. Specify the directory you'd like to make the annex at the top of the file.  ALL CONTENTS OF THIS DIRECTORY WILL BE REMOVED!!! 
+
+	#!/bin/bash
+
+	#This is the folder you'd like to create and unannex
+	FOLDERTOUNANNEX='/tmp/badAnnex'
+
+	pushd .
+
+	if [ ! -d \"$FOLDERTOUNANNEX\" ] ; then
+		mkdir \"$FOLDERTOUNANNEX\"
+	fi
+
+	cd \"$FOLDERTOUNANNEX\"
+
+	rm -rf *
+
+	mkdir subdir
+	echo \"hi\" > 1one.txt
+	echo \"hi\" > 2two.txt
+	echo \"hi\" > \"3thr re ee.boo\"
+	echo \"hi\" > \"4f o u r.boo\"
+	echo \"hi\" > 5
+	echo \"hi\" > \"6\"
+	echo \"hi\" > \"subdir/7\"
+	echo \"hi\" > \"subdir/8.cat\"
+	echo \"hi\" > \"subdir/9.cat\"
+
+	echo \"* annex.backend=SHA512E\" > .gitattributes
+
+	chmod g-r 5 6
+	chmod o-r 6
+
+	ls -la
+
+	git init
+	git annex init \"stupid\"
+	git annex add *
+	ls -la
+	git annex unannex *
+	ls -la
+
+	popd
+
+
+Then, a script to recover the files left missing by the above script.  Note this might be very slow as it has to generate SHA512 hashes for all the files in your annex.  Again, change the paths at the top of this file to work in your environment:
+
+	#!/bin/bash
+
+	#Set this to some place outside your annex, where we can store our hashes while we search for them
+	#It will be fastest if this is on a different physical disk than the annexed folder
+	#You can manually delete the file afterwards
+	HASHFILE='/backup3/tmp.sha'
+	#This is the folder you'd like to unannex
+	FOLDERTOUNANNEX='/tmp/badAnnex'
+
+
+
+
+	HASHLEN=128
+
+	pushd .
+	cd \"$FOLDERTOUNANNEX\"
+
+	find \"$FOLDERTOUNANNEX\" ! -path '*.git*' -exec sha512sum \{\} \; > \"$HASHFILE\"
+
+	find -L \"$FOLDERTOUNANNEX\" -type l | while read BROKENFILE; do
+		POINTSTO=`file \"$BROKENFILE\" | sed -r 's/^.*broken symbolic link to .(.*).$/\1/g'`
+		
+		HASH=`echo \"$POINTSTO\" | sed -r \"s/^.*--([^-\/.]{$HASHLEN}).*$/\1/g\"`
+
+		EXT=`echo \"$POINTSTO\" | sed -r \"s/^.*--[^-\/.]{$HASHLEN}(.[^.]+)?$/\1/g\"`
+		
+		echo \"-\"
+		echo \"FILE:$BROKENFILE\"
+		echo \"POINTSTO:$POINTSTO\"
+		echo \"HASH:$HASH\"
+		echo \"EXT:$EXT\"
+		
+		SOURCEFILE=`grep $HASH $HASHFILE | grep -m 1 \"$EXT\" | sed -r \"s/^.{$HASHLEN}  (.*)$/\1/g\"`
+		
+		echo \"SOURCEFILE:$SOURCEFILE\"
+		if [ -f \"$SOURCEFILE\" ];
+		then
+			cp --backup --suffix=\"~GIT_ANNEX_IS_DANGEROUS~\" -a \"$SOURCEFILE\" \"$BROKENFILE\"
+		else
+			echo \"ERROR: Cant find sourcefile\"
+		fi
+	done;
+
+	popd
+
+I have not yet run this repair script on my rather large broken annex.  I cannot seem to figure out how to restore file ownership and permissions which seem to have been lost when the second file is just linked to the matching previously annexed file (note: this is visible after \"fixing\" the bad annex created by the first script above in that after \"fixing\" file \"6\" is readable by other, whereas originally he was NOT readable by other.  The permissions of 6 have been copied from 5.)  Any thoughts or improvements on this are appreciated. 
+"""]]
diff --git a/doc/bugs/S3_buckets_with_capital_letters_breaks_authentication.mdwn b/doc/bugs/S3_buckets_with_capital_letters_breaks_authentication.mdwn
new file mode 100644
index 000000000..886f0491e
--- /dev/null
+++ b/doc/bugs/S3_buckets_with_capital_letters_breaks_authentication.mdwn
@@ -0,0 +1,30 @@
+### Please describe the problem.
+
+As described in [[tips/Internet_Archive_via_S3]], there is a problem using S3 with buckets that have capital letters. The bug lies either in the hS3 library or in archive.org itself.
+
+### What steps will reproduce the problem?
+
+Try to add an [[special_remotes/S3]] remote with capital letters in the bucket name.
+
+### What version of git-annex are you using? On what operating system?
+
+[[!format txt """
+git-annex version: 4.20130921-g434dc22
+build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi
+local repository version: 3
+default repository version: 3
+supported repository versions: 3 4
+upgrade supported from repository versions: 0 1 2
+"""]]
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+anarcat@angela:video$ git annex initremote archive-moglenrepublica type=S3 host=s3.us.archive.org bucket=Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia
+initremote archive-moglenrepublica (Internet Archive mode) git-annex: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
+# End of transcript or log.
+"""]]
+
+Just thought it would be better to have a separate thread for this bug. :)
diff --git a/doc/bugs/fatal:_unable_to_access___39__..__47__..__47__..__47__..__47__C:__92__Users__92____91__...__93____92__annex__92__.git__47__config__39__:_Invalid_argument___40__Windows__41__.mdwn b/doc/bugs/fatal:_unable_to_access___39__..__47__..__47__..__47__..__47__C:__92__Users__92____91__...__93____92__annex__92__.git__47__config__39__:_Invalid_argument___40__Windows__41__.mdwn
new file mode 100644
index 000000000..05d984bd2
--- /dev/null
+++ b/doc/bugs/fatal:_unable_to_access___39__..__47__..__47__..__47__..__47__C:__92__Users__92____91__...__93____92__annex__92__.git__47__config__39__:_Invalid_argument___40__Windows__41__.mdwn
@@ -0,0 +1,59 @@
+### What steps will reproduce the problem?
+[[!format sh """
+C:\Users\Bruno>mkdir annex
+
+C:\Users\Bruno>cd annex
+
+C:\Users\Bruno\annex>git init
+Initialized empty Git repository in C:/Users/Bruno/annex/.git/
+
+C:\Users\Bruno\annex>git annex init
+init
+  Detected a crippled filesystem.
+
+  Enabling direct mode.
+
+  Detected a filesystem without fifo support.
+
+  Disabling ssh connection caching.
+ok
+(Recording state in git...)
+
+C:\Users\Bruno\annex>echo test > test
+
+C:\Users\Bruno\annex>git annex add .
+add test (checksum...) ok
+(Recording state in git...)
+
+C:\Users\Bruno\annex>git commit -a -m added
+[master (root-commit) 2eea610] added
+ 1 file changed, 1 insertion(+)
+ create mode 120000 test
+
+C:\Users\Bruno\annex>git annex sync
+(Recording state in git...)
+fatal: unable to access '../../../../C:\Users\Bruno\annex\.git/config': Invalid argument
+
+git-annex: user error (xargs ["-0","git","--git-dir=C:\\Users\\Bruno\\annex\\.git","--work-tree=C:\\Users\\Bruno\\annex","add","-f"] exited 123)
+failed
+git-annex: sync: 1 failed
+"""]]
+
+### What version of git-annex are you using? On what operating system?
+Windows 8 (64 bits)
+
+git version 1.8.4.msysgit.0
+
+[[!format sh """
+git-annex version: 4.20131008-ge115441
+build flags: Pairing Testsuite S3 WebDAV DNS Feeds Quvi TDFA CryptoHash
+key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
+remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
+local repository version: 4
+default repository version: 3
+supported repository versions: 3 4
+upgrade supported from repository versions: 2
+"""]]
+
+### Please provide any additional information below.
+C:\Users\Bruno\annex\.git\config exists
diff --git a/doc/bugs/git-annex_happily_deleted_most_of_my_files___36____35____38____33__/comment_2_d9b65fe4cb4bfd58f37e7da5350c6401._comment b/doc/bugs/git-annex_happily_deleted_most_of_my_files___36____35____38____33__/comment_2_d9b65fe4cb4bfd58f37e7da5350c6401._comment
new file mode 100644
index 000000000..8574d974b
--- /dev/null
+++ b/doc/bugs/git-annex_happily_deleted_most_of_my_files___36____35____38____33__/comment_2_d9b65fe4cb4bfd58f37e7da5350c6401._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="http://cstork.org/"
+ nickname="Chris Stork"
+ subject="git annex get/sync don't delete files"
+ date="2013-10-10T11:43:29Z"
+ content="""
+AFAIU it's impossible for the 'get' subcommand to delete anything because it only copies files to .git/annex/objects.
+
+'sync' should also never delete files because it only adds files to the annex (i.e. checksums them and sets up the links and directories in .git/annex), commits things to git and pulls/pushes the git-annex specific branches to/from other repos.
+
+So I think some other 'nono' must have messed up your files.
+
+If you ever used (maybe unintentionally) used indirect mode there seem to be a rather high chance that some of your otherwise lost content is still in .git/annex/objects.  Did you check if there's are any files in .git/annex/objects ? 
+"""]]
diff --git a/doc/devblog/day_30__cronner/comment_1_53dfd9310e92f5225db52a13e20a46d4._comment b/doc/devblog/day_30__cronner/comment_1_53dfd9310e92f5225db52a13e20a46d4._comment
new file mode 100644
index 000000000..dc0c7dfe1
--- /dev/null
+++ b/doc/devblog/day_30__cronner/comment_1_53dfd9310e92f5225db52a13e20a46d4._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="http://nullroute.eu.org/~grawity/"
+ nickname="Mantas"
+ subject="comment 1"
+ date="2013-10-09T06:04:15Z"
+ content="""
+The SysV Unix *cron* learned this same thing [in 1979][1].
+
+[1]: http://en.wikipedia.org/wiki/Cron#Multi-user_capability
+"""]]
diff --git a/doc/forum/bash_completion/comment_1_5c42c0c8e7fc3224bf5406880f9fd0c4._comment b/doc/forum/bash_completion/comment_1_5c42c0c8e7fc3224bf5406880f9fd0c4._comment
new file mode 100644
index 000000000..7f2f0afd0
--- /dev/null
+++ b/doc/forum/bash_completion/comment_1_5c42c0c8e7fc3224bf5406880f9fd0c4._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="ringprince"
+ ip="134.76.140.110"
+ subject="comment 1"
+ date="2013-10-10T07:08:21Z"
+ content="""
+I hope it is.  I'd love to use it.
+"""]]
diff --git a/doc/forum/bash_completion/comment_2_6cbe3c825db99bf9188a0de8bb937d5b._comment b/doc/forum/bash_completion/comment_2_6cbe3c825db99bf9188a0de8bb937d5b._comment
new file mode 100644
index 000000000..72d5e3fc4
--- /dev/null
+++ b/doc/forum/bash_completion/comment_2_6cbe3c825db99bf9188a0de8bb937d5b._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://id.koumbit.net/anarcat"
+ ip="72.0.72.144"
+ subject="comment 2"
+ date="2013-10-10T09:22:00Z"
+ content="""
+it is, please publish.
+"""]]
diff --git a/doc/forum/git_annex_sync_dies___40__sometimes__41__.mdwn b/doc/forum/git_annex_sync_dies___40__sometimes__41__.mdwn
new file mode 100644
index 000000000..84f9d1125
--- /dev/null
+++ b/doc/forum/git_annex_sync_dies___40__sometimes__41__.mdwn
@@ -0,0 +1,22 @@
+I've set up git annex on my laptop and on a remote server using the gitolite v3 git-annex branch.
+When I 'git annex sync' from my laptop (mac OSX) to the server, and 'git annex copy . --to server' all works fine.
+
+Later, I tried to clone the repository on another machine (linux 2.6.32). again all is well. I 'git annex init' in the new clone and then try to 'git annex sync' -- now I run into problems.
+
+Specifically, the first request for the configlist (when I do 'git annex --debug sync' all is fine until the configlist request) somehow breaks and my ssh session shows:
+
+muxserver_listen bind(): Input/output error
+
+This seems to confuse the client, who now believes that git annex is not installed on the server (it is).
+
+If I issue the same command as git annex via ssh, all seems to work ok with the caveats below:
+
+I have a feeling it's related to the -o 'ControlPersist=yes' argument that git-annex appears to be giving ssh.  If I include this option when I run via ssh, ssh dies:
+
+>ssh git@server -o "ControlMaster=auto" -o "ControlPersist=yes" "git-annex-shell 'configlist' '/~/PRJ'"
+command-line: line 0: Bad configuration option: ControlPersist
+
+
+(note that it appears to be ok to leave "ControlMaster=auto" in).
+
+any thoughts?
diff --git a/doc/install/cabal/comment_20_0f553be2a4c666e3bed58b2bce549406._comment b/doc/install/cabal/comment_20_0f553be2a4c666e3bed58b2bce549406._comment
new file mode 100644
index 000000000..288c0e017
--- /dev/null
+++ b/doc/install/cabal/comment_20_0f553be2a4c666e3bed58b2bce549406._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawlkA6XinbeOdnEDxEGQUWyjqPGh0kdMXr4"
+ nickname="Blake"
+ subject="Updating git-annex via cabal"
+ date="2013-10-09T21:21:22Z"
+ content="""
+What is the best way to update git-annex and the dependencies via cabal? Just the `$ cabal install *` commands again?
+"""]]
diff --git a/doc/tips/Internet_Archive_via_S3.mdwn b/doc/tips/Internet_Archive_via_S3.mdwn
index d7893693f..39da2c60f 100644
--- a/doc/tips/Internet_Archive_via_S3.mdwn
+++ b/doc/tips/Internet_Archive_via_S3.mdwn
@@ -31,7 +31,7 @@ specify `x-archive-meta*` headers to add metadata as explained in their
 [documentation](http://www.archive.org/help/abouts3.txt).
 
 [[!template id=note text="""
-/!\ There seems to be a bug in either hS3 or the archive that breaks
+/!\ There seems to be a [[bug|bugs/S3 buckets with capital letters breaks authentication]] in either hS3 or the archive that breaks
 authentication when the bucket name contains spaces or upper-case letters..
 use all lowercase and no spaces when making the bucket with `initremote`.
 """]]
diff --git a/doc/tips/Internet_Archive_via_S3/comment_1_d53a3848c20dce61867283fc03c2adaa._comment b/doc/tips/Internet_Archive_via_S3/comment_1_d53a3848c20dce61867283fc03c2adaa._comment
new file mode 100644
index 000000000..7c2ce48fc
--- /dev/null
+++ b/doc/tips/Internet_Archive_via_S3/comment_1_d53a3848c20dce61867283fc03c2adaa._comment
@@ -0,0 +1,34 @@
+[[!comment format=mdwn
+ username="https://id.koumbit.net/anarcat"
+ ip="72.0.72.144"
+ subject="how to use with simply addurl?"
+ date="2013-10-09T22:27:27Z"
+ content="""
+It doesn't seem like git annex addurl by itself supports the archive.org urls...
+
+[[!format txt \"\"\"
+anarcat@marcos:presentations$ git annex addurl --file=re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm http://archive.org/download/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm
+addurl re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm
+  failed to verify url exists: http://archive.org/download/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm
+failed
+git-annex: addurl: 1 failed
+\"\"\"]]
+
+I also tried the \"details\" url (<http://archive.org/details/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia>) - but that just downloads the webpage, not the video either...
+
+Even the ultimate video URL doesn't work:
+
+[[!format txt \"\"\"
+anarcat@marcos:presentations$ git annex addurl --debug --file=re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm http://ia601009.us.archive.org/9/items/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm
+[2013-10-09 18:26:30 EDT] call: quvi [\"-v\",\"mute\",\"--support\",\"http://ia601009.us.archive.org/9/items/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm\"]
+addurl re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm [2013-10-09 18:26:30 EDT] read: curl [\"-s\",\"--head\",\"-L\",\"http://ia601009.us.archive.org/9/items/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm\",\"-w\",\"%{http_code}\"]
+
+  failed to verify url exists: http://ia601009.us.archive.org/9/items/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm
+failed
+git-annex: addurl: 1 failed
+\"\"\"]]
+
+... even though that URL actually gives out a proper 200 OK response code.
+
+Any ideas? --[[anarcat]]
+"""]]