Merge branch 'master' of ssh://git-annex.branchable.com

11 files changed, 277 insertions, 1 deletions

diff --git a/doc/bugs/Endless_SSH_password_prompts/comment_5_7490ca530d4e7d49eaa264eb5880dd17._comment b/doc/bugs/Endless_SSH_password_prompts/comment_5_7490ca530d4e7d49eaa264eb5880dd17._comment
new file mode 100644
index 000000000..7990d0ff2
--- /dev/null
+++ b/doc/bugs/Endless_SSH_password_prompts/comment_5_7490ca530d4e7d49eaa264eb5880dd17._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawkdiXn5s1wX2Zncy1S7_h-yPTv2gt0bmmY"
+ nickname="Jacob"
+ subject="Same problem on ubuntu"
+ date="2013-12-15T14:22:55Z"
+ content="""
+Same problem, I even manually setup ssh keys between the two servers but it just loops and keeps asking for a password every sync.
+"""]]
diff --git a/doc/bugs/OSX_build_broken.mdwn b/doc/bugs/OSX_build_broken.mdwn
new file mode 100644
index 000000000..ac729b429
--- /dev/null
+++ b/doc/bugs/OSX_build_broken.mdwn
@@ -0,0 +1,11 @@
+### Please describe the problem.
+
+The OSX webapp doesn't start on the current OSX 10.7.5 builds. 
+
+### What steps will reproduce the problem?
+
+Download git-annex from http://downloads.kitenet.net/git-annex/OSX/current/10.7.5_Lion/git-annex.dmg and install it by copying it to Application. Try to launch it then. The result is a message in the system log saying "git-annex unknown command webapp" and then the usage information.
+
+### What version of git-annex are you using? On what operating system?
+
+The current version as downloaded from http://downloads.kitenet.net/git-annex/OSX/current/10.7.5_Lion/git-annex.dmg on OSX 10.7.5
diff --git a/doc/bugs/Syncing_of_file_contents_seems_to_be_broken_in_recent_versions_of_the_assistant.mdwn b/doc/bugs/Syncing_of_file_contents_seems_to_be_broken_in_recent_versions_of_the_assistant.mdwn
new file mode 100644
index 000000000..c14ad54f8
--- /dev/null
+++ b/doc/bugs/Syncing_of_file_contents_seems_to_be_broken_in_recent_versions_of_the_assistant.mdwn
@@ -0,0 +1,132 @@
+### Please describe the problem.
+
+Since I last tested git-annex (version 4.20130516.1) my setup seems to no longer work in current versions (5.20131130)
+
+### What steps will reproduce the problem?
+
+[[!format sh """
+$ mkdir 1 2
+$ cd 1; git init; git-annex init; git annex direct; echo "file added to 1" > file_from_1; cd ..
+Initialized empty Git repository in /home/pedrocr/testannexbug/1/.git/
+init  ok
+(Recording state in git...)
+commit  
+# On branch master
+#
+# Initial commit
+#
+nothing to commit (create/copy files and use "git add" to track)
+ok
+direct  ok
+$ cd 2; git init; git-annex init; git annex direct; echo "file added to 2" > file_from_2; cd ..
+Initialized empty Git repository in /home/pedrocr/testannexbug/2/.git/
+init  ok
+(Recording state in git...)
+commit  
+# On branch master
+#
+# Initial commit
+#
+nothing to commit (create/copy files and use "git add" to track)
+ok
+direct  ok
+$ cd 1; git remote add 2 ssh://localhost/~pedrocr/Hacks/test-git-annex/2; git annex assistant; cd ..
+$ cd 2; git remote add 1 ssh://localhost/~pedrocr/Hacks/test-git-annex/1; git annex assistant; cd ..
+$ ls -la 1
+total 20
+drwxrwxr-x 3 pedrocr pedrocr 4096 Dec 15 11:33 .
+drwxrwxr-x 4 pedrocr pedrocr 4096 Dec 15 11:33 ..
+-rw-rw-r-- 1 pedrocr pedrocr   16 Dec 15 11:33 file_from_1
+-rw-r--r-- 1 pedrocr pedrocr   16 Dec 15 11:33 file_from_2
+drwxrwxr-x 9 pedrocr pedrocr 4096 Dec 15 11:33 .git
+$ ls -la 2
+total 20
+drwxrwxr-x 3 pedrocr pedrocr 4096 Dec 15 11:33 .
+drwxrwxr-x 4 pedrocr pedrocr 4096 Dec 15 11:33 ..
+-rw-r--r-- 1 pedrocr pedrocr   16 Dec 15 11:33 file_from_1
+-rw-rw-r-- 1 pedrocr pedrocr   16 Dec 15 11:33 file_from_2
+drwxrwxr-x 9 pedrocr pedrocr 4096 Dec 15 11:33 .git
+"""]]
+
+so far so good
+
+[[!format sh """
+$ date | tee 1/file_from_1
+Sun Dec 15 11:53:18 WET 2013
+$ date | tee 2/file_from_2
+Sun Dec 15 11:53:22 WET 2013
+$ cat 1/*
+Sun Dec 15 11:53:18 WET 2013
+file added to 2
+$ cat 2/*
+file added to 1
+Sun Dec 15 11:53:22 WET 2013
+$
+"""]]
+
+But then syncing doesn't work. There doesn't seem to be anything special in the logs
+
+[[!format sh """
+$ tail -n 20 1/.git/annex/daemon.log 
+(Recording state in git...)
+add file_from_1 (checksum...) [2013-12-15 11:53:18 WET] Committer: Committing changes to git
+[2013-12-15 11:53:18 WET] Pusher: Syncing with 2 
+
+file_from_1
+          29 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/1)
+
+sent 104 bytes  received 31 bytes  270.00 bytes/sec
+total size is 29  speedup is 0.21
+[2013-12-15 11:53:18 WET] Transferrer: Uploaded file_from_1
+Already up-to-date.
+To ssh://localhost/~pedrocr/Hacks/test-git-annex/2
+   f025062..431ac37  git-annex -> synced/git-annex
+   d393bb3..fef60ca  annex/direct/master -> synced/master
+Already up-to-date.
+[2013-12-15 11:53:21 WET] Pusher: Syncing with 2 
+To ssh://localhost/~pedrocr/Hacks/test-git-annex/2
+   431ac37..3cc7045  git-annex -> synced/git-annex
+
+
+$ tail -n 20 2/.git/annex/daemon.log 
+(Recording state in git...)
+add file_from_2 (checksum...) [2013-12-15 11:53:22 WET] Committer: Committing changes to git
+[2013-12-15 11:53:22 WET] Pusher: Syncing with 1 
+
+file_from_2
+          29 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/1)
+Already up-to-date.
+
+sent 104 bytes  received 31 bytes  270.00 bytes/sec
+total size is 29  speedup is 0.21
+[2013-12-15 11:53:22 WET] Transferrer: Uploaded file_from_2
+To ssh://localhost/~pedrocr/Hacks/test-git-annex/1
+   ebbcd4c..dae7a77  git-annex -> synced/git-annex
+   ffd5a90..67ad7c1  annex/direct/master -> synced/master
+Already up-to-date.
+[2013-12-15 11:53:25 WET] Pusher: Syncing with 1 
+To ssh://localhost/~pedrocr/Hacks/test-git-annex/1
+   dae7a77..976c899  git-annex -> synced/git-annex
+"""]]
+
+
+
+### What version of git-annex are you using? On what operating system?
+
+I'm running git-annex from the Ubuntu PPA on Ubuntu 12.04 LTS.
+
+[[!format sh """
+$ lsb_release -v
+No LSB modules are available.
+$ lsb_release -a
+No LSB modules are available.
+Distributor ID:	Ubuntu
+Description:	Ubuntu 12.04.3 LTS
+Release:	12.04
+Codename:	precise
+$ git annex version
+git-annex version: 5.20131130
+build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus XMPP Feeds Quvi TDFA
+key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
+remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
+"""]]
diff --git a/doc/bugs/direct_mode_sync_should_avoid_git_commit/comment_1_d286800081f019651cd40355b7d5518a._comment b/doc/bugs/direct_mode_sync_should_avoid_git_commit/comment_1_d286800081f019651cd40355b7d5518a._comment
new file mode 100644
index 000000000..98897593c
--- /dev/null
+++ b/doc/bugs/direct_mode_sync_should_avoid_git_commit/comment_1_d286800081f019651cd40355b7d5518a._comment
@@ -0,0 +1,27 @@
+[[!comment format=mdwn
+ username="TroisSinges"
+ ip="82.227.207.5"
+ subject="Almost done"
+ date="2013-12-15T10:13:50Z"
+ content="""
+Hi!
+
+Thank you for the fix, it's much better now. However, in some case, it doesn't work. For example, when I try to add inject a file in the direct repository (rm symlinkedFile ; mv /tmp/myFile symlinkedFile ; git annex add symlinkedFile), it tries to map each file in memory (which takes a very long time through the network).
+
+    # git annex add bigFile --debug
+    [2013-12-15 11:00:00 CET] read: git [\"--git-dir=/Volumes/Video/Annex/.git\",\"--work-tree=/Volumes/Video/Annex\",\"-c\",\"core.bare=false\",\"ls-files\",\"--others\",\"--exclude-standard\",\"-z\",\"--\",\"bigFile\"]
+    [2013-12-15 11:00:00 CET] read: git [\"--git-dir=/Volumes/Video/Annex/.git\",\"--work-tree=/Volumes/Video/Annex\",\"-c\",\"core.bare=false\",\"ls-files\",\"--modified\",\"-z\",\"--\",\"bigFile\"]
+
+If I use lsof, each another file in repository is mapped in memory:
+
+    # lsof -p 30130
+    git     30130 troissinges    3r   REG             46,11 4519526640  3093837953332665982 /Volumes/Video/Annex/AnotherBigFile
+
+My version of software:
+
+    git annex version
+    git-annex version: 5.20131213
+    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash
+    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
+    remote types: git gcrypt S3 bup directory rsync web webdav glacier hook
+"""]]
diff --git a/doc/bugs/feature_request:_addhash.mdwn b/doc/bugs/feature_request:_addhash.mdwn
new file mode 100644
index 000000000..fca74200e
--- /dev/null
+++ b/doc/bugs/feature_request:_addhash.mdwn
@@ -0,0 +1,27 @@
+### Use case 1
+
+I have a big repo using a SHA256E back-end. Along comes a new shiny SKEIN512E back-end and I would like to transition to using that, because it's faster and moves from ridiculously to ludicrously low risk of collisions.
+
+I can set `.gitattributes` to use the new back-end for any new files added, but then I when I import some arbitrary mix of existing and new files to the repo it will not deduplicate any more, it will add all the files under the new hash scheme.
+
+### Use case 2
+
+I have a big repo of files I have added using `addurl --fast`. I download the files, and they are in the repo.
+
+ - I cannot verify later that none of them have been damaged.
+ - If I come across an offline collection of some of the files, I cannot easily get them into the annex by a simple import.
+
+### Workaround
+
+In both these cases, what I can do is unlock (maybe?) or unannex (definitely) all of the files, and then re-add them using the new hash. In both cases this means I now risk having duplicates in various clones of the repo, and would have to clean them up with `drop-unused` -- after first having re-copied them from a repo that has them under the new hash. I also lose the continuity of the history of that object.
+
+In use case 2 I also lose the URLs of the files and would have to re-add them using `addurl`.
+
+... which brings me to the proposed feature.
+
+### addhash
+
+Symmetrical to `addurl`, which in one form can take an existing hashed (or URL-sourced) file and add an URL source to it, `addhash` can take an existing URL-sourced (or hashed) file and add a hash reference to it (given that the file is in the annex so that the hash may be calculated) -- an alias under which it may also be identified, in addition to the existing URL or hash.
+
+ - Any file added to the annex after `addhash` will use the symlink name of the original hash if their hash matches the `addhash`ed one.
+ - An `fsck` run will use one of the available hashes to verify the integrity of the file, maybe according to some internal order of preference, or possibly a configurable one.
diff --git a/doc/bugs/git_annex_get___60__file__62___should_verify_file_hash.mdwn b/doc/bugs/git_annex_get___60__file__62___should_verify_file_hash.mdwn
new file mode 100644
index 000000000..f680729ac
--- /dev/null
+++ b/doc/bugs/git_annex_get___60__file__62___should_verify_file_hash.mdwn
@@ -0,0 +1,32 @@
+### Please describe the problem.
+git annex get fileName-  should perform a hash check on the file content before adding to the local repository
+
+
+### What steps will reproduce the problem?
+Two scenarios:
+1) Malicious user and owner of repository being pulled from can edit his/her local .git/annex/objects directory 
+to alter the file content. For src code, this could be to insert a bug, insert a backdoor, or for example 
+to replace an image file artifact for a website, with a pornographic image.
+The user pulling the file content with "git annex get fileName"  might not be aware of the file contents 
+until they actually examine the file or perform an fsck or commit locally. 
+In the meantime a kiddy porn image could be sitting in their repository or a src code backdoor can get incorporated and deployed etc.
+2) a file could also simply get corrupted during download. An inherent hash check during the 'annex get' would
+point out the problem immediately.
+To reproduce:  create repoNum1, and clone it to create repoNum2. manual edit/replace content in repoNum1/.git/annex/objects/...
+then perform a 'git annex get <fileName>' from repoNum2 on the file that has been manipulated 
+
+
+### What version of git-annex are you using? On what operating system?
+3.2012112ubuntu2 on running linux mint 
+
+
+### Please provide any additional information below.
+
+Aside: Thanks Joey - this is fantastic work you are doing. You have really improved git. The ability to checkout 
+an entire tree - but selectively get only the content actually needed is a real killer feature.
+Kudos and again many many thanks
+M.
+
+
+# End of transcript or log.
+"""]]
diff --git a/doc/forum/Can_Not_Sync_to_Git_Repo/comment_8_36abd829ea71a44c7cded1123a7c913d._comment b/doc/forum/Can_Not_Sync_to_Git_Repo/comment_8_36abd829ea71a44c7cded1123a7c913d._comment
new file mode 100644
index 000000000..66f75327e
--- /dev/null
+++ b/doc/forum/Can_Not_Sync_to_Git_Repo/comment_8_36abd829ea71a44c7cded1123a7c913d._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo"
+ nickname="Tobias"
+ subject="comment 8"
+ date="2013-12-14T13:15:13Z"
+ content="""
+The master branch was checkout out (I don't know why). Checking out the mentioned branch fixed the syncing issue: `git checkout annex/direct/master`
+"""]]
diff --git a/doc/forum/S3_Host_Question/comment_3_797edf3ad41561ab8960f3b28d20611e._comment b/doc/forum/S3_Host_Question/comment_3_797edf3ad41561ab8960f3b28d20611e._comment
new file mode 100644
index 000000000..a0dbae5bd
--- /dev/null
+++ b/doc/forum/S3_Host_Question/comment_3_797edf3ad41561ab8960f3b28d20611e._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="https://me.yahoo.com/a/FHnTlSBo1eCGJRwueeKeB6.RCaPbGMPr5jxx8A--#ce0d8"
+ nickname="Hamza"
+ subject="comment 3"
+ date="2013-12-14T09:05:26Z"
+ content="""
+remote info is recorded in git once you clone the git repo it knows about machine a and s3 repo.
+"""]]
diff --git a/doc/forum/local_pairing_with_2_mac/comment_10_508585e72c81d197a9a1e193c25a702a._comment b/doc/forum/local_pairing_with_2_mac/comment_10_508585e72c81d197a9a1e193c25a702a._comment
new file mode 100644
index 000000000..1d69f41fd
--- /dev/null
+++ b/doc/forum/local_pairing_with_2_mac/comment_10_508585e72c81d197a9a1e193c25a702a._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawnfC02DTOZygFDDvvg0maRciYYMhEIFgkw"
+ nickname="Utku"
+ subject="comment 10"
+ date="2013-12-15T09:51:28Z"
+ content="""
+unfortunately, it is not working on the computer where ssh was not working
+I click an got annex and nothing seems to open
+I try to go on the system logs but I see nothing really obvious. The only log that may be related would be : 
+15/12/13 10:46:23,230 Dock[166]: find_shared_window: WID 21425
+15/12/13 10:46:23,230 Dock[166]: CGSGetWindowTags: Invalid window 0x53b1
+
+if you tell me where to look at I can give you something more useful. 
+
+"""]]
diff --git a/doc/forum/local_pairing_with_2_mac/comment_9_f7a1ce9627ebfc854dfde2f6c924db80._comment b/doc/forum/local_pairing_with_2_mac/comment_9_f7a1ce9627ebfc854dfde2f6c924db80._comment
new file mode 100644
index 000000000..deaf9b68c
--- /dev/null
+++ b/doc/forum/local_pairing_with_2_mac/comment_9_f7a1ce9627ebfc854dfde2f6c924db80._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="209.250.56.87"
+ subject="comment 9"
+ date="2013-12-13T18:52:40Z"
+ content="""
+As a workaround, I have removed ssh-keygen and ssh from the OSX DMG. AIUI, OSX ships ssh by default anyway.
+"""]]
diff --git a/doc/install/OSX.mdwn b/doc/install/OSX.mdwn
index 1671476ae..784e42c83 100644
--- a/doc/install/OSX.mdwn
+++ b/doc/install/OSX.mdwn
@@ -34,7 +34,7 @@ the app for OSX Lion.
 
 <pre>
 brew update
-brew install haskell-platform git ossp-uuid md5sha1sum coreutils libgsasl gnutls libidn libgsasl pkg-config libxml2
+brew install haskell-platform git ossp-uuid md5sha1sum coreutils gnutls libidn gsasl pkg-config libxml2
 brew link libxml2 --force
 cabal update
 mkdir $HOME/bin