Merge branch 'master' of ssh://git-annex.branchable.com

3 files changed, 224 insertions, 0 deletions

diff --git a/doc/bugs/issue_with_syncing_between_v5_and_v6_repos.mdwn b/doc/bugs/issue_with_syncing_between_v5_and_v6_repos.mdwn
new file mode 100644
index 000000000..c4a302dab
--- /dev/null
+++ b/doc/bugs/issue_with_syncing_between_v5_and_v6_repos.mdwn
@@ -0,0 +1,110 @@
+### Please describe the problem.
+I was accidentally syncing between repos at v5 and v6, I didn't realize they were different versions until I started digging into this issue.
+
+I have two repos, one at v5 and one at v6. I add a file to v6 annex, unlock, then sync to my v5 repo. From my v5 repo, I then move the file into a sub-directory, in my v5 repo, and accidentally commit with git add; git commit, instead of git annex add; git commit, since with my v6 repo I am used to using git add for everything. The file content is now missing from annex on my v5 repo. I then sync from both repos. Now the file in v5 repo is broken and the content is not present in annex. The content is fine on the v6 repo. The only way to actually get annex to see the content, from my v5 repo, though is to do git annex upgrade, git annex sync, fsck repair don't help.
+
+### What steps will reproduce the problem?
+
+### What version of git-annex are you using? On what operating system?
+
+    $ git annex version
+    git-annex version: 6.20171128-g58b04cd2e
+    build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi
+    dependency versions: aws-0.17.1 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.0 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
+    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1         MD5E MD5 WORM URL
+    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
+    local repository version: 6
+    supported repository versions: 3 5 6
+    upgrade supported from repository versions: 0 1 2 3 4 5
+    operating system: darwin x86_64
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+
+#
+# Create a V6 repo and add an image to it
+#
+$ mkdir /tmp/repov6
+$ cd /tmp/repov6
+$ git init
+$ git annex init
+$ /tmp/repov6$ git annex upgrade
+upgrade . (v5 to v6...) ok
+
+$ git add image1.png 
+$ git commit -m "added image 1 to annex"
+$ /tmp/repov6$ git annex info image1.png 
+file: image1.png
+size: 17.69 kilobytes
+key: SHA256E-s17691--c98e6eae515fec6a094e2b4bdcd221ba089a5e6073f6c365f6d4c829243c8aa5.png
+present: true
+$ git annex unlock image1.png
+$ git commit -m "unlocked the image"
+
+#
+# Create a V5 repo
+# and sync with first repo
+#
+$ cd /tmp
+$ git clone /tmp/repov6/ repov5
+Cloning into 'repov5'...
+done.
+$ cd repov5
+$ git annex init "repo v5"
+$ git remote add repov6 /tmp/repov6
+$ git annex sync
+andrew@bumblebee /tmp/repov5$ ls -l image1.png 
+… image1.png -> .git/annex/objects/zJ/7J/SHA256E-s17691--c98e6eae515fec6a094e2b4bdcd221ba089a5e6073f6c365f6d4c829243c8aa5.png/SHA256E-s17691--c98e6eae515fec6a094e2b4bdcd221ba089a5e6073f6c365f6d4c829243c8aa5.png
+
+# this repo is still at version 5
+$ git annex version
+…
+local repository version: 5
+…
+
+# now lets move the unlocked not present image file
+$ mkdir imagesFolder
+$ mv image1.png imagesFolder/
+$ git add -A .
+$ git commit -m "moved file"
+
+## oops I accidentally committed my annexed file using git add from a v5 repo…
+## since I am used to using git add commands in my v6 repos
+
+#
+# Now lets sync
+#
+$ cd /tmp/repov6
+$ git remote add repov5 /tmp/repov5
+$ git annex sync
+$ ls -laL imagesFolder/image1.png 
+…  17691 Dec 10 21:26 imagesFolder/image1.png
+
+# 
+# Now lets try to get this to our V5 repo
+#
+$ cd /tmp/repov5
+$ git annex sync
+$ git annex info imagesFolder2/image1.png
+fatal: Not a valid object name imagesFolder2/image1.png
+info imagesFolder2/image1.png (not a directory or an annexed file or a treeish or a remote or a uuid) failed
+a
+
+# the following does not help
+$ git annex fsck
+$ git annex repair
+
+# the following resolves the issue
+$ git annex upgrade	
+
+
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+Yes
diff --git a/doc/bugs/regression_-_yt__58___prefix_for___34__regular__34___urls/comment_5_811ba6ec07f6f30c701201178cdb00ac._comment b/doc/bugs/regression_-_yt__58___prefix_for___34__regular__34___urls/comment_5_811ba6ec07f6f30c701201178cdb00ac._comment
new file mode 100644
index 000000000..aaf8143bd
--- /dev/null
+++ b/doc/bugs/regression_-_yt__58___prefix_for___34__regular__34___urls/comment_5_811ba6ec07f6f30c701201178cdb00ac._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 5"
+ date="2017-12-11T05:27:15Z"
+ content="""
+ok, tested with 6.20171208+gitg01f78e877-1~ndall+1 -- --fast mode issue is indeed resolved.  Thanks!
+Failing tests now only relate to our special remote (datalad-archives) which interfaces urls, so I thought it might relate... I will try to distill more info tomorrow unless you beat me to it figuring out where that regression could be (upon quick look didn't spot any yt: anywhere, so probably another issue) 
+"""]]
diff --git a/doc/bugs/regression_-_yt__58___prefix_for___34__regular__34___urls/comment_6_2cd156baab77dc3b3c248a2368b59040._comment b/doc/bugs/regression_-_yt__58___prefix_for___34__regular__34___urls/comment_6_2cd156baab77dc3b3c248a2368b59040._comment
new file mode 100644
index 000000000..4dddd9af6
--- /dev/null
+++ b/doc/bugs/regression_-_yt__58___prefix_for___34__regular__34___urls/comment_6_2cd156baab77dc3b3c248a2368b59040._comment
@@ -0,0 +1,105 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 6"
+ date="2017-12-11T14:40:26Z"
+ content="""
+ok, first initial bit of information:  there is a difference in interactions with the special remote now. Here is the diff between old and new runs (sorry - content was also changing so there is difference in keys as well):
+
+[[!format sh \"\"\"
+$> diff -Nar -u6 /tmp/datalad_temp_tree_check_basic_scenarioVEOIhb/.git/bin/git-annex-remote-datalad-archive /tmp/datalad_temp_tree_check_basic_scenarioIfsLo7/.git/bin/git-annex-remote-datalad-archive
+--- /tmp/datalad_temp_tree_check_basic_scenarioVEOIhb/.git/bin/git-annex-remote-datalad-archive	2017-12-11 08:56:38.381298365 -0500
++++ /tmp/datalad_temp_tree_check_basic_scenarioIfsLo7/.git/bin/git-annex-remote-datalad-archive	2017-12-11 08:56:14.885677071 -0500
+@@ -48,66 +48,72 @@
+ ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', '--debug', '--json', 'simple.txt']
+ 
+ ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'commit', '-m', 'Added the load file']
+ 
+ ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'lookupkey', '--debug', 'a.tar.gz']
+ 
+-### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'addurl', '--debug', '--relaxed', '--file=simple.txt', 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+']
++### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'addurl', '--debug', '--relaxed', '--file=simple.txt', 'dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+']
+ #send VERSION 1
+ #recv PREPARE
+ #send PREPARE-SUCCESS
+ #recv GETCOST
+ #send COST 500
+ #recv GETAVAILABILITY
+ #send AVAILABILITY LOCAL
+-#recv CLAIMURL dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+-#send DEBUG Claiming url 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
++#recv CLAIMURL dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
++#send DEBUG Claiming url 'dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
+ #send CLAIMURL-SUCCESS
+-#recv CHECKURL dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
++#recv CHECKURL dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+ #send CHECKURL-CONTENTS UNKNOWN
+ #recv 
+ 
+ ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'drop', '--debug', '--json', 'simple.txt']
+ #send VERSION 1
+ #recv PREPARE
+ #send PREPARE-SUCCESS
+ #recv CHECKPRESENT SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
+ #send GETURLS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt dl+archive:
+-#recv VALUE dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
++#recv VALUE dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+ #recv VALUE 
+ #send CHECKPRESENT-SUCCESS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
+ #recv 
+ 
+ ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'whereis', '--debug', '--json', 'simple.txt']
+ #send VERSION 1
+ #recv PREPARE
+ #send PREPARE-SUCCESS
+ #recv WHEREIS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
+ #send WHEREIS-FAILURE
+-#recv CLAIMURL dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+-#send DEBUG Claiming url 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
+-#send CLAIMURL-SUCCESS
+ #recv 
+ 
+ ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'find', '--debug', '--json', '--not', '--in', 'here', 'simple.txt']
+ 
+ ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'get', '--debug', '--json', '--json-progress', 'simple.txt']
+ #send VERSION 1
+ #recv PREPARE
+ #send PREPARE-SUCCESS
+ #recv TRANSFER RETRIEVE SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt .git/annex/tmp/SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
+ #send GETURLS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt dl+archive:
+-#recv VALUE dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
++#recv VALUE dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+ #recv VALUE 
+ #send TRANSFER-SUCCESS RETRIEVE SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
+ #recv 
+ 
+-### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'rmurl', '--debug', 'simple.txt', 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+']
++### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'rmurl', '--debug', 'simple.txt', 'dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+']
++#send VERSION 1
++#recv PREPARE
++#send PREPARE-SUCCESS
++#recv CLAIMURL dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
++#send DEBUG Claiming url 'dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
++#send CLAIMURL-SUCCESS
++#recv 
++
++### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'drop', '--debug', '--json', 'simple.txt']
+ send VERSION 1
+ recv PREPARE
+ send PREPARE-SUCCESS
+-recv CLAIMURL dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+-send DEBUG Claiming url 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
+-send CLAIMURL-SUCCESS
++recv CHECKPRESENT SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
++send GETURLS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt dl+archive:
++recv VALUE dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
++recv VALUE 
++send CHECKPRESENT-SUCCESS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
+ recv 
+-
+-### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'drop', '--debug', '--json', 'simple.txt']
+
+
+\"\"\"]]
+[our test](https://github.com/datalad/datalad/blob/master/datalad/customremotes/tests/test_archives.py#L86) verifies that annex refuses  to drop the content if we remove the dl+archive link for  the key, and now it fails
+"""]]