Merge branch 'master' of ssh://git-annex.branchable.com

7 files changed, 192 insertions, 0 deletions

diff --git a/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__.mdwn b/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__.mdwn
new file mode 100644
index 000000000..59278794b
--- /dev/null
+++ b/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__.mdwn
@@ -0,0 +1,99 @@
+### Please describe the problem.
+
+I am a bit unsure what is going on ;)
+
+### What version of git-annex are you using? On what operating system?
+
+6.20160118+gitgdaf852e-1~ndall+1
+
+### Please provide any additional information below.
+
+[[!format sh """
+
+$ chmod a+w -R /tmp/123; rm -rf /tmp/123; mkdir /tmp/123; cd /tmp/123; git init; git annex init;
+chmod: cannot access ‘/tmp/123’: No such file or directory
+Initialized empty Git repository in /tmp/123/.git/
+init  ok
+(recording state in git...)
+
+$ git annex addurl --file svgtune_0.2.0.orig.tar.gz http://http.debian.net/debian/pool/main/s/svgtune/svgtune_0.2.0.orig.tar.gz
+addurl svgtune_0.2.0.orig.tar.gz (downloading http://http.debian.net/debian/pool/main/s/svgtune/svgtune_0.2.0.orig.tar.gz ...) 
+/tmp/123/.git/annex/tmp/URL-s5121--http&c%%http.debian. 100%[===============================================================================================================================>]   5.00K  --.-KB/s    in 0s      
+ok
+(recording state in git...)
+
+$ ls -l svgtune_0.2.0.orig.tar.gz 
+lrwxrwxrwx 1 yoh yoh 198 Jan 19 14:29 svgtune_0.2.0.orig.tar.gz -> .git/annex/objects/K6/j2/SHA256E-s5121--6d8f7d10206a120a42bec2cd29bc2365d09889fdf070ac8c67d1cff0b1539f63.tar.gz/SHA256E-s5121--6d8f7d10206a120a42bec2cd29bc2365d09889fdf070ac8c67d1cff0b1539f63.tar.gz
+
+$ git annex info
+repository mode: indirect
+trusted repositories: 0
+semitrusted repositories: 3
+	00000000-0000-0000-0000-000000000001 -- web
+ 	00000000-0000-0000-0000-000000000002 -- bittorrent
+ 	1087c63f-e325-41ff-9c45-bbc493aa42f1 -- yoh@hopa:/tmp/123 [here]
+untrusted repositories: 0
+transfers in progress: none
+available local disk space: 1 gigabyte (+1 megabyte reserved)
+local annex keys: 1
+local annex size: 5.12 kilobytes
+annexed files in working tree: 1
+size of annexed files in working tree: 5.12 kilobytes
+bloom filter size: 32 mebibytes (0% full)
+backend usage: 
+	SHA256E: 1
+
+$ git annex initremote datalad-archives externaltype=dl+archive type=external autoenable=true encryption=none 
+initremote datalad-archives ok
+(recording state in git...)
+
+$ echo "dl+archive:SHA256E-s5121--6d8f7d10206a120a42bec2cd29bc2365d09889fdf070ac8c67d1cff0b1539f63.tar.gz/svgtune-0.2.0/README.rst README.rst" | git annex addurl -c annex.largefiles=exclude=*.rst --with-files --json --batch 
+{"command":"addurl","file":"README.rst","note":"from datalad-archives","success":true}                                                                                                                                         
+
+$ ls -l README.rst 
+lrwxrwxrwx 1 yoh yoh 192 Jan 19 14:33 README.rst -> .git/annex/objects/V4/3p/SHA256E-s2126--76cea2921af6b250b9bcde3a99785d1010d657cbc6781f01cd7a7886708c441f.rst/SHA256E-s2126--76cea2921af6b250b9bcde3a99785d1010d657cbc6781f01cd7a7886708c441f.rst
+
+$ echo "http://www.onerussian.com/tmp/README2.rst README2.rst" | git annex addurl -c annex.largefiles=exclude=*.rst --with-files --json --batch        
+{"command":"addurl","file":"README2.rst","note":"downloading http://www.onerussian.com/tmp/README2.rst ...","note":"non-large file; adding content to git repository","success":true}
+
+$ ls -l README2.rst
+-rw------- 1 yoh yoh 13 Jan 19 14:34 README2.rst
+
+$ echo "dl+archive:SHA256E-s5121--6d8f7d10206a120a42bec2cd29bc2365d09889fdf070ac8c67d1cff0b1539f63.tar.gz/svgtune-0.2.0/Makefile Makefile" | git annex addurl --debug -c annex.largefiles=exclude=*Makefile --with-files --json --batch                                                                                                                                                                                                    
+{"command":"addurl","file":"Makefile","note":"from datalad-archives","note":"non-large file; adding content to git repository","success":true}
+
+$ echo "dl+archive:SHA256E-s5121--6d8f7d10206a120a42bec2cd29bc2365d09889fdf070ac8c67d1cff0b1539f63.tar.gz/svgtune-0.2.0/README.rst README-2.rst" | git annex addurl -c annex.largefiles=exclude=*.rst --with-files --json --batch                                                                                                                                          
+{"command":"addurl","file":"README-2.rst","note":"from datalad-archives","note":"non-large file; adding content to git repository","success":true}
+
+
+"""]]
+
+and to re-confirm  that only for the first file, I am redoing above but first asking to addurl Makefile (goes errorneously to annex) and then README.rst (goes to git as it should)
+
+[[!format sh """
+$ chmod a+w -R /tmp/123; rm -rf /tmp/123; mkdir /tmp/123; cd /tmp/123; git init; git annex init;
+Initialized empty Git repository in /tmp/123/.git/
+init  ok
+(recording state in git...)
+
+$ git annex addurl --file svgtune_0.2.0.orig.tar.gz http://http.debian.net/debian/pool/main/s/svgtune/svgtune_0.2.0.orig.tar.gz
+addurl svgtune_0.2.0.orig.tar.gz (downloading http://http.debian.net/debian/pool/main/s/svgtune/svgtune_0.2.0.orig.tar.gz ...) 
+/tmp/123/.git/annex/tmp/URL-s5121--http&c%%http.debian. 100%[===============================================================================================================================>]   5.00K  --.-KB/s    in 0s      
+ok
+(recording state in git...)
+
+$ git annex initremote datalad-archives externaltype=dl+archive type=external autoenable=true encryption=none
+initremote datalad-archives ok
+(recording state in git...)
+
+$ echo "dl+archive:SHA256E-s5121--6d8f7d10206a120a42bec2cd29bc2365d09889fdf070ac8c67d1cff0b1539f63.tar.gz/svgtune-0.2.0/Makefile Makefile" | git annex addurl --debug -c annex.largefiles=exclude=*Makefile --with-files --json --batch                                                                                                                                                                                                                         
+{"command":"addurl","file":"Makefile","note":"from datalad-archives","success":true}                                                                   
+
+$ echo "dl+archive:SHA256E-s5121--6d8f7d10206a120a42bec2cd29bc2365d09889fdf070ac8c67d1cff0b1539f63.tar.gz/svgtune-0.2.0/README.rst README.rst" | git annex addurl -c annex.largefiles=exclude=*.rst --with-files --json --batch
+{"command":"addurl","file":"README.rst","note":"from datalad-archives","note":"non-large file; adding content to git repository","success":true}
+
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+
diff --git a/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_1_5e9dfaa4dd781f231954c5c1cf88c418._comment b/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_1_5e9dfaa4dd781f231954c5c1cf88c418._comment
new file mode 100644
index 000000000..389442738
--- /dev/null
+++ b/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_1_5e9dfaa4dd781f231954c5c1cf88c418._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4"
+ subject="comment 1"
+ date="2016-01-19T19:48:10Z"
+ content="""
+not sure if of help, but I think it happens for the first 'addurl --batch' process invocation, not just for the first url (in my examples above I didn't have persistent --batch'ed process), since in the results of my batched runs (single addurl --batch process) I have not just a single file but many added to annex instead of git. (previously used a sequence of add/find/addurl which now replaced with a batched addurl)
+"""]]
diff --git a/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_2_4587641f1f2b358dfaa43ac69d4ade49._comment b/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_2_4587641f1f2b358dfaa43ac69d4ade49._comment
new file mode 100644
index 000000000..d0acd0fdd
--- /dev/null
+++ b/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_2_4587641f1f2b358dfaa43ac69d4ade49._comment
@@ -0,0 +1,37 @@
+[[!comment format=mdwn
+ username="https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4"
+ subject="comment 2"
+ date="2016-01-19T19:50:25Z"
+ content="""
+and actually has nothing to do with custom special remotes -- the same for regular url:
+
+[[!format sh \"\"\"
+$ chmod a+w -R /tmp/123; rm -rf /tmp/123; mkdir /tmp/123; cd /tmp/123; git init; git annex init;                                                                                                                               
+Initialized empty Git repository in /tmp/123/.git/
+init  ok
+(recording state in git...)
+
+$ git annex addurl --file svgtune_0.2.0.orig.tar.gz http://http.debian.net/debian/pool/main/s/svgtune/svgtune_0.2.0.orig.tar.gz
+addurl svgtune_0.2.0.orig.tar.gz (downloading http://http.debian.net/debian/pool/main/s/svgtune/svgtune_0.2.0.orig.tar.gz ...) 
+/tmp/123/.git/annex/tmp/URL-s5121--http&c%%http.debian. 100%[===============================================================================================================================>]   5.00K  --.-KB/s    in 0s      
+ok
+(recording state in git...)
+
+$ git annex initremote datalad-archives externaltype=dl+archive type=external autoenable=true encryption=none
+initremote datalad-archives ok
+(recording state in git...)
+
+$ echo \"http://www.onerussian.com/tmp/README2.rst README2.rst\" | git annex addurl -c annex.largefiles=exclude=*.rst --with-files --json --batch
+{\"command\":\"addurl\",\"file\":\"README2.rst\",\"note\":\"downloading http://www.onerussian.com/tmp/README2.rst ...\",\"success\":true}
+
+$ ls -l README2.rst 
+lrwxrwxrwx 1 yoh yoh 188 Jan 19 14:49 README2.rst -> .git/annex/objects/9p/28/SHA256E-s13--1c87b6727f523662df714f06a94ea27fa4d9050c38f4f7712bd4663ffbfdfa01.rst/SHA256E-s13--1c87b6727f523662df714f06a94ea27fa4d9050c38f4f7712bd4663ffbfdfa01.rst
+
+$ echo \"http://www.onerussian.com/tmp/README2.rst README2_.rst\" | git annex addurl -c annex.largefiles=exclude=*.rst --with-files --json --batch 
+{\"command\":\"addurl\",\"file\":\"README2_.rst\",\"note\":\"downloading http://www.onerussian.com/tmp/README2.rst ...\",\"note\":\"non-large file; adding content to git repository\",\"success\":true}
+
+$ ls -l README2_.rst 
+-rw------- 1 yoh yoh 13 Jan 19 14:34 README2_.rst
+
+\"\"\"]]
+"""]]
diff --git a/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_3_18b28edd16df0a54c27a9b905fe04d8c._comment b/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_3_18b28edd16df0a54c27a9b905fe04d8c._comment
new file mode 100644
index 000000000..954a20bb8
--- /dev/null
+++ b/doc/bugs/addurl_--batch_from_url_from_a_custom_special_remote_adds_to_annex_disregarding_largefiles___40__on_first_run__41__/comment_3_18b28edd16df0a54c27a9b905fe04d8c._comment
@@ -0,0 +1,35 @@
+[[!comment format=mdwn
+ username="https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4"
+ subject="actually has smth to do with remotes "
+ date="2016-01-19T19:53:25Z"
+ content="""
+since if I don't initremote -- adds to git.  If I initremote -- adds to annex:
+
+[[!format sh \"\"\"
+$ chmod a+w -R /tmp/123; rm -rf /tmp/123; mkdir /tmp/123; cd /tmp/123; git init; git annex init;
+Initialized empty Git repository in /tmp/123/.git/
+init  ok
+(recording state in git...)
+
+$ echo \"http://www.onerussian.com/tmp/README2.rst README2.rst\" | git annex addurl -c annex.largefiles=exclude=*.rst --with-files --json --batch 
+{\"command\":\"addurl\",\"file\":\"README2.rst\",\"note\":\"downloading http://www.onerussian.com/tmp/README2.rst ...\",\"note\":\"non-large file; adding content to git repository\",\"success\":true}
+
+
+
+$ chmod a+w -R /tmp/123; rm -rf /tmp/123; mkdir /tmp/123; cd /tmp/123; git init; git annex init;                                               
+Initialized empty Git repository in /tmp/123/.git/
+init  ok
+(recording state in git...)
+
+$ git annex initremote datalad-archives externaltype=dl+archive type=external autoenable=true encryption=none                                  
+initremote datalad-archives ok
+(recording state in git...)
+
+$ echo \"http://www.onerussian.com/tmp/README2.rst README2.rst\" | git annex addurl -c annex.largefiles=exclude=*.rst --with-files --json --batch
+{\"command\":\"addurl\",\"file\":\"README2.rst\",\"note\":\"downloading http://www.onerussian.com/tmp/README2.rst ...\",\"success\":true}
+
+$ ls -l README2.rst 
+lrwxrwxrwx 1 yoh yoh 188 Jan 19 14:51 README2.rst -> .git/annex/objects/9p/28/SHA256E-s13--1c87b6727f523662df714f06a94ea27fa4d9050c38f4f7712bd4663ffbfdfa01.rst/SHA256E-s13--1c87b6727f523662df714f06a94ea27fa4d9050c38f4f7712bd4663ffbfdfa01.rst
+
+\"\"\"]]
+"""]]
diff --git a/doc/todo/--batch_for_find/comment_4_830cf616968e25cee8e7d35f9709f79b._comment b/doc/todo/--batch_for_find/comment_4_830cf616968e25cee8e7d35f9709f79b._comment
new file mode 100644
index 000000000..35f2873c0
--- /dev/null
+++ b/doc/todo/--batch_for_find/comment_4_830cf616968e25cee8e7d35f9709f79b._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4"
+ subject="comment 4"
+ date="2016-01-19T19:24:11Z"
+ content="""
+you caught me! ;) indeed in this case we could use 'info' for that purpose.  But if we decide if content is present under annex (which is what actually our function does, we just used it for determining \"if under annex\" after addurl) -- then find --batch would be needed, or whois --batch could be used.  So we are both correct! ;)
+"""]]
diff --git a/doc/todo/return___34__key__34___entry_in_--json_output_for_addurl___40__and_future_add__41___--batch.mdwn b/doc/todo/return___34__key__34___entry_in_--json_output_for_addurl___40__and_future_add__41___--batch.mdwn
new file mode 100644
index 000000000..25d852d23
--- /dev/null
+++ b/doc/todo/return___34__key__34___entry_in_--json_output_for_addurl___40__and_future_add__41___--batch.mdwn
@@ -0,0 +1,3 @@
+You have noted that somewhere (may be in email), that it might help us to pipeline things if 'add' was returning the key if file was added to the annex.  I guess the same could apply to 'addurl' so decided to mark this separate todo
+
+[[!meta author=yoh]]
diff --git a/doc/todo/smudge.mdwn b/doc/todo/smudge.mdwn
index 2178154ee..8ddf337b7 100644
--- a/doc/todo/smudge.mdwn
+++ b/doc/todo/smudge.mdwn
@@ -4,12 +4,16 @@ git-annex should use smudge/clean filters.
 
 * Reconcile staged changes into the associated files database, whenever
   the database is queried. This is needed to handle eg:
+
+  ```
   	git add largefile
 	git mv largefile othername
 	git annex move othername --to foo
 	# fails to drop content from associated file othername,
 	# because it doesn't know it has that name
 	# git commit clears up this mess
+  ```
+
 * Dropping a smudged file causes git status (and git annex status)
   to show it as modified,  because the timestamp has changed. 
   Getting a smudged file can also cause this.