Merge branch 'master' into tor

11 files changed, 231 insertions, 6 deletions

diff --git a/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_2_afa6a131999feda67876859cd85ebcfc._comment b/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_2_afa6a131999feda67876859cd85ebcfc._comment
new file mode 100644
index 000000000..b0db54479
--- /dev/null
+++ b/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_2_afa6a131999feda67876859cd85ebcfc._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="luckcolorsgoo@ab4f3c1c44a7dbcbcb9d9a29315b59ad524ceaaa"
+ nickname="luckcolorsgoo"
+ avatar="http://cdn.libravatar.org/avatar/ddff84cd2a97252a05cccb4bc5010448"
+ subject="comment 2"
+ date="2016-11-16T22:56:46Z"
+ content="""
+In my case i was going to make a script for automatically downloading and updating an git portbale / git annex instance, by first fetching git portbale and then downloading the gitannex exe.
+
+So yeah it's more reliable to extract an archive rather than trying to extract the setup without executing it.
+That's why i'm asking for this feature. :)
+
+
+
+"""]]
diff --git a/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed/comment_1_39718e8a35e42421a8aaf3316ae1d76a._comment b/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed/comment_1_39718e8a35e42421a8aaf3316ae1d76a._comment
new file mode 100644
index 000000000..02b62f46e
--- /dev/null
+++ b/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed/comment_1_39718e8a35e42421a8aaf3316ae1d76a._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="andrew"
+ avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
+ subject="RESOLVED"
+ date="2016-11-17T14:59:15Z"
+ content="""
+Ooops. I am on OS-X. I use brew for my gnupg installation. It appears I had removed gpg from the path when installing something. I just needed to run to fix:
+
+    brew link gnupg2
+
+Thanks,
+
+Andrew
+"""]]
diff --git a/doc/bugs/when_you_get_a_file_but_don__39__t_actually_have_enough_space_for_it__44___the_error_message_makes_useless_suggestions.mdwn b/doc/bugs/when_you_get_a_file_but_don__39__t_actually_have_enough_space_for_it__44___the_error_message_makes_useless_suggestions.mdwn
new file mode 100644
index 000000000..19e839263
--- /dev/null
+++ b/doc/bugs/when_you_get_a_file_but_don__39__t_actually_have_enough_space_for_it__44___the_error_message_makes_useless_suggestions.mdwn
@@ -0,0 +1,21 @@
+The suggestion to make remotes available isn't really applicable, since the error was local.
+
+This is with git annex 6.20161110-gd48f4ca.
+
+[[!format sh """
+  ../git-annex.linux/git-annex get archiveteam-fire/metro.co.uk-urls-2007-04-12-20150627/metro.co.uk-urls-2007-04-12-20150627_meta.xml
+get archiveteam-fire/metro.co.uk-urls-2007-04-12-20150627/metro.co.uk-urls-2007-04-12-20150627_meta.xml 
+  not enough free space, need 98.82 GB more (use --force to override this check or adjust annex.diskreserve)
+
+  Unable to access these remotes: web
+
+  Try making some of these repositories available:
+        00000000-0000-0000-0000-000000000001 -- web
+        9f8218c0-763f-463d-9152-ecdc56d4452c -- iabak@redwyne.jwintjen.de:~/IA.BAK/shard12
+failed
+git-annex: get: 1 failed
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+mixed success
diff --git a/doc/devblog/day_426__grab_bag.mdwn b/doc/devblog/day_426__grab_bag.mdwn
index 4a3bf908b..36e32077e 100644
--- a/doc/devblog/day_426__grab_bag.mdwn
+++ b/doc/devblog/day_426__grab_bag.mdwn
@@ -45,19 +45,19 @@ authentication. Which is a whole nother ball of complexity. So, I'm leaning
 instead to using a simple custom protocol something like:
 
 		> AUTH $localuuid $token
-		< AUTH-OK $remoteuuid
+		< AUTH-SUCCESS $remoteuuid
 		> SENDPACK $length
 		> $gitdata
 		< RECVPACK $length
 		< $gitdata
 		> GET $pos $key
-		< SENDING $length
+		< DATA $length
 		< $bytes
-		> GET-OK
+		> SUCCESS
 		> PUT $key
-		< SEND $pos
-		> SENDING $length
+		< PUT-FROM $pos
+		> DATA $length
 		> $bytes
-		< PUT-OK
+		< SUCCESS
 
 Today's work was sponsored by Riku Voipio.
diff --git a/doc/devblog/day_427__free_p2p.mdwn b/doc/devblog/day_427__free_p2p.mdwn
new file mode 100644
index 000000000..7c727587b
--- /dev/null
+++ b/doc/devblog/day_427__free_p2p.mdwn
@@ -0,0 +1,51 @@
+For a Haskell programmer, and day where a big thing is implemented
+without the least scrap of code that touches the IO monad is a good day.
+And this was a good day for me!
+
+Implemented the p2p protocol for tor hidden services. Its needs are somewhat
+similar to the external special remote protocol, but the two protocols are
+not fully overlapping with one-another. Rather than try to unify them, and
+so complicate both cases, I prefer to reuse as much code as possible between
+separate protocol implementations. The generating and parsing of messages
+is largely shared between them. I let the new p2p protocol otherwise
+develop in its own direction.
+
+But, I *do* want to make this p2p protocol reusable for other types of p2p
+networks than tor hidden services. This was an opportunity to use the Free
+monad, which I'd never used before. It worked out great, letting me write
+monadic code to handle requests and responses in the protocol, that reads
+the content of files and resumes transfers and so on, all independent
+of any concrete implementation.
+
+The whole implementation of the protocol only needed 74 lines of monadic code.
+It helped that I was able to factor out functions like this one, that is used
+both for handling a download, and by the remote when an upload is sent to it:
+
+	receiveContent :: Key -> Offset -> Len -> Proto Bool
+	receiveContent key offset len = do
+	        content <- receiveBytes len
+	        ok <- writeKeyFile key offset content
+	        sendMessage $ if ok then SUCCESS else FAILURE
+	        return ok
+
+To get transcripts of the protocol in action, the Free monad can be evaluated
+purely, providing the other side of the conversation:
+
+	ghci> putStrLn $ protoDump $ runPure (put (fromJust $ file2key "WORM--foo")) [PUT_FROM (Offset 10), SUCCESS]
+	> PUT WORM--foo
+	< PUT-FROM 10
+	> DATA 90
+	> bytes
+	< SUCCESS
+	result: True
+
+	ghci> putStrLn $ protoDump $ runPure (serve (toUUID "myuuid")) [GET (Offset 0) (fromJust $ file2key "WORM--foo")]
+	< GET 0 WORM--foo
+	> PROTO-ERROR must AUTH first
+	result: ()
+
+Am very happy with all this pure code and that I'm finally using Free monads.
+Next I need to get down the the dirty business of wiring this up to
+actual IO actions, and an actual network connection.
+
+Today's work was sponsored by Jake Vosloo on Patreon.
diff --git a/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__.mdwn b/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__.mdwn
new file mode 100644
index 000000000..512e89528
--- /dev/null
+++ b/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__.mdwn
@@ -0,0 +1,9 @@
+I have a special remote that I would like to delete and have marked it as such in the assistant. Although this was before my myriad of problems with git annex itself wanting to repair the repo all the time. Right now if I take a loog into my daemon.log I see the following error over and over again:
+
+```
+drop skydrive foo.bar 
+  This file could not be removed
+failed
+```
+
+I checked if I can login into my account and it works just fine. So I assume that this might be a bug? Is it somehow possible to forego the cleaning out of the special remote and just mark it as deleted for good? Thanks in advance!
diff --git a/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__/comment_1_0b523b2b6c361346c36ad456bbbac645._comment b/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__/comment_1_0b523b2b6c361346c36ad456bbbac645._comment
new file mode 100644
index 000000000..cf808aac4
--- /dev/null
+++ b/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__/comment_1_0b523b2b6c361346c36ad456bbbac645._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2016-11-18T15:39:20Z"
+ content="""
+It could certianly be a bug in the special remote implementation. It's also
+possible for some special remotes to intentionally not be able to remove
+content (this is the case with the web special remote, and the bup special
+remote at least).
+
+You can manually remove the special remote, by editing .git/config and
+deleting the stanza for that remote. You may want to run `git annex dead
+$remotename` first, if you don't intend to ever use that special remote
+again.
+"""]]
diff --git a/doc/forum/how_to_disaster_recovery/comment_12_f2e570dc60a6f16e8f696d94e253775f._comment b/doc/forum/how_to_disaster_recovery/comment_12_f2e570dc60a6f16e8f696d94e253775f._comment
new file mode 100644
index 000000000..7c7da0ef4
--- /dev/null
+++ b/doc/forum/how_to_disaster_recovery/comment_12_f2e570dc60a6f16e8f696d94e253775f._comment
@@ -0,0 +1,7 @@
+[[!comment format=mdwn
+ username="openmedi"
+ subject="comment 12"
+ date="2016-11-18T12:03:44Z"
+ content="""
+A recent update to annex via homebrew now reslolves the issue with the weird looking webapp.
+"""]]
diff --git a/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_4_a7f476aeacf88679f25badc78fad886a._comment b/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_4_a7f476aeacf88679f25badc78fad886a._comment
new file mode 100644
index 000000000..eec45e333
--- /dev/null
+++ b/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_4_a7f476aeacf88679f25badc78fad886a._comment
@@ -0,0 +1,57 @@
+[[!comment format=mdwn
+ username="andrew"
+ avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
+ subject="git annex copy --auto --to cloud works"
+ date="2016-11-17T17:49:27Z"
+ content="""
+Yes, only `git annex sync --content` seems to fail. I am using v6 with a mix of unlocked and locked files. I did not know about the --auto flags for copy/get.
+
+* `git annex copy --auto --to cloud` works fine
+* `git annex get --auto --from cloud` works fine
+
+
+*Are there any symlinks in the path to /Users/andrew/notes ? Eg, is /Users a symlink, or /Users/andrew a symlink, or //Users/andrew/notes itself symlinked to elsewhere?*
+
+**No**
+
+*You say it's only failing for some files. Do the filenames that it's failing on contain any non-ascii characters?*
+
+**They seem normal.**
+
+*So, please paste in git annex version and git annex info output.*
+
+    git-annex version: 6.20161110-gd48f4ca
+    build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
+    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
+    remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
+    local repository version: 6
+    supported repository versions: 3 5 6
+    upgrade supported from repository versions: 0 1 2 3 4 5
+    operating system: darwin x86_64
+
+    repository mode: indirect
+    trusted repositories: 0
+    semitrusted repositories: 10
+    00000000-0000-0000-0000-000000000001 -- web
+    00000000-0000-0000-0000-000000000002 -- bittorrent
+    22de57a0-c9ca-4bfe-8349-3141b3a87c8f -- Dream Objects [cloud]
+    334791ca-c284-4a87-a233-fc29be00d31a -- [disc_May-2-2015_a]
+    4c57ac0e-b8fe-4b4b-98d3-fb0a1b6b9657 -- MacBook Air [here]
+    6a85150d-6ea2-4ba1-92ce-8f4ef575b8e0 -- prowl MacBook Mini
+    896c3d52-427a-41a1-867c-d18e6740d758 -- disc_May_4_2015_1
+    96391b13-3981-430f-ac3b-6210e3d4e759 -- [disc_May-2-2015_b]
+    b4a41e90-2398-4bba-aaf5-d8f8cd78a5bc -- 2TB USB Drive [usbdrive]
+    e42b223d-ec04-4ad8-bdf7-8429a45d844c -- disc_May-2-2015_a
+    untrusted repositories: 0
+    transfers in progress: none
+    available local disk space: 2.32 gigabytes (+1 megabyte reserved)
+    temporary object directory size: 29.47 megabytes (clean up with git-annex unused)
+    local annex keys: 4104
+    local annex size: 10.53 gigabytes
+    annexed files in working tree: 6417
+    size of annexed files in working tree: 80.75 gigabytes
+    bloom filter size: 32 mebibytes (0.8% full)
+    backend usage: 
+    SHA256E: 6417
+
+"""]]
diff --git a/doc/forum/uuid_mismatch___59___expected_Just___40__UUID_...___41___but_remote_gitrepo_has_UUID_.../comment_3_a681a4847acbe890c4e486288b3c81d3._comment b/doc/forum/uuid_mismatch___59___expected_Just___40__UUID_...___41___but_remote_gitrepo_has_UUID_.../comment_3_a681a4847acbe890c4e486288b3c81d3._comment
new file mode 100644
index 000000000..d92f3fe17
--- /dev/null
+++ b/doc/forum/uuid_mismatch___59___expected_Just___40__UUID_...___41___but_remote_gitrepo_has_UUID_.../comment_3_a681a4847acbe890c4e486288b3c81d3._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="yomguy"
+ avatar="http://cdn.libravatar.org/avatar/03db077c04f8b753f3f504d9a2b06a29"
+ subject="comment 3"
+ date="2016-11-18T14:00:51Z"
+ content="""
+Hi joey,
+
+After modifying the gcrypt-id as you proposed, I have finally managed to clone the repo with  
+
+`git clone gcrypt::ssh://my.domain/home/admin/`
+
+But now I get only unresolved symbolic links for each files, that is .git/annex/objects directory only contains .map files.
+
+Would you have an idea about the reason/source of this behavior?
+
+Thank you so much,
+Guillaume
+"""]]
diff --git a/doc/news/version_6.20161118.mdwn b/doc/news/version_6.20161118.mdwn
new file mode 100644
index 000000000..42d86282c
--- /dev/null
+++ b/doc/news/version_6.20161118.mdwn
@@ -0,0 +1,17 @@
+git-annex 6.20161118 released with [[!toggle text="these changes"]]
+[[!toggleable text="""
+   * git-annex.cabal: Loosen bounds on persistent to allow 2.5, which
+     on Debian has been patched to work with esqueleto.
+     This may break cabal's resolver on non-Debian systems;
+     if so, either use stack to build, or run cabal with
+     --constraint='persistent ==2.2.4.1'
+     Hopefully this mess with esqueleto will be resolved soon.
+   * sync: Pass --allow-unrelated-histories to git merge when used with git
+     git 2.9.0 or newer. This makes merging a remote into a freshly created
+     direct mode repository work the same as it works in indirect mode.
+   * Avoid backtraces on expected failures when built with ghc 8;
+     only use backtraces for unexpected errors.
+   * fsck --all --from was checking the existence and content of files
+     in the local repository, rather than on the special remote. Oops.
+   * Linux arm standalone: Build with a 32kb page size, which is needed
+     on several ARM NAS devices, including Drobo 5N, and WD NAS."""]]
\ No newline at end of file