15 files changed, 416 insertions, 0 deletions
diff --git a/doc/devblog/day_425__tor.mdwn b/doc/devblog/day_425__tor.mdwn
new file mode 100644
index 000000000..08fe21cdd
--- /dev/null
+++ b/doc/devblog/day_425__tor.mdwn
@@ -0,0 +1,23 @@
+Have waited too long for some next-generation encrypted P2P network, like
+telehash to emerge. Time to stop waiting; tor hidden services are not as
+cutting edge, but should work. Updated the [[design|design/assistant/telehash]]
+and started implementation in the `tor` branch.
+
+Unfortunately, Tor's default configuration does not enable the ControlPort.
+And, changing that in the configuration could be problimatic. This
+makes it harder than it ought to be to register a tor hidden service.
+So, I implemented a `git annex enable-tor` command, which can be run as root
+to set it up. The webapp will probably use `su-to-root` or `gksu` to run it. 
+There's some Linux-specific parts in there, and it uses a socket for
+communication between tor and the hidden service, which may cause problems
+for Windows porting later.
+
+Next step will be to get `git annex remotedaemon` to run as a tor hidden
+service.
+
+Also made a `no-xmpp` branch which removes xmpp support from the assistant.
+That will remove 3000 lines of code when it's merged. Will probably wait
+until after tor hidden services are working.
+
+Today's work was sponsored by Jake Vosloo on
+[Patreon](https://www.patreon.com/joeyh/).
diff --git a/doc/devblog/day_425__tor/comment_1_1dd41fa32eb3867d764f3238005b5b81._comment b/doc/devblog/day_425__tor/comment_1_1dd41fa32eb3867d764f3238005b5b81._comment
new file mode 100644
index 000000000..fe609cab0
--- /dev/null
+++ b/doc/devblog/day_425__tor/comment_1_1dd41fa32eb3867d764f3238005b5b81._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="grawity@2ea26be48562f66fcb9b66307da72b1e2e37453f"
+ nickname="grawity"
+ avatar="http://cdn.libravatar.org/avatar/7003e967f47003bae82966aa373de8ef"
+ subject="comment 1"
+ date="2016-11-15T18:01:18Z"
+ content="""
+…or `pkexec`, which is present on many systems and generally integrates better with whatever DE/non-DE the user may be running.
+
+(OTOH, pkexec does not set up X11 access – then again, root helpers shouldn't need it.)
+"""]]
diff --git a/doc/devblog/day_426__grab_bag.mdwn b/doc/devblog/day_426__grab_bag.mdwn
new file mode 100644
index 000000000..36e32077e
--- /dev/null
+++ b/doc/devblog/day_426__grab_bag.mdwn
@@ -0,0 +1,63 @@
+Fixed one howler of a bug today. Turns out that
+`git annex fsck --all --from remote` didn't actually check the content of
+the remote, but checked the local repository. Only `--all` was buggy;
+`git annex fsck --from remote` was ok. Don't think this is crash priority
+enough to make a release for, since only `--all` is affected.
+
+Somewhat uncomfortably made `git annex sync` pass
+`--allow-unrelated-histories` to git merge. While I do think that git's
+recent refusal to merge unrelated histories is good in general, the
+problem is that initializing a direct mode repository involves making an
+empty commit. So merging from a remote into such a direct mode repository
+means merging unrelated histories, while an indirect mode repository doesn't.
+Seems best to avoid such inconsistencies, and the only way I could see to
+do it is to always use `--allow-unrelated-histories`. May revisit this once
+direct mode is finally removed.
+
+Using the git-annex arm standalone bundle on some WD NAS boxes used to
+work, and then it seems they changed their kernel to use a nonstandard page
+size, and broke it. This actually seems to be a 
+[bug in the gold linker](http://bugs.debian.org/844467), which defaults to an
+unncessarily small page size on arm. The git-annex arm bundle is being
+adjusted to try to deal with this.
+
+ghc 8 made `error` include some backtrace information. While it's really
+nice to have backtraces for unexpected exceptions in Haskell, it turns
+out that git-annex used `error` a lot with the intent of showing an error
+message to the user, and a backtrace clutters up such messages. So,
+bit the bullet and checked through every `error` in git-annex and made such
+ones not include a backtrace.
+
+Also, I've been considering what protocol to use between git-annex nodes
+when communicating over tor. One way would be to make it very similar to
+`git-annex-shell`, using rsync etc, and possibly reusing code from
+git-annex-shell. However, it can take a while to make a connection across
+the tor network, and that method seems to need a new connection for each
+file transfered etc. Also thought about using a http based protocol. The
+servant library is great for that, you get both http client and server
+implementations almost for free. Resuming interrupted transfers might
+complicate it, and the hidden service side would need to listen on a unix
+socket, instead of the regular http port. It might be worth it to use http
+for tor, if it could be reused for git-annex http servers not on the tor
+network. But, then I'd have to make the http server support git pull and
+push over http in a way that's compatable with how git uses http, including
+authentication. Which is a whole nother ball of complexity. So, I'm leaning
+instead to using a simple custom protocol something like:
+
+		> AUTH $localuuid $token
+		< AUTH-SUCCESS $remoteuuid
+		> SENDPACK $length
+		> $gitdata
+		< RECVPACK $length
+		< $gitdata
+		> GET $pos $key
+		< DATA $length
+		< $bytes
+		> SUCCESS
+		> PUT $key
+		< PUT-FROM $pos
+		> DATA $length
+		> $bytes
+		< SUCCESS
+
+Today's work was sponsored by Riku Voipio.
diff --git a/doc/devblog/day_426__grab_bag/comment_1_4d01c756850032d351fa99188a3301a7._comment b/doc/devblog/day_426__grab_bag/comment_1_4d01c756850032d351fa99188a3301a7._comment
new file mode 100644
index 000000000..7b5a2949a
--- /dev/null
+++ b/doc/devblog/day_426__grab_bag/comment_1_4d01c756850032d351fa99188a3301a7._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="https://anarc.at/openid/"
+ nickname="anarcat"
+ avatar="http://cdn.libravatar.org/avatar/b36dcf65657dd36128161355d8920a99503def9461c1bb212410980fe6f07125"
+ subject="how about reusing the special remote protocol?"
+ date="2016-11-16T21:58:08Z"
+ content="""
+git-annex already has a custom protocol detailed in [[design/external_special_remote_protocol]]. it could be quite useful to have that protocol extended to support direct object transfer instead of having to mess around with temporary files like may remotes do, for example...
+
+maybe that makes no sense at all, i don't know. :) --[[anarcat]]
+"""]]
diff --git a/doc/devblog/day_427__free_p2p.mdwn b/doc/devblog/day_427__free_p2p.mdwn
new file mode 100644
index 000000000..7c727587b
--- /dev/null
+++ b/doc/devblog/day_427__free_p2p.mdwn
@@ -0,0 +1,51 @@
+For a Haskell programmer, and day where a big thing is implemented
+without the least scrap of code that touches the IO monad is a good day.
+And this was a good day for me!
+
+Implemented the p2p protocol for tor hidden services. Its needs are somewhat
+similar to the external special remote protocol, but the two protocols are
+not fully overlapping with one-another. Rather than try to unify them, and
+so complicate both cases, I prefer to reuse as much code as possible between
+separate protocol implementations. The generating and parsing of messages
+is largely shared between them. I let the new p2p protocol otherwise
+develop in its own direction.
+
+But, I *do* want to make this p2p protocol reusable for other types of p2p
+networks than tor hidden services. This was an opportunity to use the Free
+monad, which I'd never used before. It worked out great, letting me write
+monadic code to handle requests and responses in the protocol, that reads
+the content of files and resumes transfers and so on, all independent
+of any concrete implementation.
+
+The whole implementation of the protocol only needed 74 lines of monadic code.
+It helped that I was able to factor out functions like this one, that is used
+both for handling a download, and by the remote when an upload is sent to it:
+
+	receiveContent :: Key -> Offset -> Len -> Proto Bool
+	receiveContent key offset len = do
+	        content <- receiveBytes len
+	        ok <- writeKeyFile key offset content
+	        sendMessage $ if ok then SUCCESS else FAILURE
+	        return ok
+
+To get transcripts of the protocol in action, the Free monad can be evaluated
+purely, providing the other side of the conversation:
+
+	ghci> putStrLn $ protoDump $ runPure (put (fromJust $ file2key "WORM--foo")) [PUT_FROM (Offset 10), SUCCESS]
+	> PUT WORM--foo
+	< PUT-FROM 10
+	> DATA 90
+	> bytes
+	< SUCCESS
+	result: True
+
+	ghci> putStrLn $ protoDump $ runPure (serve (toUUID "myuuid")) [GET (Offset 0) (fromJust $ file2key "WORM--foo")]
+	< GET 0 WORM--foo
+	> PROTO-ERROR must AUTH first
+	result: ()
+
+Am very happy with all this pure code and that I'm finally using Free monads.
+Next I need to get down the the dirty business of wiring this up to
+actual IO actions, and an actual network connection.
+
+Today's work was sponsored by Jake Vosloo on Patreon.
diff --git a/doc/devblog/day_428-429__git_push_to_hiddden_service.mdwn b/doc/devblog/day_428-429__git_push_to_hiddden_service.mdwn
new file mode 100644
index 000000000..ec1bf12fd
--- /dev/null
+++ b/doc/devblog/day_428-429__git_push_to_hiddden_service.mdwn
@@ -0,0 +1,31 @@
+The `tor` branch is coming along nicely.
+
+This weekend, I continued working on the P2P protocol, implementing
+it for network sockets, and extending it to support connecting up
+git-send-pack/git-receive-pack. 
+
+There was a bit of a detour when I split the Free monad into two separate
+ones, one for Net operations and the other for Local filesystem operations.
+
+This weekend's work was sponsored by Thomas Hochstein on Patreon.
+
+----
+
+Today, implemented a `git-remote-tor-annex` command that git will
+use for tor-annex:: urls,  and made `git annex remotedaemon`
+serve the tor hidden service.
+
+Now I have git push/pull working to the hidden service, for example:
+
+	git pull tor-annex::eeaytkuhaupbarfi.onion:47651
+
+That works very well, but does not yet check that the user is authorized
+to use the repo, beyond knowing the onion address. And currently
+it only works in git-annex repos; with some tweaks it should
+also work in plain git repos.
+
+Next, I need to teach git-annex how to access tor-annex remotes.
+And after that, an interface in the webapp for setting them up and
+connecting them together.
+
+Today's work was sponsored by Josh Taylor on Patreon.
diff --git a/doc/devblog/day_430__tor_socket_problem.mdwn b/doc/devblog/day_430__tor_socket_problem.mdwn
new file mode 100644
index 000000000..7e7c8d1bd
--- /dev/null
+++ b/doc/devblog/day_430__tor_socket_problem.mdwn
@@ -0,0 +1,13 @@
+Debian's tor daemon is very locked down in the directories it can read
+from, and so I've had a hard time finding a place to put the unix socket
+file for git-annex's tor hidden service. Painful details in 
+<http://bugs.debian.org/846275>. At least for now, I'm putting it under
+/etc/tor/, which is probably a FHS violation, but seems to be the only
+option that doesn't involve a lot of added complexity.
+
+--- 
+
+The Windows autobuilder is moving, since
+[NEST](http://nest-initiative.org/) is shutting down the server it has been
+using. Yury Zaytsev has set up a new Windows autobuilder, hosted at
+Dartmouth College this time.
diff --git a/doc/devblog/day_431__p2p_linking.mdwn b/doc/devblog/day_431__p2p_linking.mdwn
new file mode 100644
index 000000000..1e53ffefc
--- /dev/null
+++ b/doc/devblog/day_431__p2p_linking.mdwn
@@ -0,0 +1,27 @@
+Today I finished the second-to-last big missing peice for tor hidden service
+remotes. Networks of these remotes are P2P networks, and there needs to be
+a way for peers to find one-another, and to authenticate with one-another.
+The `git annex p2p` command sets up links between peers in such a network.
+
+So far it has only a basic interface that sets up a one way link between
+two peers. In the first repository, run `git annex p2p --gen-address`.
+That outputs a long address. In the second repository, run
+`git annex p2p --link peer1`, and paste the address into it. That sets up a
+git remote named "peer1" that connects back to the first repository over tor.
+
+That is a one-directional link, while a bi-directional link would be
+much more convenient to have between peers. Worse, the address can be reused by
+anyone who sees it, to link into the repository. And, the address is far
+too long to communicate in any way except for pasting it.
+
+So I want to improve that later. What I'd really like to have is an
+interface that displays a one-time-use phrase of five to ten words, that
+can be read over the phone or across the room. Exchange phrases with a
+friend, and get your repositories securely linked together with tor.
+
+But, `git annex p2p` is good enough for now. I can move on to the final
+keystone of the tor support, which is file transfer over tor.
+That should, fingers crossed, be relatively easy, and the `tor` branch is
+close to mergeable now.
+
+Today's work was sponsored by Riku Voipio.
diff --git a/doc/devblog/day_431__p2p_linking/comment_1_1d5f809564c25e765f82594af8e174ab._comment b/doc/devblog/day_431__p2p_linking/comment_1_1d5f809564c25e765f82594af8e174ab._comment
new file mode 100644
index 000000000..9eceb71ed
--- /dev/null
+++ b/doc/devblog/day_431__p2p_linking/comment_1_1d5f809564c25e765f82594af8e174ab._comment
@@ -0,0 +1,49 @@
+[[!comment format=mdwn
+ username="https://anarc.at/openid/"
+ nickname="anarcat"
+ avatar="http://cdn.libravatar.org/avatar/b36dcf65657dd36128161355d8920a99503def9461c1bb212410980fe6f07125"
+ subject="magic wormhole"
+ date="2016-11-30T22:16:19Z"
+ content="""
+> What I'd really like to have is an interface that displays a
+> one-time-use phrase of five to ten words, that can be read over the
+> phone or across the room. Exchange phrases with a friend, and get
+> your repositories securely linked together with tor.
+
+I already mentionned the project in [[design/assistant/telehash/]],
+but [magic-wormhole](https://github.com/warner/magic-wormhole) does
+exactly that:
+
+    % wormhole send README.md
+    Sending 7924 byte file named 'README.md'
+    On the other computer, please run: wormhole receive
+    Wormhole code is: 7-crossover-clockwork
+    
+    Sending (<-10.0.1.43:58988)..
+    100%|=========================| 7.92K/7.92K [00:00<00:00, 6.02MB/s]
+    File sent.. waiting for confirmation
+    Confirmation received. Transfer complete.
+
+Receiver:
+
+    % wormhole receive
+    Enter receive wormhole code: 7-crossover-clockwork
+    Receiving file (7924 bytes) into: README.md
+    ok? (y/n): y
+    Receiving (->tcp:10.0.1.43:58986)..
+    100%|===========================| 7.92K/7.92K [00:00<00:00, 120KB/s]
+    Received file written to README.md
+
+While that example shows a file transfer, arbitrary data can be
+transfered this way. There's a documented protocol, and it's not
+completely peer-to-peer: there are relay servers to deal with NAT'd
+machines. But the [PAKE
+protocol](https://en.wikipedia.org/wiki/Password-authenticated_key_agreement)
+(basically SPAKE2) could be a good inspiration here.
+
+Otherwise, I must say that, as a user, I don't mind copy-pasting a
+hidden service string (if that's what it's about): i can do that over
+a secure medium (email + OpenPGP or IM + OTR) easily... But I
+understand it can be difficult to do for new users.
+
+"""]]
diff --git a/doc/devblog/day_432-433__almost_there.mdwn b/doc/devblog/day_432-433__almost_there.mdwn
new file mode 100644
index 000000000..b41ce3f70
--- /dev/null
+++ b/doc/devblog/day_432-433__almost_there.mdwn
@@ -0,0 +1,13 @@
+Friday and today were spent implementing both sides of the P2P protocol for
+git-annex content transfers.
+
+There were some tricky cases to deal with. For example, when a file is being
+sent from a direct mode repository, or v6 annex.thin repository, the
+content of the file can change as it's being transferred. Including being
+appended to or truncated. Had to find a way to deal with that, to avoid
+breaking the protocol by not sending the indicated number of bytes of data.
+
+It all seems to be done now, but it's not been tested at all, and there are
+probably some bugs to find. (And progress info is not wired up yet.)
+
+Today's work was sponsored by Trenton Cronholm on Patreon.
diff --git a/doc/devblog/day_434__it_works.mdwn b/doc/devblog/day_434__it_works.mdwn
new file mode 100644
index 000000000..75d096b31
--- /dev/null
+++ b/doc/devblog/day_434__it_works.mdwn
@@ -0,0 +1,27 @@
+Git annex transfers over Tor worked correctly the first time I tried them
+today. I had been expecting protocol implementation bugs, so this was a
+nice surprise!
+
+Of course there were some bugs to fix. I had forgotten to add UUID
+discovery to `git annex p2p --link`. And, resuming interrupted transfers
+was buggy.
+
+Spent some time adding progress updates to the Tor remote. I was curious to
+see what speed transfers would run. Speed will of course vary depending on
+the Tor relays being used, but this example with a 100 mb file is not bad:
+
+	copy big4 (to peer1...) 
+	62%          1.5MB/s 24s
+
+There are still a couple of [[known bugs|todo/tor]], 
+but I've merged the `tor` branch into `master` already.
+
+----
+
+Alpernebbi has built a GUI for editing git-annex metadata.
+Something I always wanted!  
+[[Read about it here|tips/a_gui_for_metadata_operations]]
+
+----
+
+Today's work was sponsored by Ethan Aubin.
diff --git a/doc/devblog/day_435-436_post_tor_merge.mdwn b/doc/devblog/day_435-436_post_tor_merge.mdwn
new file mode 100644
index 000000000..2f05e0252
--- /dev/null
+++ b/doc/devblog/day_435-436_post_tor_merge.mdwn
@@ -0,0 +1,20 @@
+More improvements to tor support. Yesterday, debugged a reversion that
+broke push/pull over tor, and made actual useful error messages be
+displayed when there were problems. Also fixed a memory leak, although I
+fixed it by reorganizing code and could not figure out quite why it happened,
+other than that the ghc runtime was not managing to be as lazy as I would
+expect. 
+
+Today, added git ref change notification to the
+P2P protocol, and made the remotedaemon automatically fetch changes from
+tor remotes. So, it should work to use the assistant to keep
+repositories in sync over tor. I have not tried it yet, and linking over tor
+still needs to be done at the command line, so it's not really ready for
+webapp users yet.
+
+Also fixed a denial of service attack in git-annex-shell and git-annex when
+talking to a remote git-annex-shell. It was possible to feed either a large
+amount of data when they tried to read a line of data, and summon the OOM
+killer. Next release will be expedited some because of that.
+
+Today's work was sponsored by Thomas Hochstein on Patreon.
diff --git a/doc/devblog/day_437__catching_up.mdwn b/doc/devblog/day_437__catching_up.mdwn
new file mode 100644
index 000000000..1deb54459
--- /dev/null
+++ b/doc/devblog/day_437__catching_up.mdwn
@@ -0,0 +1,20 @@
+Quite a backlog developed in the couple of weeks I was concentrating on tor
+support. I've taken a first pass through it and fixed the most pressing
+issues now. 
+
+Most important was an ugly memory corruption problem in the GHC runtime
+system that may have led to data corruption when using git-annex with Linux
+kernels older than 4.5. All the Linux standalone builds of git-annex have
+been updated to fix that issue.
+
+Today dealt with several more things, including fixing a buggy timestamp
+issue with `metadata --batch`, reverting the ssh ServerAliveInterval
+setting (broke on too many systems with old ssh or complicated ssh
+configurations), making batch input not be rejected when it can't be decoded
+as UTF-8, and more.
+
+Also, spent some time learning a little bit about Magic Wormhole and SPAKE,
+as a way to exchange tor remote addresses. Using Magic Wormhole for that
+seems like a reasonable plan. I did file a couple bugs on it which will
+need to get fixed, and then using it is mostly a question of whether it's
+easy enough to install that git-annex can rely on it.
diff --git a/doc/devblog/day_438__bi-directional_p2p_links.mdwn b/doc/devblog/day_438__bi-directional_p2p_links.mdwn
new file mode 100644
index 000000000..abcbed122
--- /dev/null
+++ b/doc/devblog/day_438__bi-directional_p2p_links.mdwn
@@ -0,0 +1,6 @@
+Improved `git annex p2p --link` to create a bi-directional link
+automatically. Bi-directional links are desirable more often than not, so
+it's the default behavior.
+
+Also continued thinking about using magic wormhole for communicating
+p2p addresses for pairing. And filed some more bugs on magic wormhole.
diff --git a/doc/devblog/day_439__wormhole_pairing.mdwn b/doc/devblog/day_439__wormhole_pairing.mdwn
new file mode 100644
index 000000000..cc988a2db
--- /dev/null
+++ b/doc/devblog/day_439__wormhole_pairing.mdwn
@@ -0,0 +1,51 @@
+`git annex p2p --pair` implemented, using Magic Wormhole codes
+that have to be exchanged between the repositories being paired.
+
+It looks like this, with the same thing being done at the same time
+in the other repository.
+
+	joey@elephant:~/tmp/bench3/a>git annex p2p --pair
+	p2p pair peer1 (using Magic Wormhole) 
+	
+	This repository's pairing code is: 1-select-bluebird
+	
+	Enter the other repository's pairing code: (here I entered 8-fascinate-sawdust) 
+	Exchanging pairing data...
+	Successfully exchanged pairing data. Connecting to peer1...
+	ok
+
+And just that simply, the two repositories find one another, 
+Tor onion addresses and authentication data is exchanged, and a git remote
+is set up connecting via Tor.
+
+	joey@elephant:~/tmp/bench3/a>git annex sync peer1
+	commit  
+	ok
+	pull peer1 
+	warning: no common commits
+	remote: Counting objects: 5, done.
+	remote: Compressing objects: 100% (3/3), done.
+	remote: Total 5 (delta 0), reused 0 (delta 0)
+	Unpacking objects: 100% (5/5), done.
+	From tor-annex::5vkpoyz723otbmzo.onion:61900
+	 * [new branch]      git-annex  -> peer1/git-annex
+
+Very pleased with this, and also the whole thing worked on the very first
+try!
+
+It might be slightly annoying to have to exchange two codes during pairing.
+It would be possible to make this work with only one code. I decided to go
+with two codes, even though it's only marginally more secure than one,
+mostly for UI reasons. The pairing interface and 
+[[instructions for using it|tips/peer_to_peer_network_with_tor]] is simplfied
+by being symmetric.
+
+(I also decided to revert the work I did on Friday to make `p2p --link`
+set up a bidirectional link. Better to keep `--link` the simplest possible
+primitive, and pairing makes bidirectional links more easily.)
+
+Next: Some more testing of this and the Tor hidden services, a webapp UI
+for P2P peering, and then finally removing XMPP support. I hope to finish
+that by New Years.
+
+Today's work was sponsored by Jake Vosloo on Patreon.