diff options
| -rw-r--r-- | doc/design/assistant.mdwn | 5 | ||||
| -rw-r--r-- | doc/design/assistant/rate_limiting.mdwn | 57 | ||||
| -rw-r--r-- | doc/design/assistant/syncing.mdwn | 5 |
3 files changed, 60 insertions, 7 deletions
diff --git a/doc/design/assistant.mdwn b/doc/design/assistant.mdwn index b89231361..aab4b7544 100644 --- a/doc/design/assistant.mdwn +++ b/doc/design/assistant.mdwn @@ -11,10 +11,10 @@ and use cases to add. Feel free to chip in with comments! --[[Joey]] * Month 4 "cloud": [[!traillink cloud]] [[!traillink transfer_control]] * Month 5 "cloud continued": [[!traillink xmpp]] [[!traillink more_cloud_providers]] * Month 6 "9k bonus round": [[!traillink desymlink]] -* Month 7: user-driven features and polishing; +* Month 7: user-driven features and polishing; [presentation at LCA2013](http://mirror.linux.org.au/linux.conf.au/2013/mp4/gitannex.mp4) * Month 8: [[!traillink Android]] -* Month 9: user-driven features and polishing +* Month 9: [[screencasts|videos]] and polishing We are, approximately, here: @@ -29,6 +29,7 @@ We are, approximately, here: ## not yet on the map: +* [[rate_limiting]] * [[partial_content]] * encrypted git remotes using [git-remote-gcrypt](https://github.com/blake2-ppc/git-remote-gcrypt) * [[deltas]] diff --git a/doc/design/assistant/rate_limiting.mdwn b/doc/design/assistant/rate_limiting.mdwn new file mode 100644 index 000000000..3ab804329 --- /dev/null +++ b/doc/design/assistant/rate_limiting.mdwn @@ -0,0 +1,57 @@ +Webapp needs a simple speed control knob, especially to avoid saturating +bandwidth on uploads. + +We have basic throttling support in git-annex for rsync, +but none for any special remotes. A good first step would be to expose +this in the webapp, and ensure that `git-annex-shell` also honors it when +sending/receiving data. + +We actually need two speed controls, one for upload and one for download. + +It is probably not necessary to throttle git push/pull operations, as the +data transfers tend to be small. Only throttling file transfers is +essential. + +## possibility: trickle + +Since `git-annex transferkeys` is a separate process, one easy option would +be to run it inside `trickle`. If the user changes the bandwidth limits, +it could kill the transfer and restart it with different trickle options. + +Problem: Not all special remotes support resuming transfers, so this is +suboptimal. (So too are the pause/resume buttons, when using those +remotes!) + +`trickle` is available for OSX as well as Linux and BSDs. +<http://hints.macworld.com/comment.php?mode=view&cid=65362> +<http://mac.softpedia.com/get/Network-Admin/Trickle.shtml> +It is probably not easily available for Android, as it uses `LD_PRELOAD`. + +## possibility: built in IO limiting + +A cleaner method would be to do the limiting inside git-annex. We already +have metered file IO. It should be possible to make the meter not only report +on the transfer speed, but detect when it's going too fast, and delay. This +will delay the file streaming through the special remote's transfer code, +so should work for a variety of special remotes. (Not for rsync or bup +or git-annex-shell though, so those need to be handled separately.) + +Should work well for uploads at least. I don't know how well it would work +for throttling downloads; the sender may just keep sending data and the +data buffer before it gets to the IO meter. Maybe once the buffers fill the +OS would have the TCP throttled down. Needs investigation; trickle claims +to throttle downloads. + +## communications channels + +There would need to be a communication channel for the assistant to tell +`git annex transferkeys` when the rate limit has changed. It could for +example send it a SIGUSR1, and then leave it up to the process to reload +the git config. Inside the IO meter, we could have an MVar that contains +the current throttle value, so the IO meter could check it each time it's +called and adjust its throttling appropriately. + +Ideally, the assistant could also communicate in the same way with +`git-annex-shell` to tell it when the limit has changed. Since +`git-annex-shell` uses rsync, it would need to abort the transfer, and rely +on the other side retrying to start it up with the new limit. diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn index d27a50bec..5b2a11aa6 100644 --- a/doc/design/assistant/syncing.mdwn +++ b/doc/design/assistant/syncing.mdwn @@ -25,11 +25,6 @@ syncs? ## TODO * Test MountWatcher on LXDE. -* git-annex needs a simple speed control knob, which can be plumbed - through to, at least, rsync. A good job for an hour in an - airport somewhere. -* Find a way to probe available outgoing bandwidth, to throttle so - we don't bufferbloat the network to death. * Add a hook, so when there's a change to sync, a program can be run and do its own signaling. * --debug will show often unnecessary work being done. Optimise. |