diff options
author | 2013-03-19 19:02:56 -0400 | |
---|---|---|
committer | 2013-03-19 19:02:56 -0400 | |
commit | aef6d297681566c63a0c8b1b86dfb0f73b757669 (patch) | |
tree | 2b5357229df53a6b0bfd1130bfcda4201582d293 /doc | |
parent | 166145bcba1a75baf45192517b3a90609c96867c (diff) |
blog for the day
Diffstat (limited to 'doc')
-rw-r--r-- | doc/design/assistant/blog/day_220__performance.mdwn | 40 | ||||
-rw-r--r-- | doc/design/assistant/webapp.mdwn | 9 |
2 files changed, 49 insertions, 0 deletions
diff --git a/doc/design/assistant/blog/day_220__performance.mdwn b/doc/design/assistant/blog/day_220__performance.mdwn new file mode 100644 index 000000000..a1b7b329a --- /dev/null +++ b/doc/design/assistant/blog/day_220__performance.mdwn @@ -0,0 +1,40 @@ +I've been running some large transfers with the assistant, and looking at +ways to improve performance. (I also found and fixed a zombie process +leak.) + +---- + +One thing I noticed is that the assistant pushes changes to the git-annex +location log quite frequently during a batch transfer. If the files being +transferred are reasonably sized, it'll be pushing once per file transfer. +It would be good to reduce the number of pushes, but the pushes are +important in some network topologies to inform other nodes +when a file gets near to them, so they can get the file too. + +Need to see if I can find a smart way to avoid some of the pushes. +For example, if we've just downloaded a file, and are queuing uploads +of the file to a remote, we probably don't need to push the git-annex +branch to the remote. + +---- + +Another performance problem is that having the webapp open while transfers +are running uses significant CPU just for the browser to update the progress +bar. Unsurprising, since the webapp is sending the browser a new `<div>` +each time. Updating the DOM instead from javascript would avoid that; +the webapp just needs to send the javascript either a full `<div>` or a +changed percentage and quantity complete to update a single progress bar. + +I'd prefer to wait on doing that until I'm able to use Fay to generate +Javascript from Haskell, because it would be much more pleasant.. will see. + +---- + +Also a performance problem when performing lots of transfers, particularly +of small files, is that the assistant forks off a `git annex transferkey` +for each transfer, and that has to in turn start up several git commands. + +Today I have been working to change that, so the assistant maintains a +pool of transfer processes, and dispatches each transfer it wants to make +to a process from the pool. I just got all that to build, although untested +so far, in the `transferpools` branch. diff --git a/doc/design/assistant/webapp.mdwn b/doc/design/assistant/webapp.mdwn index 08e0a9b58..f9844e73c 100644 --- a/doc/design/assistant/webapp.mdwn +++ b/doc/design/assistant/webapp.mdwn @@ -1,5 +1,14 @@ The webapp is a web server that displays a shiny interface. +## performance + +Having the webapp open while transfers are +running uses significant CPU just for the browser to update the progress +bar. Unsurprising, since the webapp is sending the browser a new `<div>` +each time. Updating the DOM instead from javascript would avoid that; +the webapp just needs to send the javascript either a full `<div>` or a +changed percentage and quantity complete to update a single progress bar. + ## other features * there could be a UI to export a file, which would make it be served up |