From d7c8b60e6bbb2101a5146451b0900e50d211e600 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 20 Sep 2012 17:39:09 -0400 Subject: blog for the day --- .../blog/day_87__more_progress_progress.mdwn | 28 ++++++++++++++++++++++ doc/design/assistant/progressbars.mdwn | 7 +++++- 2 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 doc/design/assistant/blog/day_87__more_progress_progress.mdwn (limited to 'doc/design/assistant') diff --git a/doc/design/assistant/blog/day_87__more_progress_progress.mdwn b/doc/design/assistant/blog/day_87__more_progress_progress.mdwn new file mode 100644 index 000000000..c2b266d6d --- /dev/null +++ b/doc/design/assistant/blog/day_87__more_progress_progress.mdwn @@ -0,0 +1,28 @@ +Worked more on upload progress tracking. I'm fairly happy with its state +now: + +* It's fully implemented for rsync special remotes. + +* Git remotes also fully support it, with the + notable exception of file uploads run by `git-annex-shell recvkey`. That + runs `rsync --server --sender`, and in that mode, rsync refuses to output + progress info. Not sure what to do about this case. Maybe I should + write a parser for the rsync wire protocol that can tell what chunk of the + file is being sent, and shim it in front of the rsync server? That's + rather hardcore, but it seems the best of a bad grab bag of options that + include things like `LD_PRELOAD` hacks. + +* Also optimised the rsync progress bar reader to read whole + chunks of data rather than one byte at a time. + +* Also got progress bars to actually update in the webapp for uploads. + + This turned out to be tricky because kqueue cannot be used to detect when + existing files have been modified. (One of kqueue's worst shortcomings vs + inotify.) Currently on kqueue systems it has to poll. + +I will probably upload add progress tracking to the directory special remote, +which should be very easy (it already implements its own progress bars), +and leave the other special remotes for later. I can add upload progress +tracking to each special remote when I add support for configuring it in +the webapp. diff --git a/doc/design/assistant/progressbars.mdwn b/doc/design/assistant/progressbars.mdwn index 6b1521af6..d3764e8dc 100644 --- a/doc/design/assistant/progressbars.mdwn +++ b/doc/design/assistant/progressbars.mdwn @@ -22,7 +22,12 @@ the ProgressCallback as the upload progresses. * git: Done, with one exception: `git-annex-shell sendkey` runs `rsync --server --sender` and in that mode it does not report progress info. So downloads initiated by other repos do not show progress in the repo - doing the uploading. + doing the uploading. + + Maybe I should + write a proxy for the rsync wire protocol that can tell what chunk of the + file is being sent, and shim it in front of the rsync server? + * rsync: **done** * directory * web: Not applicable; does not upload -- cgit v1.2.3