diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/design/assistant/blog/day_87__more_progress_progress.mdwn | 28 | ||||
| -rw-r--r-- | doc/design/assistant/progressbars.mdwn | 29 |
2 files changed, 47 insertions, 10 deletions
diff --git a/doc/design/assistant/blog/day_87__more_progress_progress.mdwn b/doc/design/assistant/blog/day_87__more_progress_progress.mdwn new file mode 100644 index 000000000..c2b266d6d --- /dev/null +++ b/doc/design/assistant/blog/day_87__more_progress_progress.mdwn @@ -0,0 +1,28 @@ +Worked more on upload progress tracking. I'm fairly happy with its state +now: + +* It's fully implemented for rsync special remotes. + +* Git remotes also fully support it, with the + notable exception of file uploads run by `git-annex-shell recvkey`. That + runs `rsync --server --sender`, and in that mode, rsync refuses to output + progress info. Not sure what to do about this case. Maybe I should + write a parser for the rsync wire protocol that can tell what chunk of the + file is being sent, and shim it in front of the rsync server? That's + rather hardcore, but it seems the best of a bad grab bag of options that + include things like `LD_PRELOAD` hacks. + +* Also optimised the rsync progress bar reader to read whole + chunks of data rather than one byte at a time. + +* Also got progress bars to actually update in the webapp for uploads. + + This turned out to be tricky because kqueue cannot be used to detect when + existing files have been modified. (One of kqueue's worst shortcomings vs + inotify.) Currently on kqueue systems it has to poll. + +I will probably upload add progress tracking to the directory special remote, +which should be very easy (it already implements its own progress bars), +and leave the other special remotes for later. I can add upload progress +tracking to each special remote when I add support for configuring it in +the webapp. diff --git a/doc/design/assistant/progressbars.mdwn b/doc/design/assistant/progressbars.mdwn index ade9a8370..d3764e8dc 100644 --- a/doc/design/assistant/progressbars.mdwn +++ b/doc/design/assistant/progressbars.mdwn @@ -16,13 +16,22 @@ This is one of those potentially hidden but time consuming problems. ## uploads -Options: - -* Feed rsync output into a parser and parse out a progress value. Ugly, - failure prone, but potentially the least CPU-expensive option. -* Use librsync. Note: It's not wire-compatiable with the actual rsync - command. -* Set up a FIFO, have rsync read from or write to that, and the FIFO - feeder/reader then can update the transfer info. Generic enough to - work for most (all?) special remotes, but also the most expensive option, - involving another copy through memory of the whole file contents. +Each individual remote type needs to implement its own support for calling +the ProgressCallback as the upload progresses. + +* git: Done, with one exception: `git-annex-shell sendkey` runs `rsync + --server --sender` and in that mode it does not report progress info. + So downloads initiated by other repos do not show progress in the repo + doing the uploading. + + Maybe I should + write a proxy for the rsync wire protocol that can tell what chunk of the + file is being sent, and shim it in front of the rsync server? + +* rsync: **done** +* directory +* web: Not applicable; does not upload +* S3 +* bup +* hook: Would require the hook interface to somehow do this, which seems + too complicated. So skipping. |