diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/design/assistant/blog/day_84__deferred_downloads.mdwn | 33 | ||||
| -rw-r--r-- | doc/design/assistant/syncing.mdwn | 10 |
2 files changed, 42 insertions, 1 deletions
diff --git a/doc/design/assistant/blog/day_84__deferred_downloads.mdwn b/doc/design/assistant/blog/day_84__deferred_downloads.mdwn new file mode 100644 index 000000000..0ae684e47 --- /dev/null +++ b/doc/design/assistant/blog/day_84__deferred_downloads.mdwn @@ -0,0 +1,33 @@ +Implemented deferred downloads. So my example from yesterday of three +repositories in a line keep fully in sync now! + +I punted on one problem while doing it. It might be possible to get a really +big list of deferred downloads in some situation. That all lives in memory. +I aim for git-annex to always have a constant upper bound on memory use, +so that's not really acceptable. I have TODOed a reminder to do something +about limiting the size of this list. + +---- + +I also ran into a nasty crash while implementing this, where two threads +were trying to do things to git HEAD at the same time, and so one crashed, +and in a way I don't entirely understand, that crash took down another +thread with a BlockedIndefinitelyOnSTM exception. I think I've fixed +this, but it's bothersome that this is the second time that modifications +to the Merger thread have led to a concurrency related crash that I +have not fully understood. + +My guess is that STM can get confused when it's +retrying, and the thread that was preventing it from completing a +transaction crashes, because it suddenly does not see any other +references to the TVar(s) involved in the transaction. Any GHC STM gurus +out there? + +--- + +Still work to be done on making data transfers to keep fully in sync in all +circumstances. One case I've realized needs work occurs when a USB drive is +plugged in. Files are downloaded from it to keep the repo in sync, but the +repo neglects to queue uploads of those files it just got out to other +repositories it's in contact with. Seems I still need to do something to +detecting when a successful download is done, and queue uploads. diff --git a/doc/design/assistant/syncing.mdwn b/doc/design/assistant/syncing.mdwn index f874b9932..b9c6016b2 100644 --- a/doc/design/assistant/syncing.mdwn +++ b/doc/design/assistant/syncing.mdwn @@ -39,7 +39,11 @@ all the other git clones, at both the git level and the key/value level. Possible solution: C could record a download intent. (Similar to a failed download, but with an unknown source.) When C next receives a git-annex branch push, it could try to requeue downloads that it has such intents - registered for. + registered for. **done** + + Note that this solution won't cover use cases the other does. For example, + connect a USB drive A; B syncs files from it, and then should pass them to C. + If the files are not new, C won't immediatly request them from B. ## TODO @@ -81,6 +85,10 @@ all the other git clones, at both the git level and the key/value level. need to use the TransferScanner, if we get and check a list of the changed files. * [[use multiple transfer slots|todo/Slow_transfer_for_a_lot_of_small_files.]] +* The TransferQueue's list of deferred downloads could theoretically + grow without bounds in memory. Limit it to a given number of entries, + and fall back to some other method -- either storing deferred downloads + on disk, or perhaps scheduling a TransferScanner run to get back into sync. ## data syncing |