aboutsummaryrefslogtreecommitdiff
path: root/Assistant/TransferQueue.hs
Commit message (Collapse)AuthorAge
* Deal with the MonadFail proposalGravatar Benjamin Barenblat2022-01-19
| | | | | | | | | | base-4.9 split MonadFail from Monad. Strengthen some type signatures to require MonadFail instead of just Monad, derive MonadFail in relevant places, and change a partial pattern match inside STM to one that explicitly calls error. (STM is not a MonadFail; the user must explicitly specify the desired semantics if a pattern match doesn’t work out. In this case, the failing branch of the pattern should never be reached, so crashing is fine.)
* Eliminate Data.Map.insertWith'Gravatar Benjamin Barenblat2022-01-19
| | | | | containers-0.6 removed insertWith' in favor of the Data.Map.Strict API. Switch to the new API where appropriate.
* first pass at assistant knowing about export remotesGravatar Joey Hess2017-09-20
| | | | | | | | | | | | | | | Split exportRemotes out from syncDataRemotes; the parts of the assistant that upload keys and drop keys from remotes don't apply to exports, because those operations are not supported. Some parts of the assistant and webapp do operate on both syncDataRemotes and exportRemotes. Particularly when downloading from either of them. Added a downloadRemotes that combines both. With this, the assistant should download from exports, but it won't yet upload changes to them. This commit was sponsored by Fernando Jimenez on Patreon.
* get, move, copy, mirror: Added --failed switch which retries failed copies/movesGravatar Joey Hess2016-08-03
| | | | | | | | | Note that get --from foo --failed will get things that a previous get --from bar tried and failed to get, etc. I considered making --failed only retry transfers from the same remote, but it was easier, and seems more useful, to not have the same remote requirement. Noisy due to some refactoring into Types/
* update my email address and homepage urlGravatar Joey Hess2015-01-21
|
* fix some mixed space+tab indentationGravatar Joey Hess2014-10-09
| | | | | | | | | This fixes all instances of " \t" in the code base. Most common case seems to be after a "where" line; probably vim copied the two space layout of that line. Done as a background task while listening to episode 2 of the Type Theory podcast.
* try to drop unused object if it does not need to be transferred anywhereGravatar Joey Hess2014-01-23
|
* fix transfers of key with no associated fileGravatar Joey Hess2014-01-23
| | | | | | | | | | | | | | | | | Several places assumed this would not happen, and when the AssociatedFile was Nothing, did nothing. As part of this, preferred content checks pass the Key around. Note that checkMatcher is sometimes now called with Just Key and Just File. It currently constructs a FileMatcher, ignoring the Key. However, if it constructed a FileKeyMatcher, which contained both, then it might be possible to speed up parts of Limit, which currently call the somewhat expensive lookupFileKey to get the Key. I have not made this optimisation yet, because I am not sure if the key is always the same. Will need some significant checking to satisfy myself that's the case..
* assistant unused file handlingGravatar Joey Hess2014-01-22
| | | | | | | | | | | | | | | | | | | Make sanity checker run git annex unused daily, and queue up transfers of unused files to any remotes that will have them. The transfer retrying code works for us here, so eg when a backup disk remote is plugged in, any transfers to it are done. Once the unused files reach a remote, they'll be removed locally as unwanted. If the setup does not cause unused files to go to a remote, they'll pile up, and the sanity checker detects this using some heuristics that are pretty good -- 1000 unused files, or 10% of disk used by unused files, or more disk wasted by unused files than is left free. Once it detects this, it pops up an alert in the webapp, with a button to take action. TODO: Webapp UI to configure this, and also the ability to launch an immediate cleanup of all unused files. This commit was sponsored by Simon Michael.
* revert use of Data.Map.StrictGravatar Joey Hess2014-01-07
| | | | | memory profile shows this did not contribute to the memory leaks fixed in 4cf6d95c1a9d10cb59669eaceafce4c7a3155eb6
* assistant: Fixed several minor memory leaks that manifested when adding a ↵Gravatar Joey Hess2014-01-05
| | | | large number of files.
* use DList for the transfer queueGravatar Joey Hess2013-04-25
| | | | | Some nice efficiency gains here for list appending, although mostly the small size of the transfer queue makes them irrelivant.
* use a DList for the deferred downloads queueGravatar Joey Hess2013-04-25
|
* avoid queuing transfers that are currently runningGravatar Joey Hess2013-04-02
|
* avoid queuing transfer that is already queued, with a different associated fileGravatar Joey Hess2013-04-02
|
* avoid queueing uploads to remotes that already have the contentGravatar Joey Hess2013-04-02
|
* tweakGravatar Joey Hess2013-03-13
|
* logic errorGravatar Joey Hess2013-03-07
|
* avoid queuing transfers that are already queuedGravatar Joey Hess2013-03-07
| | | | | | | | I saw this happen in real life, when syncing to a newly added usb drive. I think it got scanned twice, and files were doubled in the queue. This could be optimised a little bit more, to only read from the mvar once, rather than twice.
* add additional debug info about reasons for transfersGravatar Joey Hess2013-03-01
|
* --auto fixesGravatar Joey Hess2012-12-06
| | | | | | | * get/copy --auto: Transfer data even if it would exceed numcopies, when preferred content settings want it. * drop --auto: Fix dropping content when there are no preferred content settings.
* don't try to transfer data to/from XMPP remotesGravatar Joey Hess2012-11-11
| | | | | Partition syncRemotes into ones needing git sync (ie, non-special remotes), and ones needing data sync (ie, non-XMPP remotes).
* finished pushing Assistant monad into all relevant filesGravatar Joey Hess2012-10-30
| | | | All temporary and old functions are removed.
* pushed Assistant monad down into DaemonStatus codeGravatar Joey Hess2012-10-30
| | | | | | Currently have three old versions of functions that more reworking is needed to remove: getDaemonStatusOld, modifyDaemonStatusOld_, and modifyDaemonStatusOld
* cleanup daemonStatus accessorsGravatar Joey Hess2012-10-30
|
* split remaining assistant typesGravatar Joey Hess2012-10-30
|
* split out daemonstatus typesGravatar Joey Hess2012-10-30
|
* reorderGravatar Joey Hess2012-10-14
| | | | get list of remotes after, rather than before, a potentially blocking action
* better variable nameGravatar Joey Hess2012-10-14
|
* assistant: Now honors preferred content settings when deciding what to transfer.Gravatar Joey Hess2012-10-09
| | | | | | | | | Both when queueing downloads, and uploads, consults the preferred content settings. I didn't make it check yet when requeing failed transfers or queuing deferred downloads; dealing with the preferred content settings (or indeed, other settings) changing while the assistant is running still needs work.
* make other repositories list list all autostarted reposGravatar Joey Hess2012-09-18
| | | | And add a form to add another, unrelated repository
* avoid sending uploads right back to where the download came fromGravatar Joey Hess2012-09-18
| | | | Just an optimisation.
* deferred downloadsGravatar Joey Hess2012-09-17
| | | | | | | | | | | | | Now when a download is queued and there's no known remote to get it from, it's added to a deferred download list, which will be retried later. The Merger thread tries to queue any deferred downloads when it receives a push to the git-annex branch. Note that the Merger thread now also forces an update of the git-annex branch. The assistant was not updating this branch before, and it saw a (mostly) correct view of state, but now that incoming pushes go to synced/git-annex, it needs to be merged in.
* hlintGravatar Joey Hess2012-09-13
|
* remove the TChan component from the data structureGravatar Joey Hess2012-08-31
| | | | | | | | | The code to maintain that TChan in parallel with the list was buggy, the two were not always the same. And all that TChan was needed for was blocking on the next transfer, which can be accomplished just as well by checking the size and retrying, thanks to STM. Also, this is faster, and uses less memory. Total win.
* make start button work on queued transfersGravatar Joey Hess2012-08-29
| | | | | | When multiple downloads of a key are queued, it starts the first, but leaves the other downloads in the queue. This ensures that we don't lose a queued download if the one that got started failed.
* more genericGravatar Joey Hess2012-08-29
|
* forgot to update the tchan when dequeuing transfersGravatar Joey Hess2012-08-29
|
* when canceling a transfer, also cancel all other downloads of the same keyGravatar Joey Hess2012-08-29
|
* add support for readonly remotesGravatar Joey Hess2012-08-26
| | | | | | | Currently only the web special remote is readonly, but it'd be possible to also have readonly drives, or other remotes. These are handled in the assistant by only downloading from them, and never trying to upload to them.
* keep logs of failed transfers, and requeue them when doing a non-full scanGravatar Joey Hess2012-08-23
| | | | of a remote
* implement pausing of transfersGravatar Joey Hess2012-08-10
| | | | | | A paused transfer's thread keeps running, keeping the slot in use. This is intentional; pausing a transfer should not let other queued transfers to run in its place.
* send update notificaton when removing a queued transferGravatar Joey Hess2012-08-10
|
* transfer cancelingGravatar Joey Hess2012-08-08
| | | | | | | | | | Should work (untested) for transfers being run by other processes. Not yet by transfers being run by the assistant. killThread does not kill processes forked off by a thread. To fix this, will probably need to make `git annex getkey` and `git annex sendkey` commands that operate on keys, and write their own transfer info. Then the assistant can run them, and kill them, as needed.
* fix bug in transfer initiation checkingGravatar Joey Hess2012-07-29
| | | | | | | Putting the transfer on the currentTransfers atomically introduced a bug: It checks to see if the transfer is in progress, and cancels it. Fixed by moving that check inside the STM transaction.
* refactorGravatar Joey Hess2012-07-29
|
* send notifications when the TransferQueue is changedGravatar Joey Hess2012-07-28
| | | | | | The fun part was making it move things from TransferQueue to currentTransfers entirely atomically. Which will avoid inconsistent display if the WebApp renders the current status at just the wrong time. STM to the rescue!
* move DaemonStatus manipulation out of the Annex monad to IOGravatar Joey Hess2012-07-28
| | | | | | I've convinced myself that nothing in DaemonStatus can deadlock, as it always keepts the TMVar full. That was the only reason it was in the Annex monad.
* webapp now displays the real running and queued transfersGravatar Joey Hess2012-07-27
| | | | yowza!!!
* laziness fixGravatar Joey Hess2012-07-25
| | | | Now scanning runs fully interleaved with transferring.
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-09 21:27:44 +0000