summaryrefslogtreecommitdiff
path: root/Utility
Commit message (Collapse)AuthorAge
* unify exception handling into Utility.ExceptionGravatar Joey Hess2014-08-07
| | | | | | | | | | | | | | | | | | | | Removed old extensible-exceptions, only needed for very old ghc. Made webdav use Utility.Exception, to work after some changes in DAV's exception handling. Removed Annex.Exception. Mostly this was trivial, but note that tryAnnex is replaced with tryNonAsync and catchAnnex replaced with catchNonAsync. In theory that could be a behavior change, since the former caught all exceptions, and the latter don't catch async exceptions. However, in practice, nothing in the Annex monad uses async exceptions. Grepping for throwTo and killThread only find stuff in the assistant, which does not seem related. Command.Add.undo is changed to accept a SomeException, and things that use it for rollback now catch non-async exceptions, rather than only IOExceptions.
* generalized using the extensions packageGravatar Joey Hess2014-08-07
|
* move ugly rsync zombie workaroundGravatar Joey Hess2014-08-03
| | | | | | | | | | | | This reaping of any processes came to cause me problems when redoing the rsync special remote -- a gpg process that was running gets waited on and the place that then checks its return code fails. I cannot reproduce any zombies when using the rsync special remote. But I still can when using a normal git remote, accessed over ssh. There is 1 zombie per file downloaded without this horrible hack enabled. So, move the hack to only be used in that case.
* testremote: New command to test uploads/downloads to a remote.Gravatar Joey Hess2014-08-01
| | | | | | | | | This only performs some basic tests so far; no testing of chunking or resuming. Also, the existing encryption type of the remote is used; it would be good later to derive an encrypted and a non-encrypted version of the remote and test them both. This commit was sponsored by Joseph Liu.
* lift types from IO to AnnexGravatar Joey Hess2014-07-29
| | | | | | | | | | | Some remotes like External need to run store and retrieve actions in Annex, not IO. In order to do that lift, I had to dive pretty deep into the utilities, making Utility.Gpg and Utility.Tmp be partly converted to using MonadIO, and Control.Monad.Catch for exception handling. There should be no behavior changes in this commit. This commit was sponsored by Michael Barabanov.
* resume interrupted chunked downloadsGravatar Joey Hess2014-07-27
| | | | | | | | | | | | | | | | | | Leverage the new chunked remotes to automatically resume downloads. Sort of like rsync, although of course not as efficient since this needs to start at a chunk boundry. But, unlike rsync, this method will work for S3, WebDAV, external special remotes, etc, etc. Only directory special remotes so far, but many more soon! This implementation will also properly handle starting a download from one remote, interrupting, and resuming from another one, and so on. (Resuming interrupted chunked uploads is similarly doable, although slightly more expensive.) This commit was sponsored by Thomas Djärv.
* core implementation of new style chunkingGravatar Joey Hess2014-07-25
| | | | | | | | | | | | | | | | Not yet used by any special remotes, but should not be too hard to add it to most of them. storeChunks is the hairy bit! It's loosely based on Remote.Directory.storeLegacyChunked. The object is read in using a lazy bytestring, which is streamed though, creating chunks as needed, without ever buffering more than 1 chunk in memory. Getting the progress meter update to work right was also fun, since progress meter values are absolute. Finessed by constructing an offset meter. This commit was sponsored by Richard Collins.
* increase sleepGravatar Joey Hess2014-07-16
|
* Windows: fix crash after 10 minutesGravatar Joey Hess2014-07-16
| | | | | getLine in waitForTermination doesn't work when stdin is closed.. Just loop forever, there was no reason to getLine here I think.
* Windows: Fix locking issue that prevented the webapp starting (since ↵Gravatar Joey Hess2014-07-14
| | | | | | | | 5.20140707). Reversion introduced in 89c188fac37c20c40b0a9dabeb35403cfa4e4f52. The locking code was wrong; the webapp re-ran itself, saw pid was locked, and so didn't start!
* deal with process-1.2.0.0 throwing an exception when program DNE (may only ↵Gravatar Joey Hess2014-07-13
| | | | | | | | | | | | | | | fix the tip of the iceberg) Configure crashed on systems with that process and without eg, sha256sum. The rest of the code in configure looks to work ok, since it uses sh -c to probe for commands, and sh is always in path so it works. Dunno about all the rest of git-annex. Not a huge amount of external program use, other than git, so perhaps this won't be a large pain. Note that boolSystem can throw an exception now if the program doesn't exist. Could easily be changed back to False.
* refactorGravatar Joey Hess2014-07-10
|
* Fix minor FD leak in journal code.Gravatar Joey Hess2014-07-09
| | | | | | | | | | | | | | | | | | | | | | | | Minor because normally only 1 FD is leaked per git-annex run. However, the test suite leaks a few hundred FDs, and this broke it on the Debian autobuilders, which seem to have a tigher than usual ulimit. The leak was introduced by the lazy getDirectoryContents' that was introduced in b54de1dad4874b7561d2c5a345954b6b5c594078 in order to scale to millions of journal files -- if the lazy list was never fully consumed, the directory handle did not get closed. Instead, pull in openDirectory/readDirectory/closeDirectory code that I already developed and submitted in a patch to the haskell directory library earlier. Using this in journalDirty avoids the place that the lazy list caused a problem. And using it in stageJournal eliminates the need for getDirectoryContents'. The getJournalFiles* functions are switched back to using the regular strict getDirectoryContents. I'm not sure if those always consume the whole list, so this avoids any leak. And the things that call those are things like git annex unused, which also look at every file committed to the git-annex branch, so would need more work to scale to insane numbers of files anyway.
* Support building with bloomfilter 2.0.0.Gravatar Joey Hess2014-07-07
|
* fix windows buildGravatar Joey Hess2014-07-05
|
* work around getDirectoryContents not streaming lazilyGravatar Joey Hess2014-07-04
|
* Windows: Assistant now logs to daemon.log.Gravatar Joey Hess2014-06-17
| | | | | | | Yes, this means that git annex webapp on windows execs git-annex, which execs itself to set env, and the execs itself again to redirect logs. This is disgusting. This is Windows(TM).
* finish fixing windows timezone madnessGravatar Joey Hess2014-06-12
| | | | | | | | | | | | | | | | | | | | | Rather than calculating the TSDelta once, and caching it, this now reads the inode sential file's InodeCache file once, and then each time a new InodeCache is generated, looks at the sentinal file to get the current delta. This way, if the time zone changes while git-annex is running, it will adapt. This adds some inneffiency, but only on Windows, and only 1 stat per new file added. The worst innefficiency is that `git annex status` and `git annex sync` will now (on Windows) stat the inode sentinal file once per file in the repo. It would be more efficient to use getCurrentTimeZone, rather than needing to stat the sentinal file. This should be easy to do, once the time package gets my bugfix patch. This commit was sponsored by Jürgen Lüters.
* fix for Windows file timestamp timezone madnessGravatar Joey Hess2014-06-12
| | | | | | | | | | | | | | | | | | | | | | On Windows, changing the time zone causes the apparent mtime of files to change. This confuses git-annex, which natually thinks this means the files have actually been modified (since THAT'S WHAT A MTIME IS FOR, BILL <sheesh>). Work around this stupidity, by using the inode sentinal file to detect if the timezone has changed, and calculate a TSDelta, which will be applied when generating InodeCaches. This should add no overhead at all on unix. Indeed, I sped up a few things slightly in the refactoring. Seems to basically work! But it has a big known problem: If the timezone changes while the assistant (or a long-running command) runs, it won't notice, since it only checks the inode cache once, and so will use the old delta for all new inode caches it generates for new files it's added. Which will result in them seeming changed the next time it runs. This commit was sponsored by Vincent Demeester.
* minimal exportsGravatar Joey Hess2014-06-11
|
* deal with FAT on Linux timestamp issueGravatar Joey Hess2014-06-11
| | | | | | | | | Deal with FAT's low resolution timestamps, which in combination with Linux's caching of higher res timestamps while a FAT is mounted, caused direct mode repositories on FAT to seem to have modified files after they were unmounted and remounted. This commit was sponsored by Fabrice Rossi.
* Fix build with wai 0.3.0.Gravatar Joey Hess2014-06-11
| | | | | | | | | | This version of wai changed the type of Middleware, so I cannot seem to liftIO inside it. So, got rid of a lot of not really needed complexity to use System.Log.Logger's logging stuff, and just use the standard wai stdout logger when debug logging is enabled. Format may change some, and it logs http to stdout instead of stderr now. Doesn't matter for the webapp since both go to the same log anyway.
* export CreateProcess fields from Utility.ProcessGravatar Joey Hess2014-06-10
| | | | update code to avoid cwd and env redefinition warnings
* Windows: Fix opening webapp when repository is in a directory with spaces in ↵Gravatar Joey Hess2014-06-10
| | | | the path.
* Avoid leaving behind .tmp files when failing in some cases, including ↵Gravatar Joey Hess2014-06-09
| | | | importing files to a disk that is full.
* Ignore setsid failures.Gravatar Joey Hess2014-06-09
|
* qualify commentGravatar Joey Hess2014-06-09
|
* fix build with old versions of warpGravatar Joey Hess2014-06-04
|
* lazy bytestring fromStrict is not available in stable; use fromChunks insteadGravatar Joey Hess2014-05-29
|
* fix warning messageGravatar Joey Hess2014-05-29
| | | | This should work even back in debian stable's warp
* get rid of (completely safe) uses of Char8Gravatar Joey Hess2014-05-27
| | | | | Char8 often indicates an encoding bug. It didn't here, but I can avoid it and not worry about it.
* refactorGravatar Joey Hess2014-05-14
|
* fix windows buildGravatar Joey Hess2014-05-14
|
* setsid when running webapp in foreground tooGravatar Joey Hess2014-05-14
| | | | | | | | This avoids ssh prompting for passwords on stdin, ever. It may also change other behavior of other programs, as there is no controlling terminal now. However, setsid was already done when running the assistant in daemon mode, so any behavior changes should not be really new.
* relicense general utility library code to BSDGravatar Joey Hess2014-05-10
| | | | | Omitted a couple of files what have had significant contributions from others.
* Merge branch 'master' into bootstrap3Gravatar Joey Hess2014-05-02
|\ | | | | | | | | Conflicts: debian/changelog
| * hlintGravatar Joey Hess2014-04-26
| |
* | create a local copy of Yesod.Form.Bootstrap3Gravatar Sören Brunk2014-04-25
| |
* | bootstrap3 formsGravatar Sören Brunk2014-04-21
|/
* better approach for building distributionupdateGravatar Joey Hess2014-04-21
| | | | Use cabal include file
* fix distributionupdate buildGravatar Joey Hess2014-04-21
| | | | Including avoiding needing cabal's defines for Utility.URI
* Merge branch 'master' into remotecontrolGravatar Joey Hess2014-04-14
|\ | | | | | | | | Conflicts: doc/devblog/day_152__more_ssh_connection_caching.mdwn
| * adjust to not use cpp in modules used by configureGravatar Joey Hess2014-04-14
| |
| * add missing Network.URI Ord instance for Debian stableGravatar Joey Hess2014-04-14
| |
* | sync, assistant, remotedaemon: Use ssh connection caching for git pushes and ↵Gravatar Joey Hess2014-04-12
|/ | | | | | | | | | | | | | | | | pulls. For sync, saves 1 ssh connection per remote. For remotedaemon, the same ssh connection that is already open to run git-annex-shell notifychanges is reused to pull from the remote. Only potential problem is that this also enables connection caching when the assistant syncs with a ssh remote. Including the sync it does when a network connection has just come up. In that case, cached ssh connections are likely to be stale, and so using them would hang. Until I'm sure such problems have been dealt with, this commit needs to stay on the remotecontrol branch, and not be merged to master. This commit was sponsored by Alexandre Dupas.
* wrote test case; found bugs in date math; fixed and simplified using ↵Gravatar Joey Hess2014-04-12
| | | | Data.Time.Calendar
* Improve handling on monthly/yearly scheduling.Gravatar Joey Hess2014-04-12
| | | | Avoid back-to-back runs.
* Improve handling on monthly/yearly scheduling.Gravatar Joey Hess2014-04-11
| | | | | | | | | | | | | | | | | | | Code was still buggy, it turns out (though the recursion checker caught it). In the case of (Schedule (Monthly Nothing) AnyTime), where the last run was on yyyy-12-31, it looped forever. Also, the handling of (Schedule (Yearly Nothing) AnyTime) was wacky where the last run was yyyy-12-31. It would suggest a window starting on the 3rd for the next run (because 31 mod 28 is 3). I think that originally I was wanted to avoid running on 01-01 if it had just run on 12-31. But the code didn't accomplish this, and it's not necessary anyway. This is supposed to calculate the next window meeting the schedule, and for (Schedule (Monthly Nothing), the window starts at 01-01 and runs through 01-31. If that causes two back-to-back runs, well the next one will not be until 02-01 at the earliest. Also, back-to-back runs can be avoided, if desired, by using Divisible 2.
* guard against any remaining infinite recursion bugs; throw error if no ↵Gravatar Joey Hess2014-04-11
| | | | | | | | candidate cn be found in next hundred years Note that the exception thrown is not visible in the webapp currently because it crashes one of Cronner's 2 worker threads, which is never checked.
* Last release didn't quite fix the high cpu issue in all cases, this should.Gravatar Joey Hess2014-04-11
| | | | | | | This is supposed to look for a day past the last day it ran, not a month past. Seems to work, at least in anarcat's test case.