summaryrefslogtreecommitdiff
path: root/Remote/WebDAV.hs
Commit message (Collapse)AuthorAge
* update my email address and homepage urlGravatar Joey Hess2015-01-21
|
* reformatGravatar Joey Hess2014-12-16
|
* Expand checkurl to support recommended filename, and multi-file-urlsGravatar Joey Hess2014-12-11
| | | | This commit was sponsored by an anonymous bitcoiner.
* Urls can now be claimed by remotes. This will allow creating, for example, a ↵Gravatar Joey Hess2014-12-08
| | | | external special remote that handles magnet: and *.torrent urls.
* add stub claimUrlGravatar Joey Hess2014-12-08
|
* include creds info for glacier and webdavGravatar Joey Hess2014-10-22
| | | | | | | | That and S3 are all that uses creds currently, except that external remotes can use creds. I have not handled showing info about external remote creds because they can have 0, 1, or more separate cred pairs, and there's no way for info to enumerate them or know how they're used. So it seems ok to leave out creds info for external remotes.
* add per-remote-type infoGravatar Joey Hess2014-10-21
| | | | | | | | | | Now `git annex info $remote` shows info specific to the type of the remote, for example, it shows the rsync url. Remote types that support encryption or chunking also include that in their info. This commit was sponsored by Ævar Arnfjörð Bjarmason.
* glacier, S3: Fix bug that caused embedded creds to not be encypted using the ↵Gravatar Joey Hess2014-09-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | remote's key. encryptionSetup must be called before setRemoteCredPair. Otherwise, the RemoteConfig doesn't have the cipher in it, and so no cipher is used to encrypt the embedded creds. This is a security fix for non-shared encryption methods! For encryption=shared, there's no security problem, just an inconsistentency in whether the embedded creds are encrypted. This is very important to get right, so used some types to help ensure that setRemoteCredPair is only run after encryptionSetup. Note that the external special remote bypasses the type safety, since creds can be set after the initial remote config, if the external special remote program requests it. Also note that IA remotes never use encryption, so encryptionSetup is not run for them at all, and again the type safety is bypassed. This leaves two open questions: 1. What to do about S3 and glacier remotes that were set up using encryption=pubkey/hybrid with embedcreds? Such a git repo has a security hole embedded in it, and this needs to be communicated to the user. Is the changelog enough? 2. enableremote won't work in such a repo, because git-annex will try to decrypt the embedded creds, which are not encrypted, so fails. This needs to be dealt with, especially for ecryption=shared repos, which are not really broken, just inconsistently configured. Noticing that problem for encryption=shared is what led to commit cc54ff9e49260cd94f938e69e926a273e231ef4e, which tried to fix the problem by not decrypting the embedded creds. This commit was sponsored by Josh Taylor.
* WebDav: Fix enableremote crash when the remote already exists. (Bug ↵Gravatar Joey Hess2014-09-17
| | | | introduced in version 5.20140817.)
* testremote: Add testing of behavior when remote is not availableGravatar Joey Hess2014-08-10
| | | | | | | | | | | | | | | | | | | | Added a mkUnavailable method, which a Remote can use to generate a version of itself that is not available. Implemented for several, but not yet all remotes. This allows testing that checkPresent properly throws an exceptions when it cannot check if a key is present or not. It also allows testing that the other methods don't throw exceptions in these circumstances. This immediately found several bugs, which this commit also fixes! * git remotes using ssh accidentially had checkPresent return an exception, rather than throwing it * The chunking code accidentially returned False rather than propigating an exception when there were no chunks and checkPresent threw an exception for the non-chunked key. This commit was sponsored by Carlo Matteo Capocasa.
* avoid printing really ugly webdav exceptionsGravatar Joey Hess2014-08-09
| | | | | The responseheaders can sometimes include the entire input request, which is several pages of garbage.
* fix removeKey when not presentGravatar Joey Hess2014-08-08
|
* show missing url= parameter error soonerGravatar Joey Hess2014-08-08
|
* WebDAV: Avoid buffering whole file in memory when downloading.Gravatar Joey Hess2014-08-08
| | | | | | httpBodyRetriever will later also be used by S3 This commit was sponsored by Ethan Aubin.
* further break out legacy chunking codeGravatar Joey Hess2014-08-08
|
* unify exception handling into Utility.ExceptionGravatar Joey Hess2014-08-07
| | | | | | | | | | | | | | | | | | | | Removed old extensible-exceptions, only needed for very old ghc. Made webdav use Utility.Exception, to work after some changes in DAV's exception handling. Removed Annex.Exception. Mostly this was trivial, but note that tryAnnex is replaced with tryNonAsync and catchAnnex replaced with catchNonAsync. In theory that could be a behavior change, since the former caught all exceptions, and the latter don't catch async exceptions. However, in practice, nothing in the Annex monad uses async exceptions. Grepping for throwTo and killThread only find stuff in the assistant, which does not seem related. Command.Add.undo is changed to accept a SomeException, and things that use it for rollback now catch non-async exceptions, rather than only IOExceptions.
* WebDAV: Avoid buffering whole file in memory when uploading.Gravatar Joey Hess2014-08-07
| | | | | | The httpStorer will later also be used by S3. This commit was sponsored by Torbjørn Thorsen.
* webdav: reuse http connection when operating on the chunks of a fileGravatar Joey Hess2014-08-07
| | | | | | | | For both new and legacy chunks. Massive speed up! This commit was sponsored by Dominik Wagenknecht.
* use DAV monadGravatar Joey Hess2014-08-07
| | | | | | | | | | | | This speeds up the webdav special remote somewhat, since it often now groups actions together in a single http connection when eg, storing a file. Legacy chunks are still supported, but have not been sped up. This depends on a as-yet unreleased version of DAV. This commit was sponsored by Thomas Hochstein.
* convert WebDAV to new special remote interface, adding new-style chunking ↵Gravatar Joey Hess2014-08-06
| | | | | | | | | | | | | | | | | | | | | | | support Reusing http connection when operating on chunks is not done yet, I had to submit some patches to DAV to support that. However, this is no slower than old-style chunking was. Note that it's a fileRetriever and a fileStorer, despite DAV using bytestrings that would allow streaming. As a result, upload/download of encrypted files is made a bit more expensive, since it spools them to temp files. This was needed to get the progress meters to work. There are probably ways to avoid that.. But it turns out that the current DAV interface buffers the whole file content in memory, and I have sent in a patch to DAV to improve its interfaces. Using the new interfaces, it's certainly going to need to be a fileStorer, in order to read the file size from the file (getting the size of a bytestring would destroy laziness). It should be possible to use the new interface to make it be a byteRetriever, so I'll change that when I get to it. This commit was sponsored by Andreas Olsson.
* pushed checkPresent exception handling out of Remote implementationsGravatar Joey Hess2014-08-06
| | | | | | | | | | | | | | | | I tend to prefer moving toward explicit exception handling, not away from it, but in this case, I think there are good reasons to let checkPresent throw exceptions: 1. They can all be caught in one place (Remote.hasKey), and we know every possible exception is caught there now, which we didn't before. 2. It simplified the code of the Remotes. I think it makes sense for Remotes to be able to be implemented without needing to worry about catching exceptions inside them. (Mostly.) 3. Types.StoreRetrieve.Preparer can only work on things that return a Bool, which all the other relevant remote methods already did. I do not see a good way to generalize that type; my previous attempts failed miserably.
* roll ChunkedEncryptable into Special and improve interfaceGravatar Joey Hess2014-08-03
| | | | Allow disabling progress displays, for eg, rsync.
* WebDAV: Dropped support for DAV before 0.6.1.Gravatar Joey Hess2014-07-30
| | | | | 0.6.1 is in testing, and stable does not have DAV at all, so I can dispense with this compatability code
* better exception displayGravatar Joey Hess2014-07-26
|
* fix buildGravatar Joey Hess2014-07-26
|
* move meteredWriteFileChunks out of legacyGravatar Joey Hess2014-07-24
|
* improve chunk data typesGravatar Joey Hess2014-07-24
|
* prepare for new style chunkingGravatar Joey Hess2014-07-24
| | | | | | | | | | | | Moved old legacy chunking code, and cleaned up the directory and webdav remotes use of it, so when no chunking is configured, that code is not used. The config for new style chunking will be chunk=1M instead of chunksize=1M. There should be no behavior changes from this commit. This commit was sponsored by Andreas Laas.
* webdav: When built with a new enough haskell DAV (0.6), disable the http ↵Gravatar Joey Hess2014-03-05
| | | | response timeout, which was only 5 seconds.
* more DAV url fixes for windowsGravatar Joey Hess2014-02-25
|
* add protocol-level debugging for davGravatar Joey Hess2014-02-25
|
* Windows webdav: Fix DOS path separator bug.Gravatar Joey Hess2014-02-25
| | | | Use posix </> etc for urls.
* fix #740010 properlyGravatar Joey Hess2014-02-25
|
* webdav: When built with DAV 0.6.0, use the new DAV monad to avoid locking ↵Gravatar Joey Hess2014-02-24
| | | | files, which is not needed by git-annex's use of webdav, and does not work on Box.com.
* webapp: Fix creation of box.com, S3, and Glacier repositories, broken in ↵Gravatar Joey Hess2014-02-24
| | | | 5.20140221.
* plumb creds from webapp to initremoteGravatar Joey Hess2014-02-11
| | | | | Avoids abusing setting environment variables, which was always a hack and won't work on windows.
* add GETAVAILABILITY to external special remote protocolGravatar Joey Hess2014-01-13
| | | | | And some reworking of types, and added an annex-availability git config setting.
* Deal with box.com changing the url of their webdav endpoint.Gravatar Joey Hess2013-12-02
| | | | | | Use new url when making new remotes. Transparently rewrite old url to new for existing remotes.
* assistant: Support repairing git remotes that are locally accessibleGravatar Joey Hess2013-10-27
| | | | | | | | (eg, on removable drives) gcrypt remotes are not yet handled. This commit was sponsored by Sören Brunk.
* directory, webdav: Fix bug introduced in version 4.20131002 that caused the ↵Gravatar Joey Hess2013-10-26
| | | | chunkcount file to not be written. Work around repositories without such a file, so files can still be retreived from them.
* add remote fsck interfaceGravatar Joey Hess2013-10-11
| | | | | | | | | | | | | | | | | | | | Currently only implemented for local git remotes. May try to add support to git-annex-shell for ssh remotes later. Could concevably also be supported by some special remote, although that seems unlikely. Cronner user this when available, and when not falls back to fsck --fast --from remote git annex fsck --from does not itself use this interface. To do so, I would need to pass --fast and all other options that influence fsck on to the git annex fsck that it runs inside the remote. And that seems like a lot of work for a result that would be no better than cd remote; git annex fsck This may need to be revisited if git-annex-shell gets support, since it may be the case that the user cannot ssh to the server to run git-annex fsck there, but can run git-annex-shell there. This commit was sponsored by Damien Diederen.
* hlintGravatar Joey Hess2013-09-25
|
* Support hot-swapping of removable drives containing gcrypt repositories.Gravatar Joey Hess2013-09-12
| | | | | | | | | | | To support this, a core.gcrypt-id is stored by git-annex inside the git config of a local gcrypt repository, when setting it up. That is compared with the remote's cached gcrypt-id. When different, a drive has been changed. git-annex then looks up the remote config for the uuid mapped from the core.gcrypt-id, and tweaks the configuration appropriately. When there is no known config for the uuid, it will refuse to use the remote.
* partially complete gcrypt remote (local send done; rest not)Gravatar Joey Hess2013-09-07
| | | | | | | | | | | | | | | | | | | | | | | | This is a git-remote-gcrypt encrypted special remote. Only sending files in to the remote works, and only for local repositories. Most of the work so far has involved making initremote work. A particular problem is that remote setup in this case needs to generate its own uuid, derivied from the gcrypt-id. That required some larger changes in the code to support. For ssh remotes, this will probably just reuse Remote.Rsync's code, so should be easy enough. And for downloading from a web remote, I will need to factor out the part of Remote.Git that does that. One particular thing that will need work is supporting hot-swapping a local gcrypt remote. I think it needs to store the gcrypt-id in the git config of the local remote, so that it can check it every time, and compare with the cached annex-uuid for the remote. If there is a mismatch, it can change both the cached annex-uuid and the gcrypt-id. That should work, and I laid some groundwork for it by already reading the remote's config when it's local. (Also needed for other reasons.) This commit was sponsored by Daniel Callahan.
* fix build (seems getGpgEncOpts got renamed to getGpgEncParams)Gravatar Joey Hess2013-09-04
|
* Allow public-key encryption of file content.Gravatar guilhem2013-09-03
| | | | | | | | | | | | With the initremote parameters "encryption=pubkey keyid=788A3F4C". /!\ Adding or removing a key has NO effect on files that have already been copied to the remote. Hence using keyid+= and keyid-= with such remotes should be used with care, and make little sense unless the point is to replace a (sub-)key by another. /!\ Also, a test case has been added to ensure that the cipher and file contents are encrypted as specified by the chosen encryption scheme.
* Support building with DAV 0.4.Gravatar Joey Hess2013-04-30
|
* webapp: Now automatically fills in any creds used by an existing remote when ↵Gravatar Joey Hess2013-04-27
| | | | creating a new remote of the same type. Done for Internet Archive, S3, Glacier, and Box.com remotes.
* connect existing meters to the transfer log for downloadsGravatar Joey Hess2013-04-11
| | | | | | | | | | | | | | Most remotes have meters in their implementations of retrieveKeyFile already. Simply hooking these up to the transfer log makes that information available. Easy peasy. This is particularly valuable information for encrypted remotes, which otherwise bypass the assistant's polling of temp files, and so don't have good progress bars yet. Still some work to do here (see progressbars.mdwn changes), but this is entirely an improvement from the lack of progress bars for encrypted downloads.
* webapp: Progess bar fixes for many types of special remotes.Gravatar Joey Hess2013-03-28
| | | | | | | | | | | | | There was confusion in different parts of the progress bar code about whether an update contained the total number of bytes transferred, or the number of bytes transferred since the last update. One way this bug showed up was progress bars that seemed to stick at zero for a long time. In order to fix it comprehensively, I add a new BytesProcessed data type, that is explicitly a total quantity of bytes, not a delta. Note that this doesn't necessarily fix every problem with progress bars. Particularly, buffering can now cause progress bars to seem to run ahead of transfers, reaching 100% when data is still being uploaded.