summaryrefslogtreecommitdiff
path: root/Remote
Commit message (Collapse)AuthorAge
* assistant: Support repairing git remotes that are locally accessibleGravatar Joey Hess2013-10-27
| | | | | | | | (eg, on removable drives) gcrypt remotes are not yet handled. This commit was sponsored by Sören Brunk.
* directory, webdav: Fix bug introduced in version 4.20131002 that caused the ↵Gravatar Joey Hess2013-10-26
| | | | chunkcount file to not be written. Work around repositories without such a file, so files can still be retreived from them.
* fix inverted logic when determining whether to write a chunkcount fileGravatar Joey Hess2013-10-26
| | | | | | late-night hlint bit me on this one.. Reviewed f32cb2cf1576db1395f77bd5f7f0c0a3e86c1334 and the rest of it seems ok
* S3: Try to ensure bucket name is valid for archive.org.Gravatar Joey Hess2013-10-16
|
* fix remote fsck to run in remoteGravatar Joey Hess2013-10-14
|
* ye olde inverted logicGravatar Joey Hess2013-10-14
|
* add remote fsck interfaceGravatar Joey Hess2013-10-11
| | | | | | | | | | | | | | | | | | | | Currently only implemented for local git remotes. May try to add support to git-annex-shell for ssh remotes later. Could concevably also be supported by some special remote, although that seems unlikely. Cronner user this when available, and when not falls back to fsck --fast --from remote git annex fsck --from does not itself use this interface. To do so, I would need to pass --fast and all other options that influence fsck on to the git annex fsck that it runs inside the remote. And that seems like a lot of work for a result that would be no better than cd remote; git annex fsck This may need to be revisited if git-annex-shell gets support, since it may be the case that the user cannot ssh to the server to run git-annex fsck there, but can run git-annex-shell there. This commit was sponsored by Damien Diederen.
* url size fixesGravatar Joey Hess2013-10-11
| | | | | | | | | | addurl: Improve message when adding url with wrong size to existing file. Before the message suggested the url didn't exist. Fixed handling of URL keys that have no recorded size. Before, if the key has no size, the url also had to not declare any size, which was unlikely and wrong, or it was taken to not exist. This probably would mostly affect keys that were added to the annex with addurl --relaxed.
* remove __WINDOWS__ ifdefGravatar Joey Hess2013-10-06
|
* typoe and debug infoGravatar Joey Hess2013-10-01
|
* git-annex-shell gcryptsetup commandGravatar Joey Hess2013-10-01
| | | | | This was the least-bad alternative to get dedicated key gcrypt repos working in the assistant.
* Disable receive.denyNonFastForwards when setting up a gcrypt special remoteGravatar Joey Hess2013-10-01
| | | | | | gcrypt needs to be able to fast-forward the master branch. If a git repository is set up with git init --shared --bare, it gets that set, and pushing to it will then fail, even when it's up-to-date.
* fix probing for local gcrypt reposGravatar Joey Hess2013-10-01
|
* fix transferring to gcrypt repo from direct mode repoGravatar Joey Hess2013-10-01
| | | | | | | | | | | | | | | | | | recvkey was told it was receiving a HMAC key from a direct mode repo, and that confused it into rejecting the transfer, since it has no way to verify a key using that backend, since there is no HMAC backend. I considered making recvkey skip verification in the case of an unknown backend. However, that could lead to bad results; a key can legitimately be in the annex with a backend that the remote git-annex-shell doesn't know about. Better to keep it rejecting if it cannot verify. Instead, made the gcrypt special remote not set the direct mode flag when sending (and receiving) files. Also, added some recvkey messages when its checks fail, since otherwise all that is shown is a confusing error message from rsync when the remote git-annex-shell exits nonzero.
* Send a git-annex user-agent when downloading urls.Gravatar Joey Hess2013-09-28
| | | | | | | | | Overridable with --user-agent option. Not yet done for S3 or WebDAV due to limitations of libraries used -- nether allows a user-agent header to be specified. This commit sponsored by Michael Zehrer.
* clean up some ugly codeGravatar Joey Hess2013-09-27
|
* blind enabling gcrypt repos on rsync.netGravatar Joey Hess2013-09-27
| | | | | | | | | | | This pulls off quite a nice trick: When given a path on rsync.net, it determines if it is an encrypted git repository that the user has the key to decrypt, and merges with it. This is works even when the local repository had no idea that the gcrypt remote exists! (As previously done with local drives.) This commit sponsored by Pedro Côrte-Real
* support ssh://host/~/dirGravatar Joey Hess2013-09-26
| | | | | | | | When generating the path for rsync, /~/ is not valid, so change to just host:dir Note that git remotes specified in host:dir form are internally converted to the ssh:// url form, so this was especially needed..
* hlintGravatar Joey Hess2013-09-25
|
* add back lost check that git-annex-shell supports gcryptGravatar Joey Hess2013-09-24
|
* git-annex-shell: Added support for operating inside gcrypt repositories.Gravatar Joey Hess2013-09-24
| | | | | | * Note that the layout of gcrypt repositories has changed, and if you created one you must manually upgrade it. See http://git-annex.branchable.com/upgrades/gcrypt/
* factor out more ssh stuff from git remoteGravatar Joey Hess2013-09-24
| | | | | This has the dual benefits of making Remote.Git shorter, and letting Remote.GCrypt use these utilities.
* Use cryptohash rather than SHA for hashing.Gravatar Joey Hess2013-09-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a massive win on OSX, which doesn't have a sha256sum normally. Only use external hash commands when the file is > 1 mb, since cryptohash is quite close to them in speed. SHA is still used to calculate HMACs. I don't quite understand cryptohash's API for those. Used the following benchmark to arrive at the 1 mb number. 1 mb file: benchmarking sha256/internal mean: 13.86696 ms, lb 13.83010 ms, ub 13.93453 ms, ci 0.950 std dev: 249.3235 us, lb 162.0448 us, ub 458.1744 us, ci 0.950 found 5 outliers among 100 samples (5.0%) 4 (4.0%) high mild 1 (1.0%) high severe variance introduced by outliers: 10.415% variance is moderately inflated by outliers benchmarking sha256/external mean: 14.20670 ms, lb 14.17237 ms, ub 14.27004 ms, ci 0.950 std dev: 230.5448 us, lb 150.7310 us, ub 427.6068 us, ci 0.950 found 3 outliers among 100 samples (3.0%) 2 (2.0%) high mild 1 (1.0%) high severe 2 mb file: benchmarking sha256/internal mean: 26.44270 ms, lb 26.23701 ms, ub 26.63414 ms, ci 0.950 std dev: 1.012303 ms, lb 925.8921 us, ub 1.122267 ms, ci 0.950 variance introduced by outliers: 35.540% variance is moderately inflated by outliers benchmarking sha256/external mean: 26.84521 ms, lb 26.77644 ms, ub 26.91433 ms, ci 0.950 std dev: 347.7867 us, lb 210.6283 us, ub 571.3351 us, ci 0.950 found 6 outliers among 100 samples (6.0%) import Crypto.Hash import Data.ByteString.Lazy as L import Criterion.Main import Common testfile :: FilePath testfile = "/run/shm/data" -- on ram disk main = defaultMain [ bgroup "sha256" [ bench "internal" $ whnfIO internal , bench "external" $ whnfIO external ] ] sha256 :: L.ByteString -> Digest SHA256 sha256 = hashlazy internal :: IO String internal = show . sha256 <$> L.readFile testfile external :: IO String external = do s <- readProcess "sha256sum" [testfile] return $ fst $ separate (== ' ') s
* better probing for gcrypt repositories using new --check optionGravatar Joey Hess2013-09-19
| | | | | | | | Now can tell if a repo uses gcrypt or not, and whether it's decryptable with the current gpg keys. This closes the hole that undecryptable gcrypt repos could have before been combined into the repo in encrypted mode.
* webapp: support adding existing gcrypt special remotes from removable drivesGravatar Joey Hess2013-09-18
| | | | | | | | | | | | | | | | | | | | | | | When adding a removable drive, it's now detected if the drive contains a gcrypt special remote, and that's all handled nicely. This includes fetching the git-annex branch from the gcrypt repo in order to find out how to set up the special remote. Note that gcrypt repos that are not git-annex special remotes are not supported. It will attempt to detect such a gcrypt repo and refuse to use it. (But this is hard to do any may fail; see https://github.com/blake2-ppc/git-remote-gcrypt/issues/6) The problem with supporting regular gcrypt repos is that we don't know what the gcrypt.participants setting is intended to be for the repo. So even if we can decrypt it, if we push changes to it they might not be visible to other participants. Anyway, encrypted sneakernet (or mailnet) is now fully possible with the git-annex assistant! Assuming that the gpg key distribution is handled somehow, which the assistant doesn't yet help with. This commit was sponsored by Navishkar Rao.
* gcrypt: Ensure that signing key is set to one of the participants keys.Gravatar Joey Hess2013-09-17
| | | | | | Otherwise gcrypt will fail to pull, since it requires this to be the case. This needs a patched gcrypt, which is in my forked version.
* Support hot-swapping of removable drives containing gcrypt repositories.Gravatar Joey Hess2013-09-12
| | | | | | | | | | | To support this, a core.gcrypt-id is stored by git-annex inside the git config of a local gcrypt repository, when setting it up. That is compared with the remote's cached gcrypt-id. When different, a drive has been changed. git-annex then looks up the remote config for the uuid mapped from the core.gcrypt-id, and tweaks the configuration appropriately. When there is no known config for the uuid, it will refuse to use the remote.
* sync: support gcrypt4.20130909Gravatar Joey Hess2013-09-09
|
* Allow building without quvi support.Gravatar Joey Hess2013-09-09
|
* ignore gcrypt remotes w/o an annex-uuidGravatar Joey Hess2013-09-08
|
* gcrypt: now supports rsyncGravatar Joey Hess2013-09-08
| | | | | | | | | | | | | | | | | | | | Use rsync for gcrypt remotes that are not local to the disk. (Note that I have punted on supporting http transport for now, it doesn't seem likely to be very useful.) This was mostly quite easy, it just uses the rsync special remote to handle the transfers. The git repository url is converted to a RsyncOptions structure, which required parsing it separately, since the rsync special remote only supports rsync urls, which use a different format. Note that annexed objects are now stored at the top of the gcrypt repo, rather than inside annex/objects. This simplified the rsync suport, since it doesn't have to arrange to create that directory. And git-annex is not going to be run directly within gcrypt repos -- or if in some strance scenario it was, it would make sense for it to not see the encrypted objects. This commit was sponsored by Sheila Miguez
* local gcrypt fully working!Gravatar Joey Hess2013-09-08
|
* partially complete gcrypt remote (local send done; rest not)Gravatar Joey Hess2013-09-07
| | | | | | | | | | | | | | | | | | | | | | | | This is a git-remote-gcrypt encrypted special remote. Only sending files in to the remote works, and only for local repositories. Most of the work so far has involved making initremote work. A particular problem is that remote setup in this case needs to generate its own uuid, derivied from the gcrypt-id. That required some larger changes in the code to support. For ssh remotes, this will probably just reuse Remote.Rsync's code, so should be easy enough. And for downloading from a web remote, I will need to factor out the part of Remote.Git that does that. One particular thing that will need work is supporting hot-swapping a local gcrypt remote. I think it needs to store the gcrypt-id in the git config of the local remote, so that it can check it every time, and compare with the cached annex-uuid for the remote. If there is a mismatch, it can change both the cached annex-uuid and the gcrypt-id. That should work, and I laid some groundwork for it by already reading the remote's config when it's local. (Also needed for other reasons.) This commit was sponsored by Daniel Callahan.
* automatically derive an annex-uuid from a gcrypt-uuidsGravatar Joey Hess2013-09-05
|
* rename constructor for clariyGravatar Joey Hess2013-09-05
|
* Leverage an ambiguities between CiphersGravatar guilhem2013-09-05
| | | | | | | | | Cipher is now a datatype data Cipher = Cipher String | MacOnlyCipher String which makes more precise its interpretation MAC-only vs. MAC + used to derive a key for symmetric crypto.
* tabsGravatar Joey Hess2013-09-04
|
* unimportant tweakGravatar Joey Hess2013-09-04
| | | | fix something my internal haskell parser does a double take at
* replace an over-explained Bool with a data typeGravatar Joey Hess2013-09-04
| | | | | This also highlights several places where a Read/Show or similar for the new data type could avoid redundant strings.
* Encryption defaults to 'hybrid'Gravatar guilhem2013-09-04
| | | | When a keyid= is specified while encryption= is absent.
* fix build (seems getGpgEncOpts got renamed to getGpgEncParams)Gravatar Joey Hess2013-09-04
|
* Allow public-key encryption of file content.Gravatar guilhem2013-09-03
| | | | | | | | | | | | With the initremote parameters "encryption=pubkey keyid=788A3F4C". /!\ Adding or removing a key has NO effect on files that have already been copied to the remote. Hence using keyid+= and keyid-= with such remotes should be used with care, and make little sense unless the point is to replace a (sub-)key by another. /!\ Also, a test case has been added to ensure that the cipher and file contents are encrypted as specified by the chosen encryption scheme.
* Allow revocation of OpenPGP keys.Gravatar guilhem2013-08-29
| | | | | | | | | | | | | | | | | | | | | | | /!\ It is to be noted that revoking a key does NOT necessarily prevent the owner of its private part from accessing data on the remote /!\ The only sound use of `keyid-=` is probably to replace a (sub-)key by another, where the private part of both is owned by the same person/entity: git annex enableremote myremote keyid-=2512E3C7 keyid+=788A3F4C Reference: http://git-annex.branchable.com/bugs/Using_a_revoked_GPG_key/ * Other change introduced by this patch: New keys now need to be added with option `keyid+=`, and the scheme specified (upon initremote only) with `encryption=`. The motivation for this change is to open for new schemes, e.g., strict asymmetric encryption. git annex initremote myremote encryption=hybrid keyid=2512E3C7 git annex enableremote myremote keyid+=788A3F4C
* better casesGravatar Joey Hess2013-08-22
|
* unimportant typoGravatar Joey Hess2013-08-22
| | | | (u and u' happened to be the same)
* Youtube support! (And 53 other video hosts)Gravatar Joey Hess2013-08-22
| | | | | | | | | | When quvi is installed, git-annex addurl automatically uses it to detect when an page is a video, and downloads the video file. web special remote: Also support using quvi, for getting files, or checking if files exist in the web. This commit was sponsored by Mark Hepburn. Thanks!
* avoid more build warnings on WindowsGravatar Joey Hess2013-08-04
|
* Windows: Fixed permissions problem that prevented removing files from ↵Gravatar Joey Hess2013-08-04
| | | | | | directory special remote. Directory special remotes now fully usable.
* squash compiler warnings on WindowsGravatar Joey Hess2013-08-04
|
* get rid of __WINDOWS__, use mingw32_HOST_OSGravatar Joey Hess2013-08-02
| | | | | The latter is harder for me to remember, but avoids build failures in code used by the configure program.