summaryrefslogtreecommitdiff
path: root/Remote
Commit message (Collapse)AuthorAge
* external special remote protocol: Added GETUUID.Gravatar Joey Hess2013-12-31
|
* implement PREPARE-FAILURE for TobiasGravatar Joey Hess2013-12-29
|
* better error messages when external special remote exits unexpectedly or is ↵Gravatar Joey Hess2013-12-27
| | | | not in PATH
* add credential storage support for external special remotes & update exampleGravatar Joey Hess2013-12-27
|
* better protocol error message, indicate if the command was able to be parsed ↵Gravatar Joey Hess2013-12-27
| | | | or was misplaced
* flush handle after writing messageGravatar Joey Hess2013-12-27
|
* fix deadlock when state TMVar is emptyGravatar Joey Hess2013-12-27
|
* defer SETSTATE and GETSTATE for nowGravatar Joey Hess2013-12-27
| | | | | TAHOE-LAFS may use these eventually, but that's TBD and none of git-annex's own special remotes need that, except for the web special remote's urls.
* implement GETCONFIG and SETCONFIGGravatar Joey Hess2013-12-27
| | | | | | | | | | | Changed protocol spec to make SETCONFIG only store it persistently when run during INITREMOTE. I see no reason to support storing it persistently at other times, and doing so would unnecessarily complicate the code. Also, letting that be done would probably result in use for storing data that doesn't really belong there, and special remote authors who don't understand how the union merging works would probably be surprised the results.
* support encryptionGravatar Joey Hess2013-12-27
|
* make --debug show transcript of special remote protocol messagesGravatar Joey Hess2013-12-27
|
* don't send PREPARE before INITREMOTEGravatar Joey Hess2013-12-27
| | | | | | | | That complicated special remote programs, because they had to avoid making PREPARE fail if some configuration is missing, because the remote might not be initialized yet. Instead, complicate git-annex slightly by only sending PREPARE immediately before some other request other than INITREMOTE (or PREPARE of course).
* make some requests optional, simplify and future-proof protocol moreGravatar Joey Hess2013-12-27
|
* external special remotes mostly implemented (untested)Gravatar Joey Hess2013-12-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has not been tested at all. It compiles! The only known missing things are support for encryption, and for get/set of special remote configuration, and of key state. (The latter needs separate work to add a new per-key log file to store that state.) Only thing I don't much like is that initremote needs to be passed both type=external and externaltype=foo. It would be better to have just type=foo Most of this is quite straightforward code, that largely wrote itself given the types. The only tricky parts were: * Need to lock the remote when using it to eg make a request, because in theory git-annex could have multiple threads that each try to use a remote at the same time. I don't think that git-annex ever does that currently, but better safe than sorry. * Rather than starting up every external special remote program when git-annex starts, they are started only on demand, when first used. This will avoid slowdown, especially when running fast git-annex query commands. Once started, they keep running until git-annex stops, currently, which may not be ideal, but it's hard to know a better time to stop them. * Bit of a chicken and egg problem with caching the cost of the remote, because setting annex-cost in the git config needs the remote to already be set up. Managed to finesse that. This commit was sponsored by Lukas Anzinger.
* future-proofingGravatar Joey Hess2013-12-25
|
* basic data types and serialization for external special remote protocolGravatar Joey Hess2013-12-25
| | | | | | | | | | This is mostly straightforward, but did turn out quite nicely stronly typed, and with a quite nice automatic tokenization and parsing of received messages. Made a few minor changes to the protocol to clear up ambiguities and make it easier to parse. Note particularly that setting remote configuration is moved to a separate command, which allows a remote to set arbitrary data.
* pull in Win32-extras, to be able to get current process id in WindowsGravatar Joey Hess2013-12-11
| | | | | | | | | Fixed up a number of things that had worked around there not being a way to get that. Most notably, transfer info files on windows now include the process id, since no locking is currently done. This means the file format varies between windows and unix.
* Deal with box.com changing the url of their webdav endpoint.Gravatar Joey Hess2013-12-02
| | | | | | Use new url when making new remotes. Transparently rewrite old url to new for existing remotes.
* rsync special remote: Fix fallback mode for rsync remotes that use ↵Gravatar Joey Hess2013-12-02
| | | | hashDirMixed. Closes: #731142
* map: Work when there are gcrypt remotes.Gravatar Joey Hess2013-11-04
|
* really fix gcrypt for aaf881e8bb22408f671e566ebcd3f59be958a26fGravatar Joey Hess2013-11-02
| | | | Fixed all the other ones, but forgot to fix gcrypt!
* clean import listGravatar Joey Hess2013-11-02
|
* fix buildGravatar Joey Hess2013-11-02
|
* gcrypt, bup: Fix bug that prevented using these special remotes with ↵Gravatar Joey Hess2013-11-02
| | | | | | | | | encryption=pubkey. I think both of these are all that's affected, but I went ahead and fixed all the remotes that set their config to M.empty to instead store the actual config. Who knows what will expect it to be actually present in future, the Remote instance of getGpgEncParams came to..
* assistant: Support repairing git remotes that are locally accessibleGravatar Joey Hess2013-10-27
| | | | | | | | (eg, on removable drives) gcrypt remotes are not yet handled. This commit was sponsored by Sören Brunk.
* directory, webdav: Fix bug introduced in version 4.20131002 that caused the ↵Gravatar Joey Hess2013-10-26
| | | | chunkcount file to not be written. Work around repositories without such a file, so files can still be retreived from them.
* fix inverted logic when determining whether to write a chunkcount fileGravatar Joey Hess2013-10-26
| | | | | | late-night hlint bit me on this one.. Reviewed f32cb2cf1576db1395f77bd5f7f0c0a3e86c1334 and the rest of it seems ok
* S3: Try to ensure bucket name is valid for archive.org.Gravatar Joey Hess2013-10-16
|
* fix remote fsck to run in remoteGravatar Joey Hess2013-10-14
|
* ye olde inverted logicGravatar Joey Hess2013-10-14
|
* add remote fsck interfaceGravatar Joey Hess2013-10-11
| | | | | | | | | | | | | | | | | | | | Currently only implemented for local git remotes. May try to add support to git-annex-shell for ssh remotes later. Could concevably also be supported by some special remote, although that seems unlikely. Cronner user this when available, and when not falls back to fsck --fast --from remote git annex fsck --from does not itself use this interface. To do so, I would need to pass --fast and all other options that influence fsck on to the git annex fsck that it runs inside the remote. And that seems like a lot of work for a result that would be no better than cd remote; git annex fsck This may need to be revisited if git-annex-shell gets support, since it may be the case that the user cannot ssh to the server to run git-annex fsck there, but can run git-annex-shell there. This commit was sponsored by Damien Diederen.
* url size fixesGravatar Joey Hess2013-10-11
| | | | | | | | | | addurl: Improve message when adding url with wrong size to existing file. Before the message suggested the url didn't exist. Fixed handling of URL keys that have no recorded size. Before, if the key has no size, the url also had to not declare any size, which was unlikely and wrong, or it was taken to not exist. This probably would mostly affect keys that were added to the annex with addurl --relaxed.
* remove __WINDOWS__ ifdefGravatar Joey Hess2013-10-06
|
* typoe and debug infoGravatar Joey Hess2013-10-01
|
* git-annex-shell gcryptsetup commandGravatar Joey Hess2013-10-01
| | | | | This was the least-bad alternative to get dedicated key gcrypt repos working in the assistant.
* Disable receive.denyNonFastForwards when setting up a gcrypt special remoteGravatar Joey Hess2013-10-01
| | | | | | gcrypt needs to be able to fast-forward the master branch. If a git repository is set up with git init --shared --bare, it gets that set, and pushing to it will then fail, even when it's up-to-date.
* fix probing for local gcrypt reposGravatar Joey Hess2013-10-01
|
* fix transferring to gcrypt repo from direct mode repoGravatar Joey Hess2013-10-01
| | | | | | | | | | | | | | | | | | recvkey was told it was receiving a HMAC key from a direct mode repo, and that confused it into rejecting the transfer, since it has no way to verify a key using that backend, since there is no HMAC backend. I considered making recvkey skip verification in the case of an unknown backend. However, that could lead to bad results; a key can legitimately be in the annex with a backend that the remote git-annex-shell doesn't know about. Better to keep it rejecting if it cannot verify. Instead, made the gcrypt special remote not set the direct mode flag when sending (and receiving) files. Also, added some recvkey messages when its checks fail, since otherwise all that is shown is a confusing error message from rsync when the remote git-annex-shell exits nonzero.
* Send a git-annex user-agent when downloading urls.Gravatar Joey Hess2013-09-28
| | | | | | | | | Overridable with --user-agent option. Not yet done for S3 or WebDAV due to limitations of libraries used -- nether allows a user-agent header to be specified. This commit sponsored by Michael Zehrer.
* clean up some ugly codeGravatar Joey Hess2013-09-27
|
* blind enabling gcrypt repos on rsync.netGravatar Joey Hess2013-09-27
| | | | | | | | | | | This pulls off quite a nice trick: When given a path on rsync.net, it determines if it is an encrypted git repository that the user has the key to decrypt, and merges with it. This is works even when the local repository had no idea that the gcrypt remote exists! (As previously done with local drives.) This commit sponsored by Pedro Côrte-Real
* support ssh://host/~/dirGravatar Joey Hess2013-09-26
| | | | | | | | When generating the path for rsync, /~/ is not valid, so change to just host:dir Note that git remotes specified in host:dir form are internally converted to the ssh:// url form, so this was especially needed..
* hlintGravatar Joey Hess2013-09-25
|
* add back lost check that git-annex-shell supports gcryptGravatar Joey Hess2013-09-24
|
* git-annex-shell: Added support for operating inside gcrypt repositories.Gravatar Joey Hess2013-09-24
| | | | | | * Note that the layout of gcrypt repositories has changed, and if you created one you must manually upgrade it. See http://git-annex.branchable.com/upgrades/gcrypt/
* factor out more ssh stuff from git remoteGravatar Joey Hess2013-09-24
| | | | | This has the dual benefits of making Remote.Git shorter, and letting Remote.GCrypt use these utilities.
* Use cryptohash rather than SHA for hashing.Gravatar Joey Hess2013-09-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a massive win on OSX, which doesn't have a sha256sum normally. Only use external hash commands when the file is > 1 mb, since cryptohash is quite close to them in speed. SHA is still used to calculate HMACs. I don't quite understand cryptohash's API for those. Used the following benchmark to arrive at the 1 mb number. 1 mb file: benchmarking sha256/internal mean: 13.86696 ms, lb 13.83010 ms, ub 13.93453 ms, ci 0.950 std dev: 249.3235 us, lb 162.0448 us, ub 458.1744 us, ci 0.950 found 5 outliers among 100 samples (5.0%) 4 (4.0%) high mild 1 (1.0%) high severe variance introduced by outliers: 10.415% variance is moderately inflated by outliers benchmarking sha256/external mean: 14.20670 ms, lb 14.17237 ms, ub 14.27004 ms, ci 0.950 std dev: 230.5448 us, lb 150.7310 us, ub 427.6068 us, ci 0.950 found 3 outliers among 100 samples (3.0%) 2 (2.0%) high mild 1 (1.0%) high severe 2 mb file: benchmarking sha256/internal mean: 26.44270 ms, lb 26.23701 ms, ub 26.63414 ms, ci 0.950 std dev: 1.012303 ms, lb 925.8921 us, ub 1.122267 ms, ci 0.950 variance introduced by outliers: 35.540% variance is moderately inflated by outliers benchmarking sha256/external mean: 26.84521 ms, lb 26.77644 ms, ub 26.91433 ms, ci 0.950 std dev: 347.7867 us, lb 210.6283 us, ub 571.3351 us, ci 0.950 found 6 outliers among 100 samples (6.0%) import Crypto.Hash import Data.ByteString.Lazy as L import Criterion.Main import Common testfile :: FilePath testfile = "/run/shm/data" -- on ram disk main = defaultMain [ bgroup "sha256" [ bench "internal" $ whnfIO internal , bench "external" $ whnfIO external ] ] sha256 :: L.ByteString -> Digest SHA256 sha256 = hashlazy internal :: IO String internal = show . sha256 <$> L.readFile testfile external :: IO String external = do s <- readProcess "sha256sum" [testfile] return $ fst $ separate (== ' ') s
* better probing for gcrypt repositories using new --check optionGravatar Joey Hess2013-09-19
| | | | | | | | Now can tell if a repo uses gcrypt or not, and whether it's decryptable with the current gpg keys. This closes the hole that undecryptable gcrypt repos could have before been combined into the repo in encrypted mode.
* webapp: support adding existing gcrypt special remotes from removable drivesGravatar Joey Hess2013-09-18
| | | | | | | | | | | | | | | | | | | | | | | When adding a removable drive, it's now detected if the drive contains a gcrypt special remote, and that's all handled nicely. This includes fetching the git-annex branch from the gcrypt repo in order to find out how to set up the special remote. Note that gcrypt repos that are not git-annex special remotes are not supported. It will attempt to detect such a gcrypt repo and refuse to use it. (But this is hard to do any may fail; see https://github.com/blake2-ppc/git-remote-gcrypt/issues/6) The problem with supporting regular gcrypt repos is that we don't know what the gcrypt.participants setting is intended to be for the repo. So even if we can decrypt it, if we push changes to it they might not be visible to other participants. Anyway, encrypted sneakernet (or mailnet) is now fully possible with the git-annex assistant! Assuming that the gpg key distribution is handled somehow, which the assistant doesn't yet help with. This commit was sponsored by Navishkar Rao.
* gcrypt: Ensure that signing key is set to one of the participants keys.Gravatar Joey Hess2013-09-17
| | | | | | Otherwise gcrypt will fail to pull, since it requires this to be the case. This needs a patched gcrypt, which is in my forked version.