summaryrefslogtreecommitdiff
path: root/Remote
Commit message (Collapse)AuthorAge
* Disable http-client's default 30 second response timeout when HEADing an url ↵Gravatar Joey Hess2017-08-15
| | | | to check if it exists. Some web servers take quite a long time to answer a HEAD request.
* adeiu, MissingHGravatar Joey Hess2017-05-16
| | | | | | | | | | | | | | | | Removed dependency on MissingH, instead depending on the split library. After laying groundwork for this since 2015, it was mostly straightforward. Added Utility.Tuple and Utility.Split. Eyeballed System.Path.WildMatch while implementing the same thing. Since MissingH's progress meter display was being used, I re-implemented my own. Bonus: Now progress is displayed for transfers of files of unknown size. This commit was sponsored by Shane-o on Patreon.
* de-Maybe remoteGitConfigGravatar Joey Hess2017-05-11
| | | | It's always set, so does not need to be a Maybe.
* gcrypt: Support re-enabling to change eg, encryption parameters.Gravatar Joey Hess2017-04-07
| | | | | | | | This was never supported before. And it doesn't re-encrypt the gcrypt repo to the new gcrypt-participants, but it does at least now not crash, and set gcrypt-participants. This commit was sponsored by andrea rota.
* When a http remote does not expose an annex.uuid config, only warn about it ↵Gravatar Joey Hess2017-03-29
| | | | | | once, not every time git-annex is run. Same behavior as for a ssh remote.
* Support GIT_SSH and GIT_SSH_COMMANDGravatar Joey Hess2017-03-17
| | | | | | | | | | | | | | | | | | | | They are handled close the same as they are by git. However, unlike git, git-annex sometimes needs to pass the -n parameter when using these. So, this has the potential for breaking some setup, and perhaps there ought to be a ANNEX_USE_GIT_SSH=1 needed to use these. But I'd rather avoid that if possible, so let's see if anyone complains. Almost all places where "ssh" was run have been changed to support the env vars. Anything still calling sshOptions does not support them. In particular, rsync special remotes don't. Seems that annex-rsync-transport already gives sufficient control there. (Fixed in passing: Remote.Helper.Ssh.toRepo used to extract remoteAnnexSshOptions and pass them to sshOptions, which was redundant since sshOptions also extracts those.) This commit was sponsored by Jeff Goeke-Smith on Patreon.
* AssociatedFile newtypeGravatar Joey Hess2017-03-10
| | | | | | To prevent any further mistakes like 1a497cefb47557f0b4788c606f9071be422b2511 This commit was sponsored by Francois Marier on Patreon.
* Windows: Improve handling of shebang in external special remote program, ↵Gravatar Joey Hess2017-03-08
| | | | | | | | | | | | | | | | | | | searching for the program in the PATH. findShellCommand needs a full path to a file in order to check it for a shebang on Windows. It was being run with only the base name of the external special remote program, which would only work when it was in the current directory. This is why users in https://github.com/DanielDent/git-annex-remote-rclone/pull/10 and elsewhere were complaining that the previous improvements to git-annex didn't make git-remote-rclone work on Windows. Also, reworked checkearlytermination, which while it worked, seemed to rely on a race condition. And, improved its error messages. This commit was sponsored by Shane-o on Patreon.
* sync hack to make updateInstead work on eg FATGravatar Joey Hess2017-02-17
| | | | | | | | | | | | sync: When syncing with a local repository located on a crippled filesystem, run the post-receive hook there, since it wouldn't get run otherwise. This makes pushing to repos on FAT-formatted removable drives update them when receive.denyCurrentBranch=updateInstead. Made Remote.Git export onLocal, which was cleaned up to not have so many caveats about its use. This commit was sponsored by Jeff Goeke-Smith on Patreon.
* have onLocal stop any coprocesses, not only cat-fileGravatar Joey Hess2017-02-17
| | | | | I have not seen any other coprocesses being started, but let's avoid problems if any do for whatever reason.
* Run ssh with -n whenever input is not being piped into itGravatar Joey Hess2017-02-15
| | | | | | | | | | | | | | | | | | | | ... to avoid it consuming stdin that it shouldn't. This fixes git-annex-checkpresentkey --batch remote, which didn't output results for all keys passed into it. Other git-annex commands that communicate with a remote over ssh may also have been consuming stdin that they shouldn't have, which could have impacted using them in eg, shell scripts. For example, a shell script reading files from stdin and passing them to git annex drop would be impacted by this bug, whenever git annex drop ran git-annex-shell checkpresent, it would consume part/all of the stdin that the shell script was supposed to consume. Fixed by adding a ConsumeStdin parameter to Annex.Ssh.sshOptions, which is used throughout git-annex to run ssh (in order for ssh connection caching to work). Every call site was checked to see if it used CreatePipe for stdin, and if not was marked NoConsumeStdin.
* S3: Fix check of uuid file stored in bucket, which was not working.Gravatar Joey Hess2017-02-13
| | | | | | | | | | | | | | The check was broken in two ways.. First, nowhere did it error out when checkUUIDFile found a different UUID already in the file. Instead, it overwrote the uuid file. And, checkUUIDFile's implementation was for some reason always failing with a ConnectionClosed exception. Apparently something to do with using two different runResourceT's and a response getting GCed inbetween. I'm pretty sure that used to work, but changed to a more obviously correct implementation. This commit was sponsored by Peter Hogg on Patreon.
* correct spelling mistakesGravatar Edward Betts2017-02-12
|
* add SetupStage parameter to RemoteType.setupGravatar Joey Hess2017-02-07
| | | | | | | | | | | | | | | | | Most remotes have an idempotent setup that can be reused for enableremote, but in a few cases, it needs to tell which, and whether a UUID was provided to setup was used. This is groundwork for making initremote be able to provide a UUID. It should not change any behavior. Note that it would be nice to make the UUID always be provided to setup, and make setup not need to generate and return a UUID. What prevented this simplification is Remote.Git.gitSetup, which needs to reuse the UUID of the git remote when setting it up, and so has to return that UUID. This commit was sponsored by Thom May on Patreon.
* Fix build with aws 0.16. Thanks, aristidb.Gravatar Joey Hess2017-02-07
|
* Some optimisations to string splitting code.Gravatar Joey Hess2017-01-31
| | | | | | | | | | | | | | | | | | | Turns out that Data.List.Utils.split is slow and makes a lot of allocations. Here's a much simpler single character splitter that behaves the same (even in wacky corner cases) while running in half the time and 75% the allocations. As well as being an optimisation, this helps move toward eliminating use of missingh. (Data.List.Split.splitOn is nearly as slow as Data.List.Utils.split and allocates even more.) I have not benchmarked the effect on git-annex, but would not be surprised to see some parsing of eg, large streams from git commands run twice as fast, and possibly in less memory. This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
* Increase default cost for p2p remotes from 200 to 1000. This makes git-annex ↵Gravatar Joey Hess2017-01-06
| | | | prefer transferring data from special remotes when possible.
* Always use filesystem encoding for all file and handle reads and writes.Gravatar Joey Hess2016-12-24
| | | | | This is a big scary change. I have convinced myself it should be safe. I hope!
* fix build warningGravatar Joey Hess2016-12-10
|
* Remove http-conduit (<2.2.0) constraintGravatar Alper Nebi Yasak2016-12-10
| | | | | | | | | | Since https://github.com/aristidb/aws/issues/206 is resolved, this constraint is no longer necessary. However, http-conduit (>=2.2.0) requires http-client (>=0.5.0) which introduces some breaking changes. This commit also implements those changes depending on the version. Fixes: https://git-annex.branchable.com/bugs/Build_with_aws_head_fails/ Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
* git-annex-shell, remotedaemon, git remote: Fix some memory DOS attacks.Gravatar Joey Hess2016-12-09
| | | | | | | | | | | | | | | | | | | | | The attacker could just send a very lot of data, with no \n and it would all be buffered in memory until the kernel killed git-annex or perhaps OOM killed some other more valuable process. This is a low impact security hole, only affecting communication between local git-annex and git-annex-shell on the remote system. (With either able to be the attacker). Only those with the right ssh key can do it. And, there are probably lots of ways to construct git repositories that make git use a lot of memory in various ways, which would have similar impact as this attack. The fix in P2P/IO.hs would have been higher impact, if it had made it to a released version, since it would have allowed DOSing the tor hidden service without needing to authenticate. (The LockContent and NotifyChanges instances may not be really exploitable; since the line is read and ignored, it probably gets read lazily and does not end up staying buffered in memory.)
* fixGravatar Joey Hess2016-12-09
|
* make clear that log is only updated after successful removalGravatar Joey Hess2016-12-09
| | | | | This does not change behavior, because an exception is thrown on unsuccessful removal. But is clearer.
* improve warning on connection lossGravatar Joey Hess2016-12-09
|
* better format errorGravatar Joey Hess2016-12-08
|
* convert P2P runners from Maybe to Either StringGravatar Joey Hess2016-12-08
| | | | | | So we get some useful error messages when things fail. This commit was sponsored by Peter Hogg on Patreon.
* more p2p progress metersGravatar Joey Hess2016-12-07
| | | | | | | | | Display progress meter on send and receive from remote. Added a new hGetMetered that can read an exact number of bytes (or less), updating a meter as it goes. This commit was sponsored by Andreas on Patreon.
* update progress meter when sending to p2p remoteGravatar Joey Hess2016-12-07
| | | | This commit was sponsored by Thom May on Patreon.
* validate peer uuid each time we talk to itGravatar Joey Hess2016-12-07
| | | | | | | | | | In case the repo on the peer changes uuid (eg by a new repo being moved into place). Also, added some warning messages when unable to communicate with a peer. This commit was sponsored by Anthony DeRobertis on Patreon.
* need to auth with the peerGravatar Joey Hess2016-12-06
|
* refactorGravatar Joey Hess2016-12-06
|
* finish implementation of Remote.P2P (untested)Gravatar Joey Hess2016-12-06
| | | | | | | | Not tested at all, but it just might work. Only known problem is that progress is not updated when storing to a P2P remote. This commit was sponsored by Nick Daly on Patreon.
* stub Remote.P2PGravatar Joey Hess2016-12-06
| | | | | | | Similar to GCrypt remotes, P2P remotes have an url, so Remote.Git has to separate them out and handle them, passing off to Remote.P2P. This commit was sponsored by Ignacio on Patreon.
* use P2P auth for git-remote-tor-annexGravatar Joey Hess2016-11-30
| | | | | | | This changes the environment variable name to the more generic GIT_ANNEX_P2P_AUTHTOKEN. This commit was sponsored by andrea rota.
* reorgGravatar Joey Hess2016-11-22
|
* unified AuthToken type between webapp and torGravatar Joey Hess2016-11-22
|
* remove debugGravatar Joey Hess2016-11-21
|
* detect EOF on socket and cleanly shutdown the service processGravatar Joey Hess2016-11-21
|
* stop cleanly when there's a IO error accessing the HandleGravatar Joey Hess2016-11-21
| | | | | All other exceptions are let through, but IO errors accessing the handle are to be expected, so quietly ignore.
* try to gather scattered writesGravatar Joey Hess2016-11-21
| | | | | | | | | git upload-pack makes some uncessary writes in sequence, this tries to gather them together to avoid needing to send multiple DATA packets when just one will do. In a small pull, this reduces the average number of DATA packets from 4.5 to 2.5.
* fix parse of CONNECTDONEGravatar Joey Hess2016-11-21
|
* pull/push over tor working nowGravatar Joey Hess2016-11-21
| | | | | | | | | | | | | | Still a couple bugs: * Closing the connection to the server leaves git upload-pack / receive-pack running, which could be used to DOS. * Sometimes the data is transferred, but it fails at the end, sometimes with: git-remote-tor-annex: <socket: 10>: commitBuffer: resource vanished (Broken pipe) Must be a race condition around shutdown.
* Added git-remote-tor-annex, which allows git pull and push to the tor hidden ↵Gravatar Joey Hess2016-11-21
| | | | | | | | | | | service. Almost working, but there's a bug in the relaying. Also, made tor hidden service setup pick a random port, to make it harder to port scan. This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
* improve p2p protocol implementationGravatar Joey Hess2016-11-20
| | | | Tested it in ghci a little now.
* remotedaemon: serve tor hidden serviceGravatar Joey Hess2016-11-20
|
* implement p2p protocol for HandleGravatar Joey Hess2016-11-20
| | | | | | | | | | | This is most of the way to having the p2p protocol working over tor hidden services, at least enough to do git push/pull. The free monad was split into two, one for network operations and the other for local (Annex) operations. This will allow git-remote-tor-annex to run only an IO action, not needing the Annex monad. This commit was sponsored by Remy van Elst on Patreon.
* extend p2p protocol to support gitremote-helpers connectGravatar Joey Hess2016-11-19
| | | | | | | | A bit tricky since Proto doesn't support threads. Rather than adding threading support to it, ended up using a callback that waits for both data on a Handle, and incoming messages at the same time. This commit was sponsored by Denis Dzyubenko on Patreon.
* Add content locking to P2P protocolGravatar Joey Hess2016-11-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | Is content locking needed in the P2P protocol? Based on re-reading bugs/concurrent_drop--from_presence_checking_failures.mdwn, I think so: Peers can form cycles, and multiple peers can all be trying to drop the same content. So, added content locking to the protocol, with some difficulty. The implementation is fine as far as it goes, but note the warning comment for lockContentWhile -- if the connection to the peer is dropped unexpectedly, the peer will then unlock the content, and yet the local side will still think it's locked. To be honest I'm not sure if Remote.Git's lockKey for ssh remotes doesn't have the same problem. It checks that the "ssh remote git-annex-shell lockcontent" process has not exited, but if the connection closes afer that check, the lockcontent command will unlock it, and yet the local side will still think it's locked. Probably this needs to be fixed by eg, making lockcontent catch any execptions due to the connection closing, and in that case, wait a significantly long time before dropping the lock. This commit was sponsored by Anthony DeRobertis on Patreon.
* renameGravatar Joey Hess2016-11-17
|
* refactorGravatar Joey Hess2016-11-17
|