summaryrefslogtreecommitdiff
path: root/Remote
Commit message (Collapse)AuthorAge
* don't support removing content from export with removeKeyGravatar Joey Hess2017-09-17
| | | | | | | | There does not seem to be a use case for supporting that, and it would need a lot of complication to support it in a way that allows eventual consistency when two repositories are updating the same export. This commit was sponsored by Henrik Riomar on Patreon.
* split out Types.ExportGravatar Joey Hess2017-09-15
|
* avoid unncessary db queries when exported directory can't be emptyGravatar Joey Hess2017-09-15
| | | | | | In rename foo/bar to foo/baz, foo can't be empty. In delete zxyyz, there's no exported directory (top doesn't count).
* webdav: Changed path used on webdav server for temporary files.Gravatar Joey Hess2017-09-15
| | | | | | | | | Done to avoid a "tmp" directory appearing in webdav exports. Also affects non-export webdav remotes, so interrupted uploads using the old path will not overwrite it. However, PUT is quite likely to be implemented atomically on web servers anyway, so I doubt this will cause problems.
* remove empty directories when removing from exportGravatar Joey Hess2017-09-15
| | | | | | | | | | | | | | | The subtle part of this is what happens when the remote fails to remove an empty directory. The removal from the export needs to fail in that case, so the removal will be tried again later. However, removeExportLocation has already been run and changed the export db, so if the next run checks getExportLocation, it might decide nothing remains to be done, leaving the empty directory. Dealt with that by making removeEmptyDirectories, handle a failure by calling addExportLocation, reverting the database changes so the next run will be guaranteed to try deleting the empty directory again. This commit was sponsored by Thomas Hochstein on Patreon.
* add missing caseGravatar Joey Hess2017-09-15
|
* implement removeExportDirectoryGravatar Joey Hess2017-09-15
| | | | | | | | | | | | | | | Not yet called by Command.Export. WebDAV needs this to clean up empty collections. Also, example.sh turned out to not be cleaning up directories when removing content from them, so it made sense for it to use this. Remote.Directory did not need it, and since its cleanup method for empty directories is more efficient than what Command.Export will need to do to find empty directories, it uses Nothing so that extra work can be avoided. This commit was sponsored by Thom May on Patreon.
* work around box.com webdav rename bugGravatar Joey Hess2017-09-13
| | | | | | | | | | | | | | | | Apparently box.com renaming is just buggy. I tried a couple of fixes: * In case the http Manager was opening multiple connections and reaching different backend servers, I tried limiting the number of connections to 1. Didn't help. * To make sure it was not a http connection reuse problem, I tried rewriting how exportAction works, so that the same http connection is clearly open. Didn't help. So, disable renaming of exports for box.com. It would be good to test it with some other webdav server. This commit was sponsored by John Peloquin on Patreon.
* fix exporting files in subdirectories to webdavGravatar Joey Hess2017-09-13
| | | | | | | Use tmp/key when exporting, so the whole export directory structure does not have to be created under tmp/ This commit was sponsored by Denis Dzyubenko on Patreon.
* clear exportSupported for non-export remotesGravatar Joey Hess2017-09-13
| | | | | Non-export remotes were being treated as untrusted, so the test suite failed, and probably other things broke.
* export: cache connections for S3 and webdavGravatar Joey Hess2017-09-12
|
* webdav: Fix lack of url-escaping of filenames.Gravatar Joey Hess2017-09-12
| | | | | | | | | | | | | inDAVLocation does not url-escape, and so exporting a filename with spaces to box.com at least resulted in a error 400. It might also have affected storing keys on a webdav remote, if the key contained a space or other problem character. Pretty unlikely. I emailed Clint about the inDAVLocation gotcha, but seems best to fix it here. This commit was supported by the NSF-funded DataLad project.
* deal with box.com horrible infinite redirect behaviorGravatar Joey Hess2017-09-12
| | | | | | | | | | | | | | | | | | | webdav: Checking if a non-existent file is present on Box.com triggered a bug in its webdav support that generates an infinite series of redirects. It seems to redirect foo to foo/ to foo/index.php to foo/index.php/index.php ... Why a webdav endpoint would behave this way who knows. Deal with such problems by assuming such behavior means the file is not present. Can't simply disable following redirects, because the webdav endpoint could legitimately be redirected to a new endpoint. So, when this happens 10 redirects have to be followed, before it gives up and assumes this means the file does not exist. This commit was supported by the NSF-funded DataLad project.
* export to webdavGravatar Joey Hess2017-09-12
| | | | | | | | | | | This basically works, but there's a bug when renaming a file that leaves a .git-annex-temp-content-key file in the webdav store, that never gets cleaned up. Also, exporting files with spaces to box.com seems to fail; perhaps it does not support it? This commit was supported by the NSF-funded DataLad project.
* update copyright yearGravatar Joey Hess2017-09-12
|
* propigate exception in checkPresentExportS3Gravatar Joey Hess2017-09-12
| | | | checkPresentExport is supposed to throw exceptions
* stop warning about removals from IAGravatar Joey Hess2017-09-12
| | | | | | | In a test, I uploaded a pdf, and several files were derived from it. After removing the pdf, the derived files went away after approximatly half an hour. This window does not seem worth warning about every time. Documented it in the tip.
* avoid showing error when copy failsGravatar Joey Hess2017-09-12
| | | | | | | | | | | | | Since renameExport is allowed to fail for any reason, and its failure is always recovered from by doing a new upload and deleting the old content, this avoids unnecessary noise. Copying a file on the IA failed, apparently something wrong with their emulation of S3: S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "InvalidArgument", s3ErrorMessage = "Invalid Argument", s3ErrorResource = Just "x-(amz|archive)-copy-source header is bad: 'joeyh-public-test2/foo'", s3ErrorHostId = Nothing, s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing} This commit was sponsored by Jake Vosloo on Patreon.
* S3: Allow removing files from IA, but warn about derived versions ↵Gravatar Joey Hess2017-09-12
| | | | | | | | | | | | | potentially still existing there. Removal works, only derives are a potential issue, so allow removing with a warning. This way, unexporting a file works, and behavior is consistent with IA remotes whether or not exporttree=yes. Also tested exporting filenames containing unicode, spaces, underscores. All worked, despite the IA's faq saying it doesn't. This commit was sponsored by Trenton Cronholm on Patreon.
* don't show key urls in whereis for S3 with public=yes and exporttree=yesGravatar Joey Hess2017-09-08
|
* S3 export finalizationGravatar Joey Hess2017-09-08
| | | | Fixed ACL issue, and updated some documentation.
* S3 export (untested)Gravatar Joey Hess2017-09-08
| | | | | | | | | | | It opens a http connection per file exported, but then so does git annex copy --to s3. Decided not to munge exported filenames for IA. Too large a chance of the munging having confusing results. Instead, export of files not supported by IA, eg with spaces in their name, will fail. This commit was supported by the NSF-funded DataLad project.
* External special remote protocol extended to support export.Gravatar Joey Hess2017-09-08
| | | | | | Also updated example.sh to support export. This commit was supported by the NSF-funded DataLad project.
* prevent exporttree=yes on remotes that don't support exportsGravatar Joey Hess2017-09-07
| | | | | | | | | Don't allow "exporttree=yes" to be set when the special remote does not support exports. That would be confusing since the user would set up a special remote for exports, but `git annex export` to it would later fail. This commit was supported by the NSF-funded DataLad project.
* rename bug fixGravatar Joey Hess2017-09-06
|
* add export infication to git-annex infoGravatar Joey Hess2017-09-04
|
* git annex get from exportsGravatar Joey Hess2017-09-04
| | | | | | | | | | | | | | Straightforward enough, except for the needed belt-and-suspenders sanity checks to avoid foot shooting due to exports not being key/value stores. * Even when annex.verify=false, always verify from exports. * Only get files from exports that use a backend that supports checksum verification. * Never trust exports, even if the user says to, because then `git annex drop` would drop content if the export seemed to contain a copy. This commit was supported by the NSF-funded DataLad project.
* implement exporttree=yes configurationGravatar Joey Hess2017-09-04
| | | | | | | | | | | | | | | | * Only export to remotes that were initialized to support it. * Prevent storing key/value on export remotes. * Prevent enabling exporttree=yes and encryption in the same remote. SetupStage Enable was changed to take the old RemoteConfig. This allowed only setting exporttree when initially setting up a remote, and not configuring it later after stuff might already be stored in the remote. Went with =yes rather than =true for consistency with other parts of git-annex. Changed docs accordingly. This commit was supported by the NSF-funded DataLad project.
* refactor ExportActionsGravatar Joey Hess2017-09-01
| | | | | | | | This will allow disabling exports for remotes that are not configured to allow them. Also, exportSupported will be useful for the external special remote to probe. This commit was supported by the NSF-funded DataLad project
* make storeExport atomicGravatar Joey Hess2017-08-31
| | | | | | | | | | | | | | | | | | This avoids needing to deal with the complexity of partially transferred files in the export. We'd not be able to resume uploading to such a file anyway, so just avoid them. The implementation in Remote.Directory is not completely ideal, because it could leave the temp file hanging around in the export directory. This only happens if it's killed with -9, or there's a power failure; normally viaTmp cleans up after itself, even when interrupted. I could not see a better way to do it though, since the export directory might be the root of a filesystem. Also some design thoughts on resuming, which depend on storeExport being atomic. This commit was sponsored by Fernando Jimenez on Partreon.
* remove empty parent dirs when removing from exportGravatar Joey Hess2017-08-31
|
* initial export commandGravatar Joey Hess2017-08-29
| | | | | | Very basic operation works, but of course this is only the beginning. This commit was sponsored by Nick Daly on Patreon.
* provide file with content to exportGravatar Joey Hess2017-08-29
| | | | | | | | | | | | | Rather than providing the key to export, provide the file. When exporting a treeish that contains files that are not annexed, this will let the content of those files also be exported. There's still a Key in the interface; it will be used by the external special remote protocol. A SHA1 key can be used when exporting non-annexed files. This commit was sponsored by Brock Spratlen on Patreon.
* add API for exportingGravatar Joey Hess2017-08-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implemented so far for the directory special remote. Several remotes don't make sense to export to. Regular Git remotes, obviously, do not. Bup remotes almost certianly do not, since bup would need to be used to extract the export; same store for Ddar. Web and Bittorrent are download-only. GCrypt is always encrypted so exporting to it would be pointless. There's probably no point complicating the Hook remotes with exporting at this point. External, S3, Glacier, WebDAV, Rsync, and possibly Tahoe should be modified to support export. Thought about trying to reuse the storeKey/retrieveKeyFile/removeKey interface, rather than adding a new interface. But, it seemed better to keep it separate, to avoid a complicated interface that sometimes encrypts/chunks key/value storage and sometimes users non-key/value storage. Any common parts can be factored out. Note that storeExport is not atomic. doc/design/exporting_trees_to_special_remotes.mdwn has some things in the "resuming exports" section that bear on this decision. Basically, I don't think, at this time, that an atomic storeExport would help with resuming, because exports are not key/value storage, and we can't be sure that a partially uploaded file is the same content we're currently trying to export. Also, note that ExportLocation will always use unix path separators. This is important, because users may export from a mix of windows and unix, and it avoids complicating the API with path conversions, and ensures that in such a mix, they always use the same locations for exports. This commit was sponsored by Bruno BEAUFILS on Patreon.
* avoid the dashed ssh hostname class of security holesGravatar Joey Hess2017-08-17
| | | | | | | | | | | | | | | | | | | | | | | | Security fix: Disallow hostname starting with a dash, which would get passed to ssh and be treated an option. This could be used by an attacker who provides a crafted ssh url (for eg a git remote) to execute arbitrary code via ssh -oProxyCommand. No CVE has yet been assigned for this hole. The same class of security hole recently affected git itself, CVE-2017-1000117. Method: Identified all places where ssh is run, by git grep '"ssh"' Converted them all to use a SshHost, if they did not already, for specifying the hostname. SshHost was made a data type with a smart constructor, which rejects hostnames starting with '-'. Note that git-annex already contains extensive use of Utility.SafeCommand, which fixes a similar class of problem where a filename starting with a dash gets passed to a program which treats it as an option. This commit was sponsored by Jochen Bartl on Patreon.
* external: nice error message for keys with spaces in their nameGravatar Joey Hess2017-08-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | External special remotes will refuse to operate on keys with spaces in their names. That has never worked correctly due to the design of the external special remote protocol. Display an error message suggesting migration. Not super happy with this, but it's a pragmatic solution. Better than complicating the external special remote interface and all external special remotes. Note that I only made it use SafeKey in Request, not Response. git-annex does not construct a Response, so that would not add any safety. And presumably, if git-annex avoids feeding any such keys to an external special remote, it will never have a reason to make a Response using such a key. If it did, it would result in a protocol error anyway. There's still a Serializeable instance for Key; it's used by P2P.Protocol. There, the Key is always in the final position, so it's ok if it contains spaces. Note that the protocol documentation has been fixed to say that the File may contain spaces. One way that can happen, even though the Key can't, is when using direct mode, and the work tree filename contains spaces. When sending such a file to the external special remote the worktree filename is used. This commit was sponsored by Thom May on Patreon.
* add annex-ignore-command and annex-sync-command configsGravatar Joey Hess2017-08-17
| | | | | | | | | | | | | | | | Added remote configuration settings annex-ignore-command and annex-sync-command, which are dynamic equivilants of the annex-ignore and annex-sync configurations. For this I needed a new DynamicConfig infrastructure. Its implementation should be as fast as before when there is no dynamic config, and it caches so shell commands are only run once. Note that annex-ignore-command exits nonzero when the remote should be ignored. While that may seem backwards, it allows using the same command for it as for annex-sync-command when you want to disable both. This commit was sponsored by Trenton Cronholm on Patreon.
* fix build with old http-client versionsGravatar Joey Hess2017-08-17
|
* Disable http-client's default 30 second response timeout when HEADing an url ↵Gravatar Joey Hess2017-08-15
| | | | to check if it exists. Some web servers take quite a long time to answer a HEAD request.
* adeiu, MissingHGravatar Joey Hess2017-05-16
| | | | | | | | | | | | | | | | Removed dependency on MissingH, instead depending on the split library. After laying groundwork for this since 2015, it was mostly straightforward. Added Utility.Tuple and Utility.Split. Eyeballed System.Path.WildMatch while implementing the same thing. Since MissingH's progress meter display was being used, I re-implemented my own. Bonus: Now progress is displayed for transfers of files of unknown size. This commit was sponsored by Shane-o on Patreon.
* de-Maybe remoteGitConfigGravatar Joey Hess2017-05-11
| | | | It's always set, so does not need to be a Maybe.
* gcrypt: Support re-enabling to change eg, encryption parameters.Gravatar Joey Hess2017-04-07
| | | | | | | | This was never supported before. And it doesn't re-encrypt the gcrypt repo to the new gcrypt-participants, but it does at least now not crash, and set gcrypt-participants. This commit was sponsored by andrea rota.
* When a http remote does not expose an annex.uuid config, only warn about it ↵Gravatar Joey Hess2017-03-29
| | | | | | once, not every time git-annex is run. Same behavior as for a ssh remote.
* Support GIT_SSH and GIT_SSH_COMMANDGravatar Joey Hess2017-03-17
| | | | | | | | | | | | | | | | | | | | They are handled close the same as they are by git. However, unlike git, git-annex sometimes needs to pass the -n parameter when using these. So, this has the potential for breaking some setup, and perhaps there ought to be a ANNEX_USE_GIT_SSH=1 needed to use these. But I'd rather avoid that if possible, so let's see if anyone complains. Almost all places where "ssh" was run have been changed to support the env vars. Anything still calling sshOptions does not support them. In particular, rsync special remotes don't. Seems that annex-rsync-transport already gives sufficient control there. (Fixed in passing: Remote.Helper.Ssh.toRepo used to extract remoteAnnexSshOptions and pass them to sshOptions, which was redundant since sshOptions also extracts those.) This commit was sponsored by Jeff Goeke-Smith on Patreon.
* AssociatedFile newtypeGravatar Joey Hess2017-03-10
| | | | | | To prevent any further mistakes like 1a497cefb47557f0b4788c606f9071be422b2511 This commit was sponsored by Francois Marier on Patreon.
* Windows: Improve handling of shebang in external special remote program, ↵Gravatar Joey Hess2017-03-08
| | | | | | | | | | | | | | | | | | | searching for the program in the PATH. findShellCommand needs a full path to a file in order to check it for a shebang on Windows. It was being run with only the base name of the external special remote program, which would only work when it was in the current directory. This is why users in https://github.com/DanielDent/git-annex-remote-rclone/pull/10 and elsewhere were complaining that the previous improvements to git-annex didn't make git-remote-rclone work on Windows. Also, reworked checkearlytermination, which while it worked, seemed to rely on a race condition. And, improved its error messages. This commit was sponsored by Shane-o on Patreon.
* sync hack to make updateInstead work on eg FATGravatar Joey Hess2017-02-17
| | | | | | | | | | | | sync: When syncing with a local repository located on a crippled filesystem, run the post-receive hook there, since it wouldn't get run otherwise. This makes pushing to repos on FAT-formatted removable drives update them when receive.denyCurrentBranch=updateInstead. Made Remote.Git export onLocal, which was cleaned up to not have so many caveats about its use. This commit was sponsored by Jeff Goeke-Smith on Patreon.
* have onLocal stop any coprocesses, not only cat-fileGravatar Joey Hess2017-02-17
| | | | | I have not seen any other coprocesses being started, but let's avoid problems if any do for whatever reason.
* Run ssh with -n whenever input is not being piped into itGravatar Joey Hess2017-02-15
| | | | | | | | | | | | | | | | | | | | ... to avoid it consuming stdin that it shouldn't. This fixes git-annex-checkpresentkey --batch remote, which didn't output results for all keys passed into it. Other git-annex commands that communicate with a remote over ssh may also have been consuming stdin that they shouldn't have, which could have impacted using them in eg, shell scripts. For example, a shell script reading files from stdin and passing them to git annex drop would be impacted by this bug, whenever git annex drop ran git-annex-shell checkpresent, it would consume part/all of the stdin that the shell script was supposed to consume. Fixed by adding a ConsumeStdin parameter to Annex.Ssh.sshOptions, which is used throughout git-annex to run ssh (in order for ssh connection caching to work). Every call site was checked to see if it used CreatePipe for stdin, and if not was marked NoConsumeStdin.
* S3: Fix check of uuid file stored in bucket, which was not working.Gravatar Joey Hess2017-02-13
| | | | | | | | | | | | | | The check was broken in two ways.. First, nowhere did it error out when checkUUIDFile found a different UUID already in the file. Instead, it overwrote the uuid file. And, checkUUIDFile's implementation was for some reason always failing with a ConnectionClosed exception. Apparently something to do with using two different runResourceT's and a response getting GCed inbetween. I'm pretty sure that used to work, but changed to a more obviously correct implementation. This commit was sponsored by Peter Hogg on Patreon.