summaryrefslogtreecommitdiff
path: root/Remote
Commit message (Collapse)AuthorAge
* disable whereisKey for encrypted or chunked remotesGravatar Joey Hess2015-08-19
| | | | | | | This only makes sense for public repos, that are not chunked, so that there's a 1:1 from Key in the git-annex repo to file on the remote. Rather than making every remote implementation deal with that, just disable whereisKey when it doesn't make sense.
* make whereis show urls when web remote does not have contentGravatar Joey Hess2015-08-17
| | | | This is needed when external special remotes register an url for a key.
* External special remotes can now be built that can be used in readonly mode, ↵Gravatar Joey Hess2015-08-17
| | | | | | | | | | | | | | | | | where git-annex downloads content from the remote using regular http. Note that, if an url is added to the web log for such a remote, it's not distinguishable from another url that might be added for the web remote. (Because the web log doesn't distinguish which remote owns a plain url. Urls with a downloader set are distinguishable, but we're not using them here.) This seems ok-ish.. In such a case, both remotes will try to use both urls, and both remotes should be able to. The only issue I see is that dropping a file from the web remote will remove both urls in this case. This is not often done, and could even be considered a feature, I suppose.
* export some always failing methods for readonly remotesGravatar Joey Hess2015-08-17
|
* refactorGravatar Joey Hess2015-08-17
|
* refactorGravatar Joey Hess2015-08-17
|
* Added WHEREIS to external special remote protocol.Gravatar Joey Hess2015-08-13
|
* add some debugs to get timingsGravatar Joey Hess2015-08-13
| | | | | | | Note that I had one in Annex.Action.startup too, but it resulted in a weird message printed by ssh, "channel 2: bad ext data". I don't know why, but it only happened when transferinfo was run, so I wonder if 1d71ad072e13c8ed1cb8b34367b57d59e651f0a9 introduced a fragility somehow.
* --debug is passed along to git-annex-shell when git-annex is in debug mode.Gravatar Joey Hess2015-08-13
|
* Sped up downloads of files from ssh remotes, reducing the non-data-transfer ↵Gravatar Joey Hess2015-08-13
| | | | overhead 6x.
* remove debug printGravatar Joey Hess2015-08-13
|
* Added support for SHA3 hashed keys (in 8 varieties), when git-annex is built ↵Gravatar Joey Hess2015-08-06
| | | | | | | | using the cryptonite library. While cryptohash has SHA3 support, it has not been updated for the final version of the spec. Note that cryptonite has not been ported to all arches that cryptohash builds on yet.
* Simplify setup process for a ssh remote.Gravatar Joey Hess2015-08-05
| | | | | | | | | | | | | | | | | | | | | | Now it suffices to run git remote add, followed by git-annex sync. Now the remote is automatically initialized for use by git-annex, where before the git-annex branch had to manually be pushed before using git-annex sync. Note that this involved changes to git-annex-shell, so if the remote is using an old version, the manual push is still needed. Implementation required git-annex-shell be changed, so configlist can autoinit a repository even when no git-annex branch has been pushed yet. Unfortunate because we'll have to wait for it to get deployed to servers before being able to rely on this change in the documentation. Did consider making git-annex sync push the git-annex branch to repos that didn't have a uuid, but this seemed difficult to do without complicating it in messy ways. It would be cleaner to split a command out from configlist to handle the initialization. But this is difficult without sacrificing backwards compatability, for users of old git-annex versions which would not use the new command.
* remove unused importsGravatar Joey Hess2015-07-31
|
* Fix rsync special remote to work when -Jn is used for concurrent uploads.Gravatar Joey Hess2015-07-30
|
* Fix bug that prevented uploads to remotes using new-style chunking from ↵Gravatar Joey Hess2015-07-16
| | | | | | | | | | | | | | | | resuming after the last successfully uploaded chunk. "checkPresent baser" was wrong; the baser has a dummy checkPresent action not the real one. So, to fix this, we need to call preparecheckpresent to get a checkpresent action that can be used to check if chunks are present. Note that, for remotes like S3, this means that the preparer is run, which opens a S3 handle, that will be used for each checkpresent of a chunk. That's a good thing; if we're resuming an upload that's already many chunks in, it'll reuse that same http connection for each chunk it checks. Still, it's not a perfectly ideal thing, since this is a different http connection that the one that will be used to upload chunks. It would be nice to improve the API so that both use the same http connection.
* layoutGravatar Joey Hess2015-06-15
|
* tahoe: Use ~/.tahoe-git-annex/ rather than ~/.tahoe/git-annex/ to avoid old ↵Gravatar Joey Hess2015-06-09
| | | | versions of tahoe create-client choking.
* show S3 urls for public repos in whereisGravatar Joey Hess2015-06-05
| | | | | | Note that it's possible for a S3 bucket to be configured to allow public access, but for git-annex to not know that it is. I chose to not show the url unless public=yes.
* S3: Publically accessible buckets can be used without creds.Gravatar Joey Hess2015-06-05
|
* public=yes config to send AclPublicReadGravatar Joey Hess2015-06-05
| | | | | | | | | | | In my tests, this has to be set when uploading a file to the bucket and then the file can be accessed using the bucketname.s3.amazonaws.com url. Setting it when creating the bucket didn't seem to make the whole bucket public, or allow accessing files stored in it. But I have gone ahead and also sent it when creating the bucket just in case that is needed in some case.
* groundwork for readonly accessGravatar Joey Hess2015-06-05
| | | | Split S3Info out of S3Handle and added some stubs
* remove Params constructor from Utility.SafeCommandGravatar Joey Hess2015-06-01
| | | | | | | | | | | | | | | | | | This removes a bit of complexity, and should make things faster (avoids tokenizing Params string), and probably involve less garbage collection. In a few places, it was useful to use Params to avoid needing a list, but that is easily avoided. Problems noticed while doing this conversion: * Some uses of Params "oneword" which was entirely unnecessary overhead. * A few places that built up a list of parameters with ++ and then used Params to split it! Test suite passes.
* fromkey, registerurl: Allow urls to be specified instead of keys, and ↵Gravatar Joey Hess2015-05-22
| | | | | | | | generate URL keys. This is especially useful because the caller doesn't need to generate valid url keys, which involves some escaping of characters, and may involve taking a md5sum of the url if it's too long.
* use lock pools throughout git-annexGravatar Joey Hess2015-05-19
| | | | | | | | | | | | | The one exception is in Utility.Daemon. As long as a process only daemonizes once, which seems reasonable, and as long as it avoids calling checkDaemon once it's already running as a daemon, the fcntl locking gotchas won't be a problem there. Annex.LockFile has it's own separate lock pool layer, which has been renamed to LockCache. This is a persistent cache of locks that persist until closed. This is not quite done; lockContent stil needs to be converted.
* Avoid accumulating transfer failure log files unless the assistant is being ↵Gravatar Joey Hess2015-05-12
| | | | | | | | | | | | used. Only the assistant uses these, and only the assistant cleans them up, so make only git annex transferkeys write them, There is one behavior change from this. If glacier is being used, and a manual git annex get --from glacier fails because the file isn't available yet, the assistant will no longer later see that failed transfer file and retry the get. Hope no-one depended on that old behavior.
* Take space that will be used by running downloads into account when checking ↵Gravatar Joey Hess2015-05-12
| | | | annex.diskreserve.
* Merge branch 'master' into concurrentprogressGravatar Joey Hess2015-05-12
|\ | | | | | | | | | | | | | | | | | | | | | | Conflicts: Command/Fsck.hs Messages.hs Remote/Directory.hs Remote/Git.hs Remote/Helper/Special.hs Types/Remote.hs debian/changelog git-annex.cabal
| * generalied elem/notElem in ghc 7.10 require some additional type signatures ↵Gravatar Joey Hess2015-05-10
| | | | | | | | when using OverloadedStrings
| * remaining dataenc to sandi conversionsGravatar Joey Hess2015-05-07
| | | | | | | | | | | | I've tested all the dataenc to sandi conversions except Assistant.XMPP, and all have unchanged behavior, including behavior on large unicode code points.
| * S3: Fix incompatability with bucket names used by hS3; the aws library ↵Gravatar Joey Hess2015-04-27
| | | | | | | | | | | | | | | | | | cannot handle upper-case bucket names. git-annex now converts them to lower case automatically. For example, it failed to get files from a bucket named S3. Also fixes `git annex initremote UPPERCASE type=S3`, which failed with the new aws library, with a signing error message.
| * Fix bogus failure of fsck --fast.Gravatar Joey Hess2015-04-27
| |
| * S3: git annex enableremote will not create a bucket name, which failed since ↵Gravatar Joey Hess2015-04-23
| | | | | | | | the bucket already exists.
| * S3: git annex info will show additional information about a S3 remote ↵Gravatar Joey Hess2015-04-23
| | | | | | | | (endpoint, port, storage class)
| * convert all log prorities, not just debugGravatar Joey Hess2015-04-21
| | | | | | | | In particular, error should go to stderr
| * S3: Enable debug logging when annex.debug or --debug is set.Gravatar Joey Hess2015-04-21
| | | | | | | | To debug a bug report, but generally useful.
| * removed all uses of undefined from code baseGravatar Joey Hess2015-04-19
| | | | | | | | It's a code smell, can lead to hard to diagnose error messages.
| * Fix fsck --from a git remote in a local directory, and from a directory ↵Gravatar Joey Hess2015-04-18
| | | | | | | | | | | | | | | | | | | | | | special remote. This was a reversion caused by the relative path changes in 5.20150113. The directory special remote was not affected in its normal configuration, since annex-directory is an absolute path normally. But it could fail when a relative path was used. The git remote was affected even when an absolute path to it was used in .git/config, since git-annex now converts all such paths to relative.
* | add filename to progress bar, and display ok/failed at endGravatar Joey Hess2015-04-14
| | | | | | | | This needed plumbing an AssociatedFile through retrieveKeyFileCheap.
* | add missing progress barGravatar Joey Hess2015-04-14
| |
* | Merge branch 'master' into concurrentprogressGravatar Joey Hess2015-04-14
|\| | | | | | | | | Conflicts: debian/changelog
| * bittorrent: Fix handling of magnet links.Gravatar Joey Hess2015-04-14
| |
* | only display built-in meters in parallel modeGravatar Joey Hess2015-04-10
| |
* | use built-in progress meters for git when in parallel modeGravatar Joey Hess2015-04-10
|/
* relay external special remote stderr through progress suppression machinery ↵Gravatar Joey Hess2015-04-04
| | | | | | | | | | (eep!) It sounds worse than it is. ;) Some external special remotes may run commands that display progress on stderr. If git-annex is run with --quiet, this should filter out such displays while letting the errors through.
* well along the way to fully quiet --quietGravatar Joey Hess2015-04-04
| | | | | | | Came up with a generic way to filter out progress messages while keeping errors, for commands that use stderr for both. --json mode will disable command outputs too.
* propigate outer output state into inner state when running onLocalGravatar Joey Hess2015-04-03
| | | | | Otherwise, progress displays would not be suppressed here when running with --quiet. Interesting wrinkle!
* WIP on making --quiet silence progress, and infra for concurrent progress barsGravatar Joey Hess2015-04-03
|
* rename bothHandles -> ioHandlesGravatar Joey Hess2015-04-03
|
* Fix GETURLS in external special remote protocol to strip downloader prefix ↵Gravatar Joey Hess2015-03-27
| | | | | | | from logged url info before checking for the specified prefix. This doesn't change what GETURLS returns, but only whether it matches any prefix that the external special remote asked for.