diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/bugs/S3_memory_leaks.mdwn | 4 | ||||
| -rw-r--r-- | doc/bugs/S3_upload_not_using_multipart.mdwn | 8 | ||||
| -rw-r--r-- | doc/special_remotes/S3.mdwn | 10 |
3 files changed, 20 insertions, 2 deletions
diff --git a/doc/bugs/S3_memory_leaks.mdwn b/doc/bugs/S3_memory_leaks.mdwn index 94bbdc398..7dc1e5757 100644 --- a/doc/bugs/S3_memory_leaks.mdwn +++ b/doc/bugs/S3_memory_leaks.mdwn @@ -2,9 +2,13 @@ S3 has memory leaks Sending a file to S3 causes a slow memory increase toward the file size. +> This is fixed, now that it uses aws. --[[Joey]] + Copying the file back from S3 causes a slow memory increase toward the file size. +> [[fixed|done]] too! --[[Joey]] + The author of hS3 is aware of the problem, and working on it. I think I have identified the root cause of the buffering; it's done by hS3 so it can resend the data if S3 sends it a 307 redirect. --[[Joey]] diff --git a/doc/bugs/S3_upload_not_using_multipart.mdwn b/doc/bugs/S3_upload_not_using_multipart.mdwn index 5e5d97c6a..cd40e9d2b 100644 --- a/doc/bugs/S3_upload_not_using_multipart.mdwn +++ b/doc/bugs/S3_upload_not_using_multipart.mdwn @@ -52,3 +52,11 @@ Please provide any additional information below. upgrade supported from repository versions: 0 1 2 [[!tag confirmed]] + +> [[fixed|done]] This is now supported, when git-annex is built with a new +> enough version of the aws library. You need to configure the remote to +> use an appropriate value for multipart, eg: +> +> git annex enableremote cloud multipart=1GiB +> +> --[[Joey]] diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn index fe46948b3..c7c6f76c5 100644 --- a/doc/special_remotes/S3.mdwn +++ b/doc/special_remotes/S3.mdwn @@ -21,6 +21,11 @@ the S3 remote. * `chunk` - Enables [[chunking]] when storing large files. `chunk=1MiB` is a good starting point for chunking. +* `partsize` - Specifies the largest object to attempt to store in the + bucket. Multipart uploads will be used when storing larger objects. + This is not enabled by default, but can be enabled or changed at any + time. Setting `partsize=1GiB` is reasonable for S3. + * `keyid` - Specifies the gpg key to use for [[encryption]]. * `embedcreds` - Optional. Set to "yes" embed the login credentials inside @@ -33,7 +38,8 @@ the S3 remote. embedcreds without gpg encryption. * `datacenter` - Defaults to "US". Other values include "EU", - "us-west-1", and "ap-southeast-1". + "us-west-1", "us-west-2", "ap-southeast-1", "ap-southeast-2", and + "sa-east-1". * `storageclass` - Default is "STANDARD". If you have configured git-annex to preserve multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY" @@ -52,5 +58,5 @@ the S3 remote. and to "bar/" in another special remote, and both special remotes could then use the same bucket. -* `x-amz-*` are passed through as http headers when storing keys +* `x-amz-meta-*` are passed through as http headers when storing keys in S3. |