summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
* work around minimum part size problemGravatar Joey Hess2014-11-04
| | | | | | | | | When uploading the last part of a file, which was 640229 bytes, S3 rejected that part: "Your proposed upload is smaller than the minimum allowed size" I don't know what the minimum is, but the fix is just to include the last part into the previous part. Since this can result in a part that's double-sized, use half-sized parts normally.
* fix a couple type errors and the progress barGravatar Joey Hess2014-11-04
|
* fix memory leakGravatar Joey Hess2014-11-04
| | | | | | | | | | Unfortunately, I don't fully understand why it was leaking using the old method of a lazy bytestring. I just know that it was leaking, despite neither hGetUntilMetered nor byteStringPopper seeming to leak by themselves. The new method avoids the lazy bytestring, and simply reads chunks from the handle and streams them out to the http socket.
* combine 2 checksGravatar Joey Hess2014-11-04
|
* casts; now fully working.. but still leakingGravatar Joey Hess2014-11-03
| | | | | Still seems to buffer the whole partsize in memory, but I'm pretty sure my code is not what's doing it. See https://github.com/aristidb/aws/issues/142
* this should avoid leaking memoryGravatar Joey Hess2014-11-03
|
* switch from hGetSome to hGetGravatar Joey Hess2014-11-03
| | | | | | | | | | | | This should be essentially no-op change for hGetContentsMetered, since it always gets the entire contents. So the only difference is that each chunk of the lazy bytestring will always be the full chunk size. So, I'm pretty sure this is safe. Also, the only current users of hGetContentsMetered are reading files, so the stream won't block for long in the middle. The improvement is that hGetUntilMetered will always get some multiple of the defaultChunkSize. This will allow the S3 multipart code to pick a fixed size and know that hGetUntilMetered will really get that size.
* logic errorGravatar Joey Hess2014-11-03
|
* WIP 3Gravatar Joey Hess2014-11-03
|
* WIP 2Gravatar Joey Hess2014-11-03
|
* WIP try sending using RequestBodyStreamChunkedGravatar Joey Hess2014-11-03
| | | | | May not work; if it does this is gonna be the simplest way to get good memory size and progress reporting.
* Merge branch 'master' into s3-aws-multipartGravatar Joey Hess2014-11-03
|\
| * hGetUntilMeteredGravatar Joey Hess2014-11-03
| |
* | link to memory leak bugGravatar Joey Hess2014-11-03
| |
| * commentGravatar Joey Hess2014-11-03
| |
* | improve info display for multipartGravatar Joey Hess2014-11-03
| |
* | fix buildGravatar Joey Hess2014-11-03
| |
* | adjust version checkGravatar Joey Hess2014-11-03
| | | | | | | | | | I assume 0.10.6 will have the fix for the bug I reported, which got fixed in master already..
* | show multipart configuration in git annex info s3remoteGravatar Joey Hess2014-11-03
| |
* | Merge branch 'master' into s3-aws-multipartGravatar Joey Hess2014-11-03
|\|
* | finish multipart support using unreleased update to aws lib to yield etagsGravatar Joey Hess2014-11-03
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Untested and not even compiled yet. Testing should include checks that file content streams through without buffering in memory. Note that CL.consume causes all the etags to be buffered in memory. This is probably nearly unavoidable, since a request has to be constructed that contains the list of etags in its body. (While it might be possible to stream generation of the body, that would entail making a http request that dribbles out parts of the body as the multipart uploads complete, which is not likely to work well.. To limit this being a problem, it's best for partsize to be set to some suitably large value, like 1gb. Then a full terabyte file will need only 1024 etags to be stored, which will probably use around 1 mb of memory.
| * commentGravatar Joey Hess2014-11-03
| |
| * todo itemGravatar Joey Hess2014-11-03
| |
| * commentGravatar Joey Hess2014-11-03
| |
| * followupGravatar Joey Hess2014-11-03
| |
| * close non-bugGravatar Joey Hess2014-11-03
| |
| * commentGravatar Joey Hess2014-11-03
| |
| * (no commit message)Gravatar https://gregy.mojeid.cz/#RQxWK6xlii2014-11-03
| |
| * (no commit message)Gravatar https://gregy.mojeid.cz/#RQxWK6xlii2014-11-03
| |
| * (no commit message)Gravatar https://gregy.mojeid.cz/#RQxWK6xlii2014-11-03
| |
| * (no commit message)Gravatar https://gregy.mojeid.cz/#RQxWK6xlii2014-11-03
| |
| * removedGravatar https://www.google.com/accounts/o8/id?id=AItOawlVB7wf1TFFrNPvM62fg78X-simVV-R1Cg2014-11-03
| |
| * Added a commentGravatar https://www.google.com/accounts/o8/id?id=AItOawlVB7wf1TFFrNPvM62fg78X-simVV-R1Cg2014-11-03
| |
| * Added a commentGravatar https://www.google.com/accounts/o8/id?id=AItOawlVB7wf1TFFrNPvM62fg78X-simVV-R1Cg2014-11-03
| |
| * (no commit message)Gravatar https://www.google.com/accounts/o8/id?id=AItOawk9SYh6N-JUMkYkW4aOk55zC3Vr9KonDV42014-11-02
| |
| * Added a comment: why worry about modifications ?Gravatar https://www.google.com/accounts/o8/id?id=AItOawlUGPPMvAP5Hu0NyNqeRMPC4pANeJNHZ0o2014-11-02
| |
| * (no commit message)Gravatar https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo2014-11-02
| |
| * (no commit message)Gravatar https://id.koumbit.net/lavamind2014-11-02
| |
| * new bug "git annex status does not report added files"Gravatar http://svario.it/gioele2014-11-01
| |
| * new wish "--dry-run option for all commands"Gravatar http://svario.it/gioele2014-11-01
| |
| * new forum post "Why is git annex status slow?"Gravatar http://svario.it/gioele2014-11-01
| |
| * (no commit message)Gravatar https://renaud.casenave.fr/2014-11-01
| |
| * commentGravatar Joey Hess2014-10-31
| |
| * commentGravatar Joey Hess2014-10-31
| |
| * comment and associated todoGravatar Joey Hess2014-10-31
| |
| * sssheeeshGravatar Joey Hess2014-10-31
| |
| * commentGravatar Joey Hess2014-10-31
| |
| * commentGravatar Joey Hess2014-10-31
| |
| * Windows: Fix crash when user.name is not set in git config.Gravatar Joey Hess2014-10-31
| |
| * commntGravatar Joey Hess2014-10-31
| |