summaryrefslogtreecommitdiff
path: root/Remote/S3.hs
diff options
context:
space:
mode:
authorGravatar Joey Hess <joey@kitenet.net>2014-11-04 16:06:13 -0400
committerGravatar Joey Hess <joey@kitenet.net>2014-11-04 16:06:13 -0400
commit2ba5af49c94b97c586220c3553367988ef095934 (patch)
treec8591d5bebf9ce66de94a638417a30189f995369 /Remote/S3.hs
parent4134c7f356afbd52bdbc660cba6ffd584cae7ee5 (diff)
work around minimum part size problem
When uploading the last part of a file, which was 640229 bytes, S3 rejected that part: "Your proposed upload is smaller than the minimum allowed size" I don't know what the minimum is, but the fix is just to include the last part into the previous part. Since this can result in a part that's double-sized, use half-sized parts normally.
Diffstat (limited to 'Remote/S3.hs')
-rw-r--r--Remote/S3.hs15
1 files changed, 11 insertions, 4 deletions
diff --git a/Remote/S3.hs b/Remote/S3.hs
index e0ff93bb3..8d30c7c9b 100644
--- a/Remote/S3.hs
+++ b/Remote/S3.hs
@@ -181,9 +181,16 @@ store r h = fileStorer $ \k f p -> do
}
uploadid <- S3.imurUploadId <$> sendS3Handle h startreq
- -- The actual part size will be a even multiple of the
- -- 32k chunk size that hGetUntilMetered uses.
- let partsz' = (partsz `div` toInteger defaultChunkSize) * toInteger defaultChunkSize
+ {- The actual part size will be a even multiple of the
+ - 32k chunk size that hGetUntilMetered uses.
+ -
+ - Also, half-size parts are used. This is so that
+ - the final part of a file can be rolled into the
+ - last full-size part, which avoids a problem when the
+ - final part could otherwise be too small for S3 to accept
+ - it.
+ -}
+ let partsz' = (partsz `div` toInteger defaultChunkSize `div` 2) * toInteger defaultChunkSize
-- Send parts of the file, taking care to stream each part
-- w/o buffering in memory, since the parts can be large.
@@ -195,7 +202,7 @@ store r h = fileStorer $ \k f p -> do
else do
-- Calculate size of part that will
-- be read.
- let sz = if fsz - pos < partsz'
+ let sz = if fsz - pos < partsz' * 2
then fsz - pos
else partsz'
let p' = offsetMeterUpdate p (toBytesProcessed pos)