summaryrefslogtreecommitdiff
path: root/doc/bugs/S3_upload_not_using_multipart.mdwn
blob: ac3817c65ee99f5c91117afbfa4ae3522ed68631 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
What steps will reproduce the problem?

> Try to copy/move a file greater than 5G to S3.

    git annex copy large_file.tgz --to cloud

What is the expected output? What do you see instead?

> Looks like git-annex may not be using the Multipart Upload API: http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html

> Expected transfer to succeed, instead this error is output:

    copy large-file.tgz (gpg) (checking cloud...) (to cloud...) Reading passphrase from file descriptor 12    


     Your proposed upload exceeds the maximum allowed size
     failed
     git-annex: copy: 1 failed

What version of git-annex are you using? On what operating system?

> OSX 10.8.2

Please provide any additional information below.

    annex [master●] % git annex status
	supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
	supported remote types: git S3 bup directory rsync web webdav glacier hook
	repository mode: indirect
	trusted repositories: 0
	semitrusted repositories: 3
		00000000-0000-0000-0000-000000000001 -- web
	 	BE1D8EC7-C64B-47DE-AD4E-2A50437532B4 -- cloud
	 	E84568BA-6A4B-4AA1-B622-605B9248EDB1 -- here (eric laptop)
	untrusted repositories: 0
	dead repositories: 0
	transfers in progress: none
	available local disk space: 169 gigabytes (+1 megabyte reserved)
	temporary directory size: 218 megabytes (clean up with git-annex unused)
	local annex keys: 24
	local annex size: 8 gigabytes
	known annex keys: 25
	known annex size: 8 gigabytes
	bloom filter size: 16 mebibytes (0% full)
	backend usage: 
		SHA256E: 49
	annex [master●] % git annex version
	git-annex version: 3.20130114
	local repository version: 3
	default repository version: 3
	supported repository versions: 3
	upgrade supported from repository versions: 0 1 2

[[!tag confirmed]]

> [[fixed|done]] This is now supported, when git-annex is built with a new
> enough version of the aws library. You need to configure the remote to
> use an appropriate value for multipart, eg:
> 
> git annex enableremote cloud partsize=1GiB
> 
> --[[Joey]]