summaryrefslogtreecommitdiff
path: root/doc
diff options
context:
space:
mode:
Diffstat (limited to 'doc')
-rw-r--r--doc/special_remotes/S3.mdwn9
-rw-r--r--doc/tips/public_Amazon_S3_remote.mdwn45
-rw-r--r--doc/tips/publishing_your_files_to_the_public.mdwn54
-rw-r--r--doc/tips/using_Amazon_S3.mdwn15
-rw-r--r--doc/todo/credentials-less_access_to_s3.mdwn2
-rw-r--r--doc/todo/credentials-less_access_to_s3/comment_3_26de94e8e3fefc9b47d1510bfb2dac9b._comment10
6 files changed, 107 insertions, 28 deletions
diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn
index 2b3d07e55..33f0410bb 100644
--- a/doc/special_remotes/S3.mdwn
+++ b/doc/special_remotes/S3.mdwn
@@ -48,6 +48,15 @@ the S3 remote.
so by default, a bucket name is chosen based on the remote name
and UUID. This can be specified to pick a bucket name.
+* `public` - Set to "yes" to allow public read access to files sent
+ to the S3 remote. This is accomplished by setting an ACL when each
+ file is uploaded to the remote. So, changes to this setting will
+ only affect subseqent uploads.
+
+* `publicurl` - Configure the URL that is used to download files
+ from the bucket when they are available publically.
+ (This is automatically configured for Amazon S3 and the Internet Archive.)
+
* `partsize` - Amazon S3 only accepts uploads up to a certian file size,
and storing larger files requires a multipart upload process.
diff --git a/doc/tips/public_Amazon_S3_remote.mdwn b/doc/tips/public_Amazon_S3_remote.mdwn
new file mode 100644
index 000000000..dd36b8350
--- /dev/null
+++ b/doc/tips/public_Amazon_S3_remote.mdwn
@@ -0,0 +1,45 @@
+Here's how to create a Amazon [[S3 special remote|special_remotes/S3]] that
+can be read by anyone who gets a clone of your git-annex repository,
+without them needing Amazon AWS credentials.
+
+## create remote
+
+First, export your Amazon AWS credentials:
+
+ # export AWS_ACCESS_KEY_ID="08TJMT99S3511WOZEP91"
+ # export AWS_SECRET_ACCESS_KEY="s3kr1t"
+
+Now, create the remote:
+
+ # git annex initremote pubs3 type=S3 encryption=none public=yes
+ initremote pubs3 (checking bucket) (creating bucket in US) ok
+
+In the above example, no encryption was used, but it will also work
+if you enable eg, encryption=shared. It's also ok to enable chunking.
+
+The public=yes is the crtical part; this lets the public read the contents
+of the bucket.
+
+Now, copy some files to the remote, in the usual way, and push your
+git repository to someplace where someone else can access it.
+
+## use public remote
+
+Once the S3 remote is set up, anyone who can clone the git repositry
+can get files from the remote, without needing any Amazon AWS credentials.
+
+Start by checking out the git repository.
+
+In the checkout, enable the S3 remote:
+
+ # git annex enableremote pubs3
+ enableremote pubs3 ok
+
+Now, git-annex can be used as usual to download files from that remote.
+
+----
+
+See [[special_remotes/S3]] for details about configuring S3 remotes.
+
+See [[publishing_your_files_to_the_public]] for other ways to use a public
+S3 bucket.
diff --git a/doc/tips/publishing_your_files_to_the_public.mdwn b/doc/tips/publishing_your_files_to_the_public.mdwn
index d2c074503..50fb64db3 100644
--- a/doc/tips/publishing_your_files_to_the_public.mdwn
+++ b/doc/tips/publishing_your_files_to_the_public.mdwn
@@ -2,28 +2,19 @@
(In this example, I'll assume you'll be creating a bucket in S3 named **public-annex** and a special remote in git-annex, which will store its files in the previous bucket, named **public-s3**, but change these names if you are going to do the thing for real)
-First, in the AWS dashboard, go to (or create) the bucket you will use at S3 and add a public get policy to it:
+Set up your special [S3](http://git-annex.branchable.com/special_remotes/S3/) remote with (at least) these options:
- {
- "Version": "2008-10-17",
- "Statement": [
- {
- "Sid": "AllowPublicRead",
- "Effect": "Allow",
- "Principal": {
- "AWS": "*"
- },
- "Action": "s3:GetObject",
- "Resource": "arn:aws:s3:::public-annex/*"
- }
- ]
- }
-
-Then set up your special [S3](http://git-annex.branchable.com/special_remotes/S3/) remote with (at least) these options:
+ git annex initremote public-s3 type=s3 encryption=none bucket=public-annex chunk=0 public=yes
- git annex initremote public-s3 type=s3 encryption=none bucket=public-annex chunk=0
+This way git-annex will upload the files to this repo, (when you call `git
+annex copy [FILES...] --to public-s3`) without encrypting them and without
+chunking them. And, thanks to the public=yes, they will be
+accessible by anyone with the link.
-This way git-annex will upload the files to this repo, (when you call `git annex copy [FILES...] --to public-s3`) without encrypting them and without chunking them, and, because of the policy of the bucket, they will be accessible by anyone with the link.
+(Note that public=yes was added in git-annex version 5.20150605.
+If you have an older version, it will be silently ignored, and you
+will instead need to use the AWS dashboard to configure a public get policy
+for the bucket.)
Following the example, the files will be accessible at `http://public-annex.s3.amazonaws.com/KEY` where `KEY` is the file key created by git-annex and which you can discover running
@@ -31,8 +22,6 @@ Following the example, the files will be accessible at `http://public-annex.s3.a
This way you can share a link to each file you have at your S3 remote.
-___________________
-
## Sharing all links in a folder
To share all the links in a given folder, for example, you can go to that folder and run (this is an example with the _fish_ shell, but I'm sure you can do the same in _bash_, I just don't know exactly):
@@ -74,3 +63,26 @@ done | git annex registerurl
</pre>
`registerurl` was introduced in `5.20150317`. There's a todo open to ensure we don't have to do this by hand: [[todo/credentials-less access to s3]].
+
+## Manually configuring a public get policy
+
+Here is how to manually configure a public get policy
+for a bucket, in the AWS dashboard.
+
+ {
+ "Version": "2008-10-17",
+ "Statement": [
+ {
+ "Sid": "AllowPublicRead",
+ "Effect": "Allow",
+ "Principal": {
+ "AWS": "*"
+ },
+ "Action": "s3:GetObject",
+ "Resource": "arn:aws:s3:::public-annex/*"
+ }
+ ]
+ }
+
+This should not be necessary if using a new enough version
+of git-annex, which can instead be configured with public=yet.
diff --git a/doc/tips/using_Amazon_S3.mdwn b/doc/tips/using_Amazon_S3.mdwn
index d6f621786..85d5bc958 100644
--- a/doc/tips/using_Amazon_S3.mdwn
+++ b/doc/tips/using_Amazon_S3.mdwn
@@ -22,16 +22,17 @@ Next, create the S3 remote, and describe it.
The configuration for the S3 remote is stored in git. So to make another
repository use the same S3 remote is easy:
- # cd /media/usb/annex
+ # export AWS_ACCESS_KEY_ID="08TJMT99S3511WOZEP91"
+ # export AWS_SECRET_ACCESS_KEY="s3kr1t"
# git pull laptop
# git annex enableremote cloud
enableremote cloud (gpg) (checking bucket) ok
-Now the remote can be used like any other remote.
+Notice that to enable an existing S3 remote, you have to provide the Amazon
+AWS credentials because they were not stored in the repository. (It is
+possible to configure git-annex to do that, but not the default.)
- # git annex copy my_cool_big_file --to cloud
- copy my_cool_big_file (gpg) (checking cloud...) (to cloud...) ok
- # git annex move video/hackity_hack_and_kaxxt.mov --to cloud
- move video/hackity_hack_and_kaxxt.mov (checking cloud...) (to cloud...) ok
+See [[public_Amazon_S3_remote]] for how to set up a Amazon S3 remote that
+can be used by the public, without them needing AWS credentials.
-See [[special_remotes/S3]] for details.
+See [[special_remotes/S3]] for details about configuring S3 remotes.
diff --git a/doc/todo/credentials-less_access_to_s3.mdwn b/doc/todo/credentials-less_access_to_s3.mdwn
index 39835ac1f..6816b3ff7 100644
--- a/doc/todo/credentials-less_access_to_s3.mdwn
+++ b/doc/todo/credentials-less_access_to_s3.mdwn
@@ -9,3 +9,5 @@ Besides, you never know if and when the file really is available on s3, so runni
How hard would it be to fix that in the s3 remote?
Thanks! --[[anarcat]]
+
+> [[done]] --[[Joey]]
diff --git a/doc/todo/credentials-less_access_to_s3/comment_3_26de94e8e3fefc9b47d1510bfb2dac9b._comment b/doc/todo/credentials-less_access_to_s3/comment_3_26de94e8e3fefc9b47d1510bfb2dac9b._comment
new file mode 100644
index 000000000..8a9ff4db2
--- /dev/null
+++ b/doc/todo/credentials-less_access_to_s3/comment_3_26de94e8e3fefc9b47d1510bfb2dac9b._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2015-06-05T20:17:38Z"
+ content="""
+The remote can indeed fallback when there are no creds.
+
+Also, git-annex can set an ACL on files it uploads, if the remote is
+configured with public=yes, so no manual ACL setting will be needed.
+"""]]