diff options
Diffstat (limited to 'doc/special_remotes')
117 files changed, 1862 insertions, 0 deletions
diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn new file mode 100644 index 000000000..5291a4eb6 --- /dev/null +++ b/doc/special_remotes/S3.mdwn @@ -0,0 +1,53 @@ +This special remote type stores file contents in a bucket in Amazon S3 +or a similar service. + +See [[tips/using_Amazon_S3]] and +[[tips/Internet_Archive_via_S3]] for usage examples. + +## configuration + +The standard environment variables `AWS_ACCESS_KEY_ID` and +`AWS_SECRET_ACCESS_KEY` are used to supply login credentials +for Amazon. You need to set these only when running +`git annex initremote`, as they will be cached in a file only you +can read inside the local git repository. + +A number of parameters can be passed to `git annex initremote` to configure +the S3 remote. + +* `encryption` - One of "none", "hybrid", "shared", or "pubkey". + See [[encryption]]. + +* `keyid` - Specifies the gpg key to use for [[encryption]]. + +* `embedcreds` - Optional. Set to "yes" embed the login credentials inside + the git repository, which allows other clones to also access them. This is + the default when gpg encryption is enabled; the credentials are stored + encrypted and only those with the repository's keys can access them. + + It is not the default when using shared encryption, or no encryption. + Think carefully about who can access your repository before using + embedcreds without gpg encryption. + +* `datacenter` - Defaults to "US". Other values include "EU", + "us-west-1", and "ap-southeast-1". + +* `storageclass` - Default is "STANDARD". If you have configured git-annex + to preserve multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY" + to save money. + +* `host` and `port` - Specify in order to use a different, S3 compatable + service. + +* `bucket` - S3 requires that buckets have a globally unique name, + so by default, a bucket name is chosen based on the remote name + and UUID. This can be specified to pick a bucket name. + +* `fileprefix` - By default, git-annex places files in a tree rooted at the + top of the S3 bucket. When this is set, it's prefixed to the filenames + used. For example, you could set it to "foo/" in one special remote, + and to "bar/" in another special remote, and both special remotes could + then use the same bucket. + +* `x-amz-*` are passed through as http headers when storing keys + in S3. diff --git a/doc/special_remotes/S3/comment_10_c366f020c9b97a365e21878a33360079._comment b/doc/special_remotes/S3/comment_10_c366f020c9b97a365e21878a33360079._comment new file mode 100644 index 000000000..bc06156cf --- /dev/null +++ b/doc/special_remotes/S3/comment_10_c366f020c9b97a365e21878a33360079._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 10" + date="2013-05-23T20:04:03Z" + content=""" +You can enable a special remote on a clone by running `git annex enableremote $name`, where $name is the name you used to originally create the special remote. (Older versions of git-annex used `git annex initremote` to enable the special remote on the clone.) + +(Just in case, I have verified that embedcreds does cause the cipher= to be stored in the remote.log. It does.) +"""]] diff --git a/doc/special_remotes/S3/comment_11_c1da387e082d91feec13dde91ccb111a._comment b/doc/special_remotes/S3/comment_11_c1da387e082d91feec13dde91ccb111a._comment new file mode 100644 index 000000000..5cd9b7311 --- /dev/null +++ b/doc/special_remotes/S3/comment_11_c1da387e082d91feec13dde91ccb111a._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="basak" + ip="2001:8b0:1c8::2" + subject="comment 11" + date="2013-05-24T09:38:40Z" + content=""" +Thanks Joey - initremote on my slightly older version appears to work. I'll use `enableremote` when I can. + +> (Just in case, I have verified that embedcreds does cause the cipher= to be stored in the remote.log. It does.) + +This doesn't do what I expect. The documentation suggests that my S3 _login_ credentials would be stored. I understand that the cipher would be stored; but isn't this a separate concept? Instead, I'm being asked to set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`; my understanding was that git-annex will keep them in the repository for me, so that I don't have to set them after running `initremote` before cloning. This works, apart from surviving the cloning. I'm using `encryption=shared`; does this affect anything? Or am I using a version of git-annex (3.20121112ubuntu3) that's too old? +"""]] diff --git a/doc/special_remotes/S3/comment_12_59c3ecab7dbc8be53258460473cac21c._comment b/doc/special_remotes/S3/comment_12_59c3ecab7dbc8be53258460473cac21c._comment new file mode 100644 index 000000000..47b0aaefa --- /dev/null +++ b/doc/special_remotes/S3/comment_12_59c3ecab7dbc8be53258460473cac21c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 12" + date="2013-05-24T15:33:12Z" + content=""" +Ah -- No, your AWS creds are not stored. While some other special remotes, like webdav, can store all necessary credentials, it's not done for AWS. I didn't want git-annex to be responsible for someone accidentially publishing their AWS creds to their friends, since that could cost them a lot of money. +"""]] diff --git a/doc/special_remotes/S3/comment_13_0789a21d980825188bb09f7fc8bba8be._comment b/doc/special_remotes/S3/comment_13_0789a21d980825188bb09f7fc8bba8be._comment new file mode 100644 index 000000000..c8e84021b --- /dev/null +++ b/doc/special_remotes/S3/comment_13_0789a21d980825188bb09f7fc8bba8be._comment @@ -0,0 +1,33 @@ +[[!comment format=mdwn + username="basak" + ip="2001:8b0:1c8::2" + subject="comment 13" + date="2013-05-24T15:47:14Z" + content=""" +That's not what the documentation here says! It even warns me: \"Think carefully about who can access your repository before using embedcreds without gpg encryption.\" + +My use case: + +Occasional use of EC2, and a desire to store some persistent stuff in S3, since the dataset is large and I have limited bandwidth. I want to destroy the EC2 instance when I'm not using it, leaving the data in S3 for later. + +If I use git-annex to manage the S3 store, then I get the ability to clone the repository and destroy the instance. Later, I can start a new instance, push the repo back up, and would like to be able to then pull the data back out of S3 again. + +I'd really like the login credentials to persist in the repository (as the documentation here says it should). Even if I have to add a --yes-i-know-my-s3-credentials-will-end-up-available-to-anyone-who-can-see-my-git-repo flag. This is because I use some of my git repos to store private data, too. + +If I use an Amazon IAM policy as follows, I can generate a set of credentials that are limited to access to a particular prefix of a specific S3 bucket only - effectively creating a sandboxed area just for git-annex: + + { + \"Statement\": [{\"Sid\": \"Stmt1368780615583\", + \"Action\": [\"s3:GetObject\", \"s3:PutObject\", \"s3:DeleteObject\"], + \"Effect\": \"Allow\", + \"Resource\": [\"arn:aws:s3:::bucketname/prefix/*\"]} + ], + \"Statement\": [{\"Sid\": \"Stmt1368781573129\", + \"Action\": [\"s3:GetBucketLocation\"], + \"Effect\": \"Allow\", + \"Resource\": [\"arn:aws:s3:::bucketname\"]} + ] + } + +Doing this means that I have a different set of credentials for every annex, so it would be really useful to be able have these stored and managed within the repository itself. Each set is limited to what the annex stores, so there is no bigger compromise I have to worry about apart from the compromise of the data that the annex itself manages. +"""]] diff --git a/doc/special_remotes/S3/comment_14_29574a51d5831c51e2e765eb2c06e567._comment b/doc/special_remotes/S3/comment_14_29574a51d5831c51e2e765eb2c06e567._comment new file mode 100644 index 000000000..2285e9062 --- /dev/null +++ b/doc/special_remotes/S3/comment_14_29574a51d5831c51e2e765eb2c06e567._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 14" + date="2013-05-24T16:45:25Z" + content=""" +I apologise for incorrect information. I was thinking about defaults when using the webapp. + +I have verified that embedcreds=yes stores the AWS creds, always. + + +"""]] diff --git a/doc/special_remotes/S3/comment_15_ceb9048c743135f6beca57a23505f0a3._comment b/doc/special_remotes/S3/comment_15_ceb9048c743135f6beca57a23505f0a3._comment new file mode 100644 index 000000000..a8b5573e7 --- /dev/null +++ b/doc/special_remotes/S3/comment_15_ceb9048c743135f6beca57a23505f0a3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawne_amN4fko4p5cRY_9EYwaYuJKNn7LRio" + nickname="Tobias" + subject="different s3 storage URLs" + date="2013-08-23T08:59:32Z" + content=""" +Is it possible to change the S3 endpoint hosts? I'm running a radosgw with S3 support which I'd like to define as S3 remote for git-annex +"""]] diff --git a/doc/special_remotes/S3/comment_16_7b79f8b5ef88a2775d61b5ac5774d3e0._comment b/doc/special_remotes/S3/comment_16_7b79f8b5ef88a2775d61b5ac5774d3e0._comment new file mode 100644 index 000000000..508cedca4 --- /dev/null +++ b/doc/special_remotes/S3/comment_16_7b79f8b5ef88a2775d61b5ac5774d3e0._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 16" + date="2013-08-23T17:39:56Z" + content=""" +Yes, you can specify the host to use when setting up the remote. It's actually documented earlier on this very page, if ou search for \"host\". Any S3 compatabile host will probably work -- the Internet Archive's S3 does, for example. +"""]] diff --git a/doc/special_remotes/S3/comment_1_4a1f7a230dad6caa84831685b236fd73._comment b/doc/special_remotes/S3/comment_1_4a1f7a230dad6caa84831685b236fd73._comment new file mode 100644 index 000000000..17e35e7d9 --- /dev/null +++ b/doc/special_remotes/S3/comment_1_4a1f7a230dad6caa84831685b236fd73._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnoUOqs_lbuWyZBqyU6unHgUduJwDDgiKY" + nickname="Matt" + subject="environment variables" + date="2012-05-29T12:40:25Z" + content=""" +Just noting that the environment variables `ANNEX_S3_ACCESS_KEY_ID` and `ANNEX_S3_SECRET_ACCESS_KEY` seem to have been changed to `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` +"""]] diff --git a/doc/special_remotes/S3/comment_2_5b22d67de946f4d34a4a3c7449d32988._comment b/doc/special_remotes/S3/comment_2_5b22d67de946f4d34a4a3c7449d32988._comment new file mode 100644 index 000000000..f535559ae --- /dev/null +++ b/doc/special_remotes/S3/comment_2_5b22d67de946f4d34a4a3c7449d32988._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.81.112" + subject="comment 2" + date="2012-05-29T19:10:46Z" + content=""" +Thanks, I've fixed that. (You could have too.. this is a wiki ;) +"""]] diff --git a/doc/special_remotes/S3/comment_3_bcab2bd0f168954243aa9bcc9671bd94._comment b/doc/special_remotes/S3/comment_3_bcab2bd0f168954243aa9bcc9671bd94._comment new file mode 100644 index 000000000..abb6aacc9 --- /dev/null +++ b/doc/special_remotes/S3/comment_3_bcab2bd0f168954243aa9bcc9671bd94._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnoUOqs_lbuWyZBqyU6unHgUduJwDDgiKY" + nickname="Matt" + subject="comment 3" + date="2012-05-30T00:26:33Z" + content=""" +Thanks! Being new here, I didn't want to overstep my boundaries. I've gone ahead and made a small edit and will do so elsewhere as needed. +"""]] diff --git a/doc/special_remotes/S3/comment_4_38c0b062997fde1ad28facc05d973e83._comment b/doc/special_remotes/S3/comment_4_38c0b062997fde1ad28facc05d973e83._comment new file mode 100644 index 000000000..8c17f7d64 --- /dev/null +++ b/doc/special_remotes/S3/comment_4_38c0b062997fde1ad28facc05d973e83._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmX5gPNK35Dub-HzR0Yb3KXllbqc0rYRYs" + nickname="Eduardo" + subject="bucket/folder s3 remotes" + date="2012-08-09T10:52:07Z" + content=""" +it'd be really nice being able to configure a S3 remote of the form `<bucket>/<folder>` (not really a folder, of course, just the usual prefix trick used to simulate folders at S3). The remote = bucket architecture is not scalable at all, in terms of number of repositories. + +how hard would it be to support this? + +thanks, this is the only thing that's holding us back from using git-annex, nice tool! +"""]] diff --git a/doc/special_remotes/S3/comment_5_409bc2b56382417cf26bb222fb783ba7._comment b/doc/special_remotes/S3/comment_5_409bc2b56382417cf26bb222fb783ba7._comment new file mode 100644 index 000000000..325db6799 --- /dev/null +++ b/doc/special_remotes/S3/comment_5_409bc2b56382417cf26bb222fb783ba7._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + subject="comment 5" + date="2012-08-09T18:01:06Z" + content=""" +I guess this could be useful if you have a *lot* of buckets already in use at S3, or if you want to be able to have a lot of distinct S3 special remotes. Implemented the `fileprefix` setting. Note that I have not tested it, beyond checking it builds, since I let my S3 account expire. Your testing would be appreciated. + +"""]] diff --git a/doc/special_remotes/S3/comment_6_78da9e233882ec0908962882ea8c4056._comment b/doc/special_remotes/S3/comment_6_78da9e233882ec0908962882ea8c4056._comment new file mode 100644 index 000000000..742dbedc2 --- /dev/null +++ b/doc/special_remotes/S3/comment_6_78da9e233882ec0908962882ea8c4056._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnY9ObrNrQuRp8Xs0XvdtJJssm5cp4NMZA" + nickname="alan" + subject="Rackspace Cloud Files support?" + date="2012-08-23T21:00:11Z" + content=""" +Any chance I could bribe you to setup Rackspace Cloud Files support? We are using them and would hate to have a S3 bucket only for this. + +https://github.com/rackspace/python-cloudfiles +"""]] diff --git a/doc/special_remotes/S3/comment_7_6af9781004d982d8e6b20a83ad29eead._comment b/doc/special_remotes/S3/comment_7_6af9781004d982d8e6b20a83ad29eead._comment new file mode 100644 index 000000000..c243cde13 --- /dev/null +++ b/doc/special_remotes/S3/comment_7_6af9781004d982d8e6b20a83ad29eead._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmyFvkaewo432ELwtCoecUGou4v3jCP0Pc" + nickname="Eric" + subject="S3 Remote Future Proof?" + date="2013-01-20T09:21:50Z" + content=""" +Joey, I'm curious to understand how future proof an S3 remote is. Can I restore my files without git-annex? +"""]] diff --git a/doc/special_remotes/S3/comment_8_0fa68d584ee7f6b5c9058fba7e911a11._comment b/doc/special_remotes/S3/comment_8_0fa68d584ee7f6b5c9058fba7e911a11._comment new file mode 100644 index 000000000..6997719d1 --- /dev/null +++ b/doc/special_remotes/S3/comment_8_0fa68d584ee7f6b5c9058fba7e911a11._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="68.119.100.5" + subject="comment 8" + date="2013-01-20T20:37:09Z" + content=""" +If encryption is not used, the files are stored in S3 as-is, and can be accessed directly. They are stored in a hashed directory structure with the names of their key used, rather than the original filename. To get back to the original filename, a copy of the git repo would also be needed. + +With encryption, you need the gpg key used in the encryption, or, for shared encryption, a symmetric key which is stored in the git repo. + +See [[future proofing]] for non-S3 specific discussion of this topic. +"""]] diff --git a/doc/special_remotes/S3/comment_9_7ad757b3865b04967c79af0a263bb3b0._comment b/doc/special_remotes/S3/comment_9_7ad757b3865b04967c79af0a263bb3b0._comment new file mode 100644 index 000000000..51b7ab16a --- /dev/null +++ b/doc/special_remotes/S3/comment_9_7ad757b3865b04967c79af0a263bb3b0._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="basak" + ip="2001:8b0:1c8::2" + subject="Recovering from a clone" + date="2013-05-22T18:32:05Z" + content=""" +How do I recover a special remote from a clone, please? I see that `remote.log` has most of the details, but my remote is not configured on my clone and I see no obvious way to do it. And I used `embedcreds`, but the only credentials I can see are stored in .git/annex/creds/ so did not survive a clone. I'm confused because the documentation here for `embedcreds` says that clones should have access. + +As a workaround, it looks like copying the remote over from `.git/config` as well as the credentials from `.git/annex/creds/` seems to work. Is there some other way I'm supposed to do this, or is this the intended way? +"""]] diff --git a/doc/special_remotes/bup.mdwn b/doc/special_remotes/bup.mdwn new file mode 100644 index 000000000..f2d465e77 --- /dev/null +++ b/doc/special_remotes/bup.mdwn @@ -0,0 +1,39 @@ +This special remote type stores file contents in a +[bup](http://github.com/bup/bup) repository. By using git-annex +in the front-end, and bup as a remote, you get an easy git-style +interface to large files, and easy backups of the file contents using git. + +This is particularly well suited to collaboration on projects involving +large files, since both the git-annex and bup repositories can be +accessed like any other git repository. + +See [[walkthrough/using_bup]] for usage examples. + +Each individual key is stored in a bup remote using `bup split`, with +a git branch named the same as the key name. Content is retrieved from +bup using `bup join`. All other bup operations are up to you -- consider +running `bup fsck --generate` in a cron job to generate recovery blocks, +for example; or clone bup's git repository to further back it up. + +## configuration + +These parameters can be passed to `git annex initremote` to configure bup: + +* `encryption` - One of "none", "hybrid", "shared", or "pubkey". + See [[encryption]]. + +* `keyid` - Specifies the gpg key to use for [[encryption]]. + +* `buprepo` - Required. This is passed to `bup` as the `--remote` + to use to store data. To create the repository,`bup init` will be run. + Example: "buprepo=example.com:/big/mybup" or "buprepo=/big/mybup" + (To use the default `~/.bup` repository on the local host, specify "buprepo=") + +Options to pass to `bup split` when sending content to bup can also +be specified, by using `git config annex.bup-split-options`. This +can be used to, for example, limit its bandwidth. + +## notes + +[[git-annex-shell]] does not support bup, due to the wacky way that bup +starts its server. So, to use bup, you need full shell access to the server. diff --git a/doc/special_remotes/bup/comment_10_f78c1ed97d2e4c6ebffaa7482cfe0c9b._comment b/doc/special_remotes/bup/comment_10_f78c1ed97d2e4c6ebffaa7482cfe0c9b._comment new file mode 100644 index 000000000..97f9b9ea9 --- /dev/null +++ b/doc/special_remotes/bup/comment_10_f78c1ed97d2e4c6ebffaa7482cfe0c9b._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="http://sekenre.wordpress.com/" + nickname="sekenre" + subject="Synchronizing Bup repositories" + date="2013-05-07T16:46:34Z" + content=""" +Hi All, + +I managed to answer my questions above about copying changes between local bup repositories efficiently. + +You run the following commands + + git annex copy . --to bup_repo_1 # Uses bup split in the background (slow) + rsync -av /mnt/repodisk1/repo/ /mnt/repodisk2/repo/ \ + --exclude=config --exclude=*.bloom --exclude=*.midx # rsync without bup-specific indices (speed depends on delta between repositories) + BUP_DIR=/mnt/repodisk2/repo/ bup midx -a && bup bloom # rebuild bup-specific indices on the target (this is extremely fast) + git annex copy . --to bup_repo_2 # Records file is now available in repo2 (also extremely fast) + +Now `git annex whereis` will show the correct location and `git annex get <file> --from bup_repo_2` will work. + +So far in my testing I haven't found any problems... + +"""]] diff --git a/doc/special_remotes/bup/comment_11_b53bceb0058acf4d1ab12ea4853ee443._comment b/doc/special_remotes/bup/comment_11_b53bceb0058acf4d1ab12ea4853ee443._comment new file mode 100644 index 000000000..c699c6241 --- /dev/null +++ b/doc/special_remotes/bup/comment_11_b53bceb0058acf4d1ab12ea4853ee443._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnAvbXOnK57sqgvZvxkbG74NUKBDwKDcuk" + nickname="Tim" + subject="bup location data not synced through annex assistant" + date="2013-05-15T15:08:54Z" + content=""" +I set up 2 servers running git annex assistant, both with a ~/annex dir and an additional ~/annex-bup bup repo. There is no additional cloud repository. +For test, I added my /etc dir which uploaded correctly from server1, but which never arrived on server2 + + bup@bup1:~/annex/etc$ git annex whereis updatedb.conf + whereis updatedb.conf (3 copies) + 687d3a7f-4798-4dbe-8774-1785b8ab6b7d -- here (bup@bup1:~/annex) + adfc1307-771f-40e9-b794-bae2e1f21b8b -- bup2-annex-bup + e4e0ac0b-992a-4312-a4ac-fc8d3d9f7c0f -- bup1-annex-bup + ok + + bup@bup2:~/annex/etc$ git annex whereis updatedb.conf + whereis updatedb.conf (1 copy) + 687d3a7f-4798-4dbe-8774-1785b8ab6b7d -- bup1 (bup@bup1:~/annex) + ok + +As you can see, server 2 just doesn't know the data is already on it's own disk in it's local bup repo. +Is there a reason this data does not get synced? Should I set up a transfer repo? +"""]] diff --git a/doc/special_remotes/bup/comment_12_65d923226cf6120349d807c5c60f640c._comment b/doc/special_remotes/bup/comment_12_65d923226cf6120349d807c5c60f640c._comment new file mode 100644 index 000000000..a790daea3 --- /dev/null +++ b/doc/special_remotes/bup/comment_12_65d923226cf6120349d807c5c60f640c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnAvbXOnK57sqgvZvxkbG74NUKBDwKDcuk" + nickname="Tim" + subject="my bad" + date="2013-05-15T15:39:31Z" + content=""" +Sorry, looks like I did initremote twice on the same folder, instead of enableremote the second time... +"""]] diff --git a/doc/special_remotes/bup/comment_1_96179a003da4444f6fc08867872cda0a._comment b/doc/special_remotes/bup/comment_1_96179a003da4444f6fc08867872cda0a._comment new file mode 100644 index 000000000..02691c690 --- /dev/null +++ b/doc/special_remotes/bup/comment_1_96179a003da4444f6fc08867872cda0a._comment @@ -0,0 +1,43 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkgbXwQtPQSG8igdS7U8l031N8sqDmuyvk" + nickname="Albert" + subject="Error with bup and gnupg" + date="2012-10-22T20:56:56Z" + content=""" +Hello, + +I get this error when trying to use git-annex with bup and gnupg: + +<pre> +move importable_pilot_surveys.tar (gpg) (checking localaseebup...) (to localaseebup...) +Traceback (most recent call last): + File \"/usr/lib/bup/cmd/bup-split\", line 133, in <module> + progress=prog) + File \"/usr/lib/bup/bup/hashsplit.py\", line 167, in split_to_shalist + for (sha,size,bits) in sl: + File \"/usr/lib/bup/bup/hashsplit.py\", line 118, in _split_to_blobs + for (blob, bits) in hashsplit_iter(files, keep_boundaries, progress): + File \"/usr/lib/bup/bup/hashsplit.py\", line 86, in _hashsplit_iter + bnew = next(fi) + File \"/usr/lib/bup/bup/helpers.py\", line 86, in next + return it.next() + File \"/usr/lib/bup/bup/hashsplit.py\", line 49, in blobiter + for filenum,f in enumerate(files): + File \"/usr/lib/bup/cmd/bup-split\", line 128, in <genexpr> + files = extra and (open(fn) for fn in extra) or [sys.stdin] +IOError: [Errno 2] No such file or directory: '-' +</pre> + + +I was able to work-around this issue by altering /usr/lib/bup/cmd/bup-split (though I don't think its a bup bug) to just pull from stdin: + +files = [sys.stdin] + +on ~ line 128. + +Any ideas? Also, do you think that bup's data-deduplication does anything when gnupg is enabled, i.e. is it just as well to use a directory remote with gnupg? + +Thanks! Git annex rules! + +Albert +"""]] diff --git a/doc/special_remotes/bup/comment_2_612b038c15206f9f3c2e23c7104ca627._comment b/doc/special_remotes/bup/comment_2_612b038c15206f9f3c2e23c7104ca627._comment new file mode 100644 index 000000000..97af184f3 --- /dev/null +++ b/doc/special_remotes/bup/comment_2_612b038c15206f9f3c2e23c7104ca627._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.23" + subject="comment 2" + date="2012-10-23T20:01:43Z" + content=""" +@Albert, thanks for reporting this bug (but put them in [[bugs]] in future please). + +This is specific to using the bup special remote with encryption. Without encryption it works. And no, it won't manage to deduplicate anything that's encrypted, as far as I know. + +I think bup-split must have used - for stdin in the past, but now, it just reads from stdin when no file is specified, so I've updated git-annex. +"""]] diff --git a/doc/special_remotes/bup/comment_3_1186def82741ddab1ade256fb2e59e6f._comment b/doc/special_remotes/bup/comment_3_1186def82741ddab1ade256fb2e59e6f._comment new file mode 100644 index 000000000..2e3e38992 --- /dev/null +++ b/doc/special_remotes/bup/comment_3_1186def82741ddab1ade256fb2e59e6f._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="http://sekenre.wordpress.com/" + nickname="sekenre" + subject="Bup remotes in git-annex assistant" + date="2013-03-13T12:54:56Z" + content=""" +Hi, + +Is the bup remote available via the Assistant user interface? + +Unrelated question; + +If you are syncing files between two bup repos on local usb drives, does it use git to sync the changes or does it use \"bup split\" to re-add the file? (Basically, is the syncing as efficient as possible using git-annex or would I have to go to a lower level) + +Many Thanks, +Sek +"""]] diff --git a/doc/special_remotes/bup/comment_4_7d22a805dd2914971e7ca628ceea69be._comment b/doc/special_remotes/bup/comment_4_7d22a805dd2914971e7ca628ceea69be._comment new file mode 100644 index 000000000..0607e5f6c --- /dev/null +++ b/doc/special_remotes/bup/comment_4_7d22a805dd2914971e7ca628ceea69be._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 4" + date="2013-03-13T16:05:50Z" + content=""" +I don't plan to support creating bup spefial remotes in the assistant, currently. Of course the assistant can use bup special remotes you set up. + +Your two bup repos would be synced using bup-split. +"""]] diff --git a/doc/special_remotes/bup/comment_6_5942333cde09fd98e26c4f1d389cb76f._comment b/doc/special_remotes/bup/comment_6_5942333cde09fd98e26c4f1d389cb76f._comment new file mode 100644 index 000000000..288f8765d --- /dev/null +++ b/doc/special_remotes/bup/comment_6_5942333cde09fd98e26c4f1d389cb76f._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnZWCbRYPVnwscdkdEDwgQHZJLwW6H_AHo" + nickname="Tobias" + subject="bup fail?" + date="2013-03-31T21:05:32Z" + content=""" +I've run into problems storing a huge number of files in the bup repo. It seems that thousands of branches are a problem. I don't know if it's a problem of git-annex, bup, or the filesystem. + +How about adding an option to store tree/commit ids in git-annex instead of using branches in bup? +"""]] diff --git a/doc/special_remotes/bup/comment_7_cb1a0d3076e9d06e7a24204478f6fa98._comment b/doc/special_remotes/bup/comment_7_cb1a0d3076e9d06e7a24204478f6fa98._comment new file mode 100644 index 000000000..a2284c3e1 --- /dev/null +++ b/doc/special_remotes/bup/comment_7_cb1a0d3076e9d06e7a24204478f6fa98._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 7" + date="2013-04-02T21:24:06Z" + content=""" +`bup-split` uses a git branch to name the objects stored in the bup repository. So it will be limited by any scalability issues affecting large numbers of git branches. I don't know what those are. + +Yes, it would be possible to make git-annex store this in the git-annex branch instead. +"""]] diff --git a/doc/special_remotes/bup/comment_8_4cbc67e5911748d13cee3c483d7ece8a._comment b/doc/special_remotes/bup/comment_8_4cbc67e5911748d13cee3c483d7ece8a._comment new file mode 100644 index 000000000..62376a6e4 --- /dev/null +++ b/doc/special_remotes/bup/comment_8_4cbc67e5911748d13cee3c483d7ece8a._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlsXhOlsW6RaGR83VNSMxPh159l5dFau70" + nickname="Yung-Chin" + subject="re scaling issue" + date="2013-05-03T14:57:51Z" + content=""" +Tobias/joey, + +I think there are at least two scaling issues that may be causing you trouble. One is that bup writes pack+idx files rather than bare objects, and if you send 1 file per call to bup-split, you end up with a pair of pack and idx files for each such call. When you later try to retrieve a blob, bup currently just calls git, and git will have to traverse all these tiny idx files looking for the right hash (bare objects you could at least find by name). You can probably ameliorate the pain by calling git repack (look at the -a and --max-pack-size switches) on your bup repository. The other is the \"thousands of branches\" issue, and I think \"git pack-refs --all\" (that's again on your bup repository) might help a little bit. + +It would certainly help performance if you could store blob/tree ids in git-annex instead of branch names. For small files, all bup would need to store is a blob, but currently you end up storing a blob, a tree, and a commit (and looking-up all of those, plus the ref too, on calling bup-join). (you might want to patch bup-split, so it would allow you to ask it for \"--blob-or-tree\", because currently if you say you pass it -b for blob-ids, then for bigger files you get a series of IDs, whereas you'd be much better off with a tree-id there) +"""]] diff --git a/doc/special_remotes/bup/comment_9_ca7096a759961af375e6bd49663b45b3._comment b/doc/special_remotes/bup/comment_9_ca7096a759961af375e6bd49663b45b3._comment new file mode 100644 index 000000000..d6021f764 --- /dev/null +++ b/doc/special_remotes/bup/comment_9_ca7096a759961af375e6bd49663b45b3._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlsXhOlsW6RaGR83VNSMxPh159l5dFau70" + nickname="Yung-Chin" + subject="comment 9" + date="2013-05-03T16:34:05Z" + content=""" +Thinking about this some more, a very elegant way to make a bup remote could actually be to just pass the whole .git/annex tree into bup-index/save (you could avoid sending some files by only bup-indexing select subtrees, or by using --exclude-*'s, but you'd run bup-save over the whole .git/annex tree). You could then use bup-restore to retrieve files or whole subtrees, and you'd refer to the files you're retrieving by their actual pathname under which they live in .git/annex (if that doesn't make sense it's because I've misunderstood how git-annex is organised!), so something like \"bup restore branch_name/latest/.git/annex/aa/bb/sha-of-some-sort\" would work - that's cute, right? And you'd only have 1 branch. + +However... somebody who is good with lazy-evaluation would need to rework bup.vfs: currently, if you'd call bup-restore on a path like that, it would instantiate a _lot_ of vfs-nodes you don't need - to begin with, it would make a node for every commit you ever made (on any branch!) - on a big repository you'd wait ages for it to just find the commit objects... +"""]] diff --git a/doc/special_remotes/comment_10_e9881290486a1770bd260f8650ada9c6._comment b/doc/special_remotes/comment_10_e9881290486a1770bd260f8650ada9c6._comment new file mode 100644 index 000000000..79cbe3713 --- /dev/null +++ b/doc/special_remotes/comment_10_e9881290486a1770bd260f8650ada9c6._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkQqKSVY98PVGDIaYZdK9CodJdbh7cFfhY" + nickname="Ashwin" + subject="Rackspace Cloud Files support" + date="2013-03-22T08:20:40Z" + content=""" +It'd be really cool to have Rackspace cloud files support. Like the guy above me said, I would submit a patch but not if I have to learn Haskell first :) +"""]] diff --git a/doc/special_remotes/comment_11_e01b5cc5a0d81b071e93e27e7b91fe2a._comment b/doc/special_remotes/comment_11_e01b5cc5a0d81b071e93e27e7b91fe2a._comment new file mode 100644 index 000000000..f2f9c4a8f --- /dev/null +++ b/doc/special_remotes/comment_11_e01b5cc5a0d81b071e93e27e7b91fe2a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="andy" + ip="99.48.75.171" + subject="Re: Webhook special remote" + date="2013-04-12T08:54:47Z" + content=""" +@Alex: You might see if the newly-added [[todo/wishlist: allow configuration of downloader for addurl]] could be made to do what you need... I've not played around with it yet, but perhaps you could set the downloader to be something that can sort out the various URLs and send them to the correct downloading tool? +"""]] diff --git a/doc/special_remotes/comment_12_13237170ef5b6646e0e25d3421af3fe5._comment b/doc/special_remotes/comment_12_13237170ef5b6646e0e25d3421af3fe5._comment new file mode 100644 index 000000000..5146cca36 --- /dev/null +++ b/doc/special_remotes/comment_12_13237170ef5b6646e0e25d3421af3fe5._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://yarikoptic.myopenid.com/" + nickname="site-myopenid" + subject="How to establish local preference for (special) remotes" + date="2013-05-22T14:06:48Z" + content=""" +Sorry if it is RTFM... If I have multiple original (reachable) remotes, how could I establish my preference for which one to be used in any given location? + +usecase: if I clone a repository within amazon cloud instance -- I would now prefer if this (or all -- user-wide configuration somehow?) repository 'get's load from URLs originating in the cloud of this zone (e.g. having us-east-1.s3.amazonaws.com/ in their URLs). +"""]] diff --git a/doc/special_remotes/comment_13_1a36a0483a9db04d36e0234a192ebad8._comment b/doc/special_remotes/comment_13_1a36a0483a9db04d36e0234a192ebad8._comment new file mode 100644 index 000000000..8e5a8015d --- /dev/null +++ b/doc/special_remotes/comment_13_1a36a0483a9db04d36e0234a192ebad8._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8" + nickname="develop" + subject="Remote costs" + date="2013-05-22T14:15:03Z" + content=""" +This should be implemented with costs + +I refer you too: http://git-annex.branchable.com/design/assistant/blog/day_213__costs/ + +This has been implemented in the assistant, so if you use that, changing priority should be as simple as changing the order of the remotes on the web interface. Whichever remote is highest on the list, is the one your client will fetch from. +"""]] diff --git a/doc/special_remotes/comment_14_a8419963dc024b1d9eb73807596012dc._comment b/doc/special_remotes/comment_14_a8419963dc024b1d9eb73807596012dc._comment new file mode 100644 index 000000000..cd0a8eb58 --- /dev/null +++ b/doc/special_remotes/comment_14_a8419963dc024b1d9eb73807596012dc._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 14" + date="2013-05-22T14:30:00Z" + content=""" +You do not need to use the assistant to configure the costs of remotes. Just set `remote.<name>.annex-cost` to appropriate values. See also the documentation for the `remote.<name>.annex-cost-command` which allows your own code to calculate costs. +"""]] diff --git a/doc/special_remotes/comment_15_95ccfdd22a2391daa99e0beb04adedd6._comment b/doc/special_remotes/comment_15_95ccfdd22a2391daa99e0beb04adedd6._comment new file mode 100644 index 000000000..3e2ea9948 --- /dev/null +++ b/doc/special_remotes/comment_15_95ccfdd22a2391daa99e0beb04adedd6._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="http://yarikoptic.myopenid.com/" + nickname="site-myopenid" + subject="remotes costs" + date="2013-05-22T18:33:11Z" + content=""" +Thank you -- that is nice! + +Could costs be presented in 'whereis' and 'status' commands? e.g. like we know APT repositories priorities from apt-cache policy -- now I do not see them (at least in 4.20130501... updating to sid's 0521 now) + +"""]] diff --git a/doc/special_remotes/comment_16_b9d238fb15ad7628e33c90b071e07bb0._comment b/doc/special_remotes/comment_16_b9d238fb15ad7628e33c90b071e07bb0._comment new file mode 100644 index 000000000..8b1fcd831 --- /dev/null +++ b/doc/special_remotes/comment_16_b9d238fb15ad7628e33c90b071e07bb0._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://yarikoptic.myopenid.com/" + nickname="site-myopenid" + subject="compression -- storage and transfer" + date="2013-05-22T18:48:59Z" + content=""" +Is there any remote which would not only compress during transfer (I believe rsync does that, right?) but also store objects compressed? + +I thought bup would do both -- but it seems that git annex receives data uncompressed from a bup remote, and bup remote requires ssh access. + +In my case I want to make publicly available files which are binary blobs which could be compressed very well. It would be a pity if I waste storage on my end and also incur significant traffic, which could be avoided if data load was transferred compressed. May be HTTP compression (http://en.wikipedia.org/wiki/HTTP_compression) could somehow be used efficiently for this purpose (not sure if load then originally could already reside in a compressed form to avoid server time to re-compress it)? +"""]] diff --git a/doc/special_remotes/comment_17_cc21b81a8f809f6efa5f5b6332513fc3._comment b/doc/special_remotes/comment_17_cc21b81a8f809f6efa5f5b6332513fc3._comment new file mode 100644 index 000000000..f576e2723 --- /dev/null +++ b/doc/special_remotes/comment_17_cc21b81a8f809f6efa5f5b6332513fc3._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://yarikoptic.myopenid.com/" + nickname="site-myopenid" + subject="Re: compression -- storage and transfer" + date="2013-05-22T19:17:33Z" + content=""" +ha -- apparently it is trivial to configure apache to serve pre-compressed files (e.g. see http://stackoverflow.com/questions/75482/how-can-i-pre-compress-files-with-mod-deflate-in-apache-2-x) and they arrive compressed to client with + +Content-Encoding: gzip + +but unfortunately git-annex doesn't like those (fails to \"verify\") -- do you think it could be implemented for web \"special remotes\"? that would be really nice -- then I could store such load on another website, and addurl links to the compressed content +"""]] diff --git a/doc/special_remotes/comment_18_3fe750118ff1edbe91a110b86fb5b662._comment b/doc/special_remotes/comment_18_3fe750118ff1edbe91a110b86fb5b662._comment new file mode 100644 index 000000000..2345bba21 --- /dev/null +++ b/doc/special_remotes/comment_18_3fe750118ff1edbe91a110b86fb5b662._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 18" + date="2013-05-23T23:25:02Z" + content=""" +All special remotes store files compressed when you enable encryption. Not otherwise, though. + +As far as the web special remote and pre-compressed files, files are downloaded from the web using `wget` or (of wget is not available) `curl`. So if you can make it work with those commands, it should work. +"""]] diff --git a/doc/special_remotes/comment_19_6794eb52bd87c28ef1df3172aa7d5780._comment b/doc/special_remotes/comment_19_6794eb52bd87c28ef1df3172aa7d5780._comment new file mode 100644 index 000000000..b8c701ade --- /dev/null +++ b/doc/special_remotes/comment_19_6794eb52bd87c28ef1df3172aa7d5780._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://yarikoptic.myopenid.com/" + nickname="site-myopenid" + subject="compressed storage/transfer -- gzip Content-Type" + date="2013-05-25T06:41:37Z" + content=""" +FWIW -- eh -- unfortunately it seems not that transparent. wget seems to not support decompression at all, curl can do with explicit --compressed, BUT it doesn't distinguish url to a \"natively\" .gz file and pre-compressed content. And I am not sure if it is possible to anyhow reliably distinguish the two urls. In the case of obtaining pre-compressed file from my sample apache server the only difference in the http response header is that it gets \"compound\" ETag: +compare ETag: \"3acb0e-17b38-4dd5343744660\" (for directly asking zeros100.gz) vs \"3acb0e-17b38-4dd5343744660;4dd5344e1537e\" (requesting zeros100) where portion past \";\" I guess signals the caching tag for gzipping, but not exactly sure on that since it seems to be not a part of standard. Also for zeros100 I am getting \"TCN: choice\"... once again not sure if that is any how reliably indicative for my purpose. So I guess there is no good way ATM via Content-Type request. +"""]] diff --git a/doc/special_remotes/comment_1_961276c18e9353ca8e25cad53e7ec51f._comment b/doc/special_remotes/comment_1_961276c18e9353ca8e25cad53e7ec51f._comment new file mode 100644 index 000000000..ec1bea419 --- /dev/null +++ b/doc/special_remotes/comment_1_961276c18e9353ca8e25cad53e7ec51f._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8" + nickname="Jon Ander" + subject="MediaFire" + date="2013-01-17T12:17:54Z" + content=""" +MediaFire offers 50GB of free storage (max size 200MB). It would be great to support it as a new special remote. +"""]] diff --git a/doc/special_remotes/comment_20_6b7242721f2f2c77b634568cb737e3e3._comment b/doc/special_remotes/comment_20_6b7242721f2f2c77b634568cb737e3e3._comment new file mode 100644 index 000000000..b17896a68 --- /dev/null +++ b/doc/special_remotes/comment_20_6b7242721f2f2c77b634568cb737e3e3._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg" + nickname="Bence" + subject="Testing a special remote" + date="2013-11-24T08:24:36Z" + content=""" +Is there a unit test or integration test to check for the behavior of a special remote implementation and/or validity? + +I don't speak Haskell, so maybe there are some in the source but maybe I wouldn't recognize, so I haven't checked. If there are any tests how should I use it? + +Thank you, +Bence +"""]] diff --git a/doc/special_remotes/comment_21_5c11e69c28b9ed4cbe238a36c0839a47._comment b/doc/special_remotes/comment_21_5c11e69c28b9ed4cbe238a36c0839a47._comment new file mode 100644 index 000000000..1645e03e6 --- /dev/null +++ b/doc/special_remotes/comment_21_5c11e69c28b9ed4cbe238a36c0839a47._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.64" + subject="comment 21" + date="2013-11-24T15:58:30Z" + content=""" +@Bence the closest I have is some tests of particular special remotes inside Test.hs. The shell equivilant of that code is: + +[[!format sh \"\"\" +set -e +git annex copy file --to remote # tests store +git annex drop file # tests checkpresent when remote has file +git annex move file --from remote # tests retrieve and remove +\"\"\"]] +"""]] diff --git a/doc/special_remotes/comment_2_97543acfa7434e332ebea5672e446317._comment b/doc/special_remotes/comment_2_97543acfa7434e332ebea5672e446317._comment new file mode 100644 index 000000000..840a9d626 --- /dev/null +++ b/doc/special_remotes/comment_2_97543acfa7434e332ebea5672e446317._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.7.238" + subject="comment 2" + date="2013-01-17T16:44:25Z" + content=""" +Mediafire does not appear to offer any kind of API for its storage. +"""]] diff --git a/doc/special_remotes/comment_3_9229776623c234204c8b164edff95da0._comment b/doc/special_remotes/comment_3_9229776623c234204c8b164edff95da0._comment new file mode 100644 index 000000000..84fe385e0 --- /dev/null +++ b/doc/special_remotes/comment_3_9229776623c234204c8b164edff95da0._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8" + nickname="Jon Ander" + subject="MediaFire REST API" + date="2013-01-17T16:53:41Z" + content=""" +Wouldn't this be enough? http://developers.mediafire.com/index.php/REST_API +"""]] diff --git a/doc/special_remotes/comment_4_3bbda479d13f6bf393dcd59ed94ddeaa._comment b/doc/special_remotes/comment_4_3bbda479d13f6bf393dcd59ed94ddeaa._comment new file mode 100644 index 000000000..c5979f819 --- /dev/null +++ b/doc/special_remotes/comment_4_3bbda479d13f6bf393dcd59ed94ddeaa._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q" + nickname="Andrew" + subject="JABOF special remote" + date="2013-01-19T08:34:32Z" + content=""" +Similar to a JABOD, this would be Just A Bunch Of Files. I already have a NAS with a file structure conducive to serving media to my TV. However, it's not capable (currently) of running git-annex locally. It would be great to be able to tell annex the path to a file there as a remote much like a web remote from \"git annex addurl\". That way I can safely drop all the files I took with me on my trip, while annex still verifies and counts the file on the NAS as a location. + +There are some interesting things to figure out for this to be efficient. For example, SHAs of the files. Maybe store that in a metadata file in the directory of the files? Or perhaps use the WORM backend by default? +"""]] diff --git a/doc/special_remotes/comment_5_f7000975d38077828ab11a99095b39eb._comment b/doc/special_remotes/comment_5_f7000975d38077828ab11a99095b39eb._comment new file mode 100644 index 000000000..1a9eb390a --- /dev/null +++ b/doc/special_remotes/comment_5_f7000975d38077828ab11a99095b39eb._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.3.194" + subject="comment 5" + date="2013-01-19T16:05:13Z" + content=""" +The web special remote is recently able to use file:// URL's, so you can just point to files on some arbitrary storage if you want to. +"""]] diff --git a/doc/special_remotes/comment_6_5d2bd7c1e1493d3c3784708a9b0bc001._comment b/doc/special_remotes/comment_6_5d2bd7c1e1493d3c3784708a9b0bc001._comment new file mode 100644 index 000000000..0f11973f7 --- /dev/null +++ b/doc/special_remotes/comment_6_5d2bd7c1e1493d3c3784708a9b0bc001._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlBia1J9-PoXgZYj2LASf7Bs__IqK3T8qQ" + nickname="Greg" + subject="Rackspace US/UK" + date="2013-01-30T11:33:12Z" + content=""" +It'd be awesome to be able to use Rackspace as remote storage as an alternative to S3, I would submit a patch, but know 0 Haskell :D +"""]] diff --git a/doc/special_remotes/comment_7_af01ee5ce31b1490af565cb087d65277._comment b/doc/special_remotes/comment_7_af01ee5ce31b1490af565cb087d65277._comment new file mode 100644 index 000000000..043ad8eb5 --- /dev/null +++ b/doc/special_remotes/comment_7_af01ee5ce31b1490af565cb087d65277._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawn-UoTjMBsVh6q4HNViGwJi-5FNaCVQB7E" + nickname="Nico" + subject="Rapidshare" + date="2013-02-02T16:49:58Z" + content=""" +Would it be possible to support Rapidshare as a new special remote? +They offer unlimited storage for 6-10€ per month. It would be great for larger backups. +Their API can be found here: http://images.rapidshare.com/apidoc.txt +"""]] diff --git a/doc/special_remotes/comment_8_3d4ffec566d68d601eafe8758a616756._comment b/doc/special_remotes/comment_8_3d4ffec566d68d601eafe8758a616756._comment new file mode 100644 index 000000000..85e663296 --- /dev/null +++ b/doc/special_remotes/comment_8_3d4ffec566d68d601eafe8758a616756._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY" + nickname="Alex" + subject="'webhook' special remote?" + date="2013-02-24T15:05:27Z" + content=""" +Is there any chance a special remote that functions like a hybrid of 'web' and 'hook'? At least in theory, it should be relatively simple, since it would only support 'get' and the only meaningful parameters to pass would be the URL and the output file name. + +Maybe make it something like git config annex.myprogram-webhook 'myprogram $ANNEX_URL $ANNEX_FILE', and fetching could work by adding a --handler or --type parameter to addurl. + +The use case here is anywhere that simple 'fetch the file over HTTP/FTP/etc' isn't workable - maybe it's on rapidshare and you need to use plowshare to download it; maybe it's a youtube video and you want to use youtube-dl, maybe it's a chapter of a manga and you want to turn it into a CBZ file when you fetch it. + +"""]] diff --git a/doc/special_remotes/comment_9_26af468952f0403171370b56e127830a._comment b/doc/special_remotes/comment_9_26af468952f0403171370b56e127830a._comment new file mode 100644 index 000000000..88f021c6a --- /dev/null +++ b/doc/special_remotes/comment_9_26af468952f0403171370b56e127830a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY" + nickname="Alex" + subject="comment 9" + date="2013-02-24T15:13:16Z" + content=""" +A *ridiculously* cool possibility would be to allow them to match against URLs and then handle those (youtube-dl for youtube video URLs, for instance), but that would be additional work on your end and isn't really necessary. +"""]] diff --git a/doc/special_remotes/directory.mdwn b/doc/special_remotes/directory.mdwn new file mode 100644 index 000000000..4d72e8bee --- /dev/null +++ b/doc/special_remotes/directory.mdwn @@ -0,0 +1,33 @@ +This special remote type stores file contents in directory. + +One use case for this would be if you have a removable drive that +you want to use it to sneakernet files between systems (possibly with +[[encrypted|encryption]] contents). Just set up both systems to use +the drive's mountpoint as a directory remote. + +If you just want two copies of your repository with the files "visible" +in the tree in both, the directory special remote is not what you want. +Instead, you should use a regular `git clone` of your git-annex repository. + +## configuration + +These parameters can be passed to `git annex initremote` to configure the +remote: + +* `encryption` - One of "none", "hybrid", "shared", or "pubkey". + See [[encryption]]. + +* `keyid` - Specifies the gpg key to use for [[encryption]]. + +* `chunksize` - Avoid storing files larger than the specified size in the + directory. For use on directories on mount points that have file size + limitations. The default is to never chunk files. + The value can use specified using any commonly used units. + Example: `chunksize=100 megabytes` + Note that enabling chunking on an existing remote with non-chunked + files is not recommended. + +Setup example: + + # git annex initremote usbdrive type=directory directory=/media/usbdrive/ encryption=none + # git annex describe usbdrive "usb drive on /media/usbdrive/" diff --git a/doc/special_remotes/directory/comment_11_86f8c1b09cbd82bcd76378dfa1b3ca07._comment b/doc/special_remotes/directory/comment_11_86f8c1b09cbd82bcd76378dfa1b3ca07._comment new file mode 100644 index 000000000..f29d54b59 --- /dev/null +++ b/doc/special_remotes/directory/comment_11_86f8c1b09cbd82bcd76378dfa1b3ca07._comment @@ -0,0 +1,49 @@ +[[!comment format=mdwn + username="dietz" + ip="128.119.40.196" + subject="annexing external files" + date="2013-07-18T20:57:53Z" + content=""" +This is great work. I've developed a serious annex-addiction and now I want to use it everywhere! In particular I was hoping to apply it to this use case: + +I have large files/directories (approx 5 TB) on an nfs mount to which is a) write-protected (think \"read-only medium\") and b) used by non-git users. Both reasons prevent me from setting up a git-annex repos there. However, I would like to use git-annex to keep track of the paths and get/drop files from my different computers. + +On one of my servers, I set up a git annex repos, hoping to only manage the structure, the locations, and the number of copies. I don't want to have copies of the 5TB files in that repository, as disk space is not unlimited (just for the sake of making them available to my laptop). + +I as banking on using a special remote (either directory or rsync) to tell the git-annex repos where the actual data is. + +I am not concerned with data loss, as it is backed up in regular time intervals by our sysadmin. + + +I tried both directory remote and rsync remote but there seems to be a missing piece (I suppose its add). Any ideas? + +This is what I did: + + +I added the directory remote and an rsync remote + +- $ git annex initremote collections type=directory directory=/my/nfs/dir encryption=none +- $ git annex initremote rsync type=rsync rsyncurl=ssh://myserver/mnt/nfs/dir encryption=none + +the copy command fails without complaints + $ git annex copy --from collections + +I tried adding virtual files to git annex + +- $ git annex add file/inside/dir + +but still any kind of get/copy command does not get any new files + + +It would be awesome if I could use git-annex for this, to keep track of my copies and copies of copies. And then I could also keep track of data on my write-protected DVDs. + +Is there any chance? + +Thanks a lot! + +-- Laura + + + + +"""]] diff --git a/doc/special_remotes/directory/comment_12._comment b/doc/special_remotes/directory/comment_12._comment new file mode 100644 index 000000000..608fb62cd --- /dev/null +++ b/doc/special_remotes/directory/comment_12._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.2.25" + subject="comment 2" + date="2013-07-19T13:54:10Z" + content=""" +@Laura the directory special remote requires files to +be in a particular directory structure with special names +git-annex comes up with. So you can't use it on an existing +tree of files like that. + +What you can do is use the [[web_special_remote|web]], +with a `file://` url to point to the files wherever +they are stored. So for example, +`git annex addurl file:///media/dvd/file` +"""]] diff --git a/doc/special_remotes/directory/comment_12_311cd013fd8db47856d84161119e059d._comment b/doc/special_remotes/directory/comment_12_311cd013fd8db47856d84161119e059d._comment new file mode 100644 index 000000000..21ab5aaa9 --- /dev/null +++ b/doc/special_remotes/directory/comment_12_311cd013fd8db47856d84161119e059d._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="dietz" + ip="128.119.40.196" + subject="comment 12" + date="2013-07-20T06:06:31Z" + content=""" +Using the web remote is a pretty nice trick! + +Thanks, Joey - I would not have guessed that. + +-- Laura +"""]] diff --git a/doc/special_remotes/directory/comment_1_e8a53592adb13f7d7f212a2eb5a18a31._comment b/doc/special_remotes/directory/comment_1_e8a53592adb13f7d7f212a2eb5a18a31._comment new file mode 100644 index 000000000..b2a041c53 --- /dev/null +++ b/doc/special_remotes/directory/comment_1_e8a53592adb13f7d7f212a2eb5a18a31._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlc1og3PqIGudOMkFNrCCNg66vB7s-jLpc" + nickname="Paul" + subject="how is this different than rsync?" + date="2012-06-22T22:10:19Z" + content=""" +Thanks for this great tool! I was wondering what the differences are between using `type=directory`, `type=rsync`, or a bare git repo for directories? + +I guess I can't use just a regular repo because my USB drive is formatted as `vfat` -- which threw me for a loop the first time I heard about `git-annex` about a year ago, because I followed the walkthrough, and it didn't work as expected and gave up (now I know it was just a case of PEBKAC). It might be worth adding a note about [vfat](http://git-annex.branchable.com/bugs/fat_support/) to the \"Adding a remote\" section of the [walkthrough](http://git-annex.branchable.com/walkthrough/), since the unstated assumption there is that the USB drive is formatted as a filesystem that supports symlinks. + +Thanks again, my scientific data management just got a lot more sane! +"""]] diff --git a/doc/special_remotes/directory/comment_2_d949edad6a330079f9e15f703f9091e3._comment b/doc/special_remotes/directory/comment_2_d949edad6a330079f9e15f703f9091e3._comment new file mode 100644 index 000000000..77b4c4d22 --- /dev/null +++ b/doc/special_remotes/directory/comment_2_d949edad6a330079f9e15f703f9091e3._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.2.25" + subject="comment 2" + date="2012-06-25T15:29:29Z" + content=""" +The directory and rsync special remotes intentionally use the same layout. So the same directory could be set up as both types of special remotes. + +The main reason to use this rather than a bare git repo is that it supports encryption. +"""]] diff --git a/doc/special_remotes/directory/comment_3_49009f4e9e335c9a9d0422aa59c9a432._comment b/doc/special_remotes/directory/comment_3_49009f4e9e335c9a9d0422aa59c9a432._comment new file mode 100644 index 000000000..b8ec51049 --- /dev/null +++ b/doc/special_remotes/directory/comment_3_49009f4e9e335c9a9d0422aa59c9a432._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://dzsino.myopenid.com/" + nickname="dzsino" + subject="dropping a directory remote?" + date="2013-01-15T22:29:15Z" + content=""" +How do I drop a directory remote after initremote? Say I want to start over and enable chunking (not supposed to be enabled on an existing directory), can I simply git remote rm it? +"""]] diff --git a/doc/special_remotes/directory/comment_4_f5e9b0b477c4e521f8633fd274757fa3._comment b/doc/special_remotes/directory/comment_4_f5e9b0b477c4e521f8633fd274757fa3._comment new file mode 100644 index 000000000..53fc857a3 --- /dev/null +++ b/doc/special_remotes/directory/comment_4_f5e9b0b477c4e521f8633fd274757fa3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.7.238" + subject="comment 4" + date="2013-01-17T18:22:26Z" + content=""" +The best way to remove a special remote is to first `git annex move --from $remote` to get all the content out of it, then `git annex dead $remote` and finally you can `git remote rm $remote` +"""]] diff --git a/doc/special_remotes/directory/comment_5_e790718423c41f5ea8047ea5225bfacd._comment b/doc/special_remotes/directory/comment_5_e790718423c41f5ea8047ea5225bfacd._comment new file mode 100644 index 000000000..dae22fc8f --- /dev/null +++ b/doc/special_remotes/directory/comment_5_e790718423c41f5ea8047ea5225bfacd._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawk3HGoDpnOPob5jOjvIootmkve1-nCpRiI" + nickname="Kalle" + subject="Directory remote with files accessible from non git-annex system?" + date="2013-01-20T15:44:16Z" + content=""" + + + + +"""]] diff --git a/doc/special_remotes/directory/comment_6_325aac80b86588912c4fd61339ccbd0b._comment b/doc/special_remotes/directory/comment_6_325aac80b86588912c4fd61339ccbd0b._comment new file mode 100644 index 000000000..7b9d21605 --- /dev/null +++ b/doc/special_remotes/directory/comment_6_325aac80b86588912c4fd61339ccbd0b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://nicolas-schodet.myopenid.com/" + ip="2a01:e35:8ae6:f130:1e4b:d6ff:fe78:1ddb" + subject="comment 6" + date="2013-06-26T17:52:12Z" + content=""" +I tried the suggestion on comment 4, but when I add again a remote with the same path, it gets the same repository identifier and is considered dead. Is that expected? + +My use case: I use a usb drive to transfer some large files from one git annex to another, then I use the usb drive for something else and the special remote is removed. Later I want to use the same usb drive again, but when I create the repository, it starts in the dead state. +"""]] diff --git a/doc/special_remotes/directory/comment_7_4206db69d68d9917623ce02500387021._comment b/doc/special_remotes/directory/comment_7_4206db69d68d9917623ce02500387021._comment new file mode 100644 index 000000000..b1a3b953e --- /dev/null +++ b/doc/special_remotes/directory/comment_7_4206db69d68d9917623ce02500387021._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.193" + subject="comment 7" + date="2013-06-26T18:30:39Z" + content=""" +@nicolas, I suspect you are using `git annex initremote` with the same name that you used for the now dead-and-buried remote. That causes it to be reanimated, which is not what you want. + +Since git-annex version 4.20130501, `git annex initremote` is reserved for creating new remotes, not enabling old ones, so it will refuse to do this. That's to avoid exactly this confusion. + +Using `git annex initremote` with a different remote name, and the same directory should work just fine. +"""]] diff --git a/doc/special_remotes/directory/comment_8_acd9023511fe43817718bc89430f96c3._comment b/doc/special_remotes/directory/comment_8_acd9023511fe43817718bc89430f96c3._comment new file mode 100644 index 000000000..29801fa36 --- /dev/null +++ b/doc/special_remotes/directory/comment_8_acd9023511fe43817718bc89430f96c3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q" + nickname="Andrew" + subject="comment 8" + date="2013-06-26T18:46:40Z" + content=""" +For the use case you're describing, it might be better to define the usb key as a remote set to \"manual.\" Then, you can copy over the things you want with git annex copy --to=usbkey and when you're done drop everything with git annex drop --from=usbkey and never destroy the remote. +"""]] diff --git a/doc/special_remotes/directory/comment_9_d330eb808a990bb71034613c297a265e._comment b/doc/special_remotes/directory/comment_9_d330eb808a990bb71034613c297a265e._comment new file mode 100644 index 000000000..47ac7541d --- /dev/null +++ b/doc/special_remotes/directory/comment_9_d330eb808a990bb71034613c297a265e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://nicolas-schodet.myopenid.com/" + ip="2a01:e35:8ae6:f130:1e4b:d6ff:fe78:1ddb" + subject="comment 9" + date="2013-06-26T20:05:20Z" + content=""" +Thanks for your answers. You're right, the simplest solution for me is to never remove the remote. If my directory is lost, I realized that I can simply create an empty directory. +"""]] diff --git a/doc/special_remotes/gcrypt.mdwn b/doc/special_remotes/gcrypt.mdwn new file mode 100644 index 000000000..ac98c43bb --- /dev/null +++ b/doc/special_remotes/gcrypt.mdwn @@ -0,0 +1,45 @@ +[git-remote-gcrypt](https://github.com/joeyh/git-remote-gcrypt/) +adds support for encrypted remotes to git. The git-annex gcrypt special +remote allows git-annex to also store its files in such repositories. +Naturally, git-annex encrypts the files it stores too, so everything +stored on the remote is encrypted. + +See [[tips/fully_encrypted_git_repositories_with_gcrypt]] for some examples +of using gcrypt. + +## configuration + +These parameters can be passed to `git annex initremote` to configure +gcrypt: + +* `encryption` - One of "none", "hybrid", "shared", or "pubkey". + See [[encryption]]. + +* `keyid` - Specifies the gpg key to use for encryption of both the files + git-annex stores in the repository, as well as to encrypt the git + repository itself. May be repeated when multiple participants + should have access to the repository. + +* `gitrepo` - Required. The path or url to the git repository + for gcrypt to use. This repository should be either empty, or an existing + gcrypt repositry. + +* `shellescape` - See [[rsync]] for the details of this option. + +## notes + +For git-annex to store files in a repository on a remote server, you need +shell access, and `rsync` must be installed. Those are the minimum +requirements, but it's also recommended to install git-annex on the remote +server, so that [[git-annex-shell]] can be used. + +While you can use git-remote-gcrypt with servers like github, git-annex +can't store files on them. In such a case, you can just use +git-remote-gcrypt directly. + +If you use encryption=hybrid, you can add more gpg keys that can access +the files git-annex stored in the gcrypt repository. However, due to the +way git-remote-gcrypt encrypts the git repository, you will need to somehow +force it to re-push everything again, so that the encrypted repository can +be decrypted by the added keys. Probably this can be done by setting +`GCRYPT_FULL_REPACK` and doing a forced push of branches. diff --git a/doc/special_remotes/glacier.mdwn b/doc/special_remotes/glacier.mdwn new file mode 100644 index 000000000..b55b9adbb --- /dev/null +++ b/doc/special_remotes/glacier.mdwn @@ -0,0 +1,47 @@ +This special remote type stores file contents in Amazon Glacier. + +To use it, you need to have [glacier-cli](http://github.com/basak/glacier-cli) +installed. + +The unusual thing about Amazon Glacier is the multiple-hour delay it takes +to retrieve information out of Glacier. To deal with this, commands like +"git-annex get" request Glacier start the retrieval process, and will fail +due to the data not yet being available. You can then wait appriximately +four hours, re-run the same command, and this time, it will actually +download the data. + +## configuration + +The standard environment variables `AWS_ACCESS_KEY_ID` and +`AWS_SECRET_ACCESS_KEY` are used to supply login credentials +for Amazon. You need to set these only when running +`git annex initremote`, as they will be cached in a file only you +can read inside the local git repository. + +A number of parameters can be passed to `git annex initremote` to configure +the Glacier remote. + +* `encryption` - One of "none", "hybrid", "shared", or "pubkey". + See [[encryption]]. + +* `keyid` - Specifies the gpg key to use for [[encryption]]. + +* `embedcreds` - Optional. Set to "yes" embed the login credentials inside + the git repository, which allows other clones to also access them. This is + the default when gpg encryption is enabled; the credentials are stored + encrypted and only those with the repository's keys can access them. + + It is not the default when using shared encryption, or no encryption. + Think carefully about who can access your repository before using + embedcreds without gpg encryption. + +* `datacenter` - Defaults to "us-east-1". + +* `vault` - By default, a vault name is chosen based on the remote name + and UUID. This can be specified to pick a vault name. + +* `fileprefix` - By default, git-annex places files in a tree rooted at the + top of the Glacier vault. When this is set, it's prefixed to the filenames + used. For example, you could set it to "foo/" in one special remote, + and to "bar/" in another special remote, and both special remotes could + then use the same vault. diff --git a/doc/special_remotes/glacier/comment_1_fcd856b99dc6b3f9141b65fe639ef76b._comment b/doc/special_remotes/glacier/comment_1_fcd856b99dc6b3f9141b65fe639ef76b._comment new file mode 100644 index 000000000..1d6d89433 --- /dev/null +++ b/doc/special_remotes/glacier/comment_1_fcd856b99dc6b3f9141b65fe639ef76b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c" + nickname="Justin" + subject="comment 1" + date="2013-05-16T00:54:21Z" + content=""" +The glacier-cli tool seems to have been abandoned, and there are a number of outstanding issues with it. boto has a `glacier` tool, but it doesn't seem to include caching, which seems to be something git annex needs. + +Looking through the PRs, it seems like we should build a tool specifically tailored to git annex's needs. It seems that there are at least three of us willing to hack on this if it's in Python. I'm not sure any of us knows haskell, though... +"""]] diff --git a/doc/special_remotes/glacier/comment_2_38fcca87074f6ea31a12569a822aa8c9._comment b/doc/special_remotes/glacier/comment_2_38fcca87074f6ea31a12569a822aa8c9._comment new file mode 100644 index 000000000..939832b32 --- /dev/null +++ b/doc/special_remotes/glacier/comment_2_38fcca87074f6ea31a12569a822aa8c9._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="basak" + ip="2001:8b0:1c8::2" + subject="comment 2" + date="2013-05-17T08:35:10Z" + content=""" +I'm the glacier-cli author. It is not abandoned! + +glacier-cli is supposed to map to Glacier exactly, so that it is compatible with all other tools. Most of the outstanding PRs break this essential behaviour, so I have not merged them. Many of the feature requests and bugs related to the upstream boto library, which is just about the best maintained client library that exists for AWS on any platform (and Amazon have adopted it now, IIRC). I have written appropriate reviews on all the PRs. + +If there is specific behaviour that git-annex needs, them I am happy to accept PRs for this, provided that they do not break the ability (and default) for glacier-cli to talk to Glacier natively without an extra layer of interpretation. If an extra layer of interpretation is needed (eg. forbidding duplicate \"keys\"), then this needs to be an option, or wrapped in a separate tool, or written into git-annex's Glacier special remote. +"""]] diff --git a/doc/special_remotes/glacier/comment_3_cea5bcb162e4288847ba5f25464a0406._comment b/doc/special_remotes/glacier/comment_3_cea5bcb162e4288847ba5f25464a0406._comment new file mode 100644 index 000000000..8a8b914ad --- /dev/null +++ b/doc/special_remotes/glacier/comment_3_cea5bcb162e4288847ba5f25464a0406._comment @@ -0,0 +1,28 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c" + nickname="Justin" + subject="comment 3" + date="2013-05-19T05:56:45Z" + content=""" +Hi! :) + +The main issue I'm hitting is the \"Multiple rows were found for one()\" error. I think I get this when git-annex tries to upload the same file twice (which may be a bug in git-annex, which could apply de-duplication earlier), but I think I also get it when trying to upload a file whose upload I've canceled in the past. + +I don't quite understand what git-annex needs here, and I totally understand that you're writing a general-purpose tool. But there does seem to be an issue that git-annex needs fixed one way or another. + +I'm happy to try fixing it myself if you can help me understand what's going on (I didn't quite understand your review in the PR), but if I'm the only person in the world using git-annex to back up to glacier, that scares me a little! + + copy foo/bar/baz (checking glacier...) Traceback (most recent call last): + File \"/home/jlebar/code/glacier-cli/glacier\", line 694, in <module> + App().main() + File \"/home/jlebar/code/glacier-cli/glacier\", line 680, in main + args.func(args) + File \"/home/jlebar/code/glacier-cli/glacier\", line 579, in archive_checkpresent + last_seen = self.cache.get_archive_last_seen(args.vault, args.name) + File \"/home/jlebar/code/glacier-cli/glacier\", line 157, in get_archive_last_seen + result = self._get_archive_query_by_ref(vault, ref).one() + File \"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py\", line 2182, in one + \"Multiple rows were found for one()\") + sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one() + +"""]] diff --git a/doc/special_remotes/glacier/comment_4_0c92cc82c7ac513130f862391a02d329._comment b/doc/special_remotes/glacier/comment_4_0c92cc82c7ac513130f862391a02d329._comment new file mode 100644 index 000000000..2de6632eb --- /dev/null +++ b/doc/special_remotes/glacier/comment_4_0c92cc82c7ac513130f862391a02d329._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="basak" + ip="2001:8b0:1c8::2" + subject="comment 4" + date="2013-05-22T18:10:32Z" + content=""" +Let's discuss this in a bug. I've created http://git-annex.branchable.com/bugs/Glacier_remote_uploads_duplicates/ +"""]] diff --git a/doc/special_remotes/glacier/comment_5_8d1dcb4bf48386314bfb248ea6eeeb68._comment b/doc/special_remotes/glacier/comment_5_8d1dcb4bf48386314bfb248ea6eeeb68._comment new file mode 100644 index 000000000..ee535e1f1 --- /dev/null +++ b/doc/special_remotes/glacier/comment_5_8d1dcb4bf48386314bfb248ea6eeeb68._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://id.clacke.se/" + nickname="clacke" + subject="comment 5" + date="2013-06-03T09:03:57Z" + content=""" +You are not the only one, Justin. I am just getting into git-annex and I am setting up a glacier remote as I write this. +"""]] diff --git a/doc/special_remotes/glacier/comment_6_adb1db354dc4941e4b004e4ba34660d7._comment b/doc/special_remotes/glacier/comment_6_adb1db354dc4941e4b004e4ba34660d7._comment new file mode 100644 index 000000000..717060fcb --- /dev/null +++ b/doc/special_remotes/glacier/comment_6_adb1db354dc4941e4b004e4ba34660d7._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY" + nickname="Jimmy" + subject="I know this thread is old but..." + date="2013-11-18T00:27:50Z" + content=""" +It's working well for me, though I seem to sometimes still hit the duplicate bug listed above. +"""]] diff --git a/doc/special_remotes/glacier/comment_7_747e403aac5acaba00e220931e926951._comment b/doc/special_remotes/glacier/comment_7_747e403aac5acaba00e220931e926951._comment new file mode 100644 index 000000000..045fb6041 --- /dev/null +++ b/doc/special_remotes/glacier/comment_7_747e403aac5acaba00e220931e926951._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlq4ClC5EMN1Vq1DpWXAqP5TiDnCK1mSfk" + nickname="Jonas" + subject="Expected cost for a repository" + date="2013-11-24T22:32:16Z" + content=""" +Is there a way to estimate the cost of storing a repo on glacier? + +I'm especially worried because of the cost of STORE and RETRIEVE requests; there are hundreds of thousands of small files in my annex repo, so that request cost could easily dominate storage cost. Does the glacier remote do anything to minimize the number of objects stored in glacier? +"""]] diff --git a/doc/special_remotes/hook.mdwn b/doc/special_remotes/hook.mdwn new file mode 100644 index 000000000..eaea940a7 --- /dev/null +++ b/doc/special_remotes/hook.mdwn @@ -0,0 +1,103 @@ +This special remote type lets you store content in a remote of your own +devising. + +It's not recommended to use this remote type when another like [[rsync]] +or [[directory]] will do. If your hooks are not carefully written, data +could be lost. + +## example + +Here's a simple example that stores content on clay tablets. If you +implement this example in the real world, I'd appreciate a tour +next Apert! :) --[[Joey]] + + # git config annex.cuneiform-store-hook 'tocuneiform < "$ANNEX_FILE" | tablet-writer --implement=stylus --title="$ANNEX_KEY" | tablet-proofreader | librarian --shelve --floor=$ANNEX_HASH_1 --shelf=$ANNEX_HASH_2' + # git config annex.cuneiform-retrieve-hook 'librarian --get --floor=$ANNEX_HASH_1 --shelf=$ANNEX_HASH_2 --title="$ANNEX_KEY" | tablet-reader --implement=coffee --implement=glasses --force-monastic-dedication | fromcuneiform > "$ANNEX_FILE"' + # git config annex.cuneiform-remove-hook 'librarian --get --floor=$ANNEX_HASH_1 --shelf=$ANNEX_HASH_2 --title="$ANNEX_KEY" | goon --hit-with-hammer' + # git config annex.cuneiform-checkpresent-hook 'librarian --find --force-distrust-catalog --floor=$ANNEX_HASH_1 --shelf=$ANNEX_HASH_2 --title="$ANNEX_KEY" --shout-title' + # git annex initremote library type=hook hooktype=cuneiform encryption=none + # git annex describe library "the reborn Library of Alexandria (upgrade to bronze plates pending)" + +Can you spot the potential data loss bugs in the above simple example? +(Hint: What happens when the `tablet-proofreader` exits nonzero?) + +## configuration + +These parameters can be passed to `git annex initremote`: + +* `hooktype` - Required. This specifies a collection of hooks to use for + this remote. + +* `encryption` - One of "none", "hybrid", "shared", or "pubkey". + See [[encryption]]. + +* `keyid` - Specifies the gpg key to use for [[encryption]]. + +## hooks + +Each type of hook remote is specified by a collection of hook commands. +Each hook command is run as a shell command line, and should return nonzero +on failure, and zero on success. + +These environment variables are used to communicate with the hook commands: + +* `ANNEX_KEY` - name of a key to store, retrieve, remove, or check. +* `ANNEX_FILE` - a file containing the key's content +* `ANNEX_HASH_1` - short stable value, based on the key, can be used for hashing + into 1024 buckets. +* `ANNEX_HASH_2` - another hash value, can be used for a second level of hashing + +The settings to use in git config for the hook commands are as follows: + +* `annex.$hooktype-store-hook` - Command run to store a key in the special remote. + `ANNEX_FILE` contains the content to be stored. + +* `annex.$hooktype-retrieve-hook` - Command run to retrieve a key from the special remote. + `ANNEX_FILE` is a file that the retrieved content should be written to. + The file may already exist with a partial + copy of the content (or possibly just garbage), to allow for resuming + of partial transfers. + +* `annex.$hooktype-remove-hook` - Command to remove a key from the special remote. + +* `annex.$hooktype-checkpresent-hook` - Command to check if a key is present + in the special remote. Should output the key name to stdout, on its own line, + if and only if the key has been actively verified to be present in the + special remote (caching presence information is a very bad idea); + all other output to stdout will be ignored. + +## combined hook program + +Rather than setting all of the above hooks, you can write a single +program that handles everything, and set a single hook to make it be used. + + # git config annex.demo-hook /usr/local/bin/annexdemo + # git annex initremote mydemorepo type=hook hooktype=demo encryption=none + +The program just needs to look at the `ANNEX_ACTION` environment variable +to see what it's being asked to do For example: + +[[!format sh """ +#!/bin/sh +set -e +case "$ANNEX_ACTION" in + store) + demo-upload "$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY" < "$ANNEX_FILE" + ;; + retrieve) + demo-download "$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY" > "$ANNEX_FILE" + ;; + remove) + demo-nuke "$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY" + ;; + checkpresent) + if demo-exists "$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY"; then + echo "$ANNEX_KEY" + fi + ;; + *) + echo "unknown ANNEX_ACTION: $ANNEX_ACTION" >&2 + exit 1 + ;; +esac +"""]] diff --git a/doc/special_remotes/hook/comment_1_6a74a25891974a28a8cb42b87cb53c26._comment b/doc/special_remotes/hook/comment_1_6a74a25891974a28a8cb42b87cb53c26._comment new file mode 100644 index 000000000..2163ba76d --- /dev/null +++ b/doc/special_remotes/hook/comment_1_6a74a25891974a28a8cb42b87cb53c26._comment @@ -0,0 +1,32 @@ +[[!comment format=mdwn + username="helmut" + ip="89.0.176.236" + subject="Asynchronous hooks?" + date="2012-10-13T09:46:14Z" + content=""" +Is there a way to use asynchronous remotes? Interaction with git annex would have to +split the part of initiating some action from completing it. + +I imagine I could `git annex copy` a file to an asynchronous remote and the command +would almost immediately complete. Later I would learn that the transfer is +completed, so the hook must be able to record that information in the `git-annex` +branch. An additional plumbing command seems required here as well as a way to +indicate that even though the store-hook completed, the file is not transferred. + +Similarly `git annex get` would immediately return without actually fetching the +file. This should already be possible by returning non-zero from the retrieve-hook. +Later the hook could use plumbing level commands to actually stick the received file +into the repository. + +The remove-hook should need no changes, but the checkpresent-hook would be more like +a trigger without any actual result. The extension of the plumbing required for the +extension to the receive-hook could update the location log. A downside here is that +you never know when a fsck has completed. + +My proposal does not include a way to track the completion of actions, but relies on +the hook to always complete them reliably. It is not clear that this is the best road +for asynchronous hooks. + +One use case for this would be a remote that is only accessible via uucp. Are there +other use cases? Is the drafted interface useful? +"""]] diff --git a/doc/special_remotes/hook/comment_2_ee7c43b93c5b787216334f019643f6a0._comment b/doc/special_remotes/hook/comment_2_ee7c43b93c5b787216334f019643f6a0._comment new file mode 100644 index 000000000..ec64dd96d --- /dev/null +++ b/doc/special_remotes/hook/comment_2_ee7c43b93c5b787216334f019643f6a0._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg" + nickname="Bence" + subject="More environment variables" + date="2013-07-09T10:28:58Z" + content=""" +Could you please include the original filename+path in the environment variables (_next to ANNEX_KEY & ANNEX_FILE_)? Like ANNEX_FILENAME and ANNEX_PATH. + +Having these infos in a hook would help eg. a flickr backend to be more usefull. [Tags](http://www.flickr.com/help/tags/) would contain the ANNEX_KEY and the image title could be the original filename (ANNEX_FILENAME). Also, having directory path (ANNEX_PATH) for the given file, the uploading process could put images into the proper sets/collections. Voila, you have a \"filesystem based\" flickr image gallery (almost like flickrfs). + +Other backends, like AmazonS3 having [meta data](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html) also could benefit from this. + +To build the Death Star further, an _annex.$hooktype-**sync**-hook_ would instruct the backend to sync data, eg. place or move images/files in the proper image-sets/directories after they are moved/repositioned in git-annex, but that would be the backend's job, not git-annex's. Maybe the sync-hook would be called when _git annex sync_ is called. This is just an idea. + +While writing this, a new hook for sharing came into my mind: _annex.$hooktype-**share**-hook_. +Calling this on a file/directory (_git annex share my_image_to_share.jpg_) would return a publicly shareable (short)url pointing to the file/directory. This would work for web-backends like AmazonS3, flickr, DropBox, Google Drive, ... +"""]] diff --git a/doc/special_remotes/hook/comment_3_2593291795e732994862d08bf2ed467b._comment b/doc/special_remotes/hook/comment_3_2593291795e732994862d08bf2ed467b._comment new file mode 100644 index 000000000..0f8c80254 --- /dev/null +++ b/doc/special_remotes/hook/comment_3_2593291795e732994862d08bf2ed467b._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA" + nickname="Tobias" + subject="comment 3" + date="2013-07-09T10:54:58Z" + content=""" +I have requested this before. But it doesn't seem to be entirely doable because some items may have multiple equally correct filenames/paths. And some items may have zero filenames/paths. + +That said I hope a solution can be found because I really want this feature too. And would implement it in all my hooks. + +And for some of the cases i don't really see it as an issue. If you have a public flickr repo with clean(unencrypted) files. It is because you want to access existing files. If an object has no filename/path the hook could/would/should just ignore the file, sure this means no backup of old versions of files, but you can have other backends for those versions. + +The bigger issue is with the same file existing multiple places in the annex, which filename/path should be used? And the filename/path can change between sync(if it is deleted from one of the positions). I personally still see this as being entirely doable. The key for downloading would always be the same, so worst case scenario is the image may be duplicated on flickr. Or that the picture only one of the multiple folders it should be in on flickr. Still, i see these issues as being minor, and that usability would increase if this was implemented, even with these caveats. + +And there probably is some issues I haven't realized/know about. + +"""]] diff --git a/doc/special_remotes/hook/comment_4_35d79b5ffa5a19056efcdc805070bc4b._comment b/doc/special_remotes/hook/comment_4_35d79b5ffa5a19056efcdc805070bc4b._comment new file mode 100644 index 000000000..988d17def --- /dev/null +++ b/doc/special_remotes/hook/comment_4_35d79b5ffa5a19056efcdc805070bc4b._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg" + nickname="Bence" + subject="checkpresent success and failure" + date="2013-07-31T13:06:21Z" + content=""" +What value should be returned in the \"checkpresent-hook\" to signal that the given file does not exist in the given backend? + +Should the called hook process return an exit code less or greater then zero? In this case the following is displayed: +>(user error (sh [\"-c\",\"name_of_the_process\"] exited 1)) failed + +This tells that the process failed (no internet connection or something that prevents the process from doing its job) and not that result is false, which would mean the file/entry does not exist in the given backend. +If the return code is zero the file is treated as existing file/entry (no matter what I write to stderr). + +Also I think, the \"checkpresent\" block misses the ending ;; in the example. + +Here is my work-in-progress hook: https://gist.github.com/parhuzamos/31bf4516eea434e0d248 +"""]] diff --git a/doc/special_remotes/hook/comment_5_6fbf1e963fa3ea4b2eb8ca5a3819762d._comment b/doc/special_remotes/hook/comment_5_6fbf1e963fa3ea4b2eb8ca5a3819762d._comment new file mode 100644 index 000000000..a4c7b525e --- /dev/null +++ b/doc/special_remotes/hook/comment_5_6fbf1e963fa3ea4b2eb8ca5a3819762d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.21" + subject="comment 5" + date="2013-07-31T16:25:55Z" + content=""" +The checkpresent hook should always exit 0 unless there was an exceptional condition (eg, perhaps it cannot check if the file exists one way or the other). Like the documentation for it says, the important thing is what it outputs to stdout, which should contain the key name if it's present, and should not contain the key name if it's not present. + +I hope you post to this website about your special remote when you get it fully working! +"""]] diff --git a/doc/special_remotes/hook/comment_6_e0ab48d5333e5de85f016b097e6fdac1._comment b/doc/special_remotes/hook/comment_6_e0ab48d5333e5de85f016b097e6fdac1._comment new file mode 100644 index 000000000..e7c218f57 --- /dev/null +++ b/doc/special_remotes/hook/comment_6_e0ab48d5333e5de85f016b097e6fdac1._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg" + nickname="Bence" + subject="comment 6" + date="2013-07-31T17:34:50Z" + content=""" +Roger that. + +If this is acceptable: [terminal output screenshot](http://i.imgur.com/lsJJYwF.png), than I'm almost done and will publish soon. +(Of course a REST API using client would much be better, but this is just the start.) + +"""]] diff --git a/doc/special_remotes/hook/comment_7_cc2b1243c2c36e63241513bcaddfea67._comment b/doc/special_remotes/hook/comment_7_cc2b1243c2c36e63241513bcaddfea67._comment new file mode 100644 index 000000000..68cdf6545 --- /dev/null +++ b/doc/special_remotes/hook/comment_7_cc2b1243c2c36e63241513bcaddfea67._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.21" + subject="comment 7" + date="2013-07-31T17:42:22Z" + content=""" +If I were you I'd suppress that \"File not found\" error. + +Hook special remotes *can* output messages to stderr, and it's also fine to output eg, progress bars to stdout when seding/receving files. But unnecessary cluttery output should be avoided. +"""]] diff --git a/doc/special_remotes/hook/comment_8_bbae315233bda48eb04662dfd48cf1ae._comment b/doc/special_remotes/hook/comment_8_bbae315233bda48eb04662dfd48cf1ae._comment new file mode 100644 index 000000000..81cc32511 --- /dev/null +++ b/doc/special_remotes/hook/comment_8_bbae315233bda48eb04662dfd48cf1ae._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg" + nickname="Bence" + subject="checkpresent again" + date="2013-08-01T23:18:38Z" + content=""" +In the current [HEAD](https://github.com/joeyh/git-annex/commit/bb74db6ef094324062adcf26a677113ee6fd0e58) the \"checkpresent\" method in [Hook.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Hook.hs#L145) is missing a \"return\" while other hooks have a return value eg. [Directory.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Directory.hs#L241), [Rsync.hs](https://github.com/joeyh/git-annex/blob/master/Remote/Rsync.hs#L272), ... + + +I noticed that if my *checkpresent* hook does not output anything (to be exact: I commented out all the lines in the block) it behaves strangely. I copied the test file to \"copyrepo\", removed it by hand (so git-annex does not know about the change) and executed a fsck. + + ~/annex5/test1/123 $ git annex fsck --from copyrepo --fast + fsck somefile.1 (checking copyrepo...) (fixing location log) + ** Based on the location log, somefile.1 + ** was expected to be present, but its content is missing. + failed + +running the check again + + ~/annex5/test1/123 $ ga fsck --from copyrepo --fast + fsck somefile.1 (checking copyrepo...) ok + +and running the check again without --fast + + ~/annex5/test1/123 $ ga fsck --from copyrepo + fsck somefile.1 (checking copyrepo...) ok + +It thinks, the file is in the repo but it is not. + +"""]] diff --git a/doc/special_remotes/hook/comment_9_037523d1994c702239ca96791156fe65._comment b/doc/special_remotes/hook/comment_9_037523d1994c702239ca96791156fe65._comment new file mode 100644 index 000000000..89719adfb --- /dev/null +++ b/doc/special_remotes/hook/comment_9_037523d1994c702239ca96791156fe65._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.145" + subject="comment 9" + date="2013-08-01T23:51:48Z" + content=""" +The behavior you show with `fsck --from` is that the first time it's run against the damaged remote it notices the file is not present using the checkpresent hook. It then updates the location log. The subsequent times it's run, it sees that the location log says the file is not present in the remote. It verifies this is the case by calling the checkpresent hook. Since the two data sources agree, and numcopies is still satisfied, it prints \"ok\". There does not seem to be a bug here. + +(`return` in Haskell does not do what you would expect to happen in a traditional imperative language. It does not alter control flow, and any function using `return` can be mechanically converted to one that does not use `return`.) +"""]] diff --git a/doc/special_remotes/rsync.mdwn b/doc/special_remotes/rsync.mdwn new file mode 100644 index 000000000..b2a9d23f5 --- /dev/null +++ b/doc/special_remotes/rsync.mdwn @@ -0,0 +1,56 @@ +This special remote type rsyncs file contents to somewhere else. + +Setup example: + + # git annex initremote myrsync type=rsync rsyncurl=rsync://rsync.example.com/myrsync keyid=joey@kitenet.net + # git annex describe myrsync "rsync server" + +Or for using rsync over SSH + + # git annex initremote myrsync type=rsync rsyncurl=ssh.example.com:/myrsync keyid=joey@kitenet.net + # git annex describe myrsync "rsync server" + +## configuration + +These parameters can be passed to `git annex initremote` to configure rsync: + +* `encryption` - One of "none", "hybrid", "shared", or "pubkey". + See [[encryption]]. + +* `keyid` - Specifies the gpg key to use for [[encryption]]. + +* `rsyncurl` - Required. This is the url or `hostname:/directory` to + pass to rsync to tell it where to store content. + +* `shellescape` - Optional. Set to "no" to avoid shell escaping normally + done when using rsync over ssh. That escaping is needed with typical + setups, but not with some hosting providers that do not expose rsynced + filenames to the shell. You'll know you need this option if `git annex get` + from the special remote fails with an error message containing a single + quote (`'`) character. If that happens, you can run enableremote + setting shellescape=no. + +The `annex-rsync-options` git configuration setting can be used to pass +parameters to rsync. + +## annex-rsync-transport + +You can use the `annex-rsync-transport` git configuration setting to choose +whether we run rsync over ssh or rsh. This setting is also used to specify +parameters that git annex will pass to ssh/rsh. + +ssh is the default transport; if you'd like to run rsync over rsh, modify your +.git/config to include + + annex-rsync-transport = rsh + +under the appropriate remote. + +To pass parameters to ssh/rsh, include the parameters after "rsh" or +"ssh". For example, to configure ssh to use the private key at +`/path/to/private/key`, specify + + annex-rsync-transport = ssh -i /path/to/private/key + +Note that environment variables aren't expanded here, so for example, you +cannot specify `-i $HOME/.ssh/private_key`. diff --git a/doc/special_remotes/rsync/comment_10_43e8fa3517c1f5935f02ad06fbed63dc._comment b/doc/special_remotes/rsync/comment_10_43e8fa3517c1f5935f02ad06fbed63dc._comment new file mode 100644 index 000000000..185bd97ec --- /dev/null +++ b/doc/special_remotes/rsync/comment_10_43e8fa3517c1f5935f02ad06fbed63dc._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://cstork.org/" + nickname="Chris Stork" + subject="comment 10" + date="2013-08-25T20:59:41Z" + content=""" +@joey I don't understand you last comment where you state that special remotes can act as transfer repositories \"to transfer the files between computers that do not communicate directly\". If there's no communication, ie git pushes or pulls, between the computers then they don't know what file names the files on the special remote map to. They need to somehow communicate the git repo too, don't they? +"""]] diff --git a/doc/special_remotes/rsync/comment_11_8cafc1a8b37e6fb056185ec058c0c3e8._comment b/doc/special_remotes/rsync/comment_11_8cafc1a8b37e6fb056185ec058c0c3e8._comment new file mode 100644 index 000000000..c8fc8831a --- /dev/null +++ b/doc/special_remotes/rsync/comment_11_8cafc1a8b37e6fb056185ec058c0c3e8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 11" + date="2013-08-26T18:36:03Z" + content=""" +@Chris yes in that case you still need, a central git repository (which need not be on a host that supports git-annex), or the assistant can use xmpp to sync the git data. +"""]] diff --git a/doc/special_remotes/rsync/comment_1_9e180c397486989beab21699b8e8f103._comment b/doc/special_remotes/rsync/comment_1_9e180c397486989beab21699b8e8f103._comment new file mode 100644 index 000000000..a4c403e7e --- /dev/null +++ b/doc/special_remotes/rsync/comment_1_9e180c397486989beab21699b8e8f103._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="diepes" + ip="105.227.34.36" + subject="rsync description and example" + date="2013-03-29T15:57:20Z" + content=""" +Hi, I would like to see a example of setting up / using e.g. rsync.net as a repo. + + +Please also provide a highlevel description of how the rsync repo fits in with git-annex. + + +Q1. can you adds files to the remote rsync repo, and will they be detected and synced back ? +Q2. is all the git history rsync'd to remote ? how do i recover if i loose all data except the remote rsync repo ? + +"""]] diff --git a/doc/special_remotes/rsync/comment_2_25545dc0b53f09ae73b29899c8884b02._comment b/doc/special_remotes/rsync/comment_2_25545dc0b53f09ae73b29899c8884b02._comment new file mode 100644 index 000000000..3a6e25b0d --- /dev/null +++ b/doc/special_remotes/rsync/comment_2_25545dc0b53f09ae73b29899c8884b02._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 2" + date="2013-03-29T17:12:47Z" + content=""" +No, special remotes do not contain a copy of the git repository, and no, git-annex does not notice when files are added to remote rsync repositories. Suggest you read [[special_remotes]]. +"""]] diff --git a/doc/special_remotes/rsync/comment_3_960a89b1ae7e3888ffba06baa963dc21._comment b/doc/special_remotes/rsync/comment_3_960a89b1ae7e3888ffba06baa963dc21._comment new file mode 100644 index 000000000..37911a7e7 --- /dev/null +++ b/doc/special_remotes/rsync/comment_3_960a89b1ae7e3888ffba06baa963dc21._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="diepes" + ip="105.227.34.36" + subject="sshfs ? and no sync from special remote to 2nd git-annex ?" + date="2013-03-29T22:09:49Z" + content=""" +* It sound to me it would be 1st prize if the cloud provider supported the git-annex functionality over ssh. +* Then it could be a full git-annex repo, and used for recovery if my laptop with the git info gets lost. +* From special remotes \"These can be used just like any normal remote by git-annex\" +* Your comment \"No, special remotes do not contain a copy of the git repository\" + + * so a special remote is + +1. \"A. just a remote filesystem, that contains the objects with sha1 names ? \" +2. \"B. there is no git info and thus no file name & directory details or am i missing something ?\" +3. \"C. you can't use the remote as a cloud drive to sync changes e.g. file moves, renames between two other git-annex repositories ? (no meta data)\" +4. \"D. the data on the cloud/rsync storage can not be used directly it has to moved into a git-annex capable storage location. \" + +* Would it not be better to mount the remote storage over ssh(sshfs) and then use full git-annex on mounted directory ? +"""]] diff --git a/doc/special_remotes/rsync/comment_4_db84816c31239953dd21f23a8c557b43._comment b/doc/special_remotes/rsync/comment_4_db84816c31239953dd21f23a8c557b43._comment new file mode 100644 index 000000000..4e95fac20 --- /dev/null +++ b/doc/special_remotes/rsync/comment_4_db84816c31239953dd21f23a8c557b43._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="diepes" + ip="41.160.8.49" + subject="rsync.net support git and rsync, not git-annex" + date="2013-05-13T14:44:53Z" + content=""" +how would i use rsync.net as a full repository ? (annex files and git repo) + +It support's rsync, and I discovered now git as well. + +I had a look at the webapp and can't figure out how to set it up, so I can have multiple remote's sync to rsync.net for annex files and git sync. + +Link: http://www.rsync.net/resources/howto/github_integration.html + +"""]] diff --git a/doc/special_remotes/rsync/comment_5_ccaffa4aded9dab88c76a856b96ea5b9._comment b/doc/special_remotes/rsync/comment_5_ccaffa4aded9dab88c76a856b96ea5b9._comment new file mode 100644 index 000000000..3368b59ed --- /dev/null +++ b/doc/special_remotes/rsync/comment_5_ccaffa4aded9dab88c76a856b96ea5b9._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="cehteh" + ip="217.8.62.137" + subject="rsync daemon mode" + date="2013-07-27T01:35:37Z" + content=""" +rsync has a --daemon mode with a simple challenge-response authentication but no encryption. This offers a nice lightweight alternative to ssh, especially when we +store/transfer encrypted content anyways. Is this already supported in git-annex, if yes how to set it up? +"""]] diff --git a/doc/special_remotes/rsync/comment_6_e687b9482b177e1351c8c65ea617d3fa._comment b/doc/special_remotes/rsync/comment_6_e687b9482b177e1351c8c65ea617d3fa._comment new file mode 100644 index 000000000..a5cf0514b --- /dev/null +++ b/doc/special_remotes/rsync/comment_6_e687b9482b177e1351c8c65ea617d3fa._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.246.110" + subject="comment 6" + date="2013-07-28T00:11:38Z" + content=""" +The first example on this page shows using rsync:// to store files on a system using the rsync daemon. +"""]] diff --git a/doc/special_remotes/rsync/comment_7_e122979ea811d9ef835ba05bb936190f._comment b/doc/special_remotes/rsync/comment_7_e122979ea811d9ef835ba05bb936190f._comment new file mode 100644 index 000000000..8cb1d72b4 --- /dev/null +++ b/doc/special_remotes/rsync/comment_7_e122979ea811d9ef835ba05bb936190f._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://olivier.berger.myopenid.com/" + nickname="obergix" + subject="rsync remote is basically more intended for backups ?" + date="2013-08-17T17:40:47Z" + content=""" +If I get it correctly, it is mainly useable as a backup, which will accumulate contents of the objects managed by git-annex over time. + +It would be great to have a use case illustrating its use in concrete matters. Thanks in advance. +"""]] diff --git a/doc/special_remotes/rsync/comment_8_d566113318d0aa7500d76ffe1bd46069._comment b/doc/special_remotes/rsync/comment_8_d566113318d0aa7500d76ffe1bd46069._comment new file mode 100644 index 000000000..1e6bf8aa4 --- /dev/null +++ b/doc/special_remotes/rsync/comment_8_d566113318d0aa7500d76ffe1bd46069._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 8" + date="2013-08-22T18:02:00Z" + content=""" +There are many use cases for a rsync special remote. You could use it as a backup. You could use it to archive files offline in a drive with encryption enabled so if the drive is stolen your data is not. You could `git annex move --to rsyncremote` large files when your local drive is getting full, and then `git annex move` the files back when free space is again available. You could have one repository copy files to a rsync remote, and then `git annex get` them on another repository, to transfer the files between computers that do not communicate directly. The git-annex assistant makes it easy to set up rsync remotes using this last scenario, which is referred to as a transfer repository, and arranges to drop files from the transfer repository once they have been transferred to all known clients. + +None of these use cases are particular to rsync remotes. Most special remotes can all be used in these and other ways. It largely doesn't matter for your use what underlying transport the special remote uses. +"""]] diff --git a/doc/special_remotes/rsync/comment_9_5dcf10a502b2d4feac46b620d43e9d00._comment b/doc/special_remotes/rsync/comment_9_5dcf10a502b2d4feac46b620d43e9d00._comment new file mode 100644 index 000000000..b7690cf68 --- /dev/null +++ b/doc/special_remotes/rsync/comment_9_5dcf10a502b2d4feac46b620d43e9d00._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://olivier.berger.myopenid.com/" + nickname="obergix" + subject="Added use cases to "special remotes"" + date="2013-08-22T20:23:13Z" + content=""" +Thanks @joeyh. I've taken the liberty to add your use case description to [[special remotes]]. Hope this helps. +"""]] diff --git a/doc/special_remotes/web.mdwn b/doc/special_remotes/web.mdwn new file mode 100644 index 000000000..cd20a93bb --- /dev/null +++ b/doc/special_remotes/web.mdwn @@ -0,0 +1,11 @@ +git-annex can use the WWW as a special remote, downloading urls to files. +See [[tips/using_the_web_as_a_special_remote]] for usage examples. + +## notes + +Currently git-annex only supports downloading content from the web; +it cannot upload to it or remove content. + +This special remote uses arbitrary urls on the web as the source for content. +git-annex can also download content from a normal git remote, accessible by +http. diff --git a/doc/special_remotes/web/comment_1_0bd570025f6cd551349ea88a4729ac8e._comment b/doc/special_remotes/web/comment_1_0bd570025f6cd551349ea88a4729ac8e._comment new file mode 100644 index 000000000..d01e17da3 --- /dev/null +++ b/doc/special_remotes/web/comment_1_0bd570025f6cd551349ea88a4729ac8e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://olivier.berger.myopenid.com/" + nickname="obergix" + subject="Which URL prefix are supported ?" + date="2013-08-17T08:44:05Z" + content=""" +It is not clear whether only http:// URLs are supported. Can you list others ? +"""]] diff --git a/doc/special_remotes/web/comment_2_333141cc9ec6c26ffd19aa95303a91e3._comment b/doc/special_remotes/web/comment_2_333141cc9ec6c26ffd19aa95303a91e3._comment new file mode 100644 index 000000000..ff2018117 --- /dev/null +++ b/doc/special_remotes/web/comment_2_333141cc9ec6c26ffd19aa95303a91e3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 2" + date="2013-08-17T08:59:11Z" + content=""" +When it says \"arbitrary urls\", it means it. The only requirement is that the url be well formed and that wget or whatever command you have it configured to use via annex.web-download-command knows how to download it. +"""]] diff --git a/doc/special_remotes/webdav.mdwn b/doc/special_remotes/webdav.mdwn new file mode 100644 index 000000000..251075b09 --- /dev/null +++ b/doc/special_remotes/webdav.mdwn @@ -0,0 +1,42 @@ +This special remote type stores file contents in a WebDAV server. + +## configuration + +The environment variables `WEBDAV_USERNAME` and `WEBDAV_PASSWORD` are used +to supply login credentials. You need to set these only when running +`git annex initremote`, as they will be cached in a file only you +can read inside the local git repository. + +A number of parameters can be passed to `git annex initremote` to configure +the webdav remote. + +* `encryption` - One of "none", "hybrid", "shared", or "pubkey". + See [[encryption]]. + +* `keyid` - Specifies the gpg key to use for [[encryption]]. + +* `embedcreds` - Optional. Set to "yes" embed the login credentials inside + the git repository, which allows other clones to also access them. This is + the default when gpg encryption is enabled; the credentials are stored + encrypted and only those with the repository's keys can access them. + + It is not the default when using shared encryption, or no encryption. + Think carefully about who can access your repository before using + embedcreds without gpg encryption. + +* `url` - Required. The URL to the WebDAV directory where files will be + stored. This can be a subdirectory of a larger WebDAV repository, and will + be created as needed. Use of a https URL is strongly + encouraged, since HTTP basic authentication is used. + +* `chunksize` - Avoid storing files larger than the specified size in + WebDAV. For use when the WebDAV server has file size + limitations. The default is to never chunk files. + The value can use specified using any commonly used units. + Example: `chunksize=75 megabytes` + Note that enabling chunking on an existing remote with non-chunked + files is not recommended. + +Setup example: + + # WEBDAV_USERNAME=joey@kitenet.net WEBDAV_PASSWORD=xxxxxxx git annex initremote box.com type=webdav url=https://www.box.com/dav/git-annex chunksize=75mb keyid=joey@kitenet.net diff --git a/doc/special_remotes/webdav/comment_1_6b523eea78eae1d19fe2a9950ee33e3a._comment b/doc/special_remotes/webdav/comment_1_6b523eea78eae1d19fe2a9950ee33e3a._comment new file mode 100644 index 000000000..768720fe5 --- /dev/null +++ b/doc/special_remotes/webdav/comment_1_6b523eea78eae1d19fe2a9950ee33e3a._comment @@ -0,0 +1,26 @@ +[[!comment format=mdwn + username="http://rfhbuk.myopenid.com/" + ip="109.145.123.81" + subject="git-annex initremote failing for webdav servers" + date="2012-12-01T10:18:04Z" + content=""" +Unfortunately, trying to set up the following webdav servers fail. Some of the terminal output of git-annex --debug initremote ... is below, but it may not really helpful. Can I help any further, can a webdav remote be set up in a different way? (The box.com webdav initremote worked fine.) Cheers + + +With livedrive.com: + + [2012-12-01 10:15:12 GMT] chat: gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--quiet\",\"--trust-model\",\"always\",\"--encrypt\",\"--no-encrypt-to\",\"--no-default-recipient\",\"--recipient\",\"xxxxxxxxxxxxxx\"] + (encryption setup with gpg key xxxxxxxxxxxxxx) (testing WebDAV server...) + git-annex: \"Bad Request\": user error + failed + git-annex: initremote: 1 failed + + +With sd2dav.1und1.de: + + [2012-12-01 10:13:10 GMT] chat: gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--quiet\",\"--trust-model\",\"always\",\"--encrypt\",\"--no-encrypt-to\",\"--no-default-recipient\",\"--recipient\",\"xxxxxxxxxxxxxx\"] + (encryption setup with gpg key xxxxxxxxxxxxxx) (testing WebDAV server...) + git-annex: \"Locked\": user error + failed + git-annex: initremote: 1 failed +"""]] diff --git a/doc/special_remotes/webdav/comment_2_83fc4e7d9ba7a05c8500da659f561b8f._comment b/doc/special_remotes/webdav/comment_2_83fc4e7d9ba7a05c8500da659f561b8f._comment new file mode 100644 index 000000000..8a23ad493 --- /dev/null +++ b/doc/special_remotes/webdav/comment_2_83fc4e7d9ba7a05c8500da659f561b8f._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.49" + subject="comment 2" + date="2012-12-01T18:34:05Z" + content=""" +I tried signing up for livedrive, but I cannot log into it with WebDav at all. Do they require a Pro account to use WebDav? + +When it's \"testing webdav server\", it tries to make a collection (a subdirectory), and uploads a file to it, and sets the file's properties, and deletes the file. One of these actions must be failing, perhaps because the webdav server implementation does not support it. Or perhaps because the webdav client library is doing something wrong. I've instrumented the test, so it'll say which one. +"""]] diff --git a/doc/special_remotes/webdav/comment_3_239367ad639c61ecdf87a89f7ac53efe._comment b/doc/special_remotes/webdav/comment_3_239367ad639c61ecdf87a89f7ac53efe._comment new file mode 100644 index 000000000..2d559b7af --- /dev/null +++ b/doc/special_remotes/webdav/comment_3_239367ad639c61ecdf87a89f7ac53efe._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.49" + subject="comment 3" + date="2012-12-01T20:13:36Z" + content=""" +I've identified the problem keeping it working with livedrive. Once a patch I've written is applied to the Haskell DAV library, I'll be able to update git-annex to support it. + +I don't know about sd2dav.1und1.de. The error looks like it doesn't support WebDAV file locking. +"""]] diff --git a/doc/special_remotes/webdav/comment_4_ffa52f7776cdc8caa28667b5eadae123._comment b/doc/special_remotes/webdav/comment_4_ffa52f7776cdc8caa28667b5eadae123._comment new file mode 100644 index 000000000..24dd85bf5 --- /dev/null +++ b/doc/special_remotes/webdav/comment_4_ffa52f7776cdc8caa28667b5eadae123._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.49" + subject="comment 4" + date="2012-12-01T21:14:34Z" + content=""" +Good news, livedrive is supported now, by git-annex's master branch. +"""]] diff --git a/doc/special_remotes/webdav/comment_5_5b8cbdb5e9a1b90d748a5074997e1cd5._comment b/doc/special_remotes/webdav/comment_5_5b8cbdb5e9a1b90d748a5074997e1cd5._comment new file mode 100644 index 000000000..7d5f3af5d --- /dev/null +++ b/doc/special_remotes/webdav/comment_5_5b8cbdb5e9a1b90d748a5074997e1cd5._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://peter-simons.myopenid.com/" + ip="192.100.130.7" + subject="WebDAV without locking" + date="2013-01-16T12:44:16Z" + content=""" +Hi Joey, + +you are right, the 1-und-1.de implementation of WebDAV does not support locking. + +Do you think there is a way to make that service work with git-annex anyhow -- say, by allowing the user to disable file-locking? I've worked around the problem with 1-und-1.de by using fuse+davfs2 to mount the storage in my file system and access it through the directory backend, but FUSE is dead slow, unfortunately, and this solution really sucks. +"""]] diff --git a/doc/special_remotes/webdav/comment_6_d3be3e588c3a2abb2025ceb82c18b0ef._comment b/doc/special_remotes/webdav/comment_6_d3be3e588c3a2abb2025ceb82c18b0ef._comment new file mode 100644 index 000000000..150c984a8 --- /dev/null +++ b/doc/special_remotes/webdav/comment_6_d3be3e588c3a2abb2025ceb82c18b0ef._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.7.238" + subject="comment 6" + date="2013-01-17T18:23:27Z" + content=""" +It seems it must advertise it supports the LOCK and UNLOCK http actions, but fails when they're used. + +The DAV library I am using always locks if it seems the server supports it. So this will need changes to that library. I've filed a bug requesting the changes. <http://bugs.debian.org/698379> +"""]] diff --git a/doc/special_remotes/webdav/comment_7_6fa7e11331db5a943015bd5367eb3d73._comment b/doc/special_remotes/webdav/comment_7_6fa7e11331db5a943015bd5367eb3d73._comment new file mode 100644 index 000000000..324760c4c --- /dev/null +++ b/doc/special_remotes/webdav/comment_7_6fa7e11331db5a943015bd5367eb3d73._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo" + nickname="Tobias" + subject="SSL certificates" + date="2013-03-25T16:09:54Z" + content=""" +Trying to use webdav leads in: + + git-annex: HandshakeFailed (Error_Protocol (\"certificate rejected: not coping with anything else Start (Container Context 0)\",True,CertificateUnknown)) + +How can I disable the SSL certificate check? +"""]] diff --git a/doc/special_remotes/webdav/comment_8_2627b41f80c7511b27464e2040b128a8._comment b/doc/special_remotes/webdav/comment_8_2627b41f80c7511b27464e2040b128a8._comment new file mode 100644 index 000000000..96ae19d11 --- /dev/null +++ b/doc/special_remotes/webdav/comment_8_2627b41f80c7511b27464e2040b128a8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 8" + date="2013-03-27T16:38:47Z" + content=""" +I don't think the checking of SSL certificates can currently be disabled. +"""]] diff --git a/doc/special_remotes/xmpp.mdwn b/doc/special_remotes/xmpp.mdwn new file mode 100644 index 000000000..86f6a7c0b --- /dev/null +++ b/doc/special_remotes/xmpp.mdwn @@ -0,0 +1,39 @@ +XMPP (Jabber) is used by the [[assistant]] as a git remote. This is, +technically not a git-annex special remote (large files are not transferred +over XMPP; only git commits are sent). + +Typically XMPP will be set up using the web app, but here's how a manual +set up could be accomplished: + +1. xmpp login credentials need to be stored in `.git/annex/creds/xmpp`. + Obviously this file should be mode 600. An example file: + + XMPPCreds {xmppUsername = "joeyhess", xmppPassword = "xxxx", xmppHostname = "xmpp.l.google.com.", xmppPort = 5222, xmppJID = "joeyhess@gmail.com"} + +2. A git remote is created using a special url, of the form `xmpp::user@host` + For the above example, it would be `url = xmpp::joeyhess@gmail.com` + +3. The uuid of one of the other clients using XMPP should be configured + using the `annex.uuid` setting, the same as is set up for other remotes. + +With the above configuration, the [[assistant]] will use xmpp remotes much as +any other git remote. Since XMPP requires a client that is continually running +to see incoming pushes, the XMPP remote cannot be used with git at the +command line. + +## XMPP server support status +[[!table data=""" +Provider|Status|Type|Notes +[[Gmail|http://gmail.com]]|Working|?|Google Apps: [setup your SRV records](http://www.olark.com/gtalk/check_srv) or configure `.git/annex/creds/xmpp` manually +[[Coderollers|http://www.coderollers.com/xmpp-server/]]|Working|[[Openfire|http://www.igniterealtime.org/projects/openfire/]] +[[jabber.me|http://jabber.me/]]|Working|[[Tigase|http://www.tigase.org/]] +[[xmpp.ru.net|https://www.xmpp.ru.net]]|Working|[[jabberd2|http://jabberd2.org/]] +[[jabber.org|http://jabber.org]]|Working|[[Isode M-Link|http://www.isode.com/products/m-link.html]] +-|Working|[[Prosody|http://prosody.im/]]|No providers tested. +-|Working|[[Metronome|http://www.lightwitch.org/]]|No providers tested. +-|[[Failing|http://git-annex.branchable.com/forum/XMPP_authentication_failure/]]|ejabberd|[[Authentication bug|https://support.process-one.net/browse/EJAB-1632]]: Fixed in debian unstable with version 2.1.10-5 +-|[[Failing|http://git-annex.branchable.com/forum/XMPP_authentication_failure/#comment-4ce5aeabd12ca3016290b3d8255f6ef1]]|jabberd14|No further information +"""]] +List of providers: [[http://xmpp.net/]] + +See also: [[xmpp_protocol_design_notes|design/assistant/xmpp]] diff --git a/doc/special_remotes/xmpp/comment_10_c7c2e2e81cb5b2b9a5272430c835dd39._comment b/doc/special_remotes/xmpp/comment_10_c7c2e2e81cb5b2b9a5272430c835dd39._comment new file mode 100644 index 000000000..0380af359 --- /dev/null +++ b/doc/special_remotes/xmpp/comment_10_c7c2e2e81cb5b2b9a5272430c835dd39._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmZilYULa6CDEGfuagoDlesyakBgnf-dF8" + nickname="Maarten" + subject="Google Apps & SRV" + date="2013-11-27T18:13:18Z" + content=""" +To make XMPP via a Google Apps account work out of the box, all you need to do is add the correct SRV records to your domain's DNS zone. If you do, you don't need to manually create creds/xmpp, you can just use the web interface, enter your full apps email address and password (remember, if 2-factor, this needs to be an application-specific password). + +To create the correct SRV records (and check whether they're correct), see http://www.olark.com/gtalk/check_srv +"""]] diff --git a/doc/special_remotes/xmpp/comment_1_568247938929a2934e8198fca80b7184._comment b/doc/special_remotes/xmpp/comment_1_568247938929a2934e8198fca80b7184._comment new file mode 100644 index 000000000..060aa5bd7 --- /dev/null +++ b/doc/special_remotes/xmpp/comment_1_568247938929a2934e8198fca80b7184._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA" + nickname="Tobias" + subject="User defined server" + date="2013-04-17T09:45:59Z" + content=""" +It would be nice if you could expand the XMPP setup in the assistant to support an \"advanced\" settings view where a custom server could be defined. + +Example: I have a google Apps domain called mytest.com, with the users bla1@mytest.com and bla2@mytest.com. When trying to add either of those accounts to the assistant XMPP will try to use mytest.com as the jabber server, and not googles server. + +"""]] diff --git a/doc/special_remotes/xmpp/comment_2_9fc3f512020b7eb2591d6b7b2e8de2d7._comment b/doc/special_remotes/xmpp/comment_2_9fc3f512020b7eb2591d6b7b2e8de2d7._comment new file mode 100644 index 000000000..02f70d0f2 --- /dev/null +++ b/doc/special_remotes/xmpp/comment_2_9fc3f512020b7eb2591d6b7b2e8de2d7._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q" + nickname="Andrew" + subject="comment 2" + date="2013-04-17T22:28:39Z" + content=""" +Yeah, I agree, this would be nice. + +For your own domain, you can configure DNS like this: [[http://support.google.com/a/bin/answer.py?hl=en&answer=34143]] to make XMPP find the right server. But for some that's not an option and the \"advanced\" mode would be useful in that case. +"""]] diff --git a/doc/special_remotes/xmpp/comment_3_48ddbba1402d89acaea07cff747c48e0._comment b/doc/special_remotes/xmpp/comment_3_48ddbba1402d89acaea07cff747c48e0._comment new file mode 100644 index 000000000..7643d4d7d --- /dev/null +++ b/doc/special_remotes/xmpp/comment_3_48ddbba1402d89acaea07cff747c48e0._comment @@ -0,0 +1,28 @@ +[[!comment format=mdwn + username="RaspberryPie" + ip="46.19.143.203" + subject="Missing prerequisites for XMPP syncing?" + date="2013-09-17T06:53:59Z" + content=""" +I set up two fresh annexes that can talk via XMPP and no other way. After I fire up the assistants I expect them to sync their metadata, but nothing happens. One log gives me an 'XMPPClient: received: [\"Unknown message\"]' message every two minutes. The other one doesn't contain the string XMPP at all, not once. So my suspicion is that this particular version of git-annex doesn't support XMPP, which is weird because: + + $ git annex version + git-annex version: 4.20130909 + build flags: Assistant Pairing Testsuite S3 Inotify XMPP DNS Feeds + local repository version: 3 + default repository version: 3 + supported repository versions: 3 4 + upgrade supported from repository versions: 0 1 2 + +This is the version output from the other machine: + + $ git annex version + git-annex version: 4.20130827 + build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP + local repository version: 3 + default repository version: 3 + supported repository versions: 3 4 + upgrade supported from repository versions: 0 1 2 + +What am I missing? Are there more build flags for XMPP than the one called XMPP? (Also, no, I can't just copy versions between machines b/c the architectures are different. And yep, the one giving me trouble is ARM.) +"""]] diff --git a/doc/special_remotes/xmpp/comment_4_59857879abaae22bde444a215e00bf18._comment b/doc/special_remotes/xmpp/comment_4_59857879abaae22bde444a215e00bf18._comment new file mode 100644 index 000000000..db217cff9 --- /dev/null +++ b/doc/special_remotes/xmpp/comment_4_59857879abaae22bde444a215e00bf18._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 4" + date="2013-09-19T21:07:35Z" + content=""" +If you have the XMPP flag in your git-annex build, it will support XMPP. Are you sure you set up the xmpp creds file and the xmpp special remote correctly on the ARM machine? (I assume it has no webapp, so you had to set that up manually..) + +Here's how you can do that manually: + +1. Run git-annex on a machine with the webapp, set up XMPP, and copy the .git/annex/creds/xmpp to the machine without the webapp. +2. On the machine without the webapp, add a git remote that has its \"url = xmpp::loginname@xmppserver.com\" and its annex-uuid set to the annex.uuid of the repository on the first machine. +3. Run git-annex assistant on the machine without the webapp. +"""]] diff --git a/doc/special_remotes/xmpp/comment_5_583ee374bd34fcc9ae26c2fd690e8c47._comment b/doc/special_remotes/xmpp/comment_5_583ee374bd34fcc9ae26c2fd690e8c47._comment new file mode 100644 index 000000000..298c4392a --- /dev/null +++ b/doc/special_remotes/xmpp/comment_5_583ee374bd34fcc9ae26c2fd690e8c47._comment @@ -0,0 +1,73 @@ +[[!comment format=mdwn + username="RaspberryPie" + ip="37.221.161.234" + subject="Nope" + date="2013-09-24T22:05:55Z" + content=""" +Your guess is right, Joey, I'm configuring by hand as the ARM machine has no webapp. And yes, I'm mostly sure I set up everything correctly. The XMPP account is working, and my configuration of git-annex is all but identical to your example. + +Here's what I do. First on the machine with the webapp: + + mkdir ~/test + cd ~/test + git init + git annex init + git annex webapp + +I set up XMPP from within the webapp. The file ~/test/.git/annex/creds/xmpp is created with the correct credentials. (BTW: The file's default permissions are 620 instead of 600 - is that a bug?) + +I add a file or two to the annex for good measure. Then, on the ARM machine: + + mkdir ~/test + cd ~/test + git init + git annex init + mkdir .git/annex/creds + scp -2 webappmachine:~/test/.git/annex/creds/xmpp .git/annex/creds + chmod 600 .git/annex/creds/xmpp + git remote add webappmachine xmpp::login@server + +The final step is to edit .git/config on the ARM machine. The [remote] section now looks like this: + + [remote \"webappmachine\"] + url = xmpp::login@server + fetch = +refs/heads/*:refs/remotes/webappmachine/* + annex-uuid = aaaaaaaa-bbbb-cccc-dddddddddddd + +where aaaaaaaa-bbbb-cccc-dddddddddddd is the return value of `git config --get annex.uuid` on the webapp machine. + +I then run `git annex assistant` on the ARM machine and expect the two machines to synchronize their metadata, e.g. the number of knownn annex keys in the repo. But it doesn't happen. + +So I set `debug = true`, restart the assistants and check the log. This is what I get on the webapp machine: + + [2013-09-24 17:45:41 EDT] XMPPClient: connected a5/25577ac4-3248-4c83-8391-bd93708bcf2b + [2013-09-24 17:45:41 EDT] XMPPClient: received: [\"Presence from a5/dc9bcde8-fe18-47de-807c-c620019279f2 Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [(Name {nameLocalName = \\"query\\", nameNamespace = Nothing, namePrefix = Nothing},[ContentText \\"\\"])], elementNodes = []})\",\"QueryPresence\"] + [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Presence from a5/900e3b6e-a7f4-4a6a-8d12-ed94de429258 Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [(Name {nameLocalName = \\"push\\", nameNamespace = Nothing, namePrefix = Nothing},[ContentText \\"43357474-abbb-4667-a334-e4615ea6d4a2\\"])], elementNodes = []})\",\"NotifyPush [UUID \\"43357474-abbb-4667-a334-e4615ea6d4a2\\"]\"] + [2013-09-24 17:45:42 EDT] XMPPClient: push notification for + [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"symbolic-ref\",\"HEAD\"] + [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"show-ref\",\"refs/heads/master\"] + [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Pushing \\"a59\\" (CanPush (UUID \\"d50c4cc9-e7c0-4ef0-84c6-f11012051eb9\\") [34f875cc7fa1198414f93990af9ab78e6cee893e,6fad42234060361435d6cf2ab4bd40e438c2d05c])\"] + [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"show-ref\",\"git-annex\"] + [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] + [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"log\",\"refs/heads/git-annex..6fad42234060361435d6cf2ab4bd40e438c2d05c\",\"--oneline\",\"-n1\"] + [2013-09-24 17:45:42 EDT] chat: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"cat-file\",\"--batch\"] + [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Ignorable Presence from a5/25577ac4-3248-4c83-8391-bd93708bcf2b Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [], elementNodes = []})\"] + [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Unknown message\"] + [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Pushing \\"a59\\" (PushRequest (UUID \\"d50c4cc9-e7c0-4ef0-84c6-f11012051eb9\\"))\"] + [2013-09-24 17:45:42 EDT] XMPPSendPack: started running push Pushing \"a59\" (PushRequest (UUID \"d50c4cc9-e7c0-4ef0-84c6-f11012051eb9\")) + [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"symbolic-ref\",\"HEAD\"] + [2013-09-24 17:45:42 EDT] XMPPClient: received: [\"Ignorable Presence from a5/25577ac4-3248-4c83-8391-bd93708bcf2b Just (Element {elementName = Name {nameLocalName = \\"git-annex\\", nameNamespace = Just \\"git-annex\\", namePrefix = Nothing}, elementAttributes = [], elementNodes = []})\"] + [2013-09-24 17:45:42 EDT] read: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"show-ref\",\"refs/heads/master\"] + [2013-09-24 17:45:42 EDT] call: git [\"--git-dir=/home/pi/test/.git\",\"--work-tree=/home/pi/test\",\"branch\",\"-f\",\"synced/master\"] + [2013-09-24 17:45:42 EDT] XMPPSendPack: finished running push Pushing \"a59\" (PushRequest (UUID \"d50c4cc9-e7c0-4ef0-84c6-f11012051eb9\")) False + +And from then on, in two-minute intervals: + + [2013-09-24 17:47:42 EDT] XMPPClient: received: [\"Unknown message\"] + [2013-09-24 17:49:42 EDT] XMPPClient: received: [\"Unknown message\"] + [2013-09-24 17:51:42 EDT] XMPPClient: received: [\"Unknown message\"] + +The log on the ARM machine is rather unhelpful. Actually it doesn't even contain the string \"XMPP\". This looks to me like the webapp machine tries to communicate via Jabber but doesn't get any intelligible answer. And this is the reason I wondered whether the problem lies with my self-compiled ARM git-annex binary. I actually spent a while compiling 4.20130909 with all flags but webapp and webdav, but the result is still the same. + +Any other ideas what I'm doing wrong here? +"""]] diff --git a/doc/special_remotes/xmpp/comment_6_8f0b5bba1271d031a67e7f0c175d67d5._comment b/doc/special_remotes/xmpp/comment_6_8f0b5bba1271d031a67e7f0c175d67d5._comment new file mode 100644 index 000000000..750e0874a --- /dev/null +++ b/doc/special_remotes/xmpp/comment_6_8f0b5bba1271d031a67e7f0c175d67d5._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.220" + subject="comment 6" + date="2013-09-25T18:13:24Z" + content=""" +If you're not getting an \"XMPPClient: connected\", then my guess would be that your git-annex build's XMPP is screwed up somehow. For example, if it hung forever when connecting to the XMPP server, it would never get as far as printing that message. (If it tried and failed to connect, you'd get a message about the connection having failed.) +"""]] diff --git a/doc/special_remotes/xmpp/comment_7_ac7acbded03325b015959d82ae77faf1._comment b/doc/special_remotes/xmpp/comment_7_ac7acbded03325b015959d82ae77faf1._comment new file mode 100644 index 000000000..0ad65336b --- /dev/null +++ b/doc/special_remotes/xmpp/comment_7_ac7acbded03325b015959d82ae77faf1._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="RaspberryPie" + ip="46.165.221.166" + subject="comment 7" + date="2013-09-26T03:46:18Z" + content=""" +I see. Is there a way to check whether the build is corrupt? The build logs gave me nothing. + +Anyway, XMPP is not the most important feature to me. It still bugs me though that it doesn't work when it should. +"""]] diff --git a/doc/special_remotes/xmpp/comment_8_81a9636a1e8a36a58185468a26f8633d._comment b/doc/special_remotes/xmpp/comment_8_81a9636a1e8a36a58185468a26f8633d._comment new file mode 100644 index 000000000..79dd2892b --- /dev/null +++ b/doc/special_remotes/xmpp/comment_8_81a9636a1e8a36a58185468a26f8633d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnZEanlyzay_QlEAL0CWpyZcRTyN7vay8U" + nickname="Carlo" + subject="comment 8" + date="2013-10-22T14:30:58Z" + content=""" +Same setup, same problem... no log output on raspberry pi. +"""]] diff --git a/doc/special_remotes/xmpp/comment_9_eda76b826491c96b1ce072aacf9d3adf._comment b/doc/special_remotes/xmpp/comment_9_eda76b826491c96b1ce072aacf9d3adf._comment new file mode 100644 index 000000000..3505f3aaf --- /dev/null +++ b/doc/special_remotes/xmpp/comment_9_eda76b826491c96b1ce072aacf9d3adf._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlu-fdXIt_RF9ggvg4zP0yBbtjWQwHAMS4" + nickname="Jörn" + subject="Same problem, no XMPP showing up in daemon.log" + date="2013-11-21T21:13:16Z" + content=""" +I have the same setup like @RaspberryPie, except that my server is not running on the Pi but on Debian7-amd64. On my client (OSX, self-compiled using cabal) I can see XMPP log entries like @RaspberryPi, however, on the Debian7 machine (also self-compiled) I do not see any XMPP entry in the daemon.log. Setup regarding .git/annex/creds/xmpp and the special xmpp remote is correct (checked a thousand times). + +Do you have any idea what could be wrong, Joey? Thanks a lot. + +Output of git annex version: + + git-annex version: 5.20131120 + build flags: Assistant Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS Feeds Quvi CryptoHash + key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL + remote types: git gcrypt S3 bup directory rsync web webdav glacier hook + local repository version: 4 + default repository version: 3 + supported repository versions: 3 5 + upgrade supported from repository versions: 0 1 2 4 + +Jörn +"""]] |