diff options
Diffstat (limited to 'doc/tips')
178 files changed, 4110 insertions, 0 deletions
diff --git a/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__.mdwn b/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__.mdwn new file mode 100644 index 000000000..c32eb966f --- /dev/null +++ b/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__.mdwn @@ -0,0 +1,111 @@ +I've been wrestling with git-annex to try to make it build on Debian, or more specifically, wrestling with Haskell dependencies. + +After a fair amount of futzing around, and pestering a bunch of people in the process (thanks for the help! :) ) I finally managed to make it build. + +I figured I would post the steps here, since it's not completely trivial, and I expect that a few others might be interested in building newer versions as well. + +There appears to currently be two methods: + +* Debian packages on Wheezy plus Sid + * Starting out on Wheezy, and then picking the rest from Sid (it seems at least libghc-safesemaphore-dev from Sid is critical for newer git-annex) + * WebDAV suport will not be available with this method +* Cabal packages + + +#Debian packages on Wheezy plus Sid + +##Start off with a clean wheezy chroot + + sudo debootstrap wheezy debian-wheezy + sudo chroot debian-wheezy + +##Install some build tools + + apt-get update + apt-get install devscripts git + +##Get git-annex (either by cloning or simply moving the source into the chroot) + + mkdir /src + cd /src + git clone git://git-annex.branchable.com/source.git git-annex + cd git-annex + +##Remove WebDAV dependency which can't be satisfied anywhere + + sed '/libghc-dav-dev/d' -i debian/control + +##Create dummy build-depends package and install all available Wheezy dependencies using it + + mk-build-deps + dpkg -i git-annex-build-deps*.deb + apt-get install -f + +(this will remove the build-depends package) + +##Add Sid sources and install all available Sid dependencies + + echo "deb http://http.debian.net/debian sid main" >>/etc/apt/sources.list + apt-get update + dpkg -i git-annex-build-deps*.deb + apt-get install -f + +(the build-depends package should now be fully installed) + +##Disable the 'make test' that fails due to missing hothasktags + + echo >>debian/rules + echo "override_dh_auto_test:" >>debian/rules + +##Build! + + debuild -us -uc -Igit + + +#Cabal packages + +##Start off with a clean Sid(/Wheezy) chroot + + sudo debootstrap sid debian-sid + sudo chroot debian-sid + +##Install a smaller set of tools and build-depends from Debian (cabal needs these to compile the Haskell stuff) + + apt-get update + apt-get install ghc cabal-install devscripts libz-dev pkg-config c2hs libgsasl7-dev libxml2-dev libgnutls-dev c2hs git debhelper ikiwiki perlmagick uuid rsync openssh-client fakeroot + +##Get git-annex (either by cloning or simply moving the source into the chroot) + + mkdir /src + cd /src + git clone git://git-annex.branchable.com/source.git git-annex + cd git-annex + +##Install the Haskell build-dependencies from cabal + + cabal update + cabal install --only-dependencies + +##Optional step which doesn't work (might in the future) +If we want to run the 'make test' after build we need hothasktags, which is only available via cabal + + apt-get install happy + cabal install hothasktags + export PATH=$PATH:~/.cabal/bin + +But this currently fails silently inside make test->fast->tags, and if you dig a bit (manually edit the makefile to be more verbose) you see + + hothasktags: ./Command/AddUnused.hs: hGetContents: invalid argument (invalid byte sequence) + +##Disable the 'make test' that fails + + echo >>debian/rules + echo "override_dh_auto_test:" >>debian/rules + +##Remove all Debian package haskell depends (taken care of by cabal instead) + + sed '/\tlibghc/d' -i debian/control + +## Build! + + debuild -us -uc -Igit diff --git a/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__/comment_1_835a3608df3e9d044cabe822d0f3e7e4._comment b/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__/comment_1_835a3608df3e9d044cabe822d0f3e7e4._comment new file mode 100644 index 000000000..55cf0b97b --- /dev/null +++ b/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__/comment_1_835a3608df3e9d044cabe822d0f3e7e4._comment @@ -0,0 +1,27 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkCw26IdxXXPBoLcZsQFslM67OJSJynb1w" + nickname="Alexander" + subject="can't install git-annex on OS X Mountain Lion without disabling WebDAV support" + date="2013-04-29T17:57:03Z" + content=""" +possibly related to this Debian issue: + +trying to install git-annex with cabal on OS X 10.8.3, the build fails with + + + Loading package DAV-0.4 ... linking ... ghc: + lookupSymbol failed in relocateSection (relocate external) + ~/.cabal/lib/DAV-0.4/ghc-7.4.2/HSDAV-0.4.o: unknown symbol `_DAVzm0zi4_PathszuDAV_version1_closure' + ghc: unable to load package `DAV-0.4' + Failed to install git-annex-4.20130417 + cabal: Error: some packages failed to install: + git-annex-4.20130417 failed during the building phase. The exception was: + ExitFailure 1 + + +This was after following all of the instructions for the Homebrew install at [http://git-annex.branchable.com/install/OSX/](http://git-annex.branchable.com/install/OSX/) +I was able to work around this issue by installing with the WebDAV flag disabled (ie, added the option --flags=\"-WebDAV\" to last command in the OS X install instructions): + + cabal install git-annex --bindir=$HOME/bin --flags=\"-WebDAV\" + +"""]] diff --git a/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__/comment_2_080b30cba72a718e73ea715e259e1cfb._comment b/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__/comment_2_080b30cba72a718e73ea715e259e1cfb._comment new file mode 100644 index 000000000..7e8e295fd --- /dev/null +++ b/doc/tips/Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__/comment_2_080b30cba72a718e73ea715e259e1cfb._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 2" + date="2013-04-30T21:51:50Z" + content=""" +@Alexander that DAV-0.4 problem is a bug in DAV, not git-annex. I've informed its author and it should be fixed soon, in a new version of DAV. +"""]] diff --git a/doc/tips/Crude_Windows_Sync.mdwn b/doc/tips/Crude_Windows_Sync.mdwn new file mode 100644 index 000000000..88138a0b1 --- /dev/null +++ b/doc/tips/Crude_Windows_Sync.mdwn @@ -0,0 +1,35 @@ +Here's a workaround to start syncing folders on Windows right now. It's a bit command line heavy, so you might need to set this up for your users. But I would much rather do this than use some other syncing solution and then have to migrate. + +(1) Create a remote server git annex repository with the assistant on Linux or Mac. + +(2) [Install git](http://git-scm.com/) on the Windows machine. + +(3) [Install git-annex for Windows](http://git-annex.branchable.com/install/Windows/) on the Windows machine. Don't forget to run the installer as administrator. + +(4) Run _Git Bash_ from the system menu, and run these commands to clone your repository. + + ssh-keygen + cat .ssh/id_rsa.pub | ssh username@my-server.com "cat >> ~/.ssh/authorized_keys" + git clone username@my-server.com:/path/to/annex + cd annex + git annex init + +(5) Create a script that will trigger a full sync + + echo ' + #!/bin/bash + git annex sync + git annex get * + git annex add . + git annex sync + git annex copy * --to origin + ' > sync.sh + chmod +x sync.sh + ./sync.sh + +(6) Copy the "Git Bash" shortcut from your windows menu to your desktop, and change the link target to: + + C:\Program Files\Git\bin\sh.exe" --login -i "annex/sync.sh" + +Now ask your users to run this shortcut before and after they change files. You can also put it into the "autostart" folder to sync at boot. + diff --git a/doc/tips/Decentralized_repository_behind_a_Firewall.mdwn b/doc/tips/Decentralized_repository_behind_a_Firewall.mdwn new file mode 100644 index 000000000..9e347c73f --- /dev/null +++ b/doc/tips/Decentralized_repository_behind_a_Firewall.mdwn @@ -0,0 +1,59 @@ +If you're anything like me¹, you have a copy of your annex on a computer running at home², set up so you can access it from anywhere like this: + + ssh myhome.no-ip.org + +This is totally great! Except, there is no way for your home computer to pull your changes, because there is no *on-the-go.no-ip.org*. You can get clunky and use a *bare git repository and git push*, but there is a better way. + +First, install *openssh-server* on your *on-the-go* computer + + sudo apt-get install openssh-server # Adjust to your flavor of unix + +Then, log into your *home* computer, with *port forwarding*: + + ssh me@myhome.no-ip.org -R 2201:localhost:22 + +Your *home* computer can now ssh into your *on-the-go* computer, as long as you keep the above shell running. + +You can now add your *on-the-go* computer as a remote on your *home* computer. Use the port forwarding shell you just connected with the command above, if you like. + + ssh-keygen -t rsa + ssh-copy-id "me@localhost -p 2201" + cd ~/annex + git remote add on-the-go ssh://me@localhost:2201/home/myuser/annex + +Now you can run normal annex operations, as long as the port forwarding shell is running³. + + git annex sync + git annex get on-the-go some/big/file + git annex info + +You can add more computers by repeating with a different port, e.g. 2202 or 2203 (or any other). + +If you're security paranoid (like me), read on. If you're not, that's it! Thanks for reading! + +--- +Paranoid Area + +Note you're granting passwordless access to your on-the-go computer to your home computer. I believe that's all right, as long as: + +* Your home computer is really in your home, and not at a friend's house or some datacenter +* Your home computer can be accessed only by ssh, and not HTTP or Samba or NTP or (shoot me now!) FTP +* Only you (and perhaps trustworthy family) have access to your home computer +* You have reasonably strong passwords or key-only logins on both your home and on-the-go computers. +* You regularly install security updates on both computers (sudo apt-get update && sudo apt-get upgrade) + +In any case, the setup is much, much, much more secure than Dropbox. With Dropbox, you have exactly the same setup, but: + +* Your data is stored in some datacenter. It's supposed to be encrypted. It might not be. +* Lot's of people have routine access to your files, and plausible reason to. Bored employees might regularly be doing some 'maintenance work' involving your pictures. +* The dropbox software can do anything it likes on your computer, and it's closed source so you don't know if it does. A disgruntled employee could put a trojan into it. +* Dropbox might have a backdoor for employee access to any file on your computer. This might be done with the best of intentions, but a mal-intentioned or careless employee might still erase things or send sensitive files from your computer by email. +* A truly huge amount of eyes connected to incredibly smart brains have looked at openssh and found it secure. Everybody trusts openssh. With dropbox, there is, well, dropbox. Whoever that is. + +----- + +¹ Me=Carlo, not Joey. I'm pretty sure doing what I wrote here is a good idea, but in case it turns out to be catastrophically dumb, it's my fault, not his. + +² My always-on computer at home is a raspberry pi with a 32GB USB stick. Best self-hosted dropbox you could imagine. + +³ You can just forward the port, but not open a shell, by adding the -N command. This could be useful for connecting on startup, e.g. in /etc/rc.local. I prefer to open the shell to forward the ports, maybe use it, and close it to stop it. diff --git a/doc/tips/Decentralized_repository_behind_a_Firewall/comment_1_78b9035234a690ca5a7c9f3cc78fa092._comment b/doc/tips/Decentralized_repository_behind_a_Firewall/comment_1_78b9035234a690ca5a7c9f3cc78fa092._comment new file mode 100644 index 000000000..71a1db9c8 --- /dev/null +++ b/doc/tips/Decentralized_repository_behind_a_Firewall/comment_1_78b9035234a690ca5a7c9f3cc78fa092._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.49" + subject="comment 1" + date="2012-11-30T16:25:58Z" + content=""" +If you don't trust your home computer with shell access, you can lock it down in `.ssh/authorized_keys` to only be able to run git-annex-shell. See [[forum/Restricting_git-annex-shell_to_a_specific_repository]] +"""]] diff --git a/doc/tips/Delay_Assistant_Startup_on_Login.mdwn b/doc/tips/Delay_Assistant_Startup_on_Login.mdwn new file mode 100644 index 000000000..74652308a --- /dev/null +++ b/doc/tips/Delay_Assistant_Startup_on_Login.mdwn @@ -0,0 +1,13 @@ +# Problem +I noticed that after installing git-annex assistant, my start up times greatly increased because the assistant does a startup scan while everything else is loading. +# Solution (for people using Gnome) +The solution I came up with is to delay the assistant's startup, as well as setting its IO priority as idle. To do this in Gnome 3, run: + + gnome-session-properties +Find the "Git Annex Assistant" entry in the Startup Programs tab, then click edit. Change this: + + /usr/local/bin/git-annex assistant --autostart (your location of git-annex may be different) +to this: + + bash -c "sleep 30; ionice -c3 /usr/local/bin/git-annex assistant --autostart" (replace /usr/local/bin to wherever git-annex is installed) +The "sleep 30" command delays the startup of the assistant by 30 seconds, and "ionice -c3" sets git-annex's IO priority to "idle," the lowest level. diff --git a/doc/tips/Delay_Assistant_Startup_on_Login/comment_1_c63917150527efab4b1106183b3aa7ef._comment b/doc/tips/Delay_Assistant_Startup_on_Login/comment_1_c63917150527efab4b1106183b3aa7ef._comment new file mode 100644 index 000000000..fe8cb80ba --- /dev/null +++ b/doc/tips/Delay_Assistant_Startup_on_Login/comment_1_c63917150527efab4b1106183b3aa7ef._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://launchpad.net/~alphapapa" + nickname="alphapapa" + subject="ionice not supported by deadline scheduler" + date="2013-06-28T17:43:47Z" + content=""" +Linux's deadline I/O scheduler does not support ionice. It is now the default on some distros, including Ubuntu. CFQ does support ionice. +"""]] diff --git a/doc/tips/Git_annex_and_Calibre.mdwn b/doc/tips/Git_annex_and_Calibre.mdwn new file mode 100644 index 000000000..71f955656 --- /dev/null +++ b/doc/tips/Git_annex_and_Calibre.mdwn @@ -0,0 +1,120 @@ +The problem +=========== + +[Calibre](http://calibre-ebook.com/) is a ebook manager that is +available in [debian](http://packages.debian.org/sid/calibre). I use +it to maintain my library, but also to dowload every day an epub +version of a French newspaper and then put it on my kobo. + +Configuring git annex for this +============================== + +I wanted to use git-annex, so + + $ git init + $ git annex init "some useful name" + +But I don't want every thing in annex, because Calibre use some text +file to save some metadata, so I used: + + $ git config annex.largefiles "include=* exclude=*.opf exclude=*.json" + +then lets add everything + + $ git annex add * + $ git add * + $ git commit -m "first commit" + +Calibre need read and write access on the its database, so let unlock it: + + $ git annex unlock metadata.db + +On my other computer I only need to do + + $ git clone $user@$host:Calibre\ library + $ cd Calibre\ library + $ git annex init "another useful name" + $ git annex get . + $ git annex unlock metadata.db + +The problem is that every time you will `git annex sync`, git annex +will lock again the metadata.db, so lets unlock it automatically. I +use git hooks, in `.git/hooks/post-commit` I have + + #!/bin/bash + + git annex edit metadata.db + +don't forget to make this file executable + + $ chmod a+x .git/hooks/post-commit + +Day to day operation +==================== + + $ git annex add . + +Will put new file into the annex + + $ git add . + +Will take care of the files that should no go into annex + + $ git annex sync + +Will make the repositories exchange informations about all this, and +make remote change local + + $ git annex get . + +Will make remote book locally available + +Merge conflict +-------------- +You should not run calibre on the two computer simultaneously, or +without syncing before it. If you do, you will have a conflict that +git-annex will automatically *solve* by rename both of the file. + +You can then either: + + - Choose one. If no books have been changed or added on one of the + computer, to use the other `metadata.db` will not make you loose + any information + - rebuild it. `calibredb restore_database` won't do it, but will tell + you how to do it. + +Checking the library +-------------------- +You can use `calibredb check_library` to check you library is +correct. If you use git for it, it will always tell you that it is not +correct: there is this author ".git" it doesn't know about. Just don't +care about it. + +Maybe this can be solved by using `vcsh` but apparently +`vcsh`+`git annex` it not well tested yet. + +Automatic stuff +--------------- +I use `mr` to automatically run all this, but some config could be +done (I believe) to have `git annex copy --auto` do what it should. + +There are also the git annex assistant for this kind of automatic +synchronizations of contents, but I don't know if my automatic +unlocking of one file will break this. + +It might be interesting to find someway to unlock and lock the library +only when running calibre, a simple script to launch calibre will do +that. Note that each time you will lock and unlock, you will have a +new commit in git. + +Another solution +=================== +You could also use direct mode in place of the auto unlock feature + + git annex direct + +The remove the `post-commit` git hook (or do not add it). Its a +simpler solution, but remember that interaction between git annex direct +repositories and plain git are complex and sometimes downright dangerous. See [[direct mode]] for details. + +In particular, do *not* called `git add *` in the above steps, as that will commit all books into git. diff --git a/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo.mdwn b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo.mdwn new file mode 100644 index 000000000..97f5828d3 --- /dev/null +++ b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo.mdwn @@ -0,0 +1,19 @@ +I worked out how to retroactively annex a large file that had been checked into a git repo some time ago. I thought this might be useful for others, so I am posting it here. + +Suppose you have a git repo where somebody had checked in a large file you would like to have annexed, but there are a bunch of commits after it and you don't want to loose history, but you also don't want everybody to have to retrieve the large file when they clone the repo. This will re-write history as if the file had been annexed when it was originally added. + +This command works for me, it relies on the current behavior of git which is to use a directory named .git-rewrite/t/ at the top of the git tree for the extracted tree. This will not be fast and it will rewrite history, so be sure that everybody who has a copy of your repo is OK with accepting the new history. If the behavior of git changes, you can specify the directory to use with the -d option. Currently, the t/ directory is created inside the directory you specify, so "-d ./.git-rewrite/" should be roughly equivalent to the default. + +Enough with the explanation, on to the command: +<pre> +git filter-branch --tree-filter 'for FILE in file1 file2 file3;do if [ -f "$FILE" ] && [ ! -L "$FILE" ];then git rm --cached "$FILE";git annex add "$FILE";ln -sf `readlink "$FILE"|sed -e "s:^../../::"` "$FILE";fi;done' --tag-name-filter cat -- --all +</pre> + +replace file1 file2 file3... with whatever paths you want retroactively annexed. If you wanted bigfile1.bin in the top dir and subdir1/bigfile2.bin to be retroactively annexed try: +<pre> +git filter-branch --tree-filter 'for FILE in bigfile1.bin subdir1/bigfile2.bin;do if [ -f "$FILE" ] && [ ! -L "$FILE" ];then git rm --cached "$FILE";git annex add "$FILE";ln -sf `readlink "$FILE"|sed -e "s:^../../::"` "$FILE";fi;done' --tag-name-filter cat -- --all +</pre> + +**If your repo has tags** then you should take a look at the git-filter-branch man page about the --tag-name-filter option and decide what you want to do. By default this will re-write the tags "nearly properly". + +You'll probably also want to look at the git-filter-branch man page's section titled "CHECKLIST FOR SHRINKING A REPOSITORY" if you want to free up the space in the existing repo that you just changed history on. diff --git a/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_1_7eaf73fb3355bd706ab18a43790b3c10._comment b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_1_7eaf73fb3355bd706ab18a43790b3c10._comment new file mode 100644 index 000000000..d4e34e8cd --- /dev/null +++ b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_1_7eaf73fb3355bd706ab18a43790b3c10._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://edheil.wordpress.com/" + ip="173.162.44.162" + subject="comment 1" + date="2012-12-16T00:11:38Z" + content=""" +Man, I wish you'd written this a couple weeks ago. :) I was never able to figure that incantation out and ended up unannexing and re-annexing the whole thing to get rid of the file I inadvertently checked into git instead of the annex. +"""]] diff --git a/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_2_dac1a171204f30d7c906e878eb6bd461._comment b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_2_dac1a171204f30d7c906e878eb6bd461._comment new file mode 100644 index 000000000..a3ea62385 --- /dev/null +++ b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_2_dac1a171204f30d7c906e878eb6bd461._comment @@ -0,0 +1,45 @@ +[[!comment format=mdwn + username="https://launchpad.net/~arand" + nickname="arand" + subject="comment 2" + date="2013-03-13T12:05:49Z" + content=""" +Based on the hints given here I've worked on a filter to both annex and add urls via filter-branch: + +[https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-filter](https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-filter) + +The script above is very specific but I think there are a few ideas that can be used in general, the general structure is + + #!/bin/bash + + # links that already exist + links=$(mktemp) + find . -type l >\"$links\" + + # remove from staging area first to not block and then annex + git rm --cached --ignore-unmatch -r bin* + git annex add -c annex.alwayscommit=false bin* + + # compare links before and after annexing, remove links that existed before + newlinks=$(mktemp -u) + mkfifo \"$newlinks\" + comm -13 <(sort \"$links\") <(find . -type l | sort) > \"$newlinks\" & + + # rewrite links + while IFS= read -r file + do + # link is created below .git-rewrite/t/ during filter-branch, strip two parents for correct target + ln -sf \"$(readlink \"$file\" | sed -e 's%^\.\./\.\./%%')\" \"$file\" + done < \"$newlinks\" + + git annex merge + +which would be run using + + git filter-branch --tree-filter path/annex-filter --tag-filter cat -- --all + +or similar. + +* I'm using `find` to make sure the only rewritten symlinks are for the newly annexed files, this way it is possible to annex an unknown set of filenames +* If doing several git annex commands using `-c annex.alwayscommit=false` and doing a `git annex merge` at the end instead might be faster. +"""]] diff --git a/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_3_b62ec0b848d2487d68d7032682622193._comment b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_3_b62ec0b848d2487d68d7032682622193._comment new file mode 100644 index 000000000..9b8aa58f8 --- /dev/null +++ b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_3_b62ec0b848d2487d68d7032682622193._comment @@ -0,0 +1,36 @@ +[[!comment format=mdwn + username="arand" + ip="130.238.245.202" + subject="comment 3" + date="2013-03-18T14:39:52Z" + content=""" +One thing I noticed is that git-annex needs to checksum each file even if they were previously annexed (rather obviously since there is no general way to tell if the file is the same as the old one without checksumming), but in the specific case that we are replacing files that are already in git, we do actually have the sha1 checksum for each file in question, which could be used. + +So, trying to work with this, I wrote a filter script that starts out annexing stuff in the first commit, and continously writes out sha1<->filename<->git-annex-object triplets to a global file, when it then starts with the next commit, it compares the sha1s of the index with those of the global file, and any matches are manually symlinked directly to the corresponding git-annex-object without checksumming. + +I've done a few tests and this seems to be considerably faster than letting git-annex checksum everything. + +This is from a git-svn import of the (free software) Red Eclipse game project, there are approximately 3500 files (images, maps, models, etc.) being annexed in each commit (and around 5300 commits, hence why I really, really care about speed): + +10 commits: ~7min + +100 commits: ~38min + +For comparison, the old and new method (the difference should increase with the amount of commits): + +old, 20 commits ~32min + +new, 20 commits: ~11min + +The script itself is a bit of a monstrosity in bash(/grep/sed/awk/git), and the files that are annexed are hardcoded (removed in forming $oldindexfiles), but should be fairly easy to adapt: + +[https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-ffilter](https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-ffilter) + +The usage would be something like: + + rm /tmp/annex-ffilter.log; git filter-branch --tree-filter 'ANNEX_FFILTER_LOG=/tmp/annex-ffilter.log ~/utv/scripts/annex-ffilter' --tag-name-filter cat -- branchname + +I suggest you use it with at least two orders of magnitude more caution than normal filter-branch. + +Hope it might be useful for someone else wrestling with filter-branch and git-annex :) +"""]] diff --git a/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_4_2423904e41a86cd1c6bc155d7b733642._comment b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_4_2423904e41a86cd1c6bc155d7b733642._comment new file mode 100644 index 000000000..ab1d4e006 --- /dev/null +++ b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_4_2423904e41a86cd1c6bc155d7b733642._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawknOATcOkmzX4jKuET5Z2RsaFUNnLKnQsU" + nickname="Stephen" + subject="comment 4" + date="2013-06-22T07:43:09Z" + content=""" +Thanks for the tip :) One question though: how do I push this new history out throughout my other Annexes? +All I managed to make it do was revert the rewrite so the raw file appeared again... +"""]] diff --git a/doc/tips/Internet_Archive_via_S3.mdwn b/doc/tips/Internet_Archive_via_S3.mdwn new file mode 100644 index 000000000..eba28961d --- /dev/null +++ b/doc/tips/Internet_Archive_via_S3.mdwn @@ -0,0 +1,58 @@ +[The Internet Archive](http://www.archive.org/) allows members to upload +collections using an Amazon S3 +[compatible API](http://www.archive.org/help/abouts3.txt), and this can +be used with git-annex's [[special_remotes/S3]] support. + +So, you can locally archive things with git-annex, define remotes that +correspond to "items" at the Internet Archive, and use git-annex to upload +your files to there. Of course, your use of the Internet Archive must +comply with their [terms of service](http://www.archive.org/about/terms.php). + +A nice added feature is that whenever git-annex sends a file to the +Internet Archive, it records its url, the same as if you'd run `git annex +addurl`. So any users who can clone your repository can download the files +from archive.org, without needing any login or password info. This makes +the Internet Archive a nice way to publish the large files associated with +a public git repository. + +---- + +Sign up for an account, and get your access keys here: +<http://www.archive.org/account/s3.php> + + # export AWS_ACCESS_KEY_ID=blahblah + # export AWS_SECRET_ACCESS_KEY=xxxxxxx + +Specify `host=s3.us.archive.org` when doing `initremote` to set up +a remote at the Archive. This will enable a special Internet Archive mode: +Encryption is not allowed; you are required to specify a bucket name +rather than having git-annex pick a random one; and you can optionally +specify `x-archive-meta*` headers to add metadata as explained in their +[documentation](http://www.archive.org/help/abouts3.txt). + + # git annex initremote archive-panama type=S3 \ + host=s3.us.archive.org bucket=panama-canal-lock-blueprints \ + x-archive-meta-mediatype=texts x-archive-meta-language=eng \ + x-archive-meta-title="original Panama Canal lock design blueprints" + initremote archive-panama (Internet Archive mode) ok + # git annex describe archive-panama "a man, a plan, a canal: panama" + describe archive-panama ok + +Then you can annex files and copy them to the remote as usual: + + # git annex add photo1.jpeg --backend=SHA1E + add photo1.jpeg (checksum...) ok + # git annex copy photo1.jpeg --fast --to archive-panama + copy (to archive-panama...) ok + +Once a file has been stored on archive.org, it cannot be (easily) removed +from it. Also, git-annex whereis will tell you a public url for the file +on archive.org. (It may take a while for archive.org to make the file +publically visibile.) + +Note the use of the SHA1E [[backend|backends]] when adding files. That is +the default backend used by git-annex, but even if you don't normally use +it, it makes most sense to use the WORM or SHA1E backend for files that +will be stored in the Internet Archive, since the key name will be exposed +as the filename there, and since the Archive does special processing of +files based on their extension. diff --git a/doc/tips/Internet_Archive_via_S3/comment_1_d53a3848c20dce61867283fc03c2adaa._comment b/doc/tips/Internet_Archive_via_S3/comment_1_d53a3848c20dce61867283fc03c2adaa._comment new file mode 100644 index 000000000..7c2ce48fc --- /dev/null +++ b/doc/tips/Internet_Archive_via_S3/comment_1_d53a3848c20dce61867283fc03c2adaa._comment @@ -0,0 +1,34 @@ +[[!comment format=mdwn + username="https://id.koumbit.net/anarcat" + ip="72.0.72.144" + subject="how to use with simply addurl?" + date="2013-10-09T22:27:27Z" + content=""" +It doesn't seem like git annex addurl by itself supports the archive.org urls... + +[[!format txt \"\"\" +anarcat@marcos:presentations$ git annex addurl --file=re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm http://archive.org/download/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm +addurl re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm + failed to verify url exists: http://archive.org/download/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm +failed +git-annex: addurl: 1 failed +\"\"\"]] + +I also tried the \"details\" url (<http://archive.org/details/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia>) - but that just downloads the webpage, not the video either... + +Even the ultimate video URL doesn't work: + +[[!format txt \"\"\" +anarcat@marcos:presentations$ git annex addurl --debug --file=re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm http://ia601009.us.archive.org/9/items/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm +[2013-10-09 18:26:30 EDT] call: quvi [\"-v\",\"mute\",\"--support\",\"http://ia601009.us.archive.org/9/items/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm\"] +addurl re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm [2013-10-09 18:26:30 EDT] read: curl [\"-s\",\"--head\",\"-L\",\"http://ia601009.us.archive.org/9/items/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm\",\"-w\",\"%{http_code}\"] + + failed to verify url exists: http://ia601009.us.archive.org/9/items/Republica2012-EbenMoglen-FreedomOfThoughtRequiresFreeMedia/re_publica_2012___Eben_Moglen___Freedom_of_Thought_Requires_Free_Media.webm +failed +git-annex: addurl: 1 failed +\"\"\"]] + +... even though that URL actually gives out a proper 200 OK response code. + +Any ideas? --[[anarcat]] +"""]] diff --git a/doc/tips/Internet_Archive_via_S3/comment_2_91c1472da27b00e5d682d22bc1ef04e0._comment b/doc/tips/Internet_Archive_via_S3/comment_2_91c1472da27b00e5d682d22bc1ef04e0._comment new file mode 100644 index 000000000..9b4ac58b2 --- /dev/null +++ b/doc/tips/Internet_Archive_via_S3/comment_2_91c1472da27b00e5d682d22bc1ef04e0._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.4.22" + subject="comment 2" + date="2013-10-11T17:08:27Z" + content=""" +This was a misleading error message. The url you are trying to add to the file does not match the size recorded for the file already in the annex. (Or possibly the file's key has no recorded size). If you really want to add the url to the file despite it being a different encoding, you can use --relaxed, although fsck may not like the result if you ever end up downloading that url.. + +(Please file bug reports for problems in the future, rather than posting comments on only vaguely related pages which as we can see here can turn out to be entirely offtopic.) +"""]] diff --git a/doc/tips/Internet_Archive_via_S3/comment_3_e23cf781c532f80d47d52265f2b2c87e._comment b/doc/tips/Internet_Archive_via_S3/comment_3_e23cf781c532f80d47d52265f2b2c87e._comment new file mode 100644 index 000000000..3745d544c --- /dev/null +++ b/doc/tips/Internet_Archive_via_S3/comment_3_e23cf781c532f80d47d52265f2b2c87e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://id.koumbit.net/anarcat" + ip="72.0.72.144" + subject="still a bug, filed separately!" + date="2013-10-11T18:49:06Z" + content=""" +Aaah, of course, sorry for the noise here. It turns out that this is *not* because the filesize (or even the checksum, for that matter) are different, so there's clearly a bug there, and i filed it in [[bugs/addurl_fails_on_the_internet_archive]]. Thanks! +"""]] diff --git a/doc/tips/The_perfect_preferred_content_settings_for_my_android_phone.mdwn b/doc/tips/The_perfect_preferred_content_settings_for_my_android_phone.mdwn new file mode 100644 index 000000000..1afac15d5 --- /dev/null +++ b/doc/tips/The_perfect_preferred_content_settings_for_my_android_phone.mdwn @@ -0,0 +1,32 @@ +I have an annex that syncs my personal files on all my computers. It works great. Phones are different. + +For one, everything's a bit slower to sync, there's battery considerations, and I just don't need every last old file on my phone. Then there's some files I explicitly don't want on my phone in case it gets lost, like family pictures, passport scans, or private keys. + +But I still want photos, videos and voice recordings I make on my phone to be synced to my server. A transfer repo would work, but I want to keep them. Then there's my PDF book collection; that would certainly be nice to always have around in case I have half on hour on a bus. And my music collection ought to be around as well. + +So I came up with this solution, and I'm very happy with it. + + include=Music/* or include=Books/* or present + +This will sync my music and book collections to my phone whenever I add something new on my computers, and it will sync and keep anything I add to the annex on my phone. Best of all worlds! Impressed how flexible preferred content is. More full-sync folders can be added like this: + + include=Music/* or include=Books/* or Notes/* or present + +To add them, I first had to figure out the uuid of my phone repo. So I added a new tab on android, and did + + cd /sdcard/annex + git config annex.uuid + +Then I went to one of my computers, and did + + git annex vicfg + +And changed the line + + content [phone-uuid] = standard + +to + + content [phone-uuid] = include=Music/* or include=Books/* or Notes/* or present + +And waited for it to sync. diff --git a/doc/tips/The_perfect_preferred_content_settings_for_my_android_phone/comment_1_393d1636bb313530be383a075bd3440a._comment b/doc/tips/The_perfect_preferred_content_settings_for_my_android_phone/comment_1_393d1636bb313530be383a075bd3440a._comment new file mode 100644 index 000000000..8938684f8 --- /dev/null +++ b/doc/tips/The_perfect_preferred_content_settings_for_my_android_phone/comment_1_393d1636bb313530be383a075bd3440a._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.246" + subject="comment 1" + date="2013-11-16T17:29:03Z" + content=""" +That's great, that's how I hoped people would be able to use preferred content settings. + +I'd suggest adding support for archive directories to this. So if you create a file on the phone and are done with it, you can move it to an archive directory, and it will then be dropped from the phone once it reaches an archive repository. + +This should accomplish that. (Untested) + +`((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1))) and (include=Music/* or include=Books/* or present)` +"""]] diff --git a/doc/tips/Using_Git-annex_as_a_web_browsing_assistant.mdwn b/doc/tips/Using_Git-annex_as_a_web_browsing_assistant.mdwn new file mode 100644 index 000000000..4ee023de3 --- /dev/null +++ b/doc/tips/Using_Git-annex_as_a_web_browsing_assistant.mdwn @@ -0,0 +1,46 @@ +[[todo/wishlist: an "assistant" for web-browsing -- tracking the sources of the downloads]] suggests using git-annex as a tool to store downloads tied +to their URLs. This also enables people to have their files stored offline, +while being able to git annex drop them at any time and redownload them +with git annex get. Additionally, a clone of the repo can be used to +download whatever files are desired from online. + +This tip explains how to implement a similar system to the one described in +the linked wishlist with existing software and features of git-annex. + +The first step is to install the Firefox plugin +[FlashGot](http://flashgot.net/). We will use it to provide the Firefox +shortcuts to add things to our annex. + +We also need a normal download manager, if we want to get status updates as +the download is done. We'll need to configure git-annex to use it by +setting `annex.web-download-command` as Joey describes in his comment on +[[todo/wishlist: allow configuration of downloader for addurl]]. See the +manpage [[git-annex]] for more information on setting configuration. + +Once we have installed all that, we need a script that has an interface +which FlashGot can treat as a downloader, but which calls git-annex to do +the actual downloading. Such a script is available from +<https://gist.github.com/andyg0808/5342434>. Download it and store it +somewhere it can live, or cut and paste: + +[[!format sh """ +#!/bin/bash +# $1=folder to cd to (must be a git annex repo) +# $2=URL to download + +cd "$1" +git-annex addurl "$2" +"""]] + +Finally, we need to configure FlashGot to use the script as a downloader. +Go to Tools > Add-ons in Firefox. Click "Preferences" on FlashGot. Click +the Add button next to the list of download managers. Enter a name for the +git-annex downloader. Choose the script that was downloaded from the +"Locate executable file" dialog that appears. Now set the command line +arguments template to be "[FOLDER] [URL]" (you can find more substitution +expressions in the Placeholders dropdown above the Command line arguments +template field). You're done! + +Go ahead and test it by trying to download a file using FlashGot. It should +offer as one of its available download managers the new manager you created +just above. Select it and have fun! diff --git a/doc/tips/Using_Git-annex_as_a_web_browsing_assistant/comment_1_74167f9fff400f148916003468c77de4._comment b/doc/tips/Using_Git-annex_as_a_web_browsing_assistant/comment_1_74167f9fff400f148916003468c77de4._comment new file mode 100644 index 000000000..a091b8e48 --- /dev/null +++ b/doc/tips/Using_Git-annex_as_a_web_browsing_assistant/comment_1_74167f9fff400f148916003468c77de4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 1" + date="2013-04-11T20:16:02Z" + content=""" +As of my last commit, you don't really need a separate download manager. The webapp will now display urls that `git annex addurl` is downloading in amoung the other transfers. +"""]] diff --git a/doc/tips/assume-unstaged.mdwn b/doc/tips/assume-unstaged.mdwn new file mode 100644 index 000000000..63f5f820a --- /dev/null +++ b/doc/tips/assume-unstaged.mdwn @@ -0,0 +1,31 @@ +[[!meta title="using assume-unstages to speed up git with large trees of annexed files"]] + +Git update-index's assume-unstaged feature can be used to speed +up `git status` and stuff by not statting the whole tree looking for changed +files. + +This feature works quite well with git-annex. Especially because git +annex's files are immutable, so aren't going to change out from under it, +this is a nice fit. If you have a very large tree and `git status` is +annoyingly slow, you can turn it on: + + git config core.ignoreStat true + +When `git mv` and `git rm` are used, those changes *do* get noticed, even +on assume-unchanged files. When new files are added, eg by `git annex add`, +they are also noticed. + +There are two gotchas. Both occur because `git add` does not stage +assume-unchanged files. + +1. When an annexed file is moved to a different directory, it updates + the symlink, and runs `git add` on it. So the file will move, + but the changed symlink will not be noticed by git and it will commit a + dangling symlink. +2. When using `git annex migrate`, it changes the symlink and `git adds` + it. Again this won't be committed. + +These can be worked around by running `git update-index --really-refresh` +after performing such operations. I hope that `git add` will be changed +to stage changes to assume-unchanged files, which would remove this +only complication. --[[Joey]] diff --git a/doc/tips/assume-unstaged/comment_1_44abd811ef79a85e557418e17a3927be._comment b/doc/tips/assume-unstaged/comment_1_44abd811ef79a85e557418e17a3927be._comment new file mode 100644 index 000000000..d253feb5b --- /dev/null +++ b/doc/tips/assume-unstaged/comment_1_44abd811ef79a85e557418e17a3927be._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://me.yahoo.com/a/2djv2EYwk43rfJIAQXjYt_vfuOU-#a11a6" + nickname="Olivier R" + subject="It doesn't work 100%" + date="2012-05-03T21:42:54Z" + content=""" +When you remove tracked files... it doesn't show the new status. it's like if the file was ignored. + + +"""]] diff --git a/doc/tips/assume-unstaged/comment_2_5b589f37cfc03bf7be33a51826cc4dba._comment b/doc/tips/assume-unstaged/comment_2_5b589f37cfc03bf7be33a51826cc4dba._comment new file mode 100644 index 000000000..474d7b399 --- /dev/null +++ b/doc/tips/assume-unstaged/comment_2_5b589f37cfc03bf7be33a51826cc4dba._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnxlx1UrzVhdy6_gFjzmF42x6QXxBUxg00" + nickname="Jakukyo" + subject="comment 2" + date="2013-09-05T12:14:42Z" + content=""" +> There are two gotchas... + +So just always run `git annex add` after editing a file +and `git update-index --really-refresh` after migrating +backend? + +"""]] diff --git a/doc/tips/automatically_getting_files_on_checkout.mdwn b/doc/tips/automatically_getting_files_on_checkout.mdwn new file mode 100644 index 000000000..bbb3b302e --- /dev/null +++ b/doc/tips/automatically_getting_files_on_checkout.mdwn @@ -0,0 +1,15 @@ +Normally git-annex does not retrieve file contents when checking out a +tree. In some use cases, it makes sense to always have the contents of +files available after a `git checkout` or `git update`. This can be +accomplished by installing the following as `.git/hooks/post-checkout` + + #!/bin/sh + # Uses git-annex to get all files in the specified directories + # (relative to the top of the repository) on checkout. + dirs=. + top="$(git rev-parse --show-toplevel)" + for dir in "$dirs"; do git annex get $top/$dir"; done + +By default, all files in the whole repository will be made available. The +`dirs` setting can be configured if you only want to get files in certian +directories. diff --git a/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes.mdwn b/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes.mdwn new file mode 100644 index 000000000..0983c7d31 --- /dev/null +++ b/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes.mdwn @@ -0,0 +1,2 @@ +When git annex does fsck on (for example) a GPG-encrypted special directory remote, it first transfers the whole file into .git/annex/tmp directory. +If your annex is on an SSD, it's a good idea to make .git/annex/tmp a symlink to say /var/tmp so SSD isn't worn down. This actually may be a better default. diff --git a/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_1_e7c5c46112a2406b873d08bbf53c40d8._comment b/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_1_e7c5c46112a2406b873d08bbf53c40d8._comment new file mode 100644 index 000000000..9c7bc2ed1 --- /dev/null +++ b/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_1_e7c5c46112a2406b873d08bbf53c40d8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog" + nickname="Michael" + subject="comment 1" + date="2013-07-31T15:15:41Z" + content=""" +Of course, this only works when /var/tmp isn't on SSD itself. Perhaps tmpfs (e.g. a /tmp on many distros) is good -- after checking that there's enough space to transfer a particular file. +"""]] diff --git a/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_2_daf45ce29fed986fa9aa8b173760d0b7._comment b/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_2_daf45ce29fed986fa9aa8b173760d0b7._comment new file mode 100644 index 000000000..929019705 --- /dev/null +++ b/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_2_daf45ce29fed986fa9aa8b173760d0b7._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog" + nickname="Michael" + subject="there's a problem" + date="2013-08-04T17:15:05Z" + content=""" +If .git/annex/tmp is a symlink to another fs, then adding doesn't work: + + add file1.jpg (checksum...) + git-annex: /path/to/.git/annex/tmp/tmp30148: rename: unsupported operation (Invalid cross-device link) + +It looks like it would be good to have two types of tmp directories here, one for adding, another one for verifying (and that one could be redirected off SSD). + +"""]] diff --git a/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_3_72d222020af4a9c6c753eb1ee7e1f1cf._comment b/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_3_72d222020af4a9c6c753eb1ee7e1f1cf._comment new file mode 100644 index 000000000..2624a4fd3 --- /dev/null +++ b/doc/tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/comment_3_72d222020af4a9c6c753eb1ee7e1f1cf._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="guilhem" + ip="46.239.117.180" + subject="comment 3" + date="2013-08-19T01:05:40Z" + content=""" +A nice feature would be to perform the `fsck` on the (encrypted) remote itself, as it would avoid to clutter either the network or the tmpdir. However, that requires some changes in git-annex's backend. Indeed it would no longer be enough to store a single digest per (plain) file: a new digest needs to be stored for each encrypted copy. It is not necessarily a big deal, but the backend would need to be reorganized carefully. +"""]] diff --git a/doc/tips/centralised_repository:_starting_from_nothing.mdwn b/doc/tips/centralised_repository:_starting_from_nothing.mdwn new file mode 100644 index 000000000..b12246d36 --- /dev/null +++ b/doc/tips/centralised_repository:_starting_from_nothing.mdwn @@ -0,0 +1,75 @@ +If you are starting from nothing (no existing `git` or `git-annex` repository) and want to use a server as a centralised repository, try the following steps. + +On the server where you'll hold the "master" repository: + + server$ cd /one/git + server$ mkdir m + server$ cd m + server$ git init --bare + Initialized empty Git repository in /one/git/m/ + server$ git annex init origin + init origin ok + server$ + +Clone that to the laptop: + + laptop$ cd /other + laptop$ git clone ssh://server//one/git/m + Cloning into 'm'... + Warning: No xauth data; using fake authentication data for X11 forwarding. + remote: Counting objects: 5, done. + remote: Compressing objects: 100% (3/3), done. + remote: Total 5 (delta 0), reused 0 (delta 0) + Receiving objects: 100% (5/5), done. + warning: remote HEAD refers to nonexistent ref, unable to checkout. + + laptop$ cd m + laptop$ git annex init laptop + init laptop ok + laptop$ + +Merge the `git-annex` repository (this is the bit that is often +overlooked!): + + laptop$ git annex merge + merge . (merging "origin/git-annex" into git-annex...) + ok + laptop$ + +Add some content: + + laptop$ git annex addurl http://kitenet.net/~joey/screencasts/git-annex_coding_in_haskell.ogg + "kitenet.net_~joey_screencasts_git-annex_coding_in_haskell.ogg" + addurl kitenet.net_~joey_screencasts_git-annex_coding_in_haskell.ogg (downloading http://kitenet.net/~joey/screencasts/git-annex_coding_in_haskell.ogg ...) --2011-12-15 08:13:10-- http://kitenet.net/~joey/screencasts/git-annex_coding_in_haskell.ogg + Resolving kitenet.net (kitenet.net)... 2001:41c8:125:49::10, 80.68.85.49 + Connecting to kitenet.net (kitenet.net)|2001:41c8:125:49::10|:80... connected. + HTTP request sent, awaiting response... 200 OK + Length: 39362757 (38M) [audio/ogg] + Saving to: `/other/m/.git/annex/tmp/URL--http&c%%kitenet.net%~joey%screencasts%git-annex_coding_in_haskell.ogg' + + 100%[======================================>] 39,362,757 2.31M/s in 17s + + 2011-12-15 08:13:27 (2.21 MB/s) - `/other/m/.git/annex/tmp/URL--http&c%%kitenet.net%~joey%screencasts%git-annex_coding_in_haskell.ogg' saved [39362757/39362757] + + (checksum...) ok + (Recording state in git...) + laptop$ git commit -m 'See Joey play.' + [master (root-commit) 106e923] See Joey play. + 1 files changed, 1 insertions(+), 0 deletions(-) + create mode 120000 kitenet.net_~joey_screencasts_git-annex_coding_in_haskell.ogg + laptop$ + +All fine, now push it back to the centralised master: + + laptop$ git push + Counting objects: 20, done. + Delta compression using up to 4 threads. + Compressing objects: 100% (11/11), done. + Writing objects: 100% (18/18), 1.50 KiB, done. + Total 18 (delta 1), reused 1 (delta 0) + To ssh://server//one/git/m + 3ba1386..ad3bc9e git-annex -> git-annex + laptop$ + +You can add more "client" repositories by following the `laptop` +sequence of operations. diff --git a/doc/tips/centralised_repository:_starting_from_nothing/comment_1_b0d22822017646775869ce1292e676f4._comment b/doc/tips/centralised_repository:_starting_from_nothing/comment_1_b0d22822017646775869ce1292e676f4._comment new file mode 100644 index 000000000..22857af3e --- /dev/null +++ b/doc/tips/centralised_repository:_starting_from_nothing/comment_1_b0d22822017646775869ce1292e676f4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-12-23T19:19:53Z" + content=""" +See also: [[centralized_git_repository_tutorial]] +"""]] diff --git a/doc/tips/centralized_git_repository_tutorial.mdwn b/doc/tips/centralized_git_repository_tutorial.mdwn new file mode 100644 index 000000000..00283829f --- /dev/null +++ b/doc/tips/centralized_git_repository_tutorial.mdwn @@ -0,0 +1,140 @@ +The [[walkthrough]] builds up a decentralized git repository setup, but +git-annex can also be used with a centralized bare repository, just like +git can. This tutorial shows how to set up a centralized repository hosted on +GitHub. + +## set up the repository, and make a checkout + +I've created a repository for technical talk videos, which you can +[fork on Github](https://github.com/joeyh/techtalks). +Or make your own repository on GitHub (or elsewhere) now. + +On your laptop, [[install]] git-annex, and clone the repository: + + # git clone git@github.com:joeyh/techtalks.git + # cd techtalks + +Tell git-annex to use the repository, and describe where this clone is +located: + + # git annex init 'my laptop' + init my laptop ok + +Let's tell git-annex that GitHub doesn't support running git-annex-shell there. +This means you can't store annexed file *contents* on GitHub; it would +really be better to host the bare repository on your own server, which +would not have this limitation. (If you want to do that, check out +[[using_gitolite_with_git-annex]].) + + # git config remote.origin.annex-ignore true + +## add files to the repository + +Add some files, obtained however. + + # youtube-dl -t 'http://www.youtube.com/watch?v=b9FagOVqxmI' + # git annex add *.mp4 + add Haskell_Amuse_Bouche-b9FagOVqxmI.mp4 (checksum) ok + (Recording state in git...) + # git commit -m "added a video. I have not watched it yet but it sounds interesting" + +This file is available directly from the web; so git-annex can download it: + + # git annex addurl http://kitenet.net/~joey/screencasts/git-annex_coding_in_haskell.ogg + addurl kitenet.net_~joey_screencasts_git-annex_coding_in_haskell.ogg + (downloading http://kitenet.net/~joey/screencasts/git-annex_coding_in_haskell.ogg ...) + (checksum...) ok + (Recording state in git...) + # git commit -a -m 'added a screencast I made' + +Feel free the rename the files, etc, using normal git commands: + + # git mv Haskell_Amuse_Bouche-b9FagOVqxmI.mp4 Haskell_Amuse_Bouche.mp4 + # git mv kitenet.net_~joey_screencasts_git-annex_coding_in_haskell.ogg git-annex_coding_in_haskell.ogg + # git commit -m 'better filenames' + +Now push your changes back to the central repository. This first time, +remember to push the git-annex branch, which is used to track the file +contents. + + # git push origin master git-annex + To git@github.com:joeyh/techtalks.git + * [new branch] master -> master + * [new branch] git-annex -> git-annex + +That push went fast, because it didn't upload large videos to GitHub. +To check this, you can ask git-annex where the contents of the videos are: + + # git annex whereis + whereis Haskell_Amuse_Bouche.mp4 (1 copy) + 767e8558-0955-11e1-be83-cbbeaab7fff8 -- here + ok + whereis git-annex_coding_in_haskell.ogg (2 copies) + 00000000-0000-0000-0000-000000000001 -- web + 767e8558-0955-11e1-be83-cbbeaab7fff8 -- here + ok + +## make more checkouts + +So far you have a central repository, and a checkout on a laptop. +Let's make another checkout that's used as a backup. You can put it anywhere +you like, just make it be somewhere your laptop can access. A few options: + +* Put it on a USB drive that you can plug into the laptop. +* Put it on a desktop. +* Put it on some server in the local network. +* Put it on a remote VPS. + +I'll use the VPS option, but these instructions should work for +any of the above. + + # ssh server + server# sudo apt-get install git-annex + +Clone the central repository as before. (If the clone fails, you need +to add your server's ssh public key to github -- see +[this page](http://help.github.com/ssh-issues/).) + + server# git clone git@github.com:joeyh/techtalks.git + server# cd techtalks + server# git config remote.origin.annex-ignore true + server# git annex init 'backup' + init backup (merging origin/git-annex into git-annex...) ok + +Notice that the server does not have the contents of any of the files yet. +If you run `ls`, you'll see broken symlinks. We want to populate this +backup with the file contents, by copying them from your laptop. + +Back on your laptop, you need to configure a git remote for the backup. +Adjust the ssh url as needed to point to wherever the backup is. (If it +was on a local USB drive, you'd use the path to the repository instead.) + + # git remote add backup ssh://server/~/techtalks + +Now git-annex on your laptop knows how to reach the backup repository, +and can do things like copy files to it: + + # git annex copy --to backup git-annex_coding_in_haskell.ogg + copy git-annex_coding_in_haskell.ogg (checking backup...) + 12877824 2% 255.11kB/s 00:00 + ok + +You can also `git annex move` files to it, to free up space on your laptop. +And then you can `git annex get` files back to your laptop later on, as +desired. + +After you use git-annex to move files around, remember to push, +which will broadcast its updated location information. + + # git push + +## take it farther + +Of course you can create as many checkouts as you desire. If you have a +desktop machine too, you can make a checkout there, and use `git remote +add` to also let your desktop access the backup repository. + +You can add remotes for each direct connection between machines you find you +need -- so make the laptop have the desktop as a remote, and the desktop +have the laptop as a remote, and then on either machine git-annex can +access files stored on the other. diff --git a/doc/tips/centralized_git_repository_tutorial/comment_1_9072ebc0c61446d7b151fcfab616fea9._comment b/doc/tips/centralized_git_repository_tutorial/comment_1_9072ebc0c61446d7b151fcfab616fea9._comment new file mode 100644 index 000000000..c509d7579 --- /dev/null +++ b/doc/tips/centralized_git_repository_tutorial/comment_1_9072ebc0c61446d7b151fcfab616fea9._comment @@ -0,0 +1,33 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkC0W3ZQERUaTkHoks6k68Tsp1tz510nGo" + nickname="Georg" + subject="sync, push, pull with/to/from centralized bare repository" + date="2013-10-07T06:45:19Z" + content=""" +Hi Joey, + +thanks for tutorial with the centralized repo. I am currently trying to set up a central bare repo for two clients (they cannot communicate directly with each other). I am not sure if I am pushing/pulling the right way. + +On the server I did: + + git init --bare + git annex init origin + +On Cĺient Alice (I want to give Bob a chance get call \"git annex get\" from \"origin\"): + + git clone ssh://tktest@192.168.56.104/~/annex . + git annex init Alice + git annex merge + git annex add . + git commit -a -m \"Added tutorial\" + git push origin master git-annex + git annex copy . --to origin + +On Client Bob I have called \"clone, init, merge, add, push, copy\" also. + +Now the tricky part - do I have to call \"git annex sync\" at Alice's side to get the updates from Bob over origin? +I ran into troubles if I called \"copy --to origin\" before \"git push origin master git-annex\". How can I resolve a non-fast-forware on the git-annex branch? +Some notes about how to sync over a central bare repo would be nice here =) + +Thanks a lot, Georg +"""]] diff --git a/doc/tips/centralized_git_repository_tutorial/comment_2_528e92b21f0551fde4adb956654953ae._comment b/doc/tips/centralized_git_repository_tutorial/comment_2_528e92b21f0551fde4adb956654953ae._comment new file mode 100644 index 000000000..21b286fff --- /dev/null +++ b/doc/tips/centralized_git_repository_tutorial/comment_2_528e92b21f0551fde4adb956654953ae._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.253.80" + subject="How can I resolve a non-fast-forware on the git-annex branch?" + date="2013-10-07T17:08:32Z" + content=""" +By either running `git annex sync`, or if you want to pull and push yourself, by running `git annex merge` before pushing. +"""]] diff --git a/doc/tips/downloading_podcasts.mdwn b/doc/tips/downloading_podcasts.mdwn new file mode 100644 index 000000000..2e0ec0e30 --- /dev/null +++ b/doc/tips/downloading_podcasts.mdwn @@ -0,0 +1,63 @@ +You can use git-annex as a podcatcher, to download podcast contents. +No additional software is required, but your git-annex must be built +with the Feeds feature (run `git annex version` to check). + +All you need to do is put something like this in a cron job: + +`cd somerepo && git annex importfeed http://url/to/podcast http://other/podcast/url` + +This downloads the urls, and parses them as RSS, Atom, or RDF feeds. +All enclosures are downloaded and added to the repository, the same as if you +had manually run `git annex addurl` on each of them. + +git-annex will avoid downloading a file from a feed if its url has already +been stored in the repository before. So once a file is downloaded, +you can move it around, delete it, `git annex drop` its content, etc, +and it will not be downloaded again by repeated runs of +`git annex importfeed`. Just how a podcatcher should behave. + +## templates + +To control the filenames used for items downloaded from a feed, +there's a --template option. The default is +`--template='${feedtitle}/${itemtitle}${extension}'` + +Other available template variables: +feedauthor, itemauthor, itemsummary, itemdescription, itemrights, itemid + +## catching up + +To catch up on a feed without downloading its contents, +use `git annex importfeed --relaxed`, and delete the symlinks it creates. +Next time you run `git annex addurl` it will only fetch any new items. + +## fast mode + +To add a feed without downloading its contents right now, +use `git annex importfeed --fast`. Then you can use `git annex get` as +usual to download the content of an item. + +## storing the podcast list in git + +You can check the list of podcast urls into git right next to the +files it downloads. Just make a file named feeds and add one podcast url +per line. + +Then you can run git-annex on all the feeds: + +`xargs git-annex importfeed < feeds` + +## distributed podcatching + +A nice benefit of using git-annex as a podcatcher is that you can +run `git annex importfeed` on the same url in different clones +of a repository, and `git annex sync` will sync it all up. + +## centralized podcatching + +You can also have a designated machine which always fetches all podcstas +to local disk and stores them. That way, you can archive podcasts with +time-delayed deletion of upstream content. You can also work around slow +downloads upstream by podcatching to a server with ample bandwidth or work +around a slow local Internet connection by podcatching to your home server +and transferring to your laptop on demand. diff --git a/doc/tips/downloading_podcasts/comment_10_4d4f6c22070b58918ee8d34c5e7290ad._comment b/doc/tips/downloading_podcasts/comment_10_4d4f6c22070b58918ee8d34c5e7290ad._comment new file mode 100644 index 000000000..3bf5afe68 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_10_4d4f6c22070b58918ee8d34c5e7290ad._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 10" + date="2013-08-05T16:47:30Z" + content=""" +`cabal install feed` should get the necessary library installed so that git-annex will build with feeds support. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_11_d8d77048c7e2524968c188e1ad517873._comment b/doc/tips/downloading_podcasts/comment_11_d8d77048c7e2524968c188e1ad517873._comment new file mode 100644 index 000000000..fd3459926 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_11_d8d77048c7e2524968c188e1ad517873._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="http://a-or-b.myopenid.com/" + ip="220.244.41.108" + subject="comment 11" + date="2013-08-06T04:20:16Z" + content=""" + $ cabal install feed + Resolving dependencies... + All the requested packages are already installed: + feed-0.3.9.1 + Use --reinstall if you want to reinstall anyway. + +Then I reinstalled `git-annex` but it still doesn't find the feeds flag. + + $ git annex version + git-annex version: 4.20130802 + build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS + +Do I need to do something like: + + cabal install git-annex --bindir=$HOME/bin -f\"-assistant -webapp -webdav -pairing -xmpp -dns -feed\" + +...but what are the default flags to include in addition to `-feed` +"""]] diff --git a/doc/tips/downloading_podcasts/comment_12_0859317471b43c88744dd3df95c879f7._comment b/doc/tips/downloading_podcasts/comment_12_0859317471b43c88744dd3df95c879f7._comment new file mode 100644 index 000000000..e75a44a8c --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_12_0859317471b43c88744dd3df95c879f7._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 12" + date="2013-08-06T04:24:10Z" + content=""" +-f-Feed will disable the feature. -fFeed will try to force it on. + +You can probably work out what's going wrong using cabal install -v3 +"""]] diff --git a/doc/tips/downloading_podcasts/comment_13_e8c3c97282d17e2a1d47fb9d5e2b2f7b._comment b/doc/tips/downloading_podcasts/comment_13_e8c3c97282d17e2a1d47fb9d5e2b2f7b._comment new file mode 100644 index 000000000..8d1242818 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_13_e8c3c97282d17e2a1d47fb9d5e2b2f7b._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="http://a-or-b.myopenid.com/" + ip="220.244.41.108" + subject="comment 13" + date="2013-08-06T05:42:45Z" + content=""" +So I ran `cabal install -v3` and looked at the output, + + Flags chosen: feed=True, tdfa=True, testsuite=True, android=False, + production=True, dns=True, xmpp=True, pairing=True, webapp=True, + assistant=True, dbus=True, inotify=True, webdav=True, s3=True + +This looks like feed should be on. + +There doesn't appear to be any errors in the compile either. + +Is it as simple as a bug where this flag just doesn't show in the `git annex version` command? +"""]] diff --git a/doc/tips/downloading_podcasts/comment_14_05a3694052de36848fbbad6eeeada895._comment b/doc/tips/downloading_podcasts/comment_14_05a3694052de36848fbbad6eeeada895._comment new file mode 100644 index 000000000..4bc831f7f --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_14_05a3694052de36848fbbad6eeeada895._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 14" + date="2013-08-07T16:03:12Z" + content=""" +Yes, it did turn out to be as simple as my having forgotten that I have to manually add features to the version list. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_15_21028bed8858c2dae1ac9c2d014fd2a1._comment b/doc/tips/downloading_podcasts/comment_15_21028bed8858c2dae1ac9c2d014fd2a1._comment new file mode 100644 index 000000000..0f998d066 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_15_21028bed8858c2dae1ac9c2d014fd2a1._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://23.gs/" + ip="46.165.197.5" + subject="No file extension?" + date="2013-08-12T13:21:50Z" + content=""" +It seems git-annex is a bit overzealous when sanitizing the file extension, currently I get: \"Nerdkunde/Let_s_go_to_the_D_M_C_A_m4a\" from http://www.nerdkunde.de/episodes.m4a.rss with the default template and only \"Nerdkunde/Let_s_go_to_the_D_M_C_A._m4a\" if I add the \".\" in the template myself... +"""]] diff --git a/doc/tips/downloading_podcasts/comment_16_4869fb5c9f896acc477c44de06c36ca7._comment b/doc/tips/downloading_podcasts/comment_16_4869fb5c9f896acc477c44de06c36ca7._comment new file mode 100644 index 000000000..4419d02a8 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_16_4869fb5c9f896acc477c44de06c36ca7._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="arand" + ip="130.243.226.21" + subject="comment 16" + date="2013-08-12T13:32:46Z" + content=""" +The filename extension is a known issue and already fixed in the development version, see <http://git-annex.branchable.com/bugs/importfeed_uses___34____95__foo__34___as_extension/> +"""]] diff --git a/doc/tips/downloading_podcasts/comment_17_2e278ff200c1c15efd27c46a3e0aed40._comment b/doc/tips/downloading_podcasts/comment_17_2e278ff200c1c15efd27c46a3e0aed40._comment new file mode 100644 index 000000000..bc49e5dd0 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_17_2e278ff200c1c15efd27c46a3e0aed40._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlpKmTa1OPwy5Jk24pOoD8Vlo2jahzTPnw" + nickname="Stephen" + subject="rss authentication" + date="2013-08-13T13:32:52Z" + content=""" +If a podcast requires authentication, is there a way to pass credentials through? I tried `http://user:pass@site.com/rss.xml` but it didn't work. + +"""]] diff --git a/doc/tips/downloading_podcasts/comment_18_382f2b970738d9b1af577955c3083e90._comment b/doc/tips/downloading_podcasts/comment_18_382f2b970738d9b1af577955c3083e90._comment new file mode 100644 index 000000000..9e3244315 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_18_382f2b970738d9b1af577955c3083e90._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="http://www.joachim-breitner.de/" + nickname="nomeata" + subject="--fast and --relaxed" + date="2013-08-16T07:27:59Z" + content=""" +Hi, + +the explanations to --fast and --relaxed on this page could be extended a bit. I looked it up in the man page, but it is not yet clear to me when I would use one or the other with feeds. Also, does “Next time you run git annex addurl it will only fetch any new items.” really only apply to --relaxed, and not --fast? + +Furthermore, it would be good if there were a template variable `itemnum` that I can use to ensure that `ls` prints the casts in the right order, even when the titles of the items are not helpful. + +Greetings, +Joachim +"""]] diff --git a/doc/tips/downloading_podcasts/comment_19_f76fc6835e5787b0156380bf09fd81ca._comment b/doc/tips/downloading_podcasts/comment_19_f76fc6835e5787b0156380bf09fd81ca._comment new file mode 100644 index 000000000..41313a87d --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_19_f76fc6835e5787b0156380bf09fd81ca._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 19" + date="2013-08-22T15:25:02Z" + content=""" +I would expect user:pass@site.com to work if the site is using http basic auth. `importfeed` just runs `wget` (or `curl`) to do all downloads, and wget's documentation says that works. It also says you can use ~/.netrc to store the password for a site. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_1_f04bc32a34baeeffcd691e9f7cce0230._comment b/doc/tips/downloading_podcasts/comment_1_f04bc32a34baeeffcd691e9f7cce0230._comment new file mode 100644 index 000000000..014fe3f50 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_1_f04bc32a34baeeffcd691e9f7cce0230._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="ckeen" + ip="79.249.110.228" + subject="Filename too long" + date="2013-07-30T14:39:44Z" + content=""" +It seems that some of my feeds get stored into keys that generate a too long filename: + + podcasts/.git/annex/tmp/b1f_325_URL-s143660317--http&c%%feedproxy.google.com%~r%mixotic%~5%urTIRWQK2OQ%Mixotic__258__-__Michael__Miller__-__Galactic__Technolgies.mp3.log.web: + openBinaryFile: invalid argument (File name too long) + +Is there a way to work around this? +"""]] diff --git a/doc/tips/downloading_podcasts/comment_20_65ebf3a3bbf0a2aebd2b69640b757e16._comment b/doc/tips/downloading_podcasts/comment_20_65ebf3a3bbf0a2aebd2b69640b757e16._comment new file mode 100644 index 000000000..6060be655 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_20_65ebf3a3bbf0a2aebd2b69640b757e16._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 20" + date="2013-08-22T15:29:11Z" + content=""" +The git-annex man page has a bit more to say about --relaxed and --fast. Their behavior when used with `importfeed` is the same as with `addurl`. + +If the podcast feed provides an `itemid`, you can use that in the filename template. I don't know how common that is. Due to the way `importfeed` works, it cannot keep track of eg, an incrementing item number itself. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_2_a9a98cad7358d16792853a2ee413fe6c._comment b/doc/tips/downloading_podcasts/comment_2_a9a98cad7358d16792853a2ee413fe6c._comment new file mode 100644 index 000000000..f8ba1155c --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_2_a9a98cad7358d16792853a2ee413fe6c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.21" + subject="comment 2" + date="2013-07-30T17:16:07Z" + content=""" +@ckeen You seem to be using a filesystem that does not support filenames 150 characters long. This is unusual -- even windows and android can support a filename up to 255 characters in length. `git-annex addurl` already deals with this sort of problem by limiting the filename to 255 characters. If you'd like to file a bug report with details about your system, I can try to make git-annex support its limitations, I suppose. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_3_5a8068a5cb0fd864581157a3aa5d1113._comment b/doc/tips/downloading_podcasts/comment_3_5a8068a5cb0fd864581157a3aa5d1113._comment new file mode 100644 index 000000000..7e5633865 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_3_5a8068a5cb0fd864581157a3aa5d1113._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://www.joachim-breitner.de/" + nickname="nomeata" + subject="Great stuff!" + date="2013-07-30T21:21:57Z" + content=""" +Looking forward to seeing it in Debian unstable; where it will definitely replace my hpodder setup. + +I guess there is no easy way to re-use the files already downloaded with hpodder? At first I thought that `git annex importfeed --relaxed` followed by adding the files to the git annex would work, but `importfeed` stores URLs, not content-based hashes, so it wouldn’t match up. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_4_e7072a9da30b4c4b4c526013144238d4._comment b/doc/tips/downloading_podcasts/comment_4_e7072a9da30b4c4b4c526013144238d4._comment new file mode 100644 index 000000000..1693c4bdc --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_4_e7072a9da30b4c4b4c526013144238d4._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.21" + subject="comment 4" + date="2013-07-30T21:29:50Z" + content=""" +@nomeata, well, you can, but it has to download the files again. + +When run without --fast, `importfeed` does use content based hashes, so if you run it in a temporary directory, it will download the content redundantly, hash it and see it's the same, and add the url to that hash. You can then delete the temporary directory, and the files hpodder had downloaded will have the url attached to them now. I don't know if this really buys you anything over deleting the hpodder files and starting over though. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_5_79b3f8d678ac9f67df4c0cd649657283._comment b/doc/tips/downloading_podcasts/comment_5_79b3f8d678ac9f67df4c0cd649657283._comment new file mode 100644 index 000000000..f5df9910f --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_5_79b3f8d678ac9f67df4c0cd649657283._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="ckeen" + ip="79.249.110.228" + subject="Force a reload of a feed?" + date="2013-07-31T10:35:50Z" + content=""" +Currently I have my podcasts imported with --fast. For some reason there are podcast episodes missing. This has been done propably during my period of toying with the feature. If I retry on a clean annex I see all episodes. My suspicion is that git-annex has been interrupted during downloading a feed but now somehow thinks it's already there. How can I debug this situation and/or force git annex to retry all the links in a feed? +"""]] diff --git a/doc/tips/downloading_podcasts/comment_6_35106fee5458bdd5c21868fbc49d3616._comment b/doc/tips/downloading_podcasts/comment_6_35106fee5458bdd5c21868fbc49d3616._comment new file mode 100644 index 000000000..caeca0151 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_6_35106fee5458bdd5c21868fbc49d3616._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.21" + subject="use the force" + date="2013-07-31T16:20:39Z" + content=""" +The only way it can skip downloading a file is if its url has already been seen before. Perhaps you deleted them? + +I've made `importfeed --force` re-download files it's seen before. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_7_ceb16498b7aadbf04a27acd5d6561d46._comment b/doc/tips/downloading_podcasts/comment_7_ceb16498b7aadbf04a27acd5d6561d46._comment new file mode 100644 index 000000000..ac2c89a36 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_7_ceb16498b7aadbf04a27acd5d6561d46._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="ckeen" + ip="78.108.63.46" + subject="--force reload all URLs" + date="2013-08-01T09:47:34Z" + content=""" +Is it intentionally saving URLs with a prefixed 2_? I have sorted out all missing URLs and renamed it, so no harm done, but it has been a bit of a hassle to get there. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_8_147397603f0b3fdb42ca387d1da7c5ef._comment b/doc/tips/downloading_podcasts/comment_8_147397603f0b3fdb42ca387d1da7c5ef._comment new file mode 100644 index 000000000..0995d8075 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_8_147397603f0b3fdb42ca387d1da7c5ef._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.145" + subject="comment 8" + date="2013-08-01T16:05:10Z" + content=""" +I've now made importfeed --force a bit smarter about reusing existing files. +"""]] diff --git a/doc/tips/downloading_podcasts/comment_9_6a26a6cc7683d38fae0f23c5a52d1e23._comment b/doc/tips/downloading_podcasts/comment_9_6a26a6cc7683d38fae0f23c5a52d1e23._comment new file mode 100644 index 000000000..3045c9894 --- /dev/null +++ b/doc/tips/downloading_podcasts/comment_9_6a26a6cc7683d38fae0f23c5a52d1e23._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="http://a-or-b.myopenid.com/" + ip="220.244.41.108" + subject="How do I switch on the 'feeds' feature?" + date="2013-08-05T04:52:41Z" + content=""" +Joey - your initial post said: + + git-annex must be built with the Feeds feature (run git annex version to check). + +...but how do I actually switch on the feeds feature? + +I install git-annex from cabal, so I do + + cabal update + cabal install git-annex + +which I did this morning and now `git annex version` gives me: + + git-annex version: 4.20130802 + build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS + +So it is the latest version, but without Feeds. :-( +"""]] diff --git a/doc/tips/dropboxannex.mdwn b/doc/tips/dropboxannex.mdwn new file mode 100644 index 000000000..926e142ca --- /dev/null +++ b/doc/tips/dropboxannex.mdwn @@ -0,0 +1,28 @@ +dropboxannex +========= + +Hook program for gitannex to use dropbox as backend + +# Requirements: + + python2 + +Credit for the Dropbox api interface goes to Dropbox. + +# Install +Clone the git repository in your home folder. + + git clone git://github.com/TobiasTheViking/dropboxannex.git + +This should make a ~/dropboxannex folder + +# Setup +Run the program once to set it up. + + cd ~/dropboxannex; python2 dropboxannex.py + +# Commands for gitannex: + + git config annex.dropbox-hook '/usr/bin/python2 ~/dropboxannex/dropboxannex.py' + git annex initremote dropbox type=hook hooktype=dropbox encryption=shared + git annex describe dropbox "the dropbox library" diff --git a/doc/tips/emacs_integration.mdwn b/doc/tips/emacs_integration.mdwn new file mode 100644 index 000000000..12f16888a --- /dev/null +++ b/doc/tips/emacs_integration.mdwn @@ -0,0 +1,20 @@ +bergey has developed an emacs mode for browsing git-annex repositories, +dired style. + +<https://gitorious.org/emacs-contrib/annex-mode> + +Locally available files are colored differently, and pressing g runs +`git annex get` on the file at point. + +---- + +John Wiegley has developed a brand new git-annex interaction mode for +Emacs, which aims to integrate with the standard facilities +(C-x C-q, M-x dired, etc) rather than invent its own interface. + +<https://github.com/jwiegley/git-annex-el> + +He has also added support to org-attach; if +`org-attach-git-annex-cutoff' is non-nil and smaller than the size + of the file you're attaching then org-attach will `git annex add the +file`; otherwise it will "git add" it. diff --git a/doc/tips/finding_duplicate_files.mdwn b/doc/tips/finding_duplicate_files.mdwn new file mode 100644 index 000000000..94fc85400 --- /dev/null +++ b/doc/tips/finding_duplicate_files.mdwn @@ -0,0 +1,21 @@ +Maybe you had a lot of files scattered around on different drives, and you +added them all into a single git-annex repository. Some of the files are +surely duplicates of others. + +While git-annex stores the file contents efficiently, it would still +help in cleaning up this mess if you could find, and perhaps remove +the duplicate files. + +Here's a command line that will show duplicate sets of files grouped together: + + git annex find --include '*' --format='${file} ${escaped_key}\n' | \ + sort -k2 | uniq --all-repeated=separate -f1 | \ + sed 's/ [^ ]*$//' + +Here's a command line that will remove one of each duplicate set of files: + + git annex find --include '*' --format='${file} ${escaped_key}\n' | \ + sort -k2 | uniq --repeated -f1 | sed 's/ [^ ]*$//' | \ + xargs -d '\n' git rm + +--[[Joey]] diff --git a/doc/tips/finding_duplicate_files/comment_10_2ed5aa8c632048b13e01d358883fa383._comment b/doc/tips/finding_duplicate_files/comment_10_2ed5aa8c632048b13e01d358883fa383._comment new file mode 100644 index 000000000..77a308b90 --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_10_2ed5aa8c632048b13e01d358883fa383._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmTNrhkVQ26GBLaLD5-zNuEiR8syTj4mI8" + nickname="Juan" + subject="comment 10" + date="2013-08-31T18:20:58Z" + content=""" +I'm already spreading the word. Handling scientific papers, data, simulations and code has been quite a challenge during my academic career. While code was solved long ago, the three first items remained a huge problem. +I'm sure many of my colleagues will be happy to use it. +Is there any hashtag or twitter account? I've seen that you collected some of my tweets, but I don't know how you did it. Did you search for git-annex? +Best, + Juan +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_1_ddb477ca242ffeb21e0df394d8fdf5d2._comment b/doc/tips/finding_duplicate_files/comment_1_ddb477ca242ffeb21e0df394d8fdf5d2._comment new file mode 100644 index 000000000..d1bd4475e --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_1_ddb477ca242ffeb21e0df394d8fdf5d2._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://adamspiers.myopenid.com/" + nickname="Adam" + subject="Cool" + date="2011-12-23T19:16:50Z" + content=""" +Very nice :) Just for reference, here's [my Perl implementation](https://github.com/aspiers/git-config/blob/master/bin/git-annex-finddups). As per [this discussion](http://git-annex.branchable.com/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/#comment-fb15d5829a52cd05bcbd5dc53edaffb2) it would be interesting to benchmark these two approaches and see if one is substantially more efficient than the other w.r.t. CPU and memory usage. +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_2_900eafe0a781018ff44b35ac232e3ad3._comment b/doc/tips/finding_duplicate_files/comment_2_900eafe0a781018ff44b35ac232e3ad3._comment new file mode 100644 index 000000000..605c804dd --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_2_900eafe0a781018ff44b35ac232e3ad3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="bremner" + ip="156.34.89.108" + subject="problems with spaces in filenames" + date="2012-09-05T02:12:18Z" + content=""" +note that the sort -k2 doesn't work right for filenames with spaces in them. On the other hand, git-rm doesn't seem to like the escaped names from escaped_file. +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_3._comment b/doc/tips/finding_duplicate_files/comment_3._comment new file mode 100644 index 000000000..44eeb5075 --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_3._comment @@ -0,0 +1,39 @@ +[[!comment format=mdwn + username="mhameed" + ip="82.32.202.53" + subject="problems with spaces in filenames" + date="Wed Sep 5 09:38:56 BST 2012" + content=""" + +Spaces, and other special chars can make filename handeling ugly. +If you don't have a restriction on keeping the exact filenames, then +it might be easiest just to get rid of the problematic chars. + + #!/bin/bash + + function process() { + dir="$1" + echo "processing $dir" + pushd $dir >/dev/null 2>&1 + + for fileOrDir in *; do + nfileOrDir=`echo "$fileOrDir" | sed -e 's/\[//g' -e 's/\]//g' -e 's/ /_/g' -e "s/'//g" ` + if [ "$fileOrDir" != "$nfileOrDir" ]; then + echo renaming $fileOrDir to $nfileOrDir + git mv "$fileOrDir" "$nfileOrDir" + else + echo "skipping $fileOrDir, no need to rename." + fi + done + + find ./ -mindepth 1 -maxdepth 1 -type d | while read d; do + process "$d" + done + popd >/dev/null 2>&1 + } + + process . + +Maybe you can run something like this before checking for duplicates. + +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_4_1494143a74cc1e9fbe4720c14b73d42b._comment b/doc/tips/finding_duplicate_files/comment_4_1494143a74cc1e9fbe4720c14b73d42b._comment new file mode 100644 index 000000000..f1a86f43c --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_4_1494143a74cc1e9fbe4720c14b73d42b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="bremner" + ip="156.34.89.108" + subject="more about spaces..." + date="2012-09-09T19:33:01Z" + content=""" +Ironically, previous renaming to remove spaces, plus some synching is how I ended up with these duplicates. For what it is worth, aspiers perl script worked out for me with a small modification. I just only printed out the duplicates with spaces in them (quoted). +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_5_1a35ca360468bcb84a67ad8d62a2ef7d._comment b/doc/tips/finding_duplicate_files/comment_5_1a35ca360468bcb84a67ad8d62a2ef7d._comment new file mode 100644 index 000000000..23beb779f --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_5_1a35ca360468bcb84a67ad8d62a2ef7d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkaBh9VNJ-RZ26wJZ4BEhMN1IlPT-DK6JA" + nickname="Alex" + subject="printing keys first is the easiest workaround" + date="2013-04-01T23:32:23Z" + content=""" +Since the keys are sure to have nos paces in them, putting them first makes working with the output with tools like sort, uniq, and awk simpler. +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_6_a6e88c93b31f67c933523725ff61b287._comment b/doc/tips/finding_duplicate_files/comment_6_a6e88c93b31f67c933523725ff61b287._comment new file mode 100644 index 000000000..31601a989 --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_6_a6e88c93b31f67c933523725ff61b287._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnkBYpLu_NOj7Uq0-acvLgWhxF8AUEIJbo" + nickname="Chris" + subject="Find files by key" + date="2013-05-03T04:14:55Z" + content=""" +Is there any simple way to search for files with a given key? + +At the moment, the best I've come up with is this: + +```` +git annex find --include '*' --format='${key} ${file}' | grep <KEY> +```` + +where `<KEY>` is the key. This seems like an awfully longwinded approach, but I don't see anything in the docs indicating a simpler way to do it. Am I missing something? +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_7_347b0186755a809594bd42feda6363e2._comment b/doc/tips/finding_duplicate_files/comment_7_347b0186755a809594bd42feda6363e2._comment new file mode 100644 index 000000000..d97b0d500 --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_7_347b0186755a809594bd42feda6363e2._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 7" + date="2013-05-13T18:42:14Z" + content=""" +@Chris I guess there's no really easy way because searching for a given key is not something many people need to do. + +However, git does provide a way. Try `git log --stat -S $KEY` +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_8_3af51722da0980b724facb184f0f66e9._comment b/doc/tips/finding_duplicate_files/comment_8_3af51722da0980b724facb184f0f66e9._comment new file mode 100644 index 000000000..26c34fcfa --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_8_3af51722da0980b724facb184f0f66e9._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmTNrhkVQ26GBLaLD5-zNuEiR8syTj4mI8" + nickname="Juan" + subject="This is an awesome feature" + date="2013-08-28T13:40:23Z" + content=""" +Thanks. I have quite a lot of papers in PDF formats. Now I'm saving space, have them controlled, synchronized with many devices and found more than 200 duplicates. +Is there a way to donate to the project? You really deserve it. +Thanks. +"""]] diff --git a/doc/tips/finding_duplicate_files/comment_9_7b4b78a5cd253abfe4f6001049bf64f3._comment b/doc/tips/finding_duplicate_files/comment_9_7b4b78a5cd253abfe4f6001049bf64f3._comment new file mode 100644 index 000000000..a20ca16ed --- /dev/null +++ b/doc/tips/finding_duplicate_files/comment_9_7b4b78a5cd253abfe4f6001049bf64f3._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.8.7" + subject="comment 9" + date="2013-08-28T20:25:20Z" + content=""" +@Juan the best thing to do is tell people about git-annex, help them use it, and file bug reports. Just generally be part of the git-annex community. + +(If you really want to donate to me, <http://campaign.joeyh.name/> is still open.) +"""]] diff --git a/doc/tips/flickrannex.mdwn b/doc/tips/flickrannex.mdwn new file mode 100644 index 000000000..d8e54b4c3 --- /dev/null +++ b/doc/tips/flickrannex.mdwn @@ -0,0 +1,62 @@ +# Latest version 0.1.10 +Hook program for gitannex to use flickr as backend. + +This allows storing any type of file on flickr, not only images and movies. + +# Requirements: + + python2 + +Credit for the flickr api interface goes to: <http://stuvel.eu/flickrapi> +Credit for the png library goes to: <https://github.com/drj11/pypng> +Credit for the png tEXt patch goes to: <https://code.google.com/p/pypng/issues/detail?id=65> + +# Install + +Clone the git repository in your home folder. + + git clone git://github.com/TobiasTheViking/flickrannex.git + +This should make a ~/flickrannex folder + +# Setup + +Run the program once to set it up. + + cd ~/flickrannex; python2 flickrannex.py + +After the setup has finished, it will print the git-annex configure lines. + +# Configuring git-annex + + git config annex.flickr-hook '/usr/bin/python2 ~/flickrannex/flickrannex.py' + git annex initremote flickr type=hook hooktype=flickr encryption=shared + git annex describe flickr "the flickr library" + +# Notes + +## Unencrypted mode +The photo name on flickr is currently the GPGHMACSHA1 version. + +Run the following command in your annex directory + git annex wanted flickr uuid include=*.jpg or include=*.jpeg or include=*.gif or include=*.png + +## Encrypted mode +The current version base64 encodes all the data, which results in ~35% larger filesize. + +I might look into yyenc instead. I'm not sure if it will work in the tEXt field. + +Run the following command in your annex directory + git annex wanted flickr exclude=largerthan=30mb + +## Including directories as tags +Get get each of the directories below the top level git directory added as tags to uploads: + + git config annex.flickr-hook 'GIT_TOP_LEVEL=`git rev-parse --show-toplevel` /usr/bin/python2 %s/flickrannex.py' + +In this case the image: + /home/me/annex-photos/holidays/2013/Greenland/img001.jpg +would get the following tags: "holidays" "2013" "Greenland" +(assuming "/home/me/annex-photos" is the top level in the annex...) + +Caveat Emptor - Tags will *always* be NULL for indirect repos - we don't (easily) know the human-readable file name. diff --git a/doc/tips/flickrannex/comment_10_50707f259abe5829ce075dfbecd5a4ba._comment b/doc/tips/flickrannex/comment_10_50707f259abe5829ce075dfbecd5a4ba._comment new file mode 100644 index 000000000..7bda45e5c --- /dev/null +++ b/doc/tips/flickrannex/comment_10_50707f259abe5829ce075dfbecd5a4ba._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8" + nickname="develop" + subject="comment 10" + date="2013-06-07T09:39:59Z" + content=""" +I'm not even sure if chunksize is exposed to the hooks at all. + +As it is, the hook will check the filesize, and if the filesize is more than 30mbyte it will exit 1. + +Chunking may be implemented down the road. I do believe joeyh might have some plans that will touch this issue, so I'd rather wait. Than re-invent the wheel yet again. + +"""]] diff --git a/doc/tips/flickrannex/comment_11_ab5bcb025381b3da4d7c6dfd0c7310dd._comment b/doc/tips/flickrannex/comment_11_ab5bcb025381b3da4d7c6dfd0c7310dd._comment new file mode 100644 index 000000000..e59aa6500 --- /dev/null +++ b/doc/tips/flickrannex/comment_11_ab5bcb025381b3da4d7c6dfd0c7310dd._comment @@ -0,0 +1,46 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y" + nickname="Nigel" + subject="git annex get failed" + date="2013-08-02T14:29:30Z" + content=""" +Hi, I am coming back to this and testing Flickr as a repository for moving files about and have run into what may be my very basic misunderstanding with vanilla annex. + +I copied one file to Flickr and dropped it elsewhere (--force). I assumed that the file was on Flickr ok but that the numcopies setting required the force because of the semi-trust level of the Flickr remote. + +Then I find I can't get the file back, even though there is a record of it from whereis. + +Can you help enlighten me as to what am I missing? I assumed whereis would only report files that exist and can be copied back. If not my error, I can raise bug or search for logs. Thanks in advance for any help. + +[[!format perl \"\"\" + + +nrb@nrb-ThinkPad-T61:~/tmp$ git annex whereis +whereis libpeerconnection.log (3 copies) + 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) + 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) + ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive +ok +whereis test.cgi (1 copy) + 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) +ok +whereis walkthrough.sh (3 copies) + 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) + 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) + ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive +ok +whereis walkthrough.sh~ (3 copies) + 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) + 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) + ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive +ok +nrb@nrb-ThinkPad-T61:~/tmp$ git annex get test.cgi +get test.cgi (from flickr...) + +git-annex: /home/nrb/tmp/.git/annex/tmp/SHA256E-s48--a01eedbee949120aeda41e566f9ae8faef1c2bacaa6d7bb8e45050fb8df6d09d.cgi: rename: does not exist (No such file or directory) +failed +git-annex: get: 1 failed +nrb@nrb-ThinkPad-T61:~/tmp$ + +\"\"\"]] +"""]] diff --git a/doc/tips/flickrannex/comment_12_90a331275d888221bc695003c8acbe46._comment b/doc/tips/flickrannex/comment_12_90a331275d888221bc695003c8acbe46._comment new file mode 100644 index 000000000..003755f30 --- /dev/null +++ b/doc/tips/flickrannex/comment_12_90a331275d888221bc695003c8acbe46._comment @@ -0,0 +1,58 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y" + nickname="Nigel" + subject="re: git annex get failed" + date="2013-08-02T15:02:14Z" + content=""" +Another try - this time a slightly simpler setup using my version of the walkthrough commands + +[[!format bash \"\"\" + +nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex drop walkthrough.sh --from usbdrive +drop usbdrive walkthrough.sh ok +(Recording state in git...) +nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex move walkthrough.sh --to flickr +move walkthrough.sh (gpg) (checking flickr...) (to flickr...) +/home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. + if res: +/home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. + if res: +ok +(Recording state in git...) +nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex whereis +whereis walkthrough.sh (1 copy) + 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) +ok +whereis walkthrough.sh~ (3 copies) + 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) + 7803d853-d231-4bb4-b696-f12a950fb96b -- here (my laptop) + d60d75f9-d878-4214-af20-fa055134ae77 -- usbdrive (portable USB drive) +ok +nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex get walkthrough.sh +get walkthrough.sh (from flickr...) (gpg) +git-annex: /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--02f600d7e8b071d2945270fd5e7fc26dd066ff31: openBinaryFile: does not exist (No such file or directory) +gpg: decrypt_message failed: eof + + Unable to access these remotes: flickr + + Try making some of these repositories available: + 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) +failed +git-annex: get: 1 failed +nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex fsck --from flickr +fsck walkthrough.sh (gpg) (checking flickr...) (fixing location log) + ** Based on the location log, walkthrough.sh + ** was expected to be present, but its content is missing. + + ** No known copies exist of walkthrough.sh +failed +fsck walkthrough.sh~ (checking flickr...) (fixing location log) + ** Based on the location log, walkthrough.sh~ + ** was expected to be present, but its content is missing. +failed +(Recording state in git...) +git-annex: fsck: 2 failed +nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ + +\"\"\" ]] +"""]] diff --git a/doc/tips/flickrannex/comment_13_1596e70dca71c853fd1d6fc9bde02b18._comment b/doc/tips/flickrannex/comment_13_1596e70dca71c853fd1d6fc9bde02b18._comment new file mode 100644 index 000000000..19faa585e --- /dev/null +++ b/doc/tips/flickrannex/comment_13_1596e70dca71c853fd1d6fc9bde02b18._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8" + nickname="develop" + subject="Version 0.1.10 pushed" + date="2013-09-11T20:31:25Z" + content=""" +Since the initial release of this hook a lot of issues have been fixed, and a few features added. + +I would highly suggest that everyone who is using this hook update to the latest version as i would consider one of the bugs to be fairly major. + + +"""]] diff --git a/doc/tips/flickrannex/comment_2_d74c4fc7edf8e47f7482564ce0ef4d12._comment b/doc/tips/flickrannex/comment_2_d74c4fc7edf8e47f7482564ce0ef4d12._comment new file mode 100644 index 000000000..d015dc195 --- /dev/null +++ b/doc/tips/flickrannex/comment_2_d74c4fc7edf8e47f7482564ce0ef4d12._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8" + nickname="develop" + subject="comment 2" + date="2013-06-05T21:33:42Z" + content=""" +Get the statically linked version from here http://git-annex.branchable.com/install/Linux_standalone/ + +I believe the new hook format was introduced in version 4.20130521 +"""]] diff --git a/doc/tips/flickrannex/comment_2_f53d0d5520e2835e9705bea4e75556f0._comment b/doc/tips/flickrannex/comment_2_f53d0d5520e2835e9705bea4e75556f0._comment new file mode 100644 index 000000000..14d7a1b7c --- /dev/null +++ b/doc/tips/flickrannex/comment_2_f53d0d5520e2835e9705bea4e75556f0._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y" + nickname="Nigel" + subject="missing configuration for flickr-checkpresent-hook" + date="2013-06-05T20:44:25Z" + content=""" +<https://github.com/TobiasTheViking/flickrannex/issues/3> + +9 days ago: [the annex] \"hook format a few versions ago, and this is using the new hook format\". + +Looks very handy. I am just starting with this, but can't seem to get it working as a remote after following the simple walkthrough. All goes well until: + + $ git annex copy . --to flickr + copy walkthrough.sh (checking flickr...) + missing configuration for flickr-checkpresent-hook + git-annex: checkpresent hook misconfigured + +my Ubuntu 12.04: + + $ git annex version + git-annex version: 4.20130516.1 + build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP + local repository version: 3 + default repository version: 3 + supported repository versions: 3 4 + upgrade supported from repository versions: 0 1 2 + +I guess my \"git-annex version is still too old\"? Any idea what version is needed? Even better if I can figure out which Linux distribution/release has the most up to date version of annex. + +"""]] diff --git a/doc/tips/flickrannex/comment_4_9ebba4d61140f6c2071e988c9328cf7e._comment b/doc/tips/flickrannex/comment_4_9ebba4d61140f6c2071e988c9328cf7e._comment new file mode 100644 index 000000000..741b0c5ba --- /dev/null +++ b/doc/tips/flickrannex/comment_4_9ebba4d61140f6c2071e988c9328cf7e._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8" + nickname="develop" + subject="comment 4" + date="2013-06-05T22:02:29Z" + content=""" +The path for the binary \"/usr/bin/python2\" is wrong. + +It could be any of /usr/bin/python /usr/bin/python2.6 /usr/bin/python2.7 + +Or maybe in /usr/local/bin + +you can try running \"which python\" or \"which python2\" to get the real path. +"""]] diff --git a/doc/tips/flickrannex/comment_5_4470dae270613dd8712623474bc80ab0._comment b/doc/tips/flickrannex/comment_5_4470dae270613dd8712623474bc80ab0._comment new file mode 100644 index 000000000..1c19711df --- /dev/null +++ b/doc/tips/flickrannex/comment_5_4470dae270613dd8712623474bc80ab0._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y" + nickname="Nigel" + subject="missing configuration for flickr-checkpresent-hook" + date="2013-06-05T22:00:48Z" + content=""" +Many thanks. + +I used gitannex-install and was left with a slight anomaly: + + Installing...........done + git-annex version 4.20130601 has been installed + $ git-annex version + git-annex version: 4.20130531-g5df09b5 + +But I guess this includes the new hook format. I get a bit further: + + $ git annex copy . --to flickr + copy walkthrough.sh (checking flickr...) (user error (sh [\"-c\",\"/usr/bin/python2 /home/nrb/repos/gits/flickrannex/flickrannex.py\"] exited 1)) failed + copy walkthrough.sh~ (checking flickr...) (user error (sh [\"-c\",\"/usr/bin/python2 /home/nrb/repos/gits/flickrannex/flickrannex.py\"] exited 1)) failed + git-annex: copy: 2 failed + + +"""]] diff --git a/doc/tips/flickrannex/comment_5_d395cdcf815cb430e374ff05c1a63ff4._comment b/doc/tips/flickrannex/comment_5_d395cdcf815cb430e374ff05c1a63ff4._comment new file mode 100644 index 000000000..dbeaafb73 --- /dev/null +++ b/doc/tips/flickrannex/comment_5_d395cdcf815cb430e374ff05c1a63ff4._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y" + nickname="Nigel" + subject="comment 5" + date="2013-06-05T22:11:14Z" + content=""" +Thanks, but on my machine I get: + + $ which python2 + /usr/bin/python2 + +I have scripted all my walkthrough commands, blowing away the test repositories and flickr settings first each time. This re-runs the flickr scripts and git config annex.flickr-hook etc. + +I can't spot anything here. + + +"""]] diff --git a/doc/tips/flickrannex/comment_6_8cf730097001ffe106f2c743edce9d0a._comment b/doc/tips/flickrannex/comment_6_8cf730097001ffe106f2c743edce9d0a._comment new file mode 100644 index 000000000..8e7b15ed0 --- /dev/null +++ b/doc/tips/flickrannex/comment_6_8cf730097001ffe106f2c743edce9d0a._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA" + nickname="Tobias" + subject="comment 6" + date="2013-06-06T09:44:11Z" + content=""" +That's weird... + +You could try adding \"--dbglevel 1 --stderr\" arguments to the hook command and give me the output. But the way i read the log it seems like it doesn't even launch the python intrepreter. I might be wrong though. + + +"""]] diff --git a/doc/tips/flickrannex/comment_7_a80c8087c4e1562a4c98a24edc182e5a._comment b/doc/tips/flickrannex/comment_7_a80c8087c4e1562a4c98a24edc182e5a._comment new file mode 100644 index 000000000..9e0eb0a73 --- /dev/null +++ b/doc/tips/flickrannex/comment_7_a80c8087c4e1562a4c98a24edc182e5a._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y" + nickname="Nigel" + subject="Unencrypted flickr can only accept picture and video files" + date="2013-06-06T10:24:58Z" + content=""" +Thanks and sorry to trouble you, it is my error, I picked unencrypted option (thinking it would be less of an issue) and am using a text file for test, gave an error line: + + 10:53:07 [flickrannex-0.1.5] main : 'Unencrypted flickr can only accept picture and video files' + +I've not looked through your code yet, but could that message be printed when not in debug mode? +"""]] diff --git a/doc/tips/flickrannex/comment_8_94f84254c32cf0f7dd1441b7da5d2bc6._comment b/doc/tips/flickrannex/comment_8_94f84254c32cf0f7dd1441b7da5d2bc6._comment new file mode 100644 index 000000000..ff11a618a --- /dev/null +++ b/doc/tips/flickrannex/comment_8_94f84254c32cf0f7dd1441b7da5d2bc6._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA" + nickname="Tobias" + subject="comment 8" + date="2013-06-06T10:51:39Z" + content=""" +I'll make it so, in the next version i push. +"""]] diff --git a/doc/tips/flickrannex/comment_9_5299b4cab4a4cb8e8fd4d2b39f0ea59c._comment b/doc/tips/flickrannex/comment_9_5299b4cab4a4cb8e8fd4d2b39f0ea59c._comment new file mode 100644 index 000000000..f25cd04c1 --- /dev/null +++ b/doc/tips/flickrannex/comment_9_5299b4cab4a4cb8e8fd4d2b39f0ea59c._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawleVyKk2kQsB_HgEdS7w1s0BmgRGy1aay0" + nickname="Milan" + subject="chunksize" + date="2013-06-07T09:09:56Z" + content=""" +Hi! Does this backend support chunksize option? If yes, is it possible to set it after the remote has been added to the repository? +Thanks, Milan. +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt.mdwn b/doc/tips/fully_encrypted_git_repositories_with_gcrypt.mdwn new file mode 100644 index 000000000..5934747f0 --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt.mdwn @@ -0,0 +1,127 @@ +[git-remote-gcrypt](https://github.com/joeyh/git-remote-gcrypt/) +adds support for encrypted remotes to git. The git-annex +[[gcrypt special remote|special_remotes/gcrypt]] allows git-annex to +also store its files in such repositories. Naturally, git-annex encrypts +the files it stores too, so everything stored on the remote is encrypted. + +Here are some ways you can use this awesome stuff.. + +[[!toc ]] + +This page will show how to set it up at the command line, but the git-annex +[[assistant]] can also be used to help you set up encrypted git +repositories. + +## prerequisites + +* Install +[git-remote-gcrypt](https://github.com/joeyh/git-remote-gcrypt/) +* Install git-annex version 4.20130909 or newer. + +## encrypted backup drive + +Let's make a USB drive into an encrypted backup repository. It will contain +both the full contents of your git repository, and all the files you +instruct git-annex to store on it, and everything will be encrypted so that +only you can see it. + +First, you need to set up a gpg key. You might consider generating a +special purpose key just for this use case, since you may end up wanting to +put the key on multiple machines that you would not trust with your +main gpg key. + +You need to tell git-annex the keyid of the key when setting up the +encrypted repository: + + git init --bare /mnt/encryptedbackup + git annex initremote encryptedbackup type=gcrypt gitrepo=/mnt/encryptedbackup keyid=$mykey + git annex sync encryptedbackup + +Now you can copy (or even move) files to the repository. After +sending files to it, you'll probably want to do a sync, which pushes +the git repository changes to it as well. + + git annex copy --to encryptedbackup ... + git annex sync encryptedbackup + +Note that if you lose your gpg key, it will be *impossible* to get the +data out of your encrypted backup. You need to find a secure way to store a +backup of your gpg key. Printing it out and storing it in a safe deposit box, +for example. + +You can actually specifiy keyid= as many times as you like to allow any one +of a set of gpg keys to access this repository. So you could add a friend's +key, or another gpg key you have. + +To restore from the backup, just plug the drive into any machine that has +the gpg key used to encrypt it, and then: + + git clone gcrypt::/mnt/encryptedbackup restored + cd restored + git annex enableremote encryptedbackup gitrepo=/mnt/encryptedbackup + git annex get --from encryptedbackup + +## encrypted git-annex repository on a ssh server + +If you have a ssh server that has rsync installed, you can set up an +encrypted repository there. Works just like the encrypted drive except +without the cable. + +First, on the server, run: + + git init --bare encryptedrepo + +(Also, install git-annex on the server if it's possible & easy to do so. +While this will work without git-annex being installed on the server, it +is recommended to have it installed.) + +Now, in your existing git-annex repository, set up the encrypted remote: + + git annex initremote encryptedrepo type=gcrypt gitrepo=ssh://my.server/home/me/encryptedrepo keyid=$mykey + git annex sync encryptedrepo + +If you're going to be sharing this repository with others, be sure to also +include their keyids, by specifying keyid= repeatedly. + +Now you can copy (or even move) files to the repository. After +sending files to it, you'll probably want to do a sync, which pushes +the git repository changes to it as well. + + git annex copy --to encryptedrepo ... + git annex sync encryptedbackup + +Anyone who has access to the repo it and has one of the keys +used to encrypt it can check it out: + + git clone gcrypt::ssh://my.server/home/me/encryptedrepo myrepo + cd myrepo + git annex enableremote encryptedrepo gitrepo=ssh://my.server/home/me/encryptedrepo + git annex get --from encryptedrepo + +## private encrypted git remote on hosting site + +You can use gcrypt to store your git repository in encrypted form on any +hosting site that supports git. Only you can decrypt its contents. +Using it this way, git-annex does not store large files on the hosting site; it's +only used to store your git repository itself. + + git remote add encrypted gcrypt::ssh://hostingsite/myrepo.git + git push encrypted master git-annex + +Now you can carry on using git-annex with your new repository. For example, +`git annex sync` will sync with it. + +To check out the repository from the hosting site, use the same gcrypt:: +url you used when setting it up: + + git clone gcrypt::ssh://hostingsite/myrepo.git + +## multiuser encrypted git remote on hosting site + +Suppose two users want to share an encrypted git remote. Both of you +need to set up the remote, and configure gcrypt to encrypt it so that both +of you can see it. + + git remote add sharedencrypted gcrypt::ssh://hostingsite/myrepo.git + git config remote.sharedencrypted.gcryt-participants "$mykey $friendkey" + git config git push sharedencrypted master git-annex diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_10_4440a80d64c60c7312d5c405d54e607a._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_10_4440a80d64c60c7312d5c405d54e607a._comment new file mode 100644 index 000000000..4ee70bcd7 --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_10_4440a80d64c60c7312d5c405d54e607a._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="tanen" + ip="83.128.159.25" + subject="comment 10" + date="2013-11-04T17:58:36Z" + content=""" +> \"We could symetrically encrypt the repository with a keyfile that's stored in the repository itself\" +> Then you would need to decrypt the repository in order get the key you need to decrypt the repository. The impossibility of this design is why I didn't do that! + +Sorry, I ment that the file containing the symmetric encryption key should obviously not be used to encrypt itself, it would be stored in the repository \"unencrypted\" (but protected with a passphrase) + +> store a non-encrypted gpg key alongside the repsitory encrypted with it, but then you have to rely on a passphrase for all your security. + +Exactly. I think such a mode be a great addition. It might not be as secure as encryption based on a private key - depending on the passphrase strength -, but it would certainly be a lot more convenient and portable (and still much more secure than the shared encryption method). +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_1_5c54690586f2a781905ea4b25aa1147f._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_1_5c54690586f2a781905ea4b25aa1147f._comment new file mode 100644 index 000000000..71305e650 --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_1_5c54690586f2a781905ea4b25aa1147f._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkbpbjP5j8MqWt_K4NASwv0WvB8T4rQ-pM" + nickname="Fabrice" + subject="Is there a way to specify a preferred pgp key?" + date="2013-11-01T18:57:38Z" + content=""" +Hi, + +I think the current behavior of the special remote is a bit annoying when one has several pgp keys. + +Indeed, I've followed the encrypted backup drive example specifying the id of a dedicated key in the initremote step, so far so good. Doing that, I was prompted for my key phrase by the gnome keyring daemon, as expected. + +The annoying part starts right at the git annex sync step. Indeed, when git-remote-gcrypt tries to decrypt the manifest from the encrypted remote, rather than trying only the key specified during the initremote step, it tries all my (secret) keys. This means that I get prompted for the key phrase of all those keys (minus the correct one which is already unlocked...). + +In the future, this might possible to avoid by allowing gcrypt to fetch a preferred key from git config and to use with the --try-secret-key option available gnupg 2.1.x. But for 1.x or 2.0.x, the simpler option --default-key does not seem to alter the order in which keys are tried to decrypt the manifest. Also, it does not seem to be a problem of the gnome keyring daemon, but rather a gpg problem as when the daemon is replaced by the standard gpg-agent, the same problem occurs. + +Meanwhile, is there any way to avoid this problem? +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_2_07feedb4348f8c31176cc744c19368a1._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_2_07feedb4348f8c31176cc744c19368a1._comment new file mode 100644 index 000000000..b154263fe --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_2_07feedb4348f8c31176cc744c19368a1._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkbpbjP5j8MqWt_K4NASwv0WvB8T4rQ-pM" + nickname="Fabrice" + subject="A possible solution" + date="2013-11-02T14:22:13Z" + content=""" +I'm answering to myself :-). A possible solution to the annoying pass phrase asking with current gnupg is to use a specialized secret keyring. One first exports the secret key used for this repository in a specific keyring as follows: + +`gpg --export-secret-keys keyid | gpg --import --no-default-keyring --secret-keyring mygitannexsecret.gpg` + +This will create a keyring in $HOME/.gnupg with only the specific key. + +Then, in the git-remote-gcrypt shell script, gpg should be called as follows + +`gpg --no-default-keyring --secret-keyring mygitannexsecret.gpg -q -d ...` + +when decrypting the manifest in order to try only the specific key. This behavior can be easily triggered via some git configuration variable. + +Any comment? + +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_3_c2f873dffa015f1d72ad0c3921909316._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_3_c2f873dffa015f1d72ad0c3921909316._comment new file mode 100644 index 000000000..0ce74d767 --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_3_c2f873dffa015f1d72ad0c3921909316._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.47" + subject="comment 3" + date="2013-11-02T17:32:28Z" + content=""" +Fabrice, I've filed a bug report about this: <https://github.com/blake2-ppc/git-remote-gcrypt/issues/9> +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_4_f8a6e4415f4fe6da14f6a3b7334bc952._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_4_f8a6e4415f4fe6da14f6a3b7334bc952._comment new file mode 100644 index 000000000..9b1307df4 --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_4_f8a6e4415f4fe6da14f6a3b7334bc952._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="tanen" + ip="83.128.159.25" + subject="comment 4" + date="2013-11-03T22:35:07Z" + content=""" +The way I would want to setup git-annex (assistant) is \"Wuala/Spideroak style\": two computers with a full checkout of the repository, changes automatically being synced between them, even if the two computers are never online simultaneously, and encryption should be done locally: the (special) remote should not be able to view file listings or content. + +Do I understand it correctly that the gcrypt remote is the only way to make this happen? I tried to create such a setup via the webapp but failed. Adding the repository and remote (via \"Encrypt with GnuPG key\") on the first computer went OK*, but trying to enable that remote on the other computer fails: clicking enable asks me for the SSH password, but after that I just get redirected to a blank screen, with nothing to see in the logfile after the succesful call to ssh-keygen. No entry for the second computer is being added to authorized_keys on the remote. + +Perhaps this is because at this point the assistant is unable to actually parse the content of the encrypted repository? I tried importing the private key that was used while creating the repository on the other computer, but that made no difference. + +Thinking about this for a while, I believe gpg keys aren't actually particularly suited for this usecase. Even without the bug above, one would either have to awkwardly copy a private key to all hosts that are syncing to the repository; or, every time a new (or reinstalled) host wants to sync the repository, you would manually have to add the new keyid to the config and do the forced push + GCRYPT_FULL_REPACK, presumably having to reupload your entire history. Apart from this, having to backup a private key (outside of your git-annex based backups!) would be quite inconvenient. + +How would you feel about adding a new mode of operation where encryption is simply based on a passphrase? We could symetrically encrypt the repository with a keyfile that's stored in the repository itself, protecting the keyfile with a passphrase which - if stored at all - would be stored on the individual computers, outside of the repository. + +*although it erroneously used \"E0D2F776E7F674E3\" as key-id while the actual id is E7F674E3; where did that other half come from? +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_5_11b8e82d2a234f65b58b823e71c6d6a2._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_5_11b8e82d2a234f65b58b823e71c6d6a2._comment new file mode 100644 index 000000000..8e767254c --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_5_11b8e82d2a234f65b58b823e71c6d6a2._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" + nickname="Adam" + subject="comment 5" + date="2013-11-04T04:40:53Z" + content=""" +> How would you feel about adding a new mode of operation where encryption is simply based on a passphrase? We could symetrically encrypt the repository with a keyfile that's stored in the repository itself, protecting the keyfile with a passphrase which - if stored at all - would be stored on the individual computers, outside of the repository. + +Isn't that what the regular shared-encryption remote already does? Except it doesn't put a passphrase on the key, because anyone who has access to the local repo wouldn't need access to the remote one anyway. +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_6_3e41948e1beffcf279bb05ef8e61cc07._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_6_3e41948e1beffcf279bb05ef8e61cc07._comment new file mode 100644 index 000000000..1a5d7f6e1 --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_6_3e41948e1beffcf279bb05ef8e61cc07._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkbpbjP5j8MqWt_K4NASwv0WvB8T4rQ-pM" + nickname="Fabrice" + subject="comment 6" + date="2013-11-04T07:39:21Z" + content=""" +> _How would you feel about adding a new mode of operation where encryption is simply based on a passphrase? We could symetrically encrypt the repository with a keyfile that's stored in the repository itself, protecting the keyfile with a passphrase which - if stored at all - would be stored on the individual computers, outside of the repository._ + +As Adam wrote, without a passphrase, this is the shared encryption method. With an encrypted key, this is more or less the hybrid (default) scheme. The thing is that you have to share a secret to have a encrypted remote. I don't use the webapp, so I don't know what's happening in your case, but this is how it should work with the command line tools. First Alice create the encrypted remote with her pgp key. As far as I understand, git annex creates (via gpg) a key for a symmetric cypher which is stored in the repository, encrypted with Alice public key. If Alice wants to share the repository with Bob, she must either give a key pair (so the private key also, of course) to Bob or ask Bob for his public key. In the first case, Bob can clone the repository directly (upon reception of the key pair), while in the second case, Alice has to active Bob's public key (with `git annex enableremote myremote keyid+=bobsId`). In this case, again as far as I understand, the symmetric key is reencrypted for both Alice and Bob in the repo. + +I understand that you tried the first case with the webapp and that it did not work. I had a similar problem documented in this [http://git-annex.branchable.com/bugs/git-annex-shell:_gcryptsetup_permission_denied](bug). Maybe you could had some comments to this bug description? + +> _*although it erroneously used \"E0D2F776E7F674E3\" as key-id while the actual id is E7F674E3; where did that other half come from?_ + +This is the long id of your pgp key (16 characters as opposed to 8 for the short id). +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_7_4ce0b26b25b336f07b2e27246cdfba3e._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_7_4ce0b26b25b336f07b2e27246cdfba3e._comment new file mode 100644 index 000000000..dfb2a3b92 --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_7_4ce0b26b25b336f07b2e27246cdfba3e._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="tanen" + ip="83.128.159.25" + subject="comment 7" + date="2013-11-04T09:01:13Z" + content=""" +Thanks for the responses. Please correct me if I'm wrong, but the way I understood it, using the shared encryption scheme creates a conflict between \"changes being synced between them, even if the two computers are never online simultaneously\" and \"encryption should be done locally: the (special) remote should not be able to view file listings or content.\" + +- If I use shared encryption \"the webapp way\", only the file contents will be rsynced to the remote, not the repository itself. This means that different hosts are unable to sync unless they are online simultaneously, so that commit data can be sent directly between them via XMPP. In practice, this would mean my hosts are never synced (because I don't keep my home computer running when I leave for work, and vice versa) + +- If I use shared encryption and additionally put the repository itself on a remote, that remote would have the keys to fully decrypt the repository, that's not acceptable. + +Reading through the docs again, the hybrid scheme actually seems to be closer to what I want than the shared scheme, but it still has a major downside: the encryption only applies to the files itself, so in order to get \"offline sync\" there still has to be a 'remote' for the repository itself, which will contain all your metadata unencrypted. And also it would depend on the user being able to manually setup and backup a set of gpg keys instead of just memorizing a secure passphrase. + +@Fabrice Looks like the bug you found could very well be the cause of the problem I had; I'll try it again when a new version is available. +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_8_49aa34d75d24a2066baa8a585bc9c2e9._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_8_49aa34d75d24a2066baa8a585bc9c2e9._comment new file mode 100644 index 000000000..86f3f5cad --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_8_49aa34d75d24a2066baa8a585bc9c2e9._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkbpbjP5j8MqWt_K4NASwv0WvB8T4rQ-pM" + nickname="Fabrice" + subject="comment 8" + date="2013-11-04T10:31:56Z" + content=""" +I think you are (at least partially) right. Of course, the only way to sync completely computers that are not on together is to use either a usb drive or a third always on computer. (I've to confess I did not understand first when I read git annex docs, shame on me ;-) If you don't want to trust completely this computer (I don't, for instance), you must : + +* use an encrypted git repository on this computer; + +* and use either hybrid or pubkey encryption. + +But contrarily to what you seem to imply (I hope I understand you correctly), if you do that, the third computer can still figure out a few things (usage patterns, such as where connections come from), but that's all. You've got full sync and everything is encrypted, both the git part and the files handled by the annex. This applied only to encrypted git special remotes as other remotes do not store the git part. +"""]] diff --git a/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_9_3784e0c828cd60b6a9075c2d32d070cc._comment b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_9_3784e0c828cd60b6a9075c2d32d070cc._comment new file mode 100644 index 000000000..24e5f5b83 --- /dev/null +++ b/doc/tips/fully_encrypted_git_repositories_with_gcrypt/comment_9_3784e0c828cd60b6a9075c2d32d070cc._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.47" + subject="comment 9" + date="2013-11-04T17:07:55Z" + content=""" +\"We could symetrically encrypt the repository with a keyfile that's stored in the repository itself\" + +Then you would need to decrypt the repository in order get the key you need to decrypt the repository. The impossibility of this design is why I didn't do that! + +It would certainly be possible to store a non-encrypted gpg key alongside the repsitory encrypted with it, but then you have to rely on a passphrase for all your security. + +You should file a bug report for the bug you saw.. +"""]] diff --git a/doc/tips/googledriveannex.mdwn b/doc/tips/googledriveannex.mdwn new file mode 100644 index 000000000..abecf10cf --- /dev/null +++ b/doc/tips/googledriveannex.mdwn @@ -0,0 +1,28 @@ +googledriveannex +========= + +Hook program for gitannex to use Google Drive as backend + +# Requirements: + + python2 + +Credit for the googledrive api interface goes to google + +## Install +Clone the git repository in your home folder. + + git clone git://github.com/TobiasTheViking/googledriveannex.git + +This should make a ~/googledriveannex folder + +## Setup +Run the program once to make an empty config file + + cd ~/googledriveannex; python2 googledriveannex.py + +## Commands for gitannex: + + git config annex.googledrive-hook '/usr/bin/python2 ~/googledriveannex/googledriveannex.py' + git annex initremote googledrive type=hook hooktype=googledrive encryption=shared + git annex describe googledrive "the googledrive library" diff --git a/doc/tips/imapannex.mdwn b/doc/tips/imapannex.mdwn new file mode 100644 index 000000000..e9963df34 --- /dev/null +++ b/doc/tips/imapannex.mdwn @@ -0,0 +1,27 @@ +imapannex +========= + +Hook program for gitannex to use imap as backend + +# Requirements: + + python2 + +# Install +Clone the git repository in your home folder. + + git clone git://github.com/TobiasTheViking/imapannex.git + +This should make a ~/imapannex folder + +# Setup +Run the program once to set it up. + + cd ~/imapannex; python2 imapannex.py + +# Commands for gitannex: + + git config annex.imap-hook '/usr/bin/python2 ~/imapannex/imapannex.py' + git annex initremote imap type=hook hooktype=imap encryption=shared + git annex describe imap "the imap library" + git annex wanted imap exclude=largerthan=30mb diff --git a/doc/tips/megaannex.mdwn b/doc/tips/megaannex.mdwn new file mode 100644 index 000000000..8edbf4421 --- /dev/null +++ b/doc/tips/megaannex.mdwn @@ -0,0 +1,41 @@ +[Megaannex](https://github.com/TobiasTheViking/megaannex) +is a hook program for git-annex to use mega.co.nz as backend + +# Requirements: + + python2 + requests>=0.10 + pycrypto + +Credit for the mega api interface goes to: +<https://github.com/richardasaurus/mega.py> + +## Install + +Clone the git repository in your home folder. + + git clone git://github.com/TobiasTheViking/megaannex.git + +This should make a ~/megannex folder + +## Setup + +Run the program once to make an empty config file. + + cd ~/megaannex; python2 megaannex.py + +Edit the megaannex.conf file. Add your mega.co.nz username, password, and folder name. + +## Configuring git-annex + + git config annex.mega-hook '/usr/bin/python2 ~/megaannex/megaannex.py' + + git annex initremote mega type=hook hooktype=mega encryption=shared + git annex describe mega "the mega.co.nz library" + +## Notes + +You may need to use a different command than "python2", depending +on your python installation. + +-- Tobias diff --git a/doc/tips/migrating_data_to_a_new_backend.mdwn b/doc/tips/migrating_data_to_a_new_backend.mdwn new file mode 100644 index 000000000..b9acb8bd1 --- /dev/null +++ b/doc/tips/migrating_data_to_a_new_backend.mdwn @@ -0,0 +1,16 @@ +Maybe you started out using the WORM backend, and have now configured +git-annex to use SHA1. But files you added to the annex before still +use the WORM backend. There is a simple command that can migrate that +data: + + # git annex migrate my_cool_big_file + migrate my_cool_big_file (checksum...) ok + +You can only migrate files whose content is currently available. Other +files will be skipped. + +After migrating a file to a new backend, the old content in the old backend +will still be present. That is necessary because multiple files +can point to the same content. The `git annex unused` subcommand can be +used to clear up that detritus later. Note that hard links are used, +to avoid wasting disk space. diff --git a/doc/tips/migrating_two_seperate_disconnected_directories_to_git_annex.mdwn b/doc/tips/migrating_two_seperate_disconnected_directories_to_git_annex.mdwn new file mode 100644 index 000000000..1209d1217 --- /dev/null +++ b/doc/tips/migrating_two_seperate_disconnected_directories_to_git_annex.mdwn @@ -0,0 +1,77 @@ +Scenario +-------- + +You are a new git-annex user. You have already files spread around many computers and wish to migrate those into git-annex, without having to recopy all files all over the place. + +Let's say, for example, you have a server, named `marcos` and a workstation named `angela`. You have your audio collection stored in `/srv/mp3` in `marcos` and `~/mp3` on `angela`, but only `marcos` has all the files, and `angela` only has a subset. + +We also assume that `marcos` has an SSH server. + +How do you add all this stuff to git-annex? + +Create the biggest git-annex repository +--------------------------------------- + +Start with `marcos`, with the complete directory: + + cd /srv/mp3 + git init + git annex init + git annex add . + git commit -m"git annex yay" + +This will checksum all files and add them to the `git-annex` branch of the git repository. Wait for this process to complete. + +Create the smaller repo and synchronise +--------------------------------------- + +On `angela`, we want to synchronise the git annex metadata with `marcos`. We need to initialize a git repo with `marcos` as a remote: + + cd ~/mp3 + git init + git remote add marcos marcos.example.com:/srv/mp3 + git fetch marcos + git annex info # this should display the two repos + git annex add . + +This will, again, checksum all files and add them to git annex. Once that is done, you can verify that the files are really the same as marcos with `whereis`: + + git annex whereis + +This should display something like: + + whereis Orange Seeds/I remember.wav (2 copies) + b7802161-c984-4c9f-8d05-787a29c41cfe -- marcos (anarcat@marcos:/srv/mp3) + c2ca4a13-9a5f-461b-a44b-53255ed3e2f9 -- here (anarcat@angela) + ok + +Once you are sure things went on okay, you can synchronise this with `marcos`: + + git annex sync + +This will push the metadata information to marcos, so it knows which files are available on `angela`. From there on, you can freely get and move files between the two repos! + +Importing files from a third directory +-------------------------------------- + +Say that some files on `angela` are actually spread out outside of the `~/mp3` directory. You can use the `git annex import` command to add those extra directories: + + cd ~/mp3 + git annex import ~/music/ + +(!) Be careful that `~/music` is not a git-annex repository, or this will [[destroy it!|bugs/git annex import destroys a fellow git annex repository]]. + +Deleting deleted files +---------------------- + +It is quite possible some files were removed (or renamed!) on `marcos` but not on `angela`, since it was synchronised only some time ago. A good way to find out about those files is to use the `--not --in` argument, for example, on `angela`: + + git annex whereis --in here --not --in marcos + +This will show files that are on `angela` and not on `marcos`. They could be new files that were only added on `angela`, so be careful! A manual analysis is necessary, but let's say you are certain those files are not relevant anymore, you can delete them from `angela`: + + git annex drop <file> + +If the file is a renamed or modified version from the original, you may need to use `--force`, but be careful! If you delete the wrong file, it will be lost forever! + +> (!) Maybe this wouldn't happen with [[direct mode]] and an fsck? --[[anarcat]] diff --git a/doc/tips/offline_archive_drives.mdwn b/doc/tips/offline_archive_drives.mdwn new file mode 100644 index 000000000..eff123e8b --- /dev/null +++ b/doc/tips/offline_archive_drives.mdwn @@ -0,0 +1,69 @@ +After you've used git-annex for a while, you will have data in your repository +that you don't want to keep in the limited disk space of a laptop or a server, +but that you don't want to entirely delete. + +This is where git-annex's support for offline archive drives shines. +You can move old files to an archive drive, which can be kept offline if +it's not practical to keep it spinning. Better, you can move old files to +two or more archive drives, in case one of them later fails to spin up. +(One consideration when [[future_proofing]] your archive.) + +To set up an archive drive, you can take any removable drive, format +it with a filesystem you'll be able to read some years later, and then follow +the [[walkthrough]] to set up a repository on it that is a git remote of +the repository in your computer you want to archive. In short: + + cd /media/archive + git clone ~/annex + cd ~/annex + git remote add archivedrive /media/archive/annex + git annex sync archivedrive + +Don't forget to tell git-annex this is an archive drive (or perhaps a backup +drive). Also, give the drive a description that matches something you write on +its label, so you can find it later: + + git annex group archivedrive archive + git annex wanted archivedrive standard + git annex describe archivedrive "my first archive drive (SATA)" + +Or you can use the assistant to set up the drive for you. +(Nice video tutorial here: [[videos/git-annex_assistant_archiving]]) + +(Keeping the archive drive in an offsite location? Consider encrypting +it! See [[fully_encrypted_git_repositories_with_gcrypt]].) + +Then, when the archive drive is plugged in, you can easily copy files to +it: + + cd ~/annex + git-annex copy --auto --to archivedrive + +Or, if you're using the assistant, it will automatically notice when the drive +gets plugged in and copy files that need to be archived. + +When you want to get rid of the local file, leaving only the copy on the +archive, you can just: + + git annex drop file + +The archive drive has to be plugged in for this to work, so git-annex +can verify it still has the file. If you had configured git-annex to +always store 2 [[copies]], it will need 2 archive drives plugged in. +You may find it useful to configure a [[trust]] setting for the drive to +avoid needing to haul it out of storage to drop a file. + +Now the really nice thing. When your archive drive gets filled up, you +can simply remove it, store it somewhere safe, and replace it with a new +drive, which can be mounted at the same location for simplicity. Set up +the new drive the same way described above, and use it to archive even more +files. + +Finally, when you want to access one of the files you archived, you can +just ask for it: + + git annex get file + +If necessary git-annex will tell you which archive drive you need to +pull out of storage to get the file back. This is where the description +you entered earlier comes in handy. diff --git a/doc/tips/offline_archive_drives/comment_1_3d4fdf42191a9d81434d00f51c3609ff._comment b/doc/tips/offline_archive_drives/comment_1_3d4fdf42191a9d81434d00f51c3609ff._comment new file mode 100644 index 000000000..5855b0f7a --- /dev/null +++ b/doc/tips/offline_archive_drives/comment_1_3d4fdf42191a9d81434d00f51c3609ff._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkC0W3ZQERUaTkHoks6k68Tsp1tz510nGo" + nickname="Georg" + subject="git annex sync" + date="2013-10-06T08:59:24Z" + content=""" +Shouldn't it be +git annex sync archivedrive +instead of +git annex sync archive +in the first examples. As the name of the remote is \"archivedrive\", IMO \"sync\" should be called with the name of the remote. +"""]] diff --git a/doc/tips/offline_archive_drives/comment_2_864c752aa8d064791a4b14dbbe2e6882._comment b/doc/tips/offline_archive_drives/comment_2_864c752aa8d064791a4b14dbbe2e6882._comment new file mode 100644 index 000000000..3ff074ff3 --- /dev/null +++ b/doc/tips/offline_archive_drives/comment_2_864c752aa8d064791a4b14dbbe2e6882._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkC0W3ZQERUaTkHoks6k68Tsp1tz510nGo" + nickname="Georg" + subject="git annex copy not working" + date="2013-10-06T10:18:10Z" + content=""" +Is the feature \"git-annex copy --auto --to usb\" working? +I have created a backup repo on my usb drive but when I call \"git-annex copy --auto --to usb\" nothing happens. + +I have called \"git annex group usb backup\" and \"git annex sync usb\" to set up the repo. + +What is the correct way to get the data out to the backup repo? + +Best regards, Georg +"""]] diff --git a/doc/tips/offline_archive_drives/comment_3_7be2ccaf70c9ecfc9a34384e0e31f490._comment b/doc/tips/offline_archive_drives/comment_3_7be2ccaf70c9ecfc9a34384e0e31f490._comment new file mode 100644 index 000000000..cd5888009 --- /dev/null +++ b/doc/tips/offline_archive_drives/comment_3_7be2ccaf70c9ecfc9a34384e0e31f490._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.243" + subject="thanks for your checking.." + date="2013-10-06T17:04:24Z" + content=""" +The example was missing a preferred content setting, without which --auto doesn't copy anything unless needed to satisfy numcopies: + +git annex wanted archivedrive standard +"""]] diff --git a/doc/tips/owncloudannex.mdwn b/doc/tips/owncloudannex.mdwn new file mode 100644 index 000000000..ad40c67e3 --- /dev/null +++ b/doc/tips/owncloudannex.mdwn @@ -0,0 +1,28 @@ +owncloudannex +========= + +Hook program for gitannex to use owncloud as backend + +# Requirements: + + python2 + +Credit for the Owncloud api interface goes to me. + +# Install +Clone the git repository in your home folder. + + git clone git://github.com/TobiasTheViking/owncloudannex.git + +This should make a ~/owncloudannex folder + +# Setup +Run the program once to set it up. + + cd ~/owncloudannex; python2 owncloudannex.py + +# Commands for gitannex: + + git config annex.owncloud-hook '/usr/bin/python2 ~/owncloudannex/owncloudannex.py' + git annex initremote owncloud type=hook hooktype=owncloud encryption=shared + git annex describe owncloud "the owncloud library" diff --git a/doc/tips/owncloudannex/comment_1_129652308c3c499462828dcaf8e747a4._comment b/doc/tips/owncloudannex/comment_1_129652308c3c499462828dcaf8e747a4._comment new file mode 100644 index 000000000..47e33042e --- /dev/null +++ b/doc/tips/owncloudannex/comment_1_129652308c3c499462828dcaf8e747a4._comment @@ -0,0 +1,40 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkQvRLq7nMMEGoEKuYx9oaf67IC0nZfmVI" + nickname="chung yan" + subject="owncloud hook exited nonzero" + date="2013-07-10T05:03:47Z" + content=""" +hi + +I got the above message, and cannot sync to my owncloud. Here is the log file i had: + +>[2013-07-10 12:30:31 HKT] main: starting assistant version 4.20130627 +(scanning...) [2013-07-10 12:30:31 HKT] Watcher: Performing startup scan +(started...) git-annex: Daemon is already running. +git-annex: Daemon is already running. +[2013-07-10 12:44:04 HKT] Committer: Adding sync2Servers.sh + +>add syncByRsync/sync2Servers.sh (checksum...) [2013-07-10 12:44:04 HKT] Committer: Committing changes to git +(gpg) + +>owncloud hook exited nonzero! +>git-annex: Daemon is already running. +[2013-07-10 12:51:07 HKT] main: Syncing with owncloud + +>owncloud hook exited nonzero! +>[2013-07-10 12:53:10 HKT] Committer: Adding 2 files +ok +(Recording state in git...) +(Recording state in git...) +add syncByRsync/DSCN1810.JPG (checksum...) ok +add syncByRsync/DSCN1810.JPG (checksum...) [2013-07-10 12:53:10 HKT] Committer: Committing changes to git + +>owncloud hook exited nonzero! +[2013-07-10 12:53:50 HKT] main: Syncing with owncloud + owncloud hook exited nonzero! + owncloud hook exited nonzero! + +thanks + +yan +"""]] diff --git a/doc/tips/owncloudannex/comment_2_38604990368666f654d41891ba99ac61._comment b/doc/tips/owncloudannex/comment_2_38604990368666f654d41891ba99ac61._comment new file mode 100644 index 000000000..6ea0b033c --- /dev/null +++ b/doc/tips/owncloudannex/comment_2_38604990368666f654d41891ba99ac61._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0" + nickname="Tobias" + subject="comment 2" + date="2013-07-10T08:21:39Z" + content=""" +Personally i've only seen that when the server ran out of space. But lets see what is going on. + +Please run this command + +git config annex.owncloud-hook '/usr/bin/python2 ~/owncloudannex/owncloudannex.py --dbglevel 1 --stderr' + +And then replicate the error. It should give me some debug information to work with. + +"""]] diff --git a/doc/tips/owncloudannex/comment_3_1bfd290d00d6536da7d31818db46f8ec._comment b/doc/tips/owncloudannex/comment_3_1bfd290d00d6536da7d31818db46f8ec._comment new file mode 100644 index 000000000..f7e3f0ee4 --- /dev/null +++ b/doc/tips/owncloudannex/comment_3_1bfd290d00d6536da7d31818db46f8ec._comment @@ -0,0 +1,87 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkQvRLq7nMMEGoEKuYx9oaf67IC0nZfmVI" + nickname="chung yan" + subject="comment 3" + date="2013-07-10T08:46:27Z" + content=""" +hi Tobias + +Thanks your suggestion for list of my error log, here it is: + +[2013-07-10 16:27:58 HKT] main: starting assistant version 4.20130627 +(scanning...) [2013-07-10 16:27:58 HKT] Watcher: Performing startup scan +(started...) [2013-07-10 16:28:14 HKT] Committer: Adding 2 files + +add syncByRsync/sync2Servers.sh (checksum...) ok +add syncByRsync/sync2Servers.sh (checksum...) [2013-07-10 16:28:14 HKT] Committer: Committing changes to git +[2013-07-10 16:31:12 HKT] Watcher: add direct DSCN1810.JPG +[2013-07-10 16:31:12 HKT] read: lsof [\"-F0can\",\"+d\",\"/home/yan/annex_testing/.git/annex/tmp/\"] +[2013-07-10 16:31:12 HKT] Committer: Adding DSCN1810.JPG +ok +(Recording state in git...) +(Recording state in git...) +add DSCN1810.JPG [2013-07-10 16:31:12 HKT](checksum...) Watcher: add direct DSCN1810.JPG +[2013-07-10 16:31:12 HKT] read: sha256sum [\"/home/yan/annex_testing/.git/annex/tmp/DSCN181012023.JPG\"] + + DSCN1810.JPG changed while it was being added +[2013-07-10 16:31:12 HKT] Committer: delaying commit of 1 changes +[2013-07-10 16:31:13 HKT] read: lsof [\"-F0can\",\"+d\",\"/home/yan/annex_testing/.git/annex/tmp/\"] +[2013-07-10 16:31:13 HKT] Committer: Adding 2 files +failed +add DSCN1810.JPG (checksum...) [2013-07-10 16:31:13 HKT] read: sha256sum [\"/home/yan/annex_testing/.git/annex/tmp/DSCN181012023.JPG\"] +[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"hash-object\",\"-t\",\"blob\",\"-w\",\"--stdin\",\"--no-filters\"] +ok +add DSCN1810.JPG (checksum...) [2013-07-10 16:31:13 HKT] read: sha256sum [\"/home/yan/annex_testing/.git/annex/tmp/DSCN181012024.JPG\"] +[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"hash-object\",\"-t\",\"blob\",\"-w\",\"--stdin\",\"--no-filters\"] +[2013-07-10 16:31:13 HKT] Committer: committing 2 changes +[2013-07-10 16:31:13 HKT] Committer: Committing changes to git +[2013-07-10 16:31:13 HKT] feed: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"update-index\",\"-z\",\"--index-info\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"commit\",\"--allow-empty-message\",\"--no-edit\",\"-m\",\"\",\"--quiet\",\"--no-verify\"] +[2013-07-10 16:31:13 HKT] Committer: queued Upload UUID \"df02d32a-7e3a-4e12-a417-7f1d1a1cf1a6\" DSCN1810.JPG Nothing : new file created +[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"] +[2013-07-10 16:31:13 HKT] feed: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"update-index\",\"-z\",\"--index-info\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"write-tree\"] +[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"commit-tree\",\"a8337a989b58b29eee5fd2fa7a4c0b8ec45d5e59\",\"-p\",\"refs/heads/git-annex\"] +[2013-07-10 16:31:13 HKT] call: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"update-ref\",\"refs/heads/git-annex\",\"267bf35da4d9abe5ed7fe82ea5df8a8df2ddf940\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"symbolic-ref\",\"HEAD\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"refs/heads/master\"] +[2013-07-10 16:31:13 HKT] Transferrer: Transferring: Upload UUID \"df02d32a-7e3a-4e12-a417-7f1d1a1cf1a6\" DSCN1810.JPG Nothing +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"git-annex\"] +[2013-07-10 16:31:13 HKT] call: git-annex [\"transferkeys\",\"--readfd\",\"46\",\"--writefd\",\"36\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"log\",\"refs/heads/git-annex..267bf35da4d9abe5ed7fe82ea5df8a8df2ddf940\",\"--oneline\",\"-n1\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"git-annex\"] +[2013-07-10 16:31:13 HKT] read:[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"ls-tree\",\"-z\",\"--\",\"refs/heads/git-annex\",\"uuid.log\",\"remote.log\",\"trust.log\",\"group.log\",\"preferred-content.log\"] + git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] +[2013-07-10 16:31:13 HKT] read: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"log\",\"refs/heads/git-annex..267bf35da4d9abe5ed7fe82ea5df8a8df2ddf940\",\"--oneline\",\"-n1\"] +[2013-07-10 16:31:13 HKT] chat: git [\"--git-dir=/home/yan/annex_testing/.git\",\"--work-tree=/home/yan/annex_testing\",\"cat-file\",\"--batch\"] +(gpg) [2013-07-10 16:31:13 HKT] TransferWatcher: transfer starting: Upload UUID \"df02d32a-7e3a-4e12-a417-7f1d1a1cf1a6\" DSCN1810.JPG Nothing +[2013-07-10 16:31:13 HKT] chat: gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--quiet\",\"--trust-model\",\"always\",\"--batch\",\"--passphrase-fd\",\"48\",\"--symmetric\",\"--force-mdc\"] +[2013-07-10 16:31:14 HKT] call: sh [\"-c\",\"/usr/bin/python2 /home/yan/annex/SocialBusiness/Resources/Infrastructure/Computer_Tools/Records/AsusU24/owncloudannex/owncloudannex.py --dbglevel 1 --stderr\"] + + 16:31:16 [owncloudannex-0.1.1] <module> : 'START' + 16:31:16 [owncloudannex-0.1.1] main : 'ARGS: 'ANNEX_ACTION=store ANNEX_KEY=GPGHMACSHA1--0179fbbf559d5c25cf69b4e92025ff1f007d4f1f ANNEX_HASH_1=fp ANNEX_HASH_2=23 ANNEX_FILE=/home/yan/annex_testing/.git/annex/tmp/GPGHMACSHA1--0179fbbf559d5c25cf69b4e92025ff1f007d4f1f /home/yan/annex/SocialBusiness/Resources/Infrastructure/Computer_Tools/Records/AsusU24/owncloudannex/owncloudannex.py --dbglevel 1 --stderr'' + 16:31:16 [owncloudannex-0.1.1] readFile : ''/home/yan/annex/SocialBusiness/Resources/Infrastructure/Computer_Tools/Records/AsusU24/owncloudannex/owncloudannex.conf' - 'r'' + 16:31:16 [owncloudannex-0.1.1] readFile : 'Done' + 16:31:16 [owncloudannex-0.1.1] login : '' + 16:31:16 [owncloudannex-0.1.1] login : 'Using base: wingyan.no-ip.org - {'Authorization': 'Basic Y2h1bmd5YW41QGdtYWlsLmNvbTp5YW5fd2lraQ=='}' + 16:31:16 [owncloudannex-0.1.1] login : 'res: <davlib.DAV instance at 0x12915f0>' + 16:31:16 [owncloudannex-0.1.1] findInFolder : 'u'gitannex'(<type 'unicode'>) - '/'(<type 'str'>)' + 16:31:17 [owncloudannex-0.1.1] findInFolder : 'propfind: /owncloud/remote.php/webdav - '<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n <s:exception>Sabre_DAV_Exception_NotAuthenticated</s:exception>\n <s:message>Username or password does not match</s:message>\n <s:sabredav-version>1.7.6</s:sabredav-version>\n</d:error>\n'' + 16:31:17 [owncloudannex-0.1.1] findInFolder : 'Failure' + 16:31:17 [owncloudannex-0.1.1] createFolder : '/gitannex' + 16:31:18 [owncloudannex-0.1.1] createFolder : 'Failure: 401 - '<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n <s:exception>Sabre_DAV_Exception_NotAuthenticated</s:exception>\n <s:message>Username or password does not match</s:message>\n <s:sabredav-version>1.7.6</s:sabredav-version>\n</d:error>\n'' + + + owncloud hook exited nonzero! +[2013-07-10 16:31:18 HKT] TransferWatcher: transfer finishing: Transfer {transferDirection = Upload, transferUUID = UUID \"df02d32a-7e3a-4e12-a417-7f1d1a1cf1a6\", transferKey = Key {keyName = \"911ba6148fbcbe4afe53772f1216b8204f403ed4ee06cb90c3c3ac25e56d9402.JPG\", keyBackendName = \"SHA256E\", keySize = Just 1893431, keyMtime = Nothing}} + +I figured out it is a *Username or password does not match*, and i saw the content of owncloudannex.conf as \"uname\": \"myGmail@gmail.com\", but i quickly saw my WebDAV login to owncloud as my user name, not gmail address, so i changed this \"uname\": \"my_normal_user_name_not_gmail_acc\" inside owncloudannex.conf. Finally, i got it work. So, i think, user name should not be a gmail address, should be owncloud login user name. + +Another issue, i had a look into /WebDAV.../gitannex/, it is git repos. file, for my user opinion, it is better that it is a real file content that we can see the file(such as photos) by owncloud web client directly, rather that owncloud is a file server to keep git repos only. + +Thanks + +yan +"""]] diff --git a/doc/tips/owncloudannex/comment_4_492b6922a7c5bb5464fedb46b0c5303b._comment b/doc/tips/owncloudannex/comment_4_492b6922a7c5bb5464fedb46b0c5303b._comment new file mode 100644 index 000000000..c7a0875df --- /dev/null +++ b/doc/tips/owncloudannex/comment_4_492b6922a7c5bb5464fedb46b0c5303b._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0" + nickname="Tobias" + subject="comment 4" + date="2013-07-10T08:54:13Z" + content=""" +Hm, an error like that it really should print without the need of debug information. I'll look into it. + +And if you want the real file content, not encrypted, just change \"shared\" to \"none\" in the line: + +git annex initremote owncloud type=hook hooktype=owncloud encryption=shared + +I'm not sure if you can change this after you initialized the remote though. And also, the folder structure and filenames will be as you see them now on the owncloud. I'd need some more data from gitannex to make it show the real directory structure. But that doesn't seem feasible. + +Why would you even want the data unencrypted on owncloud anyways? i mean on flickr, or googledocs, i kinda get it. but for owncloud? + +"""]] diff --git a/doc/tips/owncloudannex/comment_5_1d48ac08714fadcb06d874570d745bd8._comment b/doc/tips/owncloudannex/comment_5_1d48ac08714fadcb06d874570d745bd8._comment new file mode 100644 index 000000000..170e8b857 --- /dev/null +++ b/doc/tips/owncloudannex/comment_5_1d48ac08714fadcb06d874570d745bd8._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkQvRLq7nMMEGoEKuYx9oaf67IC0nZfmVI" + nickname="chung yan" + subject="comment 5" + date="2013-07-10T09:43:55Z" + content=""" +hi Tobias + +thanks your sharing, i am still in have a trial of git-annex, so i do not yet have real data on the server, so i just del. all and re-create both client and server repos in owncloud, i can got what you said, and see my photo. + +Actually, i am studying git-annex. 1) It can syncs the data from my client computer to server. 2) i would like that other and anywhere computers through browser can view my server data which do not need to download all data into a new client. So my original goal is i can have a web to display the content of my files, and see owncloud have a nice web. + +Regarding to GDoc, flickr, etc. they have space limitation, so i install owncloud as my own computer. I also see the <http://git-annex.branchable.com/design/assistant/partial_content/>, but it is not yet at roadmap. + +thanks +"""]] diff --git a/doc/tips/owncloudannex/comment_6_65959f49a2f56bffd6fe48670c0c8d5a._comment b/doc/tips/owncloudannex/comment_6_65959f49a2f56bffd6fe48670c0c8d5a._comment new file mode 100644 index 000000000..7b10068c5 --- /dev/null +++ b/doc/tips/owncloudannex/comment_6_65959f49a2f56bffd6fe48670c0c8d5a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0" + nickname="Tobias" + subject="comment 6" + date="2013-07-10T09:50:14Z" + content=""" +A 1 terabyte limit(for flickr) should be enough for most people. No? +"""]] diff --git a/doc/tips/owncloudannex/comment_7_7482002991672ef67836bae43b8d0be8._comment b/doc/tips/owncloudannex/comment_7_7482002991672ef67836bae43b8d0be8._comment new file mode 100644 index 000000000..56e71276d --- /dev/null +++ b/doc/tips/owncloudannex/comment_7_7482002991672ef67836bae43b8d0be8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkQvRLq7nMMEGoEKuYx9oaf67IC0nZfmVI" + nickname="chung yan" + subject="comment 7" + date="2013-07-10T09:56:54Z" + content=""" +i have not got this 1 terabyte limit from flick before. I will have a look. I still consider some my own data, it is better to keep at my own server. +"""]] diff --git a/doc/tips/powerful_file_matching.mdwn b/doc/tips/powerful_file_matching.mdwn new file mode 100644 index 000000000..47f8c8a64 --- /dev/null +++ b/doc/tips/powerful_file_matching.mdwn @@ -0,0 +1,36 @@ +git-annex has a powerful syntax for making it act on only certain files. + +The simplest thing is to exclude some files, using wild cards: + + git annex get --exclude '*.mp3' --exclude '*.ogg' + +But you can also exclude files that git-annex's [[location_tracking]] +information indicates are present in a given repository. For example, +if you want to populate newarchive with files, but not those already +on oldarchive, you could do it like this: + + git annex copy --not --in oldarchive --to newarchive + +Without the --not, --in makes it act on files that *are* in the specified +repository. So, to remove files that are on oldarchive: + + git annex drop --in oldarchive + +Or maybe you're curious which files have a lot of copies, and then +also want to know which files have only one copy: + + git annex find --copies 7 + git annex find --not --copies 2 + +The above are the simple examples of specifying what files git-annex +should act on. But you can specify anything you can dream up by combining +the things above, with --and --or -( and -). Those last two strange-looking +options are parentheses, for grouping other options. You will probably +have to escape them from your shell. + +Here are the mp3 files that are in either of two repositories, but have +less than 3 copies: + + git annex find --not --exclude '*.mp3' --and \ + -\( --in usbdrive --or --in archive -\) --and \ + --not --copies 3 diff --git a/doc/tips/recover_data_from_lost+found.mdwn b/doc/tips/recover_data_from_lost+found.mdwn new file mode 100644 index 000000000..48ef2a1d7 --- /dev/null +++ b/doc/tips/recover_data_from_lost+found.mdwn @@ -0,0 +1,19 @@ +Suppose something goes wrong, and fsck puts all the files in lost+found. +It's actually very easy to recover from this disaster. + +First, check out the git repository again. Then, in the new checkout: + + $ mkdir recovered-content + $ sudo mv ../lost+found/* recovered-content + $ sudo chown you:you recovered-content + $ chmod -R u+w recovered-content + $ git annex add recovered-content + $ git rm recovered-content + $ git commit -m "recovered some content" + $ git annex fsck + +The way that works is that when git-annex adds the same content that was in +the repository before, all the old links to that content start working +again. This works particularly well if the SHA* backends are used, but even +with the default backend it will work pretty well, as long as fsck +preserved the modification time of the files. diff --git a/doc/tips/recovering_from_a_corrupt_git_repository.mdwn b/doc/tips/recovering_from_a_corrupt_git_repository.mdwn new file mode 100644 index 000000000..084f76852 --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository.mdwn @@ -0,0 +1,17 @@ +I have found this the most reliable way to recover from a corrupt git repository. I have had a lot of them lately, there might be a regression in btrfs in Ubuntu's Linux 3.8.0-33 (!). + +1. Create a clone of a known good repository. +2. Add the clone as an object alternate to the broken repository. +3. Do a `git-repack -a -d` to lift the external objects into repo-local packs. +4. Remove the clone + +[[!format sh """ +$ cd /tmp/ +$ git clone good-host:/path/to/good-repo +$ cd /home/user/broken-repo +$ echo /tmp/good-repo/.git/objects/ > .git/objects/info/alternates +$ git repack -a -d +$ rm -rf /tmp/good-repo +"""]] + +... and push early, push often. ;-) diff --git a/doc/tips/recovering_from_a_corrupt_git_repository/comment_1_f5827be97f78dbae113a5ba0c9ced896._comment b/doc/tips/recovering_from_a_corrupt_git_repository/comment_1_f5827be97f78dbae113a5ba0c9ced896._comment new file mode 100644 index 000000000..d212e23f3 --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository/comment_1_f5827be97f78dbae113a5ba0c9ced896._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.246" + subject="comment 1" + date="2013-11-11T16:01:59Z" + content=""" +Better, since version 4.20131024, `git annex repair` can be run to automatically do this, and more. (Including recovering data in corrupt git repositories that you forgot to push!) +"""]] diff --git a/doc/tips/recovering_from_a_corrupt_git_repository/comment_2_e98df7275bb032308bb87e3607bdde32._comment b/doc/tips/recovering_from_a_corrupt_git_repository/comment_2_e98df7275bb032308bb87e3607bdde32._comment new file mode 100644 index 000000000..c018c4ca5 --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository/comment_2_e98df7275bb032308bb87e3607bdde32._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://id.clacke.se/" + nickname="clacke" + subject="comment 2" + date="2013-11-12T16:14:50Z" + content=""" +I am using git-annex 20131106 and I tried `git annex repair` first. It seems git is making assumptions that if I have object A then I must have object B that A depends on. Or maybe it freaks out because the object is not missing, just full of zeroes. I haven't done any analysis on exactly what situation causes this. When I have time, I will. +"""]] diff --git a/doc/tips/recovering_from_a_corrupt_git_repository/comment_3_11bece6dfac090edbcd783b266c482a3._comment b/doc/tips/recovering_from_a_corrupt_git_repository/comment_3_11bece6dfac090edbcd783b266c482a3._comment new file mode 100644 index 000000000..63f443a31 --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository/comment_3_11bece6dfac090edbcd783b266c482a3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://id.clacke.se/" + nickname="clacke" + subject="comment 3" + date="2013-11-12T16:17:42Z" + content=""" +Luckily, so far the objects getting corrupted have been in past commits (maybe only in packfiles?), not the latest ones, so I have been able to recover even though HEAD has been unique to the local repo. +"""]] diff --git a/doc/tips/recovering_from_a_corrupt_git_repository/comment_4_86e99017f7585ac2f76753051214637e._comment b/doc/tips/recovering_from_a_corrupt_git_repository/comment_4_86e99017f7585ac2f76753051214637e._comment new file mode 100644 index 000000000..8d454fc3e --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository/comment_4_86e99017f7585ac2f76753051214637e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://id.clacke.se/" + nickname="clacke" + subject="comment 4" + date="2013-11-12T17:10:06Z" + content=""" +Hm. This is bad. My hunch that btrfs was the culprit seems to have been wrong. After having switched things around, along with lots of `git annex sync` in various places, I now have a corrupt repo on ext4. It must be git or git-annex that does something wrong. +"""]] diff --git a/doc/tips/recovering_from_a_corrupt_git_repository/comment_6_c8953732ce353cdf0c4fb81ddc98c04a._comment b/doc/tips/recovering_from_a_corrupt_git_repository/comment_6_c8953732ce353cdf0c4fb81ddc98c04a._comment new file mode 100644 index 000000000..eeb3ceef2 --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository/comment_6_c8953732ce353cdf0c4fb81ddc98c04a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.246" + subject="comment 6" + date="2013-11-12T18:10:21Z" + content=""" +If you can reliably corrupt a git repository, it's highly likely your hardware (disk or memory) is broken. +"""]] diff --git a/doc/tips/recovering_from_a_corrupt_git_repository/comment_6_d0da84df0241dc6ccf0aa0c7598917df._comment b/doc/tips/recovering_from_a_corrupt_git_repository/comment_6_d0da84df0241dc6ccf0aa0c7598917df._comment new file mode 100644 index 000000000..16b2329b9 --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository/comment_6_d0da84df0241dc6ccf0aa0c7598917df._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://id.clacke.se/" + nickname="clacke" + subject="comment 6" + date="2013-11-12T17:22:01Z" + content=""" +This is my symptom too: [[forum/Git annex 'corrupting' itself]] +"""]] diff --git a/doc/tips/recovering_from_a_corrupt_git_repository/comment_7_addf49556e4c33d55a41c393f519d0a4._comment b/doc/tips/recovering_from_a_corrupt_git_repository/comment_7_addf49556e4c33d55a41c393f519d0a4._comment new file mode 100644 index 000000000..5047bcca4 --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository/comment_7_addf49556e4c33d55a41c393f519d0a4._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.246" + subject="comment 7" + date="2013-11-12T18:12:49Z" + content=""" +> I tried git annex repair first. It seems git is making assumptions that if I have object A then I must have object B that A depends on. Or maybe it freaks out because the object is not missing + +`git annex repair` is supposed to deal with these situations. If it fails to fix such a broken repository, please file a detailed bug report, ideally with a link to a copy of the repository. +"""]] diff --git a/doc/tips/recovering_from_a_corrupt_git_repository/comment_8_505a2fc2b841fe8eb419801f923ef35f._comment b/doc/tips/recovering_from_a_corrupt_git_repository/comment_8_505a2fc2b841fe8eb419801f923ef35f._comment new file mode 100644 index 000000000..78144b893 --- /dev/null +++ b/doc/tips/recovering_from_a_corrupt_git_repository/comment_8_505a2fc2b841fe8eb419801f923ef35f._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://id.clacke.se/" + nickname="clacke" + subject="comment 8" + date="2013-11-14T10:00:20Z" + content=""" +Several machines started showing this behavior around 4.20131106 or 4.20131101. I will find a way to reproduce when I can find the time. +"""]] diff --git a/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant.mdwn b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant.mdwn new file mode 100644 index 000000000..4363dc85d --- /dev/null +++ b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant.mdwn @@ -0,0 +1,41 @@ +Sparkleshare and dvcs-autosync are tools to automatically commit your +changes to git and keep them in sync with other repositories. Unlike +git-annex, they don't store the file content on the side, but directly in +the git repository. Great for small files, less good for big files. + +Here's how to use the [[git-annex assistant|/assistant]] to do the same +thing, but even better! + +---- + +First, get git-annex version 4.20130329 or newer. + +---- + +Let's suppose you're delveloping a video game, written in C. You have +source code, and some large game assets. You want to ensure the source +code is stored in git -- that's what git's for! And you want to store +the game assets in the git annex -- to avod bloating your git repos with +possibly enormous files, but still version control them. + +All you need to do is configure git-annex to treat your C files +as small files. And treat any file larger than, say, 100kb as a large +file that is stored in the annex. + + git config annex.largefiles "largerthan=100kb and not (include=*.c or include=*.h)" + +Now if you run `git annex add`, it will only add the large files to the +annex. You can `git add` the small files directly to git. + +Better, if you run `git annex assistant`, it will *automatically* +add the large files to the annex, and store the small files in git. +It'll notice every time you modify a file, and immediately commit it, +too. And sync it out to other repositories you configure using `git annex +webapp`. + +---- + +It's also possible to disable the use of the annex entirely, and just +have the assistant *always* put every file into git, no matter its size: + + git config annex.largefiles "exclude=*" diff --git a/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_1_907e4032ca4a39adb846cf16dbf447dc._comment b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_1_907e4032ca4a39adb846cf16dbf447dc._comment new file mode 100644 index 000000000..b0ff0114a --- /dev/null +++ b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_1_907e4032ca4a39adb846cf16dbf447dc._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://hands.com/~phil/" + nickname="hands" + subject="software from the future?" + date="2013-03-31T17:30:34Z" + content=""" +I think you probably meant at least version 4.20130323 ;-) +"""]] diff --git a/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_2_902d001ba86657ef0f8cca5b175f99ca._comment b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_2_902d001ba86657ef0f8cca5b175f99ca._comment new file mode 100644 index 000000000..2367c938d --- /dev/null +++ b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_2_902d001ba86657ef0f8cca5b175f99ca._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 2" + date="2013-03-31T18:50:35Z" + content=""" +I meant 4.20130329 +"""]] diff --git a/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_3_a1cf93f9b29658f0f26e9e0ae6057ee3._comment b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_3_a1cf93f9b29658f0f26e9e0ae6057ee3._comment new file mode 100644 index 000000000..91122360a --- /dev/null +++ b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_3_a1cf93f9b29658f0f26e9e0ae6057ee3._comment @@ -0,0 +1,60 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo" + nickname="Tobias" + subject="Trying this feature" + date="2013-04-14T13:04:55Z" + content=""" +I just gave this feature a try, but it seems it doesn't work as expected or maybe I don't understand it: + + ~/annex/largefilestest % git init + ~/annex/largefilestest (git)-[master] % git annex init \"test repo\" + ~/annex/largefilestest (git)-[master] % git config annex.largefiles \"not include=*.txt\" + +Now I copy two files to this directory and add both to the annex + + ~/annex/largefilestest (git)-[master] % ll + total 100 + -rw-rw-r-- 1 tobru tobru 93709 Oct 19 16:14 dpkg-get-selections.txt + -rw-rw-r-- 1 tobru tobru 7256 Jan 6 15:52 x3400.jpg + ~/annex/largefilestest (git)-[master] % git annex add . + add x3400.jpg (checksum...) ok + (Recording state in git...) + ~/annex/largefilestest (git)-[master] % git status + # On branch master + # + # Initial commit + # + # Changes to be committed: + # (use \"git rm --cached <file>...\" to unstage) + # + # new file: x3400.jpg + # + # Untracked files: + # (use \"git add <file>...\" to include in what will be committed) + # + # dpkg-get-selections.txt + ~/annex/largefilestest (git)-[master] % ll + total 96 + -rw-rw-r-- 1 tobru tobru 93709 Oct 19 16:14 dpkg-get-selections.txt + lrwxrwxrwx 1 tobru tobru 192 Jan 6 15:52 x3400.jpg -> .git/annex/objects/vf/QX/SHA256E-s7256--60e5b69ade5619e37f7fcaa964626da9c415959d861241aa13e2516fffc2dddf.jpg/SHA256E-s7256--60e5b69ade5619e37f7fcaa964626da9c415959d861241aa13e2516fffc2dddf.jpg + +So the picture is added to the annex as expected. But the .txt file is not added to git. Do I have to manually add this to git? And why is the picture seen as new file by git? + +The second question could be answered by: \"run git annex sync\". Is this correct? Because after running this command, git does not see this file as a new file anymore: + + ~/annex/largefilestest (git)-[master] % git annex sync + commit + [master (root-commit) a0afb14] git-annex automatic sync + 1 file changed, 1 insertion(+) + create mode 120000 x3400.jpg + ok + git-annex: no branch is checked out + ~/annex/largefilestest (git)-[master] % git status + # On branch master + # Untracked files: + # (use \"git add <file>...\" to include in what will be committed) + # + # dpkg-get-selections.txt + nothing added to commit but untracked files present (use \"git add\" to track) + +"""]] diff --git a/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_4_e10671908b58c554375787d0f76e2366._comment b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_4_e10671908b58c554375787d0f76e2366._comment new file mode 100644 index 000000000..14a909014 --- /dev/null +++ b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_4_e10671908b58c554375787d0f76e2366._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 4" + date="2013-04-14T18:37:50Z" + content=""" +Like it says in the tip, `git annex add` will add the large files to git. You can add the small files with `git add`; git-annex won't do that for you. + +To automatically add both sorts of files, you can use the `git annex watch` or `git annex assistant` daemons. The latter also keeps files in sync between repositories automatically. + +(Why did the picture show up as a new file in git? Because you hadn't committed it. This is the same as when you `git add` a file; +it's only staged in the index; `git status` will show it is new until you `git commit`) +"""]] diff --git a/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_5_4114380f66b6376c851e93f6876d590b._comment b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_5_4114380f66b6376c851e93f6876d590b._comment new file mode 100644 index 000000000..5d638a3b8 --- /dev/null +++ b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_5_4114380f66b6376c851e93f6876d590b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo" + nickname="Tobias" + subject="mimetypes" + date="2013-05-01T20:37:33Z" + content=""" +Does `annex.largefiles` support mimetypes? F.e. `git config annex.largefiles \"not mimetype=text/plain\"` +"""]] diff --git a/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_6_6a5d6af107b297afd008b021f73d787b._comment b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_6_6a5d6af107b297afd008b021f73d787b._comment new file mode 100644 index 000000000..bd2212ffb --- /dev/null +++ b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_6_6a5d6af107b297afd008b021f73d787b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnPOttrEmm9CQYxzWrmgGN7LXy98gDkrlM" + nickname="binet" + subject="annex.largefiles and direct mode" + date="2013-09-16T22:50:48Z" + content=""" +I was wondering if the annex.largefiles feature was compatible with direct mode? +"""]] diff --git a/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_7_74d57cf503a86d8f7ace2d769dbb58be._comment b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_7_74d57cf503a86d8f7ace2d769dbb58be._comment new file mode 100644 index 000000000..2144b0ec8 --- /dev/null +++ b/doc/tips/replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant/comment_7_74d57cf503a86d8f7ace2d769dbb58be._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 7" + date="2013-09-19T18:03:29Z" + content=""" +annex.largefiles does not support mime types. I agree it would be a useful addition. + +annex.largefiles can be used with direct mode. I would only recommending using it this way using the assistant, which will keep straight which files are which and commit them appropriately. +"""]] diff --git a/doc/tips/setup_a_public_repository_on_a_web_site.mdwn b/doc/tips/setup_a_public_repository_on_a_web_site.mdwn new file mode 100644 index 000000000..8251cabf6 --- /dev/null +++ b/doc/tips/setup_a_public_repository_on_a_web_site.mdwn @@ -0,0 +1,55 @@ +Let's say you want to distribute some big files to the whole world. +You can of course, just drop them onto a website. But perhaps you'd like to +use git-annex to manage those files. And as an added bonus, why not let +anyone in the world clone your site and use `git-annex get`! + +My site like this is [downloads.kitenet.net](https://downloads.kitenet.net). +Here's how I set it up. --[[Joey]] + +1. Set up a web site. I used Apache, and configured it to follow symlinks. + `Options FollowSymLinks` +2. Put some files on the website. Make sure it works. +3. `git init; git annex init` +4. `git config core.sharedrepository world` (Makes sure files + are always added with permissions that allow everyone to read them.) +5. We want users to be able to clone the git repository over http, because + git-annex can download files from it over http as well. For this to + work, `git update-server-info` needs to get run after commits. The + git `post-update` hook will take care of this, you just need to enable + the hook. `chmod +x .git/hooks/post-update` +6. `git annex add; git commit -m added` +7. Make sure users can still download files from the site directly. +8. Instruct advanced users to clone a http url that ends with the "/.git/" + directory. For example, for downloads.kitenet.net, the clone url + is `https://downloads.kitenet.net/.git/` +9. Set up a git `post-receive` hook to update the repository's working tree + when changes are pushed to it. See below for details. + +When users clone over http, and run git-annex, it will +automatically learn all about your repository and be able to download files +right out of it, also using http. + +## post-receive hook + +If you have git-annex 4.20130703, the post-receive hook mentioned above +in step 8 just needs to run `git annex merge`. + +With older versions of git-annex, you can instead use `git annex sync`. + +There are two gotchas with some versions of git to be aware of when writing +this post-receive hook. + +1. The hook may be run with the current directory set to the `.git` + directory, and not the top of your work tree. So you need to `cd ..` or + similar in the hook. +2. `GIT_DIR` may be set to `.`, which will not be right after changing + directory. So you will probably want to unset it. + +Here's a post-receive hook that takes these problems into account: + +<pre> +#!/bin/sh +unset GIT_DIR +cd .. +git annex merge +</pre> diff --git a/doc/tips/setup_a_public_repository_on_a_web_site/comment_1_1d0fa6da33e401df1d7ff31979247fec._comment b/doc/tips/setup_a_public_repository_on_a_web_site/comment_1_1d0fa6da33e401df1d7ff31979247fec._comment new file mode 100644 index 000000000..cc0a0f2b3 --- /dev/null +++ b/doc/tips/setup_a_public_repository_on_a_web_site/comment_1_1d0fa6da33e401df1d7ff31979247fec._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnYLUTs4jFpBPwOlIIJ6qD8xdqZPboJafM" + nickname="Oluf" + subject="Combine this with a 'public' repository-group" + date="2013-07-16T09:44:26Z" + content=""" +Hi, + +would it be possible to do this whith the contents of a public repository-group (a non-bare public repository)? +"""]] diff --git a/doc/tips/setup_a_public_repository_on_a_web_site/comment_2_b98b761dee9d923153e3c288c1d987ee._comment b/doc/tips/setup_a_public_repository_on_a_web_site/comment_2_b98b761dee9d923153e3c288c1d987ee._comment new file mode 100644 index 000000000..7bfb89b36 --- /dev/null +++ b/doc/tips/setup_a_public_repository_on_a_web_site/comment_2_b98b761dee9d923153e3c288c1d987ee._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.4.90" + subject="comment 2" + date="2013-07-16T17:54:28Z" + content=""" +You can choose which files get stored in the public repository, and are thus accessible to the public. +However, note that since the git repository is published, anyone could clone it and see all the names and hashes of your files, even if you've not pushed the file contents to the public repository. + +Currently the way the \"public\" [[repository group|preferred_content]] works only makes it be usable with special remotes. This is because it uses a `preferreddir` setting in the special remote configuration. +"""]] diff --git a/doc/tips/shared_git_annex_directory_between_multiple_users.mdwn b/doc/tips/shared_git_annex_directory_between_multiple_users.mdwn new file mode 100644 index 000000000..5ca3b45ec --- /dev/null +++ b/doc/tips/shared_git_annex_directory_between_multiple_users.mdwn @@ -0,0 +1,39 @@ +Scenario +======== + +You have a server where you want to welcome other people to push files, say for a family photo album. People have their own user account, so by default they will not be able to read/write from each other's repositories, due to git-annex strict restrictions. + +Solution +======== + +Setup a shared git repository: + + git init shared ; cd shared # you can also do this on an existing git annex repo + git config core.sharedrepository group + chmod g+rwX -R . + chown -R :media . + +The idea here is to use the new (since [[news/version 4.20130909]]) support for git's `sharedRepository` configuration and restrict access to a specific group (instead of the default, a single user). You can also this to make the files accessible to all users on the system: + + git config core.sharedrepository world + chmod a+rwX -R . + +This will make sure that you anyone can operate that git annex repository remotely. + +Third party applications +------------------------ + +Now if another application that is not aware of git's `sharedRepository` configuration (say a [[bittorrent]] daemon) writes files there, you may want to make sure that the files created are also writable by everyone. This is more tricky, but one way of doing this is with the [[!wikipedia setgid]] bit: + + find -type d -exec chmod g+s {} \; + +You will also need to start the process with a proper [[!wikipedia umask]] (`002` instead of `022`). + +(!) I haven't actually tested this part. --[[anarcat]] + +See also +======== + + * [[tips/setup a public repository on a web site]] + * [[news/version 4.20130909]] + * [[bugs/acl not honoured in rsync remote]]: why this does not work on encrypted remotes diff --git a/doc/tips/skydriveannex.mdwn b/doc/tips/skydriveannex.mdwn new file mode 100644 index 000000000..56acdd96c --- /dev/null +++ b/doc/tips/skydriveannex.mdwn @@ -0,0 +1,29 @@ +skydriveannex +========= + +Hook program for gitannex to use [skydrive](http://en.wikipedia.org/wiki/SkyDrive) (previously *Windows Live SkyDrive* and *Windows Live Folders*) as backend + +# Requirements: + + python2 + python-yaml + +Credit for the Skydrive api interface goes to https://github.com/mk-fg/python-skydrive + +# Install +Clone the git repository in your home folder. + + git clone git://github.com/TobiasTheViking/skydriveannex.git + +This should make a ~/skydriveannex folder + +# Setup +Run the program once to set it up. + + cd ~/skydriveannex; python2 skydriveannex.py + +# Commands for gitannex: + + git config annex.skydrive-hook '/usr/bin/python2 ~/skydriveannex/skydriveannex.py' + git annex initremote skydrive type=hook hooktype=skydrive encryption=shared + git annex describe skydrive "the skydrive library" diff --git a/doc/tips/untrusted_repositories.mdwn b/doc/tips/untrusted_repositories.mdwn new file mode 100644 index 000000000..cdb5da7c3 --- /dev/null +++ b/doc/tips/untrusted_repositories.mdwn @@ -0,0 +1,28 @@ +Suppose you have a USB thumb drive and are using it as a git annex +repository. You don't trust the drive, because you could lose it, or +accidentally run it through the laundry. Or, maybe you have a drive that +you know is dying, and you'd like to be warned if there are any files +on it not backed up somewhere else. Maybe the drive has already died +or been lost. + +You can let git-annex know that you don't trust a repository, and it will +adjust its behavior to avoid relying on that repositories's continued +availability. + + # git annex untrust usbdrive + untrust usbdrive ok + +Now when you do a fsck, you'll be warned appropriately: + + # git annex fsck . + fsck my_big_file + Only these untrusted locations may have copies of this file! + 05e296c4-2989-11e0-bf40-bad1535567fe -- portable USB drive + Back it up to trusted locations with git-annex copy. + failed + +Also, git-annex will refuse to drop a file from elsewhere just because +it can see a copy on the untrusted repository. + +It's also possible to tell git-annex that you have an unusually high +level of trust for a repository. See [[trust]] for details. diff --git a/doc/tips/using_Amazon_Glacier.mdwn b/doc/tips/using_Amazon_Glacier.mdwn new file mode 100644 index 000000000..5aac7fa91 --- /dev/null +++ b/doc/tips/using_Amazon_Glacier.mdwn @@ -0,0 +1,75 @@ +Amazon Glacier provides low-cost storage, well suited for archiving and +backup. But it takes around 4 hours to get content out of Glacier. + +Recent versions of git-annex support Glacier. To use it, you need to have +[glacier-cli](http://github.com/basak/glacier-cli) installed. + +First, export your Amazon AWS credentials: + + # export AWS_ACCESS_KEY_ID="08TJMT99S3511WOZEP91" + # export AWS_SECRET_ACCESS_KEY="s3kr1t" + +Now, create a gpg key, if you don't already have one. This will be used +to encrypt everything stored in Glacier, for your privacy. Once you have +a gpg key, run `gpg --list-secret-keys` to look up its key id, something +like "2512E3C7" + +Next, create the Glacier remote. + + # git annex initremote glacier type=glacier keyid=2512E3C7 + initremote glacier (encryption setup with gpg key C910D9222512E3C7) (gpg) ok + +The configuration for the Glacier remote is stored in git. So to make another +repository use the same Glacier remote is easy: + + # cd /media/usb/annex + # git pull laptop + # git annex initremote glacier + initremote glacier (gpg) ok + +Now the remote can be used like any other remote. + + # git annex move my_cool_big_file --to glacier + copy my_cool_big_file (gpg) (checking glacier...) (to glacier...) ok + +But, when you try to get a file out of Glacier, it'll queue a retrieval +job: + + # git annex get my_cool_big_file + get my_cool_big_file (from glacier...) (gpg) + glacier: queued retrieval job for archive 'GPGHMACSHA1--862afd4e67e3946587a9ef7fa5beb4e8f1aeb6b8' + Recommend you wait up to 4 hours, and then run this command again. + failed + +Like it says, you'll need to run the command again later. Let's remember to +do that: + + # at now + 4 hours + at> git annex get my_cool_big_file + +Another oddity of Glacier is that git-annex is never entirely sure +if a file is still in Glacier. Glacier inventories take hours to retrieve, +and even when retrieved do not necessarily represent the current state. + +So, git-annex plays it safe, and avoids trusting the inventory: + + # git annex copy important_file --to glacier + copy important_file (gpg) (checking glacier...) (to glacier...) ok + # git annex drop important_file + drop important_file (gpg) (checking glacier...) + Glacier's inventory says it has a copy. + However, the inventory could be out of date, if it was recently removed. + (Use --trust-glacier if you're sure it's still in Glacier.) + + (unsafe) + Could only verify the existence of 0 out of 1 necessary copies + +Like it says, you can use `--trust-glacier` if you're sure +Glacier's inventory is correct and up-to-date. + +A final potential gotcha with Glacier is that glacier-cli keeps a local +mapping of file names to Glacier archives. If this cache is lost, or +you want to retrieve files on a different box than the one that put them in +glacier, you'll need to use `glacier vault sync` to rebuild this cache. + +See [[special_remotes/Glacier]] for details. diff --git a/doc/tips/using_Amazon_S3.mdwn b/doc/tips/using_Amazon_S3.mdwn new file mode 100644 index 000000000..0c68c7387 --- /dev/null +++ b/doc/tips/using_Amazon_S3.mdwn @@ -0,0 +1,37 @@ +git-annex extends git's usual remotes with some [[special_remotes]], that +are not git repositories. This way you can set up a remote using say, +Amazon S3, and use git-annex to transfer files into the cloud. + +First, export your Amazon AWS credentials: + + # export AWS_ACCESS_KEY_ID="08TJMT99S3511WOZEP91" + # export AWS_SECRET_ACCESS_KEY="s3kr1t" + +Now, create a gpg key, if you don't already have one. This will be used +to encrypt everything stored in S3, for your privacy. Once you have +a gpg key, run `gpg --list-secret-keys` to look up its key id, something +like "2512E3C7" + +Next, create the S3 remote, and describe it. + + # git annex initremote cloud type=S3 keyid=2512E3C7 + initremote cloud (encryption setup with gpg key C910D9222512E3C7) (checking bucket) (creating bucket in US) (gpg) ok + # git annex describe cloud "at Amazon's US datacenter" + describe cloud ok + +The configuration for the S3 remote is stored in git. So to make another +repository use the same S3 remote is easy: + + # cd /media/usb/annex + # git pull laptop + # git annex initremote cloud + initremote cloud (gpg) (checking bucket) ok + +Now the remote can be used like any other remote. + + # git annex copy my_cool_big_file --to cloud + copy my_cool_big_file (gpg) (checking cloud...) (to cloud...) ok + # git annex move video/hackity_hack_and_kaxxt.mov --to cloud + move video/hackity_hack_and_kaxxt.mov (checking cloud...) (to cloud...) ok + +See [[special_remotes/S3]] for details. diff --git a/doc/tips/using_Amazon_S3/comment_1_666a26f95024760c99c627eed37b1966._comment b/doc/tips/using_Amazon_S3/comment_1_666a26f95024760c99c627eed37b1966._comment new file mode 100644 index 000000000..60d96cb44 --- /dev/null +++ b/doc/tips/using_Amazon_S3/comment_1_666a26f95024760c99c627eed37b1966._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnoUOqs_lbuWyZBqyU6unHgUduJwDDgiKY" + nickname="Matt" + subject="ANNEX_S3 vs AWS for keys" + date="2012-05-29T12:24:25Z" + content=""" +The instructions state ANNEX_S3_ACCESS_KEY_ID and ANNEX_SECRET_ACCESS_KEY but git-annex cannot connect with those constants. git-annex tells me to set both \"AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY\" instead, which works. This is with Xubuntu 12.04. +"""]] diff --git a/doc/tips/using_Amazon_S3/comment_2_f5a0883be7dbb421b584c6dc0165f1ef._comment b/doc/tips/using_Amazon_S3/comment_2_f5a0883be7dbb421b584c6dc0165f1ef._comment new file mode 100644 index 000000000..dc809cb12 --- /dev/null +++ b/doc/tips/using_Amazon_S3/comment_2_f5a0883be7dbb421b584c6dc0165f1ef._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.81.112" + subject="comment 2" + date="2012-05-29T19:10:42Z" + content=""" +Thanks, I've fixed that. (You could have too.. this is a wiki ;) +"""]] diff --git a/doc/tips/using_Google_Cloud_Storage.mdwn b/doc/tips/using_Google_Cloud_Storage.mdwn new file mode 100644 index 000000000..d44e4f17f --- /dev/null +++ b/doc/tips/using_Google_Cloud_Storage.mdwn @@ -0,0 +1,9 @@ +[Google Cloud Storage](https://cloud.google.com/products/cloud-storage) +supports the same API as Amazon S3, so the +[[S3 special remote|special_remotes/S3]] can be used with it. +Here is a configuration example: + + git annex initremote cloud type=S3 encryption=none host=storage.googleapis.com port=80 + +Thanks to jterrance for the [original tip](https://gist.github.com/4576324). +--[[Joey]] diff --git a/doc/tips/using_Google_Cloud_Storage/comment_1_c576182f39563ae68767391c4227a177._comment b/doc/tips/using_Google_Cloud_Storage/comment_1_c576182f39563ae68767391c4227a177._comment new file mode 100644 index 000000000..3a4d02f32 --- /dev/null +++ b/doc/tips/using_Google_Cloud_Storage/comment_1_c576182f39563ae68767391c4227a177._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y" + nickname="Nigel" + subject="AWS credentials" + date="2013-05-31T10:23:23Z" + content=""" +This looks very valuable - Google are offering a free 5GB up until 2013 June 30th + +Sign in here[1] get your credentials here[2] under the section “Interoperable Access” (Source[3]) + + # Set up AWS credentials + $ export AWS_ACCESS_KEY_ID=\"YOUR-KEY\" + $ export AWS_SECRET_ACCESS_KEY=\"YOUR-SECRET\" + +1. http://gs-signup-redirect.appspot.com/ -or- https://developers.google.com/storage/docs/signup +2. https://storage.cloud.google.com/m +3. http://fog.io/storage/ +"""]] diff --git a/doc/tips/using_box.com_as_a_special_remote.mdwn b/doc/tips/using_box.com_as_a_special_remote.mdwn new file mode 100644 index 000000000..254d0a554 --- /dev/null +++ b/doc/tips/using_box.com_as_a_special_remote.mdwn @@ -0,0 +1,71 @@ +[Box.com](http://box.com/) is a file storage service, currently notable +for providing 50 gb of free storage if you sign up with its Android client. +(Or a few gb free otherwise.) + +git-annex can use Box as a [[special remote|special_remotes]]. +Recent versions of git-annex make this very easy to set up: + + WEBDAV_USERNAME=you@example.com WEBDAV_PASSWORD=xxxxxxx git annex initremote box.com type=webdav url=https://www.box.com/dav/git-annex chunksize=75mb encryption=shared + +Note the use of chunksize; Box has a 100 mb maximum file size, and this +breaks up large files into chunks before that limit is reached. + +# old davfs2 method + +This method is deprecated, but still documented here just in case. +Note that the files stored using this method cannot reliably be retreived +using the webdav special remote. + +## davfs2 setup + +* First, install + the [davfs2](http://savannah.nongnu.org/projects/davfs2) program, + which can mount Box using WebDAV. On Debian, just `sudo apt-get install davfs2` +* Allow users to mount davfs filesystems, by ensuring that + `/sbin/mount.davfs` is setuid root. On Debian, just `sudo dpkg-reconfigure davfs2` +* Add yourself to the davfs2 group. + + sudo adduser $(whoami) davfs2 + +* Edit `/etc/fstab`, and add a line to mount Box using davfs. + + sudo mkdir -p /media/box.com + echo "https://www.box.com/dav/ /media/box.com davfs noauto,user 0 0" | sudo tee -a /etc/fstab + +* Create `~/.davfs2/davfs2.conf` with some important settings: + + mkdir ~/.davfs2/ + echo use_locks 0 > ~/.davfs2/davfs2.conf + echo cache_size 1 >> ~/.davfs2/davfs2.conf + echo delay_upload 0 >> ~/.davfs2/davfs2.conf + +* Create `~/.davfs2/secrets`. This file contains your Box.com login and password. + Your login is probably the email address you signed up with. + + echo "/media/box.com joey@kitenet.net mypassword" > ~/.davfs2/secrets + chmod 600 ~/.davfs2/secrets + +* Now you should be able to mount Box, as a non-root user: + + mount /media/box.com + +## git-annex setup + +You need git-annex version 3.20120303 or newer, which adds support for chunking +files larger than Box's 100 mb limit. + +Create the special remote, in your git-annex repository. +** This example is non-encrypted; fill in your gpg key ID for a securely +encrypted special remote! ** + + git annex initremote box.com type=directory directory=/media/box.com chunksize=2mb encryption=none + +Now git-annex can copy files to box.com, get files from it, etc, just like +with any other special remote. + + % git annex copy bigfile --to box.com + bigfile (to box.com...) ok + % git annex drop bigfile + bigfile (checking box.com...) ok + % git annex get bigfile + bigfile (from box.com...) ok diff --git a/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn new file mode 100644 index 000000000..594d8c480 --- /dev/null +++ b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn @@ -0,0 +1,59 @@ +## Intro + +This tip is based on my (Matt Ford) experience of using `git annex` with my out-and-about netbook which hits many different wifi networks and has no fixed home or address. + +I'm not using a bare repository that allows pushing (an alternative solution) nor do I fancy allowing `git push` to run against my desktop checked out repository (perhaps I worry over nothing?) + +None of this is really `git annex` specific but I think it is useful to know... + +## Dealing with no fixed hostname + +Essentially set up two repos as per the [[walkthrough]]. + +Desktop as follows: + + cd ~/annex + git init + git annex init "desktop" + +And the laptop like this + + git clone ssh://desktop/annex + git init + git annex init "laptop" + +Now we want to add the the repos as remotes of each other. + +For the laptop it is easy: + + git remote add desktop ssh://desktop/~/annex + +However for the desktop to add an ever changing laptops hostname it's a little tricky. We make use of remote SSH tunnels to do this. Essentially we have the laptop (which always knows it's own name and address and knows the address of the desktop) create a tunnel starting on an arbitrary port at the desktop and heads back to the laptop on it's own SSH server port (22). + +To do this make part of your laptop's SSH config look like this: + + Host desktop + User matt + HostName desktop.example.org + RemoteForward 2222 localhost:22 + +Now on the desktop to connect over the tunnel to the laptop's SSH port you need this: + + Host laptop + User matt + HostName localhost + port 2222 + +So to add the desktop's remote: + +a) From the laptop ensure the tunnel is up + + ssh desktop + +b) From the desktop add the remote + + git remote add laptop ssh://laptop/~/annex + +So now you can work on the train, pop on the wifi at work upon arrival, and sync up with a `git pull && git annex get`. + +An alternative solution may be to use direct tunnels over Openvpn. diff --git a/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh/comment_1_c0b7682a2b6f3078457b85683c825baf._comment b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh/comment_1_c0b7682a2b6f3078457b85683c825baf._comment new file mode 100644 index 000000000..e627ead47 --- /dev/null +++ b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh/comment_1_c0b7682a2b6f3078457b85683c825baf._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://adamspiers.myopenid.com/" + nickname="Adam" + subject="comment 1" + date="2011-12-23T13:31:33Z" + content=""" +ControlPersist is awesome - thanks! + +Here's [an alternative, git-specific approach](http://thread.gmane.org/gmane.comp.version-control.home-dir/502). +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex.mdwn b/doc/tips/using_gitolite_with_git-annex.mdwn new file mode 100644 index 000000000..fcc3f96c3 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex.mdwn @@ -0,0 +1,89 @@ +[Gitolite](https://github.com/sitaramc/gitolite) is a git repository +manager. Here's how to add git-annex support to gitolite, so you can +`git annex copy` files to a gitolite repository, and `git annex get` +files from it. + +Warning : The method described here works with gitolite version g2, avaible in the g2 branch on github. There is an experimental support for g3 in the git-annex branch, if you tested it please add some feedback. + +A nice feature of using gitolite with git-annex is that users can be given +read-only access to a repository, and this allows them to `git annex get` +file contents, but not change anything. + +First, you need new enough versions: + +* gitolite 2.2 is needed -- this version contains a git-annex-shell ADC + and supports "ua" ADCs. +* git-annex 3.20111016 or newer needs to be installed on the gitolite + server. Don't install an older version, it wouldn't be secure! + +And here's how to set it up. The examples are for gitolite as installed +on Debian with apt-get, but the changes described can be made to any +gitolite installation, just with different paths. + +Set `$GL_ADC_PATH` in `.gitolite.rc`, if you have not already done so. + +<pre> +echo '$GL_ADC_PATH = "/usr/local/lib/gitolite/adc/";' >>~gitolite/.gitolite.rc +</pre> + +Make the ADC directory, and a "ua" subdirectory. + +<pre> +mkdir -p /usr/local/lib/gitolite/adc/ua +</pre> + +Install the git-annex-shell ADC into the "ua" subdirectory from the gitolie repository. + +<pre> +cd /usr/local/lib/gitolite/adc/ua/ +cp gitolite/contrib/adc/git-annex-shell . +</pre> + +Now all gitolite repositories can be used with git-annex just as any +ssh remote normally would be used. For example: + +<pre> +# git clone gitolite@localhost:testing +Cloning into testing... +Receiving objects: 100% (18/18), done. +# cd testing +# git annex init +init ok +# cp /etc/passwd my-cool-big-file +# git annex add my-cool-big-file +add my-cool-big-file ok +(Recording state in git...) +# git commit -m added +[master d36c8b4] added + 1 files changed, 1 insertions(+), 0 deletions(-) + create mode 120000 my-cool-big-file +# git push --all +Counting objects: 17, done. +Delta compression using up to 2 threads. +Compressing objects: 100% (12/12), done. +Writing objects: 100% (14/14), 1.39 KiB, done. +Total 14 (delta 0), reused 1 (delta 0) +To gitolite@localhost:testing + c552a38..db4653e git-annex -> git-annex + 29cd204..d36c8b4 master -> master +# git annex copy --to origin +copy my-cool-big-file (checking origin...) (to origin...) +WORM-s2502-m1318875140--my-cool-big-file + 2502 100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/1) + +sent 2606 bytes received 31 bytes 1758.00 bytes/sec +total size is 2502 speedup is 0.95 +ok +</pre> + + +### Troubleshooting + +I got an error like this when setting up gitolite *after* setting up a local git repo and git annex: + +<pre> +git-annex-shell: First run: git-annex init +Command ssh ["git@git.example.com","git-annex-shell 'configlist' '/~/myrepo.git'"] failed; exit code 1 +</pre> + +because I forgot to "git push --all" after adding the new gitolite remote. diff --git a/doc/tips/using_gitolite_with_git-annex/comment_10_8767bc8014b459a3cd76f275fd4fa8d6._comment b/doc/tips/using_gitolite_with_git-annex/comment_10_8767bc8014b459a3cd76f275fd4fa8d6._comment new file mode 100644 index 000000000..b01779cb4 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_10_8767bc8014b459a3cd76f275fd4fa8d6._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://ertai.myopenid.com/" + nickname="npouillard" + subject="git-annex no longer supported by gitolite g3" + date="2013-03-25T12:47:21Z" + content=""" +See http://gitolite.com/gitolite/dev-status.html for some details. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_11_00715e0b47f09130e0e536e29f7b9258._comment b/doc/tips/using_gitolite_with_git-annex/comment_11_00715e0b47f09130e0e536e29f7b9258._comment new file mode 100644 index 000000000..1fbbd9b8a --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_11_00715e0b47f09130e0e536e29f7b9258._comment @@ -0,0 +1,31 @@ +[[!comment format=mdwn + username="http://mildred.fr/" + nickname="mildred" + subject="Problems with URL ending with ".git"" + date="2013-05-24T12:15:16Z" + content=""" +Hi, + +I noticed using the git-annex branch of gitolite v3 that the same URL with \".git\" at the end would not work in git-annex. For example my test repository was `git@git2.mildred.fr:u/mildred/Annex.git` but it didn't work until I converted it to `git@git2.mildred.fr:u/mildred/Annex` + +On the server, the repository is in `repositories/u/mildred/Annex.git` + +If I try a copy with git-annex for example, I would get: + + $ git annex copy titi --to test + copy titi (checking test...) FATAL: u/mildred/Annex.git mildred DENIED + + (unable to check test) failed + git-annex: copy: 1 failed + +(test is the name of my remote and titi is my file) + +Note, in my gitolite conf, I have: + + repo u/CREATOR/[a-zA-Z0-9].* + C = @all + RW+D = CREATOR + RW = WRITERS + R = READERS + +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_12_7027ce60265b8f24c8ab54553e544068._comment b/doc/tips/using_gitolite_with_git-annex/comment_12_7027ce60265b8f24c8ab54553e544068._comment new file mode 100644 index 000000000..f692dd93e --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_12_7027ce60265b8f24c8ab54553e544068._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawn5RcmefXjrl1vmbHIiOWQyXGXVKxlm3rg" + nickname="Kavin" + subject="comment 12" + date="2013-07-25T03:20:15Z" + content=""" +latest code of gitolite does not support git-annex ? I could not find a way to make it work ? +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_13_75218b7409c0e281cb01c9b2791e8cdf._comment b/doc/tips/using_gitolite_with_git-annex/comment_13_75218b7409c0e281cb01c9b2791e8cdf._comment new file mode 100644 index 000000000..bc674a592 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_13_75218b7409c0e281cb01c9b2791e8cdf._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawm9tgeFE5v-arAYYftSv3yUTI5Q4qB2C9M" + nickname="Khaije" + subject="git-annex with gitolite FTW" + date="2013-08-13T15:13:07Z" + content=""" +The steps to activate git-annex integration have changed/simplified for v3. + + +1) during install, be sure to use the 'git-annex' branch, rather than master[fn:1]. + +2) to enable git-annex-shell, open ~/.gitolite.rc and insert 'git-annex-shell' => 'ua' into the hash list in the COMMANDS array.[fn:2] +#'git-annex-shell' => 'ua', + + + + +[fn:1] We'd like to have this feature-branch merged to master, so please send Sitaram feedback, positive and negative, based on your experiences. +[fn:2] There is no GL_ADC_PATH and no \"ua\" subdirectory here, and nothing to \"install\"; the command now comes with gitolite. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_14_7d4d4515218d1259d32be3baeb5ee56e._comment b/doc/tips/using_gitolite_with_git-annex/comment_14_7d4d4515218d1259d32be3baeb5ee56e._comment new file mode 100644 index 000000000..048dfa698 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_14_7d4d4515218d1259d32be3baeb5ee56e._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSbvo_NbY-ev1VKtzwo7nEqUmvRO6rXGA" + nickname="François" + subject="comment 14" + date="2013-09-22T18:30:45Z" + content=""" +@khaije + +Could you paste your config file? Here is mine: http://paste.debian.net/44856/ +I don't have any COMMANDS array. Could you elaborate your modifications please? + +Thanks. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_15_dc6f21b5a3d5931c8d949a9753411b9e._comment b/doc/tips/using_gitolite_with_git-annex/comment_15_dc6f21b5a3d5931c8d949a9753411b9e._comment new file mode 100644 index 000000000..f07116983 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_15_dc6f21b5a3d5931c8d949a9753411b9e._comment @@ -0,0 +1,29 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnRaueN1AcM8pIMofH5-wQ1Kr4T0GBe8sA" + nickname="wayne" + subject="git-annex with gitolite-rc" + date="2013-10-19T17:03:32Z" + content=""" +@François + +The proper array in .gitolite.rc seems to be the \"ENABLE\" array, which it appears is parsed into the COMMANDS array in src/lib/Gitolite/Rc.pm + +@Khaije + +Using 'git-annex-shell' => 'ua' doesn't seem to work for me. The program still fails in src/gitolite-shell around line 163 (gitolite repo version b1d3c05): + + _die \"suspicious characters loitering about '$soc'\" + if $rc{COMMANDS}{ words[0] } ne 'ua' and $soc !~ $REMOTE_COMMAND_PATT; + +When I insert $rc{COMMANDS}{ words[0] } into the _die message, it shows up as \"1\" instead of \"ua\" as I was expecting. + +When I manually set $rc{COMMANDS}{ words[0] } to 'ua' slightly earlier in the script, the git-annex-shell command gets run but it seems to fail to parse the result of the configlist command properly because then I get + + Failed to get annex.uuid configuration of repository origin + + Instead, got: \"annex.uuid=\ncore.gcrypt=\n\" + + +I am actually about to give up on the notion of using git-annex and gitolite together. Maybe. I am interested to know if anyone else is having similar problems. + +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_16_8e5039e6655fc80dc863b6cdf44ef02a._comment b/doc/tips/using_gitolite_with_git-annex/comment_16_8e5039e6655fc80dc863b6cdf44ef02a._comment new file mode 100644 index 000000000..5e0e6acd6 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_16_8e5039e6655fc80dc863b6cdf44ef02a._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmbo-yMzW_lkBrS4ICNn5XMr8saYtI1_WY" + nickname="Douglas" + subject="Any further info regarding gitolite support?" + date="2013-11-14T03:20:31Z" + content=""" +We have reached the same point as the previous poster from 25 days ago. +$ git annex copy --to origin +FATAL: suspicious characters loitering about 'git-annex-shell 'configlist' '/~/testing'' + + Remote origin does not have git-annex installed; setting remote.origin.annex-ignore +git-annex: cannot determine uuid for origin + +Anyone actually have this working? +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_17_9c40e1da8bb44f7207e802377f5cf923._comment b/doc/tips/using_gitolite_with_git-annex/comment_17_9c40e1da8bb44f7207e802377f5cf923._comment new file mode 100644 index 000000000..57152bad7 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_17_9c40e1da8bb44f7207e802377f5cf923._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawm9tgeFE5v-arAYYftSv3yUTI5Q4qB2C9M" + nickname="Khaije" + subject="comment 17" + date="2013-11-23T02:14:12Z" + content=""" +my gitolite.rc is available at https://gist.github.com/khaije1/7609848 + +For whatever reason I've found this to be very simple to get working so I'd guess there's a missing ingredient somewhere. The combination of gitolite and git-annex is valuable to me so I'll add documents to the url above in hopes it will assist some people with getting the same value. + +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_1_9a2a2a8eac9af97e0c984ad105763a73._comment b/doc/tips/using_gitolite_with_git-annex/comment_1_9a2a2a8eac9af97e0c984ad105763a73._comment new file mode 100644 index 000000000..807180660 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_1_9a2a2a8eac9af97e0c984ad105763a73._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="http://www.openid.albertlash.com/openid/" + ip="71.178.29.218" + subject="comment 1" + date="2011-12-24T06:08:45Z" + content=""" +Looks like you are missing a closing double quote on the line: + + +echo '$GL_ADC_PATH = \"/usr/local/lib/gitolite/adc/;' >>~gitolite/.gitolite.rc + +right after /; + +I got this working by the way - great stuff. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_2_d8efea4ab9576555fadbb47666ecefa9._comment b/doc/tips/using_gitolite_with_git-annex/comment_2_d8efea4ab9576555fadbb47666ecefa9._comment new file mode 100644 index 000000000..007a009ea --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_2_d8efea4ab9576555fadbb47666ecefa9._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 2" + date="2011-12-24T16:54:31Z" + content=""" +I've fixed the typo (anyone can edit pages in this wiki FWIW.) +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_3_807035f38509ccb9f93f1929ecd37417._comment b/doc/tips/using_gitolite_with_git-annex/comment_3_807035f38509ccb9f93f1929ecd37417._comment new file mode 100644 index 000000000..243764054 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_3_807035f38509ccb9f93f1929ecd37417._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="bremner" + ip="156.34.79.193" + subject="repo name conventions?" + date="2011-12-30T21:41:13Z" + content=""" +I'm confused by the fact that the git-annex-shell adc rejects any repo names that don't start with /~/ since none of my repos start that way. It seems work ok if I just delete /\~ from the front of the regex, but I feel like I must be missing something. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_4_eb81f824aadc97f098379c5f7e4fba4c._comment b/doc/tips/using_gitolite_with_git-annex/comment_4_eb81f824aadc97f098379c5f7e4fba4c._comment new file mode 100644 index 000000000..c53ce01d9 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_4_eb81f824aadc97f098379c5f7e4fba4c._comment @@ -0,0 +1,33 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 4" + date="2011-12-31T00:29:45Z" + content=""" +Well a repo url like `gitolite@localhost:testing` puts it in the gitolite user's /~/testing + +This worked when I added the gitolite stuff, anyway.. Let's see if it still does: + +<pre> +joey@gnu:~/tmp>mkdir g +joey@gnu:~/tmp>cd g +joey@gnu:~/tmp/g>git init +Initialized empty Git repository in /home/joey/tmp/g/.git/ +joey@gnu:~/tmp/g>git annex init +init ok +joey@gnu:~/tmp/g>git remote add test 'gitolite@localhost:testing' +joey@gnu:~/tmp/g>touch foo +joey@gnu:~/tmp/g>git annex add foo +add foo (checksum...) ok +(Recording state in git...) +joey@gnu:~/tmp/g>git annex copy foo --to test --debug +git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"ls-files\",\"--cached\",\"-z\",\"--\",\"foo\"] +git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"check-attr\",\"annex.numcopies\",\"-z\",\"--stdin\"] +git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] +git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"show-ref\",\"git-annex\"] +git [\"--git-dir=/home/joey/tmp/g/.git\",\"--work-tree=/home/joey/tmp/g\",\"cat-file\",\"--batch\"] +Running: ssh [\"-4\",\"gitolite@localhost\",\"git-annex-shell 'configlist' '/~/testing'\"] +</pre> + +Still seems right, the ADC's regexp will match this the git-annex shell command. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_5_f688309532d2993630e9e72e87fb9c46._comment b/doc/tips/using_gitolite_with_git-annex/comment_5_f688309532d2993630e9e72e87fb9c46._comment new file mode 100644 index 000000000..052fc90d6 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_5_f688309532d2993630e9e72e87fb9c46._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="bremner" + ip="156.34.79.193" + subject="gitolite gets different paths for different urls" + date="2011-12-31T01:50:49Z" + content=""" +I guess there is some path rewriting going in in gitolite proper because if try a url of the form +ssh://git@localhost/testing, then it still works with gitolite, but fails with the ADC because +the repo is passed as /testing: +<pre> +Running: ssh [\"git@host\",\"git-annex-shell 'configlist' '/recommend'\"] +Running: ssh [\"git@host\",\"git-annex-shell 'configlist' '/recommend'\"] +</pre> + +What I have to ask Sitaram and or find in the docs is if this is a bug or a feature in gitolite. I can see how the leading slash would get swallowed up by this line +<pre> +$repo = \"'$REPO_BASE/$repo.git'\" +</pre> +in gl-auth-command, but I guess that isn't the whole story. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_6_3e203e010a4df5bf03899f867718adc5._comment b/doc/tips/using_gitolite_with_git-annex/comment_6_3e203e010a4df5bf03899f867718adc5._comment new file mode 100644 index 000000000..ce888cb13 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_6_3e203e010a4df5bf03899f867718adc5._comment @@ -0,0 +1,25 @@ +[[!comment format=mdwn + username="bremner" + ip="156.34.79.193" + subject="ssh://gitolite-host/repo-name is supposed to work" + date="2011-12-31T03:34:17Z" + content=""" +I confirmed with Sitaram that this is intentional, if probably under-documented. +Since the ADC strips the leading /~/ in assigning $start anyway, I guess something like the following will work +<pre> + +diff --git a/contrib/adc/git-annex-shell b/contrib/adc/git-annex-shell +index 7f9f5b8..523dfed 100755 +--- a/contrib/adc/git-annex-shell ++++ b/contrib/adc/git-annex-shell +@@ -28,7 +28,7 @@ my $cmd=$ENV{SSH_ORIGINAL_COMMAND}; + # the second parameter. + # Further parameters are not validated here (see below). + die \"bad git-annex-shell command: $cmd\" +- unless $cmd =~ m#^(git-annex-shell '\w+' ')/\~/([0-9a-zA-Z][0-9a-zA-Z._\@/+- ++ unless $cmd =~ m#^(git-annex-shell '\w+' ')/(?:\~\/)?([0-9a-zA-Z][0-9a-zA-Z. + my $start = $1; + my $repo = $2; + my $end = $3; +</pre> +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_7_f8fd08b6ab47378ad88c87348057220d._comment b/doc/tips/using_gitolite_with_git-annex/comment_7_f8fd08b6ab47378ad88c87348057220d._comment new file mode 100644 index 000000000..bdbecd4d9 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_7_f8fd08b6ab47378ad88c87348057220d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 7" + date="2011-12-31T18:32:28Z" + content=""" +That patch seems ok, it doesn't seem to allow through any repo locations that were blocked before. + +So, it has my blessing.. but the ADC is in gitolite and will need to be patched there. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_8_8249772c142117f88e37975d058aa936._comment b/doc/tips/using_gitolite_with_git-annex/comment_8_8249772c142117f88e37975d058aa936._comment new file mode 100644 index 000000000..0717bab1c --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_8_8249772c142117f88e37975d058aa936._comment @@ -0,0 +1,29 @@ +[[!comment format=mdwn + username="bremner" + ip="156.34.79.193" + subject="afaict git annex normalizes urls on the client side." + date="2011-12-31T22:29:38Z" + content=""" +After some debugging printing, here is my current understanding. + +- urls of the form git@host:~repo or ssh://git@host + + - git sends commands like \"git-receive-pack '~/repo' + - gitolite converts these to $REPO_BASE/~/repo which fails. ~/repo would also fail fwiw. + - git-annex sends seems /~/repo, which works + +- urls of the form git@host:/repo or ssh://git@host/repo + + - git sends \"git-receive-pack '/db/cs3383'\" + - gitolite converts this to $REPO_BASE/repo which works + - git annex sends \"git-annex-shell 'inannex' '/repo' ...\" which works, but only with the patch above. + +- urls of the form git@host:repo + + - git sends \"git-receive-pack 'repo' + - gitolite converts this to $REPO_BASE/repo, which works + - git-annex sends \"git-annex-shell 'inannex' '/~/db/cs3383'...\", which also works for git-annex-shell. + +So the weird case is the last one where git and git-annex are sending different things over the wire. +I don't know if you have other motivations for doing the url normalization on the client side, but it isn't needed for gitolite, and in some sense complicates things a little. On the other hand, now that I see what is going on, it isn't a big deal to just strip the leading /~ off in the adc. It does lead to the odd situation of some URLs working for git-annex but not git. +"""]] diff --git a/doc/tips/using_gitolite_with_git-annex/comment_9_28418635a6ed7231b89e02211cd3c236._comment b/doc/tips/using_gitolite_with_git-annex/comment_9_28418635a6ed7231b89e02211cd3c236._comment new file mode 100644 index 000000000..fc297ff17 --- /dev/null +++ b/doc/tips/using_gitolite_with_git-annex/comment_9_28418635a6ed7231b89e02211cd3c236._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 9" + date="2012-01-02T16:27:55Z" + content=""" +Ah right. git-annex normalizes all git ssh style user@host:dir to valid uris, which is where the `/~/` comes from. I don't anticipate this changing on the git-annex side. +"""]] diff --git a/doc/tips/using_the_SHA1_backend.mdwn b/doc/tips/using_the_SHA1_backend.mdwn new file mode 100644 index 000000000..70dc2ef75 --- /dev/null +++ b/doc/tips/using_the_SHA1_backend.mdwn @@ -0,0 +1,11 @@ +A handy alternative to the default [[backend|backends]] is the +SHA1 backend. This backend provides more git-style assurance that your data +has not been damaged. And the checksum means that when you add the same +content to the annex twice, only one copy need be stored in the backend. + +The only reason it's not the default is that it needs to checksum +files when they're added to the annex, and this can slow things down +significantly for really big files. To make SHA1 the default, just +add something like this to `.gitattributes`: + + * annex.backend=SHA1 diff --git a/doc/tips/using_the_web_as_a_special_remote.mdwn b/doc/tips/using_the_web_as_a_special_remote.mdwn new file mode 100644 index 000000000..e04ba800d --- /dev/null +++ b/doc/tips/using_the_web_as_a_special_remote.mdwn @@ -0,0 +1,101 @@ +The web can be used as a [[special_remote|special_remotes]] too. + + # git annex addurl http://example.com/video.mpeg + addurl example.com_video.mpeg (downloading http://example.com/video.mpeg) + ########################################################## 100.0% + ok + +Now the file is downloaded, and has been added to the annex like any other +file. So it can be renamed, copied to other repositories, and so on. + +To add a lot of urls at once, just list them all as parameters to +`git annex addurl`. + +## trust issues + +Note that git-annex assumes that, if the web site does not 404, and has the +right file size, the file is still present on the web, and this counts as +one [[copy|copies]] of the file. If the file still seems to be present +on the web, it will let you remove your last copy, trusting it can be +downloaded again: + + # git annex drop example.com_video.mpeg + drop example.com_video.mpeg (checking http://example.com/video.mpeg) ok + +If you don't [[trust]] the web to this degree, just let git-annex know: + + # git annex untrust web + untrust web ok + +With the result that it will hang onto files: + + # git annex drop example.com_video.mpeg + drop example.com_video.mpeg (unsafe) + Could only verify the existence of 0 out of 1 necessary copies + Also these untrusted repositories may contain the file: + 00000000-0000-0000-0000-000000000001 -- web + (Use --force to override this check, or adjust annex.numcopies.) + failed + +## attaching urls to existing files + +You can also attach urls to any file already in the annex: + + # git annex addurl --file my_cool_big_file http://example.com/cool_big_file + addurl my_cool_big_file ok + # git annex whereis my_cool_big_file + whereis my_cool_big_file (2 copies) + 00000000-0000-0000-0000-000000000001 -- web + 27a9510c-760a-11e1-b9a0-c731d2b77df9 -- here + +## configuring filenames + +By default, `addurl` will generate a filename for you. You can use +`--file=` to specify the filename to use. + +If you're adding a bunch of related files to a directory, or just don't +like the default filenames generated by `addurl`, you can use `--pathdepth` +to specify how many parts of the url are put in the filename. +A positive number drops that many paths from the beginning, while a negative +number takes that many paths from the end. + + # git annex addurl http://example.com/videos/2012/01/video.mpeg + addurl example.com_videos_2012_01_video.mpeg (downloading http://example.com/videos/2012/01/video.mpeg) + # git annex addurl http://example.com/videos/2012/01/video.mpeg --pathdepth=2 + addurl 2012_01_video.mpeg (downloading http://example.com/videos/2012/01/video.mpeg) + # git annex addurl http://example.com/videos/2012/01/video.mpeg --pathdepth=-2 + addurl 01_video.mpeg (downloading http://example.com/videos/2012/01/video.mpeg) + +## videos + +<a name=quvi></a> + +There's support for downloading videos from sites like YouTube, Vimeo, +and many more. This relies on [quvi](http://quvi.sourceforge.net/) to find +urls to the actual videos files. + +When you have quvi installed, you can just +`git annex addurl http://youtube.com/foo` and it will detect that +it is a video and download the video content for offline viewing. + +Later, in another clone of the repository, you can run `git annex get` on +the file and it will also be downloaded with the help of quvi. This works +even if the video host has transcoded or otherwise changed the video +in the meantime; the assumption is that these video files are equivilant. + +There is an `annex.quvi-options` configuration setting that can be used +to pass parameters to quvi. For example, you could set `git config +annex.quvi-options "--format low"` to configure it to download low +quality videos from YouTube. + +Note that for performance reasons, the url is not checked for redirects, +so some shortened urls will not be detected. You can +either load the short url in a browser to get the full url, or you +can force use of quvi with redirect detection, by prepending "quvi:" to the +url. + +Downloading whole YouTube playlists is not currently supported by quvi. + +## podcasts + +This is done using `git annex importfeed`. See [[downloading podcasts]]. diff --git a/doc/tips/using_the_web_as_a_special_remote/comment_1_321a41d611c6fe45e047af9c96c5176c._comment b/doc/tips/using_the_web_as_a_special_remote/comment_1_321a41d611c6fe45e047af9c96c5176c._comment new file mode 100644 index 000000000..ee1a271ea --- /dev/null +++ b/doc/tips/using_the_web_as_a_special_remote/comment_1_321a41d611c6fe45e047af9c96c5176c._comment @@ -0,0 +1,26 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlc1og3PqIGudOMkFNrCCNg66vB7s-jLpc" + nickname="Paul" + subject="can addurl use hashing once the file is downloaded?" + date="2012-09-20T21:01:30Z" + content=""" +There are resources that I want to add to my annex that are currently available +via a URL, but it seems like if I add these using `git-annex addurl`, they get +symlinked to file in the annex/objects directory that starts with `URL-...`, +instead of the more typical `SHA256-...`, and this does not change even after +the files are downloaded. + +My concern is that I really want to ensure that these files don't change, which +is the appeal of content-addressable symlinking of normal files (as opposed to +URL addressable ones). + +Would there be a way to automate the injection of hash-based symlinking for +files that are added via addurl? Sometimes I add a bunch of files via ``addurl +--fast``, and after I've download them via ``get``, it would be nice to have +those files have the same level of data integrity as when I download them using +something outside of git-annex, add them to the annex, and do an ``addurl +--file`` afterward. + +Thanks for all of your hard work! + +"""]] diff --git a/doc/tips/using_the_web_as_a_special_remote/comment_2_dfe9c8c49aadff80d2020288584e0390._comment b/doc/tips/using_the_web_as_a_special_remote/comment_2_dfe9c8c49aadff80d2020288584e0390._comment new file mode 100644 index 000000000..b015cdcec --- /dev/null +++ b/doc/tips/using_the_web_as_a_special_remote/comment_2_dfe9c8c49aadff80d2020288584e0390._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + subject="comment 2" + date="2012-09-20T21:55:57Z" + content=""" +`addurl` only uses the URL- keys if you run it with --fast. Otherwise it downloads the content and hashes it the same as `add` does. + +If you use `--fast`, you can go back and `git annex migrate` the file once it's been downloaded, to convert +it to the SHA backend. +"""]] diff --git a/doc/tips/using_the_web_as_a_special_remote/comment_3_ed8dd3bbd9b9ae7f2309b72b94f61eb1._comment b/doc/tips/using_the_web_as_a_special_remote/comment_3_ed8dd3bbd9b9ae7f2309b72b94f61eb1._comment new file mode 100644 index 000000000..0601f3005 --- /dev/null +++ b/doc/tips/using_the_web_as_a_special_remote/comment_3_ed8dd3bbd9b9ae7f2309b72b94f61eb1._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnx8kHW66N3BqmkVpgtXDlYMvr8TJ5VvfY" + nickname="Yaroslav" + subject="how to drop one of the urls?" + date="2013-04-12T14:53:29Z" + content=""" +is there a way to remove one of the urls? e.g. if I have + + $> git annex whereis fail2ban_logo.png + whereis fail2ban_logo.png (1 copy) + 00000000-0000-0000-0000-000000000001 -- web + + web: http://www.fail2ban.org/fail2ban_logo.png + web: http://www.onerussian.com/tmp/statsmodes.png + ok + +and would like to remove the fail2ban.org one... ? +"""]] diff --git a/doc/tips/using_the_web_as_a_special_remote/comment_4_c1133a524989a940f1b5db588707157a._comment b/doc/tips/using_the_web_as_a_special_remote/comment_4_c1133a524989a940f1b5db588707157a._comment new file mode 100644 index 000000000..bd55a7872 --- /dev/null +++ b/doc/tips/using_the_web_as_a_special_remote/comment_4_c1133a524989a940f1b5db588707157a._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 4" + date="2013-04-22T21:28:03Z" + content=""" +You can use `git annex rmurl $file $url`, which I just added to git-annex. + +(Also, `git annex drop $file --from web` will remove all the urls..) +"""]] diff --git a/doc/tips/visualizing_repositories_with_gource.mdwn b/doc/tips/visualizing_repositories_with_gource.mdwn new file mode 100644 index 000000000..25a69c1b7 --- /dev/null +++ b/doc/tips/visualizing_repositories_with_gource.mdwn @@ -0,0 +1,22 @@ +[Gource](http://code.google.com/p/gource/) is an amazing animated +visualisation of a git repository. + +Normally, gource shows files being added, removed, and changed in +the repository, and the user(s) making the changes. Of course it can be +used in this way in a repository using git-annex too; just run `gource`. + +The other way to use gource with git-annex is to visualise the movement of +annexed file contents between repositories. In this view, the "users" are +repositories, and they move around the file contents that are being added +or removed from them with git-annex. + +[[!img screenshot.jpg]] + +To use gource this way, first go into the directory you want to visualize, +and use `git annex log` to make an input file for `gource`: + + git annex log --gource | tee gource.log + sort gource.log | gource --log-format custom - + +The `git annex log` can take a while, to speed it up you can use something +like `--after "4 months ago"` to limit how far back it goes. diff --git a/doc/tips/visualizing_repositories_with_gource/screenshot.jpg b/doc/tips/visualizing_repositories_with_gource/screenshot.jpg Binary files differnew file mode 100644 index 000000000..9d3096b99 --- /dev/null +++ b/doc/tips/visualizing_repositories_with_gource/screenshot.jpg diff --git a/doc/tips/what_to_do_when_a_repository_is_corrupted.mdwn b/doc/tips/what_to_do_when_a_repository_is_corrupted.mdwn new file mode 100644 index 000000000..80cb046d9 --- /dev/null +++ b/doc/tips/what_to_do_when_a_repository_is_corrupted.mdwn @@ -0,0 +1,22 @@ +A git-annex repository on a removable USB drive is great, until the cable +falls out at the wrong time and git's repository gets trashed. The way +git checksums everything and the poor quality of USB media makes this +perhaps more likely than you would expect. If this happens to you, +here's a way to recover that makes the most of whatever data is left +on the drive. + +* First, run `git fsck`. If it does not report any problems, your data + is fine, and you don't need to proceed further. +* So `git fsck` says the git repository is corrupted. But probably the data + git-annex stored is fine. Your first step is to clone another copy + of the git repository from somewhere else. Let's call this clone + "$good", and the corrupted repository "$bad". +* Preserve your git configuration changes, and the `annex.uuid` setting: + `mv $bad/.git/config $good/.git/config` +* Move annexed data into the new repository: `mkdir $good/.git/annex; mv + $bad/.git/annex/objects $good/.git/annex/objects` +* Reinitalize git-annex: `cd $good; git annex init` +* Check for any problems with the annexed data: `cd $good; git annex fsck` +* Now you can remove the corrupted repository, the new one is ready to use. + +--[[Joey]] diff --git a/doc/tips/what_to_do_when_you_lose_a_repository.mdwn b/doc/tips/what_to_do_when_you_lose_a_repository.mdwn new file mode 100644 index 000000000..363eeea4e --- /dev/null +++ b/doc/tips/what_to_do_when_you_lose_a_repository.mdwn @@ -0,0 +1,19 @@ +So you lost a thumb drive containing a git-annex repository. Or a hard +drive died or some other misfortune has befallen your data. + +Unless you configured backups, git-annex can't get your data back. But it +can help you deal with the loss. + +Go somewhere that knows about the lost repository, and mark it as +dead: + + git annex dead usbdrive + +This retains the [[location_tracking]] information for the repository, +but avoids trying to access it, or list it as a location where files +are present. + +If you later found the drive, you could let git-annex know it's found +like so: + + git annex semitrust usbdrive diff --git a/doc/tips/what_to_do_when_you_lose_a_repository/comment_1_cf19b8dc304dc37c26717174c4a98aa4._comment b/doc/tips/what_to_do_when_you_lose_a_repository/comment_1_cf19b8dc304dc37c26717174c4a98aa4._comment new file mode 100644 index 000000000..a7fce26ef --- /dev/null +++ b/doc/tips/what_to_do_when_you_lose_a_repository/comment_1_cf19b8dc304dc37c26717174c4a98aa4._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="http://dlaxalde.myopenid.com/" + nickname="dl" + subject="comment 1" + date="2012-05-31T14:36:33Z" + content=""" +Is there a way to have git-annex completely ignore a repository? I see that +the `dead` command adds the uuid of the repository to `trust.log` but does +not change `uuid.log`. Is it enough to remove the corresponding line in +`uuid.log` and `trust.log`? +"""]] diff --git a/doc/tips/what_to_do_when_you_lose_a_repository/comment_3_fa9ca81668f5faebf2f61b10f82c97d2._comment b/doc/tips/what_to_do_when_you_lose_a_repository/comment_3_fa9ca81668f5faebf2f61b10f82c97d2._comment new file mode 100644 index 000000000..a8d044c28 --- /dev/null +++ b/doc/tips/what_to_do_when_you_lose_a_repository/comment_3_fa9ca81668f5faebf2f61b10f82c97d2._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.8.243" + subject="comment 3" + date="2012-05-31T17:01:37Z" + content=""" +`dead` is the best we can do. The automatic merging used on the git-annex branch tends to re-add lines that are deleted in one repo when merging with another that still has them. +"""]] diff --git a/doc/tips/yet_another_simple_disk_usage_like_utility.mdwn b/doc/tips/yet_another_simple_disk_usage_like_utility.mdwn new file mode 100644 index 000000000..961776e19 --- /dev/null +++ b/doc/tips/yet_another_simple_disk_usage_like_utility.mdwn @@ -0,0 +1,9 @@ +Here's the annex-du script that I use: + +#!/bin/sh +git annex find "$@" --include '*' --format='${bytesize}\n' |awk '{ sum += $1; nfiles++; } END { printf "%d files, %.3f MB\n", nfiles, sum/1000000 } ' + +This one can be slow on a large number of files, but it has an advantage of being able to use all of the filtering available in git annex find. +For example, to figure out how much is stored in remote X, do + +annex-du --in=X diff --git a/doc/tips/yet_another_simple_disk_usage_like_utility/comment_1_41b212bde8bc88d2a5dea93bd0dc75f1._comment b/doc/tips/yet_another_simple_disk_usage_like_utility/comment_1_41b212bde8bc88d2a5dea93bd0dc75f1._comment new file mode 100644 index 000000000..4c3e3c22b --- /dev/null +++ b/doc/tips/yet_another_simple_disk_usage_like_utility/comment_1_41b212bde8bc88d2a5dea93bd0dc75f1._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog" + nickname="Michael" + subject="comment 1" + date="2013-07-12T19:36:28Z" + content=""" +Ah, I just found that git annex info can do the same :) +Disregard this. +"""]] diff --git a/doc/tips/yet_another_simple_disk_usage_like_utility/comment_2_73698913837bfd5a58cf15721211e43e._comment b/doc/tips/yet_another_simple_disk_usage_like_utility/comment_2_73698913837bfd5a58cf15721211e43e._comment new file mode 100644 index 000000000..fe4b3d0d2 --- /dev/null +++ b/doc/tips/yet_another_simple_disk_usage_like_utility/comment_2_73698913837bfd5a58cf15721211e43e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89" + nickname="John" + subject="comment 2" + date="2013-08-30T06:09:29Z" + content=""" +You may want to try my `sizes` tool on Hackage. Just pass `-A` and it will be aware of the annex and report sizes as if no files were annexed. The only downside is that it reports file usage for replicated content multiple times, as if you'd copied the data out of the annex rather than hardlinked all duplicate copies (although, this may be exactly the behavior some people want). +"""]] |
