Youtube support! (And 53 other video hosts)

When quvi is installed, git-annex addurl automatically uses it to detect
when an page is a video, and downloads the video file.

web special remote: Also support using quvi, for getting files,
or checking if files exist in the web.

This commit was sponsored by Mark Hepburn. Thanks!

4 files changed, 58 insertions, 5 deletions

diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 00c7b2e50..7cac9087d 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -195,6 +195,9 @@ subdirectories).
   alternate locations from which the file can be downloaded. In this mode,
   addurl can be used both to add new files, or to add urls to existing files.
 
+  When quvi is installed, urls are automatically tested to see if they
+  are on a video hosting site, and the video is downloaded instead.
+
 * rmurl file url
 
   Record that the file is no longer available at the url.
@@ -1112,6 +1115,11 @@ Here are all the supported configuration settings.
   (wget is always used in preference to curl if available.)
   For example, to force ipv4 only, set it to "-4"
 
+* `annex.quvi-options`
+
+  Options to pass to quvi when using it to find the url to download for a
+  video.
+
 * `annex.http-headers`
 
   HTTP headers to send when downloading from the web. Multiple lines of
diff --git a/doc/install/fromscratch.mdwn b/doc/install/fromscratch.mdwn
index 63d4b1cbb..17444a4fd 100644
--- a/doc/install/fromscratch.mdwn
+++ b/doc/install/fromscratch.mdwn
@@ -11,6 +11,7 @@ quite a lot.
   * [monad-control](http://hackage.haskell.org/package/monad-control)
   * [QuickCheck 2](http://hackage.haskell.org/package/QuickCheck)
   * [json](http://hackage.haskell.org/package/json)
+  * [aeson](http://hackage.haskell.org/package/aeson)
   * [IfElse](http://hackage.haskell.org/package/IfElse)
   * [dlist](http://hackage.haskell.org/package/dlist)
   * [bloomfilter](http://hackage.haskell.org/package/bloomfilter)
diff --git a/doc/tips/using_the_web_as_a_special_remote.mdwn b/doc/tips/using_the_web_as_a_special_remote.mdwn
index 3ce02a56a..227d8e75c 100644
--- a/doc/tips/using_the_web_as_a_special_remote.mdwn
+++ b/doc/tips/using_the_web_as_a_special_remote.mdwn
@@ -8,10 +8,16 @@ The web can be used as a [[special_remote|special_remotes]] too.
 Now the file is downloaded, and has been added to the annex like any other
 file. So it can be renamed, copied to other repositories, and so on.
 
+To add a lot of urls at once, just list them all as parameters to
+`git annex addurl`.
+
+## trust issues
+
 Note that git-annex assumes that, if the web site does not 404, and has the
 right file size, the file is still present on the web, and this counts as
-one [[copy|copies]] of the file. So it will let you remove your last copy,
-trusting it can be downloaded again:
+one [[copy|copies]] of the file. If the file still seems to be present 
+on the web, it will let you remove your last copy, trusting it can be
+downloaded again:
 
 	# git annex drop example.com_video.mpeg
 	drop example.com_video.mpeg (checking http://example.com/video.mpeg) ok
@@ -31,7 +37,9 @@ With the result that it will hang onto files:
 	  (Use --force to override this check, or adjust annex.numcopies.)
 	failed
 
-You can also add urls to any file already in the annex:
+## attaching urls to existing files
+
+You can also attach urls to any file already in the annex:
 
 	# git annex addurl --file my_cool_big_file http://example.com/cool_big_file
 	addurl my_cool_big_file ok
@@ -40,8 +48,10 @@ You can also add urls to any file already in the annex:
   	00000000-0000-0000-0000-000000000001 -- web
    	27a9510c-760a-11e1-b9a0-c731d2b77df9 -- here
 
-To add a lot of urls at once, just list them all as parameters to
-`git annex addurl`. 
+## configuring filenames
+
+By default, `addurl` will generate a filename for you. You can use
+`--file=` to specify the filename to use.
 
 If you're adding a bunch of related files to a directory, or just don't
 like the default filenames generated by `addurl`, you can use `--pathdepth`
@@ -55,3 +65,35 @@ number takes that many paths from the end.
 	addurl 2012_01_video.mpeg (downloading http://example.com/videos/2012/01/video.mpeg)
 	# git annex addurl http://example.com/videos/2012/01/video.mpeg --pathdepth=-2
 	addurl 01_video.mpeg (downloading http://example.com/videos/2012/01/video.mpeg)
+
+## videos
+
+There's support for downloading videos from sites like YouTube, Vimeo,
+and many more. This relies on [quvi](http://quvi.sourceforge.net/) to find
+urls to the actual videos files.
+
+When you have quvi installed, you can just 
+`git annex addurl http://youtube.com/foo` and it will detect that
+it is a video and download the video content for offline viewing.
+
+Later, in another clone of the repository, you can run `git annex get` on
+the file and it will also be downloaded with the help of quvi. This works
+even if the video host has transcoded or otherwise changed the video
+in the meantime; the assumption is that these video files are equivilant.
+
+There is an `annex.quvi-options` configuration setting that can be used
+to pass parameters to quvi. For example, you could set `git config
+annex.quvi-options "--format low"` to configure it to download low
+quality videos from YouTube.
+
+Note that for performance reasons, the url is not checked for redirects,
+so shortened urls to sites like youtu.be will not be detected. You can
+either load the short url in a browser to get the full url, or you
+can force use of quvi with redirect detection, by prepending "quvi:" to the
+url. For example, `git annex addurl quvi:http://youtu.be/foo`
+
+Downloading whole YouTube playlists is not currently supported by quvi.
+
+## podcasts
+
+This is done using `git annex importfeed`. See [[downloading podcasts]].
diff --git a/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote.mdwn b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote.mdwn
index cfe07324e..229dc258b 100644
--- a/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote.mdwn
+++ b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote.mdwn
@@ -18,3 +18,5 @@ The [[Web special remote|special remotes/web]] could possibly be improved by det
 > > URL may yield different file contents depending on the quality
 > > chosen. Also, it seems that the URLs guessed by quvi may be
 > > ephemeral. --[[anarcat]]
+
+> [[done]]!!! --[[Joey]]