summaryrefslogtreecommitdiff
path: root/Command/ImportFeed.hs
Commit message (Collapse)AuthorAge
* toFeed was unused so removeGravatar Joey Hess2017-08-28
|
* Support building with feed-1.0, while still supporting older versions.Gravatar Joey Hess2017-08-28
| | | | This commit was sponsored by Jeff Goeke-Smith on Patreon.
* Drop support for building with old versions of directory, feed, and http-types.Gravatar Joey Hess2017-03-10
|
* improve display of checking known urlsGravatar Joey Hess2017-02-28
| | | | Display it as a separate action, so it ends with a newline
* remove old debug printGravatar Joey Hess2017-02-28
|
* Always use filesystem encoding for all file and handle reads and writes.Gravatar Joey Hess2016-12-24
| | | | | This is a big scary change. I have convinced myself it should be safe. I hope!
* analysisGravatar Joey Hess2016-12-19
|
* Avoid backtraces on expected failures when built with ghc 8; only use ↵Gravatar Joey Hess2016-11-15
| | | | | | | | | | | | | backtraces for unexpected errors. ghc 8 added backtraces on uncaught errors. This is great, but git-annex was using error in many places for a error message targeted at the user, in some known problem case. A backtrace only confuses such a message, so omit it. Notably, commands like git annex drop that failed due to eg, numcopies, used to use error, so had a backtrace. This commit was sponsored by Ethan Aubin.
* importfeed: Drop URL parameters from file extension.Gravatar Joey Hess2016-10-17
| | | | Thanks, James MacMahon.
* remove 3 build flagsGravatar Joey Hess2016-01-26
| | | | | | | | | | | | | * Removed the webapp-secure build flag, rolling it into the webapp build flag. * Removed the quvi and tahoe build flags, which only adds aeson to the core dependencies. * Removed the feed build flag, which only adds feed to the core dependencies. Build flags have cost in both code complexity and also make Setup configure have to work harder to find a usable set of build flags when some dependencies are missing.
* remove 163 lines of code without changing anything except importsGravatar Joey Hess2016-01-20
|
* use action, not sideActionGravatar Joey Hess2015-10-11
| | | | | | | | sideAction is for things not generally related to the current action being performed. And, it adds a newline after the side action. This was not the right thing to use for stuff like "checksum", where doing a checksum is part of the git annex get process, and indeed we want it to display "(checksum...) ok"
* ImportFeed: fix signature of mkquvi without libquviGravatar Ben Boeckel2015-08-23
|
* importfeed --relaxed: Avoid hitting the urls of items in the feed.Gravatar Joey Hess2015-08-19
|
* importfeed: Look at not only permalinks, but now also guids to identify ↵Gravatar Joey Hess2015-07-20
| | | | | | | | | | | | | | | | | previously downloaded files. I've seen rss feeds that have no permalinks, only guids (which are sometimes in the form of permalinks, argh/sigh). I had previously avoided trusting guids to be globally unique, because my survey of rss feeds that I subscribe to shows a lot of pretty bad "guids" like "2 at http://serialpodcast.org" or even worse "oth20150401-hq". Worry was that two podcasts that are generating guids so badly, that there's no guarantee they're actually globally unique. But, I'm seeing too many url changes that result in redundant files, so let's try this. If feeds are so broken that guids overlap, they could just as well incorrectly call them permalinks too.
* converted ImportFeedGravatar Joey Hess2015-07-13
|
* convert all commands to work with optparse-applicativeGravatar Joey Hess2015-07-08
| | | | Still no options though.
* started converting to use optparse-applicativeGravatar Joey Hess2015-07-08
| | | | | | | | | | | | This is a work in progress. It compiles and is able to do basic command dispatch, including git autocorrection, while using optparse-applicative for the core commandline parsing. * Many commands are temporarily disabled before conversion. * Options are not wired in yet. * cmdnorepo actions don't work yet. Also, removed the [Command] list, which was only used in one place.
* fromkey, registerurl: Allow urls to be specified instead of keys, and ↵Gravatar Joey Hess2015-05-22
| | | | | | | | generate URL keys. This is especially useful because the caller doesn't need to generate valid url keys, which involves some escaping of characters, and may involve taking a md5sum of the url if it's too long.
* more time-1.5 fixesGravatar Joey Hess2015-05-10
|
* Improve quvi 0.4 output parsing to handle cases wher there is no known ↵Gravatar Joey Hess2015-05-08
| | | | filename extension. This is currently the case when using quvi with youtube. In this case, the extension ".m" will be used.
* importfeed: Error out when passed a non-url.Gravatar Joey Hess2015-04-09
|
* importfeed: Fix feed download when curl is used.Gravatar Joey Hess2015-04-09
| | | | | This bug used to only afflict file:/// urls, but became more exposed by b1483ee577dded9e00f8d82d52a806ee02888b2c.
* importfeed: Avoid downloading a redundant item from a feed whose guid has ↵Gravatar Joey Hess2015-03-31
| | | | | | | been downloaded before, even when the url has changed. To support this, always store itemid in metadata; before this was only done when annex.genmetadata was set.
* addurl: Added --raw option, which bypasses special handling of quvi, ↵Gravatar Joey Hess2015-03-05
| | | | bittorrent etc urls.
* addurl: When a Content-Disposition header suggests a filename to use, addurl ↵Gravatar Joey Hess2015-01-22
| | | | will consider using it, if it's reasonable and doesn't conflict with an existing file. (--file overrides this)
* update my email address and homepage urlGravatar Joey Hess2015-01-21
|
* revert parentDir changeGravatar Joey Hess2015-01-09
| | | | | | | | Reverts 2bba5bc22d049272d3328bfa6c452d3e2e50e86c Unfortunately, this caused breakage on Windows, and possibly elsewhere, because parentDir and takeDirectory do not behave the same when there is a trailing directory separator.
* made parentDir return a Maybe FilePath; removed most uses of itGravatar Joey Hess2015-01-06
| | | | | | | | parentDir is less safe than takeDirectory, especially when working with relative FilePaths. It's really only useful in loops that want to terminate at / This commit was sponsored by Audric SCHILTKNECHT.
* move dummy uuids to Annex.UUIDGravatar Joey Hess2014-12-17
|
* sanitize filepaths provided by checkUrlGravatar Joey Hess2014-12-11
|
* importfeed: Support for downloading urls claimed by remotesGravatar Joey Hess2014-12-11
|
* add missing quvi ifdefGravatar Joey Hess2014-12-11
|
* doh't use "def" for command definitions, it conflicts with Data.Default.defGravatar Joey Hess2014-10-14
|
* fix some mixed space+tab indentationGravatar Joey Hess2014-10-09
| | | | | | | | | This fixes all instances of " \t" in the code base. Most common case seems to be after a "where" line; probably vim copied the two space layout of that line. Done as a background task while listening to episode 2 of the Type Theory podcast.
* fix build with old feedGravatar Joey Hess2014-07-19
|
* include feed url in repeated problems messageGravatar Joey Hess2014-07-14
|
* really add non-date metadata tooGravatar Joey Hess2014-07-03
|
* import metadata from feedsGravatar Joey Hess2014-07-03
| | | | | | | | | | | | | | | | | | | | | | When annex.genmetadata is set, metadata from the feed is added to files that are imported from it. Reused the same feedtitle and itemtitle, feedauthor, itemauthor, etc names that are used in --template. Also added title and author, which are the item title/author if available, falling back to the feed title/author. These are more likely to be common metadata fields. (There is a small bit of dupication here, but once git gets around to packing the object, it will compress it away.) The itempubdate field is not included in the metadata as a string; instead it is used to generate year and month fields, same as is done when adding files with annex.genmetadata set. This commit was sponsored by Amitai Schlair, who cooincidentially is responsible for ikiwiki generating nice feed metadata!
* allow building with old feed lib, w/o pubdateGravatar Joey Hess2014-04-21
|
* replace (Key, Backend) with KeyGravatar Joey Hess2014-04-17
| | | | | | | | | | Only fsck and reinject and the test suite used the Backend, and they can look it up as needed from the Key. This simplifies the code and also speeds it up. There is a small behavior change here. Before, all commands would warn when acting on an annexed file with an unknown backend. Now, only fsck and reinject show that warning.
* importfeed: Filename template can now contain an itempubdate variable. Needs ↵Gravatar Joey Hess2014-04-07
| | | | feed 0.3.9.2.
* Probe for quvi version at run time.Gravatar Joey Hess2014-02-28
| | | | | Overhead: git annex addurl runs quvi --version once. And more bloat to Annex state..
* add UrlOptions sum typeGravatar Joey Hess2014-02-24
|
* reorganize some files and importsGravatar Joey Hess2014-01-26
|
* fix inversion of control in CommandSeek (no behavior changes)Gravatar Joey Hess2014-01-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | I've been disliking how the command seek actions were written for some time, with their inversion of control and ugly workarounds. The last straw to fix it was sync --content, which didn't fit the Annex [CommandStart] interface well at all. I have not yet made it take advantage of the changed interface though. The crucial change, and probably why I didn't do it this way from the beginning, is to make each CommandStart action be run with exceptions caught, and if it fails, increment a failure counter in annex state. So I finally remove the very first code I wrote for git-annex, which was before I had exception handling in the Annex monad, and so ran outside that monad, passing state explicitly as it ran each CommandStart action. This was a real slog from 1 to 5 am. Test suite passes. Memory usage is lower than before, sometimes by a couple of megabytes, and remains constant, even when running in a large repo, and even when repeatedly failing and incrementing the error counter. So no accidental laziness space leaks. Wall clock speed is identical, even in large repos. This commit was sponsored by an anonymous bitcoiner.
* fix form of quvi urlGravatar Joey Hess2014-01-05
|
* check if quvi page url has already been downloaded beforeGravatar Joey Hess2014-01-05
| | | | | | That is the url that is typically stored in the git-annex branch. Kept the check for the video url too, just in case.
* importfeed: Support youtube playlists.Gravatar Joey Hess2013-12-29
|
* add readFileStrictAnyEncodingGravatar Joey Hess2013-11-20
|