summaryrefslogtreecommitdiff
path: root/Annex
Commit message (Collapse)AuthorAge
* reorgGravatar Joey Hess2017-12-14
|
* .noannex fileGravatar Joey Hess2017-12-13
| | | | | | | | | | | | | A top-level .noannex file will prevent git-annex init from being used in a repository. This is useful for repositories that have a policy reason not to use git-annex. The content of the file will be displayed to the user who tries to run git-annex init. This also affects git annex reinit and initialization via the webapp. It does not affect automatic inits, when there's a sibling git-annex branch already. This commit was supported by the NSF-funded DataLad project.
* avoid trying youtube-dl for ftp and file url schemesGravatar Joey Hess2017-12-11
| | | | This commit was sponsored by John Peloquin on Patreon.
* fix regression in addurl --fast caused by youtube-dl supportGravatar Joey Hess2017-12-08
| | | | | | | | | | | | | Similar to 9ec6bdfb526fa6b75a264b6417b24aa7f01adc25 but another code path. As well as using youtube-dl unecessarily, it used the filename it comes up with, which while nice for youtube videos, is not right for other files. This means more work is done for urls that youtube-dl does support, but is probably more efficient for other urls, since it only downloads the first chunk of content, while youtube-dl probably downloads more. This commit was supported by the NSF-funded DataLad project.
* fix regression in addurl --file caused by youtube-dl supportGravatar Joey Hess2017-12-06
| | | | | | | | | | | | | | | | | | Now youtubeDlCheck downloads the beginning of the url's content and checks if it's html, only when it is does it pass it off the youtube-dl to check if it supports it. This means more work is done for urls that youtube-dl does support, but is probably more efficient for other urls, since it only downloads the first chunk of content, while youtube-dl probably downloads more. As well as the reported bug, this also fixes behavior when an url was added with youtube-dl, but the url content has now changed from a html page to something else. Remote.Web.checkKey used to wrongly succeed in that situation, since youtube-dl said sure it can download that something else. This commit was supported by the NSF-funded DataLad project.
* avoid build warning when built w/o dbusGravatar Joey Hess2017-12-06
|
* more lambda-case conversionGravatar Joey Hess2017-12-05
|
* fix windows buildGravatar Joey Hess2017-12-05
|
* honor annex.diskreserve when running youtube-dlGravatar Joey Hess2017-11-30
| | | | This commit was sponsored by André Pereira on Patreon.
* check youtube-dl for --fast and --relaxed when adding new fileGravatar Joey Hess2017-11-30
| | | | | | The filename comes from youtube-dl also. This commit was sponsored by Denis Dzyubenko on Patreon.
* rethought --relaxed changeGravatar Joey Hess2017-11-30
| | | | | | | | | Better to make it not be surprising and slow, than surprising and fast. --raw can be used when it needs to be really fast. Implemented adding a youtube-dl supported url to an existing file. This commit was sponsored by andrea rota.
* avoid warning when youtube-dl is not installedGravatar Joey Hess2017-11-30
| | | | | If a user does not have it installed, don't warn on every imported item about it.
* fix buildGravatar Joey Hess2017-11-30
|
* pass git config options to youtube-dl --simulateGravatar Joey Hess2017-11-29
| | | | | | Decided not to --ignore-config by default. It the user has something in their youtube-dl config files that breaks git-annex they can configure it to use that option.
* convert importfeed to youtube-dlGravatar Joey Hess2017-11-29
| | | | | | | | | | | | | | | | | | | | | | | | Fully working, including --fast/--relaxed. Note that, while git-annex addurl --relaxed is not going to check youtube-dl, I kept git annex importfeed --relaxed checking it. Thinking is that, let's not break people's importfeed cron jobs, and importfeed does not typically have to check a large number of new items, so it's ok if it's a little bit slower when used with youtube playlist feeds. importfeed's behavior is also improved (?) when a feed has links in it to non-media files. Before, those were skipped. Now, the content of the link is downloaded. This had to be done, because trying to use youtube-dl is slow, and if those were skipped, it would have to check every time importfeed was run. While this behavior change may not be desirable for some feeds, that intersperse links to web pages with enclosures, it will be desirable for other feeds, that have non-enclosure directy links to media files. Remove old quvi modules. This commit was sponsored by Øyvind Andersen Holm.
* youtube-dl workingGravatar Joey Hess2017-11-29
| | | | | | | | | Including resuming and cleanup of incomplete downloads. Still todo: --fast, --relaxed, importfeed, disk reserve checking, quvi code cleanup. This commit was sponsored by Anthony DeRobertis on Patreon.
* add gitAnnexTmpWorkDir and withTmpWorkDirGravatar Joey Hess2017-11-29
| | | | | | | | | Needed to run youtube-dl in, but could also be useful for other stuff. The tricky part of this was making the workdir be cleaned up whenever the tmp object file is cleaned up. This commit was sponsored by Ole-Morten Duesund on Patreon.
* generalize notifyTransferGravatar Joey Hess2017-11-28
| | | | support not only AssociatedFile but also URLString
* typoGravatar Joey Hess2017-11-16
|
* enable LambdaCase and convert around 10% of places that could use itGravatar Joey Hess2017-11-15
| | | | | | | | | | | Needs ghc 7.6.1, so minimum base version increased slightly. All builds are well above this version of ghc, and debian oldstable is as well. Code that could use lambdacase can be found by running: git grep -B 1 'case ' | less and searching in less for "<-" This commit was sponsored by andrea rota.
* still can't express custom-setup depsGravatar Joey Hess2017-11-14
| | | | | | | | They need unix on non-windows, for Utility.Env, which Build.Configure uses, but cabal can't express that in a custom-setup stanza. To avoid this problem, Utility.Env would need to be moved into unix-compat..
* split out setEnv to avoid adding depGravatar Joey Hess2017-11-14
| | | | | | | | | | | Windows needs the setenv package in custom-setup, but I don't want to pull it in on unix, which would probably break some builds and need more work. Instead, split out setEnv to a separate module. Quite likely, unix-compat will get a portable environment layer, and then both modules can be removed from here. This commit was sponsored by Øyvind Andersen Holm.
* use unix-compat 0.5 on windowsGravatar Joey Hess2017-11-14
| | | | Re-applying ac57659e61f9743aebd35258e89752ced0040f9f
* Revert "use unix-compat 0.5 on windows"Gravatar Joey Hess2017-11-09
| | | | | | This reverts commit ac57659e61f9743aebd35258e89752ced0040f9f. Too early for this; needs newer Win32 version. Le sigh.
* use unix-compat 0.5 on windowsGravatar Joey Hess2017-11-09
| | | | | | | | | | | That version has my patches for the problems that Utility.PosixFiles was working around, so am able to get rid of that module now. This will later allow bringing back the custom-setup stanza in the cabal file. It will need to depend on unix-compat 0.5 on all OS's, which I'm not ready to do yet. This commit was sponsored by Nick Daly on Patreon.
* Fix exporting of non-annexed files to external special remotes.Gravatar Joey Hess2017-10-30
| | | | | | | | | | | | Actual problem is the keyName was set to "Ref \"sha\"", which led to this follow-on failure since it contained a space. The bad data would also get into the export database when exporting to a non-external special remote. Looking briefly at that, I don't think the bad data will lead to anything more than a re-upload of the file content now that the problem has been fixed. This commit was sponsored by Peter Hogg on Patreon.
* Add day to metadata when annex.genmetadata is enabled.Gravatar Joey Hess2017-10-25
| | | | Thanks, Sean T Parsons
* windows build fixGravatar Joey Hess2017-10-24
|
* better dup key with -J fixGravatar Joey Hess2017-10-17
| | | | | | | | | | | | | | | This avoids all the complication about redundant work discussed in the previous try at fixing this. At the expense of needing each command that could have the problem to be patched to simply wrap the action in onlyActionOn once the key is known. But there do not seem to be many such commands. onlyActionOn' should not be used with a CommandStart (or CommandPerform), although the types do allow it. onlyActionOn handles running the whole CommandStart chain. I couldn't immediately see a way to avoid mistken use of onlyActionOn'. This commit was supported by the NSF-funded DataLad project.
* Improve behavior when -J transfers multiple files that point to the same keyGravatar Joey Hess2017-10-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After a false start, I found a fairly non-intrusive way to deal with it. Although it only handles transfers -- there may be issues with eg concurrent dropping of the same key, or other operations. There is no added overhead when -J is not used, other than an added inAnnex check. When -J is used, it has to maintain and check a small Set, which should be negligible overhead. It could output some message saying that the transfer is being done by another thread. Or it could even display the same progress info for both files that are being downloaded since they have the same content. But I opted to keep it simple, since this is rather an edge case, so it just doesn't say anything about the transfer of the file until the other thread finishes. Since the deferred transfer action still runs, actions that do more than transfer content will still get a chance to do their other work. (An example of something that needs to do such other work is P2P.Annex, where the download always needs to receive the content from the peer.) And, if the first thread fails to complete a transfer, the second thread can resume it. But, this unfortunately means that there's a risk of redundant work being done to transfer a key that just got transferred. That's not ideal, but should never cause breakage; the same thing can occur when running two separate git-annex processes. The get/move/copy/mirror --from commands had extra inAnnex checks added, inside the download actions. Without those checks, the first thread downloaded the content, and then the second thread woke up and downloaded the same content redundantly. move/copy/mirror --to is left doing redundant uploads for now. It would need a second checkPresent of the remote inside the upload to avoid them, which would be expensive. A better way to avoid redundant work needs to be found.. This commit was supported by the NSF-funded DataLad project.
* add: Replace work tree file atomically.Gravatar Joey Hess2017-10-16
| | | | | | | Before, there was a window where interrupting an add could result in the file being moved into the annex, with no symlink yet created. This commit was supported by the NSF-funded DataLad project.
* fix process and FD leakGravatar Joey Hess2017-09-29
| | | | | | | | | | | | | Fix process and file descriptor leak that was exposed when git-annex was built with ghc 8.2.1. Apparently ghc has changed its behavior of GC of open file handles that are pipes to running processes. That broke git-annex test on OSX due to running out of FDs. Audited for all uses of Annex.new and made stopCoProcesses be called once it's done with the state. Fixed several places that might have leaked in other situations than running the test suite. This commit was sponsored by Ewen McNeill.
* test: Fix reversion that made it only run inside a git repository.Gravatar Joey Hess2017-09-29
| | | | | | | | Using annexeval to run probeCrippledFileSystem' caused Git.CurrentRepo.get to be run. Fixed easily since probeCrippledFileSystem' had no need to use the Annex monad. This commit was sponsored by Ethan Aubin.
* sync: Added --cleanup, which removes local and remote synced/ branches.Gravatar Joey Hess2017-09-28
| | | | | | | Also deletes any tagged pushes that the assistant might have done, since those would also prevent resetting a branch back. This commit was sponsored by andrea rota.
* Warn when metadata is inherited from a previous version of a fileGravatar Joey Hess2017-09-28
| | | | | | to avoid the user being surprised in cases where that behavior is not desired or expected This commit was supported by the NSF-funded DataLad project.
* fix build with old ghcGravatar Joey Hess2017-09-25
|
* git annex sync --content to exportsGravatar Joey Hess2017-09-19
| | | | | | Assistant still todo. This commit was sponsored by Boyd Stephen Smith Jr. on Patreon
* merge changes made on other repos into ExportTreeGravatar Joey Hess2017-09-18
| | | | | | | | | | | Now when one repository has exported a tree, another repository can get files from the export, after syncing. There's a bug: While the database update works, somehow the database on disk does not get updated, and so the database update is run the next time, etc. Wasn't able to figure out why yet. This commit was sponsored by Ole-Morten Duesund on Patreon.
* add ExportTree table to export dbGravatar Joey Hess2017-09-18
| | | | | | | | | | | | New table needed to look up what filenames are used in the currently exported tree, for reasons explained in export.mdwn. Also, added smart constructors for ExportLocation and ExportDirectory to make sure they contain filepaths with the right direction slashes. And some code refactoring. This commit was sponsored by Francois Marier on Patreon.
* lock to avoid more than one export to a remote at a timeGravatar Joey Hess2017-09-18
| | | | This commit was sponsored by Jack Hill on Patreon.
* fix compaction of export.logGravatar Joey Hess2017-09-12
| | | | | | | | It was not getting old lines removed, because the tree graft confused the updater, so it union merged from the previous git-annex branch, which still contained the old lines. Fixed by carefully using setIndexSha. This commit was supported by the NSF-funded DataLad project.
* prevent exporttree=yes on remotes that don't support exportsGravatar Joey Hess2017-09-07
| | | | | | | | | Don't allow "exporttree=yes" to be set when the special remote does not support exports. That would be confusing since the user would set up a special remote for exports, but `git annex export` to it would later fail. This commit was supported by the NSF-funded DataLad project.
* git annex get from exportsGravatar Joey Hess2017-09-04
| | | | | | | | | | | | | | Straightforward enough, except for the needed belt-and-suspenders sanity checks to avoid foot shooting due to exports not being key/value stores. * Even when annex.verify=false, always verify from exports. * Only get files from exports that use a backend that supports checksum verification. * Never trust exports, even if the user says to, because then `git annex drop` would drop content if the export seemed to contain a copy. This commit was supported by the NSF-funded DataLad project.
* track exported files in a sqlite databaseGravatar Joey Hess2017-09-04
| | | | | | | | | Went with a separate db per export remote, rather than a single export database. Mostly because there will probably not be a lot of separate export remotes, and it might be convenient to be able to delete a given remote's export database. This commit was supported by the NSF-funded DataLad project.
* implement exporttree=yes configurationGravatar Joey Hess2017-09-04
| | | | | | | | | | | | | | | | * Only export to remotes that were initialized to support it. * Prevent storing key/value on export remotes. * Prevent enabling exporttree=yes and encryption in the same remote. SetupStage Enable was changed to take the old RemoteConfig. This allowed only setting exporttree when initially setting up a remote, and not configuring it later after stuff might already be stored in the remote. Went with =yes rather than =true for consistency with other parts of git-annex. Changed docs accordingly. This commit was supported by the NSF-funded DataLad project.
* graft exported tree into git-annex branchGravatar Joey Hess2017-08-31
| | | | | | | | | | | So it will be available later and elsewhere, even after GC. I first though to use git update-index to do this, but feeding it a line with a tree object seems to always cause it to generate a git subtree merge. So, fell back to using the Git.Tree interface to maniupulate the trees, and not involving the git-annex branch index file at all. This commit was sponsored by Andreas Karlsson.
* init: Display an additional message when it detects a filesystem that allows ↵Gravatar Joey Hess2017-08-28
| | | | writing to files whose write bit is not set.
* avoid the dashed ssh hostname class of security holesGravatar Joey Hess2017-08-17
| | | | | | | | | | | | | | | | | | | | | | | | Security fix: Disallow hostname starting with a dash, which would get passed to ssh and be treated an option. This could be used by an attacker who provides a crafted ssh url (for eg a git remote) to execute arbitrary code via ssh -oProxyCommand. No CVE has yet been assigned for this hole. The same class of security hole recently affected git itself, CVE-2017-1000117. Method: Identified all places where ssh is run, by git grep '"ssh"' Converted them all to use a SshHost, if they did not already, for specifying the hostname. SshHost was made a data type with a smart constructor, which rejects hostnames starting with '-'. Note that git-annex already contains extensive use of Utility.SafeCommand, which fixes a similar class of problem where a filename starting with a dash gets passed to a program which treats it as an option. This commit was sponsored by Jochen Bartl on Patreon.
* migrate: WORM keys containing spaces will be migrated to not contain spaces ↵Gravatar Joey Hess2017-08-17
| | | | | | | | | anymore To work around the problem that the external special remote protocol does not support keys containing spaces. This commit was sponsored by Denis Dzyubenko on Patreon.
* Prevent spaces from being embedded in the name of new WORM keys, as that ↵Gravatar Joey Hess2017-08-17
| | | | handing spaces in keys would complicate things like the external special remote protocol.
generated by cgit v1.2.3 (git 2.39.1) at 2024-05-24 02:09:57 +0000