summaryrefslogtreecommitdiff
path: root/Backend/URL.hs
Commit message (Collapse)AuthorAge
* add KeyVariety typeGravatar Joey Hess2017-02-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Where before the "name" of a key and a backend was a string, this makes it a concrete data type. This is groundwork for allowing some varieties of keys to be disabled in file2key, so git-annex won't use them at all. Benchmarks ran in my big repo: old git-annex info: real 0m3.338s user 0m3.124s sys 0m0.244s new git-annex info: real 0m3.216s user 0m3.024s sys 0m0.220s new git-annex find: real 0m7.138s user 0m6.924s sys 0m0.252s old git-annex find: real 0m7.433s user 0m7.240s sys 0m0.232s Surprising result; I'd have expected it to be slower since it now parses all the key varieties. But, the parser is very simple and perhaps sharing KeyVarieties uses less memory or something like that. This commit was supported by the NSF-funded DataLad project.
* remove 163 lines of code without changing anything except importsGravatar Joey Hess2016-01-20
|
* rename fsckKey to verifyKeyContentGravatar Joey Hess2015-10-01
| | | | No behavior changes.
* fromkey, registerurl: Allow urls to be specified instead of keys, and ↵Gravatar Joey Hess2015-05-22
| | | | | | | | generate URL keys. This is especially useful because the caller doesn't need to generate valid url keys, which involves some escaping of characters, and may involve taking a md5sum of the url if it's too long.
* update my email address and homepage urlGravatar Joey Hess2015-01-21
|
* Generate shorter keys for WORM and URL, avoiding keys that are longer than ↵Gravatar Joey Hess2015-01-06
| | | | used for SHA256, so as to not break on systems like Windows that have very small maximum path length limits.
* add key stability checking interfaceGravatar Joey Hess2014-07-27
| | | | | | | Needed for resuming from chunks. Url keys are considered not stable. I considered treating url keys with a known size as stable, but just don't feel that is enough information.
* migrate: Avoid re-checksumming when migrating from hashE to hash backend.Gravatar Joey Hess2014-07-10
|
* Better sanitization of problem characters when generating URL and WORM keys.Gravatar Joey Hess2013-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | FAT has a lot of characters it does not allow in filenames, like ? and * It's probably the worst offender, but other filesystems also have limitiations. In 2011, I made keyFile escape : to handle FAT, but missed the other characters. It also turns out that when I did that, I was also living dangerously; any existing keys that contained a : had their object location change. Oops. So, adding new characters to escape to keyFile is out. Well, it would be possible to make keyFile behave differently on a per-filesystem basis, but this would be a real nightmare to get right. Consider that a rsync special remote uses keyFile to determine the filenames to use, and we don't know the underlying filesystem on the rsync server.. Instead, I have gone for a solution that is backwards compatable and simple. Its only downside is that already generated URL and WORM keys might not be able to be stored on FAT or some other filesystem that dislikes a character used in the key. (In this case, the user can just migrate the problem keys to a checksumming backend. If this became a big problem, fsck could be made to detect these and suggest a migration.) Going forward, new keys that are created will escape all characters that are likely to cause problems. And if some filesystem comes along that's even worse than FAT (seems unlikely, but here it is 2013, and people are still using FAT!), additional characters can be added to the set that are escaped without difficulty. (Also, made WORM limit the part of the filename that is embedded in the key, to deal with filesystem filename length limits. This could have already been a problem, but is more likely now, since the escaping of the filename can make it longer.) This commit was sponsored by Ian Downes
* Fix a few bugs involving filenames that are at or near the filesystem's ↵Gravatar Joey Hess2013-07-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | maximum filename length limit. Started with a problem when running addurl on a really long url, because the whole url is munged into the filename. Ended up doing a fairly extensive review for places where filenames could get too large, although it's hard to say I'm not missed any.. Backend.Url had a 128 character limit, which is fine when the limit is 255, but not if it's a lot shorter on some systems. So check the pathconf() limit. Note that this could result in fromUrl creating different keys for the same url, if run on systems with different limits. I don't see this is likely to cause any problems. That can already happen when using addurl --fast, or if the content of an url changes. Both Command.AddUrl and Backend.Url assumed that urls don't contain a lot of multi-byte unicode, and would fail to truncate an url that did properly. A few places use a filename as the template to make a temp file. While that's nice in that the temp file name can be easily related back to the original filename, it could lead to `git annex add` failing to add a filename that was at or close to the maximum length. Note that in Command.Add.lockdown, the template is still derived from the filename, just with enough space left to turn it into a temp file. This is an important optimisation, because the assistant may lock down a bunch of files all at once, and using the same template for all of them would cause openTempFile to iterate through the same set of names, looking for an unused temp file. I'm not very happy with the relatedTemplate hack, but it avoids that slowdown. Backend.WORM does not limit the filename stored in the key. I have not tried to change that; so git annex add will fail on really long filenames when using the WORM backend. It seems better to preserve the invariant that a WORM key always contains the complete filename, since the filename is the only unique material in the key, other than mtime and size. Since nobody has complained about add failing (I think I saw it once?) on WORM, probably it's ok, or nobody but me uses it. There may be compatability problems if using git annex addurl --fast or the WORM backend on a system with the 255 limit and then trying to use that repo in a system with a smaller limit. I have not tried to deal with those. This commit was sponsored by Alexander Brem. Thanks!
* handle sha*sum's leading \ in checksum with certian unsual filenamesGravatar Joey Hess2012-12-20
| | | | | | | | * Bugfix: Remove leading \ from checksums output by sha*sum commands, when the filename contains \ or a newline. Closes: #696384 * fsck: Still accept checksums with a leading \ as valid, now that above bug is fixed. * migrate: Remove leading \ in checksums
* where indentingGravatar Joey Hess2012-11-11
|
* separate source of content from the filename associated with the key when ↵Gravatar Joey Hess2012-06-05
| | | | | | generating a key This already made migrate's code a lot simpler.
* handle really long urlsGravatar Joey Hess2012-02-16
| | | | | Using the whole url as a key can make the filename too long. Truncate and use a md5sum for uniqueness if necessary.
* addurl --fast: Verifies that the url can be downloaded (only getting its ↵Gravatar Joey Hess2012-02-10
| | | | head), and records the size in the key.
* convert fsckKey to a MaybeGravatar Joey Hess2012-01-19
| | | | This way it's clear when a backend does not implement its own fsck checks.
* type alias cleanupGravatar Joey Hess2011-12-31
|
* renameGravatar Joey Hess2011-10-05
|
* renameGravatar Joey Hess2011-10-04
|
* factor out common importsGravatar Joey Hess2011-10-03
| | | | no code changes
* addurl: --fast can be used to avoid immediately downloading the url.Gravatar Joey Hess2011-08-06
| | | | | | The tricky part about this is that to generate a key, the file must be present already. Worked around by adding (back) an URL key type, which is used for addurl --fast.
* remove URL backendGravatar Joey Hess2011-07-01
|
* rename modules for data types into Types/ directoryGravatar Joey Hess2011-06-01
|
* migrate: Support migrating v1 SHA keys to v2 SHA keys with size information ↵Gravatar Joey Hess2011-03-23
| | | | that can be used for free space checking.
* Fix dropping of files using the URL backend.Gravatar Joey Hess2011-03-17
|
* fromkey, and url backend download work nowGravatar Joey Hess2011-03-15
|
* rename fileGravatar Joey Hess2011-03-15
|
* only enable SHA backends that configure finds support forGravatar Joey Hess2011-03-02
|
* use ShellParam typeGravatar Joey Hess2011-02-28
| | | | | So, I have a type checked safe handling of filenames starting with dashes, throughout the code.
* Support filenames that start with a dash; when such a file is passed to a ↵Gravatar Joey Hess2011-02-25
| | | | utility it will be escaped to avoid it being interpreted as an option.
* make filename available to fsck messagesGravatar Joey Hess2011-01-26
|
* rename TypeInternals to BackendTypesGravatar Joey Hess2011-01-26
| | | | Now that it only contains types used by the backends
* successfully split Annex and AnnexState out of TypeInternalsGravatar Joey Hess2011-01-25
|
* parameterize Backend typeGravatar Joey Hess2011-01-25
| | | | | This allows the Backend type to not depend on the Annex type, and so the Annex type can later be moved out of TypeInternals.
* In .gitattributes, the git-annex-numcopies attribute can be used to control ↵Gravatar Joey Hess2010-11-28
| | | | the number of copies to retain of different types of files.
* fsck improvementsGravatar Joey Hess2010-11-13
| | | | | | | | | * fsck: Check if annex.numcopies is satisfied. * fsck: Verify the sha1 of files when the SHA1 backend is used. * fsck: Verify the size of files when the WORM backend is used. * fsck: Allow specifying individual files to fsk if fscking everything is not desired. * fsck: Fix bug, introduced in 0.04, in detection of unused data.
* refactoring, no code changes reallyGravatar Joey Hess2010-11-08
|
* more Wall cleaningGravatar Joey Hess2010-10-31
|
* avoid unnessary newlines before progress in quiet modeGravatar Joey Hess2010-10-29
|
* convert safeSystem to boolSystemGravatar Joey Hess2010-10-29
| | | | to fix ctrl-c handling
* copyright statements0.02Gravatar Joey Hess2010-10-27
|
* use safesystemGravatar Joey Hess2010-10-19
|
* gratuitous renameGravatar Joey Hess2010-10-18
|
* cleanup outputGravatar Joey Hess2010-10-17
|
* tweaksGravatar Joey Hess2010-10-16
|
* rename backends moreGravatar Joey Hess2010-10-15