aboutsummaryrefslogtreecommitdiff
path: root/Locations.hs
Commit message (Collapse)AuthorAge
* optimise ObjectHashLower handlingGravatar Joey Hess2015-06-12
| | | | | | | A repo tuned with ObjectHashLower only ever uses lower-case hash directories, so unlike a bare repo which could have old mixed case directories, there is no need to do an expensive check of the filesystem in this case.
* Fix bug that prevented enumerating locally present objects in repos tuned ↵Gravatar Joey Hess2015-06-11
| | | | | | | | with annex.tune.objecthash1=true Need to walk 1 level of subdirs less in this case. The git-annex branch traversal code didn't have a similar bug.
* a few hlintsGravatar Joey Hess2015-04-11
|
* fixup annex link target calculation when submodules are used in filesystems ↵Gravatar Joey Hess2015-03-04
| | | | not supporting symlinks
* avoid checking location of content when calculating gitAnnexLinkGravatar Joey Hess2015-03-04
| | | | | | | It doesn't matter if the object is present or not, gitAnnexLink should always yield the same symlink target. This is an optimisation; no behavior should be changed.
* use WAL mode to ensure read from db always works, even when it's being ↵Gravatar Joey Hess2015-02-18
| | | | | | | | | | | | | | | | written to Also, moved the database to a subdir, as there are multiple files. This seems to work well with concurrent fscks, although they still do redundant work due to the commit granularity. Occasionally two writes will conflict, and one is then deferred and happens later. Except, with 3 concurrent fscks, I got failures: git-annex: user error (SQLite3 returned ErrorBusy while attempting to perform prepare "SELECT \"fscked\".\"key\"\nFROM \"fscked\"\nWHERE \"fscked\".\"key\" = ?\n": database is locked) Argh!!!
* fsck: Multiple incremental fscks of different repos (some remote) can now be ↵Gravatar Joey Hess2015-02-17
| | | | in progress at the same time in the same repo without it getting confused about which files have been checked for which remotes.
* avoid crash when starting fsck --incremental when one is already runningGravatar Joey Hess2015-02-17
| | | | | | | | | | | Turns out sqlite does not like having its database deleted out from underneath it. It might suffice to empty the table, but I would rather start each fsck over with a new database, so I added a lock file, and running incremental fscks use a shared lock. This leaves one concurrency bug left; running two concurrent fsck --more will lead to: "SQLite3 returned ErrorBusy while attempting to perform step." and one or both will fail. This is a concurrent writers problem.
* convert incremental fsck to using sqlite databaseGravatar Joey Hess2015-02-16
| | | | | | | | | | | | | | | | | Did not keep backwards compat for sticky bit records. An incremental fsck that is already in progress will start over on upgrade to this version. This is not yet ready for merging. The autobuilders need to have sqlite installed. Also, interrupting a fsck --incremental does not commit the database. So, resuming with fsck --more restarts from beginning. Memory: Constant during a fsck of tens of thousands of files. (But, it does seem to buffer whole transation in memory, so may really scale with number of files.) CPU: ?
* Windows: Fix bug in dropping an annexed file, which caused a symlink to be ↵Gravatar Joey Hess2015-02-09
| | | | staged that contained backslashes.
* use a SetGravatar Joey Hess2015-01-28
|
* test suite found a problem with today's workGravatar Joey Hess2015-01-28
| | | | ". def" did not do what I thought it would, at all.
* implement annex.tune.objecthashlowerGravatar Joey Hess2015-01-28
| | | | Split out Annex.DirHashes which never really belonged in Locations.
* groundwork for parameterizing hash depthGravatar Joey Hess2015-01-28
|
* rework Differences data typeGravatar Joey Hess2015-01-28
| | | | | | | | | | | | | | Eliminated complexity and future proofed. The most important change is that all functions over Difference are now total; any Difference that can be expressed should be handled. Avoids needs for sanity checking of inputs, and version skew with the future. Also, the difference.log now serializes a [Difference], not a Differences. This saves space and keeps it simpler. Note that [Difference] might contain conflicting differences (eg, [Version5, Version6]. In this case, one of them needs to consistently win over the others, probably based on Ord.
* Repository tuning parameters can now be passed when initializing a ↵Gravatar Joey Hess2015-01-27
| | | | | | | | | | repository for the first time. * init: Repository tuning parameters can now be passed when initializing a repository for the first time. For details, see http://git-annex.branchable.com/tuning/ * merge: Refuse to merge changes from a git-annex branch of a repo that has been tuned in incompatable ways.
* Revert "remove absNormPathUnix, using my absPathFrom replacement"Gravatar Joey Hess2015-01-21
| | | | | | | | | This reverts commit b6b368ed036f2e34ee4b7d39e5b41b1ba2d0a76c. Consider: relPathDirToFile (absPathFrom "/tmp/repo/xxx" "y/bar") "/tmp/repo/.git/annex/objects/xxx" This needs to always yield "../../../.git/annex/objects/xxx" but on Windows, it is "..\\..\\/tmp/repo/.git/annex/objects/xxx"
* remove absNormPathUnix, using my absPathFrom replacementGravatar Joey Hess2015-01-21
|
* update my email address and homepage urlGravatar Joey Hess2015-01-21
|
* revert parentDir changeGravatar Joey Hess2015-01-09
| | | | | | | | Reverts 2bba5bc22d049272d3328bfa6c452d3e2e50e86c Unfortunately, this caused breakage on Windows, and possibly elsewhere, because parentDir and takeDirectory do not behave the same when there is a trailing directory separator.
* Merge branch 'master' into relativepathsGravatar Joey Hess2015-01-06
|\ | | | | | | | | | | Conflicts: Locations.hs debian/changelog
| * made parentDir return a Maybe FilePath; removed most uses of itGravatar Joey Hess2015-01-06
| | | | | | | | | | | | | | | | parentDir is less safe than takeDirectory, especially when working with relative FilePaths. It's really only useful in loops that want to terminate at / This commit was sponsored by Audric SCHILTKNECHT.
* | Switch to using relative paths to the git repository.Gravatar Joey Hess2015-01-06
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the git repository to be moved while git-annex is running in it, with fewer problems. On Windows, this avoids some of the problems with the absurdly small MAX_PATH of 260 bytes. In particular, git-annex repositories should work in deeper/longer directory structures than before. See http://git-annex.branchable.com/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/ There are several possible ways this change could break git-annex: 1. If it changes its working directory while it's running, that would be Bad News. Good news everyone! git-annex never does so. It would also break thread safety, so all such things were stomped out long ago. 2. parentDir "." -> "" which is not a valid path. I had to fix one instace of this, and I should probably wipe all calls to parentDir out of the git-annex code base; it was never a good idea. 3. Things like relPathDirToFile require absolute input paths, and code assumes that the git repo path is absolute and passes it to it as-is. In the case of relPathDirToFile, I converted it to not make this assumption. Currently, the test suite has 16 failures.
* fix some mixed space+tab indentationGravatar Joey Hess2014-10-09
| | | | | | | | | This fixes all instances of " \t" in the code base. Most common case seems to be after a "where" line; probably vim copied the two space layout of that line. Done as a background task while listening to episode 2 of the Type Theory podcast.
* use same hash directories for chunked key as are used for its parentGravatar Joey Hess2014-07-25
| | | | | | | | | | | This avoids a proliferation of hash directories when using new-style chunking, and should improve performance since chunks are accessed in sequence and so should have a common locality. Of course, when a chunked key is encrypted, its hash directories have no relation to the parent key. This commit was sponsored by Christian Kellermann.
* prospective fix for bad_merge_commit_deleting_all_filesGravatar Joey Hess2014-07-09
| | | | | Assuming my analysis of a race is correct. In any case, this certianly closes a race..
* export CreateProcess fields from Utility.ProcessGravatar Joey Hess2014-06-10
| | | | update code to avoid cwd and env redefinition warnings
* unannex, uninit: Avoid committing after every file is unannexed, for massive ↵Gravatar Joey Hess2014-03-21
| | | | | | | | | speedup. pre-commit hook lock added, so unannex can prevent the hook from running in a confusing state. This commit was sponsored by Fredrik Hammar
* use https when .git/annex/privkey.pem and .git/annex/certificate.pem exist ↵Gravatar Joey Hess2014-02-28
| | | | | | | (untested) I have not managed to generate a key that is accepted by the old version of warp-tls I have here.
* Put non-object tmp files in .git/annex/misctmp, leaving .git/annex/tmp for ↵Gravatar Joey Hess2014-02-26
| | | | | | | | | | | | | | | | | | | | only partially transferred objects. This allows eg, putting .git/annex/tmp on a ram disk, if the disk IO of temp object files is too annoying (and if you don't want to keep partially transferred objects across reboots). .git/annex/misctmp must be on the same filesystem as the git work tree, since files are moved to there in a way that will not work cross-device, as well as symlinked into there. I first wanted to put the tmp objects in .git/annex/objects/tmp, but that would pose transition problems on upgrade when partially transferred objects existed. git annex info does not currently show the size of .git/annex/misctemp, since it should stay small. It would also be ok to make something clean it out, periodically.
* add git annex view commandGravatar Joey Hess2014-02-18
| | | | | | | | | | | | | | | | | | (And a vpop command, which is still a bit buggy.) Still need to do vadd and vrm, though this also adds their documentation. Currently not very happy with the view log data serialization. I had to lose the TDFA regexps temporarily, so I can have Read/Show instances of View. I expect the view log format will change in some incompatable way later, probably adding last known refs for the parent branch to View or something like that. Anyway, it basically works, although it's a bit slow looking up the metadata. The actual git branch construction is about as fast as it can be using the current git plumbing. This commit was sponsored by Peter Hogg.
* more hlintGravatar Joey Hess2014-02-11
|
* work around absNormPath not working on WindowsGravatar Joey Hess2014-02-06
| | | | When making git-annex links, we want unix-style paths in the link targets.
* rework annexed object locking in direct mode & support WindowsGravatar Joey Hess2014-01-28
| | | | | | | | | | | | | | | | Seems that locking of annexed objects when they're being dropped was broken in direct mode: * When taking the lock before dropping, it created the .git/annex/objects file, as an empty file. It seems that the dropping code deleted that, but that is not right, and for all I know could in some situation cause a corrupted object to leak out. * When the lock was checked, it actually tried to open each direct mode file, and checked if it was locked. Not the same lock used above, and could also fail if some consumer of the file locked it. Fixed this, and added windows support by switching direct mode to lock a .lck file.
* add git fsck to cronner, and UI for repository repair (not yet wired up)Gravatar Joey Hess2013-10-22
|
* tighten file2key to not produce invalid keys with no keyNameGravatar Joey Hess2013-10-16
| | | | | | A file named "foo-" or "foo-bar" was taken as a key's file, with a backend of "foo", and an empty keyName. This led to various problems, especially because converting that key back to a file did not yeild the same filename.
* half way complete cronner thread to run scheduled activitiesGravatar Joey Hess2013-10-08
|
* assistant: Detect stale git lock files at startup time, and remove them.Gravatar Joey Hess2013-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Extends the index.lock handling to other git lock files. I surveyed all lock files used by git, and found more than I expected. All are handled the same in git; it leaves them open while doing the operation, possibly writing the new file content to the lock file, and then closes them when done. The gc.pid file is excluded because it won't affect the normal operation of the assistant, and waiting for a gc to finish on startup wouldn't be good. All threads except the webapp thread wait on the new startup sanity checker thread to complete, so they won't try to do things with git that fail due to stale lock files. The webapp thread mostly avoids doing that kind of thing itself. A few configurators might fail on lock files, but only if the user is explicitly trying to run them. The webapp needs to start immediately when the user has opened it, even if there are stale lock files. Arranging for the threads to wait on the startup sanity checker was a bit of a bear. Have to get all the NotificationHandles set up before the startup sanity checker runs, or they won't see its signal. Perhaps the NotificationBroadcaster is not the best interface to have used for this. Oh well, it works. This commit was sponsored by Michael Jakl
* Better sanitization of problem characters when generating URL and WORM keys.Gravatar Joey Hess2013-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | FAT has a lot of characters it does not allow in filenames, like ? and * It's probably the worst offender, but other filesystems also have limitiations. In 2011, I made keyFile escape : to handle FAT, but missed the other characters. It also turns out that when I did that, I was also living dangerously; any existing keys that contained a : had their object location change. Oops. So, adding new characters to escape to keyFile is out. Well, it would be possible to make keyFile behave differently on a per-filesystem basis, but this would be a real nightmare to get right. Consider that a rsync special remote uses keyFile to determine the filenames to use, and we don't know the underlying filesystem on the rsync server.. Instead, I have gone for a solution that is backwards compatable and simple. Its only downside is that already generated URL and WORM keys might not be able to be stored on FAT or some other filesystem that dislikes a character used in the key. (In this case, the user can just migrate the problem keys to a checksumming backend. If this became a big problem, fsck could be made to detect these and suggest a migration.) Going forward, new keys that are created will escape all characters that are likely to cause problems. And if some filesystem comes along that's even worse than FAT (seems unlikely, but here it is 2013, and people are still using FAT!), additional characters can be added to the set that are escaped without difficulty. (Also, made WORM limit the part of the filename that is embedded in the key, to deal with filesystem filename length limits. This could have already been a problem, but is more likely now, since the escaping of the filename can make it longer.) This commit was sponsored by Ian Downes
* move some code aroundGravatar Joey Hess2013-10-05
|
* rename confusing functionGravatar Joey Hess2013-10-03
| | | | | The index.lck file is not a lock file. Kept the historical name for now as changing it would be work.
* lockJournal when running performTransitionsGravatar Joey Hess2013-10-03
| | | | | | | This may not strictly be needed -- the transition code bypasses the journal. However, this ensures that the git-annex branch is only committed with the journal locked. This will allow for further improvements.
* git-annex-shell: Added support for operating inside gcrypt repositories.Gravatar Joey Hess2013-09-24
| | | | | | * Note that the layout of gcrypt repositories has changed, and if you created one you must manually upgrade it. See http://git-annex.branchable.com/upgrades/gcrypt/
* untested transition detection on merging, and transition running codeGravatar Joey Hess2013-08-28
|
* importfeed: Ignores transient problems with feeds. Only exits nonzero when a ↵Gravatar Joey Hess2013-08-03
| | | | feed has repeatedly had a problems for at least 1 day.
* fuzz testerGravatar Joey Hess2013-05-23
|
* fix the day's windows permissions damageGravatar Joey Hess2013-05-12
|
* fix path separatorGravatar Joey Hess2013-05-12
|
* Use lower case hash directories for storing files on crippled filesystems, ↵Gravatar Joey Hess2013-04-04
| | | | | | | | | | | | | | | same as is already done for bare repositories. * since this is a crippled filesystem anyway, git-annex doesn't use symlinks on it * so there's no reason to use the mixed case hash directories that we're stuck using to avoid breaking everyone's symlinks to the content * so we can do what is already done for all bare repos, and make non-bare repos on crippled filesystems use the all-lower case hash directories * which are, happily, all 3 letters long, so they cannot conflict with mixed case hash directories * so I was able to 100% fix this and even resuming `git annex add` in the test case will recover and it will all just work.
* Update working tree files fully atomicallyGravatar Joey Hess2013-04-02
| | | | | | | | | | | This avoids commit churn by the assistant when eg, replacing a file with a symlink. But, just as importantly, it prevents the working tree being left with a deleted file if git-annex, or perhaps the whole system, crashes at the wrong time. (It also probably avoids confusing displays in file managers.)