summaryrefslogtreecommitdiff
path: root/Annex/Branch.hs
Commit message (Collapse)AuthorAge
* improve alwayscommit=false modeGravatar Joey Hess2012-02-25
| | | | | | | | | | | | | | Now changes are staged into the branch's index, but not committed, which avoids growing a large journal. And sync and merge always explicitly commit, ensuring that even when they do nothing else, they commit the staged changes. Added a flag file to indicate that the branch's journal contains uncommitted changes. (Could use git ls-files, but don't want to run that every time.) In the future, this ability to have uncommitted changes staged in the journal might be used on remotes after a series of oneshot commands.
* hlintGravatar Joey Hess2012-02-16
|
* tweakGravatar Joey Hess2012-02-14
|
* fix memory leak when staging the journalGravatar Joey Hess2012-02-14
| | | | | | The list of files had to be retained until the end so it could be deleted. Also, a list of update-index lines was generated and only then fed into it. Now everything streams in constant space.
* Fixed a memory leak due to excessive strictness when committing journal files.Gravatar Joey Hess2012-02-14
| | | | | | When hashing the files, the entire list of shas was read strictly. That was entirely unnecessary, since there's a cleanup action run after they're consumed.
* avoid multiple unnecessary stats of the index fileGravatar Joey Hess2012-01-14
| | | | Up to one per file processed.
* tweaksGravatar Joey Hess2012-01-11
|
* log: New command that displays the location log for file, showing each ↵Gravatar Joey Hess2012-01-06
| | | | | | | | | | | | | | | repository they were added to and removed from. This needs to run git log on the location log files to get at all past versions of the file, which tends to be a bit slow. It would be possible to make a version optimised for showing the location logs for every key. That would only need to run git log once, so would be faster, but it would need to process an enormous amount of data, so would not speed up the individual file case. In the future it would be nice to support log --format. log --json also doesn't work right yet.
* refactorGravatar Joey Hess2011-12-30
|
* add forceUpdateGravatar Joey Hess2011-12-30
| | | | | This code is picked from my tweak-fetch branch, which already did the needed refactoring.
* split out Git/Command.hsGravatar Joey Hess2011-12-14
|
* split more stuff out of Git.hsGravatar Joey Hess2011-12-14
|
* move commit to Git.BranchGravatar Joey Hess2011-12-13
|
* avoid closing pipe before all the shas are read from itGravatar Joey Hess2011-12-12
| | | | | | | Could have just used hGetContentsStrict here, but that would require storing all the shas in memory. Since this is called at the end of a git-annex run, it may have created a *lot* of shas, so I avoid that memory use and stream them out like before.
* broke out Git/HashObject.hsGravatar Joey Hess2011-12-12
|
* broke out Git/Branch.hs and reorganizedGravatar Joey Hess2011-12-12
|
* split out Git/Ref.hsGravatar Joey Hess2011-12-12
|
* split out Annex/Journal.hsGravatar Joey Hess2011-12-12
|
* split out Annex/BranchState.hsGravatar Joey Hess2011-12-12
|
* update commentGravatar Joey Hess2011-12-12
|
* optimisationGravatar Joey Hess2011-12-12
| | | | avoids a redundant call to git show-ref
* optimisationGravatar Joey Hess2011-12-12
| | | | avoids a useless diff from git-annex..refs/heads/git-annex
* cleanupGravatar Joey Hess2011-12-12
|
* avoid redundant call to updateIndexGravatar Joey Hess2011-12-11
| | | | commitBranch calls updateIndex
* detect and recover from branch push/commit raceGravatar Joey Hess2011-12-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Dealing with a race without using locking is exceedingly difficult and tricky. Fully tested, I hope. There are three places left where the branch can be updated, that are not covered by the race recovery code. Let's prove they're all immune to the race: 1. tryFastForwardTo checks to see if a fast-forward can be done, and then does git-update-ref on the branch to fast-forward it. If a push comes in before the check, then either no fast-forward will be done (ok), or the push set the branch to a ref that can still be fast-forwarded (also ok) If a push comes in after the check, the git-update-ref will undo the ref change made by the push. It's as if the push did not come in, and the next git-push will see this, and try to re-do it. (acceptable) 2. When creating the branch for the very first time, an empty index is created, and a commit of it made to the branch. The commit's ref is recorded as the current state of the index. If a push came in during that, it will be noticed the next time a commit is made to the branch, since the branch will have changed. (ok) 3. Creating the branch from an existing remote branch involves making the branch, and then getting its ref, and recording that the index reflects that ref. If a push creates the branch first, git-branch will fail (ok). If the branch is created and a racing push is then able to change it (highly unlikely!) we're still ok, because it first records the ref into the index.lck, and then updating the index. The race can cause the index.lck to have the old branch ref, while the index has the newly pushed branch merged into it, but that only results in an unnecessary update of the index file later on.
* optimize index updatingGravatar Joey Hess2011-12-11
| | | | | | | | | | | The last branch ref that the index was updated to is stored in .git/annex/index.lck, and the index only updated when the current branch ref differs. (The .lck file should later be used for locking too.) Some more optimization is still needed, since there is some redundancy in calls to git show-ref.
* slow, stupid, and safe index updatingGravatar Joey Hess2011-12-11
| | | | | | | | | | | | | | | | | | Always merge the git-annex branch into .git/annex/index before making a commit from the index. This ensures that, when the branch has been changed in any way (by a push being received, or changes pulled directly into it, or even by the user checking it out, and committing a change), the index reflects those changes. This is much too slow; it needs to be optimised to only update the index when the branch has really changed, not every time. Also, there is an unhandled race, when a change is made to the branch right after the index gets updated. I left it in for now because it's unlikely and I didn't want to complicate things with additional locking yet.
* move a file location to Locations.hsGravatar Joey Hess2011-12-11
|
* no need to show, it's a stringGravatar Joey Hess2011-12-10
|
* hslintGravatar Joey Hess2011-12-09
|
* cleanupGravatar Joey Hess2011-11-30
|
* support .git/annex on a different disk than the rest of the repoGravatar Joey Hess2011-11-28
| | | | | | | | | | | | | | | | | | The only fully supported thing is to have the main repository on one disk, and .git/annex on another. Only commands that move data in/out of the annex will need to copy it across devices. There is only partial support for putting arbitrary subdirectories of .git/annex on different devices. For one thing, but this can require more copies to be done. For example, when .git/annex/tmp is on one device, and .git/annex/journal on another, every journal write involves a call to mv(1). Also, there are a few places that make hard links between various subdirectories of .git/annex with createLink, that are not handled. In the common case without cross-device, the new moveFile is actually faster than renameFile, avoiding an unncessary stat to check that a file (not a directory) is being moved. Of course if a cross-device move is needed, it is as slow as mv(1) of the data.
* ensure branch exists before trying to update itGravatar Joey Hess2011-11-16
| | | | | | The branch may not exist, if .git/annex has been copied over from another repo (or a corrupted repo). I suppose it could also have gotten deleted somehow. Without this, there is a confusing failure.
* improve type signatures with a Ref newtypeGravatar Joey Hess2011-11-16
| | | | | | | | | | | In git, a Ref can be a Sha, or a Branch, or a Tag. I added type aliases for those. Note that this does not prevent mixing up of eg, refs and branches at the type level. Since git really doesn't care, except rare cases like git update-ref, or git tag -d, that seems ok for now. There's also a tree-ish, but let's just use Ref for it. A given Sha or Ref may or may not be a tree-ish, depending on the object type, so there seems no point in trying to represent it at the type level.
* better nameGravatar Joey Hess2011-11-16
|
* merge: Now runs in constant space.Gravatar Joey Hess2011-11-15
| | | | | | | | | | | | | | | Before, a merge was first calculated, by running various actions that called git and built up a list of lines, which were at the end sent to git update-index. This necessarily used space proportional to the size of the diff between the trees being merged. Now, lines are streamed into git update-index from each of the actions in turn. Runtime size of git-annex merge when merging 50000 location log files drops from around 100 mb to a constant 4 mb. Presumably it runs quite a lot faster, too.
* Optimised union merging; now only runs git cat-file once.Gravatar Joey Hess2011-11-12
|
* avoid unnecessary auto-merge when only changing a file in the branch.Gravatar Joey Hess2011-11-12
| | | | | | | | | | | | | | | | | | | | | | | | | | | Avoids doing auto-merging in commands that don't need fully current information from the git-annex branch. In particular, git annex add no longer needs to auto-merge. Affected commands: Anything that doesn't look up data from the branch, but does write a change to it. It might seem counterintuitive that we can change a value without first making sure we have the current value. This optimisation works because these two sequences are equivilant: 1. pull from remote 2. union merge 3. read file from branch 4. modify file and write to branch vs. 1. read file from branch 2. modify file and write to branch 3. pull from remote 4. union merge After either sequence, the git-annex branch contains the same logical content for the modified file. (Possibly with lines in a different order or additional old lines of course).
* merge: Improve commit messages to mention what was merged.Gravatar Joey Hess2011-11-12
|
* lintGravatar Joey Hess2011-11-11
|
* factored out some useful error catching methodsGravatar Joey Hess2011-11-10
|
* reorder repo parameters lastGravatar Joey Hess2011-11-08
| | | | | | | | | | | | | Many functions took the repo as their first parameter. Changing it consistently to be the last parameter allows doing some useful things with currying, that reduce boilerplate. In particular, g <- gitRepo is almost never needed now, instead use inRepo to run an IO action in the repo, and fromRepo to get a value from the repo. This also provides more opportunities to use monadic and applicative combinators.
* optimizationGravatar Joey Hess2011-11-06
| | | | | The last commit added some git-log calls to a merge. This removes some, by only merging branches that have unique refs.
* merge: Use fast-forward merges when possible.Gravatar Joey Hess2011-11-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Thanks Valentin Haenel for a test case showing how non-fast-forward merges could result in an ongoing pull/merge/push cycle. While the git-annex branch is fast-forwarded, git-annex's index file is still updated using the union merge strategy as before. There's no other way to update the index that would be any faster. It is possible that a union merge and a fast-forward result in different file contents: Files should have the same lines, but a union merge may change their order. If this happens, the next commit made to the git-annex branch will have some unnecessary changes to line orders, but the consistency of data should be preserved. Note that when the journal contains changes, a fast-forward is never attempted, which is fine, because committing those changes would be vanishingly unlikely to leave the git-annex branch at a commit that already exists in one of the remotes. The real difficulty is handling the case where multiple remotes have all changed. git-annex does find the best (ie, newest) one and fast forwards to it. If the remotes are diverged, no fast-forward is done at all. It would be possible to pick one, fast forward to it, and make a merge commit to the rest, I see no benefit to adding that complexity. Determining the best of N changed remotes requires N*2+1 calls to git-log, but these are fast git-log calls, and N is typically small. Also, typically some or all of the remote refs will be the same, and git-log is not called to compare those. In the real world I expect this will almost always add only 1 git-log call to the merge process. (Which already makes N anyway.)
* ensure directory exists when locking journalGravatar Joey Hess2011-11-02
| | | | | Fixes git annex init in a bare repository that already has a git-annex branch.
* cleanupGravatar Joey Hess2011-10-27
|
* Sped up some operations on remotes that are on the same host.Gravatar Joey Hess2011-10-27
| | | | | | | | | | | | | | Specifically, disabled trying to update the git-annex branch on the remote, since that data is never used by operations that act on such remotes. Also, when copying content to such a remote, skip committing the presence information changes to its git-annex branch. Leaving it in the journal there is ok: Any command run on the remote that needs the info will flush the journal. This may partially solve this bug: http://git-annex.branchable.com/bugs/fails_to_handle_lot_of_files/ Although I still see unreaped git processes piling up when doing a copy --to.
* tweaksGravatar Joey Hess2011-10-10
|
* fix a raceGravatar Joey Hess2011-10-09
| | | | | | | | | Another process may stage journalled files before the lock is taken, so need to get the list of journalled files afterwards. It's unfortunate this means getting the directory contents twice, but it seems better to do that than sometimes take the lock unnecessarily.
* better layoutGravatar Joey Hess2011-10-07
| | | | | And a theoretical fix to branchstate cache invalidation, but not a bug that could actually happen.