Merge branch 'master' of ssh://git-annex.branchable.com

13 files changed, 191 insertions, 0 deletions

diff --git a/doc/devblog/day_219__catching_up_and_looking_back/comment_3_9699d5a9de5ea64fbc876352e20261c4._comment b/doc/devblog/day_219__catching_up_and_looking_back/comment_3_9699d5a9de5ea64fbc876352e20261c4._comment
new file mode 100644
index 000000000..91fc49409
--- /dev/null
+++ b/doc/devblog/day_219__catching_up_and_looking_back/comment_3_9699d5a9de5ea64fbc876352e20261c4._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="anarcat"
+ ip="70.83.139.100"
+ subject="camlistore"
+ date="2014-09-17T20:18:56Z"
+ content="""
+have you looked at [camlistore](http://camlistore.org/) at all? it's a fairly new project, but it seems like it could be an interesting backend, or at least inspiration for git-annex's design. --[[anarcat]]
+"""]]
diff --git a/doc/devblog/day_219__catching_up_and_looking_back/comment_4_23c4ede3db0ea8165311466881cfa6a2._comment b/doc/devblog/day_219__catching_up_and_looking_back/comment_4_23c4ede3db0ea8165311466881cfa6a2._comment
new file mode 100644
index 000000000..8e24916db
--- /dev/null
+++ b/doc/devblog/day_219__catching_up_and_looking_back/comment_4_23c4ede3db0ea8165311466881cfa6a2._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawmH7o6q2l99M-PQolOfbR3_i5B_jtTIcAE"
+ nickname="Giovanni"
+ subject="Camlistore"
+ date="2014-09-17T20:36:43Z"
+ content="""
+anarcat, have you used it? I tried, but it is buggy. They seem to be at [\"the Archivist\"](http://git-annex.branchable.com/) group of people, but if you don't have a hard drive to store the things, everything breaks up. I paid a lot of money to Amazon because I believed I could use Camlistore to organize data stored at S3, but apparently S3 is \"just for backup\", and if it is your only storage, then Camlistore will keep fetching data over and over \"to index it\" and in the end you pay.
+
+Yes, it keeps working, so you need some server online at all times, with Camlistore running.
+"""]]
diff --git a/doc/devblog/day_219__catching_up_and_looking_back/comment_5_7997305d7ec7db072b78dd0c31ecd824._comment b/doc/devblog/day_219__catching_up_and_looking_back/comment_5_7997305d7ec7db072b78dd0c31ecd824._comment
new file mode 100644
index 000000000..91e4dc5f5
--- /dev/null
+++ b/doc/devblog/day_219__catching_up_and_looking_back/comment_5_7997305d7ec7db072b78dd0c31ecd824._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="anarcat"
+ ip="70.83.139.100"
+ subject="comment 5"
+ date="2014-09-17T20:53:01Z"
+ content="""
+i haven't tried it at all - only looked at [this one demo video](https://www.youtube.com/watch?v=yxSzQIwXM1k) that reminded me of git-annex a lot...
+"""]]
diff --git a/doc/forum/Git_remote__63_____40__bitbucket__44___github__41__/comment_1_8a6de753ac0aa56f470b2aefca628388._comment b/doc/forum/Git_remote__63_____40__bitbucket__44___github__41__/comment_1_8a6de753ac0aa56f470b2aefca628388._comment
new file mode 100644
index 000000000..c6c7f6e1c
--- /dev/null
+++ b/doc/forum/Git_remote__63_____40__bitbucket__44___github__41__/comment_1_8a6de753ac0aa56f470b2aefca628388._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 1"
+ date="2014-09-18T17:43:44Z"
+ content="""
+You can use any git repository as a git remote in git-annex, since git-annex uses plain old git repos.
+
+You will need to add some kind of [[special_remote|special_remotes]] to hold the content of the files stored in git-annex however.
+"""]]
diff --git a/doc/forum/How_do_I_sync_files_from_mobile_to_a_repo__63__/comment_4_404a8f9daa86c20a046b4c9f9051dfc0._comment b/doc/forum/How_do_I_sync_files_from_mobile_to_a_repo__63__/comment_4_404a8f9daa86c20a046b4c9f9051dfc0._comment
new file mode 100644
index 000000000..cbe73a00b
--- /dev/null
+++ b/doc/forum/How_do_I_sync_files_from_mobile_to_a_repo__63__/comment_4_404a8f9daa86c20a046b4c9f9051dfc0._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 4"
+ date="2014-09-18T18:00:42Z"
+ content="""
+I have double-checked, and when I already have an existing, non-bare repository, pointing the webapp at it over ssh keeps it as a non-bare repository. As I would expect.
+
+> I even tried to do git remote add B ssh://machineB:/~/annex but to no avail, the created annex on machine B becomes a bare repo.
+
+I didn't try this because it's such a violation of the way git actually works that I just can't believe it could happen. If it does, you've found the git bug of the year.
+
+But, I think you just got confused about whether the repository existed before, or gave the wrong path to the existing repository which would result in a new, bare repository being created in the location you told it.
+
+If you really think this happens, show a transcript with enough details for me, or the git developers, to reproduce the problem.
+"""]]
diff --git a/doc/forum/How_do_I_sync_files_from_mobile_to_a_repo__63__/comment_5_3dcdaef370d0df38e7285f1fa11c6bb3._comment b/doc/forum/How_do_I_sync_files_from_mobile_to_a_repo__63__/comment_5_3dcdaef370d0df38e7285f1fa11c6bb3._comment
new file mode 100644
index 000000000..7bcce9706
--- /dev/null
+++ b/doc/forum/How_do_I_sync_files_from_mobile_to_a_repo__63__/comment_5_3dcdaef370d0df38e7285f1fa11c6bb3._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 5"
+ date="2014-09-18T18:05:33Z"
+ content="""
+It occurs to me that another way you could have gotten confused is, if ssh://machineB:/~/annex was eg, created in the first place by running the git-annex webapp on machineB, then it would be a direct mode repo. In this case, yes core.bare=true, but so does annex.direct=true. And that repository will not be a bare repo really; it will contain the same file tree as you have on your mobile.
+"""]]
diff --git a/doc/forum/Local_and_remote_in_direct_mode/comment_3_859ec2b3a8e938073b2099fdc5781109._comment b/doc/forum/Local_and_remote_in_direct_mode/comment_3_859ec2b3a8e938073b2099fdc5781109._comment
new file mode 100644
index 000000000..f39c95c03
--- /dev/null
+++ b/doc/forum/Local_and_remote_in_direct_mode/comment_3_859ec2b3a8e938073b2099fdc5781109._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 3"
+ date="2014-09-18T17:55:21Z"
+ content="""
+Petter_petterson, I think you're mistaken about that. If you were right, that would be a massive bug in git -- nothing git-annex specific about that command after all.
+"""]]
diff --git a/doc/forum/annex_merge_creates___34__synced__47____42____34___branches/comment_3_1a0384edd20cc379e53fe7d7f650f7e2._comment b/doc/forum/annex_merge_creates___34__synced__47____42____34___branches/comment_3_1a0384edd20cc379e53fe7d7f650f7e2._comment
new file mode 100644
index 000000000..cb80c07f9
--- /dev/null
+++ b/doc/forum/annex_merge_creates___34__synced__47____42____34___branches/comment_3_1a0384edd20cc379e53fe7d7f650f7e2._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 3"
+ date="2014-09-18T17:39:19Z"
+ content="""
+`git annex merge` does not create any synced/* branches. These branches will be pulled down by `git pull` or `git annex sync`.
+"""]]
diff --git a/doc/forum/big_overhead/comment_10_d5f4e353e7f711d8c38cdcc222339bca._comment b/doc/forum/big_overhead/comment_10_d5f4e353e7f711d8c38cdcc222339bca._comment
new file mode 100644
index 000000000..2ebd56d30
--- /dev/null
+++ b/doc/forum/big_overhead/comment_10_d5f4e353e7f711d8c38cdcc222339bca._comment
@@ -0,0 +1,14 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 10"
+ date="2014-09-18T17:27:36Z"
+ content="""
+In the meantime, I've been looking over the Annex.Branch code. 
+
+`stageJournal` is only ever called in code paths that commit the updated index, so those code paths cannot result in dangling objects unless git-annex is interrupted before it can commit. (This may explain some of my own repos having a few dangling refs, that were not commits; I could have ctrl-c'd git-annex.)
+
+It's possible for a forced update of the local git-annex branch, done by eg a push from another repo, to overwrite a commit made to it. In this case, the git-annex index is merged with the branch, resulting in a new commit, and the old commit that was overwritten will indeed be dangling. However, `git annex sync` doesn't overwrite the git-annex branch; it pushes to synced/git-annex, or does a `taggedPush` to a private ref. It is the case that both those pushes are forced pushes, so can overwrite a branch ref and leave the old commit it pointed to dangling. In the case of `taggedPush`, the old commit should be a parent of the new, so it won't dangle. In the case of synced/git-annex being overwritten, the old commit could dangle, but only until whatever repo pushed it syncs again, at which time it should get incorporated as one of the parents of the new synced/git-annex it pushes. So, I don't see how long-term dangling commits could happen this way, except for in the case where a repository stops syncing/goes missing/rebases its git-annex branch (ie, git-annex forget is used). (This may explain the 2 dangling commits I found on elephant; we did delete some clones of that repository recently.)
+
+At this point I'm not convinced that the dangling objects I found in my own repos are due to some systematic problem, the above seems like it could explain them, and the above is not a problem on the class of the one Rasmus is having. Of course, it's hard to be sure you've spotted all possible ways that a resource leak can happen, and that's what these dangling objects basically are.
+"""]]
diff --git a/doc/forum/big_overhead/comment_11_cbf25217e4149f2cfad4e2bf94f2b4ca._comment b/doc/forum/big_overhead/comment_11_cbf25217e4149f2cfad4e2bf94f2b4ca._comment
new file mode 100644
index 000000000..e2a74068d
--- /dev/null
+++ b/doc/forum/big_overhead/comment_11_cbf25217e4149f2cfad4e2bf94f2b4ca._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 11"
+ date="2014-09-18T17:32:09Z"
+ content="""
+I knew I *had* used \"Initial commit\" somewhere ... etckeeper uses that message. And commits as root. Could an etckeeper repo have somehow gotten merged into your git-annex repo? Seems strange, and the filenames and contents don't really look like /etc to me, but it otherwise somewhat fits.
+"""]]
diff --git a/doc/forum/big_overhead/comment_7_a762eb55addf81c1c5350c7968598d0f._comment b/doc/forum/big_overhead/comment_7_a762eb55addf81c1c5350c7968598d0f._comment
new file mode 100644
index 000000000..1ec6a6eef
--- /dev/null
+++ b/doc/forum/big_overhead/comment_7_a762eb55addf81c1c5350c7968598d0f._comment
@@ -0,0 +1,16 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 7"
+ date="2014-09-17T20:20:40Z"
+ content="""
+There are a few things that can cause git to leave unreachable objects. These include: Rebasing; interrupting a pull before it updates the refs; running git add on a file and then changing the file's content and adding it a second time before committing.
+
+I can think of one case where this happens when using git-annex at the command line: `git annex add $file; git mv $file other-directory; git commit` will result in a dangling object storing the old symlink target before the file was moved.
+
+It'd be useful to investigate, by using `git fsck --unreachable` to get a list of currently unreachable objects, and then use `git show` to look at the objects and try to determine where they came from. Ie, are they symlink targets or are they git-annex location log files (formatted as columns of timestamps and uuids). Any unreachable commits would be the most useful to investigate.
+
+I see a few loose objects here and there in my annexes, but not very many, and git-gc has cleaned up old ones (> 1 month old). Some of them seem to be location log files. I see those in both repositories where I use the assistant, and repositories where I use only command line git-annex. I was able to find 2 unreachable commits in a repository that runs the assistant full-time; both commits were \"merging origin/synced/git-annex into git-annex\". This suggests to me that perhaps the assistant merged the git-annex branch but that merge was overwritten by another thread that committed changes to the branch at the same time.
+
+You should also check the size of inodes on your system; a thousand small loose objects in .git/objects does not normally take up gigabytes of space; with typical inode sizes it might use up a few megabytes. With 1 mb inodes, those same thousand files would use 1 gb..
+"""]]
diff --git a/doc/forum/big_overhead/comment_8_4a66f57c6c0bdc6123618cb69a719be5._comment b/doc/forum/big_overhead/comment_8_4a66f57c6c0bdc6123618cb69a719be5._comment
new file mode 100644
index 000000000..fdbebf4e4
--- /dev/null
+++ b/doc/forum/big_overhead/comment_8_4a66f57c6c0bdc6123618cb69a719be5._comment
@@ -0,0 +1,65 @@
+[[!comment format=mdwn
+ username="rasmus"
+ ip="217.130.110.20"
+ subject="comment 8"
+ date="2014-09-18T11:28:45Z"
+ content="""
+Hi Joey,
+
+Thanks for your careful reply.  
+
+Easy things first:
+
+I *never* add anything from the terminal, though I may do checks and `git annex get`, since sometimes the assistance actually grab the updated files.  Until recently I started git annex automatically on boot, but at the moment it simply renders my laptop useless for too long -- presumably due to the errors investigated here.
+
+I use btrfs (don't ask me why).  Searching online, I did not find a way to find the size of inodes, but I assume that it's sensible?  tune2fs doesn't work but as I understand it is designed for ext*.
+
+What takes up space in my `.git/objects` is files of several Mb.  So at the moment the `pack` folder is 700mb.  In the next biggest folder there's three files that are 73,4mb and 8 files that are 4kb.  This pattern repeats.  A couple of large files (73,4 shows up quite a bit as well as 45) and many small files. 
+
+I have an astonishing amount of dangling objects.  In the `doc.annex` `git rev-list HEAD --count` gives 27354.  In this repo I have 1108 unreachable blobs and commits, respectively 569 and 539.  This probably explains why `git prune` solves my problem but I don't understand why all these large files reappears when I sync -- even after having run `git prune` on both laptops.  Could they come from the `annex` on my remote server?
+
+`git show` isn't nice on blobs, but here is an example of a dangling commit
+
+
+    commit 478425bef867782e8ff22aca24316e9421288c49
+    Author: root <root@localhost>
+    Date:   Mon Dec 31 19:00:01 2012 -0400
+
+        Initial commit
+
+    diff --git a/6e5039464b41f39088a4aece64ced787aa2b04ec2dd5ac6f6c6ca4b9a06a99e5 b/6e5039464b41f39088a4aece64ced787aa2b04ec2dd5ac6f6c6ca4b9a06a99e5
+    new file mode 100644
+    index 0000000..af12763
+    Binary files /dev/null and b/6e5039464b41f39088a4aece64ced787aa2b04ec2dd5ac6f6c6ca4b9a06a99e5 differ
+    diff --git a/8ae4ee273eb540fb71b78152d10010ea2dd3d1bb82afe410ecf3d811cb72bd6d b/8ae4ee273eb540fb71b78152d10010ea2dd3d1bb82afe410ecf3d811cb72bd6d
+    new file mode 100644
+    index 0000000..0a6af91
+    Binary files /dev/null and b/8ae4ee273eb540fb71b78152d10010ea2dd3d1bb82afe410ecf3d811cb72bd6d differ
+    diff --git a/91bd0c092128cf2e60e1a608c31e92caf1f9c1595f83f2890ef17c0e4881aa0a b/91bd0c092128cf2e60e1a608c31e92caf1f9c1595f83f2890ef17c0e4881aa0a
+    new file mode 100644
+    index 0000000..26d921e
+    Binary files /dev/null and b/91bd0c092128cf2e60e1a608c31e92caf1f9c1595f83f2890ef17c0e4881aa0a differ
+    diff --git a/9f7728197cfcd9792eef1ff5930a4ab580e38e64291037130f1ad0914e34a1fc b/9f7728197cfcd9792eef1ff5930a4ab580e38e64291037130f1ad0914e34a1fc
+    new file mode 100644
+    index 0000000..2a92974
+    Binary files /dev/null and b/9f7728197cfcd9792eef1ff5930a4ab580e38e64291037130f1ad0914e34a1fc differ
+    diff --git a/ac801235d97275e761efa12a76ee009472cae8549a0835d5be8bd3f6657047fb b/ac801235d97275e761efa12a76ee009472cae8549a0835d5be8bd3f6657047fb
+    new file mode 100644
+    index 0000000..543430c
+    Binary files /dev/null and b/ac801235d97275e761efa12a76ee009472cae8549a0835d5be8bd3f6657047fb differ
+    diff --git a/d400d0f616a980ea5e3ef68a1f9d670d1eeccbd27f34d1cb7ea976e1f98e2fb7 b/d400d0f616a980ea5e3ef68a1f9d670d1eeccbd27f34d1cb7ea976e1f98e2fb7
+    new file mode 100644
+    index 0000000..7b7eadd
+    Binary files /dev/null and b/d400d0f616a980ea5e3ef68a1f9d670d1eeccbd27f34d1cb7ea976e1f98e2fb7 differ
+    diff --git a/e988a26fbabe3f498e2a564096948eafb289ccadfb186423c1f63c5a3b2c19db b/e988a26fbabe3f498e2a564096948eafb289ccadfb186423c1f63c5a3b2c19db
+    new file mode 100644
+    index 0000000..3bd1dfa
+    Binary files /dev/null and b/e988a26fbabe3f498e2a564096948eafb289ccadfb186423c1f63c5a3b2c19db differ
+
+There are several things I don't understand.  Why is the author root?  I never run `git annex` with `sudo` or as root.  I think the date is bogus.  I'm pretty sure I wasn't even running `git annex` in 2012 much less working with this repo. . . What is weird is that this is the date for *all* lost commits!  (Same for Author).  Over all lost commits there are 2352 binary files that differ.  Of these there are 284 unique hashes. . .  I don't know what this means other than my repo being seriously messed up.  I don't understand what I did wrong to end up in this state as I have been fairly careful in mainly using the `webapp`.
+
+I wonder if the best way to proceed is to start over, or whether this repo can be recovered.
+
+Thanks,
+Rasmus
+"""]]
diff --git a/doc/forum/big_overhead/comment_9_5fa681ea0d6bd0dcac7142d40df9d54f._comment b/doc/forum/big_overhead/comment_9_5fa681ea0d6bd0dcac7142d40df9d54f._comment
new file mode 100644
index 000000000..856f326f4
--- /dev/null
+++ b/doc/forum/big_overhead/comment_9_5fa681ea0d6bd0dcac7142d40df9d54f._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="108.236.230.124"
+ subject="comment 9"
+ date="2014-09-18T16:54:10Z"
+ content="""
+That is a very strange commit by every metric. Weird author, weird date, weird filenames in it (not files that git-annex uses!), with apparently some weird binary content (which git-annex would not be committing). Even a weird commit message -- git-annex never makes a commit with a message of \"Initial commit\", and as far as I can tell using `git log -S`, it never has. (OTOH, it's a pretty common example message used in eg, git documentation.) So, I feel pretty sure that dangling commit was not made by git-annex.
+
+I think you need to take a look at some of the 4+mb unreachable blobs, to get some idea of what these files are. One way is to use git-show on the hash of one of the blobs to get its content, and then, perhaps pass it to `file` or `strings`. Or, you could stop the assistant, `git checkout 478425bef867782e8ff22aca24316e9421288c49` and have a look at this strange tree that was apparently committed in 2012 to see what's in there.
+
+It might be possible that the dangling commits come somehow from the remote server. I'm not 100% sure, but I think that a git pack can end up with dangling objects in it, and then git can pull down that pack to get other, non-dangling objects. You should use `git show` on the server on some of the dangling shas to see if they are present there.
+"""]]