From 37d2338db1476c8d4caf996a746dc6fa8cd1f5d4 Mon Sep 17 00:00:00 2001 From: "johannes.elferich@796088e47b21d72547837ed62ae5c5cb187830b3" Date: Sat, 4 Jul 2015 02:12:52 +0000 Subject: Added a comment: Patched git-annex output --- ...mment_2_18b3ddff003bcdd78779b35c900964c2._comment | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/bugs/git-annex_sync_under_windows_fails_by_using_itself_as_the_ssh_command/comment_2_18b3ddff003bcdd78779b35c900964c2._comment diff --git a/doc/bugs/git-annex_sync_under_windows_fails_by_using_itself_as_the_ssh_command/comment_2_18b3ddff003bcdd78779b35c900964c2._comment b/doc/bugs/git-annex_sync_under_windows_fails_by_using_itself_as_the_ssh_command/comment_2_18b3ddff003bcdd78779b35c900964c2._comment new file mode 100644 index 000000000..3493cb7a4 --- /dev/null +++ b/doc/bugs/git-annex_sync_under_windows_fails_by_using_itself_as_the_ssh_command/comment_2_18b3ddff003bcdd78779b35c900964c2._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="johannes.elferich@796088e47b21d72547837ed62ae5c5cb187830b3" + nickname="johannes.elferich" + subject="Patched git-annex output" + date="2015-07-04T02:12:51Z" + content=""" +You are right something weird is going on with the GIT_ANNEX_SSHOPTION variable even though I have no idea what.... + +This is the output I get after patching: + +›git annex sync +(\"GIT_ANNEX_SSHOPTION\",Nothing) +commit (recording state in git...) +ok +pull origin fatal: protocol error: bad line length character: (\"GI +git-annex.exe: unknown command 35.89.33.17 + +Usage: git-annex command [option ...] + +"""]] -- cgit v1.2.3 From 6e773f4528ff93fc08e5975f0c8359b86b546163 Mon Sep 17 00:00:00 2001 From: "berend.van.berkum@0cc715f45d5a86c134f9192722f7adef9dc63488" Date: Sat, 4 Jul 2015 22:50:34 +0000 Subject: --- doc/forum/Robustness_on_failing_hardware.mdwn | 30 +++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 doc/forum/Robustness_on_failing_hardware.mdwn diff --git a/doc/forum/Robustness_on_failing_hardware.mdwn b/doc/forum/Robustness_on_failing_hardware.mdwn new file mode 100644 index 000000000..ec32cbfe7 --- /dev/null +++ b/doc/forum/Robustness_on_failing_hardware.mdwn @@ -0,0 +1,30 @@ +Hi everyone, + + +I've had a bad git-annex day and I left some musings on my first use of it. There's a question though at the end if you don't need to read that. + + +today I've spend some time trying to deal with a failing SHA256E backend. Or more precise, I'm patienting a refurbished Athlon board[*] without case and that seems to give a lot of errors. I've put up some shielding at the SDRAM which helped a lot on the checksumming, but now its the network department is having a hard time. + +This is basicly an effort to recover an old 1.5TB USB NTFS disks with bit rot, a failed ext4 and recover collections from other aging USB/desktop disks. There is some new hardware on the way, to restore another (proper) box with some ECC memory. That should fix my problems, I think. I've never dealt with a problem like this before. Before git-annex I would have been using rsync -azc or mostly -azu, but I've never had problems like this before. + +It has me pondering a bit though. To help git-annex i've done manual rsync -azc transfers, with some although a very low success--I wonder if its the lack of casing, lack of ground, or simply the non-ecc that is the cause. But that is not really the question. + +rsync does not at first glance have an option to configure the amount of retries, but -azc does try and gives a checksum failure another try before giving up. I had never seen rsync do that before even though I've been using it for a decade. I guess that shows how rare this problem should be, but its still creeping me a bit. I've never had checksummed media file collections, so I really cant be sure about integrity, can I? + +The behaviour of the SHA256E backend is a bit hysterical in this environment. Running a full annex fsck always results in some bad files. The bigger the file the harder it is to get a checksum match. I guess if I wanted to continue I should wrap the fsck step in a shell script that can reinject and fsck again and does do not needless re-checks in one run. It should figure it out in the end with a bit of more time. + +My other reaction would be to throw more checksums at it, from simple to complex. Something between filesize and sha256. I guess the whole would then have some different modes of retries and behaviours of failure maybe. Or levels of trust in the integrity. Maybe using simpler checksums would make the situation more bearable, since the problem is not disk-related iow. if the file is there and has checked out ok, then the re-fsck itself and bad memory/shielding are the culprits. + +tldr; + +git-annex has no retry at all. I have seen it can take a -i somewhere. For robustness, it would be nice if there was a rsync -c mode. It would confirm the object was transferred correctly too. My workflow is to 'annex fsck' at the remote after copying files there. Is it correct to presume that using 'annex move' in my case, would have made the sending repo dropping files even though the remote still has only corrupted copies? + +I'm new to git-annex but been wanting something like git annex for a long time. Made some attempts myself, and too pondered how to use venerable GIT but never figured something out. What I'm saying though is I may want to dig around to see what I can do with Haskell. + +I'm thinking a backend with some more checksums, maybe then some option to fsck to request only partial check of the key. Has something like a faster/less-thorough fsck using smaller checksums ever come up? And might this be a way to go at this? What do you think. + + + + +[*] Asus M2A7A-AM SE w/ AMD Athlon 64 X2 5200+ Dual Core 2.7 -- cgit v1.2.3 From 0db92ffbd825a2dd3d7a081be6a39e90f22b311b Mon Sep 17 00:00:00 2001 From: "xelez0@57a58225d4e5b260555ebc4a9d0a8df85d1e971a" Date: Sun, 5 Jul 2015 13:19:43 +0000 Subject: Added a comment: Backend of specified file --- doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment diff --git a/doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment b/doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment new file mode 100644 index 000000000..446960264 --- /dev/null +++ b/doc/backends/comment_14_57154dcd1041a33f220f9105b709be89._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="xelez0@57a58225d4e5b260555ebc4a9d0a8df85d1e971a" + nickname="xelez0" + subject="Backend of specified file" + date="2015-07-05T13:19:43Z" + content=""" +How can I determine backend of specified file? Looking over man pages and can't find it. +"""]] -- cgit v1.2.3 From 54131a657034872e883656683ca16019b7cd4576 Mon Sep 17 00:00:00 2001 From: CandyAngel Date: Sun, 5 Jul 2015 14:54:19 +0000 Subject: Added a comment --- .../comment_15_b3445fd1f379346c642a27211c6c798b._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/backends/comment_15_b3445fd1f379346c642a27211c6c798b._comment diff --git a/doc/backends/comment_15_b3445fd1f379346c642a27211c6c798b._comment b/doc/backends/comment_15_b3445fd1f379346c642a27211c6c798b._comment new file mode 100644 index 000000000..3fc12afee --- /dev/null +++ b/doc/backends/comment_15_b3445fd1f379346c642a27211c6c798b._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="CandyAngel" + subject="comment 15" + date="2015-07-05T14:54:18Z" + content=""" +It's not explicit, but 'git annex info $FILE' tells you the key, which has the backend as its first component: + + ## git annex info CG\ Cookie/Compositing\ in\ Blender/01_CompositingInBlender_SourceFiles.zip + file: CG Cookie/Compositing in Blender/01_CompositingInBlender_SourceFiles.zip + size: 744.51 megabytes + key: SHA256E-s744506832--08d2daced60b5eb6509044d5eefca82e7a6899350f49adc0083014229739515e.zip + +I don't think there are any situations where the first component of the key isn't the backend, but don't hold me to that, please :) +"""]] -- cgit v1.2.3 From d936f0cc1efdf4b937cb26be7f1946343f8c216d Mon Sep 17 00:00:00 2001 From: CandyAngel Date: Sun, 5 Jul 2015 15:00:46 +0000 Subject: Added a comment --- .../comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/backends/comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment diff --git a/doc/backends/comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment b/doc/backends/comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment new file mode 100644 index 000000000..64c0102e2 --- /dev/null +++ b/doc/backends/comment_16_c68dfaeee2ef18f420f7e11ff5f604b9._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="CandyAngel" + subject="comment 16" + date="2015-07-05T15:00:46Z" + content=""" +Or I could not be an idiot and tell you the command specifically looking up a key for a file: lookupkey + + ## git annex lookupkey CG\ Cookie/Compositing\ in\ Blender/01_CompositingInBlender_SourceFiles.zip + SHA256E-s744506832--08d2daced60b5eb6509044d5eefca82e7a6899350f49adc0083014229739515e.zip + +So to get the backend (if the first component is always the backend): + + ## git annex lookupkey CG\ Cookie/Compositing\ in\ Blender/01_CompositingInBlender_SourceFiles.zip | cut -d- -f1 + SHA256E +"""]] -- cgit v1.2.3 From 05dcde99331846fd6a511a0c26eeb004ddb4b2cf Mon Sep 17 00:00:00 2001 From: Mattt Date: Sun, 5 Jul 2015 19:40:30 +0000 Subject: Added a comment --- .../comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 doc/forum/How_big_can_a_git-annex_repo_get_and_still_be_reliable_and_fast__63__/comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment diff --git a/doc/forum/How_big_can_a_git-annex_repo_get_and_still_be_reliable_and_fast__63__/comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment b/doc/forum/How_big_can_a_git-annex_repo_get_and_still_be_reliable_and_fast__63__/comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment new file mode 100644 index 000000000..da8067ab2 --- /dev/null +++ b/doc/forum/How_big_can_a_git-annex_repo_get_and_still_be_reliable_and_fast__63__/comment_3_cc62f722b3e1db7d75d6976ef3854f94._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="Mattt" + subject="comment 3" + date="2015-07-05T19:40:30Z" + content=""" +Thanks CandyAngel and Joey. That helped. I have transferred pretty much my entire repo into git annex now and have it sorted the way I want. Yay! +"""]] -- cgit v1.2.3