diff options
Diffstat (limited to 'doc/design/assistant/blog')
8 files changed, 97 insertions, 0 deletions
diff --git a/doc/design/assistant/blog/day_8__speed/comment_1_a3dba537b276d5737abc8cb93f1965f4._comment b/doc/design/assistant/blog/day_8__speed/comment_1_a3dba537b276d5737abc8cb93f1965f4._comment new file mode 100644 index 000000000..d0a207c82 --- /dev/null +++ b/doc/design/assistant/blog/day_8__speed/comment_1_a3dba537b276d5737abc8cb93f1965f4._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="Battery usage" + date="2012-06-15T09:57:33Z" + content=""" +Complete fsck is good, but once a week probably enough. + +But please see if you can make fsck optional depending on if the machine is running on battery. +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness.mdwn b/doc/design/assistant/blog/day_9__correctness.mdwn new file mode 100644 index 000000000..1fa4c09d0 --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness.mdwn @@ -0,0 +1,30 @@ + git merge watch_ + +My cursor has been mentally poised here all day, but I've been reluctant to +merge watch into master. It seems solid, but is it correct? I was able to +think up a lot of races it'd be subject to, and deal with them, but did I +find them all? + +Perhaps I need to do some automated fuzz testing to reassure myself. +I looked into using [genbackupdata](http://liw.fi/genbackupdata/) to that +end. It's not quite what I need, but could be +[moved in that direction](http://bugs.debian.org/677542). Or I could write +my own fuzz tester, but it seems better to use someone else's, because +a) laziness and b) they're less likely to have the same blind spots I do. + +My reluctance to merge isn't helped by the known bugs with files that are +either already open before `git annex watch` starts, or are opened by two +processes at once, and confuse it into annexing the still-open file when one +process closes it. + +I've been thinking about just running `lsof` on every file as it's being +annexed to check for that, but in the end, `lsof` is too slow. Since its +check involves trawling through all of /proc, it takes it a good half a +second to check a file, and adding 25 seconds to the time it takes to +process 100 files is just not acceptable. + +But an option that could work is to run `lsof` after a bunch of new files +have been annexed. It can check a lot of files nearly as fast as a single +one. In the rare case that an annexed file is indeed still open, it could +be moved back out of the annex. Then when its remaining writer finally +closes it, another inotify event would re-annex it. diff --git a/doc/design/assistant/blog/day_9__correctness/comment_1_564a39cb976767e2c0a9c74fabe10be4._comment b/doc/design/assistant/blog/day_9__correctness/comment_1_564a39cb976767e2c0a9c74fabe10be4._comment new file mode 100644 index 000000000..8236002cc --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_1_564a39cb976767e2c0a9c74fabe10be4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://wiggy.net/" + nickname="Wichert" + subject="os compatibility" + date="2012-06-15T07:19:23Z" + content=""" +A downside of relying on lsof is that you might be painting yourself into a linux corner: other operating systems might not have a lsof or alternative you can rely on. Especially for Windows this might be a worry. +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_2_77924e9d50b40f05e792e427a25849a6._comment b/doc/design/assistant/blog/day_9__correctness/comment_2_77924e9d50b40f05e792e427a25849a6._comment new file mode 100644 index 000000000..8744882c9 --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_2_77924e9d50b40f05e792e427a25849a6._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://dieter-be.myopenid.com/" + nickname="dieter" + subject="filesystem number of open file handles on a file" + date="2012-06-15T08:21:37Z" + content=""" +wasn't there some filesystem functionality that could tell you the amount of open file handles on a certain file? I thought this was tracked per-file too. +Or maybe i'm just confusing it with the number of hard links (which stat can tell you), anyway something to look into. +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_3_92bd86cd06d579e23800af2e5c66a291._comment b/doc/design/assistant/blog/day_9__correctness/comment_3_92bd86cd06d579e23800af2e5c66a291._comment new file mode 100644 index 000000000..680f16ecc --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_3_92bd86cd06d579e23800af2e5c66a291._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 3" + date="2012-06-15T08:58:17Z" + content=""" +I would also be reluctant to use lsof for the sake of non-linux systems or systems that don't have lsof. I've only been playing around with the watch branch of my \"other\" laptop under archlinux. It looks usable, however I would prefer support for OSX before the watch branch gets merged to master ;) +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_4_0d12b51ccdfc2a94d3e59a5628521e0a._comment b/doc/design/assistant/blog/day_9__correctness/comment_4_0d12b51ccdfc2a94d3e59a5628521e0a._comment new file mode 100644 index 000000000..721875491 --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_4_0d12b51ccdfc2a94d3e59a5628521e0a._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 4" + date="2012-06-15T10:21:17Z" + content=""" +Corner case, but if the other program finishes writing while you are annexing and your check shows no open files, you are left with bad checksum on a correct file. This \"broken\" file with propagate and the next round of fsck will show that all copies are \"bad\". + +Without verifying if this is viable, could you set the file RO and thus block future writes before starting to annex? +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_5_208f9dd3e1d92555b05c29159538a901._comment b/doc/design/assistant/blog/day_9__correctness/comment_5_208f9dd3e1d92555b05c29159538a901._comment new file mode 100644 index 000000000..b7fbecc39 --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_5_208f9dd3e1d92555b05c29159538a901._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.135" + subject="comment 5" + date="2012-06-15T15:14:52Z" + content=""" +@wichert All this inotify stuff is entirely linux specific AFAIK anyway, so it's find for workarounds to limitations in inotify functionality to also be linux specific. + +@dieter I think you're thinking of hard links, filesystems don't track number of open file handles afaik. + +@Jimmy, I'm planning to get watch going on freebsd (and hopefully that will also cover OSX), after merging it :) + +@Richard, the file is set RO while it's being annexed, so any lsof would come after that point. +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_6_90cc6b60718896fb175919417600fdf9._comment b/doc/design/assistant/blog/day_9__correctness/comment_6_90cc6b60718896fb175919417600fdf9._comment new file mode 100644 index 000000000..622b141fd --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_6_90cc6b60718896fb175919417600fdf9._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.135" + subject="comment 6" + date="2012-06-15T15:23:21Z" + content=""" +But Rich is right, and I was thinking the same thing earlier this morning, that delaying the lsof allows the writer to change the file and exit, and only fsck can detect the problem then. Setting file permissions doesn't help once a process already has it open for write. Which has put me off the delayed lsof idea unfortunately. lsof *could* be run safely during the intial annexing. +"""]] |