diff options
Diffstat (limited to 'doc/design/assistant')
10 files changed, 123 insertions, 4 deletions
diff --git a/doc/design/assistant/android.mdwn b/doc/design/assistant/android.mdwn index 90dc55179..f2711a479 100644 --- a/doc/design/assistant/android.mdwn +++ b/doc/design/assistant/android.mdwn @@ -14,6 +14,18 @@ available in the App Store. * git (not all git commands are needed, but core plumbing and a few like `git-add` are.) +## GHC Android? + +Android's native SDK does not use glibc. GHC's runtime links with glibc. +This could be an enormous problem. Other people want to see GHC able to +target Android, of course, so someone may solve it before I get stuck on +it. + +References: + +* <http://stackoverflow.com/questions/5151858/running-a-haskell-program-on-the-android-os> +* <http://www.reddit.com/r/haskell/comments/ful84/haskell_on_android/> + ### Android specific features The app should be aware of power status, and avoid expensive background @@ -57,8 +69,6 @@ version git-annex dependend upon existing on the phone. (Maybe the phone would have to be always considered an untrusted repo, which probably makes sense anyway.) -Problem: - #### crazy `LD_PRELOAD` wrapper Need I say more? (Also, Android's linker may not even support it.) diff --git a/doc/design/assistant/blog/day_8__speed/comment_1_a3dba537b276d5737abc8cb93f1965f4._comment b/doc/design/assistant/blog/day_8__speed/comment_1_a3dba537b276d5737abc8cb93f1965f4._comment new file mode 100644 index 000000000..d0a207c82 --- /dev/null +++ b/doc/design/assistant/blog/day_8__speed/comment_1_a3dba537b276d5737abc8cb93f1965f4._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="Battery usage" + date="2012-06-15T09:57:33Z" + content=""" +Complete fsck is good, but once a week probably enough. + +But please see if you can make fsck optional depending on if the machine is running on battery. +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness.mdwn b/doc/design/assistant/blog/day_9__correctness.mdwn new file mode 100644 index 000000000..1fa4c09d0 --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness.mdwn @@ -0,0 +1,30 @@ + git merge watch_ + +My cursor has been mentally poised here all day, but I've been reluctant to +merge watch into master. It seems solid, but is it correct? I was able to +think up a lot of races it'd be subject to, and deal with them, but did I +find them all? + +Perhaps I need to do some automated fuzz testing to reassure myself. +I looked into using [genbackupdata](http://liw.fi/genbackupdata/) to that +end. It's not quite what I need, but could be +[moved in that direction](http://bugs.debian.org/677542). Or I could write +my own fuzz tester, but it seems better to use someone else's, because +a) laziness and b) they're less likely to have the same blind spots I do. + +My reluctance to merge isn't helped by the known bugs with files that are +either already open before `git annex watch` starts, or are opened by two +processes at once, and confuse it into annexing the still-open file when one +process closes it. + +I've been thinking about just running `lsof` on every file as it's being +annexed to check for that, but in the end, `lsof` is too slow. Since its +check involves trawling through all of /proc, it takes it a good half a +second to check a file, and adding 25 seconds to the time it takes to +process 100 files is just not acceptable. + +But an option that could work is to run `lsof` after a bunch of new files +have been annexed. It can check a lot of files nearly as fast as a single +one. In the rare case that an annexed file is indeed still open, it could +be moved back out of the annex. Then when its remaining writer finally +closes it, another inotify event would re-annex it. diff --git a/doc/design/assistant/blog/day_9__correctness/comment_1_564a39cb976767e2c0a9c74fabe10be4._comment b/doc/design/assistant/blog/day_9__correctness/comment_1_564a39cb976767e2c0a9c74fabe10be4._comment new file mode 100644 index 000000000..8236002cc --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_1_564a39cb976767e2c0a9c74fabe10be4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://wiggy.net/" + nickname="Wichert" + subject="os compatibility" + date="2012-06-15T07:19:23Z" + content=""" +A downside of relying on lsof is that you might be painting yourself into a linux corner: other operating systems might not have a lsof or alternative you can rely on. Especially for Windows this might be a worry. +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_2_77924e9d50b40f05e792e427a25849a6._comment b/doc/design/assistant/blog/day_9__correctness/comment_2_77924e9d50b40f05e792e427a25849a6._comment new file mode 100644 index 000000000..8744882c9 --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_2_77924e9d50b40f05e792e427a25849a6._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://dieter-be.myopenid.com/" + nickname="dieter" + subject="filesystem number of open file handles on a file" + date="2012-06-15T08:21:37Z" + content=""" +wasn't there some filesystem functionality that could tell you the amount of open file handles on a certain file? I thought this was tracked per-file too. +Or maybe i'm just confusing it with the number of hard links (which stat can tell you), anyway something to look into. +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_3_92bd86cd06d579e23800af2e5c66a291._comment b/doc/design/assistant/blog/day_9__correctness/comment_3_92bd86cd06d579e23800af2e5c66a291._comment new file mode 100644 index 000000000..680f16ecc --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_3_92bd86cd06d579e23800af2e5c66a291._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 3" + date="2012-06-15T08:58:17Z" + content=""" +I would also be reluctant to use lsof for the sake of non-linux systems or systems that don't have lsof. I've only been playing around with the watch branch of my \"other\" laptop under archlinux. It looks usable, however I would prefer support for OSX before the watch branch gets merged to master ;) +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_4_0d12b51ccdfc2a94d3e59a5628521e0a._comment b/doc/design/assistant/blog/day_9__correctness/comment_4_0d12b51ccdfc2a94d3e59a5628521e0a._comment new file mode 100644 index 000000000..721875491 --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_4_0d12b51ccdfc2a94d3e59a5628521e0a._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 4" + date="2012-06-15T10:21:17Z" + content=""" +Corner case, but if the other program finishes writing while you are annexing and your check shows no open files, you are left with bad checksum on a correct file. This \"broken\" file with propagate and the next round of fsck will show that all copies are \"bad\". + +Without verifying if this is viable, could you set the file RO and thus block future writes before starting to annex? +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_5_208f9dd3e1d92555b05c29159538a901._comment b/doc/design/assistant/blog/day_9__correctness/comment_5_208f9dd3e1d92555b05c29159538a901._comment new file mode 100644 index 000000000..b7fbecc39 --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_5_208f9dd3e1d92555b05c29159538a901._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.135" + subject="comment 5" + date="2012-06-15T15:14:52Z" + content=""" +@wichert All this inotify stuff is entirely linux specific AFAIK anyway, so it's find for workarounds to limitations in inotify functionality to also be linux specific. + +@dieter I think you're thinking of hard links, filesystems don't track number of open file handles afaik. + +@Jimmy, I'm planning to get watch going on freebsd (and hopefully that will also cover OSX), after merging it :) + +@Richard, the file is set RO while it's being annexed, so any lsof would come after that point. +"""]] diff --git a/doc/design/assistant/blog/day_9__correctness/comment_6_90cc6b60718896fb175919417600fdf9._comment b/doc/design/assistant/blog/day_9__correctness/comment_6_90cc6b60718896fb175919417600fdf9._comment new file mode 100644 index 000000000..622b141fd --- /dev/null +++ b/doc/design/assistant/blog/day_9__correctness/comment_6_90cc6b60718896fb175919417600fdf9._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.135" + subject="comment 6" + date="2012-06-15T15:23:21Z" + content=""" +But Rich is right, and I was thinking the same thing earlier this morning, that delaying the lsof allows the writer to change the file and exit, and only fsck can detect the problem then. Setting file permissions doesn't help once a process already has it open for write. Which has put me off the delayed lsof idea unfortunately. lsof *could* be run safely during the intial annexing. +"""]] diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn index c2a25673e..baa420b4e 100644 --- a/doc/design/assistant/inotify.mdwn +++ b/doc/design/assistant/inotify.mdwn @@ -18,8 +18,20 @@ There is a `watch` branch in git that adds the command. Possible fixes: * Somehow track or detect if a file is open for write by any processes. - `lsof` could be used, although it would be a little slow, and not - avoid every possible race. + `lsof` could be used, although it would be a little slow. + + Here's one way to avoid the slowdown: When a file is being added, + set it read-only, and hard-link it into a quarantine directory, + remembering both filenames. + Then use the batch change mode code to detect batch adds and bundle + them together. + Just before committing, lsof the quarantine directory. Any files in + it that are still open for write can just have their write bit turned + back on and be deleted from quarantine, to be handled when their writer + closes. Files that pass quarantine get added as usual. This avoids + repeated lsof calls slowing down adds, but does add a constant factor + overhead (0.25 seconds lsof call) before any add gets committed. + * Or, when possible, making a copy on write copy before adding the file would avoid this. * Or, as a last resort, make an expensive copy of the file and add that. |