diff options
Diffstat (limited to 'doc/design/assistant')
15 files changed, 201 insertions, 15 deletions
diff --git a/doc/design/assistant/blog/day_3__more_races.mdwn b/doc/design/assistant/blog/day_3__more_races.mdwn new file mode 100644 index 000000000..9c1182842 --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races.mdwn @@ -0,0 +1,26 @@ +Today I worked on the race conditions, and fixed two of them. Both +were fixed by avoiding using `git add`, which looks at the files currently +on disk. Instead, `git annex watch` injects symlinks directly into git's +index, using `git update-index`. + +There is one bad race condition remaining. If multiple processes have a +file open for write, one can close it, and it will be added to the annex. +But then the other can still write to it. + +---- + +Getting away from race conditions for a while, I made `git annex watch` +not annex `.gitignore` and `.gitattributes` files. + +And, I made it handle running out of inotify descriptors. By default, +`/proc/sys/fs/inotify/max_user_watches` is 8192, and that's how many +directories inotify can watch. Now when it needs more, it will print +a nice message showing how to increase it with `sysctl`. + +FWIW, DropBox also uses inotify and has the same limit. It seems to not +tell the user how to fix it when it goes over. Here's what `git annex +watch` will say: + + Too many directories to watch! (Not watching ./dir4299) + Increase the limit by running: + echo fs.inotify.max_user_watches=81920 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p diff --git a/doc/design/assistant/blog/day_3__more_races/comment_1_d6015338f602b574a3805de5481fc45e._comment b/doc/design/assistant/blog/day_3__more_races/comment_1_d6015338f602b574a3805de5481fc45e._comment new file mode 100644 index 000000000..2d330f332 --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races/comment_1_d6015338f602b574a3805de5481fc45e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkmtR6oVColYKoU0SjBORLDGrwR10G-mKo" + nickname="Jo-Herman" + subject="Dropbox Inotify" + date="2012-06-06T22:03:29Z" + content=""" +Actually, Dropbox giver you a warning via libnotify inotify. It tends to go away too quickly to properly read though, much less actually copy down the command... +"""]] diff --git a/doc/design/assistant/blog/day_3__more_races/comment_2_4d6b23fc6442e0ee0303523cb69d0fba._comment b/doc/design/assistant/blog/day_3__more_races/comment_2_4d6b23fc6442e0ee0303523cb69d0fba._comment new file mode 100644 index 000000000..523e6d85f --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races/comment_2_4d6b23fc6442e0ee0303523cb69d0fba._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.252.8.36" + subject="comment 2" + date="2012-06-06T23:25:57Z" + content=""" +When I work on the [[webapp]], I'm planning to make it display this warning, and any other similar warning messages that might come up. +"""]] diff --git a/doc/design/assistant/blog/day_3__more_races/comment_3_03f5b2344c2a47dea60086f217d60f9b._comment b/doc/design/assistant/blog/day_3__more_races/comment_3_03f5b2344c2a47dea60086f217d60f9b._comment new file mode 100644 index 000000000..92f5dcbd6 --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races/comment_3_03f5b2344c2a47dea60086f217d60f9b._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnBJ6Dv1glxzzi4qIzGFNa6F-mfHIvv9Ck" + nickname="Jim" + subject="Wording" + date="2012-06-07T03:43:19Z" + content=""" +For the unfamiliar, it's hard to tell if a command like that would persist. I'd suggest being as clear as possible, e.g.: + + Increase the limit for now by running: + sudo sysctl fs.inotify.max_user_watches=81920 + Increase the limit now and automatically at every boot by running: + echo fs.inotify.max_user_watches=81920 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p + +"""]] diff --git a/doc/design/assistant/blog/day_3__more_races/comment_4_860e90e989ec022100001c65e353a91e._comment b/doc/design/assistant/blog/day_3__more_races/comment_4_860e90e989ec022100001c65e353a91e._comment new file mode 100644 index 000000000..05b601eaf --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races/comment_4_860e90e989ec022100001c65e353a91e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.252.8.36" + subject="comment 4" + date="2012-06-07T04:48:15Z" + content=""" +Good thought Jim. I've done something like that. +"""]] diff --git a/doc/design/assistant/blog/day_4__speed.mdwn b/doc/design/assistant/blog/day_4__speed.mdwn new file mode 100644 index 000000000..badc6b7b1 --- /dev/null +++ b/doc/design/assistant/blog/day_4__speed.mdwn @@ -0,0 +1,47 @@ +Only had a few hours to work today, but my current focus is speed, and I +have indeed sped up parts of `git annex watch`. + +One thing folks don't realize about git is that despite a rep for being +fast, it can be rather slow in one area: Writing the index. You don't +notice it until you have a lot of files, and the index gets big. So I've +put a lot of effort into git-annex in the past to avoid writing the index +repeatedly, and queue up big index changes that can happen all at once. The +new `git annex watch` was not able to use that queue. Today I reworked the +queue machinery to support the types of direct index writes it needs, and +now repeated index writes are eliminated. + +... Eliminated too far, it turns out, since it doesn't yet *ever* flush +that queue until shutdown! So the next step here will be to have a worker +thread that wakes up periodically, flushes the queue, and autocommits. +(This will, in fact, be the start of the [[syncing]] phase of my roadmap!) +There's lots of room here for smart behavior. Like, if a lot of changes are +being made close together, wait for them to die down before committing. Or, +if it's been idle and a single file appears, commit it immediatly, since +this is probably something the user wants synced out right away. I'll start +with something stupid and then add the smarts. + +(BTW, in all my years of programming, I have avoided threads like the nasty +bug-prone plague they are. Here I already have three threads, and am going to +add probably 4 or 5 more before I'm done with the git annex assistant. So +far, it's working well -- I give credit to Haskell for making it easy to +manage state in ways that make it possible to reason about how the threads +will interact.) + +What about the races I've been stressing over? Well, I have an ulterior +motive in speeding up `git annex watch`, and that's to also be able to +**slow it down**. Running in slow-mo makes it easy to try things that might +cause a race and watch how it reacts. I'll be using this technique when +I circle back around to dealing with the races. + +Another tricky speed problem came up today that I also need to fix. On +startup, `git annex watch` scans the whole tree to find files that have +been added or moved etc while it was not running, and take care of them. +Currently, this scan involves re-staging every symlink in the tree. That's +slow! I need to find a way to avoid re-staging symlinks; I may use `git +cat-file` to check if the currently staged symlink is correct, or I may +come up with some better and faster solution. Sleeping on this problem. + +---- + +Oh yeah, I also found one more race bug today. It only happens at startup +and could only make it miss staging file deletions. diff --git a/doc/design/assistant/blog/day_4__speed/comment_1_bf3c9c33cc0dea5eaeb6f2af110b924b._comment b/doc/design/assistant/blog/day_4__speed/comment_1_bf3c9c33cc0dea5eaeb6f2af110b924b._comment new file mode 100644 index 000000000..fb5b95490 --- /dev/null +++ b/doc/design/assistant/blog/day_4__speed/comment_1_bf3c9c33cc0dea5eaeb6f2af110b924b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawldKnauegZulM7X6JoHJs7Gd5PnDjcgx-E" + nickname="Matt" + subject="open source?" + date="2012-06-09T22:34:30Z" + content=""" +Are you publishing the source code for git-annex assistant somewhere? +"""]] diff --git a/doc/design/assistant/blog/day_4__speed/comment_2_33aba4c9abaa3e6a05a2c87ab7df9d0e._comment b/doc/design/assistant/blog/day_4__speed/comment_2_33aba4c9abaa3e6a05a2c87ab7df9d0e._comment new file mode 100644 index 000000000..1fcc197ab --- /dev/null +++ b/doc/design/assistant/blog/day_4__speed/comment_2_33aba4c9abaa3e6a05a2c87ab7df9d0e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.8.126" + subject="comment 2" + date="2012-06-09T23:01:29Z" + content=""" +Yes, it's in [[git|download]] with the rest of git-annex. Currently in the `watch` branch. +"""]] diff --git a/doc/design/assistant/comment_3_05223be50c889b2ed6bc4abf74116450._comment b/doc/design/assistant/comment_3_05223be50c889b2ed6bc4abf74116450._comment new file mode 100644 index 000000000..a78fa3343 --- /dev/null +++ b/doc/design/assistant/comment_3_05223be50c889b2ed6bc4abf74116450._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 3" + date="2012-06-07T20:22:55Z" + content=""" +I'd agree getting it into the main distros is the way to go, if you need OSX binaries, I could volunteer to setup an autobuilder to generate binaries for OSX users, however it would rely on users to have macports with the correct ports installed to use it (things like coreutils etc...) + +"""]] diff --git a/doc/design/assistant/comment_4_fbbd93b55803ae21e6ba4b6568c2fafd._comment b/doc/design/assistant/comment_4_fbbd93b55803ae21e6ba4b6568c2fafd._comment new file mode 100644 index 000000000..cd3b5aaef --- /dev/null +++ b/doc/design/assistant/comment_4_fbbd93b55803ae21e6ba4b6568c2fafd._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + subject="comment 4" + date="2012-06-08T01:56:52Z" + content=""" +I always appreciate your OSX work Jimmy... + +Could it be put into macports? +"""]] diff --git a/doc/design/assistant/comment_5_f4e9af3fed6c27e8ff39badb9794064d._comment b/doc/design/assistant/comment_5_f4e9af3fed6c27e8ff39badb9794064d._comment new file mode 100644 index 000000000..bf8d9709e --- /dev/null +++ b/doc/design/assistant/comment_5_f4e9af3fed6c27e8ff39badb9794064d._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 5" + date="2012-06-08T07:22:34Z" + content=""" +In relation to macports, I often found that haskell in macports are often behind other distros, and I'm not willing to put much effort into maintaining or updating those ports. I found that to build git-annex, installing macports manually and then installing haskell-platform from the upstream to be the best way to get the most up to date dependancies for git-annex. + +fyi in macports ghc is at version 6.10.4 and haskell platform is at version 2009.2, so there are a significant number of ports to update. + +I was thinking about this a bit more and I reckon it might be easier to try and build a self contained .pkg package and have all the needed binaries in a .app styled package, that would work well when the webapp comes along. I will take a look at it in a week or two (currently moving house so I dont have much time) +"""]] diff --git a/doc/design/assistant/comment_6_c7ad07cade1f44f9a8b61f92225bb9c5._comment b/doc/design/assistant/comment_6_c7ad07cade1f44f9a8b61f92225bb9c5._comment new file mode 100644 index 000000000..9fa66d6d3 --- /dev/null +++ b/doc/design/assistant/comment_6_c7ad07cade1f44f9a8b61f92225bb9c5._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 6" + date="2012-06-08T15:21:18Z" + content=""" +It's not much for now... but see <http://www.sgenomics.org/~jtang/gitbuilder-git-annex-x00-x86_64-apple-darwin10.8.0/> I'm ignoring the debian-stable and pristine-tar branches for now, as I am just building and testing on osx 10.7. + +Hope the autobuilder will help you develop the OSX side of things without having direct access to an osx machine! I will try and get gitbuilder to spit out appropriately named tarballs of the compiled binaries in a few days when I have more time. +"""]] diff --git a/doc/design/assistant/comment_7_609d38e993267195a80fecd84c93d1e2._comment b/doc/design/assistant/comment_7_609d38e993267195a80fecd84c93d1e2._comment new file mode 100644 index 000000000..6685c6548 --- /dev/null +++ b/doc/design/assistant/comment_7_609d38e993267195a80fecd84c93d1e2._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.8.126" + subject="comment 7" + date="2012-06-09T18:07:51Z" + content=""" +Thanks, that's already been useful to me. You might as well skip the debian-specific \"bpo\" tags too. +"""]] diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn index 5d903a9b0..7cdde33ac 100644 --- a/doc/design/assistant/inotify.mdwn +++ b/doc/design/assistant/inotify.mdwn @@ -19,23 +19,22 @@ really useful, it needs to: - notice deleted files and stage the deletion (tricky; there's a race with add since it replaces the file with a symlink..) **done** +- Gracefully handle when the default limit of 8192 inotified directories + is exceeded. This can be tuned by root, so help the user fix it. + **done** - periodically auto-commit staged changes (avoid autocommitting when lots of changes are coming in) - tunable delays before adding new files, etc -- Coleasce related add/rm events. See commit - cbdaccd44aa8f0ca30afba23fc06dd244c242075 for some details of the problems - with doing this. -- don't annex `.gitignore` and `.gitattributes` files, but do auto-stage - changes to them +- coleasce related add/rm events for speed and less disk IO +- don't annex `.gitignore` and `.gitattributes` files **done** - configurable option to only annex files meeting certian size or filename criteria +- option to check files not meeting annex criteria into git directly - honor .gitignore, not adding files it excludes (difficult, probably needs my own .gitignore parser to avoid excessive running of git commands to check for ignored files) - Possibly, when a directory is moved out of the annex location, unannex its contents. -- Gracefully handle when the default limit of 8192 inotified directories - is exceeded. This can be tuned by root, so help the user fix it. - Support OSes other than Linux; it only uses inotify currently. OSX and FreeBSD use the same mechanism, and there is a Haskell interface for it, @@ -67,9 +66,18 @@ Many races need to be dealt with by this code. Here are some of them. **Currently unfixed**; This changes content in the annex, and fsck will later catch the inconsistency. - Possible fixes: Somehow track or detect if a file is open for write - by any processes. Or, when possible, making a copy on write copy - before adding the file would avoid this. + Possible fixes: + + * Somehow track or detect if a file is open for write by any processes. + * Or, when possible, making a copy on write copy before adding the file + would avoid this. + * Or, as a last resort, make an expensive copy of the file and add that. + * Tracking file opens and closes with inotify could tell if any other + processes have the file open. But there are problems.. It doesn't + seem to differentiate between files opened for read and for write. + And there would still be a race after the last close and before it's + injected into the annex, where it could be opened for write again. + Would need to detect that and undo the annex injection or something. * File is added and then replaced with another file before the annex add makes its symlink. @@ -81,16 +89,14 @@ Many races need to be dealt with by this code. Here are some of them. * File is added and then replaced with another file before the annex add stages the symlink in git. - **Currently unfixed**; `git add` will be run on the new file, which is - not at all good when it's big. Could be dealt with by using `git - update-index` to manually put the symlink into the index without git + Now fixed; `git annex watch` avoids running `git add` because of this + race. Instead, it stages symlinks directly into the index, without looking at what's currently on disk. * Link is moved, fixed link is written by fix event, but then that is removed by the user and replaced with a file before the event finishes. - **Currently unfixed**: `git add` will be run on the file. Basically same - effect as previous race above. + Now fixed; same fix as previous race above. * File is removed and then re-added before the removal event starts. @@ -102,3 +108,6 @@ Many races need to be dealt with by this code. Here are some of them. Not a problem; The removal event removes the old file from the index, and the add event adds the new one. +* At startup, `git add --update` is run, to notice deleted files. + Then inotify starts up. Files deleted in between won't have their + removals staged. diff --git a/doc/design/assistant/webapp.mdwn b/doc/design/assistant/webapp.mdwn index abf7b38c9..598c1ff3a 100644 --- a/doc/design/assistant/webapp.mdwn +++ b/doc/design/assistant/webapp.mdwn @@ -23,6 +23,8 @@ The webapp is a web server that displays a shiny interface. * there could be a UI to export a file, which would make it be served up over http by the web app +* Display any relevant warning messages. One is the `inotify max_user_watches` + exceeded message. ## implementation |