Merge branch 'master' into assistant

9 files changed, 178 insertions, 7 deletions

diff --git a/doc/bugs/Watch_command_as_of_commit_6cecc26206c4a539999b04664136c6f785211a41_segfaults/comment_10_6c872dff4fcc63c16bf69d1e96891c89._comment b/doc/bugs/Watch_command_as_of_commit_6cecc26206c4a539999b04664136c6f785211a41_segfaults/comment_10_6c872dff4fcc63c16bf69d1e96891c89._comment
new file mode 100644
index 000000000..952ffadc4
--- /dev/null
+++ b/doc/bugs/Watch_command_as_of_commit_6cecc26206c4a539999b04664136c6f785211a41_segfaults/comment_10_6c872dff4fcc63c16bf69d1e96891c89._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="jtang"
+ ip="79.97.135.214"
+ subject="comment 10"
+ date="2012-07-29T10:49:05Z"
+ content="""
+This is looking good, no more segfaulting.
+"""]]
diff --git a/doc/bugs/lsof__47__committer_thread_loops_occassionally.mdwn b/doc/bugs/lsof__47__committer_thread_loops_occassionally.mdwn
new file mode 100644
index 000000000..eb8a0ead0
--- /dev/null
+++ b/doc/bugs/lsof__47__committer_thread_loops_occassionally.mdwn
@@ -0,0 +1,49 @@
+I've noticed this to occur occassionally 
+
+<pre>
+laplace:atest jtang$ ls
+1@  2@	3@  4@	5@  readme.txt@
+laplace:atest jtang$ git annex watch -d --foreground
+watch . [2012-07-29 11:49:26 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","show-ref","git-annex"] 
+[2012-07-29 11:49:26 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","show-ref","--hash","refs/heads/git-annex"] 
+[2012-07-29 11:49:26 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","log","refs/heads/git-annex..f85faa60e73efabc2e92f837b19c3918d3ab030f","--oneline","-n1"] 
+[2012-07-29 11:49:26 IST] chat: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","cat-file","--batch"] 
+(scanning...) [2012-07-29 11:49:26 IST] Assistant: all threads started
+[2012-07-29 11:49:26 IST] Merger: watching /Users/jtang/sandbox/atest/.git/refs/heads/synced
+[2012-07-29 11:49:26 IST] TransferWatcher: watching for transfers
+[2012-07-29 11:49:26 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","symbolic-ref","HEAD"] 
+[2012-07-29 11:49:26 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","add","--update"] 
+[2012-07-29 11:49:26 IST] Merger: merging changes into Just refs/heads/master
+[2012-07-29 11:49:26 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","merge","--no-edit","refs/heads/synced/master"] 
+(started...) [2012-07-29 11:49:26 IST] Watcher: watching .
+[2012-07-29 11:49:26 IST] WebApp: running on port 60042
+Already up-to-date.
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./1
+[2012-07-29 11:49:26 IST] chat: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","cat-file","--batch"] 
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./2
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./3
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./4
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./5
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./readme.txt
+[2012-07-29 11:49:27 IST] Committer: committing 6 changes
+(Recording state in git...)
+[2012-07-29 11:49:27 IST] feed: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","update-index","-z","--index-info"] 
+[2012-07-29 11:49:27 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","commit","--allow-empty-message","-m","","--allow-empty","--quiet"] 
+[2012-07-29 11:49:28 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","symbolic-ref","HEAD"] 
+[2012-07-29 11:49:28 IST] Pusher: pushing to []
+[2012-07-29 11:49:28 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","branch","-f","synced/master"] 
+[2012-07-29 11:49:28 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","symbolic-ref","HEAD"] 
+[2012-07-29 11:49:28 IST] Merger: merging changes into Just refs/heads/master
+[2012-07-29 11:49:28 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","merge","--no-edit","refs/heads/synced/master"] 
+Already up-to-date.
+[2012-07-29 11:49:43 IST] Watcher: file added ./.gitignore
+[2012-07-29 11:49:43 IST] read: lsof ["-F0can","+d","/Users/jtang/sandbox/atest/.git/annex/tmp/"] 
+[2012-07-29 11:49:44 IST] Committer: delaying commit of 0 changes
+[2012-07-29 11:49:45 IST] read: lsof ["-F0can","+d","/Users/jtang/sandbox/atest/.git/annex/tmp/"] 
+[2012-07-29 11:49:46 IST] Committer: delaying commit of 0 changes
+[2012-07-29 11:49:47 IST] read: lsof ["-F0can","+d","/Users/jtang/sandbox/atest/.git/annex/tmp/"] 
+[2012-07-29 11:49:48 IST] Committer: delaying commit of 0 changes
+[2012-07-29 11:49:49 IST] read: lsof ["-F0can","+d","/Users/jtang/sandbox/atest/.git/annex/tmp/"] 
+</pre>
+
+I ran " git annex watch -d --foreground" to watch what was going one, and just created a .gitignore file and the the commiter/lsof thread just loops over and over.... I only noticed as my laptop battery had drained at somepoint when git-annex was running in the background.
diff --git a/doc/bugs/lsof__47__committer_thread_loops_occassionally/comment_1_f8d1720aa26c719609720acf0772606e._comment b/doc/bugs/lsof__47__committer_thread_loops_occassionally/comment_1_f8d1720aa26c719609720acf0772606e._comment
new file mode 100644
index 000000000..521d2e0cb
--- /dev/null
+++ b/doc/bugs/lsof__47__committer_thread_loops_occassionally/comment_1_f8d1720aa26c719609720acf0772606e._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="http://joeyh.name/"
+ ip="4.153.8.133"
+ subject="probably a kqueue specific problem..."
+ date="2012-07-29T18:10:12Z"
+ content="""
+I don't think this would eat your battery badly; it's looping for sure, but with a hardcoded 1 second delay I put in to guard against it eating all CPU.
+
+I tried to fix this in commit c4023f785834bc237e5fcdb69e275bbae10dd40b, but I sort of doubt I did.
+I made one more commit that will at least tell us what file it is trying to check over and over with lsof.
+"""]]
diff --git a/doc/design/assistant/android.mdwn b/doc/design/assistant/android.mdwn
index f2711a479..65569e403 100644
--- a/doc/design/assistant/android.mdwn
+++ b/doc/design/assistant/android.mdwn
@@ -26,6 +26,10 @@ References:
 * <http://stackoverflow.com/questions/5151858/running-a-haskell-program-on-the-android-os>
 * <http://www.reddit.com/r/haskell/comments/ful84/haskell_on_android/>
 
+I've heard anecdoally that ipwnstudios not only has an IPhone GHC port,
+but also Android. Need to get in touch with them.
+<http://ipwnstudios.com/>
+
 ### Android specific features
 
 The app should be aware of power status, and avoid expensive background
diff --git a/doc/design/assistant/blog/day_46__notification_pools.mdwn b/doc/design/assistant/blog/day_46__notification_pools.mdwn
new file mode 100644
index 000000000..483ad95b1
--- /dev/null
+++ b/doc/design/assistant/blog/day_46__notification_pools.mdwn
@@ -0,0 +1,68 @@
+Focus today was writing a notification broadcaster library. This is a way to
+send a notification to a set of clients, any of which can be blocked
+waiting for a new notification to arrive. A complication is that any number
+of clients may be be dead, and we don't want stale notifications for those
+clients to pile up and leak memory.
+
+It took me 3 tries to find the solution, which turns out to be head-smackingly
+simple: An array of SampleVars, one per client.
+
+Using SampleVars means that clients only see the most recent notification,
+but when the notification is just "the assistant's state changed somehow;
+display a refreshed rendering of it", that's sufficient.
+
+----
+
+First use of that was to make the thread that woke up every 10 minutes
+and checkpointed the daemon status to disk also wait for a notification
+that it changed. So that'll be more current, and use less IO.
+
+----
+
+Second use, of course, was to make the WebApp block long polling clients
+until there is really a change since the last time the client polled.
+
+To do that, I made one change to my Yesod routes:
+
+[[!format diff """
+	-/status StatusR GET
+	+/status/#NotificationId StatusR GET
+"""]]
+
+Now I find another reason to love Yesod, because after doing that,
+I hit "make".. and fixed the type error. And hit make.. and fixed
+the type error. And then it just freaking worked! Html was generated with
+all urls to /status including a `NotificationId`, and the handler for
+that route got it and was able to use it:
+
+[[!format haskell """
+	{- Block until there is an updated status to display. -}
+	b <- liftIO $ getNotificationBroadcaster webapp
+	liftIO $ waitNotification $ notificationHandleFromId b nid
+"""]]
+
+And now the WebApp is able to display transfers in realtime!
+When I have both the WebApp and `git annex get` running on the same screen,
+the WebApp displays files that git-annex is transferring about as fast
+as the terminal updates.
+
+The [[progressbars]] still need to be sorted out, but otherwise
+the WebApp is a nice live view of file transfers.
+
+---
+
+I also had some fun with Software Transactional Memory. Now when the
+assistant moves a transfer from its queue of transfers to do, to its map of
+transfers that are currently running, it does so in an atomic transaction.
+This will avoid the transfer seeming to go missing (or be listed twice) if
+the webapp refreshes at just the wrong point in time. I'm really starting
+to get into STM.
+
+----
+
+Next up, I will be making the WebApp maintain a list of notices, displayed
+on its sidebar, scrolling new notices into view, and removing ones the user
+closes, and ones that expire. This will be used for displaying errors, as
+well as other communication with the user (such as displaying a notice
+while a git sync is in progress with a remote, etc). Seems worth doing now,
+so the basic UI of the WebApp is complete with no placeholders.
diff --git a/doc/design/assistant/blog/day_47__alert_messages.mdwn b/doc/design/assistant/blog/day_47__alert_messages.mdwn
new file mode 100644
index 000000000..81551fa95
--- /dev/null
+++ b/doc/design/assistant/blog/day_47__alert_messages.mdwn
@@ -0,0 +1,14 @@
+Some days I spend 2 hours chasing red herrings (like "perhaps my JSON ajax
+calls arn't running asynchronoously?") that turn out to be a simple
+one-word typo. This was one of them.
+
+However, I did get the sidebar displaying alert messages, which can be
+easily sent to the user from any part of the assistant. This includes
+transient alerts of things it's doing, which disappear once the action
+finishes, and long-term alerts that are displayed until the user closes
+them. It even supports rendering arbitrary Yesod widgets as alerts, so
+they can also be used for asking questions, etc.
+
+Time for a screencast!
+
+<video controls src="http://joeyh.name/screencasts/git-annex-webapp.ogg"></video>
diff --git a/doc/design/assistant/partial_content/comment_1_58c4faa321a5bb71adf9fdee079849f4._comment b/doc/design/assistant/partial_content/comment_1_58c4faa321a5bb71adf9fdee079849f4._comment
new file mode 100644
index 000000000..aa65cd6b9
--- /dev/null
+++ b/doc/design/assistant/partial_content/comment_1_58c4faa321a5bb71adf9fdee079849f4._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="https://www.google.com/accounts/o8/id?id=AItOawkpEY8WTFDhjIVTWG38Ph7ppmuXUTJAHAg"
+ nickname="Justin"
+ subject="selective sync"
+ date="2012-07-28T04:08:00Z"
+ content="""
+hey joey
+
+great work!!
+
+will partial content work like selective sync in dropbox
+
+use case: on desktop i have photos/mp3s/docs, but would only want to sync the docs to my netbook
+
+cheers
+
+justin
+"""]]
diff --git a/doc/design/assistant/webapp.mdwn b/doc/design/assistant/webapp.mdwn
index 018d70886..ebf168996 100644
--- a/doc/design/assistant/webapp.mdwn
+++ b/doc/design/assistant/webapp.mdwn
@@ -16,7 +16,7 @@ The webapp is a web server that displays a shiny interface.
 
 ## interface
 
-* list of files uploading and downloading
+* list of files uploading and downloading **done**
 * progress bars for each file
 * drag and drop to reorder
 * cancel and pause
@@ -27,15 +27,14 @@ The webapp is a web server that displays a shiny interface.
 * there could be a UI to export a file, which would make it be served up
   over http by the web app
 * Display any relevant warning messages. One is the `inotify max_user_watches`
-  exceeded message. Need to lift such messages into DaemonStatus
-  so the WebApp can include them in its rendering of DaemonStatus.
+  exceeded message.
 
 ## implementation
 
-* perhaps define a custom `errorHandler`, which could avoid the potential
-  of leaking auth tokens on error pages. Or make the test suite test for
-  leakage.
 * possibly lose the ugly auth= token past the first page,
   and use a client-side session. It could be encrypted using the token
   as the `encryptKey`. Note: Would need to set the session duration
   to infinite (how?)
+* Fix notification handle leakage on pages other than the main page.
+  The javascript should use AJAX to request handles, that way
+  they won't be allocated at all in noscript.
diff --git a/doc/sync.mdwn b/doc/sync.mdwn
index 765c1e43f..057dcb355 100644
--- a/doc/sync.mdwn
+++ b/doc/sync.mdwn
@@ -1,7 +1,7 @@
 The `git annex sync` command provides an easy way to keep several
 repositories in sync. 
 
-Often git is used in a centralized fashion with a central bare repositry
+Often git is used in a centralized fashion with a central bare repository
 which changes are pulled and pushed to using normal git commands.
 That works fine, if you don't mind having a central repository.