From c8cc5cc8da5d4467a12b8723a700769d2ffb1666 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawkpEY8WTFDhjIVTWG38Ph7ppmuXUTJAHAg" Date: Sat, 28 Jul 2012 04:08:00 +0000 Subject: Added a comment: selective sync --- ...comment_1_58c4faa321a5bb71adf9fdee079849f4._comment | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/design/assistant/partial_content/comment_1_58c4faa321a5bb71adf9fdee079849f4._comment diff --git a/doc/design/assistant/partial_content/comment_1_58c4faa321a5bb71adf9fdee079849f4._comment b/doc/design/assistant/partial_content/comment_1_58c4faa321a5bb71adf9fdee079849f4._comment new file mode 100644 index 000000000..aa65cd6b9 --- /dev/null +++ b/doc/design/assistant/partial_content/comment_1_58c4faa321a5bb71adf9fdee079849f4._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkpEY8WTFDhjIVTWG38Ph7ppmuXUTJAHAg" + nickname="Justin" + subject="selective sync" + date="2012-07-28T04:08:00Z" + content=""" +hey joey + +great work!! + +will partial content work like selective sync in dropbox + +use case: on desktop i have photos/mp3s/docs, but would only want to sync the docs to my netbook + +cheers + +justin +"""]] -- cgit v1.2.3 From d7e696f38c73dfb351df187fb783d54acf275634 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawl_x1UcmuSOmX7WkBTJPQwOP1FgUPdWEWM" Date: Sat, 28 Jul 2012 21:09:23 +0000 Subject: fixed typo: repositry --- doc/sync.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sync.mdwn b/doc/sync.mdwn index 765c1e43f..057dcb355 100644 --- a/doc/sync.mdwn +++ b/doc/sync.mdwn @@ -1,7 +1,7 @@ The `git annex sync` command provides an easy way to keep several repositories in sync. -Often git is used in a centralized fashion with a central bare repositry +Often git is used in a centralized fashion with a central bare repository which changes are pulled and pushed to using normal git commands. That works fine, if you don't mind having a central repository. -- cgit v1.2.3 From 238641261c72a6850a067d970163c9213fe3efef Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Sat, 28 Jul 2012 21:53:04 -0400 Subject: blog for the day Wow, what a nice productive day this was! --- .../assistant/blog/day_46__notification_pools.mdwn | 68 ++++++++++++++++++++++ doc/design/assistant/webapp.mdwn | 2 +- 2 files changed, 69 insertions(+), 1 deletion(-) create mode 100644 doc/design/assistant/blog/day_46__notification_pools.mdwn diff --git a/doc/design/assistant/blog/day_46__notification_pools.mdwn b/doc/design/assistant/blog/day_46__notification_pools.mdwn new file mode 100644 index 000000000..d6283ddba --- /dev/null +++ b/doc/design/assistant/blog/day_46__notification_pools.mdwn @@ -0,0 +1,68 @@ +Focus today was writing a notification broadcaster library. This is a way to +send a notification to a set of clients, any of which can be blocked +waiting for a new notification to arrive. A complication is that any number +of clients may be be dead, and we don't want stale notifications for those +clients to pile up and leak memory. + +It took me 3 tries to find the solution, which turns out to be head-smackingly +simple: An array of SampleVars, one per client. + +Using SampleVars means that clients only see the most recent notification, +but when the notification is just "the assistant's state changed somehow; +display a refreshed rendering of it", that's sufficient. + +---- + +First use of that was to make the thread that woke up every 10 minutes +and checkpointed the daemon status to disk also wait for a notification +that it changed. So that'll be more current, and use less IO. + +---- + +Second use, of course, was to make the WebApp block long polling clients +until there is really a change since the last time the client polled. + +To do that, I made one change to my Yesod routes: + +[[!format diff """ + -/status StatusR GET + +/status/#NotificationId StatusR GET +"""]] + +Now I find another reason to love Yesod, because after doing that, +I hit "make".. and fixed the type error. And hit make.. and fixed +the type error. And then it just freaking worked! Html was generated with +all urls to /status including a `NotificationId`, and the handler for +that route got it and was able to use it: + +[[!format haskell """ + {- Block until there is an updated status to display. -} + b <- liftIO $ getNotificationBroadcaster webapp + liftIO $ waitNotification $ notificationHandleFromId b nid +""]] + +And now the WebApp is able to display transfers in realtime! +When I have both the WebApp and `git annex get` running on the same screen, +the WebApp displays files that git-annex is transferring about as fast +as the terminal updates. + +The [[progressbars]] still need to be sorted out, but otherwise +the WebApp is a nice live view of file transfers. + +--- + +I also had some fun with Software Transactional Memory. Now when the +assistant moves a transfer from its queue of transfers to do, to its map of +transfers that are currently running, it does so in an atomic transaction. +This will avoid the transfer seeming to go missing (or be listed twice) if +the webapp refreshes at just the wrong point in time. I'm really starting +to get into STM. + +---- + +Next up, I will be making the WebApp maintain a list of notices, displayed +on its sidebar, scrolling new notices into view, and removing ones the user +closes, and ones that expire. This will be used for displaying errors, as +well as other communication with the user (such as displaying a notice +while a git sync is in progress with a remote, etc). Seems worth doing now, +so the basic UI of the WebApp is complete with no placeholders. diff --git a/doc/design/assistant/webapp.mdwn b/doc/design/assistant/webapp.mdwn index 018d70886..1fc32282a 100644 --- a/doc/design/assistant/webapp.mdwn +++ b/doc/design/assistant/webapp.mdwn @@ -16,7 +16,7 @@ The webapp is a web server that displays a shiny interface. ## interface -* list of files uploading and downloading +* list of files uploading and downloading **done** * progress bars for each file * drag and drop to reorder * cancel and pause -- cgit v1.2.3 From ff9aeda585c2e9ad80c1f6a4e74e46fe804e5dbd Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Sat, 28 Jul 2012 21:56:37 -0400 Subject: format --- doc/design/assistant/blog/day_46__notification_pools.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/assistant/blog/day_46__notification_pools.mdwn b/doc/design/assistant/blog/day_46__notification_pools.mdwn index d6283ddba..483ad95b1 100644 --- a/doc/design/assistant/blog/day_46__notification_pools.mdwn +++ b/doc/design/assistant/blog/day_46__notification_pools.mdwn @@ -39,7 +39,7 @@ that route got it and was able to use it: {- Block until there is an updated status to display. -} b <- liftIO $ getNotificationBroadcaster webapp liftIO $ waitNotification $ notificationHandleFromId b nid -""]] +"""]] And now the WebApp is able to display transfers in realtime! When I have both the WebApp and `git annex get` running on the same screen, -- cgit v1.2.3 From 69c46c1004e70cc159dd23bdacacc0727a621acf Mon Sep 17 00:00:00 2001 From: jtang Date: Sun, 29 Jul 2012 10:49:05 +0000 Subject: Added a comment --- .../comment_10_6c872dff4fcc63c16bf69d1e96891c89._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/Watch_command_as_of_commit_6cecc26206c4a539999b04664136c6f785211a41_segfaults/comment_10_6c872dff4fcc63c16bf69d1e96891c89._comment diff --git a/doc/bugs/Watch_command_as_of_commit_6cecc26206c4a539999b04664136c6f785211a41_segfaults/comment_10_6c872dff4fcc63c16bf69d1e96891c89._comment b/doc/bugs/Watch_command_as_of_commit_6cecc26206c4a539999b04664136c6f785211a41_segfaults/comment_10_6c872dff4fcc63c16bf69d1e96891c89._comment new file mode 100644 index 000000000..952ffadc4 --- /dev/null +++ b/doc/bugs/Watch_command_as_of_commit_6cecc26206c4a539999b04664136c6f785211a41_segfaults/comment_10_6c872dff4fcc63c16bf69d1e96891c89._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="jtang" + ip="79.97.135.214" + subject="comment 10" + date="2012-07-29T10:49:05Z" + content=""" +This is looking good, no more segfaulting. +"""]] -- cgit v1.2.3 From fb2ef6090a613b6b76d5a91477e1bcf682aee1cd Mon Sep 17 00:00:00 2001 From: jtang Date: Sun, 29 Jul 2012 11:20:00 +0000 Subject: --- ...__47__committer_thread_loops_occassionally.mdwn | 49 ++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 doc/bugs/lsof__47__committer_thread_loops_occassionally.mdwn diff --git a/doc/bugs/lsof__47__committer_thread_loops_occassionally.mdwn b/doc/bugs/lsof__47__committer_thread_loops_occassionally.mdwn new file mode 100644 index 000000000..eb8a0ead0 --- /dev/null +++ b/doc/bugs/lsof__47__committer_thread_loops_occassionally.mdwn @@ -0,0 +1,49 @@ +I've noticed this to occur occassionally + +
+laplace:atest jtang$ ls
+1@  2@	3@  4@	5@  readme.txt@
+laplace:atest jtang$ git annex watch -d --foreground
+watch . [2012-07-29 11:49:26 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","show-ref","git-annex"] 
+[2012-07-29 11:49:26 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","show-ref","--hash","refs/heads/git-annex"] 
+[2012-07-29 11:49:26 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","log","refs/heads/git-annex..f85faa60e73efabc2e92f837b19c3918d3ab030f","--oneline","-n1"] 
+[2012-07-29 11:49:26 IST] chat: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","cat-file","--batch"] 
+(scanning...) [2012-07-29 11:49:26 IST] Assistant: all threads started
+[2012-07-29 11:49:26 IST] Merger: watching /Users/jtang/sandbox/atest/.git/refs/heads/synced
+[2012-07-29 11:49:26 IST] TransferWatcher: watching for transfers
+[2012-07-29 11:49:26 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","symbolic-ref","HEAD"] 
+[2012-07-29 11:49:26 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","add","--update"] 
+[2012-07-29 11:49:26 IST] Merger: merging changes into Just refs/heads/master
+[2012-07-29 11:49:26 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","merge","--no-edit","refs/heads/synced/master"] 
+(started...) [2012-07-29 11:49:26 IST] Watcher: watching .
+[2012-07-29 11:49:26 IST] WebApp: running on port 60042
+Already up-to-date.
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./1
+[2012-07-29 11:49:26 IST] chat: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","cat-file","--batch"] 
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./2
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./3
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./4
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./5
+[2012-07-29 11:49:26 IST] Watcher: add symlink ./readme.txt
+[2012-07-29 11:49:27 IST] Committer: committing 6 changes
+(Recording state in git...)
+[2012-07-29 11:49:27 IST] feed: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","update-index","-z","--index-info"] 
+[2012-07-29 11:49:27 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","commit","--allow-empty-message","-m","","--allow-empty","--quiet"] 
+[2012-07-29 11:49:28 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","symbolic-ref","HEAD"] 
+[2012-07-29 11:49:28 IST] Pusher: pushing to []
+[2012-07-29 11:49:28 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","branch","-f","synced/master"] 
+[2012-07-29 11:49:28 IST] read: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","symbolic-ref","HEAD"] 
+[2012-07-29 11:49:28 IST] Merger: merging changes into Just refs/heads/master
+[2012-07-29 11:49:28 IST] call: git ["--git-dir=/Users/jtang/sandbox/atest/.git","--work-tree=/Users/jtang/sandbox/atest","merge","--no-edit","refs/heads/synced/master"] 
+Already up-to-date.
+[2012-07-29 11:49:43 IST] Watcher: file added ./.gitignore
+[2012-07-29 11:49:43 IST] read: lsof ["-F0can","+d","/Users/jtang/sandbox/atest/.git/annex/tmp/"] 
+[2012-07-29 11:49:44 IST] Committer: delaying commit of 0 changes
+[2012-07-29 11:49:45 IST] read: lsof ["-F0can","+d","/Users/jtang/sandbox/atest/.git/annex/tmp/"] 
+[2012-07-29 11:49:46 IST] Committer: delaying commit of 0 changes
+[2012-07-29 11:49:47 IST] read: lsof ["-F0can","+d","/Users/jtang/sandbox/atest/.git/annex/tmp/"] 
+[2012-07-29 11:49:48 IST] Committer: delaying commit of 0 changes
+[2012-07-29 11:49:49 IST] read: lsof ["-F0can","+d","/Users/jtang/sandbox/atest/.git/annex/tmp/"] 
+
+ +I ran " git annex watch -d --foreground" to watch what was going one, and just created a .gitignore file and the the commiter/lsof thread just loops over and over.... I only noticed as my laptop battery had drained at somepoint when git-annex was running in the background. -- cgit v1.2.3 From 0b9ecea8ff19eec95263b0b682ec8417a1364587 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Sun, 29 Jul 2012 13:45:56 -0400 Subject: update --- doc/design/assistant/webapp.mdwn | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/doc/design/assistant/webapp.mdwn b/doc/design/assistant/webapp.mdwn index 1fc32282a..ebf168996 100644 --- a/doc/design/assistant/webapp.mdwn +++ b/doc/design/assistant/webapp.mdwn @@ -27,15 +27,14 @@ The webapp is a web server that displays a shiny interface. * there could be a UI to export a file, which would make it be served up over http by the web app * Display any relevant warning messages. One is the `inotify max_user_watches` - exceeded message. Need to lift such messages into DaemonStatus - so the WebApp can include them in its rendering of DaemonStatus. + exceeded message. ## implementation -* perhaps define a custom `errorHandler`, which could avoid the potential - of leaking auth tokens on error pages. Or make the test suite test for - leakage. * possibly lose the ugly auth= token past the first page, and use a client-side session. It could be encrypted using the token as the `encryptKey`. Note: Would need to set the session duration to infinite (how?) +* Fix notification handle leakage on pages other than the main page. + The javascript should use AJAX to request handles, that way + they won't be allocated at all in noscript. -- cgit v1.2.3 From 702fadd2837a0d3b982c86a79dae2d12d6683cec Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Sun, 29 Jul 2012 13:52:57 -0400 Subject: blog for the day + screencast --- doc/design/assistant/blog/day_46__alert_messages.mdwn | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/design/assistant/blog/day_46__alert_messages.mdwn diff --git a/doc/design/assistant/blog/day_46__alert_messages.mdwn b/doc/design/assistant/blog/day_46__alert_messages.mdwn new file mode 100644 index 000000000..d65b0b696 --- /dev/null +++ b/doc/design/assistant/blog/day_46__alert_messages.mdwn @@ -0,0 +1,16 @@ +Some days I spend 2 hours chasing red herrings (like "perhaps my JSON ajax +calls arn't running asynchronoously?") that turn out to be a simple +one-word typo. This was one of them. + +However, I did get the sidebar displaying alert messages, which can be +easily sent to the user from any part of the assistant. This includes +transient alerts of things it's doing, which disappear once the action +finishes, and long-term alerts that are displayed until the user closes +them. It even supports rendering arbitrary Yesod widgets as alerts, so +they can also be used for asking questions, etc. + +Time for a screencast! For some reason `recordmydesktop` has the sound +running around 1 second ahead of the video in this, but I think you'll +still get the idea. + + -- cgit v1.2.3 From 7f75c2574a35081ca1b1446bdf8adbda8afb2fdd Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Sun, 29 Jul 2012 18:10:13 +0000 Subject: Added a comment: probably a kqueue specific problem... --- .../comment_1_f8d1720aa26c719609720acf0772606e._comment | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/bugs/lsof__47__committer_thread_loops_occassionally/comment_1_f8d1720aa26c719609720acf0772606e._comment diff --git a/doc/bugs/lsof__47__committer_thread_loops_occassionally/comment_1_f8d1720aa26c719609720acf0772606e._comment b/doc/bugs/lsof__47__committer_thread_loops_occassionally/comment_1_f8d1720aa26c719609720acf0772606e._comment new file mode 100644 index 000000000..521d2e0cb --- /dev/null +++ b/doc/bugs/lsof__47__committer_thread_loops_occassionally/comment_1_f8d1720aa26c719609720acf0772606e._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.8.133" + subject="probably a kqueue specific problem..." + date="2012-07-29T18:10:12Z" + content=""" +I don't think this would eat your battery badly; it's looping for sure, but with a hardcoded 1 second delay I put in to guard against it eating all CPU. + +I tried to fix this in commit c4023f785834bc237e5fcdb69e275bbae10dd40b, but I sort of doubt I did. +I made one more commit that will at least tell us what file it is trying to check over and over with lsof. +"""]] -- cgit v1.2.3 From a1f4bb2f2cb35ba8d541c2dad2c43fd316c0fcef Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Sun, 29 Jul 2012 14:49:42 -0400 Subject: fix --- doc/design/assistant/blog/day_46__alert_messages.mdwn | 16 ---------------- doc/design/assistant/blog/day_47__alert_messages.mdwn | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 16 deletions(-) delete mode 100644 doc/design/assistant/blog/day_46__alert_messages.mdwn create mode 100644 doc/design/assistant/blog/day_47__alert_messages.mdwn diff --git a/doc/design/assistant/blog/day_46__alert_messages.mdwn b/doc/design/assistant/blog/day_46__alert_messages.mdwn deleted file mode 100644 index d65b0b696..000000000 --- a/doc/design/assistant/blog/day_46__alert_messages.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -Some days I spend 2 hours chasing red herrings (like "perhaps my JSON ajax -calls arn't running asynchronoously?") that turn out to be a simple -one-word typo. This was one of them. - -However, I did get the sidebar displaying alert messages, which can be -easily sent to the user from any part of the assistant. This includes -transient alerts of things it's doing, which disappear once the action -finishes, and long-term alerts that are displayed until the user closes -them. It even supports rendering arbitrary Yesod widgets as alerts, so -they can also be used for asking questions, etc. - -Time for a screencast! For some reason `recordmydesktop` has the sound -running around 1 second ahead of the video in this, but I think you'll -still get the idea. - - diff --git a/doc/design/assistant/blog/day_47__alert_messages.mdwn b/doc/design/assistant/blog/day_47__alert_messages.mdwn new file mode 100644 index 000000000..81551fa95 --- /dev/null +++ b/doc/design/assistant/blog/day_47__alert_messages.mdwn @@ -0,0 +1,14 @@ +Some days I spend 2 hours chasing red herrings (like "perhaps my JSON ajax +calls arn't running asynchronoously?") that turn out to be a simple +one-word typo. This was one of them. + +However, I did get the sidebar displaying alert messages, which can be +easily sent to the user from any part of the assistant. This includes +transient alerts of things it's doing, which disappear once the action +finishes, and long-term alerts that are displayed until the user closes +them. It even supports rendering arbitrary Yesod widgets as alerts, so +they can also be used for asking questions, etc. + +Time for a screencast! + + -- cgit v1.2.3 From e4a02f542265709f13a20b421d8b0160642499cd Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Sun, 29 Jul 2012 16:25:19 -0400 Subject: something I heard on irc --- doc/design/assistant/android.mdwn | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/design/assistant/android.mdwn b/doc/design/assistant/android.mdwn index f2711a479..65569e403 100644 --- a/doc/design/assistant/android.mdwn +++ b/doc/design/assistant/android.mdwn @@ -26,6 +26,10 @@ References: * * +I've heard anecdoally that ipwnstudios not only has an IPhone GHC port, +but also Android. Need to get in touch with them. + + ### Android specific features The app should be aware of power status, and avoid expensive background -- cgit v1.2.3