From 2bf0c6208c3252736d4b80ab661e895d3467b71f Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 25 Oct 2012 22:18:55 +0000 Subject: Added a comment --- .../comment_3_c23dc02c7487d63b0905f1b7f3ca59f5._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/bugs/git-annex:_getUserEntryForID:_failed___40__Success__41__/comment_3_c23dc02c7487d63b0905f1b7f3ca59f5._comment diff --git a/doc/bugs/git-annex:_getUserEntryForID:_failed___40__Success__41__/comment_3_c23dc02c7487d63b0905f1b7f3ca59f5._comment b/doc/bugs/git-annex:_getUserEntryForID:_failed___40__Success__41__/comment_3_c23dc02c7487d63b0905f1b7f3ca59f5._comment new file mode 100644 index 000000000..3b358b7ea --- /dev/null +++ b/doc/bugs/git-annex:_getUserEntryForID:_failed___40__Success__41__/comment_3_c23dc02c7487d63b0905f1b7f3ca59f5._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.118" + subject="comment 3" + date="2012-10-25T22:18:55Z" + content=""" +Well, git-annex needs to know the user name, and the home directory. I've made it use +USER, and HOME, when set, and only fall back to getpwent otherwise. +"""]] -- cgit v1.2.3 From 3eedd535e68b0219a4ee405f0d8d2236d9562543 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo" Date: Fri, 26 Oct 2012 05:49:47 +0000 Subject: Added a comment --- .../comment_4_0e8b28de5c173bc60ecc0126fb2209ca._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/git-annex:_getUserEntryForID:_failed___40__Success__41__/comment_4_0e8b28de5c173bc60ecc0126fb2209ca._comment diff --git a/doc/bugs/git-annex:_getUserEntryForID:_failed___40__Success__41__/comment_4_0e8b28de5c173bc60ecc0126fb2209ca._comment b/doc/bugs/git-annex:_getUserEntryForID:_failed___40__Success__41__/comment_4_0e8b28de5c173bc60ecc0126fb2209ca._comment new file mode 100644 index 000000000..63c3b474a --- /dev/null +++ b/doc/bugs/git-annex:_getUserEntryForID:_failed___40__Success__41__/comment_4_0e8b28de5c173bc60ecc0126fb2209ca._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawniayrgSdVLUc3c6bf93VbO-_HT4hzxmyo" + nickname="Tobias" + subject="comment 4" + date="2012-10-26T05:49:47Z" + content=""" +I think you mean the environment variables with \"use USER, and HOME\"? So I checked them I they are correct. +Reading man(3) getpwent says \"from the password database (e.g., the local password file /etc/passwd, NIS, and LDAP)\", so it should be no problem with the LDAP backend I'm using to log-in... +One other special thing: The home directory is on a NFS share. +"""]] -- cgit v1.2.3 From 661f55c360929a8e6679df702da66b02686dd2e0 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawlFUH5H4OUj9vMJIoXQs8bheiptgANQ6fU" Date: Fri, 26 Oct 2012 14:25:23 +0000 Subject: --- doc/forum/special_remote_for_IMAP.mdwn | 42 ++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 doc/forum/special_remote_for_IMAP.mdwn diff --git a/doc/forum/special_remote_for_IMAP.mdwn b/doc/forum/special_remote_for_IMAP.mdwn new file mode 100644 index 000000000..0d162e8b1 --- /dev/null +++ b/doc/forum/special_remote_for_IMAP.mdwn @@ -0,0 +1,42 @@ +I have implemented a special remote that stores files as email messages on an imap server. You need to install three utilities that the hooks invoke to deal with the email: mutt, imaputils, and munpack. I use mutt to send the email with the file as a mime attachment; imaputils talks to the imap server to check for and retrieve the message containing the desired file; and munpack extracts and decodes the attachment to get our file back. + +Several programs could be used in place of mutt, but the latter has a convenient command-line option for attaching files; mutt is of course available in the repositories of most linux distributions. + +imaputils is a perl program available at http://sourceforge.net/projects/imaputils/ +It has several perl library dependencies that you might need to download using the cpan tool. +What imaputils does for you is provide a command line tool for interacting with the mail server. You can search for mail with a particular subject (for example), delete mail, retrieve messages, and in general do anything that you can do with a mail client such as mutt, but from the command line rather than a curses interface. This allows you to in turn write scripts that talk to imap servers. + +munpack is part of the mpack package. This is included in the Ubuntu and Debian repositories, and can probably be easily obtained for most linuxes. munpack extracts and decodes mime attachments from the command line. + +I define the special remote with + + git annex initremote hogneygmail type=hook encryption=gitannex hooktype=hogneygmail + +The pgp key "gitannex" is a key established just for this purpose, that has no passphrase. This allows me to use encryption transparently. You could also use encryption=shared if your version of git-annex is recent enough. I also did + + git annex untrust hogneygmail + +Here are the hooks: + + hogneygmail-store-hook = mutt -n -s $ANNEX_KEY -a $ANNEX_FILE -- {email address} < /dev/null + hogneygmail-checkpresent-hook = "(imaputils.pl --conf {imap config file} --subject $ANNEX_KEY --count | grep -q \"1 messages\" -) && echo $ANNEX_KEY" + hogneygmail-retrieve-hook = "imaputils.pl --conf {imap config file} --subject $ANNEX_KEY --display | munpack -fq && mv $ANNEX_KEY $ANNEX_FILE; rm $ANNEX_KEY.desc" + hogneygmail-remove-hook = "imaputils.pl --conf {imap config file} --subject $ANNEX_KEY --delete" + +The bits inside of the curly brackets, for example {email address}, are what you need to specialize for your particular case (removing the brackets as well). The {imap config file} is a file that contains some configuration for imaputils. In my case it contains + + ssl + pass {password} + host {mail host} + user {mail username} + box {name of mailbox to check} + +The first line tells imaputils to use ssl, and the third line is the address of the mail host. I include my username and password so I won't need to type these in repeatedly. Of course this means that you need to protect this file carefully. + + + +The operation of the hooks is pretty straightforward. The store-hook attaches the file to an otherwise empty email message with a subject equal to the name of the key, and mails it off. Note that if you use encryption then the keys generated by git-annex here will not be the same ones you see on your local disc. The checkpresent-hook asks the imap server how many emails have the subject equal to the key we are looking for; only if the reply contains "1 messages" are we sure the right one is there. The retrieve-hook uses the "--display" option to imaputils to stream the message, pipes it to munpack to silently extract the attachment (which is our (encrypted) file), and moves the result into the file contents. It then cleans up by deleting the .desc file that I can't prevent munpack from leaving on the disk, even when the message is blank. The remove-hook passes the "-- delete" option to imaputils to supposedly delete the target message from the mail server. + +I've performed very limited testing of this, and my knowledge of git and, especially, git-annex is very primitive, so I'm sure this could be vastly improved. In my testing with gmail this seems to work fine, except that messages do not get deleted from the server - I don't know why. + +I've developed this as an experiment and proof of concept, and have no knowledge of whether actually using this is in accord with the terms of service of gmail or any other mail service you might be using, nor whether it is safe or a good idea. -- cgit v1.2.3 From aa82effacdfc8c0e2ecfbef7721df347506cd526 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 26 Oct 2012 14:27:21 -0400 Subject: blog for the day --- doc/assistant/xmpp.png | Bin 0 -> 27753 bytes doc/design/assistant/blog/day_115__my_new_form.mdwn | 17 +++++++++++++++++ 2 files changed, 17 insertions(+) create mode 100644 doc/assistant/xmpp.png create mode 100644 doc/design/assistant/blog/day_115__my_new_form.mdwn diff --git a/doc/assistant/xmpp.png b/doc/assistant/xmpp.png new file mode 100644 index 000000000..c3cc53ebf Binary files /dev/null and b/doc/assistant/xmpp.png differ diff --git a/doc/design/assistant/blog/day_115__my_new_form.mdwn b/doc/design/assistant/blog/day_115__my_new_form.mdwn new file mode 100644 index 000000000..d44565084 --- /dev/null +++ b/doc/design/assistant/blog/day_115__my_new_form.mdwn @@ -0,0 +1,17 @@ +Built a SRV lookup library that can use either `host` or ADNS. + +Worked on DBUS reconnection some more; found a FD leak in the dbus library, +and wrote its long-suffering author, John Millikin (also the XMPP library +author, so I've been bothering him a lot lately), who once again came +through with a quick fix. + +Built a XMPP configuration form, that tests the connection to the server. +Getting the wording right on this was hard, and it's probably still not +100% right. + +[[!img /assistant/xmpp.png]] + +Pairing over XMPP is something I'm still thinking about. It's +contingent on tunneling git over XMPP (actually not too hard), +and getting a really secure XMPP connection (needs library improvements, +as the library currently accepts any SSL certificate). -- cgit v1.2.3 From ab99a73ed01582224c401efae1f710f925d5bd46 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 26 Oct 2012 14:29:28 -0400 Subject: sign --- doc/forum/special_remote_for_IMAP.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/forum/special_remote_for_IMAP.mdwn b/doc/forum/special_remote_for_IMAP.mdwn index 0d162e8b1..2aa956551 100644 --- a/doc/forum/special_remote_for_IMAP.mdwn +++ b/doc/forum/special_remote_for_IMAP.mdwn @@ -40,3 +40,5 @@ The operation of the hooks is pretty straightforward. The store-hook attaches th I've performed very limited testing of this, and my knowledge of git and, especially, git-annex is very primitive, so I'm sure this could be vastly improved. In my testing with gmail this seems to work fine, except that messages do not get deleted from the server - I don't know why. I've developed this as an experiment and proof of concept, and have no knowledge of whether actually using this is in accord with the terms of service of gmail or any other mail service you might be using, nor whether it is safe or a good idea. + +-- Lee -- cgit v1.2.3 From 1bfd24862696258607baa4309699fc3a054a037e Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 26 Oct 2012 14:30:05 -0400 Subject: link --- doc/special_remotes.mdwn | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/special_remotes.mdwn b/doc/special_remotes.mdwn index 2f6020461..65fcb8768 100644 --- a/doc/special_remotes.mdwn +++ b/doc/special_remotes.mdwn @@ -22,6 +22,7 @@ for various cloud things: * [[tips/Internet_Archive_via_S3]] * [[tahoe-lafs|forum/tips:_special__95__remotes__47__hook_with_tahoe-lafs]] * [[tips/using_box.com_as_a_special_remote]] +* [[forum/special_remote_for_IMAP]] ## Unused content on special remotes -- cgit v1.2.3 From 3c0cc8b6c68b4b09668142b202a068b0ae23998c Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Fri, 26 Oct 2012 18:35:21 +0000 Subject: Added a comment: Neat idea --- .../comment_1_7c7d4b57a1b6508fff1a6b0508c861f8._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/forum/special_remote_for_IMAP/comment_1_7c7d4b57a1b6508fff1a6b0508c861f8._comment diff --git a/doc/forum/special_remote_for_IMAP/comment_1_7c7d4b57a1b6508fff1a6b0508c861f8._comment b/doc/forum/special_remote_for_IMAP/comment_1_7c7d4b57a1b6508fff1a6b0508c861f8._comment new file mode 100644 index 000000000..8d81dcfd0 --- /dev/null +++ b/doc/forum/special_remote_for_IMAP/comment_1_7c7d4b57a1b6508fff1a6b0508c861f8._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="Neat idea" + date="2012-10-26T18:35:21Z" + content=""" +I imagine that most mail servers have a maximum attachment size; that seems the most likely problem with using this. + +The directory special remote has support for splitting up keys into chunks to work around file size limits. It would be good, I suppose, to add that to the hook special remote too. +"""]] -- cgit v1.2.3 From 26fb68355b4ad95f99b2d14110e683109d0bab3d Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Fri, 26 Oct 2012 19:38:27 -0400 Subject: NetWatcher: When dbus connection is lost, try to reconnect. MountWatcher can't do this, because it uses the session dbus, and won't have access to the new DBUS_SESSION_BUS_ADDRESS if a new session is started. Bumped dbus library version, FD leak in it is fixed. --- Assistant/Threads/MountWatcher.hs | 5 +++++ Assistant/Threads/NetWatcher.hs | 17 +++++++++-------- Utility/DBus.hs | 16 ++++++++-------- debian/changelog | 1 + debian/control | 2 +- git-annex.cabal | 2 +- 6 files changed, 25 insertions(+), 18 deletions(-) diff --git a/Assistant/Threads/MountWatcher.hs b/Assistant/Threads/MountWatcher.hs index 294f9a972..a5200adfe 100644 --- a/Assistant/Threads/MountWatcher.hs +++ b/Assistant/Threads/MountWatcher.hs @@ -72,6 +72,11 @@ dbusThread st dstatus scanremotes = E.catch (runClient getSessionAddress go) one ) onerr :: E.SomeException -> IO () onerr e = do + {- If the session dbus fails, the user probably + - logged out of their desktop. Even if they log + - back in, we won't have access to the dbus + - session key, so polling is the best that can be + - done in this situation. -} runThreadState st $ warning $ "dbus failed; falling back to mtab polling (" ++ show e ++ ")" pollinstead diff --git a/Assistant/Threads/NetWatcher.hs b/Assistant/Threads/NetWatcher.hs index f9ca5641d..96b8007cc 100644 --- a/Assistant/Threads/NetWatcher.hs +++ b/Assistant/Threads/NetWatcher.hs @@ -24,7 +24,6 @@ import Utility.DBus import DBus.Client import DBus import Data.Word (Word32) -import qualified Control.Exception as E #else #warning Building without dbus support; will poll for network connection changes #endif @@ -57,22 +56,24 @@ netWatcherFallbackThread st dstatus scanremotes = thread $ #if WITH_DBUS dbusThread :: ThreadState -> DaemonStatusHandle -> ScanRemoteMap -> IO () -dbusThread st dstatus scanremotes = E.catch (runClient getSystemAddress go) onerr +dbusThread st dstatus scanremotes = persistentClient getSystemAddress () onerr go where go client = ifM (checkNetMonitor client) ( do - listenNMConnections client handle - listenWicdConnections client handle + listenNMConnections client handleconn + listenWicdConnections client handleconn , do runThreadState st $ warning "No known network monitor available through dbus; falling back to polling" ) - onerr :: E.SomeException -> IO () - onerr e = runThreadState st $ - warning $ "dbus failed; falling back to polling (" ++ show e ++ ")" - handle = do + handleconn = do debug thisThread ["detected network connection"] handleConnection st dstatus scanremotes + onerr e _ = do + runThreadState st $ + warning $ "lost dbus connection; falling back to polling (" ++ show e ++ ")" + {- Wait, in hope that dbus will come back -} + threadDelaySeconds (Seconds 60) {- Examine the list of services connected to dbus, to see if there - are any we can use to monitor network connections. -} diff --git a/Utility/DBus.hs b/Utility/DBus.hs index 3b34e00ac..a1a4c4804 100644 --- a/Utility/DBus.hs +++ b/Utility/DBus.hs @@ -5,7 +5,7 @@ - Licensed under the GNU GPL version 3 or higher. -} -{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE OverloadedStrings, ScopedTypeVariables #-} module Utility.DBus where @@ -65,15 +65,15 @@ runClient getaddr clientaction = do - If the connection is lost, runs onretry, which can do something like - a delay, or printing a warning, and has a state value (useful for - exponential backoff). Once onretry returns, the connection is retried. - - - - Warning: Currently connectWith can throw a SocketError and leave behind - - an open FD. So each retry leaks one FD. -} + -} persistentClient :: IO (Maybe Address) -> v -> (SomeException -> v -> IO v) -> (Client -> IO ()) -> IO () -persistentClient getaddr v onretry clientaction = do +persistentClient getaddr v onretry clientaction = {- runClient can fail with not just ClientError, but also other - - things, if dbus is not running. -} - r <- E.try (runClient getaddr clientaction) :: IO (Either SomeException ()) - either retry return r + - things, if dbus is not running. Let async exceptions through. -} + runClient getaddr clientaction `E.catches` + [ Handler (\ (e :: AsyncException) -> E.throw e) + , Handler (\ (e :: SomeException) -> retry e) + ] where retry e = do v' <- onretry e v diff --git a/debian/changelog b/debian/changelog index aa3eb9db5..36bba7c49 100644 --- a/debian/changelog +++ b/debian/changelog @@ -24,6 +24,7 @@ git-annex (3.20121018) UNRELEASED; urgency=low * configure: Check that checksum programs produce correct checksums. * Re-enable dbus, using a new version of the library that fixes the memory leak. + * NetWatcher: When dbus connection is lost, try to reconnect. * Use USER and HOME environment when set, and only fall back to getpwent, which doesn't work with LDAP or NIS. diff --git a/debian/control b/debian/control index ec648288a..ba8f571c8 100644 --- a/debian/control +++ b/debian/control @@ -22,7 +22,7 @@ Build-Depends: libghc-edit-distance-dev, libghc-hinotify-dev [linux-any], libghc-stm-dev (>= 2.3), - libghc-dbus-dev [linux-any] (>= 0.10.2), + libghc-dbus-dev [linux-any] (>= 0.10.3), libghc-yesod-dev [i386 amd64 kfreebsd-i386 kfreebsd-amd64], libghc-yesod-static-dev [i386 amd64 kfreebsd-i386 kfreebsd-amd64], libghc-yesod-default-dev [i386 amd64 kfreebsd-i386 kfreebsd-amd64], diff --git a/git-annex.cabal b/git-annex.cabal index 90620f4b3..5b7e78654 100644 --- a/git-annex.cabal +++ b/git-annex.cabal @@ -77,7 +77,7 @@ Executable git-annex C-Sources: Utility/libkqueue.c if os(linux) && flag(Dbus) - Build-Depends: dbus (>= 0.10.2) + Build-Depends: dbus (>= 0.10.3) CPP-Options: -DWITH_DBUS if flag(Webapp) && flag(Assistant) -- cgit v1.2.3