Remote usage script updated

Remote usage script updated

[Jesse Rosenthal]

Dear all,

Just a note to say that I finally got around to updating the remote
usage script on the wiki to what I'm using now. With "--format=raw" in,
it's all pretty straightforward. The only things the script does now
are:

1. Produces a slight pause in the "notmuch show" output to avoid that
   weird bug where emacs leaves off every tenth message or so.

2. Locally caches raw messages (i.e. when --format=raw) is called. This
   usually happens when getting attachments, so this is a nice way to
   avoid having to download large attachments repeatedly.

   Note this just caches based on msg-id (or a hash thereof, to avoid
   strange characters in file names). That means that if an attachment
   is deleted on the server, the cache will be out of date. An easy way
   to fix this would be to make the cache file name a concatenation of
   the msg-id hash (check that first) and the hash of the actual message
   (check that if the msg-id hash is there). I might put this in in the
   future, especially if anyone else is using the script.

3. Escapes dollar signs in the msg-id to make shell-quoting over ssh
   work.

I've actually switched over to keeping my messages on my IMAP server and
using this remote script on all of my computers. It avoids any need for
syncing. It's been working very well for me so far.

A future feature might be to integrate the ControlMaster feature of
openssh into the script, instead of having to open a connection
manually, but there are some complications there (dead sockets still
around if you go offline, etc.).

Best,
Jesse

Re: Remote usage script updated

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

On Thu, 27 Jan 2011 10:51:42 -0500, Jesse Rosenthal <jrosenthal@jhu.edu> wrote:
> Just a note to say that I finally got around to updating the remote
> usage script on the wiki to what I'm using now. With "--format=raw" in,
> it's all pretty straightforward. The only things the script does now
> are:

Thanks for passing on this note, Jesse!

> I've actually switched over to keeping my messages on my IMAP server and
> using this remote script on all of my computers. It avoids any need for
> syncing. It's been working very well for me so far.

It sounds entirely useful for a large class of users. Is there any
reason we shouldn't just host the script in the notmuch repository
itself? Let me know if there's anything I can do.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Remote usage script updated

[Jesse Rosenthal]

On Fri, 28 Jan 2011 12:54:17 +1000, Carl Worth <cworth@cworth.org> wrote:
> On Thu, 27 Jan 2011 10:51:42 -0500, Jesse Rosenthal <jrosenthal@jhu.edu> wrote
> It sounds entirely useful for a large class of users. Is there any
> reason we shouldn't just host the script in the notmuch repository
> itself? Let me know if there's anything I can do.
> 
> -Carl

I just realized I single-replied to Carl (after all the time I spent
making sure I didn't group-reply by accident).

Just to repeat briefly what I said there -- there was a slight problem
with yesterday's script (wild-cards were eaten by the shell) that should
now be fixed. And I'd be happy to house it in the repository if that
seems like it would be useful to people. But I'm curious about whether
we should have a separate tools/utilities folder. I know this came up
before with notmuch-deliver and other similar tools. What was the
outcome of that discussion?

Best,
Jesse

Re: Remote usage script updated

[Michal Sojka]

On Thu, 27 Jan 2011, Jesse Rosenthal wrote:
> Dear all,
> 
> Just a note to say that I finally got around to updating the remote
> usage script on the wiki to what I'm using now. With "--format=raw" in,
> it's all pretty straightforward. 

Hi Jesse,

thanks for this. I wanted to do this since the --format=raw was merged,
but didn't find the time for it. I have a few comments:

> The only things the script does now are:
> 
> 1. Produces a slight pause in the "notmuch show" output to avoid that
>    weird bug where emacs leaves off every tenth message or so.

I added a comment to the script to explain the sleep there. Otherwise,
people may delete the command because they think it is not needed.


> 2. Locally caches raw messages (i.e. when --format=raw) is called. This
>    usually happens when getting attachments, so this is a nice way to
>    avoid having to download large attachments repeatedly.

This is a good idea.

>    Note this just caches based on msg-id (or a hash thereof, to avoid
>    strange characters in file names). That means that if an attachment
>    is deleted on the server, the cache will be out of date. An easy way
>    to fix this would be to make the cache file name a concatenation of
>    the msg-id hash (check that first) and the hash of the actual message
>    (check that if the msg-id hash is there). I might put this in in the
>    future, especially if anyone else is using the script.
> 
> 3. Escapes dollar signs in the msg-id to make shell-quoting over ssh
>    work.

I think that quoting with
  printf -v args "%q " "$@"
instead of sed would work more reliably (see bash(1)).

> I've actually switched over to keeping my messages on my IMAP server and
> using this remote script on all of my computers. It avoids any need for
> syncing. It's been working very well for me so far.
> 
> A future feature might be to integrate the ControlMaster feature of
> openssh into the script, instead of having to open a connection
> manually, but there are some complications there (dead sockets still
> around if you go offline, etc.).

In another project I worked around the dead sockets this way:

sshgw() {
    local socket="$HOME/.ssh/cangw-connection"

    if [[ ! -S $socket ]] || ! ssh -x -a -S $socket root@192.168.2.3 true; then
	# Create master connection to speed up subsequent commands.
	ssh -N -f -M -S $socket root@192.168.2.3 >/dev/null 2>&1
    fi
    ssh -x -a -S $socket root@192.168.2.3 "$@"
}

I guess this should work even for notmuch.

-Michal

Re: Remote usage script updated

[Michal Sojka]

On Sat, 05 Feb 2011, Michal Sojka wrote:
> On Thu, 27 Jan 2011, Jesse Rosenthal wrote:
> > A future feature might be to integrate the ControlMaster feature of
> > openssh into the script, instead of having to open a connection
> > manually, but there are some complications there (dead sockets still
> > around if you go offline, etc.).
> 
> In another project I worked around the dead sockets this way:
> 
> sshgw() {
>     local socket="$HOME/.ssh/cangw-connection"
> 
>     if [[ ! -S $socket ]] || ! ssh -x -a -S $socket root@192.168.2.3 true; then
> 	# Create master connection to speed up subsequent commands.
> 	ssh -N -f -M -S $socket root@192.168.2.3 >/dev/null 2>&1
>     fi
>     ssh -x -a -S $socket root@192.168.2.3 "$@"
> }

Hmm, this code worked well with dropbear ssh server but it seems that
with openssh server the result is not that good. Namely, if the master
connection is dead, the command running true blocked for a long time. I
have always killed it before it timed out so I don't know whether it
blocks indefinitely or there is only long timeout.

I may look at this later, but I use notmuch remotely only from time to
time so this is not priority for me.

-Michal

Re: Remote usage script updated

[Jesse Rosenthal]

Michal,

On Sun, 06 Feb 2011 00:58:29 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> Hmm, this code worked well with dropbear ssh server but it seems that
> with openssh server the result is not that good. Namely, if the master
> connection is dead, the command running true blocked for a long time.

Seemed to work okay for me when I played around with it a bit (in
different circumstances, and with a confused laptop waking up from
hibernation). But I'll hold off on updating it till I can figure out the
most reliable way.

I'll definitely take your suggestion to use printf instead of manually
escaping characters. 

Thanks, in any event, for your suggestions!

By the way, I've also realized that most attachment downloading is done
not by "--format=raw" but by "notmuch part". It's possible to do caching
there as well. There are a few options there:

  * One option would be to just cache by the attachment number -- but
    this is very fragile if you delete an attachment through mutt or
    some other client that allows it.

  * cache by the hash of the attachment. The idea is that asking the
    server to fetch it, hash it, and send the hash would still save
    time over sending the whole attachment. Probably -- though most
    attachments are small enough and most connections are fast enough
    that this might not actually matter.

  * Actually stick the attachment hash in the json output in the first
    place. But this would be a lot of trouble for a small gain for
    very few.

It would come in handy when I'm trying to get a biggish pdf over a bus's
slow wireless connection. But it's certainly not a priority.

Best,
Jesse

Re: Remote usage script updated

[Michal Sojka]

On Wed, 09 Feb 2011, Jesse Rosenthal wrote:
> By the way, I've also realized that most attachment downloading is done
> not by "--format=raw" but by "notmuch part". It's possible to do caching
> there as well. There are a few options there:
> 
>   * One option would be to just cache by the attachment number -- but
>     this is very fragile if you delete an attachment through mutt or
>     some other client that allows it.

This is obviously the simplest possibility. I do not know how often do
you (and others) delete attachments, but I do it rarely and usually I
delete all the attachments of the messages older than something. So this
would work for me.

Additionally, I have added the following command to the attachment
fetching code: 

    find ${CACHE} -ctime +$CACHE_MAX_DAYS -and -mtime +$CACHE_MAX_DAYS -print0 | xargs -r -0 rm -f

It deletes attachments older than one week (in my case) so it there is
some inconsistency, it would not last more than one week.        

>   * cache by the hash of the attachment. The idea is that asking the
>     server to fetch it, hash it, and send the hash would still save
>     time over sending the whole attachment. Probably -- though most
>     attachments are small enough and most connections are fast enough
>     that this might not actually matter.

If people do more weird things with their attachments, this is probably
the best solution.

>   * Actually stick the attachment hash in the json output in the first
>     place. But this would be a lot of trouble for a small gain for
>     very few.

Agreed.

-Michal

Re: Remote usage script updated

[Michal Sojka]

On Wed, 09 Feb 2011, Jesse Rosenthal wrote:
> Michal,
> 
> On Sun, 06 Feb 2011 00:58:29 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > Hmm, this code worked well with dropbear ssh server but it seems that
> > with openssh server the result is not that good. Namely, if the master
> > connection is dead, the command running true blocked for a long time.
> 
> Seemed to work okay for me when I played around with it a bit (in
> different circumstances, and with a confused laptop waking up from
> hibernation). But I'll hold off on updating it till I can figure out the
> most reliable way.

Hi Jesse,

I've just found that there is a SSH option called ControlPersist and
this is the missing piece to reliable connection sharing for notmuch
remote script. This option is available since openssh 5.6
(http://www.openssh.org/txt/release-5.6) with a fix of race conditions
in 5.7 (http://www.openssh.org/txt/release-5.7).

Now I have in my script the following command:
  ssh -x -a -oControlMaster=auto -oControlPersist=600 -S $SOCKET $USER@$SSH_HOST $NOTMUCH_REMOTE_BIN ${CMD} ${args}

where SOCKET="${CACHE}/.ssh-socket-${USER}@${SSH_HOST}" and it works
perfectly.

Unfortunately, this option is not supported in recently released Debian
Squeeze but from http://bugs.debian.org/594295 seems that it might be
supported later, perhaps in a point release.

-Michal

Re: Remote usage script updated

[Kristoffer Ström]

Hey, one open question on the remote script, is anyone using address completion with this?

ATM i'm using the vala script for this, but i don't know if integrating this remotely would be the most elegant solution.

How do you running notmuch-remote make this work?

Best, Kristoffer

Re: Remote usage script updated

[Jesse Rosenthal]

I have mine remotely, just because I don't have a local database to
query. I made a shell wrapper, pretty much just like remote script, but
that only passes the query along, and then set that script as my
address-lookup program. 

Hmm... I guess the two could be combined, but configuring for your own
address script and so on seems like it might be more trouble than just
making the script from scratch.

On Thu, 03 Mar 2011 17:38:51 +0100, Kristoffer Ström <kristoffer@rymdkoloni.se> wrote:
> 
> Hey, one open question on the remote script, is anyone using address completion with this?
> 
> ATM i'm using the vala script for this, but i don't know if integrating this remotely would be the most elegant solution.
> 
> How do you running notmuch-remote make this work?
> 
> Best, Kristoffer