rfc for notmuch remote access script

rfc for notmuch remote access script

[Tomi Ollila]

Hi

j4ni on irc expressed interest of having an installation option for
notmuch-emacs and a notmuch remote access script as 'notmuch'

This got me thinking what kind of script would fulfill all the needs
that I know of.

and this is the result of that:

--8<----8<----8<----8<----8<----8<----8<----8<--

#!/bin/bash

set -euf

if [ "${NOTMUCH_REMOTE_XTRACE_FILE-}" ]; then
	exec 6>>"$NOTMUCH_REMOTE_XTRACE_FILE"
	BASH_XTRACEFD=6
	echo -- >&6
	set -x
	env >&6
fi

: ${NOTMUCH_REMOTE_SSH_COMMAND:=ssh}
: ${NOTMUCH_REMOTE_SSH_ARGS=}
: ${NOTMUCH_REMOTE_HOST:=notmuch}
: ${NOTMUCH_REMOTE_COMMAND:=notmuch}

printf -v ARGS '%q ' "$@"

exec "$NOTMUCH_REMOTE_SSH_COMMAND" $NOTMUCH_REMOTE_SSH_ARGS \
	"$NOTMUCH_REMOTE_HOST" $NOTMUCH_REMOTE_COMMAND $ARGS

--8<----8<----8<----8<----8<----8<----8<----8<--

out of the box this works like the instructions given in
https://notmuchmail.org/remoteusage/ (with ~/.ssh/config changes)
and https://notmuchmail.org/remoteusage/124/ (with suitable wrapping
so that contents of ~/.ssh/ can be left unmodified)

also, debugging is easy. (setenv "NOTMUCH_REMOTE_XTRACE_FILE" "xtrace.log")
inside emacs, and NOTMUCH_REMOTE_XTRACE_FILE=xtrace.log notmuch-remote
on the command line.

what do you think?

Tomi

Re: rfc for notmuch remote access script

[Matt Armstrong]

Neat.  Basics of it look correct to me.  Personally, I'd abandon most of
these environment variables and edit the script directly, but that goes
against your stated goal.

Other comments below.


Tomi Ollila <domo141@gmail.com> writes:

> Hi
>
> j4ni on irc expressed interest of having an installation option for
> notmuch-emacs and a notmuch remote access script as 'notmuch'
>
> This got me thinking what kind of script would fulfill all the needs
> that I know of.
>
> and this is the result of that:
>
> --8<----8<----8<----8<----8<----8<----8<----8<--
>
> #!/bin/bash
>
> set -euf
>
> if [ "${NOTMUCH_REMOTE_XTRACE_FILE-}" ]; then

In bash, I often see people favoring this style:

if [[ -n ${NOTMUCH_REMOTE_XTRACE_FILE-} ]]; then


> 	exec 6>>"$NOTMUCH_REMOTE_XTRACE_FILE"
> 	BASH_XTRACEFD=6
> 	echo -- >&6
> 	set -x
> 	env >&6

This BASH_XTRACEFD stuff is nice, and the technique is new to me.  Is
there a way to close the trace file descriptor so it isn't available to
the ssh command?  Maybe something like this works?

exec 6>&- "$NOTMUCH_REMOTE_SSH_COMMAND" $NOTMUCH_REMOTE_SSH_ARGS \
  "$NOTMUCH_REMOTE_HOST" $NOTMUCH_REMOTE_COMMAND $ARGS

(though that only needs to happen optionally, which makes it tricky).

> : ${NOTMUCH_REMOTE_SSH_COMMAND:=ssh}
> : ${NOTMUCH_REMOTE_SSH_ARGS=}
> : ${NOTMUCH_REMOTE_HOST:=notmuch}
> : ${NOTMUCH_REMOTE_COMMAND:=notmuch}
>
> printf -v ARGS '%q ' "$@"

This is the real secret sauce.  When I was looking at removing the
extraneous quoting we have in notmuch-show.el, it boiled down to
figuring this out.  The web has a lot of people asking how to do this
properly, but very few good answers.

Re: rfc for notmuch remote access script

[Jani Nikula]

On Thu, Oct 27, 2016 at 8:25 PM, Tomi Ollila <domo141@gmail.com> wrote:
> j4ni on irc expressed interest of having an installation option for
> notmuch-emacs and a notmuch remote access script as 'notmuch'
>
> This got me thinking what kind of script would fulfill all the needs
> that I know of.

This here fulfills all of my needs:

#!/bin/bash
printf -v ARGS "%q " "$@"
exec ssh notmuch notmuch ${ARGS}

Keep it simple, I say. The user can (and must) configure all the ssh
stuff in ~/.ssh/config under "Host notmuch". If there is something
non-trivial the user must do, taking a trivial script and editing it
is usually *much* easier than trying to understand a complicated
script and making that work using environment variables etc. that need
to be set somewhere.

BR,
Jani.

Re: rfc for notmuch remote access script

[David Bremner]

Jani Nikula <jani@nikula.org> writes:

> On Thu, Oct 27, 2016 at 8:25 PM, Tomi Ollila <domo141@gmail.com> wrote:
>> j4ni on irc expressed interest of having an installation option for
>> notmuch-emacs and a notmuch remote access script as 'notmuch'
>>
>> This got me thinking what kind of script would fulfill all the needs
>> that I know of.
>
> This here fulfills all of my needs:
>
> #!/bin/bash
> printf -v ARGS "%q " "$@"
> exec ssh notmuch notmuch ${ARGS}
>
> Keep it simple, I say. The user can (and must) configure all the ssh
> stuff in ~/.ssh/config under "Host notmuch". If there is something
> non-trivial the user must do, taking a trivial script and editing it
> is usually *much* easier than trying to understand a complicated
> script and making that work using environment variables etc. that need
> to be set somewhere.

I'm not necessarily disagreeing, but then I don't really understand your
desire to have wish (I thought) to have notmuch install this script? Do
you typically have notmuch install into your home directory?

d

Re: rfc for notmuch remote access script

[Jani Nikula]

On Fri, 28 Oct 2016, David Bremner <david@tethera.net> wrote:
> Jani Nikula <jani@nikula.org> writes:
>
>> On Thu, Oct 27, 2016 at 8:25 PM, Tomi Ollila <domo141@gmail.com> wrote:
>>> j4ni on irc expressed interest of having an installation option for
>>> notmuch-emacs and a notmuch remote access script as 'notmuch'
>>>
>>> This got me thinking what kind of script would fulfill all the needs
>>> that I know of.
>>
>> This here fulfills all of my needs:
>>
>> #!/bin/bash
>> printf -v ARGS "%q " "$@"
>> exec ssh notmuch notmuch ${ARGS}
>>
>> Keep it simple, I say. The user can (and must) configure all the ssh
>> stuff in ~/.ssh/config under "Host notmuch". If there is something
>> non-trivial the user must do, taking a trivial script and editing it
>> is usually *much* easier than trying to understand a complicated
>> script and making that work using environment variables etc. that need
>> to be set somewhere.
>
> I'm not necessarily disagreeing, but then I don't really understand your
> desire to have wish (I thought) to have notmuch install this script?

I'd like to be able to install a "notmuch-remote" setup: the notmuch
remote script, possibly bash completion, and notmuch-emacs. Basically
everything that's required on notmuch side for the complete working
notmuch remote setup. And specifically excluding notmuch binaries. Of
course, you'll still need to set up ssh to host "notmuch" and notmuch
binaries on that host.

Tomi's general approach seems to be to add a bunch of environment
variables in the remote access script so you don't have to touch
~/.ssh/config. Instead, you need to make sure you have all of these
environment variables correctly set up, both in shell and in your emacs
session. And to do that, you'll need to either read the script or read a
bunch of documentation (that someone will have to write).

Personally, I prefer a simple script that assumes a certain type of ssh
client configuration. We can typically reference documentation written
by others how to do that, and just say "put that stuff under a Host
notmuch section in ~/.ssh/config".

So I'd like notmuch remote installation that tells the user to set up
ssh, instead of telling the user to copy a script somewhere in $PATH,
and hack that up along with making compatible changes in ssh config.

> Do you typically have notmuch install into your home directory?

No. Please explain the relevance of this question.


BR,
Jani.

Re: rfc for notmuch remote access script

[David Bremner]

Jani Nikula <jani@nikula.org> writes:

>
> Personally, I prefer a simple script that assumes a certain type of ssh
> client configuration. We can typically reference documentation written
> by others how to do that, and just say "put that stuff under a Host
> notmuch section in ~/.ssh/config".
>

OK, I misunderstood, and thought you wanted to edit the script after
installation. Hence my question about installing into $HOME.

I'm not using remote access, and I don't really have opinions about the
best way to do it. I do have 2 concerns about the overall idea

1. I worry about the maintenance burden of extra code ./configure
2. I worry about promoting remote-notmuch for non-experts when the
   situation with gpg seems quite broken, at least for people not
   willing to store private key material on the server.

Re: rfc for notmuch remote access script

[Mark Walters]

On Sun, 30 Oct 2016, David Bremner <david@tethera.net> wrote:
> Jani Nikula <jani@nikula.org> writes:
>
>>
>> Personally, I prefer a simple script that assumes a certain type of ssh
>> client configuration. We can typically reference documentation written
>> by others how to do that, and just say "put that stuff under a Host
>> notmuch section in ~/.ssh/config".

Hi

The only possible downside I can see with this is if the address for the
notmuch-server varies. For example my router does NAT the address from
inside and outside my home network is different. I don't know if that is
a common setup, and quite plausibly I could configure things better. I
only mention it as it might be easier to change an environment variable,
than change the .ssh/config file.

Quite plausibly though the simpler common case outweighs that. If we did
want to support my use case above, maybe we could have an environment
variable to say which .ssh/config section to use, which most people
could completely ignore?

> OK, I misunderstood, and thought you wanted to edit the script after
> installation. Hence my question about installing into $HOME.
>
> I'm not using remote access, and I don't really have opinions about the
> best way to do it. I do have 2 concerns about the overall idea
>
> 1. I worry about the maintenance burden of extra code ./configure
> 2. I worry about promoting remote-notmuch for non-experts when the
>    situation with gpg seems quite broken, at least for people not
>    willing to store private key material on the server.

I wonder if we could do the following --

1) have 2 separate config targets for notmuch-cli and notmuch-emacs so
you can install one without the other.

2) Have things under contrib which config/make can install. Then this
script could go in contrib, which would be a clear signal that it is not
fully supported.

Moreover, point 2 would go very nicely with Jani's other patch
id:1477140605-3011-1-git-send-email-jani@nikula.org for notmuch prefixed
commands. It would mean we could have various notmuch extensions in
contrib and the user could install any they wanted.

Best wishes

Mark

Re: rfc for notmuch remote access script

[Jani Nikula]

On Sun, Oct 30, 2016 at 2:39 PM, Mark Walters <markwalters1009@gmail.com> wrote:
> The only possible downside I can see with this is if the address for the
> notmuch-server varies. For example my router does NAT the address from
> inside and outside my home network is different. I don't know if that is
> a common setup, and quite plausibly I could configure things better. I
> only mention it as it might be easier to change an environment variable,
> than change the .ssh/config file.

Arguably you should figure out how to use the ssh config Match keyword
to automagically configure the ssh client depending on which side of
the network you are. I'd imagine this is helpful outside of the remote
notmuch context too.

BR,
Jani.

Re: rfc for notmuch remote access script

[David Bremner]

Mark Walters <markwalters1009@gmail.com> writes:

> 1) have 2 separate config targets for notmuch-cli and notmuch-emacs so
> you can install one without the other.

We already have the install target install-emacs. We have a configure
option --without-emacs.  We could add "--without-lib" and
"--without-cli" options to configure, but getting the various
dependencies right might be more work then it's worth.

Re: rfc for notmuch remote access script

[Tomi Ollila]

Thanks for the discussion so far. after sending this email I continue
tomorrow with the others...


Matt Armstrong <marmstrong@google.com> writes:

> Neat.  Basics of it look correct to me.  Personally, I'd abandon most of
> these environment variables and edit the script directly, but that goes
> against your stated goal.

In this case yes... the goal being me having a chance to drop one of
my scripts from the pool...

> Other comments below.
>
>>
>> if [ "${NOTMUCH_REMOTE_XTRACE_FILE-}" ]; then
>
> In bash, I often see people favoring this style:
>
> if [[ -n ${NOTMUCH_REMOTE_XTRACE_FILE-} ]]; then

That is a non-mandatory posix shell feature, works in bash, ksh, zsh
(and possibly in busybox shell), but not in dash. to share code snippets
between all shells rather use more portable syntax (and my personal
preference is `if test "${var-}"\n then ... fi`))

( as as side note, in some systems there is /bin/[[ -- that is just plain
stupid !!! ;/ it cannot possibly handle the syntax builtin [[ accepts )

>> 	exec 6>>"$NOTMUCH_REMOTE_XTRACE_FILE"
>> 	BASH_XTRACEFD=6
>> 	echo -- >&6
>> 	set -x
>> 	env >&6
>
> This BASH_XTRACEFD stuff is nice, and the technique is new to me.  Is
> there a way to close the trace file descriptor so it isn't available to
> the ssh command?  Maybe something like this works?
>
> exec 6>&- "$NOTMUCH_REMOTE_SSH_COMMAND" $NOTMUCH_REMOTE_SSH_ARGS \
>   "$NOTMUCH_REMOTE_HOST" $NOTMUCH_REMOTE_COMMAND $ARGS

that is good question. in case of ssh having 6 is not a problem -- the
fd 6 is not available in remote host (and it is just for investigating
problems) . but for the interested, something like this would work here
(just that there is extra x in the output ;/):

x () { exec 6>&-; "$@"; }

x exec ...

>
> (though that only needs to happen optionally, which makes it tricky).
>
>> : ${NOTMUCH_REMOTE_SSH_COMMAND:=ssh}
>> : ${NOTMUCH_REMOTE_SSH_ARGS=}
>> : ${NOTMUCH_REMOTE_HOST:=notmuch}
>> : ${NOTMUCH_REMOTE_COMMAND:=notmuch}
>>
>> printf -v ARGS '%q ' "$@"
>
> This is the real secret sauce.  When I was looking at removing the
> extraneous quoting we have in notmuch-show.el, it boiled down to
> figuring this out.  The web has a lot of people asking how to do this
> properly, but very few good answers.

... and the problem with e.g. stack overflow is that without karma one
cannot answer there...

Re: rfc for notmuch remote access script

[Jani Nikula]

On Sun, 30 Oct 2016, David Bremner <david@tethera.net> wrote:
> I'm not using remote access, and I don't really have opinions about the
> best way to do it. I do have 2 concerns about the overall idea
>
> 1. I worry about the maintenance burden of extra code ./configure
> 2. I worry about promoting remote-notmuch for non-experts when the
>    situation with gpg seems quite broken, at least for people not
>    willing to store private key material on the server.

Both are valid concerns, I think especially the second one. And even the
gpg case is not that simple; personally I'd prefer not storing private
key material on the client... ymmv.

BR,
Jani.