JSON based emacs UI

JSON based emacs UI

[David Edmondson]

I'm pushed the first stage of a JSON based emacs UI to the repository at
http://github.com/dme/notmuch (it's in the "master" branch).

This is intended to be functionally equivalent to the current UI, though
HTML display is not present (see below).

This is lightly tested, but if anyone else wants to give it a try I
would be interested in feedback. There were a couple of places in the
current code where I wasn't completely sure of the intention, so I may
have inadvertently diverged (notmuch-show-rewind is a good example).

For HTML display and other improved MIME support my intention is to
enhance notmuch to allow single parts to be output (something like:
"notmuch show --part=2 id:k74hf87df.fsf@fish.org") and use that from the
UI code. No idea when I'll get to it, though.

Carl - the main changeset is large, but I couldn't see a sensible way to
break it up into smaller pieces. I'm open to suggestions.

Finally, if anyone else is working on something like this, please let me
know - I'd be interested in collaborating.

ps. Thanks to aep in #notmuch for encouraging me to just get on with it.

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[David Edmondson]

On Mon, 22 Mar 2010 14:47:39 +0000, David Edmondson <dme@dme.org> wrote:
> This is lightly tested, but if anyone else wants to give it a try I
> would be interested in feedback. There were a couple of places in the
> current code where I wasn't completely sure of the intention, so I may
> have inadvertently diverged (notmuch-show-rewind is a good example).

Oops, it doesn't remove the "unread" tag.

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[David Edmondson]

On Mon, 22 Mar 2010 14:47:39 +0000, David Edmondson <dme@dme.org> wrote:
> I've pushed the first stage of a JSON based emacs UI to the repository
> at http://github.com/dme/notmuch (it's in the "master" branch).
> 
> This is intended to be functionally equivalent to the current UI, though
> HTML display is not present (see below).
> 
> This is lightly tested, but if anyone else wants to give it a try I
> would be interested in feedback. There were a couple of places in the
> current code where I wasn't completely sure of the intention, so I may
> have inadvertently diverged (notmuch-show-rewind is a good example).
> 
> For HTML display and other improved MIME support my intention is to
> enhance notmuch to allow single parts to be output (something like:
> "notmuch show --part=2 id:k74hf87df.fsf@fish.org") and use that from the
> UI code. No idea when I'll get to it, though.

"notmuch part --part=2 id:k74hf87df.fsf@fish.org" support is pushed now
and notmuch-show.el updated to use it.

An attempt is made to inline all parts, including guessing the type of
application/octet-stream parts using mailcap (so a file called
"foo.patch" sent as type application/octet-stream will be displayed
using diff-mode formatting).

Image inlining doesn't work, even for simple image/jpeg attachments. I'm
still puzzling over this - the output from 'notmuch part' seems fine,
but the lisp code to generate the image and insert it at the right place
is broken.

Inline images in HTML are also likely problematic - I'm not sure that I
have the framework in place for handling images specified using the cid:
prefix.

Next is to fix images and persuade notmuch to be more verbose about the
detailed MIME structure of messages.

As always, feedback (especially if something breaks) appreciated.

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[David Edmondson]

On Wed, 24 Mar 2010 07:32:41 +0000, David Edmondson <dme@dme.org> wrote:
> Image inlining doesn't work, even for simple image/jpeg attachments. I'm
> still puzzling over this - the output from 'notmuch part' seems fine,
> but the lisp code to generate the image and insert it at the right place
> is broken.

This is now fixed:
     http://dme.org/images/2010-03-24-153836_868x805_scrot.png

> Inline images in HTML are also likely problematic - I'm not sure that I
> have the framework in place for handling images specified using the cid:
> prefix.

Inline HTML images are still broken. This requires improving the JSON
output from notmuch so that the message structure is available to the UI
(and then a bunch more code in the UI).

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[Aneesh Kumar K. V]

On Wed, 24 Mar 2010 07:32:41 +0000, David Edmondson <dme@dme.org> wrote:
> On Mon, 22 Mar 2010 14:47:39 +0000, David Edmondson <dme@dme.org> wrote:
> > I've pushed the first stage of a JSON based emacs UI to the repository
> > at http://github.com/dme/notmuch (it's in the "master" branch).
> > 
> > This is intended to be functionally equivalent to the current UI, though
> > HTML display is not present (see below).
> > 
> > This is lightly tested, but if anyone else wants to give it a try I
> > would be interested in feedback. There were a couple of places in the
> > current code where I wasn't completely sure of the intention, so I may
> > have inadvertently diverged (notmuch-show-rewind is a good example).
> > 
> > For HTML display and other improved MIME support my intention is to
> > enhance notmuch to allow single parts to be output (something like:
> > "notmuch show --part=2 id:k74hf87df.fsf@fish.org") and use that from the
> > UI code. No idea when I'll get to it, though.
> 
> "notmuch part --part=2 id:k74hf87df.fsf@fish.org" support is pushed now
> and notmuch-show.el updated to use it.
> 
> An attempt is made to inline all parts, including guessing the type of
> application/octet-stream parts using mailcap (so a file called
> "foo.patch" sent as type application/octet-stream will be displayed
> using diff-mode formatting).
> 

Is there a way to get the inline patches shown similar to
diff-mode not just the attached patches ?

-aneesh

Re: JSON based emacs UI

[David Edmondson]

On Thu, 25 Mar 2010 10:11:38 +0530, "Aneesh Kumar K. V" <aneesh.kumar@linux.vnet.ibm.com> wrote:
> Is there a way to get the inline patches shown similar to
> diff-mode not just the attached patches ?

A new hook function for text/plain parts called
`notmuch-wash-inline-patch' is added that attempts to do this by
converting the inline patch into a fake attachment.

It guesses whether a buffer includes a patch and also the extent of that
patch, so it's a bit risky and hence not enabled by default. It is only
lightly tested. You can try it out by setting:

   (setq notmuch-show-insert-text/plain-hook
         '(notmuch-wash-wrap-long-lines
           notmuch-wash-tidy-citations
           notmuch-wash-compress-blanks
           notmuch-wash-inline-patch
           notmuch-wash-markup-citations))

The order is important!

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[Aneesh Kumar K. V]

On Thu, 25 Mar 2010 09:14:09 +0000, David Edmondson <dme@dme.org> wrote:
> On Thu, 25 Mar 2010 10:11:38 +0530, "Aneesh Kumar K. V" <aneesh.kumar@linux.vnet.ibm.com> wrote:
> > Is there a way to get the inline patches shown similar to
> > diff-mode not just the attached patches ?
> 
> A new hook function for text/plain parts called
> `notmuch-wash-inline-patch' is added that attempts to do this by
> converting the inline patch into a fake attachment.
> 
> It guesses whether a buffer includes a patch and also the extent of that
> patch, so it's a bit risky and hence not enabled by default. It is only
> lightly tested. You can try it out by setting:
> 
>    (setq notmuch-show-insert-text/plain-hook
>          '(notmuch-wash-wrap-long-lines
>            notmuch-wash-tidy-citations
>            notmuch-wash-compress-blanks
>            notmuch-wash-inline-patch
>            notmuch-wash-markup-citations))
> 
> The order is important!
> 

That worked. Thanks. BTW is there a way to get '=' to reload the thread
details. Something similar to notmuch-search-refresh-view. for the show
window. That would make sure i can get the tag updated tag details by
using = key. right now it is difficult to find out whether i "unread"
tag is removed from a message or not. The best fix would be to show the
updated tag dynamically 

-aneesh

Re: JSON based emacs UI

[David Edmondson]

On Fri, 26 Mar 2010 10:16:12 +0530, "Aneesh Kumar K. V" <aneesh.kumar@linux.vnet.ibm.com> wrote:
> That worked. Thanks. BTW is there a way to get '=' to reload the thread
> details. Something similar to notmuch-search-refresh-view. for the show
> window.

I'll look at adding it.

> That would make sure i can get the tag updated tag details by using =
> key. right now it is difficult to find out whether i "unread" tag is
> removed from a message or not. The best fix would be to show the
> updated tag dynamically

This is now fixed.

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[Jesse Rosenthal]

Hi David,

Thanks for this -- it really looks great so far.

> As always, feedback (especially if something breaks) appreciated.

There's a complaint when I open a (multipart?) message:

invalid function: mm-make-handle

Adding "(require 'mm-decode)" to notmuch-show.el fixes this problem.

Best,
Jesse

Re: JSON based emacs UI

[David Edmondson]

On Fri, 26 Mar 2010 05:07:49 -0400, Jesse Rosenthal <jrosenthal@jhu.edu> wrote:
> There's a complaint when I open a (multipart?) message:
> 
> invalid function: mm-make-handle
> 
> Adding "(require 'mm-decode)" to notmuch-show.el fixes this problem.

Sorry about that. The repository is updated accordingly.

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[Aneesh Kumar K. V]

On Mon, 22 Mar 2010 14:47:39 +0000, David Edmondson <dme@dme.org> wrote:

I found another issue. In the notmuch-show-mode "RET" key on from and
subject doesn't seem to work.

For ex: for the last message "RET" key on from doesn't fold the message.
For all the message "RET" key on subject doesn't show the message
header(from cc and to list)

-aneesh

Re: JSON based emacs UI

[David Edmondson]

On Fri, 26 Mar 2010 20:21:46 +0530, "Aneesh Kumar K. V" <aneesh.kumar@linux.vnet.ibm.com> wrote:
> On Mon, 22 Mar 2010 14:47:39 +0000, David Edmondson <dme@dme.org> wrote:
> 
> I found another issue. In the notmuch-show-mode "RET" key on from and
> subject doesn't seem to work.
> 
> For ex: for the last message "RET" key on from doesn't fold the message.
> For all the message "RET" key on subject doesn't show the message
> header(from cc and to list)

'RET' anywhere in the message is supposed to toggle visibility of the
message. That was not working perfectly - a little forced redisplay
helped (just pushed).

Toggling the visibility of headers is done with 'h'. There's currently
no way to do that by pressing 'RET' somewhere. (You can also toggle the
body with 'b'.)

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[Fernando Carrijo]

David Edmondson <dme@dme.org> wrote:
> 'RET' anywhere in the message is supposed to toggle visibility of the
> message. That was not working perfectly - a little forced redisplay
> helped (just pushed).
> 
> Toggling the visibility of headers is done with 'h'. There's currently
> no way to do that by pressing 'RET' somewhere. (You can also toggle the
> body with 'b'.)

I fear this is not correct. I cloned notmuch from Carl's git repository,
and whenever I press 'RET' while the point is over the subject line, the
result is the same as pressing 'h': the subject information gets toggled.

I didn't read this whole thread, though.
Maybe you're talking about some JSON powered feature-branch?

> 
> dme.
> -- 
> David Edmondson, http://dme.org

Re: JSON based emacs UI

[Fernando Carrijo]

Fernando Carrijo <fcarrijo@yahoo.com.br> wrote:
> David Edmondson <dme@dme.org> wrote:
> > 'RET' anywhere in the message is supposed to toggle visibility of the
> > message. That was not working perfectly - a little forced redisplay
> > helped (just pushed).
> > 
> > Toggling the visibility of headers is done with 'h'. There's currently
> > no way to do that by pressing 'RET' somewhere. (You can also toggle the
> > body with 'b'.)
> 
> I fear this is not correct. I cloned notmuch from Carl's git repository,
> and whenever I press 'RET' while the point is over the subject line, the
> result is the same as pressing 'h': the subject information gets toggled.
> 
> I didn't read this whole thread, though.
> Maybe you're talking about some JSON powered feature-branch?

Oh, now I see that you really are.

> 
> > 
> > dme.
> > -- 
> > David Edmondson, http://dme.org
> 

Re: JSON based emacs UI

[David Edmondson]

On Sun, 28 Mar 2010 14:03:44 -0300, Fernando Carrijo <fcarrijo@yahoo.com.br> wrote:
> David Edmondson <dme@dme.org> wrote:
> > 'RET' anywhere in the message is supposed to toggle visibility of the
> > message. That was not working perfectly - a little forced redisplay
> > helped (just pushed).
> > 
> > Toggling the visibility of headers is done with 'h'. There's currently
> > no way to do that by pressing 'RET' somewhere. (You can also toggle the
> > body with 'b'.)
> 
> I fear this is not correct. I cloned notmuch from Carl's git repository,
> and whenever I press 'RET' while the point is over the subject line, the
> result is the same as pressing 'h': the subject information gets toggled.

Is this considered a feature that blocks people from switching to the
JSON based UI?

dme.
-- 
David Edmondson, http://dme.org

Re: JSON based emacs UI

[Aneesh Kumar K. V]

On Mon, 29 Mar 2010 10:17:01 +0100, David Edmondson <dme@dme.org> wrote:
> On Sun, 28 Mar 2010 14:03:44 -0300, Fernando Carrijo <fcarrijo@yahoo.com.br> wrote:
> > David Edmondson <dme@dme.org> wrote:
> > > 'RET' anywhere in the message is supposed to toggle visibility of the
> > > message. That was not working perfectly - a little forced redisplay
> > > helped (just pushed).
> > > 
> > > Toggling the visibility of headers is done with 'h'. There's currently
> > > no way to do that by pressing 'RET' somewhere. (You can also toggle the
> > > body with 'b'.)
> > 
> > I fear this is not correct. I cloned notmuch from Carl's git repository,
> > and whenever I press 'RET' while the point is over the subject line, the
> > result is the same as pressing 'h': the subject information gets toggled.
> 
> Is this considered a feature that blocks people from switching to the
> JSON based UI?
> 

For me it is not an issue and i already switched to jason based ui. The
diff-mode for inline patch itself is a big motivation for me to switch.

-aneesh

Re: JSON based emacs UI

[Sebastian Spaeth]

On Mon, 29 Mar 2010 10:17:01 +0100, David Edmondson <dme@dme.org> wrote:
> > I fear this is not correct. I cloned notmuch from Carl's git repository,
> > and whenever I press 'RET' while the point is over the subject line, the
> > result is the same as pressing 'h': the subject information gets toggled.
> 
> Is this considered a feature that blocks people from switching to the
> JSON based UI?

nahh, that is fine.

Re: JSON based emacs UI

[Fernando Carrijo]

David Edmondson <dme@dme.org> wrote:
> On Sun, 28 Mar 2010 14:03:44 -0300, Fernando Carrijo <fcarrijo@yahoo.com.br> wrote:
> > David Edmondson <dme@dme.org> wrote:
> > > 'RET' anywhere in the message is supposed to toggle visibility of the
> > > message. That was not working perfectly - a little forced redisplay
> > > helped (just pushed).
> > > 
> > > Toggling the visibility of headers is done with 'h'. There's currently
> > > no way to do that by pressing 'RET' somewhere. (You can also toggle the
> > > body with 'b'.)
> > 
> > I fear this is not correct. I cloned notmuch from Carl's git repository,
> > and whenever I press 'RET' while the point is over the subject line, the
> > result is the same as pressing 'h': the subject information gets toggled.
> 
> Is this considered a feature that blocks people from switching to the
> JSON based UI?

As long as I'm concerned, no.

Not at all!

> 
> dme.
> -- 
> David Edmondson, http://dme.org

Re: JSON based emacs UI

[Sebastian Spaeth]

On Mon, 22 Mar 2010 14:47:39 +0000, David Edmondson <dme@dme.org> wrote:
> I'm pushed the first stage of a JSON based emacs UI to the repository at
> http://github.com/dme/notmuch (it's in the "master" branch).

Despite the switch to daylight savings time stealing me an hour of this
night, I managed to test your new notmuch.el.

It works really nice and I get the (unmeasured) feeling that it is
faster too (but then it might just be the lack of my additional patches
which slow things down :-)).

I have one question (which is more build related): now, I have to pull
your branch and an make install-emacs will also always compile and
install the notmuch binary from your branch, but I might want to keep my
own notmuch binary. Is it possible to just use the notmuch frontend from
your branch, but not having to install the binary? Is there such a make
target, or any other way? Should we create a repo that just contains the
frontend and not notmuch itself, so people can mix and match more
easily? (not sure what the right answer is here).

Sebastian

Re: JSON based emacs UI

[Michal Sojka]

On Sun, 28 Mar 2010, Sebastian Spaeth wrote:
> On Mon, 22 Mar 2010 14:47:39 +0000, David Edmondson <dme@dme.org> wrote:
> > I'm pushed the first stage of a JSON based emacs UI to the repository at
> > http://github.com/dme/notmuch (it's in the "master" branch).
> 
> Despite the switch to daylight savings time stealing me an hour of this
> night, I managed to test your new notmuch.el.
> 
> It works really nice and I get the (unmeasured) feeling that it is
> faster too (but then it might just be the lack of my additional patches
> which slow things down :-)).
> 
> I have one question (which is more build related): now, I have to pull
> your branch and an make install-emacs will also always compile and
> install the notmuch binary from your branch, but I might want to keep my
> own notmuch binary. Is it possible to just use the notmuch frontend from
> your branch, but not having to install the binary? Is there such a make
> target, or any other way? Should we create a repo that just contains the
> frontend and not notmuch itself, so people can mix and match more
> easily? (not sure what the right answer is here).

Hi Sebastian,

I don't think this can be solved only in Makefile. From my look at dme's
repo, he adds a new subcomand 'part', which is used by the UI. So if you
want to use the new UI and your other features, you need to merge the
things together.

To build my version of notmuch, I use an ugly script
(http://rtime.felk.cvut.cz/gitweb/notmuch.git/blob/refs/heads/debian-wsh:/wsh-buildpackage)
which first does a big octopus merge to combine several features to one
branch and then I build notmuch from there. The current state of my
integration can be seen at
http://rtime.felk.cvut.cz/gitweb/notmuch.git/shortlog/refs/heads/integration/features.

This approach has a disadvantage that integration/features branch is
often rewritten (whenever I add, remove or change a patch) so that
others cannot track the branch. On the other side, the advantage is that
others can easily see which patches I have applied on top of master. If
Carl updates master, I just rerun the script and the updated integration
branch is ready (unless there is a merge conflict).

Cheers,
Michal

Re: JSON based emacs UI

[Sebastian Spaeth]

On Sun, 28 Mar 2010 07:46:40 +0200, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> I don't think this can be solved only in Makefile. From my look at dme's
> repo, he adds a new subcomand 'part', which is used by the UI. So if you
> want to use the new UI and your other features, you need to merge the
> things together.

I agree that notmuch and notmuch.el need to be developed deployed in
close cooperation. However, this bundling makes things a bit more
complex to untangle. I am willing to e.g. add the -part improvement to
my own branch of notmuch, but I want to follow dme's frontend closely.

> To build my version of notmuch, I use an ugly script
> (http://rtime.felk.cvut.cz/gitweb/notmuch.git/blob/refs/heads/debian-wsh:/wsh-buildpackage)
> which first does a big octopus merge to combine several features to one
> branch and then I build notmuch from there. The current state of my
> integration can be seen at
> http://rtime.felk.cvut.cz/gitweb/notmuch.git/shortlog/refs/heads/integration/features.

Interesting, but a bit more complicated that I was originally thinking off.

> This approach has a disadvantage that integration/features branch is
> often rewritten (whenever I add, remove or change a patch) so that
> others cannot track the branch. On the other side, the advantage is that
> others can easily see which patches I have applied on top of master. If
> Carl updates master, I just rerun the script and the updated integration
> branch is ready (unless there is a merge conflict).

Very nice.

Sebastian

Re: JSON based emacs UI

[Michal Sojka]

On Mon, 29 Mar 2010, Sebastian Spaeth wrote:
> On Sun, 28 Mar 2010 07:46:40 +0200, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > I don't think this can be solved only in Makefile. From my look at dme's
> > repo, he adds a new subcomand 'part', which is used by the UI. So if you
> > want to use the new UI and your other features, you need to merge the
> > things together.
> 
> I agree that notmuch and notmuch.el need to be developed deployed in
> close cooperation. However, this bundling makes things a bit more
> complex to untangle. I am willing to e.g. add the -part improvement to
> my own branch of notmuch, but I want to follow dme's frontend closely.
> 
> > To build my version of notmuch, I use an ugly script
> > (http://rtime.felk.cvut.cz/gitweb/notmuch.git/blob/refs/heads/debian-wsh:/wsh-buildpackage)
> > which first does a big octopus merge to combine several features to one
> > branch and then I build notmuch from there. The current state of my
> > integration can be seen at
> > http://rtime.felk.cvut.cz/gitweb/notmuch.git/shortlog/refs/heads/integration/features.
> 
> Interesting, but a bit more complicated that I was originally thinking off.
> 
> > This approach has a disadvantage that integration/features branch is
> > often rewritten (whenever I add, remove or change a patch) so that
> > others cannot track the branch. On the other side, the advantage is that
> > others can easily see which patches I have applied on top of master. If
> > Carl updates master, I just rerun the script and the updated integration
> > branch is ready (unless there is a merge conflict).
> 
> Very nice.

Here is a simplified version of my script. It does only the merge. If
you want to use it, I recommend adding this script to a separate branch
so that you can track which branched were added or removed to your
features branch (you will modify the $branches variable in the script).

#!/bin/sh

branches="
decode-headers-in-reply
dme/citation-suffixes
no-self-bcc-on-reply
"

set -e

die()
{
	echo >&2 "fatal: $*"
	exit 1
}

if [ -z "$EXEC_OK" ]; then
    # If this script is tracked by repository, checking out another
    # branch could delete it so we copy it to tmp and run it from
    # there.
    script=$(mktemp -t notmuch-integrate.XXXXXX)
    cp "$0" "$script"
    EXEC_OK=1
    export EXEC_OK
    exec sh "$script" "$@"
fi

[ $EXEC_OK != 1 ] && die "Error: something went wrong"

trap "rm '$0'" EXIT

git branch -D integration/features || :
git checkout -b integration/features master
git merge $branches

Re: JSON based emacs UI

[David Edmondson]

On Sun, 28 Mar 2010 03:41:32 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> It works really nice and I get the (unmeasured) feeling that it is
> faster too (but then it might just be the lack of my additional patches
> which slow things down :-)).

I'm not sure why it would be faster. If you have a long thread that
includes many text/html parts it is clearly slower.

> I have one question (which is more build related): now, I have to pull
> your branch and an make install-emacs will also always compile and
> install the notmuch binary from your branch, but I might want to keep my
> own notmuch binary. Is it possible to just use the notmuch frontend from
> your branch, but not having to install the binary? Is there such a make
> target, or any other way? Should we create a repo that just contains the
> frontend and not notmuch itself, so people can mix and match more
> easily? (not sure what the right answer is here).

As Michal says, you need the 'part' subcommand support. If you have a
patch to the Makefile to help achieve what you want I'd be happy to look
at it.

dme.
-- 
David Edmondson, http://dme.org