a temporary nmweb view of notmuch mailing list archive

a temporary nmweb view of notmuch mailing list archive

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

Hi all--

I did a bit of experimenting with Brian Sniffen's proposed notmuch-web
branch, and it's now (temporarily) publishing a view of the notmuch
mailing list here:

    https://nmbug.notmuchmail.org/btsmail/

I DO NOT EXPECT THIS URL TO BE STABLE -- please do not use links to it
in permanent places, this is just for experimentation at the moment.

I'll send a series of patches to the list shortly that include some
cleanup to nmweb, but you can also find them on the nmweb branch at:

  https://gitlab.com/dkg/notmuch


I welcome feedback on the public installation!


The outstanding things that i think need to change in the nmbug branch
before i'd be comfortable adopting this are:

 * rely on non-bundled versions of jquery -- we don't want to have to
   maintain them in the notmuch git repo.

 * ensure that the paths are configurable -- the code currently
   hardcodes "/btsmail", which is just a convenience, and also enforces
   that "static/cache" lives inside the directory -- this should be
   separable from the code.

 * the user needs need some easy way to move from message to message (next by
   thread, next by date?)


nice-to-haves:

 * be able to go from message view to thread view

 * the sender name is hyperlinked, but it's a mailto: link.  I think it
   should instead link to a view that shows messages from that person,
   or something like that -- most folks don't have their mailto: set up
   properly.



and we also need to fix a few things on nmbug:

 * delivering messages with "notmuch insert" (or invoking "notmuch new"
   after message delivery)

 * we need to make sure that the mail archive remains world-readable
   (you might find that the archive disappears -- please ping me on IRC
   if it happens and i'll try to track down how that's happening)



if you try it out, let me know what you think.

Many thanks to Brian Sniffen for getting this rolling.

      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: a temporary nmweb view of notmuch mailing list archive

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

On Sun 2018-02-04 23:09:35 -0500, Daniel Kahn Gillmor wrote:

> I did a bit of experimenting with Brian Sniffen's proposed notmuch-web
> branch, and it's now (temporarily) publishing a view of the notmuch
> mailing list here:
>
>     https://nmbug.notmuchmail.org/btsmail/
>
> I DO NOT EXPECT THIS URL TO BE STABLE -- please do not use links to it
> in permanent places, this is just for experimentation at the moment.

this is gradually getting better.  new messages to the list are
automatically archived and visible here, and the archive doesn't
spontaneously break in ways i can't predict.

it does still break every time an authorized user does "nmbug push"
though, and i don't currently understand why.  I'm putting in place a
workaround for it but if anyone has suggestions for debugging it
server-side, i'm all ears.  nmbug is backed server-side by gitolite, and
i don't know gitolite well enough to know how it sets umask or
permissions by default.

           --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: a temporary nmweb view of notmuch mailing list archive

[Brian Sniffen]

It looks like you have thread next/pref turned off. Is there a reason, including the reason that I hadn’t documented it?

-- 
Brian Sniffen

> On Feb 8, 2018, at 11:28 PM, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> 
>> On Sun 2018-02-04 23:09:35 -0500, Daniel Kahn Gillmor wrote:
>> 
>> I did a bit of experimenting with Brian Sniffen's proposed notmuch-web
>> branch, and it's now (temporarily) publishing a view of the notmuch
>> mailing list here:
>> 
>>    https://nmbug.notmuchmail.org/btsmail/
>> 
>> I DO NOT EXPECT THIS URL TO BE STABLE -- please do not use links to it
>> in permanent places, this is just for experimentation at the moment.
> 
> this is gradually getting better.  new messages to the list are
> automatically archived and visible here, and the archive doesn't
> spontaneously break in ways i can't predict.
> 
> it does still break every time an authorized user does "nmbug push"
> though, and i don't currently understand why.  I'm putting in place a
> workaround for it but if anyone has suggestions for debugging it
> server-side, i'm all ears.  nmbug is backed server-side by gitolite, and
> i don't know gitolite well enough to know how it sets umask or
> permissions by default.
> 
>           --dkg
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

Re: a temporary nmweb view of notmuch mailing list archive

[Daniel Kahn Gillmor]

On Sat 2018-02-10 13:57:34 -0500, Brian Sniffen wrote:
> It looks like you have thread next/pref turned off. Is there a reason,
> including the reason that I hadn’t documented it?

nope, i just didn't fiddle with it much beyond setting it up and trying
to make sure the system integration didn't seem wildly dangerous.  happy
to tweak it further, and will take any recommendations. :)

thanks for working on this, Brian!

          --dkg

Re: a temporary nmweb view of notmuch mailing list archive

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]

On Sun, Feb 04 2018, Daniel Kahn Gillmor wrote:
> I did a bit of experimenting with Brian Sniffen's proposed notmuch-web
> branch, and it's now (temporarily) publishing a view of the notmuch
> mailing list here:
>
>     https://nmbug.notmuchmail.org/btsmail/

Thanks, Brian and Daniel!

When I first started on notmuch, I was hoping that one immediate benefit
would be that free-software projects could finally have working web
archives of their mailing lists, (without thread views breaking at
arbitrary boundaries such as months or years).

I guess the "immediate" aspect of my hope didn't play out so much, but I
am very excited to see this now. So, thanks again!

At first glance I saw "terms ... after ... before" and thought that the
after/before fields were for some sort of contextual context, but I
finally figured out what those were actually for. So maybe it would help
to add some text along the lines of "Limit to date range" or so.

And now that we have real date searching, it might make sense to use
that in the searching instead of the time stamp.

Another thing that might be nice for selecting dates ranges is the
ability to select something less precise than a particular day.

Searching for all the messages in a month with something like:

	notmuch search date:2018-01

or even a year with:

	notmuch search date:2018

is pretty darn convenient in notmuch, or even to search across two
months with:

	notmuch search date:2018-01..2018-02

It would be nice to not lose that kind of convenience in a web
interface.

Anyway, thanks again. This is looking great, and I will look forward to
entirely removing the pipermail-based archives from the website as soon
as we can.

-Carl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: a temporary nmweb view of notmuch mailing list archive

[J. Lewis Muir]

On 02/12, Carl Worth wrote:
> At first glance I saw "terms ... after ... before" and thought that the
> after/before fields were for some sort of contextual context, but I
> finally figured out what those were actually for. So maybe it would help
> to add some text along the lines of "Limit to date range" or so.

Or maybe drop the "Terms", "After", and "Before" text fields, and
just have a single text field labeled "Query".  The date range can be
expressed in the query.  Provide a help link for some basics on the
query syntax (e.g., like [1] and [2]), including how to search by date.

Lewis

[1]: https://support.google.com/websearch/answer/134479
[2]: https://support.google.com/websearch/answer/2466433

Re: a temporary nmweb view of notmuch mailing list archive

[Daniel Kahn Gillmor]

On Sun 2018-02-11 15:03:05 -0500, Daniel Kahn Gillmor wrote:
> On Sat 2018-02-10 13:57:34 -0500, Brian Sniffen wrote:
>> It looks like you have thread next/pref turned off. Is there a reason,
>> including the reason that I hadn’t documented it?
>
> nope, i just didn't fiddle with it much beyond setting it up and trying
> to make sure the system integration didn't seem wildly dangerous.  happy
> to tweak it further, and will take any recommendations. :)

hm, looking into this a little bit, i don't actually see how to turn it
on.  if you've got pointers or documentation, i would be happy to read
it.

        --dkg

Re: a temporary nmweb view of notmuch mailing list archive

[Brian Sniffen]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

show_thread_nav = True , and take through https://github.com/briansniffen/notmuch/commit/021c914fc5cc1029778794cc5630373041066889

-- 
Brian Sniffen

> On Feb 15, 2018, at 11:06 AM, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> 
>> On Sun 2018-02-11 15:03:05 -0500, Daniel Kahn Gillmor wrote:
>>> On Sat 2018-02-10 13:57:34 -0500, Brian Sniffen wrote:
>>> It looks like you have thread next/pref turned off. Is there a reason,
>>> including the reason that I hadn’t documented it?
>> 
>> nope, i just didn't fiddle with it much beyond setting it up and trying
>> to make sure the system integration didn't seem wildly dangerous.  happy
>> to tweak it further, and will take any recommendations. :)
> 
> hm, looking into this a little bit, i don't actually see how to turn it
> on.  if you've got pointers or documentation, i would be happy to read
> it.
> 
>        --dkg

[-- Attachment #2: Type: text/html, Size: 2022 bytes --]

Re: a temporary nmweb view of notmuch mailing list archive

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 4437 bytes --]

On Thu 2018-02-15 15:47:21 -0500, Brian Sniffen wrote:
> show_thread_nav = True , and take through https://github.com/briansniffen/notmuch/commit/021c914fc5cc1029778794cc5630373041066889

ah, i'd missed some of your commits, apparently.  thanks for the
pointer.  https://notmuchmail.org/btsmail is now running my merged changes.


We're moving in the right direction!  Since i pulled these in and made a
few more changes, we've now got the following:

 * we have a threaded view from each message

 * nmweb is configurable via the environment, no need to tweak config in
   the code itself


I've also just opened a pull request on github for you to merge my
cleanup, if you're interested:

    https://github.com/briansniffen/notmuch/pull/1

We're going to need to do some git history cleanup eventually if we want
this merged on the notmuch mainline, but i figure it'd be better if we
sync up earlier rather than later :)


for the record, there are a few errors i'm seeing now from poking around
on nmweb.  in particular, i get an error here in the tree walk when
looking at what appears to be a bounced e-mail (the fact that bounced
e-mails probably shouldn't show up in this archive is a separate bug):

    https://nmbug.notmuchmail.org/btsmail/show/E1ehr4h-0000Dc-0w%40nmbug

Feb 15 18:27:13 nmbug gunicorn3[30460]: Traceback (most recent call last):
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/usr/lib/python3/dist-packages/web/application.py", line 257, in process
Feb 15 18:27:13 nmbug gunicorn3[30460]:     return self.handle()
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/usr/lib/python3/dist-packages/web/application.py", line 248, in handle
Feb 15 18:27:13 nmbug gunicorn3[30460]:     return self._delegate(fn, self.fvars, args)
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/usr/lib/python3/dist-packages/web/application.py", line 488, in _delegate
Feb 15 18:27:13 nmbug gunicorn3[30460]:     return handle_class(cls)
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/usr/lib/python3/dist-packages/web/application.py", line 466, in handle_class
Feb 15 18:27:13 nmbug gunicorn3[30460]:     return tocall(*args)
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/home/nmweb/notmuch/contrib/notmuch-web/nmweb.py", line 172, in GET
Feb 15 18:27:13 nmbug gunicorn3[30460]:     sprefix=webprefix)
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 989, in render
Feb 15 18:27:13 nmbug gunicorn3[30460]:     return self.environment.handle_exception(exc_info, True)
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 754, in handle_exception
Feb 15 18:27:13 nmbug gunicorn3[30460]:     reraise(exc_type, exc_value, tb)
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
Feb 15 18:27:13 nmbug gunicorn3[30460]:     raise value.with_traceback(tb)
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "templates/show.html", line 1, in top-level template code
Feb 15 18:27:13 nmbug gunicorn3[30460]:     {% extends "base.html" %}
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "templates/base.html", line 32, in top-level template code
Feb 15 18:27:13 nmbug gunicorn3[30460]:     {% block content %}
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "templates/show.html", line 12, in block "content"
Feb 15 18:27:13 nmbug gunicorn3[30460]:     {% for part in format_message(m.get_filename(),mid): %}{{ part|safe }}{% endfor %}
Feb 15 18:27:13 nmbug gunicorn3[30460]:   File "/home/nmweb/notmuch/contrib/notmuch-web/nmweb.py", line 241, in format_message_walk
Feb 15 18:27:13 nmbug gunicorn3[30460]:     parts.pop()
Feb 15 18:27:13 nmbug gunicorn3[30460]: IndexError: pop from empty list



Also, i noticed that just searching by date (with no terms) fails
with a Xapian error because " date:…" gets quote_plus'ed to "+date:…"
and notmuch gets confused by the leading "+" -- maybe we're missing an
unquote_plus or something?



Finally, we're clearly not escaping the subject lines correctly.  Look
at the threading here:

    https://nmbug.notmuchmail.org/btsmail/show/cover.1392309570.git.wking%40tremily.us


Does anyone have a preference for the URL to use for more permanent
web-based view of the mailing list archive?

thanks for this work, Brian!

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]