[BUG] Custom headers in `notmuch-message-headers` are broken

[BUG] Custom headers in `notmuch-message-headers` are broken

[Jan Malakhovski]

# What I did

I added "X-Github-Sender" to `notmuch-message-headers`.
Looked at a message sent via github with `notmuch-show`.

# What I expected

To see "X-Github-Sender" header displayed in `notmuch-show`.

# What I got

No such header was displayed.

# Why 

`(notmuch-show "query")` runs

```
notmuch show --format=sexp --format-version=4 query
```

internally. The latter produces a sexp with

```
:headers (:Subject "" :From "" :To "" :Reply-To "" :Date "")
```

even when the message has many more headers.

The end result is that `notmuch-message-headers` variable has no effect.

Re: [BUG] Custom headers in `notmuch-message-headers` are broken

[David Bremner]

Jan Malakhovski <oxij@oxij.org> writes:

>
> internally. The latter produces a sexp with
>
> ```
> :headers (:Subject "" :From "" :To "" :Reply-To "" :Date "")
> ```
>
> even when the message has many more headers.

Yes, you are correct that currently format_headers_sprinter in
notmuch-show.c only outputs a fixed set of headers.  Unlike indexing new
headers, I don't think there's any hidden complexity here, if someone is
looking for a project.

d

Re: [BUG] Custom headers in `notmuch-message-headers` are broken

[Amin Bandali]

I got bitten by this today.

I only had a brief look at the format_headers_sprinter function
in notmuch-show.c.  Would you, David, or anyone else be able to
point out if the following makes sense, for generalizing
format_headers_sprinter to handle any arbitrary headers?

I saw this bit near the bottom of that function:

--8<---------------cut here---------------start------------->8---
if (reply) {
	sp->map_key (sp, "In-reply-to");
	sp->string (sp, g_mime_object_get_header (GMIME_OBJECT (message), "In-reply-to"));

	sp->map_key (sp, "References");
	sp->string (sp, g_mime_object_get_header (GMIME_OBJECT (message), "References"));
}
--8<---------------cut here---------------end--------------->8---

So I had a look at GMimeObject's docs on GNOME.org [0], and saw
g_mime_object_get_header_list, which returns a GMimeHeaderList*
list of headers [1], which seems to be what we're looking for.

From there, we'd walk over 0..(g_mime_header_list_get_count-1)
indices and use g_mime_header_list_get_header_at to get each
header, and pass it to g_mime_header_get_name to get the name
which we'll pass to sp->map_key and also use to get its value
from g_mime_object_get_header.

Does that make sense?

[0]: https://developer.gnome.org/gmime/stable/GMimeObject.html
[1]: https://developer.gnome.org/gmime/stable/GMimeHeaderList.html

-amin

Re: [BUG] Custom headers in `notmuch-message-headers` are broken

[David Bremner]

Amin Bandali <aminb@gnu.org> writes:

> I got bitten by this today.
>
> I only had a brief look at the format_headers_sprinter function
> in notmuch-show.c.  Would you, David, or anyone else be able to
> point out if the following makes sense, for generalizing
> format_headers_sprinter to handle any arbitrary headers?
>
> I saw this bit near the bottom of that function:
>
> --8<---------------cut here---------------start------------->8---
> if (reply) {
> 	sp->map_key (sp, "In-reply-to");
> 	sp->string (sp, g_mime_object_get_header (GMIME_OBJECT (message), "In-reply-to"));
>
> 	sp->map_key (sp, "References");
> 	sp->string (sp, g_mime_object_get_header (GMIME_OBJECT (message), "References"));
> }
> --8<---------------cut here---------------end--------------->8---
>
> So I had a look at GMimeObject's docs on GNOME.org [0], and saw
> g_mime_object_get_header_list, which returns a GMimeHeaderList*
> list of headers [1], which seems to be what we're looking for.
>
> From there, we'd walk over 0..(g_mime_header_list_get_count-1)
> indices and use g_mime_header_list_get_header_at to get each
> header, and pass it to g_mime_header_get_name to get the name
> which we'll pass to sp->map_key and also use to get its value
> from g_mime_object_get_header.

Sounds roughly correct. Note that

1) You'll want to avoid duplicating headers already emitted

2) This will most likely require updating the test suite to add new
headers to the expected output. If that is too burdensome we could
consider bumping the version in devel/schemata, but I'd rather not.

3) We should think carefully about whether we want to blindly send
   certain large headers like "Received". Some people use notmuch via
   ssh or equivalent, and it might (dunno) be a concern.

4) Your plan _might_ only be working with gmime 3.0+.  That's ok, it
just means I would need to finally apply the patches I've got kicking
around removing gmime-2.6 support.

Re: [BUG] Custom headers in `notmuch-message-headers` are broken

[Jan Malakhovski]

> 3) We should think carefully about whether we want to blindly send
>    certain large headers like "Received". Some people use notmuch via
>    ssh or equivalent, and it might (dunno) be a concern.

I'd prefer `notmuch show` to dump everything by default and have an
option like `--headers` to limit those. I.e. to get current behavior
you'd just dump comma-separated `notmuch-message-headers` into that
option in `notmuch.el` and be happy.

> 1) You'll want to avoid duplicating headers already emitted

Why? Wouldn't that prevent you from parsing "Received" headers in
`notmuch.el`?

Re: [BUG] Custom headers in `notmuch-message-headers` are broken

[David Bremner]

Jan Malakhovski <oxij@oxij.org> writes:

>> 3) We should think carefully about whether we want to blindly send
>>    certain large headers like "Received". Some people use notmuch via
>>    ssh or equivalent, and it might (dunno) be a concern.
>
> I'd prefer `notmuch show` to dump everything by default and have an
> option like `--headers` to limit those. I.e. to get current behavior
> you'd just dump comma-separated `notmuch-message-headers` into that
> option in `notmuch.el` and be happy.

that would be an option, it would require updating the elisp.

>
>> 1) You'll want to avoid duplicating headers already emitted
>
> Why? Wouldn't that prevent you from parsing "Received" headers in
> `notmuch.el`?

I just meant that the code has already emitted e.g. "Cc" by the time it
reaches the block Amin points to.

Re: [BUG] Custom headers in `notmuch-message-headers` are broken

[Leo Gaspard]

(sorry for sending twice, forgot to Cc the list at first)

I just have been hit by this exact same issue, also for X-GitHub-Sender,
during my switch to notmuch and notmuch-mode.

>> 3) We should think carefully about whether we want to blindly send
>>    certain large headers like "Received". Some people use notmuch via
>>    ssh or equivalent, and it might (dunno) be a concern.
>
> I'd prefer `notmuch show` to dump everything by default and have an
> option like `--headers` to limit those. I.e. to get current behavior
> you'd just dump comma-separated `notmuch-message-headers` into that
> option in `notmuch.el` and be happy.

Potentially keep the `--headers` option as you propose and default to
the current behaviour? This way everything is retro-compatible, messages
still look nice when manually shown via `notmuch show` by default, but
tools that make use of the additional headers can specify which headers
they use.

And then dumping comma-separated `notmuch-message-headers` into this
option becomes the additional feature for `notmuch.el` we're all hoping
for :)

Re: [BUG] Custom headers in `notmuch-message-headers` are broken

[David Bremner]

Jan Malakhovski <oxij@oxij.org> writes:

> # What I did
>
> I added "X-Github-Sender" to `notmuch-message-headers`.
> Looked at a message sent via github with `notmuch-show`.
>
> # What I expected
>
> To see "X-Github-Sender" header displayed in `notmuch-show`.
>
> # What I got
>
> No such header was displayed.
>
> # Why 
>
> `(notmuch-show "query")` runs
>
> ```
> notmuch show --format=sexp --format-version=4 query
> ```
>
> internally. The latter produces a sexp with
>
> ```
> :headers (:Subject "" :From "" :To "" :Reply-To "" :Date "")
> ```
>
> even when the message has many more headers.
>
> The end result is that `notmuch-message-headers` variable has no effect.

Hi Jan;

Not sure if you are still interested, but this should be fixed in
notmuch 0.35 (see show.extra_headers in notmuch-config(1)).

Re: [BUG] Custom headers in `notmuch-message-headers` are broken

[Herbert Valerio Riedel]

>> The end result is that `notmuch-message-headers` variable has no effect.

> Not sure if you are still interested, but this should be fixed in
> notmuch 0.35 (see show.extra_headers in notmuch-config(1)).

I just stumbled over this issue when trying to add "Message-Id" to the
list of shown messages as the docstring of the custom variable
definition in 'notmuch-show.el' didn't mention that I needed to tweak my
config or that only a small subset of headers might be supported:

--8<---------------cut here---------------start------------->8---
(defcustom notmuch-message-headers '("Subject" "To" "Cc" "Date")
  "Headers that should be shown in a message, in this order.

For an open message, all of these headers will be made visible
according to `notmuch-message-headers-visible' or can be toggled
with `notmuch-show-toggle-visibility-headers'. For a closed message,
only the first header in the list will be visible."
  :type '(repeat string)
  :group 'notmuch-show)
--8<---------------cut here---------------end--------------->8---

I suggest adding a hint there that tweaking 'show.extra_headers' might
be necessary.