Message-ID not found

Message-ID not found

[Mukund Sivaraman]

I'm testing public-inbox with a "test-list" to try to provide a HTTP
interface to mailing list archives. I'm using stock
public-inbox-1.7.0-2.fc35.noarch on Fedora 35.

List emails are injected by public-inbox-mda. public-inbox-httpd
displays an error at this URL:

https://inbox.banu.com/test-list/Yo1VM9+M%252FsYJIZkN@d1/T/

"Message-ID <Yo1VM9+M/sYJIZkN@d1/T> not found"

The ASCII encoded Message-ID is "Yo1VM9+M/sYJIZkN@d1" generated by the
Mutt client.

The corresponding message exists:
https://inbox.banu.com/test-list/Yo1VM9+M%2FsYJIZkN@d1/

Its parent (In-Reply-To) exists:
https://inbox.banu.com/test-list/Yo1UiImlt7Sh+I68@d1/

and the parent's thread can be browsed:
https://inbox.banu.com/test-list/Yo1UiImlt7Sh+I68@d1/T/

Could someone please check what the bug could be?

Does public-inbox-httpd have a problem with the '/' character in
Message-IDs?

The inbox was init'ed using -V 2. I have tried running:

  public-inbox-index --reindex --all --rethread

but it hasn't fixed it.

		Mukund

Re: Message-ID not found

[Kyle Meyer]

Mukund Sivaraman writes:

> I'm testing public-inbox with a "test-list" to try to provide a HTTP
> interface to mailing list archives. I'm using stock
> public-inbox-1.7.0-2.fc35.noarch on Fedora 35.
>
> List emails are injected by public-inbox-mda. public-inbox-httpd
> displays an error at this URL:
>
> https://inbox.banu.com/test-list/Yo1VM9+M%252FsYJIZkN@d1/T/

What's the source of this URL?  The slash looks percent-encoded one too
many times:

                  message ID: Yo1VM9+M/sYJIZkN@d1
                     |   /  -> %2F
                     V
        percent-encoded once: Yo1VM9+M%2FsYJIZkN@d1
                     |   %  -> %25
                     V
       percent-encoded twice: Yo1VM9+M%252FsYJIZkN@d1


> "Message-ID <Yo1VM9+M/sYJIZkN@d1/T> not found"

Hmm, I'm surprised by the trailing "/T".  When I visit the above URL, I
see

  Message-ID <Yo1VM9+M%2FsYJIZkN@d1>
  not found

which is what I would expect.

> The ASCII encoded Message-ID is "Yo1VM9+M/sYJIZkN@d1" generated by the
> Mutt client.
>
> The corresponding message exists:
> https://inbox.banu.com/test-list/Yo1VM9+M%2FsYJIZkN@d1/

Yes, the looks to match the URL you'd get by percent-encoding the "/"
once (see above).

> Could someone please check what the bug could be?
>
> Does public-inbox-httpd have a problem with the '/' character in
> Message-IDs?

Here's what https://inbox.banu.com/test-list/_/text/help/ says about
slashes:

  Forward slash ('/') characters in the Message-IDs
  need to be escaped as "%2F" (without quotes).

And here's the escaping that public-inbox does on the message ID
(<https://public-inbox.org/public-inbox.git/tree/lib/PublicInbox/MID.pm?h=v1.8.0#n130>):

  # RFC3986, section 3.3:
  sub MID_ESC () { '^A-Za-z0-9\-\._~!\$\&\'\(\)\*\+,;=:@' }
  sub mid_escape ($) { uri_escape_utf8($_[0], MID_ESC) }

Re: Message-ID not found

[Mukund Sivaraman]

Hi Kyle

On Tue, May 24, 2022 at 07:39:32PM -0400, Kyle Meyer wrote:
> Mukund Sivaraman writes:
> 
> > I'm testing public-inbox with a "test-list" to try to provide a HTTP
> > interface to mailing list archives. I'm using stock
> > public-inbox-1.7.0-2.fc35.noarch on Fedora 35.
> >
> > List emails are injected by public-inbox-mda. public-inbox-httpd
> > displays an error at this URL:
> >
> > https://inbox.banu.com/test-list/Yo1VM9+M%252FsYJIZkN@d1/T/
> 
> What's the source of this URL?

In the thread index at: https://inbox.banu.com/test-list/

The 2nd thread with subject "About Pluto" links to:

https://inbox.banu.com/test-list/Yo1VM9+M%2FsYJIZkN@d1/T/#t

The webpage at that URL displays the
"Message-ID <Yo1VM9+M/sYJIZkN@d1/T> not found" error.

In this case, there is just one escape "%2F" in the URL for '/', and
from the error message it has correctly unescaped the Message-ID except
for the trailing "/T".

Could there be a bug in how the '/' is treated as a separator in
public-inbox-httpd?

		Mukund

Re: Message-ID not found

[Kyle Meyer]

Mukund Sivaraman writes:

> On Tue, May 24, 2022 at 07:39:32PM -0400, Kyle Meyer wrote:
[...]
>> > https://inbox.banu.com/test-list/Yo1VM9+M%252FsYJIZkN@d1/T/
>> 
>> What's the source of this URL?
>
> In the thread index at: https://inbox.banu.com/test-list/
>
> The 2nd thread with subject "About Pluto" links to:
>
> https://inbox.banu.com/test-list/Yo1VM9+M%2FsYJIZkN@d1/T/#t

Okay, so that link looks good.  And then when I follow the link...

> The webpage at that URL displays the
> "Message-ID <Yo1VM9+M/sYJIZkN@d1/T> not found" error.

... I see the "Message-ID <Yo1VM9+M/sYJIZkN@d1/T> not found" error you
describe.

I pulled in your inbox and ran public-inbox-httpd (v1.7.0 and v1.8.0)
locally:

  /test-list/Yo1VM9+M%2FsYJIZkN@d1/T/#t

And then when I follow that link.I see the message for
<Yo1VM9+M/sYJIZkN@d1>, as expected.  Hmm.

> [...]
> Could there be a bug in how the '/' is treated as a separator in
> public-inbox-httpd?

Sure, though my guess is that there is something more setup-specific
going on because 1) I don't see it when I try the inbox locally and 2) I
think slashes are common enough in the wild yet I don't recall seeing
similar reports for public-inbox.org inboxes, lore.kernel.org inboxes,
or the inboxes that I host.

As an example, at https://lore.kernel.org/git/?t=20220525002051 the link
for "[PATCH v5 02/17] pack-mtimes: support reading .mtimes files" is to

  https://lore.kernel.org/git/Yo1aaLDmPKJ5%2Frh5@nand.local/T/#u

I was able to successfully follow that to the message.

Could there be something in your setup that's doing the second round of
percent-encoding?

Re: Message-ID not found

[Mukund Sivaraman]

On Tue, May 24, 2022 at 09:44:33PM -0400, Kyle Meyer wrote:
> > [...]
> > Could there be a bug in how the '/' is treated as a separator in
> > public-inbox-httpd?
> 
> Sure, though my guess is that there is something more setup-specific
> going on because 1) I don't see it when I try the inbox locally and 2) I
> think slashes are common enough in the wild yet I don't recall seeing
> similar reports for public-inbox.org inboxes, lore.kernel.org inboxes,
> or the inboxes that I host.
> 
> As an example, at https://lore.kernel.org/git/?t=20220525002051 the link
> for "[PATCH v5 02/17] pack-mtimes: support reading .mtimes files" is to
> 
>   https://lore.kernel.org/git/Yo1aaLDmPKJ5%2Frh5@nand.local/T/#u
> 
> I was able to successfully follow that to the message.
> 
> Could there be something in your setup that's doing the second round of
> percent-encoding?

nginx is a reverse-proxy in front of public-inbox-httpd. I'll check if
it breaks anything.

		Mukund

Re: Message-ID not found

[Mukund Sivaraman]

Hi Kyle

On Wed, May 25, 2022 at 07:49:07AM +0530, Mukund Sivaraman wrote:
> > Could there be something in your setup that's doing the second round of
> > percent-encoding?
> 
> nginx is a reverse-proxy in front of public-inbox-httpd. I'll check if
> it breaks anything.

It was a problem with the nginx reverse-proxy config. I updated it, and
it works now. Thank you for checking, and esp. pointing out that it
worked directly with public-inbox-httpd.

For anyone else who searches and finds this thread, a working nginx
configuration with proxy_pass follows, where proxy_pass must not include
a trailing '/':

# create an upstream
upstream inbox_proxy_backend {
    server unix:/run/inbox/inbox.sock fail_timeout=0;
    keepalive 32;
}

server {
    # other server config
    # ...

    location / {
        proxy_pass http://inbox_proxy_backend;
        include proxy_params;
    }
}

		Mukund

Re: Message-ID not found

[Eric Wong]

Mukund Sivaraman <muks@mukund.org> wrote:
> Hi Kyle
> 
> On Wed, May 25, 2022 at 07:49:07AM +0530, Mukund Sivaraman wrote:
> > > Could there be something in your setup that's doing the second round of
> > > percent-encoding?
> > 
> > nginx is a reverse-proxy in front of public-inbox-httpd. I'll check if
> > it breaks anything.
> 
> It was a problem with the nginx reverse-proxy config. I updated it, and
> it works now. Thank you for checking, and esp. pointing out that it
> worked directly with public-inbox-httpd.

Thanks both.  Fwiw, the long-term plan is to allow public-inbox-httpd
to replace nginx entirely to avoid configuration pitfalls like this.