Newsgroups mailing-list gateway broken thread

Newsgroups mailing-list gateway broken thread

[Garreau, Alexandre]

Hi,

After some discussion I’ve realized that in fact mailing traversing the
newsgroups mailing-list gateway receive different message-ids in their
references, which, in my mail user-agent (gnus) break the threads.

I say that because it appears not to break anything on the mailing-list
archive web interface (and to affect most if not all of newsgroup
posters), so I’d like how does it do (does it have mappings of all
correct message-ids to other message-ids?), and how other users of the
mailing-list do, or what can be done to improve the situation.

Thanks for any information.

Re: Newsgroups mailing-list gateway broken thread

[Emanuel Berg]

Garreau, Alexandre wrote:

> After some discussion I’ve realized that in
> fact mailing traversing the newsgroups
> mailing-list gateway receive different
> message-ids in their references, which, in my
> mail user-agent (gnus) break the threads.
>
> I say that because it appears not to break
> anything on the mailing-list archive web
> interface (and to affect most if not all of
> newsgroup posters), so I’d like how does it
> do (does it have mappings of all correct
> message-ids to other message-ids?), and how
> other users of the mailing-list do, or what
> can be done to improve the situation.
>
> Thanks for any information.

(Important note: This post is *not* sent thru
gnu.emacs.help, but from gmane.emacs.help -
like I say below, it is 20 times slower. But:
how do the references look now?)

The most likely explanation is that it is
a glitch in the interface between the newsgroup
and the mailing list.

I always used gnu.emacs.help, only for a short
while I used gmane.emacs.help, but for some
reason it is much slower (to get the summary of
the new posts), so I switched back to
gnu.emacs.help.

I have no idea as to what piece of technology
it is that does the propagation, where it is
located, who runs is, and so on.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Newsgroups mailing-list gateway broken thread

[Nuno Silva]

On 2018-10-17, Garreau, Alexandre wrote:

> After some discussion I’ve realized that in fact mailing traversing the
> newsgroups mailing-list gateway receive different message-ids in their
> references, which, in my mail user-agent (gnus) break the threads.
>
> I say that because it appears not to break anything on the mailing-list
> archive web interface (and to affect most if not all of newsgroup
> posters), so I’d like how does it do (does it have mappings of all
> correct message-ids to other message-ids?), and how other users of the
> mailing-list do, or what can be done to improve the situation.

I was looking into this recently, and from what I have seen, threading
is still broken in the web archive, just not as broken as I see it in
Gnus.

There is indeed a mismatch in the references. Gnus here handles this by
starting a new thread. The web archive still keeps the reply in the same
thread, but the "missing" message-id shows up in the tree there as
"Message not available".

Here (through USENET) the message-id for your post is:
Message-ID: <mailman.2336.1539814909.1284.help-gnu-emacs@gnu.org>

-- 
Nuno Silva

Re: Newsgroups mailing-list gateway broken thread

[Garreau, Alexandre]

On 2018-10-18 at 12:05, Emanuel Berg wrote:
> Garreau, Alexandre wrote:
>
>> After some discussion I’ve realized that in
>> fact mailing traversing the newsgroups
>> mailing-list gateway receive different
>> message-ids in their references, which, in my
>> mail user-agent (gnus) break the threads.
>>
>> I say that because it appears not to break
>> anything on the mailing-list archive web
>> interface (and to affect most if not all of
>> newsgroup posters), so I’d like how does it
>> do (does it have mappings of all correct
>> message-ids to other message-ids?), and how
>> other users of the mailing-list do, or what
>> can be done to improve the situation.
>>
>> Thanks for any information.
>
> (Important note: This post is *not* sent thru
> gnu.emacs.help, but from gmane.emacs.help -
> like I say below, it is 20 times slower. But:
> how do the references look now?)

The references are just fine, they correctly references my message.

> The most likely explanation is that it is
> a glitch in the interface between the newsgroup
> and the mailing list.
>
> I always used gnu.emacs.help, only for a short
> while I used gmane.emacs.help, but for some
> reason it is much slower (to get the summary of
> the new posts), so I switched back to
> gnu.emacs.help.

I expect the newsgroups gateway *should* work.  So this is to fix,
rather than suggesting gmane as a workaround (the issue of it being
slower is a separate issue).

> I have no idea as to what piece of technology
> it is that does the propagation, where it is
> located, who runs is, and so on.

I believe this is be on the GNU side, because I saw references referring
to same garbled message-ids, with “mailman” in them: so if different
newsgroup users, such as Emanual Berg and Barry Margolin gets the same
references while using different gateways to usenet, and this one has
“mailman” in it, I think it is not only common to all newsgroups users
(thus coming from GNU), but also coming from either mailman, either
something interfacing with it, I guess.

The question is why.

Re: Newsgroups mailing-list gateway broken thread

[Emanuel Berg]

Garreau, Alexandre wrote:

> I expect the newsgroups gateway *should*
> work. So this is to fix, rather than
> suggesting gmane as a workaround (the issue
> of it being slower is a separate issue).

(This post I send thru gnu.emacs.help - so now
you can compare it to the Gmane post, if that
interests you!)

Gmane isn't exactly a workaround, either. Or if
it is, it is a HUGE one! Gmane is rather
Usenet 2.0 or Usenet Reloaded. It is the best
of both worlds, the worlds being the
mailing lists and the Usenet newsgroups.

Not all Gmane groups are that much slower by
the way. I don't know why.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Newsgroups mailing-list gateway broken thread

[Bob Proulx]

Garreau, Alexandre wrote:
> Emanuel Berg wrote:
> > Garreau, Alexandre wrote:
> >> After some discussion I’ve realized that in
> >> fact mailing traversing the newsgroups
> >> mailing-list gateway receive different
> >> message-ids in their references, which, in my
> >> mail user-agent (gnus) break the threads.

Yes.  When users post to the newsgroup and it is gatewayed into the
mailing list it breaks threading.

> >> I say that because it appears not to break
> >> anything on the mailing-list archive web
> >> interface (and to affect most if not all of

It does break on the web interface too.  It's no different there than
anywhere else.  Messages from the newsgroup break threading and start
a new thread. :-(

> >> newsgroup posters), so I’d like how does it
> >> do (does it have mappings of all correct
> >> message-ids to other message-ids?), and how
> >> other users of the mailing-list do, or what
> >> can be done to improve the situation.

There isn't anything special about the web interface.  Threading is
broken there the same as elsewhere.  Therefore there is no magic to
extract from there to add to your own to fix threading.

> >> Thanks for any information.

> > (Important note: This post is *not* sent thru
> > gnu.emacs.help, but from gmane.emacs.help -
> > like I say below, it is 20 times slower. But:
> > how do the references look now?)
> 
> The references are just fine, they correctly references my message.
...
> I expect the newsgroups gateway *should* work.  So this is to fix,
> rather than suggesting gmane as a workaround (the issue of it being
> slower is a separate issue).

Emanuel, We have talked about this before. :-)

> I believe this is be on the GNU side, because I saw references referring
> to same garbled message-ids, with “mailman” in them: so if different
> newsgroup users, such as Emanual Berg and Barry Margolin gets the same
> references while using different gateways to usenet, and this one has
> “mailman” in it, I think it is not only common to all newsgroups users
> (thus coming from GNU), but also coming from either mailman, either
> something interfacing with it, I guess.
> 
> The question is why.

This is a deja vu as we have discussed this before.  This link jumps
right into some detail from that previous discussion.

 http://lists.gnu.org/archive/html/help-gnu-emacs/2016-06/msg00147.html

Of course the web email address redactor with all of the address
hidden redactions make reading that on the web quite difficult.  One
could ftp down the mailbox and then have the full mailbox of messages.

AFAICT messages through the Stanford newsgroup to email relay do not
include the In-Reply-To header and therefore break threading.  A
message may have the References header with information about previous
messages in the thread causing the message to thread before where it
should thread, and if only one deep them be completely disconnected
from the original thread.

In the intervening two years since the previous discussion nothing
substantial has happened.

Bob

Re: Newsgroups mailing-list gateway broken thread

[Garreau, Alexandre]

On 2018-10-18 at 20:37, Emanuel Berg wrote:
> Garreau, Alexandre wrote:
>
>> I expect the newsgroups gateway *should*
>> work. So this is to fix, rather than
>> suggesting gmane as a workaround (the issue
>> of it being slower is a separate issue).
>
> (This post I send thru gnu.emacs.help - so now
> you can compare it to the Gmane post, if that
> interests you!)

Yeah still exactly the same problems as before.

> Gmane isn't exactly a workaround, either. Or if
> it is, it is a HUGE one!

I’m meaning from your side.  You’d *use* gmane *as* a workaround.  Gmane
per se is not a workaround and I agree it is great.

> Gmane is rather Usenet 2.0 or Usenet Reloaded. It is the best of both
> worlds, the worlds being the mailing lists and the Usenet newsgroups.
>
> Not all Gmane groups are that much slower by the way. I don't know
> why.

To me “the best” would be a tight integration of both smtp and nntp
clients/servers and mailing-list software, written in lisp (not python),
and work in P2P, and would primarily use cryptography for
authentication, if available (think DKIM+PGP+SASL+TLS, pick the
available ones).  But gmane is the best that actually exists right now
:)

Re: Newsgroups mailing-list gateway broken thread

[Emanuel Berg]

Bob Proulx wrote:

> Messages from the newsgroup break threading
> and start a new thread. :-( [...] There isn't
> anything special about the web interface.
> Threading is broken there the same
> as elsewhere.

Well, except for the newsgroup!

> Emanuel, We have talked about this before.
> :-)

I don't think I understood it 100% that time.

OK, from now on I use gmane.emacs.help instead!
Everyone happy?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Newsgroups mailing-list gateway broken thread

[Nuno Silva]

On 2018-10-19, Emanuel Berg wrote:

> Bob Proulx wrote:
>
>> Messages from the newsgroup break threading
>> and start a new thread. :-( [...] There isn't
>> anything special about the web interface.
>> Threading is broken there the same
>> as elsewhere.
>
> Well, except for the newsgroup!
>
>> Emanuel, We have talked about this before.
>> :-)
>
> I don't think I understood it 100% that time.
>
> OK, from now on I use gmane.emacs.help instead!
> Everyone happy?

But why is the gateway rewriting Message-IDs? 

Your post, seen from USENET, has:

    Message-ID: <mailman.2440.1539944256.1284.help-gnu-emacs@gnu.org>

And on Gmane:

    Message-ID: <86zhvab5r3.fsf@zoho.com>



(This reply is being sent through USENET, but I tried to set
In-Reply-To myself. If that works, it shouldn't break threading.)

-- 
Nuno Silva

Re: Newsgroups mailing-list gateway broken thread

[Emanuel Berg]

Nuno Silva wrote:

> But why is the gateway rewriting Message-IDs?
>
> Your post, seen from USENET, has:
>
>     Message-ID: <mailman.2440.1539944256.1284.help-gnu-emacs@gnu.org>
>
> And on Gmane:
>
>     Message-ID: <86zhvab5r3.fsf@zoho.com>

I don't know. I don't know what kind of program
the "gateway" is or how it gets its data from
either the mailing list or the newsgroup.
I don't know where it is physically or who
maintains it, or if it was just left with the
switch "on" ages ago. Whoever wrote it could be
cutting rock in Siberia for all I know.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Newsgroups mailing-list gateway broken thread

[Emanuel Berg]

Bob Proulx wrote:

> In the intervening two years since the
> previous discussion nothing substantial
> has happened.

Last words from my part in the matter is that
two years ago I didn't understand that I broke
threads on a daily basis simply by using the
newsgroup! The reason I didn't understand it
was that in the newsgroup, it didn't happen,
everything looked and looks fine.

But now when I read Proulx' summary [1] - which
I did back then as well, of course -
I understand it, and it is a bit of a mystery
why I didn't understand it even then.

So I'd like to underline that me using the
newsgroup these two years was because 1) me not
understanding the problems it caused, 2) its
speed compared to the equivalent Gmane group.

In my life, there have been a couple of
rebellious acts, some of which I today feel
shame, some pride, and some 50/50, but this
situation in particular was no such act but
pure ignorance. I didn't understand this until
this particular sentence "When users post to
the newsgroup and it is gatewayed into the
mailing list it breaks threading."

Obviously, in the grand scheme of things, all
this is a bubble, but I thought I'd say
it anyway.

[1] http://lists.gnu.org/archive/html/help-gnu-emacs/2016-06/msg00147.html

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Newsgroups mailing-list gateway broken thread

[Bob Proulx]

Emanuel Berg wrote:
> OK, from now on I use gmane.emacs.help instead!

Since you are one of the prolific posters that would reduce the number
of broken threads.  But there are also other people who post through
from the newsgroups too.  Those threads will still be broken.  So it
doesn't solve the problem.  It just means we would no longer see it
with your messages.

> Everyone happy?

That is impossible! :-)

Bob

Re: Newsgroups mailing-list gateway broken thread

[Bob Proulx]

Nuno Silva wrote:
> But why is the gateway rewriting Message-IDs?

I think that would be a bug.  But it may be a bug only in one of the
newgroup relays.  The list for the replied to message:

  Path: usenet.stanford.edu!goblin2!goblin1!goblin.stu.neva.ru!news.albasani.net!.POSTED!not-for-mail

> Your post, seen from USENET, has:
>
>     Message-ID: <mailman.2440.1539944256.1284.help-gnu-emacs@gnu.org>
>
> And on Gmane:
>
>     Message-ID: <86zhvab5r3.fsf@zoho.com>

I think that is a good clue.  But it may include a race condition.
The message id that includes "mailman" seems like something that could
be inserted by the mail to newsgroup direction of the gateway.  And
why doesn't it include an actual hostname there?  It isn't required.
It is only required that it be unique and any unique string may be
used there.  So that may not be wrong as such.  And it might be the
creation of a news reader creating that message id.  But it feels like
something that needs to be understood in order to continue to make
sense of things.

> (This reply is being sent through USENET, but I tried to set
> In-Reply-To myself. If that works, it shouldn't break threading.)

I think it worked.

What did you try setting it to?  It came to the mailing list with:

  In-Reply-To: <86zhvab5r3.fsf@zoho.com>
  References: <87lg6w43k8.fsf@portable.galex-713.eu> <861s8nefj5.fsf@zoho.com>
          <874ldjtcow.fsf@portable.galex-713.eu>
          <20181018121955500722783@bob.proulx.com>
          <mailman.2440.1539944256.1284.help-gnu-emacs@gnu.org>

It apeared that the message you replied to was:

  Message-ID: <86zhvab5r3.fsf@zoho.com>
  References: <87lg6w43k8.fsf@portable.galex-713.eu> <861s8nefj5.fsf@zoho.com>
          <874ldjtcow.fsf@portable.galex-713.eu>
          <20181018121955500722783@bob.proulx.com>

Seems that the message id should have also been pushed into the
References list too but was not.  But I know you were doing that
manually.  If it were done by software then the message id should be
pushed into References and set as the In-Reply-To.  But it is also
interesting that the References list changed beyond that too.

Bob

Re: Newsgroups mailing-list gateway broken thread

[Nuno Silva]

On 2018-10-19, Bob Proulx wrote:

> Nuno Silva wrote:
>> But why is the gateway rewriting Message-IDs?
>
> I think that would be a bug.  But it may be a bug only in one of the
> newgroup relays.  The list for the replied to message:
>
>   Path: usenet.stanford.edu!goblin2!goblin1!goblin.stu.neva.ru!news.albasani.net!.POSTED!not-for-mail
>
>> Your post, seen from USENET, has:
>>
>>     Message-ID: <mailman.2440.1539944256.1284.help-gnu-emacs@gnu.org>
>>
>> And on Gmane:
>>
>>     Message-ID: <86zhvab5r3.fsf@zoho.com>
>
> I think that is a good clue.  But it may include a race condition.
> The message id that includes "mailman" seems like something that could
> be inserted by the mail to newsgroup direction of the gateway.  And
> why doesn't it include an actual hostname there?  It isn't required.
> It is only required that it be unique and any unique string may be
> used there.  So that may not be wrong as such.  And it might be the
> creation of a news reader creating that message id.  But it feels like
> something that needs to be understood in order to continue to make
> sense of things.
>
>> (This reply is being sent through USENET, but I tried to set
>> In-Reply-To myself. If that works, it shouldn't break threading.)
>
> I think it worked.
>
> What did you try setting it to?  It came to the mailing list with:
>
>   In-Reply-To: <86zhvab5r3.fsf@zoho.com>
>   References: <87lg6w43k8.fsf@portable.galex-713.eu> <861s8nefj5.fsf@zoho.com>
>           <874ldjtcow.fsf@portable.galex-713.eu>
>           <20181018121955500722783@bob.proulx.com>
>           <mailman.2440.1539944256.1284.help-gnu-emacs@gnu.org>

That (above) is exactly what was in the post I sent. I manually added
"In-Reply-To: <86zhvab5r3.fsf@zoho.com>". This should be the Gmane
Message-ID of the post which shows up here (USENET) as
<mailman.2440.1539944256.1284.help-gnu-emacs@gnu.org>.

> It apeared that the message you replied to was:
>
>   Message-ID: <86zhvab5r3.fsf@zoho.com>
>   References: <87lg6w43k8.fsf@portable.galex-713.eu> <861s8nefj5.fsf@zoho.com>
>           <874ldjtcow.fsf@portable.galex-713.eu>
>           <20181018121955500722783@bob.proulx.com>
>
> Seems that the message id should have also been pushed into the
> References list too but was not.  But I know you were doing that
> manually.  If it were done by software then the message id should be
> pushed into References and set as the In-Reply-To.  But it is also
> interesting that the References list changed beyond that too.

It was pushed, but it was the rewritten Message-ID. This is the problem.
For some reason something is changing the Message-IDs. But instead of
changing *all* of them - Message-ID, References and In-Reply-To - (and
in both directions), it appears to be changing the Message-ID only?

-- 
Nuno Silva

Re: Newsgroups mailing-list gateway broken thread

[Emanuel Berg]

Bob Proulx wrote:

> Since you are one of the prolific posters
> that would reduce the number of broken
> threads. But there are also other people who
> post through from the newsgroups too.
> Those threads will still be broken. So it
> doesn't solve the problem. It just means we
> would no longer see it with your messages.

Yes, I understand that would be the optimal
solution. Not just for you guys but also for
me. Because then I would return to
gnu.emacs.help :)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Newsgroups mailing-list gateway broken thread

[Garreau, Alexandre]

On 2018-10-19 at 20:38, Nuno Silva wrote:
> It was pushed, but it was the rewritten Message-ID. This is the
> problem.  For some reason something is changing the Message-IDs. But
> instead of changing *all* of them - Message-ID, References and
> In-Reply-To - (and in both directions), it appears to be changing the
> Message-ID only?

Yes, as observed.  However I believe changing message-id, even it it
were consistently across all headers and in both directions, would be
erroneous, as these might be used in extra headers as well, as for
linking, archiving, etc.  One message shouldn’t see its message-id
change that way, whatever the protocol used to transmit it.

Re: Newsgroups mailing-list gateway broken thread

[Garreau, Alexandre]

On 2018-10-19 at 11:33, Nuno Silva wrote:
> (This reply is being sent through USENET, but I tried to set
> In-Reply-To myself. If that works, it shouldn't break threading.)

It looks it did, because you didn’t set the References header correctly
according here.

Re: Newsgroups mailing-list gateway broken thread

[Nuno Silva]

On 2018-10-21, Garreau, Alexandre wrote:

> On 2018-10-19 at 11:33, Nuno Silva wrote:
>> (This reply is being sent through USENET, but I tried to set
>> In-Reply-To myself. If that works, it shouldn't break threading.)
>
> It looks it did, because you didn’t set the References header correctly
> according here.

Yeah, I forgot to do that, sorry!

-- 
Nuno Silva