Rmail: new feature to deal with In-Reply-To and References headers

Rmail: new feature to deal with In-Reply-To and References headers

[rameiko87]

Rmail allows the user to move among messages by using the Subject 
header, however the In-Reply-To and References headers provide more 
accurate information about the email chain, and using them to move along 
the conversation thread is certainly more reliable than relying on the 
Subject header. How about implementing the following commands in Rmail:

Move to the next message in the same thread as the current one (that is, 
the next message whose References header includes the current 
Message-ID).
Move to the previous message in the same thread as the current one (that 
is, the message whose Message-ID is the last entry of the References 
header).
Move to the message which the current message is replying to (that is, 
the message whose Message-ID appears in the In-Reply-To header).
Move to the first message of the thread (that is, the first entry of the 
References header). With argument n, move to message number n of the 
thread.
Move to the first message of the thread (as a separate command)
Move to the last message of the thread (that is, the last message whose 
References header includes the current Message-ID).

After this, let's do the analogous for Summaries! I regard these 
improvements as the neat and elegant siblings of the Subject 
header-based walk.

Note: I wonder if any sanity-check on the message-tree is required, for 
instance what if the Message-ID of the current message appears in its 
In-Reply-To or References header.

Re: Rmail: new feature to deal with In-Reply-To and References headers

[Eli Zaretskii]

> Date: Sat, 09 Dec 2023 15:35:37 +0000
> From: rameiko87@posteo.net
> 
> Rmail allows the user to move among messages by using the Subject 
> header, however the In-Reply-To and References headers provide more 
> accurate information about the email chain, and using them to move along 
> the conversation thread is certainly more reliable than relying on the 
> Subject header. How about implementing the following commands in Rmail:

AFAIU, you should already be able to do that in Emacs 29, by using the
new command rmail-summary-by-thread.  After creating that summary, you
can use the usual message movement commands to move between messages
belonging to a thread.

Re: Rmail: new feature to deal with In-Reply-To and References headers

[rameiko87]

> AFAIU, you should already be able to do that in Emacs 29, by using the 
> new command rmail-summary-by-thread.  After creating that summary, you 
> can use the usual message movement commands to move between messages 
> belonging to a thread.

In fact it's not the case, because the order on the summary buffer 
centered on a given message does not correspond in general to the 
parents-descendants order of the thread containing that given message, 
as defined by moving up on the References header for that message.

>> Move to the previous message in the same thread as the current one 
>> (that is, the message whose Message-ID is the last entry of the 
>> References header, among messages in the Rmail file). Repeat argument 
>> goes back to the n-th last entry of the References header among 
>> messages in the Rmail file.

This feature is not implemented currently unless the parents vector is 
populated with the same order given by the References header. Someone 
remembers how the parents vector is recursively defined in rmailsum.el?

And still, going up on the chain requires regenerating the summary 
buffer by centering every time on the parent.

>> Move to the message which the current message is replying to (that is, 
>> the message whose Message-ID appears in the In-Reply-To header), error 
>> if not present on the Rmail file. Repeat argument iterates the command 
>> n-times, perhaps replacing the messages which it can't find in the 
>> Rmail file by their youngest parent.

This is not implemented currently.

>> Move to the first message of the thread (that is, the first entry of 
>> the References header). With argument n, move to message number n of 
>> the thread.
>> Move to the first message of the thread (as a separate command)
>> Move to the last message of the thread (that is, the last message 
>> whose References header includes the current Message-ID).

On 09.12.2023 16:49, Eli Zaretskii wrote:
>> Date: Sat, 09 Dec 2023 15:35:37 +0000
>> 
>> Rmail allows the user to move among messages by using the Subject
>> header, however the In-Reply-To and References headers provide more
>> accurate information about the email chain, and using them to move 
>> along
>> the conversation thread is certainly more reliable than relying on the
>> Subject header. How about implementing the following commands in 
>> Rmail:
> 
> AFAIU, you should already be able to do that in Emacs 29, by using the
> new command rmail-summary-by-thread.  After creating that summary, you
> can use the usual message movement commands to move between messages
> belonging to a thread.

Re: Rmail: new feature to deal with In-Reply-To and References headers

[Eli Zaretskii]

> Date: Sat, 09 Dec 2023 21:54:36 +0000
> From: rameiko87@posteo.net
> Cc: emacs-devel@gnu.org
> 
> > AFAIU, you should already be able to do that in Emacs 29, by using the 
> > new command rmail-summary-by-thread.  After creating that summary, you 
> > can use the usual message movement commands to move between messages 
> > belonging to a thread.
> 
> In fact it's not the case, because the order on the summary buffer 
> centered on a given message does not correspond in general to the 
> parents-descendants order of the thread containing that given message, 
> as defined by moving up on the References header for that message.
> 
> >> Move to the previous message in the same thread as the current one 
> >> (that is, the message whose Message-ID is the last entry of the 
> >> References header, among messages in the Rmail file). Repeat argument 
> >> goes back to the n-th last entry of the References header among 
> >> messages in the Rmail file.
> 
> This feature is not implemented currently unless the parents vector is 
> populated with the same order given by the References header. Someone 
> remembers how the parents vector is recursively defined in rmailsum.el?
> 
> And still, going up on the chain requires regenerating the summary 
> buffer by centering every time on the parent.
> 
> >> Move to the message which the current message is replying to (that is, 
> >> the message whose Message-ID appears in the In-Reply-To header), error 
> >> if not present on the Rmail file. Repeat argument iterates the command 
> >> n-times, perhaps replacing the messages which it can't find in the 
> >> Rmail file by their youngest parent.
> 
> This is not implemented currently.

Patches to improve the existing implementation of by-thread summary
will be welcome, of course.  It is a new feature, so I'm not surprised
that it has some rough edges.

Adding Andrea, who implemented this feature, in the hope that he will
have some comments or maybe patches.

Re: Rmail: new feature to deal with In-Reply-To and References headers

[Andrea Monaco]

I don't fully understand your needs, or maybe we use slightly different
definitions of a thread.  In my implementation, belonging to the same
thread is an equivalence relation: no matter which message of the thread
you start from, you always end up with the same thread; ie there's no
notion of "centering" on a given message.

About ordering: a thread is represented as a boolean vector indexed by
message index, so the messages in a thread summary have the same
ordering, ie by the first field of the summary line (which mirrors the
ordering in the RMAIL file, I think).



Andrea Monaco

Re: Rmail: new feature to deal with In-Reply-To and References headers

[Francesco Potortì]

Andrea Monaco:
>I don't fully understand your needs, or maybe we use slightly different
>definitions of a thread.  In my implementation, belonging to the same
>thread is an equivalence relation: no matter which message of the thread
>you start from, you always end up with the same thread; ie there's no
>notion of "centering" on a given message.
>
>About ordering: a thread is represented as a boolean vector indexed by
>message index, so the messages in a thread summary have the same
>ordering, ie by the first field of the summary line (which mirrors the
>ordering in the RMAIL file, I think).

Well, the ordering in fact is more complex as in general there is no total order.  Unless I am missing something, a thread can be partially ordered, and can be represented by a directed acyclic graph, but not generally by a linear array.

I am not sure though that I understand the issue of centering.  I added context below.

________________________________________________________________
Date: Sat, 09 Dec 2023 21:54:36 +0000
From: rameiko87@posteo.net
To: Eli Zaretskii <eliz@gnu.org>
Cc: emacs-devel@gnu.org
Subject: Re: Rmail: new feature to deal with In-Reply-To and References headers

> AFAIU, you should already be able to do that in Emacs 29, by using the 
> new command rmail-summary-by-thread.  After creating that summary, you 
> can use the usual message movement commands to move between messages 
> belonging to a thread.

In fact it's not the case, because the order on the summary buffer 
centered on a given message does not correspond in general to the 
parents-descendants order of the thread containing that given message, 
as defined by moving up on the References header for that message.

>> Move to the previous message in the same thread as the current one 
>> (that is, the message whose Message-ID is the last entry of the 
>> References header, among messages in the Rmail file). Repeat argument 
>> goes back to the n-th last entry of the References header among 
>> messages in the Rmail file.

This feature is not implemented currently unless the parents vector is 
populated with the same order given by the References header. Someone 
remembers how the parents vector is recursively defined in rmailsum.el?

And still, going up on the chain requires regenerating the summary 
buffer by centering every time on the parent.

>> Move to the message which the current message is replying to (that is, 
>> the message whose Message-ID appears in the In-Reply-To header), error 
>> if not present on the Rmail file. Repeat argument iterates the command 
>> n-times, perhaps replacing the messages which it can't find in the 
>> Rmail file by their youngest parent.

This is not implemented currently.

>> Move to the first message of the thread (that is, the first entry of 
>> the References header). With argument n, move to message number n of 
>> the thread.
>> Move to the first message of the thread (as a separate command)
>> Move to the last message of the thread (that is, the last message 
>> whose References header includes the current Message-ID).

On 09.12.2023 16:49, Eli Zaretskii wrote:
>> Date: Sat, 09 Dec 2023 15:35:37 +0000
>> 
>> Rmail allows the user to move among messages by using the Subject
>> header, however the In-Reply-To and References headers provide more
>> accurate information about the email chain, and using them to move 
>> along
>> the conversation thread is certainly more reliable than relying on the
>> Subject header. How about implementing the following commands in 
>> Rmail:
> 
> AFAIU, you should already be able to do that in Emacs 29, by using the
> new command rmail-summary-by-thread.  After creating that summary, you
> can use the usual message movement commands to move between messages
> belonging to a thread.

Re: Rmail: new feature to deal with In-Reply-To and References headers

[Eli Zaretskii]

> From: Francesco Potortì <pot@gnu.org>
> Date: Wed, 13 Dec 2023 14:17:28 +0100
> Cc: emacs-devel@gnu.org,
> 	eliz@gnu.org,
> 	rameiko87@posteo.net
> 
> Andrea Monaco:
> >I don't fully understand your needs, or maybe we use slightly different
> >definitions of a thread.  In my implementation, belonging to the same
> >thread is an equivalence relation: no matter which message of the thread
> >you start from, you always end up with the same thread; ie there's no
> >notion of "centering" on a given message.
> >
> >About ordering: a thread is represented as a boolean vector indexed by
> >message index, so the messages in a thread summary have the same
> >ordering, ie by the first field of the summary line (which mirrors the
> >ordering in the RMAIL file, I think).
> 
> Well, the ordering in fact is more complex as in general there is no total order.  Unless I am missing something, a thread can be partially ordered, and can be represented by a directed acyclic graph, but not generally by a linear array.

What Andrea says, AFAIU, is that the order is according to the summary
from which you invoked rmail-summary-by-thread.  Since the "usual"
summary is in the order of receiving the messages, that is what you
get; it is usually the actual order of responses to older messages,
unless there was some interruption in email delivery, in which case in
many cases the order could be the reverse: LIFO.

So one could first invoke rmail-sort-by-date, and after that
rmail-summary-by-thread, and get the thread messages in strict
chronological order, which should be close to the requested ordering,
no (the DAG thing aside)?