notmuch-mutt: support for messages that lack Message-ID (v2)

notmuch-mutt: support for messages that lack Message-ID (v2)

[Stefano Zacchiroli]

Heya, please find attached the second take on adding support for
messages that lack Message-ID to notmuch-mutt. For context see
http://notmuchmail.org/pipermail/notmuch/2015/thread.html#20079

In this new verison of the patch we start computing the SHA1
incrementally, in case no Message-ID will be found. As soon as the
header is over we parse it (using Mail::Header, to properly handle
line breaks). If a Message-ID is found, it is retourned and SHA1
computation on the rest of the message is aborted. If no Message-ID is
found, SHA1 computation continues, and is used in the end to
synthesize a fake Message-ID in the same way that notmuch itself would
to.

Note that, contrary to the previous version, no changes of the Debian
packaging are needed this time: Mail::Header is contained in the same
package of Mail::Internet that notmuch-mutt was already using, and
Digest::SHA is shipped with Perl itself.

Please consider applying this patch to notmuch's master branch.

Cheers.

[PATCH 1/2] notmuch-mutt README: use metacpn.org/* as deps homepages

[Stefano Zacchiroli]

---
 contrib/notmuch-mutt/README | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
index 382ac91..c661447 100644
--- a/contrib/notmuch-mutt/README
+++ b/contrib/notmuch-mutt/README
@@ -33,13 +33,13 @@ Requirements
 
 To *run* notmuch-mutt you will need Perl with the following libraries:
 
-- Mail::Box <http://search.cpan.org/~markov/Mail-Box/>
+- Mail::Box <https://metacpan.org/pod/Mail::Box>
   (Debian package: libmail-box-perl)
-- Mail::Internet <http://search.cpan.org/~markov/MailTools/>
+- Mail::Internet <https://metacpan.org/pod/Mail::Internet>
   (Debian package: libmailtools-perl)
-- String::ShellQuote <http://search.cpan.org/~rosch/String-ShellQuote/ShellQuote.pm>
+- String::ShellQuote <https://metacpan.org/pod/String::ShellQuote>
   (Debian package: libstring-shellquote-perl)
-- Term::ReadLine <http://search.cpan.org/~hayashi/Term-ReadLine-Gnu/>
+- Term::ReadLine::Gnu <https://metacpan.org/pod/Term::ReadLine::Gnu>
   (Debian package: libterm-readline-gnu-perl)
 
 To *build* notmuch-mutt documentation you will need:
-- 
2.1.4

[PATCH 2/2] notmuch-mutt: support for messages that lack Message-ID headers

[Stefano Zacchiroli]

From: "Jan N. Klug" <jan.n.klug@rub.de>

For those messages, compute a synthetic Message-ID based on the SHA1
of the whole message, in the same way that notmuch would do. See:
http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c

To do the above, rewrite get_message_id() to scan the current message
line by line, incrementally computing a SHA1. If a Message-ID is found
the SHA1 computation will be aborted; otherwise used for the synthetic
Message-ID.

Signed-off-by: Stefano Zacchiroli <zack@upsilon.cc>
---
 contrib/notmuch-mutt/README       |  4 +++-
 contrib/notmuch-mutt/notmuch-mutt | 33 ++++++++++++++++++++++++++++-----
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
index c661447..9c3379e 100644
--- a/contrib/notmuch-mutt/README
+++ b/contrib/notmuch-mutt/README
@@ -33,9 +33,11 @@ Requirements
 
 To *run* notmuch-mutt you will need Perl with the following libraries:
 
+- Digest::SHA <https://metacpan.org/release/Digest-SHA>
+  (Debian package: libdigest-sha-perl)
 - Mail::Box <https://metacpan.org/pod/Mail::Box>
   (Debian package: libmail-box-perl)
-- Mail::Internet <https://metacpan.org/pod/Mail::Internet>
+- Mail::Header <https://metacpan.org/pod/Mail::Header>
   (Debian package: libmailtools-perl)
 - String::ShellQuote <https://metacpan.org/pod/String::ShellQuote>
   (Debian package: libstring-shellquote-perl)
diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 4969e4b..84af140 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -13,11 +13,12 @@ use warnings;
 
 use File::Path;
 use Getopt::Long qw(:config no_getopt_compat);
-use Mail::Internet;
+use Mail::Header;
 use Mail::Box::Maildir;
 use Pod::Usage;
 use String::ShellQuote;
 use Term::ReadLine;
+use Digest::SHA;
 
 
 my $xdg_cache_dir = "$ENV{HOME}/.cache";
@@ -75,10 +76,32 @@ sub prompt($$) {
 }
 
 sub get_message_id() {
-    my $mail = Mail::Internet->new(\*STDIN);
-    my $mid = $mail->head->get("message-id") or return undef;
-    $mid =~ /^<(.*)>$/;	# get message-id value
-    return $1;
+    my $mid = undef;
+    my $in_header = 1;
+    my @raw_header = ();
+    my $sha = Digest::SHA->new(1);  # SHA1 hash of the whole mail
+
+    while (<STDIN>) {  # compute SHA1 as we go
+	push(@raw_header, $_) if $in_header;  # cache header lines
+        if ($_ =~ /^$/) {  # end of header, parse it and look for Message-ID
+	    $in_header = 0;
+	    my $head = Mail::Header->new(\@raw_header);
+	    $mid = $head->get("message-id") or undef;
+	    if ($mid) {
+		$mid =~ /^<(.*)>$/;  # get message-id value
+		$mid = $1;
+		last;  # stop hashing
+	    }
+        }
+	$sha->add($_);  # update hash
+    }
+
+    # If no message-id was found, generate one-id in the same way that
+    # notmuch would do.
+    # See: http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
+    $mid ||= "notmuch-sha1-".$sha->hexdigest;
+
+    return $mid;
 }
 
 sub search_action($$$@) {
-- 
2.1.4

Re: [PATCH 2/2] notmuch-mutt: support for messages that lack Message-ID headers

[Tomi Ollila]

On Mon, Feb 09 2015, Stefano Zacchiroli <zack@upsilon.cc> wrote:

> From: "Jan N. Klug" <jan.n.klug@rub.de>
>
> For those messages, compute a synthetic Message-ID based on the SHA1
> of the whole message, in the same way that notmuch would do. See:
> http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
>
> To do the above, rewrite get_message_id() to scan the current message
> line by line, incrementally computing a SHA1. If a Message-ID is found
> the SHA1 computation will be aborted; otherwise used for the synthetic
> Message-ID.

I'd suggest the (something like) following way:

First loop through headers, push lines to the array.

check for Message-Id (using Mail::Header): if found, return it

else message id not found:
 
  my $sha = Digest::SHA->new(1);
  $sha->add($_) foreach(@raw_header);
  $sha->addfile(\*STDIN);
  return  'notmuch-sha1-' . $sha->hexdigest;

(BTW: the implementation in the patch (below) would check for message-id in
@raw_header every time mail has an empty line ?)

Tomi


>
> Signed-off-by: Stefano Zacchiroli <zack@upsilon.cc>
> ---
>  contrib/notmuch-mutt/README       |  4 +++-
>  contrib/notmuch-mutt/notmuch-mutt | 33 ++++++++++++++++++++++++++++-----
>  2 files changed, 31 insertions(+), 6 deletions(-)
>
> diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
> index c661447..9c3379e 100644
> --- a/contrib/notmuch-mutt/README
> +++ b/contrib/notmuch-mutt/README
> @@ -33,9 +33,11 @@ Requirements
>  
>  To *run* notmuch-mutt you will need Perl with the following libraries:
>  
> +- Digest::SHA <https://metacpan.org/release/Digest-SHA>
> +  (Debian package: libdigest-sha-perl)
>  - Mail::Box <https://metacpan.org/pod/Mail::Box>
>    (Debian package: libmail-box-perl)
> -- Mail::Internet <https://metacpan.org/pod/Mail::Internet>
> +- Mail::Header <https://metacpan.org/pod/Mail::Header>
>    (Debian package: libmailtools-perl)
>  - String::ShellQuote <https://metacpan.org/pod/String::ShellQuote>
>    (Debian package: libstring-shellquote-perl)
> diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
> index 4969e4b..84af140 100755
> --- a/contrib/notmuch-mutt/notmuch-mutt
> +++ b/contrib/notmuch-mutt/notmuch-mutt
> @@ -13,11 +13,12 @@ use warnings;
>  
>  use File::Path;
>  use Getopt::Long qw(:config no_getopt_compat);
> -use Mail::Internet;
> +use Mail::Header;
>  use Mail::Box::Maildir;
>  use Pod::Usage;
>  use String::ShellQuote;
>  use Term::ReadLine;
> +use Digest::SHA;
>  
>  
>  my $xdg_cache_dir = "$ENV{HOME}/.cache";
> @@ -75,10 +76,32 @@ sub prompt($$) {
>  }
>  
>  sub get_message_id() {
> -    my $mail = Mail::Internet->new(\*STDIN);
> -    my $mid = $mail->head->get("message-id") or return undef;
> -    $mid =~ /^<(.*)>$/;	# get message-id value
> -    return $1;
> +    my $mid = undef;
> +    my $in_header = 1;
> +    my @raw_header = ();
> +    my $sha = Digest::SHA->new(1);  # SHA1 hash of the whole mail
> +
> +    while (<STDIN>) {  # compute SHA1 as we go
> +	push(@raw_header, $_) if $in_header;  # cache header lines
> +        if ($_ =~ /^$/) {  # end of header, parse it and look for Message-ID
> +	    $in_header = 0;
> +	    my $head = Mail::Header->new(\@raw_header);
> +	    $mid = $head->get("message-id") or undef;
> +	    if ($mid) {
> +		$mid =~ /^<(.*)>$/;  # get message-id value
> +		$mid = $1;
> +		last;  # stop hashing
> +	    }
> +        }
> +	$sha->add($_);  # update hash
> +    }
> +
> +    # If no message-id was found, generate one-id in the same way that
> +    # notmuch would do.
> +    # See: http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
> +    $mid ||= "notmuch-sha1-".$sha->hexdigest;
> +
> +    return $mid;
>  }
>  
>  sub search_action($$$@) {
> -- 
> 2.1.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

notmuch-mutt: support for messages that lack Message-ID headers (v3)

[Stefano Zacchiroli]

Take #3 on adding support for messages that lack Message-ID headers to
notmuch-mutt. For context see:
- http://notmuchmail.org/pipermail/notmuch/2015/thread.html#20171
- http://notmuchmail.org/pipermail/notmuch/2015/thread.html#20079

I've rewritten the patch taking into account the suggestions by Tomi
Ollila, who suggested a much cleaner approach for processing headers
and possibly fallback to synthetic Message-IDs (thanks Tomi!).

I'll be happy to hear further comments on this patch series.
If there are none, please merge it into master.

Cheers.

[PATCH 1/3] notmuch-mutt: bump copyright year

[Stefano Zacchiroli]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 618 bytes --]

---
 contrib/notmuch-mutt/notmuch-mutt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 4969e4b..6d4d60c 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -2,7 +2,7 @@
 #
 # notmuch-mutt - notmuch (of a) helper for Mutt
 #
-# Copyright: © 2011-2012 Stefano Zacchiroli <zack@upsilon.cc> 
+# Copyright: © 2011-2015 Stefano Zacchiroli <zack@upsilon.cc>
 # License: GNU General Public License (GPL), version 3 or above
 #
 # See the bottom of this file for more documentation.
-- 
2.1.4

[PATCH 2/3] notmuch-mutt README: use metacpn.org/* as deps homepages

[Stefano Zacchiroli]

---
 contrib/notmuch-mutt/README | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
index 382ac91..c661447 100644
--- a/contrib/notmuch-mutt/README
+++ b/contrib/notmuch-mutt/README
@@ -33,13 +33,13 @@ Requirements
 
 To *run* notmuch-mutt you will need Perl with the following libraries:
 
-- Mail::Box <http://search.cpan.org/~markov/Mail-Box/>
+- Mail::Box <https://metacpan.org/pod/Mail::Box>
   (Debian package: libmail-box-perl)
-- Mail::Internet <http://search.cpan.org/~markov/MailTools/>
+- Mail::Internet <https://metacpan.org/pod/Mail::Internet>
   (Debian package: libmailtools-perl)
-- String::ShellQuote <http://search.cpan.org/~rosch/String-ShellQuote/ShellQuote.pm>
+- String::ShellQuote <https://metacpan.org/pod/String::ShellQuote>
   (Debian package: libstring-shellquote-perl)
-- Term::ReadLine <http://search.cpan.org/~hayashi/Term-ReadLine-Gnu/>
+- Term::ReadLine::Gnu <https://metacpan.org/pod/Term::ReadLine::Gnu>
   (Debian package: libterm-readline-gnu-perl)
 
 To *build* notmuch-mutt documentation you will need:
-- 
2.1.4

[PATCH 3/3] notmuch-mutt: support for messages that lack Message-ID headers

[Stefano Zacchiroli]

For those messages, compute a synthetic Message-ID based on the SHA1
of the whole message, in the same way that notmuch would do. See:
http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c

To do the above, rewrite get_message_id() to accumulate header lines,
parse them to check for Message-ID, and fallback to SHA1 computation
if it is not present.

Thanks to:
- Jan N. Klug for preliminary versions of this patch
- Tomi Ollila for suggesting an elegant implementation
---
 contrib/notmuch-mutt/README       |  4 +++-
 contrib/notmuch-mutt/notmuch-mutt | 30 +++++++++++++++++++++++++-----
 2 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/contrib/notmuch-mutt/README b/contrib/notmuch-mutt/README
index c661447..9c3379e 100644
--- a/contrib/notmuch-mutt/README
+++ b/contrib/notmuch-mutt/README
@@ -33,9 +33,11 @@ Requirements
 
 To *run* notmuch-mutt you will need Perl with the following libraries:
 
+- Digest::SHA <https://metacpan.org/release/Digest-SHA>
+  (Debian package: libdigest-sha-perl)
 - Mail::Box <https://metacpan.org/pod/Mail::Box>
   (Debian package: libmail-box-perl)
-- Mail::Internet <https://metacpan.org/pod/Mail::Internet>
+- Mail::Header <https://metacpan.org/pod/Mail::Header>
   (Debian package: libmailtools-perl)
 - String::ShellQuote <https://metacpan.org/pod/String::ShellQuote>
   (Debian package: libstring-shellquote-perl)
diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 6d4d60c..126cbf4 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -13,11 +13,12 @@ use warnings;
 
 use File::Path;
 use Getopt::Long qw(:config no_getopt_compat);
-use Mail::Internet;
+use Mail::Header;
 use Mail::Box::Maildir;
 use Pod::Usage;
 use String::ShellQuote;
 use Term::ReadLine;
+use Digest::SHA;
 
 
 my $xdg_cache_dir = "$ENV{HOME}/.cache";
@@ -75,10 +76,29 @@ sub prompt($$) {
 }
 
 sub get_message_id() {
-    my $mail = Mail::Internet->new(\*STDIN);
-    my $mid = $mail->head->get("message-id") or return undef;
-    $mid =~ /^<(.*)>$/;	# get message-id value
-    return $1;
+    my $mid = undef;
+    my @headers = ();
+
+    while (<STDIN>) {  # collect header lines in @headers
+	push(@headers, $_);
+	last if $_ =~ /^$/;
+    }
+    my $head = Mail::Header->new(\@headers);
+    $mid = $head->get("message-id") or undef;
+
+    if ($mid) {  # Message-ID header found
+	$mid =~ /^<(.*)>$/;  # extract message id
+	$mid = $1;
+    } else {  # Message-ID header not found, synthesize a message id
+	      # based on SHA1, as notmuch would do.  See:
+	      # http://git.notmuchmail.org/git/notmuch/blob/HEAD:/lib/sha1.c
+	my $sha = Digest::SHA->new(1);
+	$sha->add($_) foreach(@headers);
+	$sha->addfile(\*STDIN);
+	$mid = 'notmuch-sha1-' . $sha->hexdigest;
+    }
+
+    return $mid;
 }
 
 sub search_action($$$@) {
-- 
2.1.4

Re: notmuch-mutt: support for messages that lack Message-ID headers (v3)

[Tomi Ollila]

On Sun, Feb 15 2015, Stefano Zacchiroli <zack@upsilon.cc> wrote:

> Take #3 on adding support for messages that lack Message-ID headers to
> notmuch-mutt. For context see:
> - http://notmuchmail.org/pipermail/notmuch/2015/thread.html#20171
> - http://notmuchmail.org/pipermail/notmuch/2015/thread.html#20079
>
> I've rewritten the patch taking into account the suggestions by Tomi
> Ollila, who suggested a much cleaner approach for processing headers
> and possibly fallback to synthetic Message-IDs (thanks Tomi!).
>
> I'll be happy to hear further comments on this patch series.
> If there are none, please merge it into master.

This series Look Good To Me.

Tomi

>
> Cheers.
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch-mutt: support for messages that lack Message-ID headers (v3)

[David Bremner]

Stefano Zacchiroli <zack@upsilon.cc> writes:

> I'll be happy to hear further comments on this patch series.
> If there are none, please merge it into master.

pushed.

d