problems with multipart/mixed

problems with multipart/mixed

[Matthias Guedemann]

Hi all,

I am using notmuch / emacs as my main mail client now for several months
and loosely follow master.

After an update yesterday I now have problems with some multipart/mixed
mails from mailing lists which are displayed for example as follows (I
could also provide the raw mail if needed):

----8<----------
Subject: [Haskell-cafe] Can't access map value with key.
To: "haskell-cafe@haskell.org" <haskell-cafe@haskell.org>                                                     
Date: Sat, 21 May 2011 04:04:31 +0200                                                           
                                                                                                               
[ multipart/mixed ]
[ text/html ]                                                                                        
_______________________________________________ Haskell-Cafe mailing
list Haskell-Cafe@haskell.org http://       
www.haskell.org/mailman/listinfo/haskell-cafe                                                            
                                                                                                                
[ ATT00001.c: text/plain ]                                                                                      
[ 4-line signature. Click/Enter to toggle visibility. ]
----8<----------

i.e. the html part is not displayed. I'd like to have it displayed
inline (using w3m), just as other html mails and just like it worked
before (at least I think it worked). I probably just missed a simple
configuration option.

My notmuch config in .emacs (and my message mode config) is at the end
of this mail. From the git logs I saw that there have been some changes
to multipart handling and new variables, but I could not really find a
solution.

Perhaps someone can point out where my current config is broken.

best regards
Matthias

;; f_r notmuch                                                                                                  
(require 'notmuch)                                                                                              
(setq message-send-mail-function 'smtpmail-send-it)                                                             
(setq smtpmail-smtp-server MYSMTP)                                                             
(setq smtpmail-smtp-service 25)                                                                                 
(setq user-mail-address MYMAIL)                                                           
(setq message-default-mail-headers "Cc: MYMAIL\nBcc: \n")                                   
(setq message-auto-save-directory "~/MailNoBackup/drafts")                                                      
(add-hook 'message-mode-hook '(lambda () (flyspell-mode t)))                                                    
(add-hook 'message-mode-hook '(lambda () (footnote-mode t)))

;; message-mode  stuff
(setq mm-text-html-renderer 'w3m)                                                                               
(setq mm-attachment-override-types '("image/.*"))                                                               
(setq w3m-display-inline-image t)                                                                              
(setq mm-inline-text-html-with-images "")                                                                      

;; gnus stuff 

...

Re: problems with multipart/mixed

[Dirk Hohndel]

On Sat, 21 May 2011 08:35:13 +0200, Matthias Guedemann <matthias.guedemann@ovgu.de> wrote:
> 
> Hi all,
> 
> I am using notmuch / emacs as my main mail client now for several months
> and loosely follow master.
> 
> After an update yesterday I now have problems with some multipart/mixed
> mails from mailing lists which are displayed for example as follows (I
> could also provide the raw mail if needed):
> 
> ----8<----------
> Subject: [Haskell-cafe] Can't access map value with key.
> To: "haskell-cafe@haskell.org" <haskell-cafe@haskell.org>                                                     
> Date: Sat, 21 May 2011 04:04:31 +0200                                                           
>                                                                                                                
> [ multipart/mixed ]
> [ text/html ]                                                                                        
> _______________________________________________ Haskell-Cafe mailing
> list Haskell-Cafe@haskell.org http://       
> www.haskell.org/mailman/listinfo/haskell-cafe                                                            
>                                                                                                                 
> [ ATT00001.c: text/plain ]                                                                                      
> [ 4-line signature. Click/Enter to toggle visibility. ]
> ----8<----------
> 
> i.e. the html part is not displayed. I'd like to have it displayed
> inline (using w3m), just as other html mails and just like it worked
> before (at least I think it worked). I probably just missed a simple
> configuration option.

If you did then I'm in the same boat. Notmuch/emacs used to display both
parts, text and html, and now only displays text with no way (that I
could find) to display the html part as well.

/D

Re: problems with multipart/mixed

[Matthias Guedemann]

> If you did then I'm in the same boat. Notmuch/emacs used to display both

welcome to the boat!

I found the reason why that happens. Basically it appears because of a
bug in the ongoing multipart development.

In my example, the problem is as follows: currently the outer multipart
message gets ID 1, the first html part ID 2, and the attachment ID 3.

This seems to be correct in the JSON output. But for whatever reason,
show in emacs does not display the html part at all and the attachment
as text/html (it is text/plain). So it seems that the show is off by 1
somewhere.

Now a quick-and-dirty way around this:

reset to c51d5b3cdb5ca0816816e88ca6f7136a24e74eee

and apply this patch:


----8<---------
diff --git a/show-message.c b/show-message.c
index ff9146e..6913d7f 100644
--- a/show-message.c
+++ b/show-message.c
@@ -30,8 +30,8 @@ show_message_part (GMimeObject *part, int *part_count,
        GMimeMultipart *multipart = GMIME_MULTIPART (part);
        int i;
 
-       *part_count = *part_count + 1;
-       (*show_part) (part, part_count);
+       //*part_count = *part_count + 1;
+       //(*show_part) (part, part_count);
 
        for (i = 0; i < g_mime_multipart_get_count (multipart); i++) {
            show_message_part (g_mime_multipart_get_part (multipart, i),
----8<---------

current master is already some steps further, so probably this will be
fixed in a correct way soon, but for now I'll use this.

regards
Matthias

Re: problems with multipart/mixed

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

On Mon, 23 May 2011 15:39:07 +0200, Matthias Guedemann <Matthias.Guedemann@ovgu.de> wrote:
> 
> > If you did then I'm in the same boat. Notmuch/emacs used to display both
> 
> welcome to the boat!

My fault. I'm sorry about this breakage.

> Now a quick-and-dirty way around this:
> 
> reset to c51d5b3cdb5ca0816816e88ca6f7136a24e74eee
> 
> and apply this patch:
..
> -       *part_count = *part_count + 1;
> -       (*show_part) (part, part_count);
> +       //*part_count = *part_count + 1;
> +       //(*show_part) (part, part_count);

Your patch here removes the change to the part numbering for "notmuch
show".

I've just pushed a change which instead adds this same part numbering to
"notmuch part". Please try it out and let me know if you have any
further trouble.

Clearly, we've got a hole in our test suite since it didn't notice this
problem at all.

Meanwhile, Jameson has proposed (in IRC) a patch series to further
rework the way that "notmuch part" works. That series also adds various
tests for "notmuch part" (which will instead be called "notmuch show
--part") to the test suite.

So hopefully we won't have this same regression appear in the future.

Thanks, everyone for reporting this bug.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: problems with multipart/mixed

[Dirk Hohndel]

Hehe, as the reply below shows... there's still something screwy even
with the latest git version... in multipart messages things just go
wrong. Whether I reply (this below should have included your text/plain
part as quote), or whether I try to see the html part of a text/plain +
text/html multipart message...

/D

On Mon, 23 May 2011 14:38:05 -0700, Carl Worth <cworth@cworth.org> wrote:
Non-text part: multipart/signed

Re: problems with multipart/mixed

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2595 bytes --]

On Mon, 23 May 2011 19:46:41 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> Hehe, as the reply below shows... there's still something screwy even
> with the latest git version... in multipart messages things just go
> wrong. Whether I reply (this below should have included your text/plain
> part as quote)

You caught me again, on two points:

	1. Our multipart testing wasn't testing "notmuch reply"

	2. I wasn't actually running the latest code in my own use

I've addressed both of those problems, which made it easy to find and
fix the segfault that was causing the missing data in the reply
buffer. I will hopefully be in a good habit now of creating a Debian
package and installing and using it locally as part of my testing of
major changes.

Meanwhile, I did just push Jameson's recent new-show-part branch (along
with some updates from me). This should complete the big upheaval of
changes to how multipart messages are handled. From here, Jameson will
rebase his crypto branch so we can verify signatures and decrypt
messages within emacs.

> or whether I try to see the html part of a text/plain +
> text/html multipart message...

This is an area where there have been some recent feature changes---and
again, sadly, there's still some missing testing of the emacs features.

The change I am seeing is that previously whenever a message had both a
text/plain part and a corresponding text/html part (withing
multipart/alternative), emacs would render both of them.

Instead, I'm now seeing the text/plain part followed by:

	[ text/html (not shown) ]

As far as that goes, this hiding of the HTML by default is exactly what
I want. (If people don't want this, there's a
notmuch-show-all-multipart/alternative-parts variable that can be
tweaked. Or just do "M-x customize-group notmuch" and find the setting
there.)

Meanwhile, I can imagine that some people might actually need to view
the HTML part that's initially not shown. I just tried hitting 'V' on
the "(not shown)" button and I got several image-viewer windows, each
showing one of the contained images. That's not ideal---it would be
better to get some web browser to display the entire message formatted
correctly.

Maybe that's just something I need to customize on my end, (though, if
so, I think notmuch could do a better job arranging that for the user).

So contributions would be welcome in this area, (both functional
improvements to the emacs interface as well as additional testing of
those emacs features).

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: problems with multipart/mixed

[Dirk Hohndel]

Pulled the latest. Fixes the reply issue - but frequently gets emacs to
dump core. Looking at the backtrace reminds me why I hate emace some
times :-) - it appears to happen in a memmove - but everything else in
the backtrave is useless

Not an improvement.

/D

On Tue, 24 May 2011 12:50:20 -0700, Carl Worth <cworth@cworth.org> wrote:
Non-text part: multipart/signed
> On Mon, 23 May 2011 19:46:41 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> > Hehe, as the reply below shows... there's still something screwy even
> > with the latest git version... in multipart messages things just go
> > wrong. Whether I reply (this below should have included your text/plain
> > part as quote)
> 
> You caught me again, on two points:
> 
> 	1. Our multipart testing wasn't testing "notmuch reply"
> 
> 	2. I wasn't actually running the latest code in my own use
> 
> I've addressed both of those problems, which made it easy to find and
> fix the segfault that was causing the missing data in the reply
> buffer. I will hopefully be in a good habit now of creating a Debian
> package and installing and using it locally as part of my testing of
> major changes.
> 
> Meanwhile, I did just push Jameson's recent new-show-part branch (along
> with some updates from me). This should complete the big upheaval of
> changes to how multipart messages are handled. From here, Jameson will
> rebase his crypto branch so we can verify signatures and decrypt
> messages within emacs.
> 
> > or whether I try to see the html part of a text/plain +
> > text/html multipart message...
> 
> This is an area where there have been some recent feature changes---and
> again, sadly, there's still some missing testing of the emacs features.
> 
> The change I am seeing is that previously whenever a message had both a
> text/plain part and a corresponding text/html part (withing
> multipart/alternative), emacs would render both of them.
> 
> Instead, I'm now seeing the text/plain part followed by:
> 
> 	[ text/html (not shown) ]
> 
> As far as that goes, this hiding of the HTML by default is exactly what
> I want. (If people don't want this, there's a
> notmuch-show-all-multipart/alternative-parts variable that can be
> tweaked. Or just do "M-x customize-group notmuch" and find the setting
> there.)
> 
> Meanwhile, I can imagine that some people might actually need to view
> the HTML part that's initially not shown. I just tried hitting 'V' on
> the "(not shown)" button and I got several image-viewer windows, each
> showing one of the contained images. That's not ideal---it would be
> better to get some web browser to display the entire message formatted
> correctly.
> 
> Maybe that's just something I need to customize on my end, (though, if
> so, I think notmuch could do a better job arranging that for the user).
> 
> So contributions would be welcome in this area, (both functional
> improvements to the emacs interface as well as additional testing of
> those emacs features).
> 
> -Carl
> 
> -- 
> carl.d.worth@intel.com
Non-text part: application/pgp-signature

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: problems with multipart/mixed

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

On Tue, 24 May 2011 14:01:35 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> Pulled the latest. Fixes the reply issue - but frequently gets emacs to
> dump core. Looking at the backtrace reminds me why I hate emace some
> times :-) - it appears to happen in a memmove - but everything else in
> the backtrave is useless

Yikes! That's definitely bad news. Let's coordinate off-list to see if I
can replicate that problem with the message you've got.

> Not an improvement.

Indeed not. The only previous cases we've hit where notmuch was reliably
crashing emacs was a buffer-overflow in emacs triggered by some giant
email messages (spam usually).

In that case, the emacs authors were really quick about finding and
fixing the bug. Will hope for something similar here once we can
replicate the problem.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]