HTML mails suddenly exposing style & markup in Gnus.

HTML mails suddenly exposing style & markup in Gnus.

[Karl Fogel]

I read mail in Gnus, and after upgrading to latest Bzr today (as I do
every couple of weeks) I am suddenly seeing lots of HTML markup exposed
in mails where it never used to be exposed.  It makes the mails very
difficult to read.  An example is below.

I haven't seen anything on emacs-devel@ about this, and looking through
the ChangeLogs and bzr logs I haven't seen anything jump out that would
explain it.  (Not sure if the recent `html-parse-string' thread has
anything to do with it, but it seems unlikely.)

Old-fashioned plain text mails are not affected, of course.  But before
this upgrade, when I got a MIME multipart email with a text/plain part
and a text/html part, Gnus would display the text/plain part.  Now it
seems to be [decoding and] displaying the text/html part!

Thus in the example message below, there are two mime parts:

  Content-Type: text/plain; charset=utf-8
  Content-Transfer-Encoding: Quoted-printable
  Content-Disposition: inline

and

  Content-Type: text/html; charset=utf-8
  Content-Transfer-Encoding: Base64
  Content-Disposition: inline

...and it seems that Gnus now displays the text/html part by default.
Below is what the body of that mail now looks like in the Gnus Article
mode display window (I can supply the full original message file if it
would help).  This example is indented by two spaces and anonymized in
some trivial ways:

  <table cellpadding="0" cellspacing="0" border="0" width="100%">
    <tr>
      <td style="padding: 10px 10px 10px 20px; text-align: left;">
        
          Reply ABOVE THIS LINE to add a comment to this message
        
      </td>
    </tr>
    <tr>
      <td style="padding: 10px 20px 5px 20px; font-family: Helvetica, Arial, sans-serif; font-size: 14px; background-color: #ffffff; text-align: left; border-bottom: 1px solid #dddddd;" colspan="2">
        <table cellpadding="0" cellspacing="0" border="0">
          <tr>
            <td style="padding: 0 0 5px 0; font-weight: normal; color: #999999; text-align: left; font-size: 14px; font-family: Helvetica, Arial, sans-serif;" width="50" valign="top">Project:</td>
            <td style="padding: 0 0 5px 10px; font-size: 14px; font-weight: normal; text-align: left; font-family: Helvetica, Arial, sans-serif;" valign="top">https://example.com/foo</td>
          </tr>
          <tr>
            <td style="padding: 0 0 5px 0; font-weight: normal; color: #999999; text-align: left; font-size: 14px; font-family: Helvetica, Arial, sans-serif;" width="50" valign="top">Company:</td>
            <td style="padding: 0 0 5px 10px; font-size: 14px; color: #000000; text-align: left; font-family: Helvetica, Arial, sans-serif;" valign="top">OpenPlans</td>
          </tr>
        </table>
      </td>
    </tr>
    <tr>
      <td style="padding-top: 0; padding-bottom: 20px; text-align: left;">
          <table cellpadding="0" cellspacing="0" border="0" align="left">
            <tr>
              <td style="padding: 0 20px 10px 20px; font-family: Helvetica, Arial, sans-serif; font-size: 14px; background-color: #ffffff; text-align: left;">
  
                
  
                
                  <table cellpadding="0" cellspacing="0" border="0">
                    <tr>
                      <td width="57" valign="top">
                        
                      </td>
                      <td style="padding-left: 15px; font-size: 14px; font-family: Helvetica, Arial, sans-serif;" valign="top">
                        <h1 style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; color: #000000; margin: 0 0 15px 0; font-weight: normal; line-height: 1.3em;">
                            Karl Fogel posted a new message:
                          <span style="font-weight: bold;">https://example.com/foo</span>
                        </h1>
                        
                          Abhi, Dan, as we discussed:
  
  	Before moving onto the dedicated server for Foo Bar, we&#8217;d needed to upgrade WordPress and MediaWiki (WordPress had some nasty security holes recently patched).
  
  	The upgrade is now done, and we&#8217;re on the most recent released versions of both.  Many thanks to Philip for timely help when I hit a snag.  I&#8217;m now preparing the new Rackspace server to receive the migrated site.  Veronica, just keep working on the site as you normally do&#8212;we&#8217;ll send you a ping when it&#8217;s time to pause for the switchover.
                        
                                            </td>
                    </tr>
                  </table>
                
  
              </td>
            </tr>
          </table>
        
        </td>
      </tr>
      <tr>
        <td style="padding: 10px 20px 15px 20px; font-family: Helvetica, Arial, sans-serif; font-size: 12px; background-color: #ffffff; text-align: left; border-top: 1px solid #dddddd;">
          https://example.com/foo/subscription/unsubscribe when comments are posted to this message
        </td>
      </tr>
      <tr>
        <table cellpadding="0" cellspacing="0" border="0" align="left" width="100%">
          <tr>
            <td style="padding: 10px 20px 20px 20px; font-size: 10px; font-family: Helvetica, Arial, sans-serif; color: #666666; text-align: left; border-top: 1px solid #dddddd;" valign="top">
              https://example.com/foo/prefers_text_emails?
            </td>
            <td style="padding: 0 20px 20px 20px; font-size: 10px; font-family: Helvetica, Arial, sans-serif; color: #666666; text-align: right; border-top: 1px solid #dddddd;">
              Delivered by http://example.com/?baz
            </td>
          </tr>
        </table>
      </tr>
  </table>

Re: HTML mails suddenly exposing style & markup in Gnus.

[Lars Magne Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> ...and it seems that Gnus now displays the text/html part by default.

Gnus has gotten a new HTML renderer based on w3m "built in", but the
priorities in what types it chooses to show shouldn't have changed.  I
think.

Do you have the w3m program installed on your system?

But if you have, then Gnus should have rendered the HTML all
pretty-like...  unless you have something other overriding that,
somehow.

What the value of mm-text-html-renderer and
mm-inline-text-html-renderer?  The former should be `gnus-article-html',
and the latter should be nil.  (If you have w3m installed.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: HTML mails suddenly exposing style & markup in Gnus.

[Karl Fogel]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>Gnus has gotten a new HTML renderer based on w3m "built in", but the
>priorities in what types it chooses to show shouldn't have changed.  I
>think.
>
>Do you have the w3m program installed on your system?

Yes:

  $ w3m -version
  w3m version w3m/0.5.2+cvs-1.1027,                                     \
      options lang=en,m17n,image,color,ansi-color,mouse,gpm,menu,cookie,\
      ssl,ssl-verify,external-uri-loader,w3mmailer,nntp,gopher,ipv6,    \
      alarm,mark,migemo
  $ 

>But if you have, then Gnus should have rendered the HTML all
>pretty-like...  unless you have something other overriding that,
>somehow.
>
>What the value of mm-text-html-renderer and
>mm-inline-text-html-renderer?  The former should be `gnus-article-html',
>and the latter should be nil.  (If you have w3m installed.)

I don't set them in any init file, and their ambient values are:

  mm-text-html-renderer is a variable defined in `mm-decode.el'.
  Its value is html2text

  mm-inline-text-html-renderer is a variable defined in `mm-decode.el'.
  Its value is nil

This is true even within a Gnus Article buffer.  (Interestingly,
`gnus-article-html' is *not* listed as a "defined renderer type" in the
help documentation for `mm-text-html-renderer'.)

After I do (setq mm-text-html-renderer 'gnus-article-html) the problem
goes away and those articles render beautifully!  Thank you for the tip.

I guess we should fix the default in Emacs?  I don't know this code very
well, but I assume it's a bug that mm-text-html-renderer does not have
the value `gnus-article-html' when w3m is present?  Should it also have
that value when w3m is not present?

In the meantime, I guess I'll just hard set it in my .emacs, unless you
have a better recommendation.

-Karl

Re: HTML mails suddenly exposing style & markup in Gnus.

[Lars Magne Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

>>Do you have the w3m program installed on your system?
>
> Yes:
>
>   $ w3m -version

[...]

> I don't set them in any init file, and their ambient values are:
>
>   mm-text-html-renderer is a variable defined in `mm-decode.el'.
>   Its value is html2text
>
>   mm-inline-text-html-renderer is a variable defined in `mm-decode.el'.
>   Its value is nil

Hm.  The definition of `mm-text-html-renderer' is:

(defcustom mm-text-html-renderer
  (cond ((and (executable-find "w3m")
	      (executable-find "curl"))
	 'gnus-article-html)
	((executable-find "links") 'links)
	((executable-find "lynx") 'lynx)
	((locate-library "w3") 'w3)
	((locate-library "html2text") 'html2text)
	(t nil))

So does that mean that you don't have curl?  I thought everybody had
curl installed.  :-)

Anyway, if html2text was the definition, then it should have done a
better job of rendering the HTML than it did.  html2text is very
simple -- it just strips tags away (basically), but you saw raw HTML,
which must mean that html2text doesn't work at all for you.

Which is puzzling, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: HTML mails suddenly exposing style & markup in Gnus.

[Karl Fogel]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>Hm.  The definition of `mm-text-html-renderer' is:
>
>(defcustom mm-text-html-renderer
>  (cond ((and (executable-find "w3m")
>	      (executable-find "curl"))
>	 'gnus-article-html)
>	((executable-find "links") 'links)
>	((executable-find "lynx") 'lynx)
>	((locate-library "w3") 'w3)
>	((locate-library "html2text") 'html2text)
>	(t nil))
>
>So does that mean that you don't have curl?  I thought everybody had
>curl installed.  :-)

This is a fresh Debian GNU/linux box, so it didn't yet have 'curl'
installed.  I've fixed my box setup script [1] to include 'curl' in my
list of standard installs now.  After installing 'curl', Emacs's
behavior is as desired, and `mm-text-html-renderer's value is
`gnus-article-html'.

>Anyway, if html2text was the definition, then it should have done a
>better job of rendering the HTML than it did.  html2text is very
>simple -- it just strips tags away (basically), but you saw raw HTML,
>which must mean that html2text doesn't work at all for you.
>
>Which is puzzling, too.

Yeah, don't know why.  The html2text program is installed.
But... "mattress in the road" situation.  My problem is solved, and I
have other more important bugs to chase :-(.  Thanks for your help,
Lars.  I'd certainly be happy to do any further testing you suggest.

-Karl

[1] http://svn.red-bean.com/repos/kfogel/trunk/bin/new-box-setup.sh

Re: HTML mails suddenly exposing style & markup in Gnus.

[Lars Magne Ingebrigtsen]

Karl Fogel <kfogel@red-bean.com> writes:

> This is a fresh Debian GNU/linux box, so it didn't yet have 'curl'
> installed. 

Ah, right.  I've now fixed up the dependencies in `gnus-article-html'
and stuff to work if you have w3m, but no curl.  (You'll get HTML
rendered nicely, but no external images.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: HTML mails suddenly exposing style & markup in Gnus.

[Kim F. Storm]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Karl Fogel <kfogel@red-bean.com> writes:
>
>> This is a fresh Debian GNU/linux box, so it didn't yet have 'curl'
>> installed. 
>
> Ah, right.  I've now fixed up the dependencies in `gnus-article-html'
> and stuff to work if you have w3m, but no curl.  (You'll get HTML
> rendered nicely, but no external images.)

I think "wget" is more widely available than curl.

-- 
Kim F. Storm  http://www.cua.dk

Re: HTML mails suddenly exposing style & markup in Gnus.

[Miles Bader]

no-spam@cua.dk (Kim F. Storm) writes:
>> Ah, right.  I've now fixed up the dependencies in `gnus-article-html'
>> and stuff to work if you have w3m, but no curl.  (You'll get HTML
>> rendered nicely, but no external images.)
>
> I think "wget" is more widely available than curl.

Probably, though, curl seems a great deal less wacky than wget...

-Miles

-- 
Patience, n. A minor form of despair, disguised as a virtue.

Re: HTML mails suddenly exposing style & markup in Gnus.

[Giuseppe Scrivano]

Miles Bader <miles@gnu.org> writes:

>> I think "wget" is more widely available than curl.
>
> Probably, though, curl seems a great deal less wacky than wget...

what exactly do not you like in wget?  If something is wacky, then it
should be changed.

Thanks,
Giuseppe