bug#21650: 24.5; mh-e keeps trying to open urls

bug#21650: 24.5; mh-e keeps trying to open urls

[Simon Gerraty]

I've used MH and Emacs to read the insane quatitites of mail I get
for many years.

With latest upgrade, I now get rubbish like:

Opening TLS connection with `gnutls-cli --insecure -p 443 ...

Which I certainly do not want (else I'd be using a browser to read mail).
^G won't stop it and a 2nd ^G makes emacs want to core dump!

After googling a bit I set

(setq starttls-use-gnutls nil)
(setq mm-discouraged-alternatives '("text/html" "text/richtext"))

but it is still doing it.

How do I tell Emacs that under absolutely no circumstances do I want it
to do this?

I can remove gnutls-cli, but then it falls back to openssl - which I
cannot remove.

Thanks
--sjg





In GNU Emacs 24.5.1 (x86_64--netbsd)
 of 2015-07-10 on amd64-nb61
Configured using:
 `configure --srcdir=/work/editors/emacs24-nox11/work/emacs-24.5
 --localstatedir=/var --without-dbus --without-m17n-flt --without-otf
 --without-rsvg --without-x --without-xft --without-gif --without-jpeg
 --without-png --without-tiff --without-xpm --prefix=/usr/pkg
 --build=x86_64--netbsd --host=x86_64--netbsd --infodir=/usr/pkg/info
 --mandir=/usr/pkg/man 'CFLAGS=-O2 -I/usr/include' 'CPPFLAGS=-DTERMINFO
 -I/usr/include' 'LDFLAGS=-L/usr/lib -Wl,-R/usr/lib -Wl,-R/usr/pkg/lib''

Important settings:
  value of $LC_ALL: C
  value of $LANG: C
  locale-coding-system: nil

Major mode: MH-Folder

Minor modes in effect:
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  hl-line-mode: t
  mh-showing-mode: t
  display-time-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  file-name-shadow-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Wrote /homes/sjg/Mail/drafts/3192
Sending...backgrounded
Processing deletes and refiles for +inbox...done
Opening TLS connection to `www.lecorpioondemand.net'...
Opening TLS connection with `gnutls-cli --insecure -p 443 www.lecorpioondemand.net'...failed
Opening TLS connection with `gnutls-cli --insecure -p 443 www.lecorpioondemand.net --protocols ssl3'...failed
Opening TLS connection with `openssl s_client -connect www.lecorpioondemand.net:443 -no_ssl2 -ign_eof'...failed
Opening TLS connection to `www.lecorpioondemand.net'...failed
Quit
Making completion list... [2 times]

Load-path shadows:
/homes/sjg/lisp/faces hides /usr/pkg/share/emacs/24.5/lisp/faces
/homes/sjg/lisp/rst hides /usr/pkg/share/emacs/24.5/lisp/textmodes/rst
/usr/pkg/share/emacs/site-lisp/ispell/ispell hides /usr/pkg/share/emacs/24.5/lisp/textmodes/ispell

Features:
(shadow sort warnings emacsbug help-mode mule-util parse-time vc-cvs
add-log shell pcomplete grep compile vc-dispatcher vc-hg sh-script smie
executable rect network-stream starttls url-http tls url-gw url-auth
url-queue gnus-html browse-url xml url-cache mm-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util url-parse auth-source eieio byte-opt bytecomp byte-compile
cl-extra cconv eieio-core url-vars diff-mode easy-mmode python-mode
comint ansi-color ring dired flow-fill misearch multi-isearch cc-langs
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs mh-alias crm ispell mh-thread mh-identity mh-letter
mh-comp qp mm-archive smiley mail-extr mh-mime mh-gnus mh-show goto-addr
thingatpt gnus-cite gnus-art mm-uu mml2015 epg-config mm-view mml-smime
smime password-cache dig mailcap gnus-sum nnoo gnus-group gnus-undo
nnmail mail-source gnus-start gnus-spec gnus-int message sendmail
format-spec rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 ietf-drums gmm-utils mailheader gnus-win
gnus-range gnus gnus-ems nnheader mail-utils mm-util mail-prsvr wid-edit
mh-seq mh-inc hl-line mh-tool-bar tool-bar mh-xface mh-utils mh-folder
which-func imenu easymenu gnus-util time-date mh-scan mh-e regexp-opt
mh-compat mailabbrev mh-acros cl-macs cl gv cl-loaddefs cl-lib
mh-buffers mh-loaddefs advice help-fns time image tooltip electric
uniquify ediff-hook vc-hooks lisp-float-type tabulated-list newcomment
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
multi-tty emacs)

Memory information:
((conses 16 656969 442054)
 (symbols 48 32268 0)
 (miscs 40 1286 4172)
 (strings 32 144770 86862)
 (string-bytes 1 2087681)
 (vectors 16 26619)
 (vector-slots 8 1209012 50622)
 (floats 8 284 595)
 (intervals 56 62208 33487)
 (buffers 960 78))

-- 
Thanks
--sjg

#include <disclaimer>   /* imagine something _very_ witty here */

bug#21650: 24.5; mh-e keeps trying to open urls

[Eli Zaretskii]

> From: Simon Gerraty <sjg@juniper.net>
> Date: Thu, 8 Oct 2015 10:03:17 -0700
> 
> I've used MH and Emacs to read the insane quatitites of mail I get
> for many years.
> 
> With latest upgrade, I now get rubbish like:
> 
> Opening TLS connection with `gnutls-cli --insecure -p 443 ...
> 
> Which I certainly do not want (else I'd be using a browser to read mail).
> ^G won't stop it and a 2nd ^G makes emacs want to core dump!
> 
> After googling a bit I set
> 
> (setq starttls-use-gnutls nil)
> (setq mm-discouraged-alternatives '("text/html" "text/richtext"))
> 
> but it is still doing it.
> 
> How do I tell Emacs that under absolutely no circumstances do I want it
> to do this?

What exactly is that "this" that you don't want Emacs to do?  It's
hard to help without understanding which part of what you described is
the problem.  (Maybe it's because I don't use MH-E; but TLS
connections are in Emacs core, not in MH-E, which just uses it.)

Thanks.

bug#21650: 24.5; mh-e keeps trying to open urls

[Simon J. Gerraty]

Eli Zaretskii <eliz@gnu.org> wrote:
> > How do I tell Emacs that under absolutely no circumstances do I want it
> > to do this?
> 
> What exactly is that "this" that you don't want Emacs to do?  

Sorry, I was refering to the opening of urls (tls or otherwise)
especially before I've had a chance to even see what they are.

That's a serious attack vector and why I would never choose an html
enabled mail reader.

Unfortunately my mail reader of choice (for the last 20 years or so;-)
seems to have morphed into one.
I'm hoping that that can be turned off.

> It's
> hard to help without understanding which part of what you described is
> the problem.  (Maybe it's because I don't use MH-E; but TLS
> connections are in Emacs core, not in MH-E, which just uses it.)

Sorry, mh-e may be a red-herring.
I want to prevent Emacs from opening http[s] urls - at the very least
while scanning mail that I've not even been able to read yet.

Thanks
--sjg

bug#21650: 24.5; mh-e keeps trying to open urls

[Glenn Morris]

Perhaps shr-related.

http://sourceforge.net/p/mh-e/bugs/478
http://debbugs.gnu.org/21519

As always, a minimal example starting from emacs -Q would be helpful.

bug#21650: 24.5; mh-e keeps trying to open urls

[Wolfgang Jenkner]

On Thu, Oct 08 2015, Simon Gerraty wrote:

> With latest upgrade, I now get rubbish like:
>
> Opening TLS connection with `gnutls-cli --insecure -p 443 ...
>
> Which I certainly do not want (else I'd be using a browser to read mail).
> ^G won't stop it and a 2nd ^G makes emacs want to core dump!

This message seems to come from `open-tls-stream' (in lisp/net/tls.el),
so it would be useful to have a backtrace from there.

E.g., instrument the function with edebug-defun and then type `d' when
a call to `open-tls-stream' activates edebug, see (info "(elisp) Edebug Misc").

bug#21650: 24.5; mh-e keeps trying to open urls

[Eli Zaretskii]

> CC: <21650@debbugs.gnu.org>, <sjg@juniper.net>
> From: "Simon J. Gerraty" <sjg@juniper.net>
> Date: Thu, 8 Oct 2015 13:19:18 -0700
> 
> Eli Zaretskii <eliz@gnu.org> wrote:
> > > How do I tell Emacs that under absolutely no circumstances do I want it
> > > to do this?
> > 
> > What exactly is that "this" that you don't want Emacs to do?  
> 
> Sorry, I was refering to the opening of urls (tls or otherwise)
> especially before I've had a chance to even see what they are.
> 
> That's a serious attack vector and why I would never choose an html
> enabled mail reader.
> 
> Unfortunately my mail reader of choice (for the last 20 years or so;-)
> seems to have morphed into one.
> I'm hoping that that can be turned off.
> 
> > It's
> > hard to help without understanding which part of what you described is
> > the problem.  (Maybe it's because I don't use MH-E; but TLS
> > connections are in Emacs core, not in MH-E, which just uses it.)
> 
> Sorry, mh-e may be a red-herring.
> I want to prevent Emacs from opening http[s] urls - at the very least
> while scanning mail that I've not even been able to read yet.

OK, it's clear now, even to me, thanks ;-)

You've got several suggestions for how to present additional
information about what triggers the URL access.  I think that
information will allow us to help you disable this.

In generally, I'd expect to see some option in MH-E to disable this.

bug#21650: 24.5; mh-e keeps trying to open urls

[Simon J. Gerraty]

Glenn Morris <rgm@gnu.org> wrote:
> Perhaps shr-related.

Thanks for the pointer!

shr-blocked-images doesn't appear to be in scope, but
gnus-blocked-images was and setting that to "."
Appears to have stopped at least the most recent offending mail item
from attempting to connect somewhere.
[Unless the fact that it was previously read is relevant]

> http://sourceforge.net/p/mh-e/bugs/478
> http://debbugs.gnu.org/21519
> 
> As always, a minimal example starting from emacs -Q would be helpful.

Looks like the offending item is indeed an img url.
Some people think e-mail is incomplete without corporate logo's and
similar noise.

bug#21650: 24.5; mh-e keeps trying to open urls

[Bill Wohler]

Eli Zaretskii <eliz@gnu.org> writes:

>> CC: <21650@debbugs.gnu.org>, <sjg@juniper.net>
>> From: "Simon J. Gerraty" <sjg@juniper.net>
>> Date: Thu, 8 Oct 2015 13:19:18 -0700
>> 
>> Eli Zaretskii <eliz@gnu.org> wrote:
>> > > How do I tell Emacs that under absolutely no circumstances do I want it
>> > > to do this?
>> > 
>> > What exactly is that "this" that you don't want Emacs to do?  
>> 
>> Sorry, I was refering to the opening of urls (tls or otherwise)
>> especially before I've had a chance to even see what they are.
>> 
>> That's a serious attack vector and why I would never choose an html
>> enabled mail reader.
>> 
>> Unfortunately my mail reader of choice (for the last 20 years or so;-)
>> seems to have morphed into one.
>> I'm hoping that that can be turned off.
>> 
>> > It's
>> > hard to help without understanding which part of what you described is
>> > the problem.  (Maybe it's because I don't use MH-E; but TLS
>> > connections are in Emacs core, not in MH-E, which just uses it.)
>> 
>> Sorry, mh-e may be a red-herring.
>> I want to prevent Emacs from opening http[s] urls - at the very least
>> while scanning mail that I've not even been able to read yet.
>
> OK, it's clear now, even to me, thanks ;-)
>
> You've got several suggestions for how to present additional
> information about what triggers the URL access.  I think that
> information will allow us to help you disable this.
>
> In generally, I'd expect to see some option in MH-E to disable this.

MH-E doesn't render HTML directly, so the options would be in the gnus
(and below) libraries.

-- 
Bill Wohler <wohler@newt.com> aka <Bill.Wohler@nasa.gov>
http://www.newt.com/wohler/
GnuPG ID:610BD9AD

bug#21650: 24.5; mh-e keeps trying to open urls

[Mike Kupfer]

Bill Wohler wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > In generally, I'd expect to see some option in MH-E to disable this.

I'm working on adding a user-visible control for use with shr: never
display images, only display local images (i.e., ones that are included
with the email), or display all (local and remote) images.

> MH-E doesn't render HTML directly, so the options would be in the gnus
> (and below) libraries.

It should be possible to inhibit all image display by setting
gnus-inhibit-images to a non-nil value.  See 
http://sourceforge.net/p/mh-e/bugs/478/ for further discussion.

Another workaround would be to change mm-text-html-renderer (e.g., set
it to 'w3m-standalone).

mike

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

After some discussion with Bill Wohler, I've looked more carefully at
the problem, paying more attention to the overall system design and
layering.

MH-E uses the MIME (mm) libraries to render emails.  There is a variable
mm-inline-text-html-with-image which, according to the documentation,
should be sufficient to disable downloading of images.

  mm-inline-text-html-with-images is a variable defined in `mm-decode.el'.
  Its value is nil
  
  Documentation:
  If non-nil, Gnus will allow retrieving images in HTML contents with
  the <img> tags.  It has no effect on Emacs/w3.  See also the
  documentation for the `mm-w3m-safe-url-regexp' variable.

Unfortunately, shr does not honor mm-inline-text-html-with-images.
Instead, it uses #'gnus-blocked-images as its control (see #'mm-shr).

MH-E could temporarily rebind gnus-blocked-images before calling
#'mm-display-part.  But really, that's a hack to work around the fact
that the documented mm API doesn't work.

For email, it appears that a simple binary control is all that's needed
(either fetch remote images or not).  So it seems like it would be
straightforward for #'mm-shr to use mm-inline-text-html-with-images, and
for Gnus to set mm-inline-text-html-with-images as needed.

But for newsgroups, it looks like finer control is desired.  So I don't
know what the fix should look like.  But MIME libraries are documented
as general-purpose, rather than private to Gnus.  So this really ought
to be resolved at the mm layer, rather than adding renderer-specific
tweaks to MH-E.

mike

bug#21650: fix should be underneath MH-E

[Glenn Morris]

Mike Kupfer wrote:

> But MIME libraries are documented as general-purpose, rather than
> private to Gnus. So this really ought to be resolved at the mm layer,
> rather than adding renderer-specific tweaks to MH-E.

Sounds absolutely right.

(cc to some relevant folks.)

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

On Tue, 02 Feb 2016 13:28:47 -0500, Glenn Morris wrote:
> Mike Kupfer wrote:

>> But MIME libraries are documented as general-purpose, rather than
>> private to Gnus. So this really ought to be resolved at the mm layer,
>> rather than adding renderer-specific tweaks to MH-E.

> Sounds absolutely right.

The first step should be anyway to fix mh-cl-flet[1], recompile
mh-e, and see how the behaviors change, I think.

[1] <http://article.gmane.org/gmane.emacs.bugs/111280>

bug#21650: fix should be underneath MH-E

[Glenn Morris]

Katsumi Yamaoka wrote:

>>> But MIME libraries are documented as general-purpose, rather than
>>> private to Gnus. So this really ought to be resolved at the mm layer,
>>> rather than adding renderer-specific tweaks to MH-E.
>
>> Sounds absolutely right.
>
> The first step should be anyway to fix mh-cl-flet[1], recompile
> mh-e, and see how the behaviors change, I think.

I don't see how that's relevant to this issue.
mm-shr _always_ consults gnus-inhibit-images and gnus-blocked-images,
using them to override shr-inhibit-images and shr-blocked-images.
If mm-shr is supposed to be a general purpose facility for use by things
other than Gnus (?), this seems obviously wrong.
Surely Gnus should make whatever buffer-local shr-related settings it
needs in its Gnus buffers, and the MH-E folks can do the same, rather
than hard-coding Gnus-specific behavior in mm-shr?

(I'd actually say that the use of mh-cl-flet in mh-display-emphasis
seems like the wrong solution as well. MH-E should rather have asked
Gnus to provide an option so that article-emphasize can do what they
want.)

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

On Tue, 02 Feb 2016 17:34:27 -0500, Glenn Morris wrote:
> Katsumi Yamaoka wrote:
>> The first step should be anyway to fix mh-cl-flet[1], recompile
>> mh-e, and see how the behaviors change, I think.

> I don't see how that's relevant to this issue.

Sorry, I'll make time to look into *this* issue.

bug#21650: fix should be underneath MH-E

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> Surely Gnus should make whatever buffer-local shr-related settings it
> needs in its Gnus buffers, and the MH-E folks can do the same, rather
> than hard-coding Gnus-specific behavior in mm-shr?

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21650: fix should be underneath MH-E

[Bill Wohler]

Katsumi Yamaoka <yamaoka@jpl.org> wrote:

> On Tue, 02 Feb 2016 13:28:47 -0500, Glenn Morris wrote:
> > Mike Kupfer wrote:
> 
> >> But MIME libraries are documented as general-purpose, rather than
> >> private to Gnus. So this really ought to be resolved at the mm layer,
> >> rather than adding renderer-specific tweaks to MH-E.
> 
> > Sounds absolutely right.
> 
> The first step should be anyway to fix mh-cl-flet[1], recompile
> mh-e, and see how the behaviors change, I think.
> 
> [1] <http://article.gmane.org/gmane.emacs.bugs/111280>

By the way, I've made your suggested mh-cl-flet fix to MH-E (thanks!!!)
and am letting it bake. I'll try to make some time to check it in this
weekend since I haven't noticed any ill effects.

-- 
Bill Wohler <wohler@newt.com> aka <Bill.Wohler@nasa.gov>
http://www.newt.com/wohler/
GnuPG ID:610BD9AD

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

On Wed, 03 Feb 2016 13:31:34 +1100, Lars Ingebrigtsen wrote:
> Glenn Morris <rgm@gnu.org> writes:

>> Surely Gnus should make whatever buffer-local shr-related settings it
>> needs in its Gnus buffers, and the MH-E folks can do the same, rather
>> than hard-coding Gnus-specific behavior in mm-shr?

> Yup.

My solution is below.  Tested briefly.  This patch moves the
binding of shr-inhibit-images and shr-blocked-images to Gnus
from mm.  So, MH-E has to do a similar thing.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2895 bytes --]

--- gnus-art.el~	2016-02-02 02:43:45.605413000 +0000
+++ gnus-art.el	2016-02-03 09:42:20.238190600 +0000
@@ -2722,9 +2722,8 @@
     (when (gmm-called-interactively-p 'any)
       (gnus-treat-article nil))))
 
-(defun article-wash-html ()
+(defun article-wash-html-1 ()
   "Format an HTML article."
-  (interactive)
   (let ((handles nil)
 	(buffer-read-only nil))
     (when (gnus-buffer-live-p gnus-original-article-buffer)
@@ -2735,6 +2734,19 @@
     (mm-enable-multibyte)
     (mm-inline-text-html handles)))
 
+(defun article-wash-html ()
+  "Format an HTML article."
+  (interactive)
+  (cond ((eq mm-text-html-renderer 'shr)
+	 (require 'shr)
+	 (let (shr-inhibit-images shr-blocked-images)
+	   (with-current-buffer gnus-summary-buffer
+	     (setq shr-inhibit-images gnus-inhibit-images
+		   shr-blocked-images (gnus-blocked-images)))
+	   (article-wash-html-1)))
+	(t
+	 (article-wash-html-1))))
+
 (defvar gnus-article-browse-html-temp-list nil
   "List of temporary files created by `gnus-article-browse-html-parts'.
 Internal variable.")
@@ -4930,7 +4942,9 @@
 	      gnus-url-button-commands)))
 
 (defmacro gnus-bind-safe-url-regexp (&rest body)
-  "Bind `mm-w3m-safe-url-regexp' according to `gnus-safe-html-newsgroups'."
+  "Bind `mm-w3m-safe-url-regexp' according to `gnus-safe-html-newsgroups'.
+Also bind `shr-inhibit-images' and `shr-blocked-images' with
+`gnus-inhibit-images' and `gnus-blocked-images' if `shr' is used."
   `(let ((mm-w3m-safe-url-regexp
 	  (let ((group (if (and (derived-mode-p 'gnus-article-mode)
 				(gnus-buffer-live-p
@@ -4948,7 +4962,15 @@
 		       (member group gnus-safe-html-newsgroups)))
 		nil
 	      mm-w3m-safe-url-regexp))))
-     ,@body))
+     (cond ((eq mm-text-html-renderer 'shr)
+	    (require 'shr)
+	    (let (shr-inhibit-images shr-blocked-images)
+	      (with-current-buffer gnus-summary-buffer
+		(setq shr-inhibit-images gnus-inhibit-images
+		      shr-blocked-images (gnus-blocked-images)))
+	      ,@body))
+	   (t
+	    ,@body))))
 
 (defun gnus-mime-button-menu (event prefix)
  "Construct a context-sensitive menu of MIME commands."
--- mm-decode.el~	2016-01-04 22:05:27.255542500 +0000
+++ mm-decode.el	2016-02-03 09:42:20.238799100 +0000
@@ -1844,15 +1844,7 @@
 				  (when handle
 				    (mm-with-part handle
 				      (buffer-string))))))
-	shr-inhibit-images shr-blocked-images charset char)
-    (if (and (boundp 'gnus-summary-buffer)
-	     (bufferp gnus-summary-buffer)
-	     (buffer-name gnus-summary-buffer))
-	(with-current-buffer gnus-summary-buffer
-	  (setq shr-inhibit-images gnus-inhibit-images
-		shr-blocked-images (gnus-blocked-images)))
-      (setq shr-inhibit-images gnus-inhibit-images
-	    shr-blocked-images (gnus-blocked-images)))
+	charset char)
     (unless handle
       (setq handle (mm-dissect-buffer t)))
     (setq charset (mail-content-type-get (mm-handle-type handle) 'charset))

bug#21650: fix should be underneath MH-E

[Bill Wohler]

Katsumi Yamaoka <yamaoka@jpl.org> wrote:

> On Wed, 03 Feb 2016 13:31:34 +1100, Lars Ingebrigtsen wrote:
> > Glenn Morris <rgm@gnu.org> writes:
> 
> >> Surely Gnus should make whatever buffer-local shr-related settings it
> >> needs in its Gnus buffers, and the MH-E folks can do the same, rather
> >> than hard-coding Gnus-specific behavior in mm-shr?
> 
> > Yup.
> 
> My solution is below.  Tested briefly.  This patch moves the
> binding of shr-inhibit-images and shr-blocked-images to Gnus
> from mm.  So, MH-E has to do a similar thing.

I disagree. This only works for shr and defeats encapsulation.

mm-text-html-renderer can also be gnus-w3m, w3m, emacs-w3m,
w3m-standalone, links, lynx, w3, and html2text. The solution should be
encapsulated in mm and work for all of those renderers rather than
duplicating an incomplete solution in Gnus and MH-E and elsewhere.

Alternatively, the solution can be applied to each renderer that is
missing that functionality.

In any event, Gnus and MH-E are not the correct level for this solution
since we delegate to mm to do the rendering.

> --- gnus-art.el~	2016-02-02 02:43:45.605413000 +0000
> +++ gnus-art.el	2016-02-03 09:42:20.238190600 +0000
> @@ -2722,9 +2722,8 @@
>      (when (gmm-called-interactively-p 'any)
>        (gnus-treat-article nil))))
>  
> -(defun article-wash-html ()
> +(defun article-wash-html-1 ()
>    "Format an HTML article."
> -  (interactive)
>    (let ((handles nil)
>  	(buffer-read-only nil))
>      (when (gnus-buffer-live-p gnus-original-article-buffer)
> @@ -2735,6 +2734,19 @@
>      (mm-enable-multibyte)
>      (mm-inline-text-html handles)))
>  
> +(defun article-wash-html ()
> +  "Format an HTML article."
> +  (interactive)
> +  (cond ((eq mm-text-html-renderer 'shr)
> +	 (require 'shr)
> +	 (let (shr-inhibit-images shr-blocked-images)
> +	   (with-current-buffer gnus-summary-buffer
> +	     (setq shr-inhibit-images gnus-inhibit-images
> +		   shr-blocked-images (gnus-blocked-images)))
> +	   (article-wash-html-1)))
> +	(t
> +	 (article-wash-html-1))))
> +
>  (defvar gnus-article-browse-html-temp-list nil
>    "List of temporary files created by `gnus-article-browse-html-parts'.
>  Internal variable.")
> @@ -4930,7 +4942,9 @@
>  	      gnus-url-button-commands)))
>  
>  (defmacro gnus-bind-safe-url-regexp (&rest body)
> -  "Bind `mm-w3m-safe-url-regexp' according to `gnus-safe-html-newsgroups'."
> +  "Bind `mm-w3m-safe-url-regexp' according to `gnus-safe-html-newsgroups'.
> +Also bind `shr-inhibit-images' and `shr-blocked-images' with
> +`gnus-inhibit-images' and `gnus-blocked-images' if `shr' is used."
>    `(let ((mm-w3m-safe-url-regexp
>  	  (let ((group (if (and (derived-mode-p 'gnus-article-mode)
>  				(gnus-buffer-live-p
> @@ -4948,7 +4962,15 @@
>  		       (member group gnus-safe-html-newsgroups)))
>  		nil
>  	      mm-w3m-safe-url-regexp))))
> -     ,@body))
> +     (cond ((eq mm-text-html-renderer 'shr)
> +	    (require 'shr)
> +	    (let (shr-inhibit-images shr-blocked-images)
> +	      (with-current-buffer gnus-summary-buffer
> +		(setq shr-inhibit-images gnus-inhibit-images
> +		      shr-blocked-images (gnus-blocked-images)))
> +	      ,@body))
> +	   (t
> +	    ,@body))))
>  
>  (defun gnus-mime-button-menu (event prefix)
>   "Construct a context-sensitive menu of MIME commands."
> --- mm-decode.el~	2016-01-04 22:05:27.255542500 +0000
> +++ mm-decode.el	2016-02-03 09:42:20.238799100 +0000
> @@ -1844,15 +1844,7 @@
>  				  (when handle
>  				    (mm-with-part handle
>  				      (buffer-string))))))
> -	shr-inhibit-images shr-blocked-images charset char)
> -    (if (and (boundp 'gnus-summary-buffer)
> -	     (bufferp gnus-summary-buffer)
> -	     (buffer-name gnus-summary-buffer))
> -	(with-current-buffer gnus-summary-buffer
> -	  (setq shr-inhibit-images gnus-inhibit-images
> -		shr-blocked-images (gnus-blocked-images)))
> -      (setq shr-inhibit-images gnus-inhibit-images
> -	    shr-blocked-images (gnus-blocked-images)))
> +	charset char)
>      (unless handle
>        (setq handle (mm-dissect-buffer t)))
>      (setq charset (mail-content-type-get (mm-handle-type handle) 'charset))

-- 
Bill Wohler <wohler@newt.com> aka <Bill.Wohler@nasa.gov>
http://www.newt.com/wohler/
GnuPG ID:610BD9AD

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

Bill Wohler wrote:

> Katsumi Yamaoka <yamaoka@jpl.org> wrote:
> > My solution is below.  Tested briefly.  This patch moves the
> > binding of shr-inhibit-images and shr-blocked-images to Gnus
> > from mm.  So, MH-E has to do a similar thing.
> 
> I disagree. This only works for shr and defeats encapsulation.

I think what makes this a non-trivial problem is wanting more
flexibility than just a binary yes-no decision, which is what
mm-inline-text-html-with-images currently provides.  That's why
gnus-blocked-images is a regexp (or a function that returns a regexp).

Could mm-inline-text-html-with-images be generalized to be more like
gnus-blocked-images?  (For example, nil means don't retrieve anything, t
means retrieve everything, a string would be a regexp of what URLs will
be retrieved.)  Then shr could use mm-inline-text-html-with-images
instead of shr-blocked-images, and MH-E users would have a single knob
that could control any of the different rendering back-ends.

mike

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

On Wed, 03 Feb 2016 19:58:46 -0800, Mike Kupfer wrote:
> Bill Wohler wrote:

>> Katsumi Yamaoka <yamaoka@jpl.org> wrote:
>>> My solution is below.  Tested briefly.  This patch moves the
>>> binding of shr-inhibit-images and shr-blocked-images to Gnus
>>> from mm.  So, MH-E has to do a similar thing.
>>
>> I disagree. This only works for shr and defeats encapsulation.

> I think what makes this a non-trivial problem is wanting more
> flexibility than just a binary yes-no decision, which is what
> mm-inline-text-html-with-images currently provides.  That's why
> gnus-blocked-images is a regexp (or a function that returns a regexp).

> Could mm-inline-text-html-with-images be generalized to be more like
> gnus-blocked-images?  (For example, nil means don't retrieve anything, t
> means retrieve everything, a string would be a regexp of what URLs will
> be retrieved.)  Then shr could use mm-inline-text-html-with-images
> instead of shr-blocked-images, and MH-E users would have a single knob
> that could control any of the different rendering back-ends.

Ok, I was too near-sighted yesterday.  Here is a second try:

Implement the new user options in mm:
`mm-html-inhibit-images' --- boolean
  Non-nil means inhibit displaying of images inline in the article body.
  The default is t.
`mm-html-blocked-images' --- regexp or nil
  Regexp matching image URLs to be blocked.  The default is "".

Make `mm-inline-text-html-with-images' an obsolete variable alias
for `mm-html-inhibit-images'.

How Gnus does when calling an mm function:

(defun gnus-function ()
  (let ((mm-html-inhibit-images gnus-inhibit-images)
	(mm-html-blocked-images (with-current-buffer gnus-summary-buffer
				  (gnus-blocked-images))))
    (mm-function)))

MH-E doesn't have to do like this if there's no need to have
options like `mh-inhibit-images'.

That's all.  Is this the right approach?

In Gnus, shr and gnus-html are controlled by both inhibit-images
and blocked-images, and w3m is controlled by only inhibit-images.
MH-E doesn't use gnus-html.el, does it?  As for mm-shr, it would
have to be changed into:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 706 bytes --]

--- mm-decode.el~	2016-01-04 22:05:27.255542500 +0000
+++ mm-decode.el	2016-02-04 06:05:08.419602200 +0000
@@ -1846,11 +1846,5 @@
 				      (buffer-string))))))
-	shr-inhibit-images shr-blocked-images charset char)
-    (if (and (boundp 'gnus-summary-buffer)
-	     (bufferp gnus-summary-buffer)
-	     (buffer-name gnus-summary-buffer))
-	(with-current-buffer gnus-summary-buffer
-	  (setq shr-inhibit-images gnus-inhibit-images
-		shr-blocked-images (gnus-blocked-images)))
-      (setq shr-inhibit-images gnus-inhibit-images
-	    shr-blocked-images (gnus-blocked-images)))
+	(shr-inhibit-images mm-html-inhibit-images)
+	(shr-blocked-images mm-html-blocked-images)
+	charset char)
     (unless handle

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

Katsumi Yamaoka wrote:

> Implement the new user options in mm:
> `mm-html-inhibit-images' --- boolean
>   Non-nil means inhibit displaying of images inline in the article body.
>   The default is t.
> `mm-html-blocked-images' --- regexp or nil
>   Regexp matching image URLs to be blocked.  The default is "".

This looks okay to me.  And the default looks safe.

Just to confirm, mm-html-blocked-images would block the downloading of
remote images, but images that are embedded in the email and referenced
using a cid would still be displayed?

> Make `mm-inline-text-html-with-images' an obsolete variable alias
> for `mm-html-inhibit-images'.

I'm afraid I don't understand well enough how variable obsolescence
works in Emacs to be sure.  
mm-inline-text-html-with-images is documented in terms of retrieving
messages, and non-nil means to allow retrieval.  mm-html-inhibit-images
would apparently control display of any images (even ones embedded in
the email), and non-nil means *not* to display.

> MH-E doesn't have to do like this if there's no need to have
> options like `mh-inhibit-images'.

Right.  We might want to add a note to the MH-E user guide that mentions
these variables.  But I don't think any changes are needed in the code.

> MH-E doesn't use gnus-html.el, does it?

I don't see any references, no.

thanks,
mike

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

On Thu, 04 Feb 2016 17:41:54 -0800, Mike Kupfer wrote:
> Just to confirm, mm-html-blocked-images would block the downloading of
> remote images, but images that are embedded in the email and referenced
> using a cid would still be displayed?

Yes, it will be displayed unless `inhibit-images' is non-nil.
In that case, the url of the embedded data begins with "cid:",
and `shr-tag-img' doesn't see what `blocked-images' is.

I have `inhibit-images'=t and `blocked-images'=nil, so images are
not displayed normally, but do `W D W' (gnus-article-show-images)
only when I want to see those images.

>> Make `mm-inline-text-html-with-images' an obsolete variable alias
>> for `mm-html-inhibit-images'.

> I'm afraid I don't understand well enough how variable obsolescence
> works in Emacs to be sure.
> mm-inline-text-html-with-images is documented in terms of retrieving
> messages, and non-nil means to allow retrieval.  mm-html-inhibit-images
> would apparently control display of any images (even ones embedded in
> the email), and non-nil means *not* to display.

Oops.  Sorry for my nonsense.  But I'd like to make it obsolete
anyway, since it is a very old option (that is what I made 14
years ago) and has been being used for only w3m.  I'll do:

(make-obsolete-variable 'mm-inline-text-html-with-images nil "25.1")

(defcustom mm-html-inhibit-images t
  "Non-nil means inhibit displaying of images inline in the article body."
  :version "25.1"
  :type 'boolean
  :group 'mime-display
  :set (lambda (symbol value)
	 (custom-set-default
	  symbol
	  (if (boundp 'mm-inline-text-html-with-images)
	      (not (symbol-value 'mm-inline-text-html-with-images))
	    value))))

The refactoring in Gnus and mm will probably be finished within
a week.

Regards,

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

On Fri, 05 Feb 2016 15:04:14 +0900, Katsumi Yamaoka wrote:
> (defcustom mm-html-inhibit-images t
>   "Non-nil means inhibit displaying of images inline in the article body."

Sorry, please let me make it supersede with:

(defcustom mm-html-inhibit-images t
  "Non-nil means inhibit displaying of images inline in the article body."
  :version "25.1"
  :type 'boolean
  :group 'mime-display
  :set (lambda (symbol value)
	 (custom-set-default
	  symbol
	  (cond ((boundp 'mm-html-inhibit-images) value)
		((boundp 'mm-inline-text-html-with-images)
		 (not (symbol-value 'mm-inline-text-html-with-images)))
		(t value)))))

This `:set' trick won't do the trick if this option is loaded
before a user sets the obsolete `mm-inline-text-html-with-images',
though.

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

Katsumi Yamaoka wrote:

> On Thu, 04 Feb 2016 17:41:54 -0800, Mike Kupfer wrote:
> > Just to confirm, mm-html-blocked-images would block the downloading of
> > remote images, but images that are embedded in the email and referenced
> > using a cid would still be displayed?
> 
> Yes, it will be displayed unless `inhibit-images' is non-nil.
> In that case, the url of the embedded data begins with "cid:",
> and `shr-tag-img' doesn't see what `blocked-images' is.

That sounds good.

As for making mm-inline-text-html-with-images obsolete, that sounds fine
to me.

> The refactoring in Gnus and mm will probably be finished within
> a week.

Great!  Thank you very much.

mike

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

On Sat, 06 Feb 2016 14:53:59 -0800, Mike Kupfer wrote:
>> The refactoring in Gnus and mm will probably be finished within
>> a week.
> Great!  Thank you very much.

Done in the emacs-25 branch:

<http://article.gmane.org/gmane.emacs.diffs/134147>

Could you verify the changes?  Thanks.

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

Katsumi Yamaoka wrote:

> Done in the emacs-25 branch:
> 
> <http://article.gmane.org/gmane.emacs.diffs/134147>
> 
> Could you verify the changes?  Thanks.

Hi Katsumi, the diffs look okay to me, though I did notice one typo in
doc/misc/emacs-mime.texi:

> +be fetched and displayed.  For instance, do block all @acronym{URL}s

"do block" -> "to block"

I'll try building and testing the changes next, though I'm not sure how
far I'll get today.

thanks,
mike

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

On Mon, 08 Feb 2016 16:41:03 -0800, Mike Kupfer wrote:

>> +be fetched and displayed.  For instance, do block all @acronym{URL}s
> "do block" -> "to block"

Ah, thanks.  I've fixed it in both emacs-mime.texi and gnus.texi.

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

Mike Kupfer wrote:

> I'll try building and testing the changes next, though I'm not sure how
> far I'll get today.

Everything works as specified.  Thanks!

I had missed the fact that mm-html-inhibit-images defaults to t.  That's
not my preference, but I don't have any compelling arguments in favor of
defaulting to nil.  Will you be creating a NEWS entry for these changes?

cheers,
mike

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

On Mon, 08 Feb 2016 19:24:31 -0800, Mike Kupfer wrote:
> Everything works as specified.  Thanks!

Thanks a lot for the test.

> I had missed the fact that mm-html-inhibit-images defaults to t.  That's
> not my preference, but I don't have any compelling arguments in favor of
> defaulting to nil.

I changed my mind. :)  I'll make both mm-html-inhibit-images and
mm-html-blocked-images default to nil following Gnus' way (enabling
most features by default).

> Will you be creating a NEWS entry for these changes?

No sweat.

bug#21650: fix should be underneath MH-E

[Bill Wohler]

Katsumi Yamaoka <yamaoka@jpl.org> wrote:

> On Mon, 08 Feb 2016 19:24:31 -0800, Mike Kupfer wrote:
> > Everything works as specified.  Thanks!
> 
> Thanks a lot for the test.
> 
> > I had missed the fact that mm-html-inhibit-images defaults to t.  That's
> > not my preference, but I don't have any compelling arguments in favor of
> > defaulting to nil.
> 
> I changed my mind. :)  I'll make both mm-html-inhibit-images and
> mm-html-blocked-images default to nil following Gnus' way (enabling
> most features by default).
> 
> > Will you be creating a NEWS entry for these changes?
> 
> No sweat.

Mike, Katsumi, you guys rock.

Many thanks for working together on this solution!

-- 
Bill Wohler <wohler@newt.com> aka <Bill.Wohler@nasa.gov>
http://www.newt.com/wohler/
GnuPG ID:610BD9AD

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

Katsumi Yamaoka wrote:

> I changed my mind. :)  I'll make both mm-html-inhibit-images and
> mm-html-blocked-images default to nil following Gnus' way (enabling
> most features by default).

Changing mm-html-inhibit-images sounds good, thanks.  For
mm-html-blocked-images, leaving it at "" would follow Gnus' default
behavior for email, which is to not retrieve remote images.  I think
that's what we want.  If mm-html-blocked-images is nil, there's a
privacy concern, in that web bugs could be used to track when an email
is viewed.

> > Will you be creating a NEWS entry for these changes?
> 
> No sweat.

Thanks!

mike

bug#21650: fix should be underneath MH-E

[Katsumi Yamaoka]

On Tue, 09 Feb 2016 07:17:16 -0800, Mike Kupfer wrote:
> For
> mm-html-blocked-images, leaving it at "" would follow Gnus' default
> behavior for email, which is to not retrieve remote images.  I think
> that's what we want.  If mm-html-blocked-images is nil, there's a
> privacy concern, in that web bugs could be used to track when an email
> is viewed.

Agreed.  I'll make it default to "" again.  In that case Gnus
users can use the `W D W' command (gnus-article-show-images) to
view images, that is what I do normally.  It would be better for
other packages to have a similar means as well, I think.

Regards,

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

Katsumi Yamaoka wrote:

> On Tue, 09 Feb 2016 07:17:16 -0800, Mike Kupfer wrote:
> > For
> > mm-html-blocked-images, leaving it at "" would follow Gnus' default
> > behavior for email, which is to not retrieve remote images.  I think
> > that's what we want.
[...]
> Agreed.  I'll make it default to "" again.  In that case Gnus
> users can use the `W D W' command (gnus-article-show-images) to
> view images, that is what I do normally.  It would be better for
> other packages to have a similar means as well, I think.

Yes, agreed.  I don't see anything like that for MH-E, so I'll open a
feature request for it.

cheers,
mike

bug#21650: fix should be underneath MH-E

[Bill Wohler]

Mike Kupfer <m.kupfer@acm.org> wrote:

> Katsumi Yamaoka wrote:
> 
> > On Tue, 09 Feb 2016 07:17:16 -0800, Mike Kupfer wrote:
> > > For
> > > mm-html-blocked-images, leaving it at "" would follow Gnus' default
> > > behavior for email, which is to not retrieve remote images.  I think
> > > that's what we want.
> [...]
> > Agreed.  I'll make it default to "" again.  In that case Gnus
> > users can use the `W D W' command (gnus-article-show-images) to
> > view images, that is what I do normally.  It would be better for
> > other packages to have a similar means as well, I think.
> 
> Yes, agreed.  I don't see anything like that for MH-E, so I'll open a
> feature request for it.

Sounds good, thanks.

-- 
Bill Wohler <wohler@newt.com> aka <Bill.Wohler@nasa.gov>
http://www.newt.com/wohler/
GnuPG ID:610BD9AD

bug#21650: fix should be underneath MH-E

[Mike Kupfer]

Mike Kupfer wrote:

> Katsumi Yamaoka wrote:
> 
> > On Tue, 09 Feb 2016 07:17:16 -0800, Mike Kupfer wrote:
> > > For
> > > mm-html-blocked-images, leaving it at "" would follow Gnus' default
> > > behavior for email, which is to not retrieve remote images.  I think
> > > that's what we want.
> [...]
> > Agreed.  I'll make it default to "" again.  In that case Gnus
> > users can use the `W D W' command (gnus-article-show-images) to
> > view images, that is what I do normally.  It would be better for
> > other packages to have a similar means as well, I think.
> 
> Yes, agreed.  I don't see anything like that for MH-E, so I'll open a
> feature request for it.

Just to follow up: this is https://sourceforge.net/p/mh-e/bugs/483/

best regards,
mike

bug#21650: 21650: fixed in Emacs 25

[Mike Kupfer]

Hi Simon, just to close the loop on the bug report that you filed: in
Emacs 25, MH-E will not download remote images by default.  Thanks for
reporting the issue.  If you have any concerns or questions, please let
me know.

thanks,
mike