problems viewing attachments in emacs ui

problems viewing attachments in emacs ui

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Hey, folks.  I've been experiencing a very annoying and persistent
problem of late where the emacs ui is unable to pass message parts to
external viewers.  Whenever I try to open an attachment or inline part
in an external viewer, the helper application is launched, but it
appears to never be able to find the temporary file.

For instance, if I launch notmuch-show-view-part on an html part, my
browser opens pointed at e.g. the following file:

  file:///home/jrollins/tmp/emm.610040w/mm.6100F_2.htm

But the browser shows the following error:

  File not found
  Iceweasel can't find the file at /home/jrollins/tmp/emm.610040w/mm.6100F_2.htm.

And sure enough, the file is not there.  html parts are definitely the
worst, since they seem to never be copied to their temporary file paths
at all, but it happens with pretty much all other part types as well,
although with slightly different behavior: e.g. image and pdf parts will
open in external viewers BRIEFLY, before the temporary file appears to
be yanked out from under them as well (at which point, depending on the
app, they complain that the file is no longer there they stop displaying
it).  I've even tried creating my own part handler to capture the temp
file immediately into *another* tempfile, in the hopes of catching it,
but that doesn't seem to help either.

This is obviously quite frustrating, since I have to view attachments
outside of emacs quite frequently.  I think I've heard speculation on
IRC that this is an emacs24 bug.  I'm kind of at a loss at this point
for how to get around the issue.  Any help would be very much
appreciated!

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: problems viewing attachments in emacs ui

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On Sun, Jul 28 2013, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> For instance, if I launch notmuch-show-view-part on an html part, my
> browser opens pointed at e.g. the following file:
>
>   file:///home/jrollins/tmp/emm.610040w/mm.6100F_2.htm
>
> But the browser shows the following error:
>
>   File not found
>   Iceweasel can't find the file at /home/jrollins/tmp/emm.610040w/mm.6100F_2.htm.

I'm now realizing that my problem with html parts is probably that
browser is attempting to open the temporary file in the background.
When the browser call returns, the caller assumes the application is
done with the temp file and purges it.  So for this issue at least I
need to either convince my browser to not open the file in the
background, or tell emacs to cleanup temp files at some later time
(session termination, for instance).

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: problems viewing attachments in emacs ui

[David Bremner]

Jameson Graef Rollins <jrollins@finestructure.net> writes:

> I'm now realizing that my problem with html parts is probably that
> browser is attempting to open the temporary file in the background.
> When the browser call returns, the caller assumes the application is
> done with the temp file and purges it.  So for this issue at least I
> need to either convince my browser to not open the file in the
> background, or tell emacs to cleanup temp files at some later time
> (session termination, for instance).

I didn't have time to think through the consequences, but the following
kludge seems to let the browser start:

--- -	2013-07-29 08:05:32.095764025 -0300
+++ ./mm-decode.el	2013-07-28 16:38:49.000000000 -0300
@@ -972,7 +972,7 @@
 		       (lambda (process state)
 			 (when (eq (process-status process) 'exit)
 			   (condition-case nil
-			       (delete-file file)
+			       ;(delete-file file)
 			     (error))
 			   (condition-case nil
 			       (delete-directory (file-name-directory file))

Re: problems viewing attachments in emacs ui

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

On 07/29/2013 07:07 AM, David Bremner wrote:
> Jameson Graef Rollins <jrollins@finestructure.net> writes:
> 
>> I'm now realizing that my problem with html parts is probably that
>> browser is attempting to open the temporary file in the background.
>> When the browser call returns, the caller assumes the application is
>> done with the temp file and purges it.  So for this issue at least I
>> need to either convince my browser to not open the file in the
>> background, or tell emacs to cleanup temp files at some later time
>> (session termination, for instance).
> 
> I didn't have time to think through the consequences, but the following
> kludge seems to let the browser start:

hm.  there are some pretty serious consequences to feeding arbitrary
html to your web browser via a file:/// URL.  In particular, your
browser might execute arbitrary javascript (which itself can interact
with the rest of your web browsing history and/or logged-in sessions),
and might fetch data from outside sources (leaking at least information
about when and from where you read your e-mail, and potentially other
things).

The consequence of commenting out the file deletion is that you start to
fill up your tmpdir with a bunch of crufty old files :)

One other approach that would address some of these other issues might
be to open the html in a separate, temporary profile for your web
browser, one that doesn't directly interact with your main web browsing
profile (or any other browser profile).  This would also have the
advantage of not terminating until the browser profile is closed.

I know that chromium offers as --temp-profile argument that behaves this
way.  I'm not sure how to do it with iceweasel -- you could do it
manually, with a combination of -ProfileManager and -no-remote, then
create a new profile, and then when done clean it up afterwards
manually, but that sounds like a real pain.

hth,

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 1027 bytes --]

Re: problems viewing attachments in emacs ui

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

On Mon, Jul 29 2013, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> hm.  there are some pretty serious consequences to feeding arbitrary
> html to your web browser via a file:/// URL.  In particular, your
> browser might execute arbitrary javascript (which itself can interact
> with the rest of your web browsing history and/or logged-in sessions),
> and might fetch data from outside sources (leaking at least information
> about when and from where you read your e-mail, and potentially other
> things).

How is this any different than feeding arbitrary html to your browser
via an http:// URL?  In both cases it might execute arbitrary javascript
and interact with the rest of your browsing history.  Obviously it's not
any different.  I would certainly prefer to interact only with
authenticated content, but that's not a reasonable requirement at the
moment.  Until the entire web is https and all emails are signed I will
continue to need to view, with discretion, some unauthenticated html
content.

I should also point out that the notmuch emacs client processes html
parts directly in the emacs session, which of course faces the same
issues.

> One other approach that would address some of these other issues might
> be to open the html in a separate, temporary profile for your web
> browser, one that doesn't directly interact with your main web browsing
> profile (or any other browser profile).  This would also have the
> advantage of not terminating until the browser profile is closed.
>
> I know that chromium offers as --temp-profile argument that behaves this
> way.  I'm not sure how to do it with iceweasel -- you could do it
> manually, with a combination of -ProfileManager and -no-remote, then
> create a new profile, and then when done clean it up afterwards
> manually, but that sounds like a real pain.

This is a somewhat useful solution, but it faces the same problems of
using multiple sessions when browsing the web.

But none of this is really addressing the underlying issue, which is
that temporary files provided to external viewers are being deleted
prematurely when the viewer opens them in the background.  A more
general solution is still desired.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: problems viewing attachments in emacs ui

[David Bremner]

Jameson Graef Rollins <jrollins@finestructure.net> writes:

>
> I should also point out that the notmuch emacs client processes html
> parts directly in the emacs session, which of course faces the same
> issues.

but only html, and not javascript.

d

Re: problems viewing attachments in emacs ui

[Austin Clements]

Quoth Jameson Graef Rollins on Jul 28 at 10:06 am:
> On Sun, Jul 28 2013, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > For instance, if I launch notmuch-show-view-part on an html part, my
> > browser opens pointed at e.g. the following file:
> >
> >   file:///home/jrollins/tmp/emm.610040w/mm.6100F_2.htm
> >
> > But the browser shows the following error:
> >
> >   File not found
> >   Iceweasel can't find the file at /home/jrollins/tmp/emm.610040w/mm.6100F_2.htm.
> 
> I'm now realizing that my problem with html parts is probably that
> browser is attempting to open the temporary file in the background.
> When the browser call returns, the caller assumes the application is
> done with the temp file and purges it.  So for this issue at least I
> need to either convince my browser to not open the file in the
> background, or tell emacs to cleanup temp files at some later time
> (session termination, for instance).

As pointed out by David, the root of this problem is in
mm-display-external in mm-decode.el.  mm-display-external was mostly
rewritten between Emacs 24.1 and Emacs 24.2 (Emacs commit 1354a694).
The Emacs 24.1 implementation would wait for the spawned process to
exit or 30 seconds to elapse, whichever was longer, before deleting
the file (Emacs 23 was much the same, but waited only 2 seconds).
Based on the source comments, this appears to be the *intent* of the
Emacs 24.2 implementation, but what the code actually does is to wait
for whichever of these events happens *first*.  So, if the spawned
process exits immediately (like in your situation), the file will be
deleted immediately, and even if the viewer sticks around, the file
will be deleted after 30 seconds anyway.

This must be affecting Gnus users the same way, but I haven't found
any evidence that they're aware of it.  The mm-display-external code
still has this problem in both the current Emacs master and the
current Gnus master.

Re: problems viewing attachments in emacs ui

[Austin Clements]

Quoth myself on Jul 30 at 10:53 pm:
> Quoth Jameson Graef Rollins on Jul 28 at 10:06 am:
> > On Sun, Jul 28 2013, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > > For instance, if I launch notmuch-show-view-part on an html part, my
> > > browser opens pointed at e.g. the following file:
> > >
> > >   file:///home/jrollins/tmp/emm.610040w/mm.6100F_2.htm
> > >
> > > But the browser shows the following error:
> > >
> > >   File not found
> > >   Iceweasel can't find the file at /home/jrollins/tmp/emm.610040w/mm.6100F_2.htm.
> > 
> > I'm now realizing that my problem with html parts is probably that
> > browser is attempting to open the temporary file in the background.
> > When the browser call returns, the caller assumes the application is
> > done with the temp file and purges it.  So for this issue at least I
> > need to either convince my browser to not open the file in the
> > background, or tell emacs to cleanup temp files at some later time
> > (session termination, for instance).
> 
> As pointed out by David, the root of this problem is in
> mm-display-external in mm-decode.el.  mm-display-external was mostly
> rewritten between Emacs 24.1 and Emacs 24.2 (Emacs commit 1354a694).
> The Emacs 24.1 implementation would wait for the spawned process to
> exit or 30 seconds to elapse, whichever was longer, before deleting
> the file (Emacs 23 was much the same, but waited only 2 seconds).
> Based on the source comments, this appears to be the *intent* of the
> Emacs 24.2 implementation, but what the code actually does is to wait
> for whichever of these events happens *first*.  So, if the spawned
> process exits immediately (like in your situation), the file will be
> deleted immediately, and even if the viewer sticks around, the file
> will be deleted after 30 seconds anyway.
> 
> This must be affecting Gnus users the same way, but I haven't found
> any evidence that they're aware of it.  The mm-display-external code
> still has this problem in both the current Emacs master and the
> current Gnus master.

Try this patch to mm-decode.el.  It should restore the original
behavior and intent of mm-display-external (in a much less gruesome
way than the old 24.1 code).  I simply copied mm-decode.el into my
personal elisp directory (which appears early in my load-path), which
seemed to work just fine for my cursory testing.

diff --git a/lisp/gnus/mm-decode.el b/lisp/gnus/mm-decode.el
index 98be1c5..c11790c 100644
--- a/lisp/gnus/mm-decode.el
+++ b/lisp/gnus/mm-decode.el
@@ -958,26 +958,30 @@ external if displayed external."
 				   shell-command-switch command)
 		    (set-process-sentinel
 		     (get-buffer-process buffer)
-		     (lexical-let ((outbuf outbuf)
-				   (file file)
-				   (buffer buffer)
-				   (command command)
-				   (handle handle))
+		     (lexical-let* ((outbuf outbuf)
+                                    (file file)
+                                    (buffer buffer)
+                                    (command command)
+                                    (handle handle)
+                                    (timer-fired nil)
+                                    (proc-exited nil)
+                                    (maybe-cleanup
+                                     (lambda ()
+                                       (when (and timer-fired proc-exited)
+                                         (ignore-errors
+                                           (delete-file file))
+                                         (ignore-errors
+                                           (delete-directory
+                                            (file-name-directory file)))))))
 		       (run-at-time
 			30.0 nil
 			(lambda ()
-			  (ignore-errors
-			    (delete-file file))
-			  (ignore-errors
-			    (delete-directory (file-name-directory file)))))
+                          (setq timer-fired t)
+                          (funcall maybe-cleanup)))
 		       (lambda (process state)
 			 (when (eq (process-status process) 'exit)
-			   (condition-case nil
-			       (delete-file file)
-			     (error))
-			   (condition-case nil
-			       (delete-directory (file-name-directory file))
-			     (error))
+                           (setq proc-exited t)
+                           (funcall maybe-cleanup)
 			   (when (buffer-live-p outbuf)
 			     (with-current-buffer outbuf
 			       (let ((buffer-read-only nil)

Re: problems viewing attachments in emacs ui

[Austin Clements]

Quoth myself on Jul 30 at 10:53 pm:
> Quoth Jameson Graef Rollins on Jul 28 at 10:06 am:
> > On Sun, Jul 28 2013, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > > For instance, if I launch notmuch-show-view-part on an html part, my
> > > browser opens pointed at e.g. the following file:
> > >
> > >   file:///home/jrollins/tmp/emm.610040w/mm.6100F_2.htm
> > >
> > > But the browser shows the following error:
> > >
> > >   File not found
> > >   Iceweasel can't find the file at /home/jrollins/tmp/emm.610040w/mm.6100F_2.htm.
> > 
> > I'm now realizing that my problem with html parts is probably that
> > browser is attempting to open the temporary file in the background.
> > When the browser call returns, the caller assumes the application is
> > done with the temp file and purges it.  So for this issue at least I
> > need to either convince my browser to not open the file in the
> > background, or tell emacs to cleanup temp files at some later time
> > (session termination, for instance).
> 
> As pointed out by David, the root of this problem is in
> mm-display-external in mm-decode.el.  mm-display-external was mostly
> rewritten between Emacs 24.1 and Emacs 24.2 (Emacs commit 1354a694).
> The Emacs 24.1 implementation would wait for the spawned process to
> exit or 30 seconds to elapse, whichever was longer, before deleting
> the file (Emacs 23 was much the same, but waited only 2 seconds).
> Based on the source comments, this appears to be the *intent* of the
> Emacs 24.2 implementation, but what the code actually does is to wait
> for whichever of these events happens *first*.  So, if the spawned
> process exits immediately (like in your situation), the file will be
> deleted immediately, and even if the viewer sticks around, the file
> will be deleted after 30 seconds anyway.
> 
> This must be affecting Gnus users the same way, but I haven't found
> any evidence that they're aware of it.  The mm-display-external code
> still has this problem in both the current Emacs master and the
> current Gnus master.

I dug into the Gnus repository to find the original source of this
change.  For those who are curious, it's commit dce35658 from Feb. 9,
2012.  Interestingly, the commit message is

    Output text from external commands in the article buffer
    
    * gnus-compat.el: Define `timer-set-function'.
    
    * mm-decode.el (mm-display-external): Output the text from the command
    in the buffer after the command finished.  This makes text-based
    commands behave better.

which says nothing about the complete restructuring of the timer
handling, which further reinforces my suspicion that the behavioral
change was unintentional.