Gnus for next release

Gnus for next release

[Miles Bader]

I've been talking with the Gnus developers, and it comes up that it would
probably be a good idea to put the current stable release of Gnus into the
next Emacs release.  

Here's what one Gnus maintainer says (just to clarify the version numbering,
Gnus 5.10.x is the branch being suggested for inclusion, but it would be
renamed 5.11 when added to the Emacs tree):

Reiner Steib <reiner.steib@gmx.de> wrote:
> It's even more than one year since the release of Gnus 5.10.1
> (2003-05-01).  After that, only bugfixes went into 5.10.[1-6].  Since
> the release of 5.10.6 (2004-01-04) there was only single commit to the
> v5-10 branch.  (Some post-5.10.6 doc-fixes could be merged from the
> trunk to v5-10).
>
> I also expect that merging Gnus v5-10 into the Emacs trunk won't lead
> to significant problems (or even to a delay of Emacs release).
> Looking at the "User-Agent" headers, you'll notice that many people
> already use Gnus v5.10.6 (or No Gnus) with Emacs 21.3.50.
>
> It would be a pity if Gnus 5.11 would not be included in the next
> Emacs release because it has many improvements over 5.9.

So what about it?  Sounds pretty good and safe to me (and Gnus is largely a
standalone part of emacs).

-Miles
-- 
=====
(^o^;
(()))
*This is the cute octopus virus, please copy it into your sig so it can spread.

Re: Gnus for next release

[John Wiegley]

Miles Bader <miles@gnu.org> writes:

> So what about it?  Sounds pretty good and safe to me (and Gnus is
> largely a standalone part of emacs).

It sounds like a good idea to me.

John

Re: Gnus for next release

[Stefan Monnier]

> I've been talking with the Gnus developers, and it comes up that it would
> probably be a good idea to put the current stable release of Gnus into the
> next Emacs release.  

I always assumed it would be done this way indeed.
But we need to find someone who's going to do the work to make sure that
all the fixes installed in Emacs-CVS's Gnus code are included in the
new code.


        Stefan

Re: Gnus for next release

[Miles Bader]

On Sun, May 09, 2004 at 07:29:31PM -0400, Stefan Monnier wrote:
> But we need to find someone who's going to do the work to make sure that
> all the fixes installed in Emacs-CVS's Gnus code are included in the
> new code.

I will probably do that.

-Miles
-- 
o The existentialist, not having a pillow, goes everywhere with the book by
  Sullivan, _I am going to spit on your graves_.

Re: Gnus for next release

[Frank Schmitt]

Miles Bader <miles@gnu.org> writes:

> On Sun, May 09, 2004 at 07:29:31PM -0400, Stefan Monnier wrote:
>> But we need to find someone who's going to do the work to make sure that
>> all the fixes installed in Emacs-CVS's Gnus code are included in the
>> new code.
>
> I will probably do that.

Way cool.

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: Gnus for next release

[David Kastrup]

Miles Bader <miles@gnu.org> writes:

> I've been talking with the Gnus developers, and it comes up that it
> would probably be a good idea to put the current stable release of
> Gnus into the next Emacs release.

[...]

> So what about it?  Sounds pretty good and safe to me (and Gnus is
> largely a standalone part of emacs).

High time.  It should have been put in at least half a year ago.
While it missed the semi-official feature freeze, this was IMO just
an oversight (didn't think of it myself): it is not that any mad
rushes of last-minute fixes were what was causing this.

I don't think it should cause any trouble, and I was short before
reporting a few problems with message-mode.  But it would be a waste
of time bothering about fixing them in 5.9 AFAICS.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Gnus for next release

[Reiner Steib]

On Mon, May 10 2004, David Kastrup wrote:

> Miles Bader <miles@gnu.org> writes:
>
>> I've been talking with the Gnus developers, and it comes up that it
>> would probably be a good idea to put the current stable release of
>> Gnus into the next Emacs release.
> [...]
>> So what about it?  Sounds pretty good and safe to me (and Gnus is
>> largely a standalone part of emacs).
>
> High time.  It should have been put in at least half a year ago.

In fact we started discussing it in January (see the thread
http://thread.gmane.org/gmane.emacs.gnus.general/55650).  (The
discussion started on the Gnus list.  Later it was Cc-ed to
emacs-devel, too.)

> While it missed the semi-official feature freeze, this was IMO just
> an oversight (didn't think of it myself): it is not that any mad
> rushes of last-minute fixes were what was causing this.

One reason for the Gnus developers' hesitation/delay might have been
the crypto problems which are solved now:
<URL:http://thread.gmane.org/E1BL6CW-0003Ge-8y@fencepost.gnu.org>.
Another reason for the delay were the bootstrap problems I encountered
starting to use Miles arch repositories
(<URL:http://thread.gmane.org/gmane.emacs.devel/21913>; Miles
suggested that merging using arch should be easier.)

I hope that we can merge Gnus 5.10 into the Emacs trunk despite of
being at little bit late.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Gnus for next release

[Richard Stallman]

Of course the updates in Gnus should be installed.
It is a problem that Gnus changes take so long
to get installed in Emacs--they should do it more often.

Re: Gnus for next release

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> Of course the updates in Gnus should be installed.
> It is a problem that Gnus changes take so long
> to get installed in Emacs--they should do it more often.

Agreed.  I remember that when I switched to CVS Emacs, I was surprised
that it did not contain the OORT Gnus everybody was talking about in
the context of Gnus for I don't know how long.  I mean: what better
test bed than CVS Emacs should one desire, once one has a more or less
consistent feature set?  Many qualified testers, and no promise that
it will work.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Possible problem with Gnus

[Richard Stallman]

We have to pay attention to an issue of how Gnus and other Emacs mail
readers treat MIME attachments.

Windows viruses often spread in attachments for Word.  We have to make
sure that attachments don't become a method for spreading viruses in
Emacs.  Some kinds of attachments run applications that perhaps can be
assumed safe, such as a gif displayer.  But attachments that run more
complex attachments, such as a browser that might execute programs
given it, have to be treated as unsafe.

I don't use Gnus.  How does a Gnus user specify to display an
attachment?  Does the user do this for one specific attachment,
or for all the attachments in one message?  Does Gnus ever display
attachments in a message without a specific direct user request
for that message?

Re: Possible problem with Gnus

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> We have to pay attention to an issue of how Gnus and other Emacs mail
> readers treat MIME attachments.
> 
> Windows viruses often spread in attachments for Word.  We have to make
> sure that attachments don't become a method for spreading viruses in
> Emacs.  Some kinds of attachments run applications that perhaps can be
> assumed safe, such as a gif displayer.  But attachments that run more
> complex attachments, such as a browser that might execute programs
> given it, have to be treated as unsafe.
> 
> I don't use Gnus.  How does a Gnus user specify to display an
> attachment?  Does the user do this for one specific attachment,
> or for all the attachments in one message?  Does Gnus ever display
> attachments in a message without a specific direct user request
> for that message?

No, and you have to explicitly ask for display/extraction of each
attachment separately.  The only exception AFAICS are inline image
attachments not exceeding a specific size, and text mode stuff like
rich text.  I don't see any application for an exploit here.  The
worst that can happen is that an invalid image manages to overflow a
decoding buffer in case that there is a bug in the library.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Possible problem with Gnus

[Stefan Monnier]

> We have to pay attention to an issue of how Gnus and other Emacs mail
> readers treat MIME attachments.

Gnus is pretty safe in this respect.

> Does Gnus ever display attachments in a message without a specific direct
> user request for that message?

Only for some specific cases like gif images and text/plain.


        Stefan

Re: Possible problem with Gnus

[Reiner Steib]

[ The following message is a courtesy copy of an article that has
  been posted to news:gmane.emacs.devel as well. ]

On Tue, May 11 2004, Richard Stallman wrote:

> We have to pay attention to an issue of how Gnus and other Emacs mail
> readers treat MIME attachments.
>
> Windows viruses often spread in attachments for Word.  We have to make
> sure that attachments don't become a method for spreading viruses in
> Emacs.  Some kinds of attachments run applications that perhaps can be
> assumed safe, such as a gif displayer.  But attachments that run more
> complex attachments, such as a browser that might execute programs
> given it, have to be treated as unsafe.

I agree with Stefan and David that Gnus is pretty safe in this
respect.

> How does a Gnus user specify to display an attachment?

For types that cannot displayed inline in Emacs, a buttons is created,
e.g. "[4. application/pdf; foo.pdf]".  To display the attachment, the
user has to press RET or mouse-2 on this button.

The viewer used to display the attachment is usually determined by
parsing the mailcap file(s), if present.  Additionally, Gnus has an
internal list of viewers, see `mailcap-mime-data' in `mailcap.el'[1].
Those viewers are designed to be as safe as possible.  Quoting from
the emacs-mime manual[2] (from Gnus 5.10):

  "When you launch an attachment through mailcap an attempt is made to
  use a safe viewer with the safest options--this isn't the case if
  you save it to disk and launch it in a different way (command line
  or double-clicking)."

E.g. xdvi is launched as "xdvi -safer %s".

> Does the user do this for one specific attachment, or for all the
> attachments in one message?

It is customizable based on the MIME type, i.e. different types of
attachment are treated differently.

> Does Gnus ever display attachments in a message without a specific
> direct user request for that message?

By default, only types that can displayed inline in Emacs are
displayed automatically, i.e. without a specific user request.
But the user can also changes this so that in principle, it can
become unsafe (but this risk is also present e.g. if the user sets
`enable-local-eval' to t).

AFAIK, you had a discussion with Florian Weimer about MIME security in
Gnus after your message[3] about "Windows viruses and GNU/Linux" on
gnu.announce.  As a result of discussing this issue on the Gnus
list[4], I have installed a variable `mm-enable-external'[2] in Gnus
5.10.5.  Setting `mm-enable-external' to `nil' disables the use of
external program through MIME completely.  But we decided not to do
this by default because using the programs from mailcap usually is
safer (as explained above and in [2]) as by saving to file and
starting the viewer from the command line.

(A related variable, e.g. for uuencoded messages is
`gnus-article-emulate-mime'[5].)

Bye, Reiner.

[1] (info "(emacs-mime)mailcap")

[2]
,----[ (info "(emacs-mime)Display Customization") ]
| `mm-enable-external'
|      Indicate whether external MIME handlers should be used.
| 
|      If `t', all defined external MIME handlers are used.  If `nil',
|      files are saved to disk (`mailcap-save-binary-file').  If it is
|      the symbol `ask', you are prompted before the external MIME
|      handler is invoked.
| 
|      When you launch an attachment through mailcap (*note mailcap::) an
|      attempt is made to use a safe viewer with the safest options--this
|      isn't the case if you save it to disk and launch it in a different
|      way (command line or double-clicking).  Anyhow, if you want to be
|      sure not to launch any external programs, set this variable to
|      `nil' or `ask'.
`----

[3]
,----[ <news:mailman.944.1061837187.29551.info-gnu@gnu.org> ]
| From: Richard Stallman <rms@gnu.org>
| Subject: Windows viruses and GNU/Linux
| Newsgroups: gnu.announce
| To: info-gnu@gnu.org
| Date: Sun, 24 Aug 2003 23:30:22 -0400
`----

[4] <URL:http://thread.gmane.org/gmane.emacs.gnus.general/54091>

,----[ <news:20030928161139.GA31465@deneb.enyo.de> ]
| From: Florian Weimer <fw@deneb.enyo.de>
| Subject: Disable mailcap support
| Newsgroups: gmane.emacs.gnus.general
| Date: Sun Sep 28 18:11:39 2003 +0200
| Original-To: ding@gnus.org
`----

[5]
,----[ (info "(gnus)MIME Commands") ]
| `gnus-article-emulate-mime'
|      There are other, non-MIME encoding methods used.  The most common
|      is `uuencode', but yEncode is also getting to be popular.  If this
|      variable is non-`nil', Gnus will look in message bodies to see if
|      it finds these encodings, and if so, it'll run them through the
|      Gnus MIME machinery.  The default is `t'.
`----
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Possible problem with Gnus

[Paul Jarc]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> wrote:
> E.g. xdvi is launched as "xdvi -safer %s".

What if the attachment's filename contains characters that would be
dangerous for the shell?  Does Gnus use the filename in the message,
or generate its own?


paul

Re: Possible problem with Gnus

[Stefan Monnier]

> For types that cannot displayed inline in Emacs, a buttons is created,
> e.g. "[4. application/pdf; foo.pdf]".  To display the attachment, the

What about postscript?  Emacs can display postscript inline, supposedly,
and postscript can contain nasty code.


        Stefan

Re: Possible problem with Gnus

[Reiner Steib]

On Tue, May 11 2004, Paul Jarc wrote:

> Reiner Steib <4.uce.03.r.s@nurfuerspam.de> wrote:
>> E.g. xdvi is launched as "xdvi -safer %s".
>
> What if the attachment's filename contains characters that would be
> dangerous for the shell?  Does Gnus use the filename in the message,
> or generate its own?

Gnus creates a new temporary directory[1].  The filename is rewritten
using `mm-file-name-rewrite-functions'[1,2] in order to avoid
dangerous characters.

(Cc-ing Jesper Harder, who improved `mm-file-name-rewrite-functions'
recently.)  I wonder if e.g. »`« and »&« should be removed too.
Jesper, could you explain why those are not deleted in
`mm-file-name-delete-gotchas'?  Testing...  Okay, Gnus already seems
to do proper quoting:

  [2. application/postscript; dan`ls`erous.ps]
  -> »Displaying gv -safer /tmp/ste/emm.11740F6T/dan\`ls\`erous.ps...«.

Bye, Reiner.

[1] See the function `mm-display-external' in `mm-decode.el':

--8<---------------cut here---------------start------------->8---
	(let* ((dir (mm-make-temp-file
		     (expand-file-name "emm." mm-tmp-directory) 'dir))
	       (filename (or
			  (mail-content-type-get
			   (mm-handle-disposition handle) 'filename)
			  (mail-content-type-get
			   (mm-handle-type handle) 'name)))
[...]
	       file buffer)
	  ;; We create a private sub-directory where we store our files.
	  (set-file-modes dir 448)
	  (if filename
	      (setq file (expand-file-name
			  (gnus-map-function mm-file-name-rewrite-functions
					     (file-name-nondirectory filename))
			  dir))
	    (setq file (mm-make-temp-file (expand-file-name "mm." dir))))
--8<---------------cut here---------------end--------------->8---

[2]
,----[ (info "(emacs-mime)Files and Directories") ]
| `mm-file-name-rewrite-functions'
|      A list of functions used for rewriting file names of MIME parts.
|      Each function is applied successively to the file name.
|      Ready-made functions include
| 
|     `mm-file-name-delete-control'
|           Delete all control characters.
| 
|     `mm-file-name-delete-gotchas'
|           Delete characters that could have unintended consequences
|           when used with flawed shell scripts, i.e. `|', `>' and `<';
|           and `-', `.' as the first character.
`----
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Possible problem with Gnus

[Reiner Steib]

On Tue, May 11 2004, Stefan Monnier wrote:

>> For types that cannot displayed inline in Emacs, a buttons is created,
>> e.g. "[4. application/pdf; foo.pdf]".  To display the attachment, the
>
> What about postscript?  Emacs can display postscript inline, supposedly,
> and postscript can contain nasty code.

By default, Gnus displays "application/postscript" using "gv -safer
%s" (and it's done only on user request, see `mm-automatic-display').

I never tried to display PostScript inline in Emacs.  Could you send
me an example?  I didn't manage to trigger Emacs/Gnus into displaying
a postscript attachment inline (even by using "image/postscript",
Content-Disposition: inline).  If Emacs can do it, maybe an addition
to `mm-inline-media-tests' would be necessary.

But even if Gnus would display PostScript inline: Shouldn't Emacs have
an option to disable "deletefile" and "renamefile" and other dangerous
PostScript commands?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Possible problem with Gnus

[David Kastrup]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> I never tried to display PostScript inline in Emacs.  Could you send
> me an example?  I didn't manage to trigger Emacs/Gnus into
> displaying a postscript attachment inline (even by using
> "image/postscript", Content-Disposition: inline).  If Emacs can do
> it, maybe an addition to `mm-inline-media-tests' would be necessary.
> 
> But even if Gnus would display PostScript inline: Shouldn't Emacs
> have an option to disable "deletefile" and "renamefile" and other
> dangerous PostScript commands?

The GhostScript option -dSAFER does this.  But anyway: the current
inline display of PostScript images suffers so many problems that it
would be a royal mistake to enable it by default.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Possible problem with Gnus

[Paul Jarc]

(Reiner, something's up with your Gnus - your message had two MFT
fields.)

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> wrote:
>   [2. application/postscript; dan`ls`erous.ps]
>   -> »Displaying gv -safer /tmp/ste/emm.11740F6T/dan\`ls\`erous.ps...«.

ISTR some systems' mailcap files had quotes around %s, and some
don't.  I don't know whether that's settled down by now, but if it's
still the case, then it would be best to generate a local filename
that doesn't contain any characters that need to be quoted.  A
filename like that will work with either kind of mailcap entry, and
will be safe.  But anyway, since Gnus does the quoting here, we are at
least safe, which answers the original question.


paul

Re: Possible problem with Gnus

[Jesper Harder]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> (Cc-ing Jesper Harder, who improved `mm-file-name-rewrite-functions'
> recently.)  I wonder if e.g. »`« and »&« should be removed too.
> Jesper, could you explain why those are not deleted in
> `mm-file-name-delete-gotchas'?

I think that change wasn't primarily related to interactive viewing of
attachments where it's just saved temporarily -- the file name
shouldn't matter here because Gnus quotes it when the viewer is
called.

It was aimed at the case where you want to save the attachment
permanently somewhere.

-- 
Jesper Harder                                <http://purl.org/harder/>

Re: Possible problem with Gnus

[Jesper Harder]

prj@po.cwru.edu (Paul Jarc) writes:

> ISTR some systems' mailcap files had quotes around %s, and some
> don't.  I don't know whether that's settled down by now, but if it's
> still the case, 

The mailcap standard (RFC 1524) is clear: mailcap files shouldn't have
quotes -- which is why Gnus does the quoting, as you note.

I think you're right, though, that a few systems (Debian, AFAIK) for
unfathomable reasons flatly refuse to follow the standard and ship
broken mailcap files.

-- 
Jesper Harder                                <http://purl.org/harder/>

Re: Possible problem with Gnus

[Reiner Steib]

On Wed, May 12 2004, Paul Jarc wrote:

> (Reiner, something's up with your Gnus - your message had two MFT
> fields.)

Oops, I post to the list via Gmane which converts "Mail-Copies-To:
never" to "Mail-Followup-To: mailing-list-address" (see
http://gmane.org/conv.php).  But it shouldn't do this if there's
already a M-F-T present or I should remove the M-C-T in this case.
Sorry.

> ISTR some systems' mailcap files had quotes around %s, and some
> don't.  I don't know whether that's settled down by now, but if it's
> still the case, then it would be best to generate a local filename
> that doesn't contain any characters that need to be quoted.  

We may (additionally) replace all but character considered to be safe
(say [-_.A-Za-z] or similar) if people really think that this makes
sense.  E.g. »a`b&c$d"e'f.foo« could become »a_b_c_d_e_f.foo«.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Possible problem with Gnus

[Richard Stallman]

    No, and you have to explicitly ask for display/extraction of each
    attachment separately.  The only exception AFAICS are inline image
    attachments not exceeding a specific size, and text mode stuff like
    rich text.

That is good.  I'm reassured that people have been paying attention
to this already.

Re: Possible problem with Gnus

[Richard Stallman]

    The GhostScript option -dSAFER does this.

Does Emacs pass that argument when it invokes GhostScript?

					       But anyway: the current
    inline display of PostScript images suffers so many problems that it
    would be a royal mistake to enable it by default.

If there are bugs in the Postscript support, please start reporting
them one by one.  This is a feature that is supposed to work,
and if it doesn't, we should fix it.

Re: Possible problem with Gnus

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     The GhostScript option -dSAFER does this.
> 
> Does Emacs pass that argument when it invokes GhostScript?
> 
> 					       But anyway: the current
>     inline display of PostScript images suffers so many problems that it
>     would be a royal mistake to enable it by default.
> 
> If there are bugs in the Postscript support, please start reporting
> them one by one.  This is a feature that is supposed to work,
> and if it doesn't, we should fix it.

It is not a feature that is supposed to work.  From the code in
lisp/gs.el:

;;; Commentary:

;; This code is experimental.  Don't use it.

;;; Code:

[...]

  (unwind-protect
      (let ((file (plist-get (cdr spec) :file))
	    gs
 	    (timeout 40))
 	;; Wait while property gets freed from a previous ghostscript process
 	;; sit-for returns nil as soon as input starts being
	;; available, so if we want to give GhostScript a reasonable
 	;; chance of starting up, we better use sleep-for.  We let
 	;; sleep-for wait only half the time because if input is
 	;; available, it is more likely that we don't care that much
 	;; about garbled redisplay and are in a hurry.
 	(while (and
 		;; Wait while the property is not yet available
 		(not (zerop (length (x-window-property "GHOSTVIEW"
 						       frame))))
 		;; The following was an alternative condition: wait
 		;; while there is still a process running.  The idea
 		;; was to avoid contention between processes.  Turned
 		;; out even more sluggish.
 		;; (get-buffer-process "*GS*")
 		(not (zerop timeout)))
 	  (unless (sit-for 0 100 t)
 	    (sleep-for 0 50))
 	  (setq timeout (1- timeout)))

So here we are, in a busy wait loop, hoping that eventually some
GhostScript process will get enough CPU time to release the X resource.

 	;; No use waiting longer.  We might want to try killing off
 	;; stuck processes, but there is no point in doing so: either
 	;; they are stuck for good, in which case the user would
 	;; probably be responsible for that, and killing them off will
 	;; make debugging harder, or they are not.  In that case, they
 	;; will cause incomplete displays.  But the same will happen
 	;; if they are killed, anyway. The whole is rather
 	;; disconcerting, and fast scrolling through a dozen images
 	;; will make Emacs freeze for a while.  The alternatives are a)
 	;; proper implementation not waiting at all but creating
 	;; appropriate queues, or b) permanently bad display due to
 	;; bad cached images.  So remember that this
 	;; is just a hack and if people don't like the behaviour, they
 	;; will most likely like the easy alternatives even less.
 	;; And at least the image cache will make the delay apparent
 	;; just once.

It is clearly documented in the code that the behavior is deficient as
soon as more than single images are concerned.  A separate GhostScript
process gets started for each image that appears on-screen.  Emacs
locks up in busy waiting until the resulting contention of possibly
dozens of GhostScript processes for the respective (GhostView) X
property calms down.  It can't start another process unless the
property has been freed again.  The only platform on which the
postscript device works in the first place is X11.  And there only
marginally.

The sane thing to do is to serialize the whole GhostScript operation
to have at most one GhostScript process running, and to not restart
this process as long as images remain to be rendered.  For this to
work, one has to stop passing the information through an XPixMap but
has to go through a file or pipe.  But file/blocking pipe access
within the display engine is not the hottest idea in the first place.

PostScript image support was the thing that prompted me to create
preview-latex because "it was easy to do so".  The deficiencies in
PostScript support, however, required that even before making the
first public release, preview-latex would have to manage GhostScript
itself to do the rendering.  I reported quite a few problems,
sometimes with patches, before I finally gave up.

There are other considerations making extensive use of PostScript
images a bad idea:

Apart from the lack of sharing resources between images (preview-latex
manages all images from a single LaTeX run in a single GhostScript
session from a single PostScript file it uses intelligently by
interpreting DSC comments like a PostScript previewer does), the
rendering of "postscript" images only starts once they appear
on-screen.  In contrast, preview-latex first deals with on-screen
images.  Once they are dealt with, it reverts to rendering the rest
off-screen.  Since GhostScript converts them all into PNG, the images
actually don't get loaded unless one actually scrolls them on-screen.
Once one does this, however, they are loaded from file.  There is no
necessity for inflating Emacs' memory requirements before such images
get loaded.  preview-latex's way of handling images is convenient, but
it takes up temporary disk space and files.  Managing this disk space
from within the display engine would be awkward, to say the least.

So even if one were to overcome the worst bugs of the postscript image
type implementation, postscript image support would still be bad for
anything but very few images per file.  Scrolling through a buffer
that explodes into rendering at any scroll-up key is not fun.  Been
there, done that.

Perhaps one should warn about this state of affairs in the manual.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Possible problem with Gnus

[Stefan Monnier]

>From your experience it seems the best approach is to do something like
what preview-latex does.  It can't easily use the same insert-image
interface, but maybe preview-latex's code could be made generic.
Anyway... just thinking out loud,


        Stefam

Re: Possible problem with Gnus

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> >From your experience it seems the best approach is to do something
> >like what preview-latex does.

Obviously.  Equally obviously, I am biased.

> >It can't easily use the same insert-image interface, but maybe
> >preview-latex's code could be made generic.

That would certainly be a good thing, all in all.  And what I can
basically offer in that line is to collect FSF copyright assignments:
this process is going to start soon, anyway, since I intend to fold
preview-latex into AUCTeX some time next year or so.  Getting hold of
the contributors to preview-latex will be much easier than with
AUCTeX, anyway.

preview-latex is a complex and specialized beast, however.  It
basically consists of four components:

a) a LaTeX style that hooks itself into selected pieces and part of
the core of LaTeX to suppress all ordinary page generation and
instead generate one page of information for each `interesting' text
passage.  Shortcoming of generality here: would need to get adapted
to plain TeX and ConTeXt for best utility.

b) a process interface for running text passages through TeX or
PDFTeX under control of AUCTeX, dumping formats to speed up matters,
parse the `error messages' from AUCTeX in order to find out where
images need to get placed and so on.  Shortcoming of generality here:
would need to get pretty much rewritten in order to work with Emacs'
default TeX modes.

c) user interface code that will replace images with text for editing
and backwards, open and fold back images, copy stuff and similar.  Not
many shortcomings here: the stuff fits the preview-latex niche very
nicely.  Interfacing into the desktop package to save and restore
images probably fits that niche as well.

d) rendering pipelines that talk to GhostScript, analyze PostScript
files to split them into resources and individual images, run material
through dvips and GhostScript, or through pdf2dsc and GhostScript (in
case PDFLaTeX was used), organize the daemon operation into proper
order (on-screen material first, the rest afterwards), manage
temporary files and stuff.

The intelligence in d) would, after suitable cleanup, pruning and
generalization, probably make a useful Emacs core addition.  For
example, it is perfectly conceivable to have the "calc" calculator
(which has a TeX mode output) use a continuing TeX session (I know how
to force TeX to flush the dvi file) connected with a named pipe for
its dvi output to dvipng (which can also run continuously in a daemon
mode) and have the resulting png files displayed in realtime in an
Emacs calc window.  The performance of those components can easily be
tuned to be workable on a 100MHz Pentium I.

There are quite more applications than just preview-latex itself: for
example Wikipedia is written with HTML tags <math>...</math> that get
run through LaTeX.  It would be convenient to have a "WYSIWYG" editing
mode that would let one see the results before the fact.

> >Anyway... just thinking out loud,

Nothing wrong with that.  Possibilities are endless.  And, as I said,
assignments won't be the problem.  Getting people to do the work is
the problem.

So I am more or less focusing on getting the stuff better for its
limited applicability right now.  This gives results.  It gives
experiences.  If I can't manage to afford the time to do anything
else, it will at least give me the qualification to tell others when
I believe that they are on the wrong track, and why.

Of all the extension projects (supporting Emacs tex-mode in addition
to AUCTeX, support ConTeXt and/or plain TeX, general image pipelines
also for, say, gnuplot and so on), few have come anywhere except great
planning.  I am glad that preview-latex now works with PDFLaTeX as
well, I am glad that dvipng came into being (it was something on my
wishlist, and one of our developers scared me out of my wits by
actually implementing it.  Took almost a year until I had it
integrated satisfactorily) and is integrated well now.  I am glad
that GhostScript interaction has been as streamlined and polished as
possible (and yes, we use GhostScript as safely as possible: getting
any Trojans through preview-latex will be rather difficult).

It needs people with a genuine interest, and the necessary resources
to expand its usefulness beyond its current scope.  This is one of the
reasons that I am planning to fold it into AUCTeX some time next year:
to get it more exposure in those areas where people can already
benefit from it.

Nothing wrong with thinking loud...  and if there are sufficient
resources for more than just thinking, it will be easy for people to
do so.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Possible problem with Gnus

[Richard Stallman]

    > If there are bugs in the Postscript support, please start reporting
    > them one by one.  This is a feature that is supposed to work,
    > and if it doesn't, we should fix it.

    It is not a feature that is supposed to work.  From the code in
    lisp/gs.el:

This makes it somewhat less urgent.  But we still want this feature to
work.  We should start fixing the problems in it, even if we do so one
by one.

To set an example, I will add -dSAFER to gs-options.

      The only platform on which the
    postscript device works in the first place is X11.

That platform is more important than all the rest.
While we do support Windows and MacOS, they should never
weigh more in our decisionmaking than the GNU operating system.

    The sane thing to do is to serialize the whole GhostScript operation
    to have at most one GhostScript process running, and to not restart
    this process as long as images remain to be rendered.

That does sound desirable.  However,

      For this to
    work, one has to stop passing the information through an XPixMap but
    has to go through a file or pipe.

Using a pixmap is preferable, in general.  Why do you think
using a single Ghostscript process is incompatible with using
a pixmap?

      In contrast, preview-latex first deals with on-screen
    images.  Once they are dealt with, it reverts to rendering the rest
    off-screen.

That would be a good optimization to add.

Re: Possible problem with Gnus

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     > If there are bugs in the Postscript support, please start
>     > reporting them one by one.  This is a feature that is supposed
>     > to work, and if it doesn't, we should fix it.
> 
>     It is not a feature that is supposed to work.  From the code in
>     lisp/gs.el:
> 
> This makes it somewhat less urgent.  But we still want this feature
> to work.  We should start fixing the problems in it, even if we do
> so one by one.
> 
> To set an example, I will add -dSAFER to gs-options.
> 
>       The only platform on which the postscript device works in the
>     first place is X11.
> 
> That platform is more important than all the rest.
> While we do support Windows and MacOS, they should never
> weigh more in our decisionmaking than the GNU operating system.
> 
>     The sane thing to do is to serialize the whole GhostScript
>     operation to have at most one GhostScript process running, and
>     to not restart this process as long as images remain to be
>     rendered.
> 
> That does sound desirable.  However,
> 
>       For this to work, one has to stop passing the information
>     through an XPixMap but has to go through a file or pipe.
> 
> Using a pixmap is preferable, in general.  Why do you think
> using a single Ghostscript process is incompatible with using
> a pixmap?

Because the interface to GhostScript that is used for passing the
XPixMap Id and the respective sizes is queried just at the start of
GhostScript.  You can't change the size after GhostScript has
started, so if you want to use one GhostScript process, it may only
connect to a single XPixMap of fixed size.

At least that's what I remember from last taking a look at the matter.

>       In contrast, preview-latex first deals with on-screen images.
>     Once they are dealt with, it reverts to rendering the rest
>     off-screen.
> 
> That would be a good optimization to add.

Rendering off-screen material is actually not as much an optimization,
but an interactivity feature.  It means that once GhostScript is
through, scrolling through the file is not computationally expensive.

However, if some document has thousands of images, it would be saner
to render them just to disk in case you'll need them, but not burden
Emacs' memory with them unless one actually moves there.

If one wanted to fold a sizable chunk of the preview-latex concept
into the display engine, then the display engine would need to

a) be able to deal with general PostScript (multi-page) instead of
isolated EPS files
b) know which images belong to the same PostScript file, so that it
will render them together, reading the PostScript resources of the
file only once, like a PS previewer.

The off-screen rendering/caching might be restricted to just images
from a single PostScript file: savings are largest in that case.

But investing serious work of that kind for the rather special case
here would probably not be worth it unless one expanded the mechanism
to be able to deal with more general conversion processes, like
dvi->ps->png or gnuplot->png or stuff like that.  If one would really
want to immerse oneself into stuff like that, one would probably want
some more general image rendering/embedding mechanism.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Possible problem with Gnus

[Richard Stallman]

Andy, we have been talking about problems in Emacs code
to display postscript in the middle of a document.

    >     The sane thing to do is to serialize the whole GhostScript
    >     operation to have at most one GhostScript process running, and
    >     to not restart this process as long as images remain to be
    >     rendered.
    > 
    > That does sound desirable.  However,
    > 
    >       For this to work, one has to stop passing the information
    >     through an XPixMap but has to go through a file or pipe.
    > 
    > Using a pixmap is preferable, in general.  Why do you think
    > using a single Ghostscript process is incompatible with using
    > a pixmap?

    Because the interface to GhostScript that is used for passing the
    XPixMap Id and the respective sizes is queried just at the start of
    GhostScript.

I think the solution for this is to make a new interface
to allow Emacs to specify the pixmap to an existing GhostScript
process when reusing it for another image.

Andy, can you implement such an interface for Emacs to use?

    >       In contrast, preview-latex first deals with on-screen images.
    >     Once they are dealt with, it reverts to rendering the rest
    >     off-screen.
    > 
    > That would be a good optimization to add.

    Rendering off-screen material is actually not as much an optimization,
    but an interactivity feature.  It means that once GhostScript is
    through, scrolling through the file is not computationally expensive.

    However, if some document has thousands of images, it would be saner
    to render them just to disk in case you'll need them, but not burden
    Emacs' memory with them unless one actually moves there.

That makes sense.  Can that be done easily with reuse of a
single GhostScript process, with the existing GhostScript features?
If not, what new feature do we need?

Re: Possible problem with Gnus

[Andy Tai]

Hi, I would be glad to develop such an interfce; I
believe ghostscript should have most infrasturcture in
place already.  I cannot promise on a time fr
Thanks,

Andy 
--- Richard Stallman <rms@gnu.org> wrote:
> Andy, we have been talking about problems in Emacs
> code
> to display postscript in the middle of a document.
> 
>     >     The sane thing to do is to serialize the
> whole GhostScript
>     >     operation to have at most one GhostScript
> process running, and
>     >     to not restart this process as long as
> images remain to be
>     >     rendered.
>     > 
>     > That does sound desirable.  However,
>     > 
>     >       For this to work, one has to stop
> passing the information
>     >     through an XPixMap but has to go through a
> file or pipe.
>     > 
>     > Using a pixmap is preferable, in general.  Why
> do you think
>     > using a single Ghostscript process is
> incompatible with using
>     > a pixmap?
> 
>     Because the interface to GhostScript that is
> used for passing the
>     XPixMap Id and the respective sizes is queried
> just at the start of
>     GhostScript.
> 
> I think the solution for this is to make a new
> interface
> to allow Emacs to specify the pixmap to an existing
> GhostScript
> process when reusing it for another image.
> 
> Andy, can you implement such an interface for Emacs
> to use?
> 
>     >       In contrast, preview-latex first deals
> with on-screen images.
>     >     Once they are dealt with, it reverts to
> rendering the rest
>     >     off-screen.
>     > 
>     > That would be a good optimization to add.
> 
>     Rendering off-screen material is actually not as
> much an optimization,
>     but an interactivity feature.  It means that
> once GhostScript is
>     through, scrolling through the file is not
> computationally expensive.
> 
>     However, if some document has thousands of
> images, it would be saner
>     to render them just to disk in case you'll need
> them, but not burden
>     Emacs' memory with them unless one actually
> moves there.
> 
> That makes sense.  Can that be done easily with
> reuse of a
> single GhostScript process, with the existing
> GhostScript features?
> If not, what new feature do we need?


=====
Andy Tai, atai@atai.org
Free Software: the software by the people, of the people and for the people! Develop! Share! Enhance! Enjoy!

Re: Possible problem with Gnus

[Andy Tai]

Hi, sorry the previous mail was sent prematurely.  I
meant I can look into implementing this interface, but
I cannot promise at a time frame right now.

Andy

--- Richard Stallman <rms@gnu.org> wrote:
> Andy, we have been talking about problems in Emacs
> code
> to display postscript in the middle of a document.
> 
>     >     The sane thing to do is to serialize the
> whole GhostScript
>     >     operation to have at most one GhostScript
> process running, and
>     >     to not restart this process as long as
> images remain to be
>     >     rendered.
>     > 
>     > That does sound desirable.  However,
>     > 
>     >       For this to work, one has to stop
> passing the information
>     >     through an XPixMap but has to go through a
> file or pipe.
>     > 
>     > Using a pixmap is preferable, in general.  Why
> do you think
>     > using a single Ghostscript process is
> incompatible with using
>     > a pixmap?
> 
>     Because the interface to GhostScript that is
> used for passing the
>     XPixMap Id and the respective sizes is queried
> just at the start of
>     GhostScript.
> 
> I think the solution for this is to make a new
> interface
> to allow Emacs to specify the pixmap to an existing
> GhostScript
> process when reusing it for another image.
> 
> Andy, can you implement such an interface for Emacs
> to use?
> 
>     >       In contrast, preview-latex first deals
> with on-screen images.
>     >     Once they are dealt with, it reverts to
> rendering the rest
>     >     off-screen.
>     > 
>     > That would be a good optimization to add.
> 
>     Rendering off-screen material is actually not as
> much an optimization,
>     but an interactivity feature.  It means that
> once GhostScript is
>     through, scrolling through the file is not
> computationally expensive.
> 
>     However, if some document has thousands of
> images, it would be saner
>     to render them just to disk in case you'll need
> them, but not burden
>     Emacs' memory with them unless one actually
> moves there.
> 
> That makes sense.  Can that be done easily with
> reuse of a
> single GhostScript process, with the existing
> GhostScript features?
> If not, what new feature do we need?