bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

[Kevin Ryde]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

The way build_annotations() makes `format-annotate-function' use the
same temp buffer " *Format Temp 0*" on each call is not reentrant.  If
TO-FN from the format-alist does anything that writes another buffer
with a buffer-file-format then the buffer which TO-FN is supposed to be
working on is clobbered.

foo.el below illustrates the problem.  Evaluating it writes a /tmp/x
containing

    THIS IS FORMAT ONE
    this is buffer yyy text

where I expected it to be

    THIS IS FORMAT TWO
    hello world

The `message's emitted by my-one-encode and my-two-encode show they get
the same " *Format Temp 0*" buffer.  The latter is what you get in
/tmp/x if taking away the "/tmp/y" blob within `my-two-encode'.

I expect this wouldn't arise often in practice, but it ought to be
reasonably easy to make build_annotations() (or wherever) do better in
the management of temporary buffers.  I expect it's a matter of finding
the right place in the write crunching that temp buffers used can be
killed.



[-- Attachment #2: foo.el --]
[-- Type: application/emacs-lisp, Size: 1955 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1076 bytes --]



In GNU Emacs 23.1.1 (i486-pc-linux-gnu, GTK+ Version 2.16.5)
 of 2009-09-14 on raven, modified by Debian
configured using `configure  '--build=i486-linux-gnu' '--host=i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.1/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/23.1/leim' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=i486-linux-gnu' 'host_alias=i486-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2' 'LDFLAGS=-g' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_AU
  value of $XMODIFIERS: nil
  locale-coding-system: iso-latin-1-unix
  default-enable-multibyte-characters: t

bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

[Andrew Hyatt]

This reproduces on Emacs 25.  But I wonder if this should be a wishlist
instead of a bug.  As the original report notes, decoding a buffer
while encoding another buffer doesn't seem like a normal use-case.

If no one objects in the next few weeks, I can wishlist this.  Making it
a minor-severity bug also seems reasonable to me.

Kevin Ryde <user42@zip.com.au> writes:

> The way build_annotations() makes `format-annotate-function' use the
> same temp buffer " *Format Temp 0*" on each call is not reentrant.  If
> TO-FN from the format-alist does anything that writes another buffer
> with a buffer-file-format then the buffer which TO-FN is supposed to be
> working on is clobbered.
>
> foo.el below illustrates the problem.  Evaluating it writes a /tmp/x
> containing
>
>     THIS IS FORMAT ONE
>     this is buffer yyy text
>
> where I expected it to be
>
>     THIS IS FORMAT TWO
>     hello world
>
> The `message's emitted by my-one-encode and my-two-encode show they get
> the same " *Format Temp 0*" buffer.  The latter is what you get in
> /tmp/x if taking away the "/tmp/y" blob within `my-two-encode'.
>
> I expect this wouldn't arise often in practice, but it ought to be
> reasonably easy to make build_annotations() (or wherever) do better in
> the management of temporary buffers.  I expect it's a matter of finding
> the right place in the write crunching that temp buffers used can be
> killed.
>
>
> (add-to-list 'format-alist '(my-one
>                              "My format."
>                              nil ;; no automatic decode
>                              my-one-decode
>                              my-one-encode
>                              t     ;; encode modifies the region
>                              nil)) ;; write removes from buffer-file-formats
>
> (defun my-one-decode (beg end)
>   end)
>
> (defun my-one-encode (beg end buffer)
>   (message "my-one-encode in %S" (current-buffer))
>
>   (save-excursion
>     (save-restriction
>       (narrow-to-region beg end)
>
>       (insert "THIS IS FORMAT ONE\n")
>
>       (point-max))))
>
>
> (add-to-list 'format-alist '(my-two
>                              "My format."
>                              nil ;; no automatic decode
>                              my-two-decode
>                              my-two-encode
>                              t     ;; encode modifies the region
>                              nil)) ;; write removes from buffer-file-formats
>
> (defun my-two-decode (beg end)
>   (save-excursion
>     (save-restriction
>       (narrow-to-region beg end)
>
>       (goto-char beg)
>       (if (looking-at "THIS IS FORMAT TWO\n")
>           (delete-region (match-beginning 0) (match-end 0)))
>
>       (point-max))))
>
> (defun my-two-encode (beg end buffer)
>   (message "my-two-encode in %S" (current-buffer))
>
>   (save-excursion
>     (save-restriction
>       (narrow-to-region beg end)
>
>       (goto-char (point-min))
>       (insert "THIS IS FORMAT TWO\n")
>
>       (save-current-buffer
>         (find-file "/tmp/y")
>         (format-decode-buffer 'my-one)
>         (erase-buffer)
>         (insert "this is buffer yyy text\n")
>         (save-buffer)
>         (kill-buffer nil))
>
>       (point-max))))
>
>
> (progn
>   (find-file "/tmp/x")
>   (format-decode-buffer 'my-two)
>   (erase-buffer)
>   (insert "hello world\n")
>   (save-buffer)
>   (kill-buffer nil)
>   (find-file "/tmp/x"))
>
>
>
> In GNU Emacs 23.1.1 (i486-pc-linux-gnu, GTK+ Version 2.16.5)
>  of 2009-09-14 on raven, modified by Debian
> configured using `configure  '--build=i486-linux-gnu' '--host=i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.1/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/23.1/leim' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=i486-linux-gnu' 'host_alias=i486-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2' 'LDFLAGS=-g' 'CPPFLAGS=''
>
> Important settings:
>   value of $LC_ALL: nil
>   value of $LC_COLLATE: nil
>   value of $LC_CTYPE: nil
>   value of $LC_MESSAGES: nil
>   value of $LC_MONETARY: nil
>   value of $LC_NUMERIC: nil
>   value of $LC_TIME: nil
>   value of $LANG: en_AU
>   value of $XMODIFIERS: nil
>   locale-coding-system: iso-latin-1-unix
>   default-enable-multibyte-characters: t

bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

[Eli Zaretskii]

> From: Andrew Hyatt <ahyatt@gmail.com>
> Date: Thu, 21 Jul 2016 00:33:17 -0400
> Cc: 5440@debbugs.gnu.org
> 
> This reproduces on Emacs 25.  But I wonder if this should be a wishlist
> instead of a bug.  As the original report notes, decoding a buffer
> while encoding another buffer doesn't seem like a normal use-case.
> 
> If no one objects in the next few weeks, I can wishlist this.  Making it
> a minor-severity bug also seems reasonable to me.

I agree.  I also suggest to amend the documentation to make this issue
more explicitly mentioned.  The ELisp manual already says

 -- Variable: format-alist
     This list contains one format definition for each defined file
     format.            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ^^^^^^

"Only one format definition per format."  But I think it won't do any
harm to specifically warn about violating that.

Thanks.

bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

[Andrew Hyatt]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrew Hyatt <ahyatt@gmail.com>
>> Date: Thu, 21 Jul 2016 00:33:17 -0400
>> Cc: 5440@debbugs.gnu.org
>> 
>> This reproduces on Emacs 25.  But I wonder if this should be a wishlist
>> instead of a bug.  As the original report notes, decoding a buffer
>> while encoding another buffer doesn't seem like a normal use-case.
>> 
>> If no one objects in the next few weeks, I can wishlist this.  Making it
>> a minor-severity bug also seems reasonable to me.
>
> I agree.  I also suggest to amend the documentation to make this issue
> more explicitly mentioned.  The ELisp manual already says
>
>  -- Variable: format-alist
>      This list contains one format definition for each defined file
>      format.            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>      ^^^^^^
>
> "Only one format definition per format."  But I think it won't do any
> harm to specifically warn about violating that.
>
> Thanks.

I think the original bug report had one format definition for each
defined file format, but was manipulating a file in one format while in
another format function.  How about I change the documentation to just
warn not to manipulate other files in FROM-FN here?

bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

[Eli Zaretskii]

> From: Andrew Hyatt <ahyatt@gmail.com>
> Cc: user42@zip.com.au,  5440@debbugs.gnu.org
> Date: Sun, 24 Jul 2016 01:06:05 -0400
> 
> >  -- Variable: format-alist
> >      This list contains one format definition for each defined file
> >      format.            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >      ^^^^^^
> >
> > "Only one format definition per format."  But I think it won't do any
> > harm to specifically warn about violating that.
> >
> > Thanks.
> 
> I think the original bug report had one format definition for each
> defined file format, but was manipulating a file in one format while in
> another format function.

That's more than one format in disguise.

> How about I change the documentation to just warn not to manipulate
> other files in FROM-FN here?

Can you show a proposed patch?

Thanks.

bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

[Andrew Hyatt]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrew Hyatt <ahyatt@gmail.com>
>> Cc: user42@zip.com.au,  5440@debbugs.gnu.org
>> Date: Sun, 24 Jul 2016 01:06:05 -0400
>> 
>> >  -- Variable: format-alist
>> >      This list contains one format definition for each defined file
>> >      format.            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> >      ^^^^^^
>> >
>> > "Only one format definition per format."  But I think it won't do any
>> > harm to specifically warn about violating that.
>> >
>> > Thanks.
>> 
>> I think the original bug report had one format definition for each
>> defined file format, but was manipulating a file in one format while in
>> another format function.
>
> That's more than one format in disguise.
>
>> How about I change the documentation to just warn not to manipulate
>> other files in FROM-FN here?
>
> Can you show a proposed patch?
>
> Thanks.

Attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: format-alist docs patch --]
[-- Type: text/x-patch, Size: 1542 bytes --]

From 18eb654c4541a49f415d4826974fd2c216f2c2db Mon Sep 17 00:00:00 2001
From: Andrew Hyatt <ahyatt@gmail.com>
Date: Sun, 24 Jul 2016 23:58:47 -0400
Subject: [PATCH] Warn against unintentional recursion in formatting.

Change documentation for format-alist to warn against formatting when
formatting, which leads to incorrect results..
---
 doc/lispref/files.texi | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/doc/lispref/files.texi b/doc/lispref/files.texi
index f3650a4..5763380 100644
--- a/doc/lispref/files.texi
+++ b/doc/lispref/files.texi
@@ -3238,7 +3238,9 @@ Format Conversion Round-Trip
 
 One responsibility of @var{from-fn} is to make sure that the beginning
 of the file no longer matches @var{regexp}.  Otherwise it is likely to
-get called again.
+get called again.  Also, @var{from-fn} must not involve other buffers or
+files other than the one being decoded, or else formatting may happen
+during formatting, leading to incorrect results.
 
 @item to-fn
 A shell command or function to encode data in this format---that is, to
@@ -3269,6 +3271,10 @@ Format Conversion Round-Trip
 positions.  All this takes place without modifying the buffer.
 @end itemize
 
+@var{to-fn} must not involve other buffers or files other than the one
+being encoded, or else formatting may happen during formatting,
+leading to incorrect results.
+
 @item modify
 A flag, @code{t} if the encoding function modifies the buffer, and
 @code{nil} if it works by returning a list of annotations.
-- 
2.4.9 (Apple Git-60)

bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

[Eli Zaretskii]

> From: Andrew Hyatt <ahyatt@gmail.com>
> Cc: user42@zip.com.au,  5440@debbugs.gnu.org
> Date: Mon, 25 Jul 2016 00:07:50 -0400
> 
> diff --git a/doc/lispref/files.texi b/doc/lispref/files.texi
> index f3650a4..5763380 100644
> --- a/doc/lispref/files.texi
> +++ b/doc/lispref/files.texi

Thanks.  A couple of comments:

> @@ -3238,7 +3238,9 @@ Format Conversion Round-Trip
>  
>  One responsibility of @var{from-fn} is to make sure that the beginning
>  of the file no longer matches @var{regexp}.  Otherwise it is likely to
> -get called again.
> +get called again.  Also, @var{from-fn} must not involve other buffers or
> +files other than the one being decoded

One of these 2 "others" should be deleted, I think.

>                                         or else formatting may happen
> +during formatting, leading to incorrect results.

I would say something like

  otherwise the internal buffer used for formatting might be
  overwritten.

> +@var{to-fn} must not involve other buffers or files other than the one
> +being encoded, or else formatting may happen during formatting,
> +leading to incorrect results.

Same here.

bug#5440: 23.1; buffer-file-format encoding temp buffers not reentrant

[Andrew Hyatt]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrew Hyatt <ahyatt@gmail.com>
>> Cc: user42@zip.com.au,  5440@debbugs.gnu.org
>> Date: Mon, 25 Jul 2016 00:07:50 -0400
>> 
>> diff --git a/doc/lispref/files.texi b/doc/lispref/files.texi
>> index f3650a4..5763380 100644
>> --- a/doc/lispref/files.texi
>> +++ b/doc/lispref/files.texi
>
> Thanks.  A couple of comments:
>
>> @@ -3238,7 +3238,9 @@ Format Conversion Round-Trip
>>  
>>  One responsibility of @var{from-fn} is to make sure that the beginning
>>  of the file no longer matches @var{regexp}.  Otherwise it is likely to
>> -get called again.
>> +get called again.  Also, @var{from-fn} must not involve other buffers or
>> +files other than the one being decoded
>
> One of these 2 "others" should be deleted, I think.

Done

>
>>                                         or else formatting may happen
>> +during formatting, leading to incorrect results.
>
> I would say something like
>
>   otherwise the internal buffer used for formatting might be
>   overwritten.

Done

>
>> +@var{to-fn} must not involve other buffers or files other than the one
>> +being encoded, or else formatting may happen during formatting,
>> +leading to incorrect results.
>
> Same here.

Done.

Thanks for the suggestions.  I'll submit this and close out the bug.