bug#66313: 29.1.50; process-mark sometimes does not yield the expected value

bug#66313: 29.1.50; process-mark sometimes does not yield the expected value

[Markus Triska]

To reproduce this issue, please do the following:

  1) download ediprolog.el with:

     $ wget https://www.metalevel.at/ediprolog/ediprolog.el

  2) download simulans_a.sh with:

     $ wget https://www.metalevel.at/ei/process-mark/simulans_a.sh   

     This shell file will illustrate this issue in lieu of Scryer Prolog.

  3) make simulans_a.sh executable with:

     $ chmod +x simulans_a.sh

  4) create test1.pl and test2.pl, with the following commands:

     $ echo "?- X = a." > test1.pl
     $ echo "?- X = a." > test2.pl

  5) launch Emacs with:

     $ emacs -Q -l ediprolog.el --eval "(add-to-list 'exec-path \".\")" \
         --eval '(setq ediprolog-program "simulans_a.sh")' test1.pl test2.pl

Emacs now shows two windows, test2.pl in the upper window, and
test1.pl in the lower window. Initially, test2.pl is the selected
window, and point is on the query "?- X = a.". Now do:

    M-x ediprolog-dwim RET

As expected, the buffer will now consist of the following text:

    ?- X = a.
    %@    X = a.

Now, switch to the below window (test1.pl) with:

    C-x o

And there, do:

    M-x ediprolog-dwim RET

Unexpectedly, the buffer will consist of the following text:

    ?- X = a.
    %@ 
       X = a.

The newline after %@ is unexpected! The expected buffer content is:

    ?- X = a.
    %@ X = a.

This already illustrates a difference compared to Emacs 26.1, 27.1,
and other versions, which behave as expected.

To see more differences, switch back to the above window (test2.pl):

    C-x o

And move point to the first line (the line of the query) with:

    C-p

And then do:

    M-x ediprolog-dwim RET

Unexpectedly, the buffer will consist of the following text:

    ?- X = a.
    %@ 
    %@    X = a.
       X = a.

The expected text (at this state) is:

    ?- X = a.
    %@    X = a.
    %@    X = a.

Repeat these steps to see more unexpected behaviour. It seems that
process-mark sometimes does not yield the expected position.

ediprolog.el goes to the process-mark in line 431, which reads:

         (goto-char (process-mark ediprolog-process))

In all cases above, the process-mark is expected to yield the value 14
at this point of the program, but it does not always yield this value.

If possible, could you please restore the behaviour that earlier Emacs
versions exhibit in these examples?

Thank you and all the best,
Markus



In GNU Emacs 29.1.50 (build 1, x86_64-apple-darwin18.2.0, X toolkit,
 cairo version 1.17.6, Xaw scroll bars) of 2023-09-14 built on
 mac
Repository revision: f0a89fa1d0e5b380ecadb57052c95b31110c0323
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description:  Mac OS X 10.14.2

Configured using:
 'configure --with-x-toolkit=lucid --without-ns
 CFLAGS=-I/opt/local/include/ LDFLAGS=-L/opt/local/lib
 --with-xpm=ifavailable'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBXML2 MODULES NOTIFY KQUEUE PDUMPER PNG RSVG SQLITE3 THREADS
TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2 LUCID
ZLIB

bug#66313: Acknowledgement (29.1.50; process-mark sometimes does not yield the expected value)

[Markus Triska]

I have reduced this to the following problem: set-process-buffer
sometimes unexpectedly changes the position of process-mark.

For a shorter test case that exhibits the issue, please download
process-mark.el from:

    https://www.metalevel.at/ei/process-mark/process-mark.el

Its contents are:

   (let ((p (start-process "bash" nil "bash")))
     (set-marker (process-mark p) (point))
     (set-process-buffer p (current-buffer))
     (marker-position (process-mark p)))

   (message "hello there!")

Place point after the first form in the file, and evaluate the form with
C-x C-e.

The expected result is: 165. It is obtained with Emacs 26.1 and several
later versions.

However, with the Emacs version I used to report this problem, the
result I get is unexpectedly: 192.

I would greatly appreciate if the previous behaviour could be restored.

Thank you and all the best,
Markus

bug#66313: Acknowledgement (29.1.50; process-mark sometimes does not yield the expected value)

[Eli Zaretskii]

> From: Markus Triska <triska@metalevel.at>
> Date: Sat, 07 Oct 2023 16:15:38 +0200
> 
> 
> I have reduced this to the following problem: set-process-buffer
> sometimes unexpectedly changes the position of process-mark.
> 
> For a shorter test case that exhibits the issue, please download
> process-mark.el from:
> 
>     https://www.metalevel.at/ei/process-mark/process-mark.el
> 
> Its contents are:
> 
>    (let ((p (start-process "bash" nil "bash")))
>      (set-marker (process-mark p) (point))
>      (set-process-buffer p (current-buffer))
>      (marker-position (process-mark p)))
> 
>    (message "hello there!")
> 
> Place point after the first form in the file, and evaluate the form with
> C-x C-e.
> 
> The expected result is: 165. It is obtained with Emacs 26.1 and several
> later versions.
> 
> However, with the Emacs version I used to report this problem, the
> result I get is unexpectedly: 192.
> 
> I would greatly appreciate if the previous behaviour could be restored.

I fixed this on the emacs-29 branch, but please note that your code is
quite problematic: the set-marker call associates the process-mark
marker with the current buffer, but the process's buffer is still nil.
So between the 2nd and the 3rd line of your snippet, you have a window
where the process-mark points to a buffer different from the buffer
associated with the process.  You should do this the other way around:
first set the process-buffer, and then update the process-mark
position to a position in that buffer.

So in a nutshell, your code was hitting "undefined behavior", and
expecting it to be bug-for-bug compatible with old versions of Emacs
was not really reasonable.

bug#66313: Acknowledgement (29.1.50; process-mark sometimes does not yield the expected value)

[Markus Triska]

Eli Zaretskii <eliz@gnu.org> writes:

> I fixed this on the emacs-29 branch

Thank you! I now found the related issue #43573, where Lars states:

   "If this breaks anything (and that is possible -- somebody could be
   using `process-mark' as a weird way to keep track of what the
   previous buffer was -- it should be reverted."

Also, I found commit 7b3e94b6648ed00c6948c09267894b548b2868e7, where the
following (then) new piece of documentation was added to processes.texi:

   if non-@code{nil}, the process mark will be
   set to point to the end of @var{buffer}.

When I wrote ediprolog.el, this was not documented as such.

Is the behaviour now consistent with the documentation, or must the
documentation or behaviour be changed? Should the change introduced in
#43573 be reverted as indicated above, because it broke something?

Thank you and all the best,
Markus

bug#66313: Acknowledgement (29.1.50; process-mark sometimes does not yield the expected value)

[Eli Zaretskii]

> From: Markus Triska <triska@metalevel.at>
> Cc: 66313-done@debbugs.gnu.org
> Date: Sat, 07 Oct 2023 17:28:32 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I fixed this on the emacs-29 branch
> 
> Thank you! I now found the related issue #43573, where Lars states:
> 
>    "If this breaks anything (and that is possible -- somebody could be
>    using `process-mark' as a weird way to keep track of what the
>    previous buffer was -- it should be reverted."
> 
> Also, I found commit 7b3e94b6648ed00c6948c09267894b548b2868e7, where the
> following (then) new piece of documentation was added to processes.texi:
> 
>    if non-@code{nil}, the process mark will be
>    set to point to the end of @var{buffer}.
> 
> When I wrote ediprolog.el, this was not documented as such.
> 
> Is the behaviour now consistent with the documentation, or must the
> documentation or behaviour be changed? Should the change introduced in
> #43573 be reverted as indicated above, because it broke something?

I see no need to revert the patch nor update the documentation.  You
have just bumped into a gray area, where the process mark was not
associated with a buffer, but set-marker (which is unrelated to
processes) just happened to associate the mark with the same buffer
which you later used in set-process-buffer.  This is
borderline-invalid code, but since it was easy to avoid moving the
marker if the marker's buffer is already the same as the one set by
set-process-buffer, I preferred to do that, rather than making any
disruptive changes so much time after that bug#43573 fix was
installed.

bug#66313: Acknowledgement (29.1.50; process-mark sometimes does not yield the expected value)

[Markus Triska]

Eli Zaretskii <eliz@gnu.org> writes:

> I see no need to revert the patch nor update the documentation.

As I see it, the documentation I cited above no longer matches what
Emacs does. This is because the documentation of `set-process-buffer'
I cited states with regards to BUFFER:

     if non-‘nil’, the process mark will be set to point to the end of
     BUFFER.

This now no longer seems to hold as unconditionally as in the way it is
described, because there are now (again) cases where the process mark
will not be set to point to the end of buffer when calling
`set-process-buffer' with a non-nil BUFFER argument. The snippet I
posted earlier shows an example of this situation.

I would greatly appreciate if the documentation could be written in such
a way that it makes clear what we can rely on when using Emacs Lisp.

Thank you and all the best,
Markus