bug#53798: 27.1; OSX: (redisplay) does not reliably redisplay

bug#53798: 27.1; OSX: (redisplay) does not reliably redisplay

[Markus Triska]

To reproduce this issue, please start Emacs with:

    $ emacs -Q

and evaluate the following form:

    (while t
      (insert "\n" (make-string 50 ?a))
      (redisplay)
      (sit-for 0.1))

At first, this works completely as intended: We see a growing number of
lines in the buffer, and we are shown each fresh line as it appears:

    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    ...

However, as soon as any key is pressed, the output becomes erratic in
the sense that for long stretches of time, we see no new lines at all,
and then several of them appear immediately at the same time.

The snippet uses (redisplay) after writing each line in order to show
the line as it appears. I therefore expect to continue to see, also when
a key is pressed, each line immediately after it is written in the
buffer, not batches of multiple lines to be shown after several of them
have already been written. Is there any way to obtain this behaviour?

Thank you a lot!

All the best,
Markus

In GNU Emacs 27.1 (build 1, x86_64-apple-darwin15.3.0, X toolkit, Xaw scroll bars)
 of 2020-12-12 built on mt-macbook
Windowing system distributor 'The X.Org Foundation', version 11.0.11502000
System Description:  Mac OS X 10.11.3

Configured using:
 'configure --prefix=/opt/local --disable-silent-rules --without-ns
 --without-dbus --without-gconf --without-libotf --without-m17n-flt
 --with-gmp --with-gnutls --with-json --with-xml2 --with-modules
 --infodir /opt/local/share/info/emacs --with-x-toolkit=lucid
 --without-xaw3d --without-imagemagick --with-xpm --with-jpeg
 --with-tiff --with-gif --with-png --with-lcms2 --without-rsvg
 --with-xft 'CFLAGS=-pipe -Os -arch x86_64'
 CPPFLAGS=-I/opt/local/include 'LDFLAGS=-L/opt/local/lib
 -Wl,-headerpad_max_install_names -lfreetype -lfontconfig -Wl,-no_pie
 -arch x86_64''

Configured features:
XPM JPEG TIFF GIF PNG GSETTINGS GLIB NOTIFY KQUEUE ACL GNUTLS LIBXML2
FREETYPE HARFBUZZ XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM
MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

bug#53798: 27.1; OSX: (redisplay) does not reliably redisplay

[Eli Zaretskii]

> From: Markus Triska <triska@metalevel.at>
> Date: Sat, 05 Feb 2022 13:42:58 +0100
> 
> 
> To reproduce this issue, please start Emacs with:
> 
>     $ emacs -Q
> 
> and evaluate the following form:
> 
>     (while t
>       (insert "\n" (make-string 50 ?a))
>       (redisplay)
>       (sit-for 0.1))
> 
> At first, this works completely as intended: We see a growing number of
> lines in the buffer, and we are shown each fresh line as it appears:
> 
>     aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>     aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>     aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>     aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>     ...
> 
> However, as soon as any key is pressed, the output becomes erratic in
> the sense that for long stretches of time, we see no new lines at all,
> and then several of them appear immediately at the same time.
> 
> The snippet uses (redisplay) after writing each line in order to show
> the line as it appears. I therefore expect to continue to see, also when
> a key is pressed, each line immediately after it is written in the
> buffer, not batches of multiple lines to be shown after several of them
> have already been written. Is there any way to obtain this behaviour?

sit-for exits immediately if some input is available, and pressing a
key makes input available.  So the loop starts iterating much faster
than before, because sit-for no longer waits for 0.1 sec.  And that is
what you see.

If I modify the snippet as below, it behaves the same no matter
whether you press a key or not.

  (while t
    (insert "\n" (make-string 50 ?a))
    (redisplay)
    (or (sit-for 0.1)
	(read-char)))

bug#53798: 27.1; OSX: (redisplay) does not reliably redisplay

[Markus Triska]

Eli Zaretskii <eliz@gnu.org> writes:

>
> If I modify the snippet as below, it behaves the same no matter
> whether you press a key or not.
>

Thank you a lot for looking into this issue! You are right: For the
snippet I originally posted, I can fix the issue as you outline.

Since the issue I faced in my original program cannot be solved in this
way, I have now constructed a snippet that shows the problem in a way
that apparently cannot be solved in this way, using the forms that
follow this message.

On OSX, if you save these forms in redisplay.el and then invoke Emacs
with:

    $ emacs -Q redisplay.el -f eval-buffer

then the forms will open UnicodeData.txt which ships with Emacs (please
adjust the path if necessary), and then will apply a certain
transformation on each line of the file.

You will see that each line is changed as intended, and we see each
change immediately as it takes effect. However, unexpectedly, when
several keys are pressed quickly in a row, the program apparently
stalls, and no longer performs any actions, for several seconds.

I found that the settings of interprogram-cut-function and
save-interprogram-paste-before-kill are relevant to reproduce this
issue. Could it be that how these settings are treated internally may
interfere with how Emacs redisplays things, and whether it stalls?

To forms to reproduce the issue follow below.

Thank you a lot!
Markus

;; please adapt this as necessary to find UnicodeData.txt
(find-file "~/emacs/admin/unidata/UnicodeData.txt")

(defun transform-line ()
  (insert "code_description(")
  (forward-word)
  (delete-char 1)
  (insert ", \"")
  (search-forward ";")
  (delete-char -1)
  (insert "\").")
  (kill-line)
  (forward-line))

(defun paste-to-osx (text &optional push)
  (let ((process-connection-type nil))
    (let ((proc (start-process "pbcopy" "*Messages*" "pbcopy")))
      (process-send-string proc text)
      (process-send-eof proc))))

(setq interprogram-cut-function 'paste-to-osx)
(setq save-interprogram-paste-before-kill t)

(while t
  (transform-line)
  (redisplay)
  (or (sit-for 0.1)
      (read-char)))

bug#53798: 27.1; OSX: (redisplay) does not reliably redisplay

[Eli Zaretskii]

> From: Markus Triska <triska@metalevel.at>
> Cc: 53798@debbugs.gnu.org
> Date: Sat, 05 Feb 2022 18:29:08 +0100
> 
>     $ emacs -Q redisplay.el -f eval-buffer
> 
> then the forms will open UnicodeData.txt which ships with Emacs (please
> adjust the path if necessary), and then will apply a certain
> transformation on each line of the file.
> 
> You will see that each line is changed as intended, and we see each
> change immediately as it takes effect. However, unexpectedly, when
> several keys are pressed quickly in a row, the program apparently
> stalls, and no longer performs any actions, for several seconds.

If you press several keys, you need to call read-char that number of
time, or empty the input queue in some other way.  Otherwise, you
still have input available after calling read-char, like in the
original recipe, because you pressed more than one key.

I'm not on macOS, so I commented out the part of the recipe that deals
with interprogram-cut-function.  When I run the result, I see no
abnormal behavior, even if I press several keys.

I'm not sure I understand what you are trying to establish with these
recipes.  What is the actual issue you are trying to solve, and why
the call to redisplay/sit-for and pressing keys are part of that?

> I found that the settings of interprogram-cut-function and
> save-interprogram-paste-before-kill are relevant to reproduce this
> issue. Could it be that how these settings are treated internally may
> interfere with how Emacs redisplays things, and whether it stalls?

What do you mean by "how Emacs redisplays things"?  In these recipes,
you actually force Emacs to do redisplay, both by calling 'redisplay'
and by calling 'sit-for', instead of letting it do it "naturally".  So
I wonder where all this is going, and why.

bug#53798: 27.1; OSX: (redisplay) does not reliably redisplay

[Markus Triska]

Eli Zaretskii <eliz@gnu.org> writes:

> If you press several keys, you need to call read-char that number of
> time, or empty the input queue in some other way.  Otherwise, you
> still have input available after calling read-char, like in the
> original recipe, because you pressed more than one key.

I tried to do this by using as the final form instead:

    (while t
      (transform-line)
      (redisplay)
      (while (input-pending-p)
        (read-char))
      (sit-for 0.1))


This reads characters as long as input-pending-p succeeds, in order to
read all characters that are pending. However, I still can reproduce the
issue.

>
> I'm not on macOS, so I commented out the part of the recipe that deals
> with interprogram-cut-function.  When I run the result, I see no
> abnormal behavior, even if I press several keys.

I can reproduce this: When I remove these settings, everything works as
intended. This confirms that these settings are relevant to reproduce
this issue.

> I'm not sure I understand what you are trying to establish with these
> recipes.  What is the actual issue you are trying to solve, and why
> the call to redisplay/sit-for and pressing keys are part of that?

The issue I am facing is the following: I have programmed Emacs to
automate several tasks for me. For instance, in the example I posted,
the task is to convert UnicodeData.txt to a collection of Prolog facts
that can be queried and reasoned about with Scryer Prolog. This is an
example of an actual task I am trying to solve. I noticed that when I
start the automation, and then accidentally press a key, the program
seems to stall instead of proceeding as intended, *even* when I take
provisions, within the program, that aim to counteract the stall. For
example, in the sample snippet I posted, I tried to manually enforce a
redisplay by calling (redisplay), and still I do not see a
redisplay. Instead, I see no display update at all for many iterations.

> What do you mean by "how Emacs redisplays things"?  In these recipes,
> you actually force Emacs to do redisplay, both by calling 'redisplay'
> and by calling 'sit-for', instead of letting it do it "naturally".  So
> I wonder where all this is going, and why.

I am *trying* to force Emacs to redisplay. The issue I am facing is that
the redisplay does *not* happen, even though I am using (redisplay).  In
other words, Emacs seems to be active in the sense that it does perform
the operations I ask of it, but it does not show the result even though
I ask them to be shown.

Does this explanation help? Please let me know if there is anything I
can do to help reproduce this.

Thank you a lot!
Markus

bug#53798: 27.1; OSX: (redisplay) does not reliably redisplay

[Eli Zaretskii]

> From: Markus Triska <triska@metalevel.at>
> Cc: 53798@debbugs.gnu.org
> Date: Sat, 05 Feb 2022 19:24:28 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If you press several keys, you need to call read-char that number of
> > time, or empty the input queue in some other way.  Otherwise, you
> > still have input available after calling read-char, like in the
> > original recipe, because you pressed more than one key.
> 
> I tried to do this by using as the final form instead:
> 
>     (while t
>       (transform-line)
>       (redisplay)
>       (while (input-pending-p)
>         (read-char))
>       (sit-for 0.1))
> 
> 
> This reads characters as long as input-pending-p succeeds, in order to
> read all characters that are pending. However, I still can reproduce the
> issue.

Well, I can't.  So maybe it's macOS specific.

> > I'm not on macOS, so I commented out the part of the recipe that deals
> > with interprogram-cut-function.  When I run the result, I see no
> > abnormal behavior, even if I press several keys.
> 
> I can reproduce this: When I remove these settings, everything works as
> intended. This confirms that these settings are relevant to reproduce
> this issue.

then maybe it really _is_ macOS specific, and I shouldn't speak up
here anymore.

> The issue I am facing is the following: I have programmed Emacs to
> automate several tasks for me. For instance, in the example I posted,
> the task is to convert UnicodeData.txt to a collection of Prolog facts
> that can be queried and reasoned about with Scryer Prolog. This is an
> example of an actual task I am trying to solve. I noticed that when I
> start the automation, and then accidentally press a key, the program
> seems to stall instead of proceeding as intended, *even* when I take
> provisions, within the program, that aim to counteract the stall. For
> example, in the sample snippet I posted, I tried to manually enforce a
> redisplay by calling (redisplay), and still I do not see a
> redisplay. Instead, I see no display update at all for many iterations.

If you want Emacs to do something, why do you care about redisplay and
its rate?  Redisplay cannot affect what Emacs does, it just reflects
the snapshot of the state.

> > What do you mean by "how Emacs redisplays things"?  In these recipes,
> > you actually force Emacs to do redisplay, both by calling 'redisplay'
> > and by calling 'sit-for', instead of letting it do it "naturally".  So
> > I wonder where all this is going, and why.
> 
> I am *trying* to force Emacs to redisplay. The issue I am facing is that
> the redisplay does *not* happen, even though I am using (redisplay).

I don't understand why you care.  Why not leave Emacs to its devices,
so it performs redisplay when it finishes whatever it is that you want
it to do?

> Does this explanation help? Please let me know if there is anything I
> can do to help reproduce this.

I think Alan or someone else who can look into this stuff on macOS
should chime in, because it sounds like its macOS specific.  I also
notice that the redisplay architecture on macOS have changed
dramatically since Emacs 27.1, so maybe the current code base no
longer behaves like what you see.

bug#53798: 27.1; OSX: (redisplay) does not reliably redisplay

[Markus Triska]

Eli Zaretskii <eliz@gnu.org> writes:

> If you want Emacs to do something, why do you care about redisplay and
> its rate?  Redisplay cannot affect what Emacs does, it just reflects
> the snapshot of the state.

In many such cases, I would like to get an expression about how far
along Emacs has already come, and also to see whether the actions I
automated contain obvious errors which would become immediately apparent
from seeing a sequence of intermediary steps which are displayed like a
movie. In addition, also in cases where I do know that everything works
correctly, I would often like to show these steps to others so that they
get an impression about what is going on internally. Seeing only a
stalled Emacs prevents a truly impressive and enjoyable presentation. In
addition, I would like to automate everything reliably in the sense that
it runs in the same way irrespective of what else is going on, such as
whether a key is accidentally pressed, the mouse is moved etc.


> I don't understand why you care.  Why not leave Emacs to its devices,
> so it performs redisplay when it finishes whatever it is that you want
> it to do?

In this concrete case, I looked at the documentation of `redisplay', and
it reads, at the start:

    Perform redisplay.

Since I see no redisplay, I wondered whether the documentation is in
fact correct. This is what led me to reporting this issue, in the hope
that someone cares sufficiently about Emacs to align the functionality
of redisplay with its documentation.

> I think Alan or someone else who can look into this stuff on macOS
> should chime in, because it sounds like its macOS specific.  I also
> notice that the redisplay architecture on macOS have changed
> dramatically since Emacs 27.1, so maybe the current code base no
> longer behaves like what you see.

Thank you a lot! If there are any questions, please let me know.

All the best,
Markus