bug#63988: 30.0.50; Recent header line format changes cause spin/seg fault with format-mode-line

[Eli Zaretskii <eliz@gnu.org>]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: aaronjensen@gmail.com,  Eshel Yaron <me@eshelyaron.com>,
>   63988@debbugs.gnu.org
> Date: Sat, 10 Jun 2023 12:16:45 -0400
> 
> >> It causes infinite recursion, since format-mode-line also calls
> >> window_wants_header_line (indirectly).
> 
> I wonder why `format-mode-line` calls `window_wants_header_line`.
> Is there a deep technical reason, or just an accident of how we
> currently implement the feature?

format-mode-line calls init_iterator (because the formatting is done
by display code), and init_iterator calls window_wants_header_line
because it needs to know the general layout of the window up front.

We could introduce some knobs to perhaps avoid this in the particular
case of format-mode-line, but the :eval form in header-line-format can
call any Lisp, not necessarily format-mode-line.  And the Lisp code
called from :eval could easily call some primitive that uses the
display code, for example vertical-motion or window-text-pixel-size or
something else.  Going over all those places and making sure
init_iterator will not call window_wants_header_line in those cases is
not my idea of fun, even if it is practical to discern between the
cases where it does and doesn't need that.

> > However, I wonder whether we should rethink this minor feature.
> > Perhaps this minor convenience is not worth the complications?
> 
> IIUC This is talking about commit
> 4f66cbbfe520ee31ef26676e09a926217d9736fe which extends the special
> treatment of a nil `header-line-format` to also hide the header line
> when its value is "equivalent" to nil.
> 
> In the past I've wished for a non-nil mode-line that is not displayed
> because it's equivalent to nil, so I think the feature makes sense, but
> I agree it's not super important.  Maybe if we want to make it work
> well, the better solution is to memoize the computation of
> (format-mode-line header-line-format) so that it's called at most once
> per redisplay?

That is possible, and I thought about it as well, but it isn't easy.
Once again: the display code is many times called for purposes other
than display, and in at least some of those cases it does need to know
whether there is a header-line, because the layout and the metrics of
the text on display depend on that.  So the question of when to record
the fact of header-line existence for a particular window -- that
question doesn't have an easy answer.  It is easy to compute that at
the beginning of redisplay_window, but what about the following
scenario in some Lisp program:

  . do some calculation that affects header-line existence
  . call window-text-pixel-size

and its many variations?  Are we going to sprinkle the code with calls
to a function that calculates the header-line and records the result
somewhere (perhaps in a window) before each call to init_iterator and
start_display, and if so, how to avoid potential recursion like the
one in this case?

Or maybe we want to use the variable-watcher feature to do that
whenever header-line-format is modified?  That is also far from being
fun, and could cause performance degradation, as some people, I hear,
use header-line to display the mode-line information.

Bottom line: this sounds like a minor convenience feature that causes
major headaches to implement, because once you allow evaluation of
arbitrary Lisp, all bets are basically off.

Let's admit it: the current Emacs display code was not designed to
support this, so it's a little wonder we face an uphill battle.  And
maybe we should just admit defeat and say we cannot support it until
and unless the display engine is redesigned with this goal in mind.