bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Fredrik Fornwall]

Starting up a CANNOT dump build of emacs 25.0.93 in tty mode while
resizing the terminal often fails. May be related to
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22975.

The error message I got when the startup fails varies (perhaps due to
timing issues) between:

"Symbol?s function definition is void: internal-echo-keystrokes-prefix"
and
"NSTATICS too small; try increasing and recompiling Emacs"

Steps to reproduce:

wget ftp://alpha.gnu.org/gnu/emacs/pretest/emacs-25.0.93.tar.xz
tar xf emacs-25.0.93.tar.xz
cd emacs-25.0.93/
CANNOT_DUMP=yes ./configure --prefix=$HOME/emacs-installation
--without-all --with-x-toolkit=no
make install
~/emacs-installation/bin/emacs
# Resize the terminal continuously during startup (may be difficult to
be quick enough,
# one approach is to run "sleep 2; ~/emacs-installation/bin/emacs" and
start resizing
# directly already while sleeping).

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Eli Zaretskii]

> Date: Mon, 25 Apr 2016 09:10:41 +0200
> From: Fredrik Fornwall <fredrik@fornwall.net>
> 
> Starting up a CANNOT dump build of emacs 25.0.93 in tty mode while
> resizing the terminal often fails. May be related to
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22975.

That bug was fixed.

Can you tell why would one want to resize the terminal during startup?
In general, during that time, a CANNOT_DUMP build is not fully
functional, so doing something like that is not recommended, I frankly
I cannot see any reasons for that.  What am I missing?

Thanks.

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Fredrik Fornwall]

On 25 April 2016 at 09:54, Eli Zaretskii <eliz@gnu.org> wrote:
> Can you tell why would one want to resize the terminal during startup?
> In general, during that time, a CANNOT_DUMP build is not fully
> functional, so doing something like that is not recommended, I frankly
> I cannot see any reasons for that.  What am I missing?

Some reasons causing terminal resize during startup:
(1) If running emacs on Android
(http://endlessparentheses.com/running-emacs-on-android.html), you may
want to show the (or switch to another) touch keyboard, which causes a
terminal resize. Android also forces CANNOT_DUMP since binaries needs
to be position-independent, and startup is also slower due to running
on mobile devices, so the window of opportunity increases.
(2) One wants to resize the terminal to e.g. put the terminal window
with emacs besides another window, or maximize the window.
(3) One wants to change font size when jumping from the terminal into emacs.

If one knows that emacs crashes if resizing during startup it's easy
to wait for startup to complete before doing any of the above.

The problem is that most new users won't know that, and a message
like"function definition is void: internal-echo-keystrokes-prefix"
won't help them understand it either, and they will instead need
support (or give up using emacs).

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Eli Zaretskii]

> Date: Mon, 25 Apr 2016 11:43:02 +0200
> From: Fredrik Fornwall <fredrik@fornwall.net>
> Cc: 23369@debbugs.gnu.org
> 
> Some reasons causing terminal resize during startup:
> (1) If running emacs on Android
> (http://endlessparentheses.com/running-emacs-on-android.html), you may
> want to show the (or switch to another) touch keyboard, which causes a
> terminal resize. Android also forces CANNOT_DUMP since binaries needs
> to be position-independent, and startup is also slower due to running
> on mobile devices, so the window of opportunity increases.
> (2) One wants to resize the terminal to e.g. put the terminal window
> with emacs besides another window, or maximize the window.
> (3) One wants to change font size when jumping from the terminal into emacs.
> 
> If one knows that emacs crashes if resizing during startup it's easy
> to wait for startup to complete before doing any of the above.
> 
> The problem is that most new users won't know that, and a message
> like"function definition is void: internal-echo-keystrokes-prefix"
> won't help them understand it either, and they will instead need
> support (or give up using emacs).

Well, if someone could provide a backtrace showing how these errors
are triggered by resizing the frame, perhaps this could be fixed.

Thanks.

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Glenn Morris]

I found this easy to reproduce but hard to debug, but it seems like
making window.el's internal--before-save-selected-window check
(featurep 'frame) before it tries to call frames-on-display-list
fixes it for me.

(Prescient comment in http://debbugs.gnu.org/22975#73 ?)

*** a/lisp/window.el
--- b/lisp/window.el
***************
*** 30,35 ****
--- 30,36 ----
  
  (defun internal--before-save-selected-window ()
    (cons (selected-window)
+         (if (featurep 'frame)
              ;; We save and restore all frames' selected windows, because
              ;; `select-window' can change the frame-selected-window of
              ;; whatever frame that window is in.  Each text terminal's
***************
*** 47,53 ****
                               (push (cons f (frame-selected-window f))
                                     alist))
                             alist))
!                        (terminal-list)))))
  
  (defun internal--after-save-selected-window (state)
    (dolist (elt (cdr state))
--- 48,54 ----
                                   (push (cons f (frame-selected-window f))
                                         alist))
                                 alist))
!                            (terminal-list))))))
  
  (defun internal--after-save-selected-window (state)
    (dolist (elt (cdr state))

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Fredrik Fornwall <fredrik@fornwall.net>,  23369@debbugs.gnu.org
> Date: Tue, 26 Apr 2016 20:53:28 -0400
> 
> I found this easy to reproduce but hard to debug, but it seems like
> making window.el's internal--before-save-selected-window check
> (featurep 'frame) before it tries to call frames-on-display-list
> fixes it for me.

Thanks.

Fredrik, does this patch solve the problems you see?  If so, I think
we should install it on the emacs-25 branch.

> (Prescient comment in http://debbugs.gnu.org/22975#73 ?)

I disagree with the conclusion there.  I think what we need is to
detect all the cases like this one and fix them one by one.  Bitrot in
areas and use cases that are rarely used is nothing new in Emacs, so
this stuff will happen from time to time, there's no way around that.
But these cases cannot be too many in this case, since not much can be
going on while 'loadup' runs anyway.  We just found one more, possibly
the last, of those dark corners.

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Glenn Morris]

Eli Zaretskii wrote:

> Fredrik, does this patch solve the problems you see? 

Remember to rebuild window.elc after applying.

> If so, I think we should install it on the emacs-25 branch.

It's ugly to (essentially) modify all uses of save-selected-window just
because of one (?) problematic use in an edge case. I wonder if we can
find and fix whatever code is calling it in this case. Presumably
something window-related that happens during resizing.

Also, it seems like if CANNOT_DUMP builds encounter an error loading
loadup.el, they go on to run the command-loop with only a partially
loaded loadup, which leads to confusing error messages that have nothing
to do with the real problem. Can they be made to abort instead?

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: fredrik@fornwall.net,  23369@debbugs.gnu.org
> Date: Wed, 27 Apr 2016 12:36:33 -0400
> 
> It's ugly to (essentially) modify all uses of save-selected-window just
> because of one (?) problematic use in an edge case. I wonder if we can
> find and fix whatever code is calling it in this case. Presumably
> something window-related that happens during resizing.

I'd still like to see a full backtrace when the problem happens, both
C and Lisp level.

> Also, it seems like if CANNOT_DUMP builds encounter an error loading
> loadup.el, they go on to run the command-loop with only a partially
> loaded loadup, which leads to confusing error messages that have nothing
> to do with the real problem. Can they be made to abort instead?

How will aborting help diagnosing the problems?  IME, it tends to hide
evidence, not make it more clear.

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Glenn Morris]

Eli Zaretskii wrote:

> I'd still like to see a full backtrace when the problem happens, both
> C and Lisp level.

I can't get one. I just get "NSTATICS too small" (even if I make it huge).

>> Also, it seems like if CANNOT_DUMP builds encounter an error loading
>> loadup.el, they go on to run the command-loop with only a partially
>> loaded loadup, which leads to confusing error messages that have nothing
>> to do with the real problem. Can they be made to abort instead?
>
> How will aborting help diagnosing the problems?  IME, it tends to hide
> evidence, not make it more clear.

In that Emacs might stop near the actual error (eg "loading foo..."),
rather than going on to fail with some totally unrelated error (eg
"Symbol's function definition is void: internal-echo-keystrokes-prefix"
as in this case).

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: fredrik@fornwall.net,  23369@debbugs.gnu.org
> Date: Wed, 27 Apr 2016 14:48:41 -0400
> 
> Eli Zaretskii wrote:
> 
> > I'd still like to see a full backtrace when the problem happens, both
> > C and Lisp level.
> 
> I can't get one. I just get "NSTATICS too small" (even if I make it huge).

If you run that command under GDB, with a breakpoint on the code which
issues this message, don't you get the backtrace?

> >> Also, it seems like if CANNOT_DUMP builds encounter an error loading
> >> loadup.el, they go on to run the command-loop with only a partially
> >> loaded loadup, which leads to confusing error messages that have nothing
> >> to do with the real problem. Can they be made to abort instead?
> >
> > How will aborting help diagnosing the problems?  IME, it tends to hide
> > evidence, not make it more clear.
> 
> In that Emacs might stop near the actual error (eg "loading foo..."),
> rather than going on to fail with some totally unrelated error (eg
> "Symbol's function definition is void: internal-echo-keystrokes-prefix"
> as in this case).

But the error message is actually quite informative; at the very
least, it tells where the problem happened, and frequently also names
the guilty function or variable.

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Glenn Morris]

Eli Zaretskii wrote:

>> I can't get one. I just get "NSTATICS too small" (even if I make it huge).
>
> If you run that command under GDB, with a breakpoint on the code which
> issues this message, don't you get the backtrace?

It's not a relevant message.

Anyway, I managed to get a useful backtrace, which revealed the real
problem, and a trivial fix:

Lisp Backtrace:
  "frames-on-display-list" (0xffff17e0)
  "let" (0xffff19d0)
  "mapcar" (0xffff1d50)
  "apply" (0xffff1ed0)
  "cons" (0xffff1ff0)
  "internal--before-save-selected-window" (0xffff2160)
  "let" (0xffff2450)
  "save-selected-window" (0xffff2570)
  "walk-windows" (0xffff26e0)
  "let" (0xffff2a20)
  "window--process-window-list" (0xffff2b90)
  "let" (0xffff2e80)
  "dolist" (0xffff2fa0)
  "window--adjust-process-windows" (0xffff31e0)
  "load" (0xffff3800)
 
*** a/lisp/window.el
--- b/lisp/window.el
***************
*** 8520,8525 ****
--- 8520,8526 ----
  displaying that processes's buffer."
    (let ((processes (process-list))
          (process-windows nil))
+     (when processes
        (walk-windows
         (lambda (window)
           (let ((buffer (window-buffer window))
***************
*** 8538,8544 ****
                          nil)
                      (setf iter (cdr iter)))))))
       1 t)
!     process-windows))
  
  (defun window--adjust-process-windows ()
    "Update process window sizes to match the current window configuration."
--- 8539,8545 ----
                            nil)
                        (setf iter (cdr iter)))))))
         1 t)
!       process-windows)))
  
  (defun window--adjust-process-windows ()
    "Update process window sizes to match the current window configuration."


>> >> Also, it seems like if CANNOT_DUMP builds encounter an error loading
>> >> loadup.el, they go on to run the command-loop with only a partially
>> >> loaded loadup, which leads to confusing error messages that have nothing
>> >> to do with the real problem. Can they be made to abort instead?
>> >
>> > How will aborting help diagnosing the problems?  IME, it tends to hide
>> > evidence, not make it more clear.
>> 
>> In that Emacs might stop near the actual error (eg "loading foo..."),
>> rather than going on to fail with some totally unrelated error (eg
>> "Symbol's function definition is void: internal-echo-keystrokes-prefix"
>> as in this case).
>
> But the error message is actually quite informative; at the very
> least, it tells where the problem happened, and frequently also names
> the guilty function or variable.

I feel like we are miscommunicating.

Neither of the errors mentioned in the original report (NSTATICS,
internal-echo-keystrokes-prefix) have anything to do with the real
problem. They are just noise caused by an Emacs with half its loadup
missing falling through to the command-loop and inevitably failing
miserably. This is not informative. If there is an error loading loadup,
it should print it and abort. (I see I am repeating
http://debbugs.gnu.org/22975#38 ).

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: fredrik@fornwall.net,  23369@debbugs.gnu.org
> Date: Wed, 27 Apr 2016 17:01:13 -0400
> 
> Anyway, I managed to get a useful backtrace, which revealed the real
> problem, and a trivial fix:
> 
> Lisp Backtrace:
>   "frames-on-display-list" (0xffff17e0)
>   "let" (0xffff19d0)
>   "mapcar" (0xffff1d50)
>   "apply" (0xffff1ed0)
>   "cons" (0xffff1ff0)
>   "internal--before-save-selected-window" (0xffff2160)
>   "let" (0xffff2450)
>   "save-selected-window" (0xffff2570)
>   "walk-windows" (0xffff26e0)
>   "let" (0xffff2a20)
>   "window--process-window-list" (0xffff2b90)
>   "let" (0xffff2e80)
>   "dolist" (0xffff2fa0)
>   "window--adjust-process-windows" (0xffff31e0)
>   "load" (0xffff3800)
>  
> *** a/lisp/window.el
> --- b/lisp/window.el
> ***************
> *** 8520,8525 ****
> --- 8520,8526 ----
>   displaying that processes's buffer."
>     (let ((processes (process-list))
>           (process-windows nil))
> +     (when processes
>         (walk-windows
>          (lambda (window)
>            (let ((buffer (window-buffer window))
> ***************
> *** 8538,8544 ****
>                           nil)
>                       (setf iter (cdr iter)))))))
>        1 t)
> !     process-windows))
>   
>   (defun window--adjust-process-windows ()
>     "Update process window sizes to match the current window configuration."
> --- 8539,8545 ----
>                             nil)
>                         (setf iter (cdr iter)))))))
>          1 t)
> !       process-windows)))
>   
>   (defun window--adjust-process-windows ()
>     "Update process window sizes to match the current window configuration."
> 

Thanks.

> > But the error message is actually quite informative; at the very
> > least, it tells where the problem happened, and frequently also names
> > the guilty function or variable.
> 
> I feel like we are miscommunicating.
> 
> Neither of the errors mentioned in the original report (NSTATICS,
> internal-echo-keystrokes-prefix) have anything to do with the real
> problem. They are just noise caused by an Emacs with half its loadup
> missing falling through to the command-loop and inevitably failing
> miserably. This is not informative. If there is an error loading loadup,
> it should print it and abort. (I see I am repeating
> http://debbugs.gnu.org/22975#38 ).

We have different experiences.  Those messages did help me when I was
debugging similar problems.  (I see I am repeating
http://debbugs.gnu.org/22975#44 ).

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Fredrik Fornwall]

On 27 April 2016 at 23:01, Glenn Morris <rgm@gnu.org> wrote:
> Anyway, I managed to get a useful backtrace, which revealed the real
> problem, and a trivial fix:
>
> Lisp Backtrace:
>   "frames-on-display-list" (0xffff17e0)
>   "let" (0xffff19d0)
>   "mapcar" (0xffff1d50)
>   "apply" (0xffff1ed0)
>   "cons" (0xffff1ff0)
>   "internal--before-save-selected-window" (0xffff2160)
>   "let" (0xffff2450)
>   "save-selected-window" (0xffff2570)
>   "walk-windows" (0xffff26e0)
>   "let" (0xffff2a20)
>   "window--process-window-list" (0xffff2b90)
>   "let" (0xffff2e80)
>   "dolist" (0xffff2fa0)
>   "window--adjust-process-windows" (0xffff31e0)
>   "load" (0xffff3800)
>
> *** a/lisp/window.el
> --- b/lisp/window.el
> ***************
> *** 8520,8525 ****
> --- 8520,8526 ----
>   displaying that processes's buffer."
>     (let ((processes (process-list))
>           (process-windows nil))
> +     (when processes
>         (walk-windows
>          (lambda (window)
>            (let ((buffer (window-buffer window))
> ***************
> *** 8538,8544 ****
>                           nil)
>                       (setf iter (cdr iter)))))))
>        1 t)
> !     process-windows))
>
>   (defun window--adjust-process-windows ()
>     "Update process window sizes to match the current window configuration."
> --- 8539,8545 ----
>                             nil)
>                         (setf iter (cdr iter)))))))
>          1 t)
> !       process-windows)))
>
>   (defun window--adjust-process-windows ()
>     "Update process window sizes to match the current window configuration."

I applied this patch (which replaces the earlier "(if (featurep
'frame)" one, right?) and can verify that resizing during startup now
works for me. Thanks!

bug#23369: 25.0.93; CANNOT_DUMP build fails if resizing terminal during startup in tty mode

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: fredrik@fornwall.net,  23369@debbugs.gnu.org
> Date: Wed, 27 Apr 2016 17:01:13 -0400
> 
> *** a/lisp/window.el
> --- b/lisp/window.el
> ***************
> *** 8520,8525 ****
> --- 8520,8526 ----
>   displaying that processes's buffer."
>     (let ((processes (process-list))
>           (process-windows nil))
> +     (when processes
>         (walk-windows
>          (lambda (window)
>            (let ((buffer (window-buffer window))
> ***************
> *** 8538,8544 ****
>                           nil)
>                       (setf iter (cdr iter)))))))
>        1 t)
> !     process-windows))
>   
>   (defun window--adjust-process-windows ()
>     "Update process window sizes to match the current window configuration."
> --- 8539,8545 ----
>                             nil)
>                         (setf iter (cdr iter)))))))
>          1 t)
> !       process-windows)))
>   
>   (defun window--adjust-process-windows ()
>     "Update process window sizes to match the current window configuration."

Btw, the change which introduced these calls (made, probably
inadvertently, at startup as well, something that doesn't seem to make
sense to me) is completely devoid of any documentation: the feature is
not mentioned in NEWS and not in any manual, even though it includes a
defcustom.