bug#39733: 26.3; Unclear direct color mode documentation can cause loading delays

bug#39733: 26.3; Unclear direct color mode documentation can cause loading delays

[Daniel Long Sockwell]

The Emacs manual describes the process for enabling direct color mode
for terminal Emacs with the following:

> Emacs can be invoked with a custom definition as shown below
>>     $ cat terminfo-24bit.src
>>     
>>     # Use colon separators.
>>     xterm-24bit|xterm with 24-bit direct color mode,
>>       use=xterm-256color,
>>       setb24=\E[48:2:%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%dm,
>>       setf24=\E[38:2:%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%dm,
>>     # Use semicolon separators.
>>     xterm-24bits|xterm with 24-bit direct color mode,
>>       use=xterm-256color,
>>       setb24=\E[48;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%dm,
>>       setf24=\E[38;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%dm,
>>     
>>     $ tic -x -o ~/.terminfo terminfo-24bit.src
>>     
>>     $ TERM=xterm-24bit emacs -nw
Source: Emacs FAQ, 5.3

I followed these directions literally, and enabled direct color support
in Emacs, apparently without issue.  However, what I did not realize was
that I ought *not* to have followed the example literally – instead, I
should have replaced "xterm" with my actual $TERM value.  (In
retrospect, that is fairly obvious, but it wasn't at the time, in part
because I've come across other areas where setting a $TERM value of
xterm was a valid workaround.  And, as I said, following the directions
literally appeared to work as shown by the output of `M-x
list-colors-display`.)

Months later, I realized that I was experiencing a very odd bug: my
Emacs instance was taking well over 2 seconds to start in a terminal,
even when invoked with `emacs -Q -nw`.  Even more oddly, I determined
`M-x emacs-init-time` showed a very low time – whatever was causing the
slowdown was outside of the normal init process.  After fairly extensive
testing, I discovered that the slowdown was caused by a pselect
timeout.  This one, according to strace:

> pselect6(5, [4], [], NULL, {tv_sec=1, tv_nsec=998526670}, {NULL, 8}) =
> 0 (Timeout) <2.000940>

After even more testing, I determined that the issue was caused by the
terminfo/$TERM settings: apparently, if the user has a $TERM value that
does not match their terminal, Emacs will have a 2-second delay before
launching, waiting for whatever that pselect call is looking for.

Once I finally figured the above out, I decided to file this bug in the
hope that the documentation can be updated and other can avoid spending
as long with this issue as I did.  Specifically, I believe that Section
5.3 should be updated to indicate that users should modify the custom
definition provided to use the appropriate $TERM value for their own
terminal.

Beyond that, the fact that an incorrect $TERM setting results in a
2-second delay in launching Emacs may, itself, be an Emacs bug – after
all, the incorrect $TERM/terminfo settings *worked* to display correct
colors in Emacs.  It may be that there is a way to avoid this delay, or
at least to signal the error to the user via *Messages*.  But I don't
know enough about Emacs' internals to be sure of that.

Either way, it was certainly an interesting process to debug!

Best regards,
Daniel




In GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu)
 of 2019-09-18 built on svetlemodry
System Description:	Antergos Linux

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --without-x --without-sound --with-modules
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt'
 CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB MODULES THREADS LIBSYSTEMD

Important settings:
  value of $LC_COLLATE: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail tool-bar rmail-loaddefs mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
regexp-opt rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
term/xterm xterm time-date elec-pair mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify multi-tty make-network-process emacs)

Memory information:
((conses 16 97828 8897)
 (symbols 48 19448 1)
 (miscs 40 40 96)
 (strings 32 27649 1398)
 (string-bytes 1 739458)
 (vectors 16 11875)
 (vector-slots 8 449294 5376)
 (floats 8 49 66)
 (intervals 56 208 0)
 (buffers 992 12))

bug#39733: 26.3; Unclear direct color mode documentation can cause loading delays

[Eli Zaretskii]

> From: Daniel Long Sockwell <daniel@codesections.com>
> Date: Fri, 21 Feb 2020 21:36:05 -0500
> 
> > Emacs can be invoked with a custom definition as shown below
> >>     $ cat terminfo-24bit.src
> >>     
> >>     # Use colon separators.
> >>     xterm-24bit|xterm with 24-bit direct color mode,
> >>       use=xterm-256color,
> >>       setb24=\E[48:2:%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%dm,
> >>       setf24=\E[38:2:%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%dm,
> >>     # Use semicolon separators.
> >>     xterm-24bits|xterm with 24-bit direct color mode,
> >>       use=xterm-256color,
> >>       setb24=\E[48;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%dm,
> >>       setf24=\E[38;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%dm,
> >>     
> >>     $ tic -x -o ~/.terminfo terminfo-24bit.src
> >>     
> >>     $ TERM=xterm-24bit emacs -nw
> Source: Emacs FAQ, 5.3
> 
> I followed these directions literally, and enabled direct color support
> in Emacs, apparently without issue.  However, what I did not realize was
> that I ought *not* to have followed the example literally – instead, I
> should have replaced "xterm" with my actual $TERM value.

If the terminfo entry uses xterm-24bits, then you _do_ have to use
xterm-24bit as the value of TERM.  If you want any other terminal, the
terminfo entry should have a different name.  The example shows it for
xterm.

> Months later, I realized that I was experiencing a very odd bug: my
> Emacs instance was taking well over 2 seconds to start in a terminal,
> even when invoked with `emacs -Q -nw`.  Even more oddly, I determined
> `M-x emacs-init-time` showed a very low time – whatever was causing the
> slowdown was outside of the normal init process.  After fairly extensive
> testing, I discovered that the slowdown was caused by a pselect
> timeout.  This one, according to strace:
> 
> > pselect6(5, [4], [], NULL, {tv_sec=1, tv_nsec=998526670}, {NULL, 8}) =
> > 0 (Timeout) <2.000940>
> 
> After even more testing, I determined that the issue was caused by the
> terminfo/$TERM settings: apparently, if the user has a $TERM value that
> does not match their terminal, Emacs will have a 2-second delay before
> launching, waiting for whatever that pselect call is looking for.

That's because for xterm we support additional features, unrelated to
colors, which need to be probed at startup.  The probe sends a special
control sequence to the terminal, and then waits for the response with
a 2 sec timeout.  Your terminal doesn't support those control
sequences, so it seems, so you get the timeout.

> Once I finally figured the above out, I decided to file this bug in the
> hope that the documentation can be updated and other can avoid spending
> as long with this issue as I did.

The latest version of the Emacs FAQ already says at the end of this
section:

     If your terminal is incompatible with XTerm, you may have to use
  another ‘TERM’ definition.  Any terminal whose name includes ‘direct’
  should be a candidate.  The ‘toe’ command can be used to find out which
  of these are installed on your system:

       $ toe | grep '\-direct'

       konsole-direct  konsole with direct-color indexing
       vte-direct      vte with direct-color indexing
       st-direct       st with direct-color indexing
       xterm-direct2   xterm with direct-color indexing (old)
       xterm-direct    xterm with direct-color indexing

     Terminals with ‘RGB’ capability treat pixels #000001 - #000007 as
  indexed colors to maintain backward compatibility with applications that
  are unaware of direct color mode.  Therefore the seven darkest blue
  shades may not be available.  If this is a problem, you can always use
  custom terminal definition with ‘setb24’ and ‘setf24’.

Given this addition, do you see anything else that needs to be changed
in the FAQ?

Thanks.

bug#39733: 26.3; Unclear direct color mode documentation can cause loading delays

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Given this addition, do you see anything else that needs to be changed
> in the FAQ?

This was half a year ago, and there was no response, so I'm closing this
bug report.  If there's anything more to be done here, please respond to
the debbugs mail address, and we'll reopen the bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no