bug#69598: 29.2; colour support based on $TERM value not terminfo database

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

(CC'd OpenBSD maintainer in case it's a packaging issue somehow)

Emacs with direct colour works only if $TERM is set to xterm-direct.
Even if $TERM refers to a terminfo entry that is identical (confirmed
by infocmp):

        turtle-direct|Test Emacs,use=xterm-direct,

Within xterm, if TERM is set to turtle-direct and emacs launched with
-nw, list-colors-display only lists 8 colours.

I am on OpenBSD -current as of a day or so ago.

I haven't tested any other values of $TERM and don't know of any
with direct colour support anyway.

--- auto report below

In GNU Emacs 29.2 (build 1, x86_64-unknown-openbsd, GTK+ Version
 3.24.41, cairo version 1.18.0) of 2024-03-04 built on
 amd64.ports.openbsd.org
System Description: OpenBSD llama.datum 7.5 GENERIC.MP#54 amd64

Configured using:
 'configure --build=amd64-unknown-openbsd --without-sound
 --with-x-toolkit=gtk3 --prefix=/usr/local --sysconfdir=/etc
 --mandir=/usr/local/man --infodir=/usr/local/info --localstatedir=/var
 --disable-silent-rules --disable-gtk-doc 'CFLAGS=-O2 -pipe -g'
 CPPFLAGS=-I/usr/local/include 'LDFLAGS=-L/usr/local/lib -g''

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBXML2 M17N_FLT MODULES NOTIFY KQUEUE PDUMPER PNG RSVG
SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM
XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: C.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  shell-dirtrack-mode: t
  windmove-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  blink-cursor-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny rfc822
mml mml-sec epa epg rfc6068 epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils tabify macros
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs cua-base align tmm text-property-search sh-script smie
treesit executable dired-aux dired dired-loaddefs misearch multi-isearch
conf-mode vc-git diff-mode vc-dispatcher cperl-mode facemenu term/screen
term/xterm xterm tramp tramp-loaddefs trampver tramp-integration files-x
tramp-compat rx parse-time iso8601 time-date format-spec evil
evil-keybindings evil-integration evil-maps evil-commands ffap url-parse
auth-source eieio eieio-core password-cache json map byte-opt bytecomp
byte-compile url-vars reveal flyspell ispell evil-jumps
evil-command-window evil-search evil-ex shell subr-x pcomplete comint
ansi-osc ansi-color evil-types evil-macros evil-repeat evil-states
evil-core evil-common windmove calc calc-loaddefs calc-macs thingatpt
rect evil-digraphs evil-vars ring edmacro kmacro undo-tree derived
easy-mmode cl-extra help-mode cl-seq cl-macs gv diff cl-loaddefs cl-lib
advice rmc iso-transl tooltip cconv eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind kqueue lcms2 dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process emacs)

Memory information:
((conses 16 320087 46442)
 (symbols 48 18980 1)
 (strings 32 53946 2567)
 (string-bytes 1 1782815)
 (vectors 16 31174)
 (vector-slots 8 1018498 154632)
 (floats 8 64 251)
 (intervals 56 7633 2000)
 (buffers 984 37))

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> Cc: Jeremie Courreges-Anglas <jca@wxcvbn.org>
> Date: Wed, 06 Mar 2024 23:01:46 +0000
> From: chohag--- via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> (CC'd OpenBSD maintainer in case it's a packaging issue somehow)
> 
> Emacs with direct colour works only if $TERM is set to xterm-direct.
> Even if $TERM refers to a terminfo entry that is identical (confirmed
> by infocmp):
> 
>         turtle-direct|Test Emacs,use=xterm-direct,
> 
> Within xterm, if TERM is set to turtle-direct and emacs launched with
> -nw, list-colors-display only lists 8 colours.

I believe this is some issue with terminfo, since I see no hits for
the "xterm-direct" string in our sources.

> I am on OpenBSD -current as of a day or so ago.
> 
> I haven't tested any other values of $TERM and don't know of any
> with direct colour support anyway.

From NEWS:

  ** Emacs can support 24-bit color TTY without terminfo database.
  If your text-mode terminal supports 24-bit true color, but your system
  lacks the terminfo database, you can instruct Emacs to support 24-bit
  true color by setting 'COLORTERM=truecolor' in the environment.  This is
  useful on systems such as FreeBSD which ships only with "etc/termcap".

  *** Emacs will now use 24-bit colors on terminals that support "Tc" capability.
  This is in addition to previously-supported ways of discovering 24-bit
  color support: either via the "RGB" or "setf24" capabilities, or if
  the 'COLORTERM' environment variable is set to the value "truecolor".

Did you try the COLORTERM=truecolor setting?

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii writes:
>
>   ** Emacs can support 24-bit color TTY without terminfo database.
>   If your text-mode terminal supports 24-bit true color, but your system
>   lacks the terminfo database, you can instruct Emacs to support 24-bit
>   true color by setting 'COLORTERM=truecolor' in the environment.  This is
>   useful on systems such as FreeBSD which ships only with "etc/termcap".
>
>   *** Emacs will now use 24-bit colors on terminals that support "Tc" capability.
>   This is in addition to previously-supported ways of discovering 24-bit
>   color support: either via the "RGB" or "setf24" capabilities, or if
>   the 'COLORTERM' environment variable is set to the value "truecolor".
>
> Did you try the COLORTERM=truecolor setting?

I did now, in a new non-xterm terminal which does display all colours
in emacs if $TERM is xterm-direct, and no joy (only lists 8 colours).

I also performed this sequence on a linux (Debian 6.1.67-1 (2023-12-12))
box in xterm (379) now that my cat has vacated it:

        $ export TERM=xterm-direct
        $ emacs -nw                   # Version 28.2
        $ echo 'fancy|Fancy Term,use=xterm-direct,' > fancy.info
        $ tic fancy.info
        $ export TERM=fancy
        $ emacs -nw

In the first emacs, list-colors-display listed (I presume) 256
colours. Certainly a lot and with the X names (it does not name
them nicely if the terminal reports 256 colours). In the second it
listed 8. Emacs has no configuration on that box (it's for compiling).

I recall testing the RGB capability late last night despite not
seeing it documented anywhere and that did not work. I shall
experiment with the Tc capability and RGB more carefully and see
what effect they have nevertheless I think the above sequence now
repeated on two distinct operating systems is quite telling.

Both systems have little or no customisation beyond the base image
apart from installed packages. Certainly nothing that should affect
terminfo (apart from running tic).

Cheers

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> From: chohag@jtan.com
> cc: chohag@jtan.com, 69598@debbugs.gnu.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Thu, 07 Mar 2024 08:47:37 +0200."
> Date: Thu, 07 Mar 2024 17:32:38 +0000
> 
> Eli Zaretskii writes:
> >
> >   ** Emacs can support 24-bit color TTY without terminfo database.
> >   If your text-mode terminal supports 24-bit true color, but your system
> >   lacks the terminfo database, you can instruct Emacs to support 24-bit
> >   true color by setting 'COLORTERM=truecolor' in the environment.  This is
> >   useful on systems such as FreeBSD which ships only with "etc/termcap".
> >
> >   *** Emacs will now use 24-bit colors on terminals that support "Tc" capability.
> >   This is in addition to previously-supported ways of discovering 24-bit
> >   color support: either via the "RGB" or "setf24" capabilities, or if
> >   the 'COLORTERM' environment variable is set to the value "truecolor".
> >
> > Did you try the COLORTERM=truecolor setting?
> 
> I did now, in a new non-xterm terminal which does display all colours
> in emacs if $TERM is xterm-direct, and no joy (only lists 8 colours).
> 
> I also performed this sequence on a linux (Debian 6.1.67-1 (2023-12-12))
> box in xterm (379) now that my cat has vacated it:
> 
>         $ export TERM=xterm-direct
>         $ emacs -nw                   # Version 28.2
>         $ echo 'fancy|Fancy Term,use=xterm-direct,' > fancy.info
>         $ tic fancy.info
>         $ export TERM=fancy
>         $ emacs -nw
> 
> In the first emacs, list-colors-display listed (I presume) 256
> colours. Certainly a lot and with the X names (it does not name
> them nicely if the terminal reports 256 colours). In the second it
> listed 8. Emacs has no configuration on that box (it's for compiling).
> 
> I recall testing the RGB capability late last night despite not
> seeing it documented anywhere and that did not work. I shall
> experiment with the Tc capability and RGB more carefully and see
> what effect they have nevertheless I think the above sequence now
> repeated on two distinct operating systems is quite telling.
> 
> Both systems have little or no customisation beyond the base image
> apart from installed packages. Certainly nothing that should affect
> terminfo (apart from running tic).

So I guess you will need to step with a debugger through the code in
term.c which discovers and initializes the color-related capabilities,
and see what's going on there on your system.  Just from the
information you provided, it is very hard to guess what could be the
culprit.

Thanks.

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii writes:
> > I also performed this sequence on a linux (Debian 6.1.67-1 (2023-12-12))
> > box in xterm (379) now that my cat has vacated it:
> > 
> >         $ export TERM=xterm-direct
> >         $ emacs -nw                   # Version 28.2
> >         $ echo 'fancy|Fancy Term,use=xterm-direct,' > fancy.info
> >         $ tic fancy.info
> >         $ export TERM=fancy
> >         $ emacs -nw
> > 
> > In the first emacs, list-colors-display listed (I presume) 256
> > colours. Certainly a lot and with the X names (it does not name
> > them nicely if the terminal reports 256 colours). In the second it
> > listed 8. Emacs has no configuration on that box (it's for compiling).
>
> So I guess you will need to step with a debugger through the code in
> term.c which discovers and initializes the color-related capabilities,

If there's something you would like me to look for then I can but
I am highly sceptical of the idea that it is isolated to something
unique about my computers here.

Besides you can see clearly in term.c that there are no references
to xterm in the strings and hopefully none hidden behind macros
(there is one to xterm+direct in a comment describing where hard-coded
defaults are from, which is one of the xterm-direct building blocks),
so whatever is going wrong is likely to be somewhere else, such as
a misunderstanding deeper within the emacs runtime.

> and see what's going on there on your system.  Just from the

Systems. Plural. Linux and BSD. I could probably open an xterm on
something else somehow but the chance of the same problem on two
distinct systems being caused by a local issue is very low.

Did you confirm presence of the bug with the transcript I provided?

> information you provided, it is very hard to guess what could be the
> culprit.

It's a lot harder for me. I only use emacs.

What more information do you need? Here's another bit from a quick
test while writing this email (in the unadulterated-Debian xterm
that's still open):

        $ echo 'xterm-naughty|Naughty Term,use=xterm-direct,' >> fancy.info
        $ tic fancy.info
        $ export TERM=xterm-naughty
        $ emacs -nw

That emacs displayed all the colours, so it's not 'xterm-direct'
in $TERM that emacs is treating magically, it's 'xterm' (naughty-xterm
lists 8 colours, so probably make that /^xterm/).

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> From: chohag@jtan.com
> cc: chohag@jtan.com, 69598@debbugs.gnu.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Thu, 07 Mar 2024 19:47:05 +0200."
> Date: Thu, 07 Mar 2024 18:31:20 +0000
> 
> Eli Zaretskii writes:
> > So I guess you will need to step with a debugger through the code in
> > term.c which discovers and initializes the color-related capabilities,
> 
> If there's something you would like me to look for then I can but
> I am highly sceptical of the idea that it is isolated to something
> unique about my computers here.

I think there might be a misunderstanding.  What meant was to see what
happens in the function init_tty in this part of it:


  #ifdef TERMINFO
	{
	  const char *fg = tigetstr ("setf24");
	  const char *bg = tigetstr ("setb24");
	  /* Non-standard support for 24-bit colors. */
	  if (fg && bg
	      && fg != (char *) (intptr_t) -1
	      && bg != (char *) (intptr_t) -1)
	    {
	      tty->TS_set_foreground = fg;
	      tty->TS_set_background = bg;
	      tty->TN_max_colors = 16777216;
	    }
	  /* Standard support for 24-bit colors.  */
	  else if (tigetflag ("RGB") > 0)
	    {
	      /* If the used Terminfo library supports only 16-bit
		 signed values, tgetnum("Co") and tigetnum("colors")
		 could return 32767.  */
	      tty->TN_max_colors = 16777216;
	    }
	  /* Fall back to xterm+direct (semicolon version) if Tc is set
	     (de-facto standard introduced by tmux) or if	requested by
	     the COLORTERM environment variable.  */
	  else if ((tigetflag ("Tc") > 0)
		   || ((bg = getenv ("COLORTERM")) != NULL
		       && strcasecmp (bg, "truecolor") == 0))
	    {
	      tty->TS_set_foreground = "\033[%?%p1%{8}%<%t3%p1%d%e38;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%d%;m";
	      tty->TS_set_background = "\033[%?%p1%{8}%<%t4%p1%d%e48;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%d%;m";
	      tty->TN_max_colors = 16777216;
	    }
	}
  #endif

By contrast, it sounds like you are only talking about a terminfo
entry that redirects to another entry.

> Did you confirm presence of the bug with the transcript I provided?

No, I cannot try that on the systems to which I have access.  Maybe
someone else could, preferably someone who knows more than I do about
terminfo.

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii writes:
> > From: chohag@jtan.com
> > cc: chohag@jtan.com, 69598@debbugs.gnu.org
> > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
> >    message dated "Thu, 07 Mar 2024 19:47:05 +0200."
> > Date: Thu, 07 Mar 2024 18:31:20 +0000
> > 
> > Eli Zaretskii writes:
> > > So I guess you will need to step with a debugger through the code in
> > > term.c which discovers and initializes the color-related capabilities,
> > 
> > If there's something you would like me to look for then I can but
> > I am highly sceptical of the idea that it is isolated to something
> > unique about my computers here.
>
> I think there might be a misunderstanding.  What meant was to see what
> happens in the function init_tty in this part of it:
>
>
>   #ifdef TERMINFO
> 	{
>         ... // calls to terminfo database requests
> 	}
>   #endif

I understand.

I haven't stepped through that piece of code in an emacs binary but
modulo terminfo bugs (which I'm going to assume for now aren't
happening) but you can see from it that it looks up strings in the
database and records the result. Unless emacs has done anything
unusual to load that database (very unlikely) it will 100% return
the strings in the terminfo entry it's directed to look at.

So it certainly (although I have not tried it) is not a problem in
that piece of code.

> By contrast, it sounds like you are only talking about a terminfo
> entry that redirects to another entry.

Yes, the problem can be isolated to creating a new terminfo entry
and using it.

But the problem is not with the terminfo entry itself. The new entry
is an exact duplicate of a working terminfo entry (where working
means that list-colors-display lists 256 named colours) and it only
works if the new entry has a name which begins "xterm-".

This means that somewhere between running the code above which does
detect that 16M colours are available, emacs discards that information
and instead (seems to) decide that support is there based on the
name of the terminal in $TERM.

> No, I cannot try that on the systems to which I have access.  Maybe

Any system with xterm and emacs should be able to reproduce the problem.

I don't know how any other terminal in which emacs detects direct
colour support would perform.

> someone else could, preferably someone who knows more than I do about
> terminfo.

It is almost certainly related to either how terminfo is used, or
because the information terminfo supplies is ignored at a critical
point.

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> From: chohag@jtan.com
> cc: chohag@jtan.com, 69598@debbugs.gnu.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Thu, 07 Mar 2024 21:26:32 +0200."
> Date: Thu, 07 Mar 2024 19:59:17 +0000
> 
> But the problem is not with the terminfo entry itself. The new entry
> is an exact duplicate of a working terminfo entry (where working
> means that list-colors-display lists 256 named colours) and it only
> works if the new entry has a name which begins "xterm-".
> 
> This means that somewhere between running the code above which does
> detect that 16M colours are available, emacs discards that information
> and instead (seems to) decide that support is there based on the
> name of the terminal in $TERM.

If you or someone else could establish that tty->TN_max_colors
receives the correct value of the number of supported colors during
initialization, and yet Emacs still doesn't realize those colors
unless the TERM name begins with "xterm", we could make some progress,
because it could point us to the code which is responsible for this
lossage.  Right now, I don't know where to look.

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii writes:
> > From: chohag@jtan.com
> > cc: chohag@jtan.com, 69598@debbugs.gnu.org
> > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
> >    message dated "Thu, 07 Mar 2024 21:26:32 +0200."
> > Date: Thu, 07 Mar 2024 19:59:17 +0000
> > 
> > But the problem is not with the terminfo entry itself. The new entry
> > is an exact duplicate of a working terminfo entry (where working
> > means that list-colors-display lists 256 named colours) and it only
> > works if the new entry has a name which begins "xterm-".
> > 
> > This means that somewhere between running the code above which does
> > detect that 16M colours are available, emacs discards that information
> > and instead (seems to) decide that support is there based on the
> > name of the terminal in $TERM.
>
> If you or someone else could establish that tty->TN_max_colors
> receives the correct value of the number of supported colors during
> initialization, and yet Emacs still doesn't realize those colors
> unless the TERM name begins with "xterm", we could make some progress,
> because it could point us to the code which is responsible for this
> lossage.  Right now, I don't know where to look.

I have (eventually, see below) followed this sequence of steps:

        $ cd ~/src/emacs-29.2
        $ ./configure
        $ gmake

launch xterm, and then within it:

        $ rm -fr ~/.terminfo
        $ echo 'notworking|Non-working clone,use=xterm-direct,' > new.info
        $ echo 'xterm-working|Working clone,use=xterm-direct,' >> new.info
        $ tic new.info

# pre-patch compiled binary:

        $ export TERM=xterm-direct
        $ ~/src/emacs-29.2/src/emacs -nw
        $ M-x list-colors-display # confirm 256 named colours
        $ export TERM=notworking
        $ ~/src/emacs-29.2/src/emacs -nw
        $ M-x list-colors-display # confirm only 8 colours
        $ export TERM=xterm-working
        $ emacs -nw               # system xterm
        $ M-x list-colors-display # confirm 256 named colours

# patch:
--- src/term.c~	Sat Jan  6 12:56:31 2024
+++ src/term.c	Thu Mar  7 20:56:36 2024
@@ -4235,6 +4235,15 @@
         tty->TN_no_color_video = 0;
     }
 
+  {
+    int bug = open("/tmp/debug", O_WRONLY | O_CREAT);
+    dprintf(bug, "pair: %s\n", tty->TS_orig_pair);
+    dprintf(bug, "bg: %s\n", tty->TS_set_background);
+    dprintf(bug, "fg: %s\n", tty->TS_set_foreground);
+    dprintf(bug, "max: %zu\n", tty->TN_max_colors);
+    close(bug);
+  }
+
   tty_default_color_capabilities (tty, 1);
 
   MagicWrap (tty) = tgetflag ("xn");

        $ gmake -C ~/src/emacs-29.2
        $ export TERM=xterm-direct
        $ ~/src/emacs-29.2/src/emacs -nw
        $ M-x list-colors-display $ # confirm 256 named colours
        $ doas cat /tmp/debug # doas because umask 0
        pair: 
        bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
        fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
        max: 16777216
        $ rm -f /tmp/debug # just in case
        $ export TERM=notworking
        $ ~/src/emacs-29.2/src/emacs -nw
        $ M-x list-colors-display # confirm only 8 colours
        $ doas cat /tmp/debug
        pair: 
        bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
        fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
        max: 16777216
        $ rm -f /tmp/debug
        $ export TERM=xterm-working
        $ ~/src/emacs-29.2/src/emacs -nw
        $ M-x list-colors-display # confirm 256 named colours
        $ doas cat /tmp/debug
        pair: 
        bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
        fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
        max: 16777216

Notably TS_orig_pair is printed here as an empty string in all cases
(NULL would print as "(null)"? and there there are no extra spaces),
created by requesting the op capability which *is* defined in
xterm-direct.

        $ infocmp dumb xterm-direct | grep op: 
                op: NULL, '\E[39;49m'.

In fact before getting the patch to this stage I included the print
statements within the block you suggested but of course this was
not reached because of op not being detected.

It didn't make any sense to me why tgetstr would return an empty
string in that instance. address where tgetstr gets its data from
is another name for tty->termcap_strings_buffer so changing the
print statement to print that and setting $TERM to xterm-direct
reveals:

        $ doas hexdump -C /tmp/debug
        00000000  60 60 60 1b 5b 4c 27 27  27 0a                    |```.[L'''.|
        0000000a

I tried to replicate this on the Debian box but emacs wouldn't build
despite the configure script being satisfied (not relevant here,
but it couldn't find mpz_* despite libgmp-dev and then libgmp3-dev
being installed).

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

chohag@jtan.com writes:
>
>         $ doas hexdump -C /tmp/debug
>         00000000  60 60 60 1b 5b 4c 27 27  27 0a                    |```.[L'''.|
>         0000000a
>

Sorry, but in case it wasn't obvious the ``` and ''' were added by
me in the print statement:

+    dprintf(bug, "```%s'''\n", tty->termcap_strings_buffer);

I was expecting cat to be enough.

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> From: chohag@jtan.com
> cc: chohag@jtan.com, 69598@debbugs.gnu.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Thu, 07 Mar 2024 22:10:30 +0200."
> Date: Thu, 07 Mar 2024 21:45:44 +0000
> 
> I have (eventually, see below) followed this sequence of steps:
> 
>         $ cd ~/src/emacs-29.2
>         $ ./configure
>         $ gmake
> 
> launch xterm, and then within it:
> 
>         $ rm -fr ~/.terminfo
>         $ echo 'notworking|Non-working clone,use=xterm-direct,' > new.info
>         $ echo 'xterm-working|Working clone,use=xterm-direct,' >> new.info
>         $ tic new.info
> 
> # pre-patch compiled binary:
> 
>         $ export TERM=xterm-direct
>         $ ~/src/emacs-29.2/src/emacs -nw
>         $ M-x list-colors-display # confirm 256 named colours
>         $ export TERM=notworking
>         $ ~/src/emacs-29.2/src/emacs -nw
>         $ M-x list-colors-display # confirm only 8 colours
>         $ export TERM=xterm-working
>         $ emacs -nw               # system xterm
>         $ M-x list-colors-display # confirm 256 named colours
> 
> # patch:
> --- src/term.c~	Sat Jan  6 12:56:31 2024
> +++ src/term.c	Thu Mar  7 20:56:36 2024
> @@ -4235,6 +4235,15 @@
>          tty->TN_no_color_video = 0;
>      }
>  
> +  {
> +    int bug = open("/tmp/debug", O_WRONLY | O_CREAT);
> +    dprintf(bug, "pair: %s\n", tty->TS_orig_pair);
> +    dprintf(bug, "bg: %s\n", tty->TS_set_background);
> +    dprintf(bug, "fg: %s\n", tty->TS_set_foreground);
> +    dprintf(bug, "max: %zu\n", tty->TN_max_colors);
> +    close(bug);
> +  }
> +
>    tty_default_color_capabilities (tty, 1);
>  
>    MagicWrap (tty) = tgetflag ("xn");
> 
>         $ gmake -C ~/src/emacs-29.2
>         $ export TERM=xterm-direct
>         $ ~/src/emacs-29.2/src/emacs -nw
>         $ M-x list-colors-display $ # confirm 256 named colours
>         $ doas cat /tmp/debug # doas because umask 0
>         pair: 
>         bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
>         fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
>         max: 16777216
>         $ rm -f /tmp/debug # just in case
>         $ export TERM=notworking
>         $ ~/src/emacs-29.2/src/emacs -nw
>         $ M-x list-colors-display # confirm only 8 colours
>         $ doas cat /tmp/debug
>         pair: 
>         bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
>         fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
>         max: 16777216
>         $ rm -f /tmp/debug
>         $ export TERM=xterm-working
>         $ ~/src/emacs-29.2/src/emacs -nw
>         $ M-x list-colors-display # confirm 256 named colours
>         $ doas cat /tmp/debug
>         pair: 
>         bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
>         fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m
>         max: 16777216
> 

Thanks.  These results AFAIU indicate that there's no problem with the
terminfo level: the number of colors known to term.c (printed as max:)
is the same no matter how you start Emacs.  Do you agree with this
conclusion?

I think the problem is that if the terminal's name doesn't begin with
"xterm", Emacs does not load lisp/term/xterm.el, which defines the
colors that will be available.  So please copy lisp/term/xterm.el to a
file named lisp/term/notworking.el, and then start Emacs with
TERM=notworking and see if that solves the problem.  That is, arrange
for a lisp/term/NAME.el file to be a copy of xterm.el when you invoke
Emacs with TERM=NAME.

> Notably TS_orig_pair is printed here as an empty string in all cases
> (NULL would print as "(null)"? and there there are no extra spaces),
> created by requesting the op capability which *is* defined in
> xterm-direct.
> 
>         $ infocmp dumb xterm-direct | grep op: 
>                 op: NULL, '\E[39;49m'.
> 
> In fact before getting the patch to this stage I included the print
> statements within the block you suggested but of course this was
> not reached because of op not being detected.
> 
> It didn't make any sense to me why tgetstr would return an empty
> string in that instance. address where tgetstr gets its data from
> is another name for tty->termcap_strings_buffer so changing the
> print statement to print that and setting $TERM to xterm-direct
> reveals:
> 
>         $ doas hexdump -C /tmp/debug
>         00000000  60 60 60 1b 5b 4c 27 27  27 0a                    |```.[L'''.|
>         0000000a

I don't think I see why TS_orig_pair's value is relevant to the issue
at hand.  Can you explain why you are interested in it?

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii writes:
> > From: chohag@jtan.com
> > cc: chohag@jtan.com, 69598@debbugs.gnu.org
> > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
> >    message dated "Thu, 07 Mar 2024 22:10:30 +0200."
> > Date: Thu, 07 Mar 2024 21:45:44 +0000
>
> Thanks.  These results AFAIU indicate that there's no problem with the
> terminfo level: the number of colors known to term.c (printed as max:)
> is the same no matter how you start Emacs.  Do you agree with this
> conclusion?

That does appear to be the case. In fact the number of colours is
set by 'tty->TN_max_colors = tgetnum ("Co")' and then set again
when the RGB or other direct-colour capability is detected.

Unfortunately the information from terminfo is sometimes ignored.

> I think the problem is that if the terminal's name doesn't begin with
> "xterm", Emacs does not load lisp/term/xterm.el, which defines the
> colors that will be available.  So please copy lisp/term/xterm.el to a
> file named lisp/term/notworking.el, and then start Emacs with
> TERM=notworking and see if that solves the problem.  That is, arrange
> for a lisp/term/NAME.el file to be a copy of xterm.el when you invoke
> Emacs with TERM=NAME.

If we're at the point where the terminal's name matters one jot,
that is to say when the value of $TERM is used for _anything_ other
than selecting a terminfo entry, that is a bug.

I don't know how emacs finds files when it's not installed so I
will be thorough:

        $ cd ~/src/emacs-29.2
        $ cp lisp/term/xterm.el lisp/term/notworking.el
        $ gmake
        $ ls -l lisp/term/{xterm,notworking}*
        -rw-r--r--  1 mking  mking  46048 Mar  8 10:46 lisp/term/notworking.el
        -rw-r--r--  1 mking  mking  32368 Mar  8 10:46 lisp/term/notworking.elc
        -rw-r--r--  1 mking  mking  46048 Jan  6 12:56 lisp/term/xterm.el
        -rw-r--r--  1 mking  mking  32368 Jan 18 10:06 lisp/term/xterm.elc
        $ infocmp -x notworking xterm-direct    # to confirm they're identical
        comparing notworking to xterm-direct.
            comparing booleans.
            comparing numbers.
            comparing strings.
        $ export TERM=notworking
        $ ./src/emacs -nw                       # confirm 8 colours
        $ export TERM=xterm-direct
        $ ./src/emacs -nw                       # confirm 256 named colours

While I was setting up this test I'd forgotten that I changed the
value in each of the three assignments to TN_max_colors in order
to confirm which condition inside the TERMINFO block was satisfied.

Before correcting that, running with TERM=notworking reported 8
colours but with TERM=xterm-direct reported 16 (the extra 8 are all
black) and said in *Messages*: xterm-register-default-colors:
Unsupported number of xterm colors (16777214).

The value ending in 4 pointed to the tigetflag("RGB") block which
reminded me of the -x option to tic telling it to accept non-standard
capabilities (of which RGB is one) and I thought perhaps the option
was also necessary when creating an entry which uses an entry
which has non-standard capabilities. It turns out that it is and
that notworking was lacking the extra capabilities.

So I got excited that maybe I'd found the problem and it was in
fact bad terminfo data. I did the whole round of tests again but
this time using "tic -x new.info" instead of plain tic (confirmed
with infocmp -x that the extended capabilities exist in the new
terminfo entries notworking and xterm-working).

The result was identical.

Summary:

My generated terminfo files were wrong. I needed to use tic -x.

That doesn't matter though because whether or not emacs detects
"RGB", direct colour availability depends on the $TERM value beginning
with "xterm-" (and, curiously, that TN_max_colors is valid --- if
TERM=notworking an invalid TN_max_colors is ignored).

Creating the file lisp/term/notworking.el had no apparent effect.

That's it. Nothing important follows.

> > Notably TS_orig_pair is printed here as an empty string in all cases
> > (NULL would print as "(null)"? and there there are no extra spaces),
> > created by requesting the op capability which *is* defined in
> > xterm-direct.
> > 
> >         $ infocmp dumb xterm-direct | grep op: 
> >                 op: NULL, '\E[39;49m'.
>
> I don't think I see why TS_orig_pair's value is relevant to the issue
> at hand.  Can you explain why you are interested in it?

Immediately prior to the TERMINFO block is this code:

          /* SVr4/ANSI color support.  If "op" isn't available, don't support
             color because we can't switch back to the default foreground and
             background.  */
          tty->TS_orig_pair = tgetstr ("op", address);
          if (tty->TS_orig_pair)
            {
              tty->TS_set_foreground = tgetstr ("AF", address);
              tty->TS_set_background = tgetstr ("AB", address);
              if (!tty->TS_set_foreground)
        	{
        	  /* SVr4.  */
        	  tty->TS_set_foreground = tgetstr ("Sf", address);
        	  tty->TS_set_background = tgetstr ("Sb", address);
        	}

              tty->TN_max_colors = tgetnum ("Co");

        #ifdef TERMINFO
              {
                ...
              }
        #endif

When I originally patched term.c (I've had enough of using one
terminal to debug another terminal that is running its own terminal)
I included the print statements at the end of the TERMINFO block
and the debug file did not get created. I moved it down to above
the next '}' line which appears to correspond to the test of
tty->TS_orig_pair to no avail (the combination of C, CPP and
indentation is hard to follow) then I moved it out of that block
and began the test which got the results I sent last night.

Probing it again with coffee it appears that TS_orig_pair is given
a value and the block is entered so I expect I misread something
when writing the patch.

That reveals a misunderstanding, again because I assumed a cat would
be enough. The pair line (printing tty->TS_orig_pair) does in fact
have a value: 1b 5b 33 39 3b 34 39 6d which is the correct SGR
control and would obviously not have an appearance when sent to a
terminal. It is in the debug file.

Focusing on tty->termcap_strings_buffer appears to be another red
herring. I was trying to figure out where tgetstr is getting its
data from and address née area née tty->termcap_strings_buffer was
the obvious candidate.

Cheers,

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> From: chohag@jtan.com
> cc: chohag@jtan.com, 69598@debbugs.gnu.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Fri, 08 Mar 2024 09:11:03 +0200."
> Date: Fri, 08 Mar 2024 11:36:18 +0000
> 
> Eli Zaretskii writes:
> > > From: chohag@jtan.com
> > > cc: chohag@jtan.com, 69598@debbugs.gnu.org
> > > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
> > >    message dated "Thu, 07 Mar 2024 22:10:30 +0200."
> > > Date: Thu, 07 Mar 2024 21:45:44 +0000
> >
> > Thanks.  These results AFAIU indicate that there's no problem with the
> > terminfo level: the number of colors known to term.c (printed as max:)
> > is the same no matter how you start Emacs.  Do you agree with this
> > conclusion?
> 
> That does appear to be the case. In fact the number of colours is
> set by 'tty->TN_max_colors = tgetnum ("Co")' and then set again
> when the RGB or other direct-colour capability is detected.

That's correct, but why is that an issue?  What I saw in your
debugging print-outs is that the value of tty->TN_max_colors was the
same in all of the 3 configurations you tried.  Since your changes
printed this value after the code that determines how many colors are
supported, I concluded that terminal initialization computed the same
number of colors in all 3 cases.  Did I miss something?

> Unfortunately the information from terminfo is sometimes ignored.

Why "unfortunately", if the result is the correct number of colors?

> > I think the problem is that if the terminal's name doesn't begin with
> > "xterm", Emacs does not load lisp/term/xterm.el, which defines the
> > colors that will be available.  So please copy lisp/term/xterm.el to a
> > file named lisp/term/notworking.el, and then start Emacs with
> > TERM=notworking and see if that solves the problem.  That is, arrange
> > for a lisp/term/NAME.el file to be a copy of xterm.el when you invoke
> > Emacs with TERM=NAME.
> 
> If we're at the point where the terminal's name matters one jot,
> that is to say when the value of $TERM is used for _anything_ other
> than selecting a terminfo entry, that is a bug.

It is not necessarily a bug.  Emacs needs to be able to convert
X-style color names to escape sequences it should send to the terminal
to produce a similar color.  That conversion is specific to the
terminal, so Emacs relies on the name of the terminal to set up the
conversion.

We could have a new feature whereby, if terminfo redirected to a
different entry, Emacs would load the lisp/term file of the redirected
entry.  But that'd be a new feature that AFAIU never existed in Emacs.

In any case, I suggest to delay the discussion of what should be the
solution to the issue until we understand the issue completely.
AFAIU, we are not there yet, see above.

> I don't know how emacs finds files when it's not installed so I
> will be thorough:
> 
>         $ cd ~/src/emacs-29.2
>         $ cp lisp/term/xterm.el lisp/term/notworking.el

This is not enough.  You need also to change this line at the end of
notworking.el:

  (provide 'term/xterm)

to say

  (provide 'term/notworking)

Sorry I didn't mention this before.

> While I was setting up this test I'd forgotten that I changed the
> value in each of the three assignments to TN_max_colors in order
> to confirm which condition inside the TERMINFO block was satisfied.
> 
> Before correcting that, running with TERM=notworking reported 8
> colours but with TERM=xterm-direct reported 16 (the extra 8 are all
> black) and said in *Messages*: xterm-register-default-colors:
> Unsupported number of xterm colors (16777214).
> 
> The value ending in 4 pointed to the tigetflag("RGB") block which
> reminded me of the -x option to tic telling it to accept non-standard
> capabilities (of which RGB is one) and I thought perhaps the option
> was also necessary when creating an entry which uses an entry
> which has non-standard capabilities. It turns out that it is and
> that notworking was lacking the extra capabilities.
> 
> So I got excited that maybe I'd found the problem and it was in
> fact bad terminfo data. I did the whole round of tests again but
> this time using "tic -x new.info" instead of plain tic (confirmed
> with infocmp -x that the extended capabilities exist in the new
> terminfo entries notworking and xterm-working).
> 
> The result was identical.

Sorry, I don't think I understand the above.  In particular, I don't
see the value 16777214 anywhere in term.c.  What did I miss?

> Summary:
> 
> My generated terminfo files were wrong. I needed to use tic -x.
> 
> That doesn't matter though because whether or not emacs detects
> "RGB", direct colour availability depends on the $TERM value beginning
> with "xterm-" (and, curiously, that TN_max_colors is valid --- if
> TERM=notworking an invalid TN_max_colors is ignored).

Is that a fact or is it a conclusion from what you observed?  If the
latter, I think we still have stuff to test, see above.

> Creating the file lisp/term/notworking.el had no apparent effect.

See above: the creation was incomplete.

Thanks.

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii writes:
> > > Thanks.  These results AFAIU indicate that there's no problem with the
> > > terminfo level: the number of colors known to term.c (printed as max:)
> > > is the same no matter how you start Emacs.  Do you agree with this
> > > conclusion?
> >
> > That does appear to be the case. In fact the number of colours is
> > set by 'tty->TN_max_colors = tgetnum ("Co")' and then set again
> > when the RGB or other direct-colour capability is detected.
>
> That's correct, but why is that an issue?  What I saw in your
> debugging print-outs is that the value of tty->TN_max_colors was the

It's not an issue, just a description of where the value comes from.

> > Unfortunately the information from terminfo is sometimes ignored.
>
> Why "unfortunately", if the result is the correct number of colors?

It's unfortunate that emacs (apparently) discards what it's told
in favour of heuristics. Especially as the heuristics appear to
draw the wrong conclusion.

> > If we're at the point where the terminal's name matters one jot,
> > that is to say when the value of $TERM is used for _anything_ other
> > than selecting a terminfo entry, that is a bug.
>
> It is not necessarily a bug.  Emacs needs to be able to convert

It's a problem inasmuch as it is a violation of the terminfo protocol
and takes a sharp turn into non-standards territory. Perhaps bug
is not the best term. It's certainly a red flag.

> X-style color names to escape sequences it should send to the terminal
> to produce a similar color.  That conversion is specific to the
> terminal, so Emacs relies on the name of the terminal to set up the
> conversion.

Hmm. If only there was some sort of database that described
terminal-specific capabilities and features and how to use them.
It could also ensure the programs were portable, even to terminals
that haven't been created yet!

> We could have a new feature whereby, if terminfo redirected to a
> different entry, Emacs would load the lisp/term file of the redirected
> entry.  But that'd be a new feature that AFAIU never existed in Emacs.

My understanding is that the inclusion by terminfo is invisible to
programs which use it. Being xterm-specific leads you down the path
of a term/xxx.el file for each terminal which has non-standard
features that you wish to use.

You are correct to note that this part of the discussion is academic
until the fault is found.

> This is not enough.  You need also to change this line at the end of
> notworking.el:
>
>   (provide 'term/xterm)
>
> to say
>
>   (provide 'term/notworking)
>
> Sorry I didn't mention this before.

No worries. It still makes no difference though unfortunately.

But to be thorough I changed all instances of xterm in that
notworking.el to notworking and that did work. After a few minutes
I isolated the change to renaming terminal-init-xterm to
terminal-init-notworking. In fact that's the only change that needs
to be made: This diff is sufficient:
        
        --- lisp/term/xterm.el  Sat Jan  6 12:56:30 2024
        +++ lisp/term/notworking.el     Fri Mar  8 14:00:28 2024
        @@ -913,7 +913,7 @@
           ;; We likewise unconditionally enable support for focus tracking.
           (xterm--init-focus-tracking))

        -(defun terminal-init-xterm ()
        +(defun terminal-init-notworking ()
           "Terminal initialization function for xterm."
           (unwind-protect
               (progn

I was actually a little surprised when I changed the provide line
back to 'term/xterm and it continued to work but this agrees with
lisp/term/README.

That explains why the workaround of using a terminal name beginning
with xterm- works. It doesn't explain why that is necessary and
using setb24/setf24, the RGB capability and/or COLORTERM=truecolor
is insufficient (ie. it doesn't explain why the terminal name
_matters_). I didn't test Tc. I don't think it would make a difference.

> > Before correcting that, running with TERM=notworking reported 8
> > colours but with TERM=xterm-direct reported 16 (the extra 8 are all
> > black) and said in *Messages*: xterm-register-default-colors:
> > Unsupported number of xterm colors (16777214).
> >
> > ...
>
> Sorry, I don't think I understand the above.  In particular, I don't
> see the value 16777214 anywhere in term.c.  What did I miss?

There were two things. First that even though the first time I ran
the test last night my compiled test terminfo entries were incomplete,
re-running the test with complete entries made no difference to the
outcome of the test.

Second and separately from any other tests, I had changed the three
instances of 16777216 within the TERMINFO block to 16777213, 16777214
and 16777215 to see which path in that code was taken (tigetflag("RGB")
in all cases).

When I left the incorrect numbers in place, the *Messages* window
complained that the number was invalid but only when TERM=xterm-direct
or TERM=xterm-working.

(And just confirmed that with a working notworking.el, TERM=notworking
also complains about this problem)

> > Summary:
> >
> > My generated terminfo files were wrong. I needed to use tic -x.
> >
> > That doesn't matter though because whether or not emacs detects
> > "RGB", direct colour availability depends on the $TERM value beginning
> > with "xterm-" (and, curiously, that TN_max_colors is valid --- if
> > TERM=notworking an invalid TN_max_colors is ignored).
>
> Is that a fact or is it a conclusion from what you observed?  If the
> latter, I think we still have stuff to test, see above.

This is a description of the symptoms I have observed.

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> From: chohag@jtan.com
> cc: chohag@jtan.com, 69598@debbugs.gnu.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Fri, 08 Mar 2024 14:22:52 +0200."
> Date: Fri, 08 Mar 2024 14:23:26 +0000
> 
> > > If we're at the point where the terminal's name matters one jot,
> > > that is to say when the value of $TERM is used for _anything_ other
> > > than selecting a terminfo entry, that is a bug.
> >
> > It is not necessarily a bug.  Emacs needs to be able to convert
> 
> It's a problem inasmuch as it is a violation of the terminfo protocol
> and takes a sharp turn into non-standards territory. Perhaps bug
> is not the best term. It's certainly a red flag.

We only do that for handling the colors, and I'm not sure I can see
how terminfo can help us here.  terminfo knows how many colors a
terminal supports and which escape sequences to use for that, but it
doesn't know which colors will be produced, in terms of the RGB
triplets.  So Emacs needs something else to support colors on text
terminals, and that something comes from the lisp/term/ files.  Those
files also set up Emacs for various non-standard function keys
produced by the terminal, and support other features, like copy/paste
etc.

So these files are not a bug, they are the foundation of several
important features in Emacs.

> > X-style color names to escape sequences it should send to the terminal
> > to produce a similar color.  That conversion is specific to the
> > terminal, so Emacs relies on the name of the terminal to set up the
> > conversion.
> 
> Hmm. If only there was some sort of database that described
> terminal-specific capabilities and features and how to use them.
> It could also ensure the programs were portable, even to terminals
> that haven't been created yet!

Yes.  But AFAIK there's no such database, so Emacs must use its own
database.

> > This is not enough.  You need also to change this line at the end of
> > notworking.el:
> >
> >   (provide 'term/xterm)
> >
> > to say
> >
> >   (provide 'term/notworking)
> >
> > Sorry I didn't mention this before.
> 
> No worries. It still makes no difference though unfortunately.
> 
> But to be thorough I changed all instances of xterm in that
> notworking.el to notworking and that did work. After a few minutes
> I isolated the change to renaming terminal-init-xterm to
> terminal-init-notworking. In fact that's the only change that needs
> to be made: This diff is sufficient:
>         
>         --- lisp/term/xterm.el  Sat Jan  6 12:56:30 2024
>         +++ lisp/term/notworking.el     Fri Mar  8 14:00:28 2024
>         @@ -913,7 +913,7 @@
>            ;; We likewise unconditionally enable support for focus tracking.
>            (xterm--init-focus-tracking))
> 
>         -(defun terminal-init-xterm ()
>         +(defun terminal-init-notworking ()
>            "Terminal initialization function for xterm."
>            (unwind-protect
>                (progn
> 
> I was actually a little surprised when I changed the provide line
> back to 'term/xterm and it continued to work but this agrees with
> lisp/term/README.

So you now agree with me that a terminal file in lisp/term is needed
for this situation?  Or are there issues that are still unaccounted
for, as far as color support is concerned?

> That explains why the workaround of using a terminal name beginning
> with xterm- works. It doesn't explain why that is necessary and
> using setb24/setf24, the RGB capability and/or COLORTERM=truecolor
> is insufficient (ie. it doesn't explain why the terminal name
> _matters_).

I hope what I wrote above explains that.  If you read the code in
xterm.el, I think you will understand it.  The other part of the
color-related support is in lisp/term/tty-colors.el, btw.

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii writes:
> > From: chohag@jtan.com
> > cc: chohag@jtan.com, 69598@debbugs.gnu.org
> > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
> >    message dated "Fri, 08 Mar 2024 14:22:52 +0200."
> > Date: Fri, 08 Mar 2024 14:23:26 +0000
> > 
> > > > If we're at the point where the terminal's name matters one jot,
> > > > that is to say when the value of $TERM is used for _anything_ other
> > > > than selecting a terminfo entry, that is a bug.
> > >
> > > It is not necessarily a bug.  Emacs needs to be able to convert
> > 
> > It's a problem inasmuch as it is a violation of the terminfo protocol
> > and takes a sharp turn into non-standards territory. Perhaps bug
> > is not the best term. It's certainly a red flag.
>
> We only do that for handling the colors, and I'm not sure I can see
> how terminfo can help us here.  terminfo knows how many colors a
> terminal supports and which escape sequences to use for that, but it
> doesn't know which colors will be produced, in terms of the RGB
> triplets.  So Emacs needs something else to support colors on text

Terminfo doesn't ``know'' what a terminal supports. Terminfo is the
database and routines to look up capabality values. It's the
applications which use terminfo that ``know'' what those capabilities
do and, of course, many of them have a conventional use as described
in terminfo(5).

In order to describe capabilities of terminals which hadn't been
made in 1980 it can be expanded with new capabilities. For example
setb24 and setf24 are terminfo extensions which as near as I can
tell originated in emacs to do exactly what you describe.

https://www.gnu.org/software/emacs/manual/html_node/efaq/Colors-on-a-TTY.html
mentions setb24 and setf24. In fact it's the only page on the web
that I can find (apart those linking to it) with any record of these
controls:

        Emacs 26.1 and later support direct color mode in terminals.
        If Emacs finds Terminfo capabilities `setb24' and `setf24',
        24-bit direct color mode is used. The capability strings
        are expected to take one 24-bit pixel value as argument and
        transform the pixel to a string that can be used to send
        24-bit colors to the terminal.

        Standard terminal definitions don't support these capabilities
        and therefore custom definition is needed.

          setb24=\E[48\:2\:\:%p1%{65536}%/%d\:%p1%{256}%/%{255}%&\
             %d\:%p1%{255}%&%dm,
          setf24=\E[38\:2\:\:%p1%{65536}%/%d\:%p1%{256}%/%{255}%&\
             %d\:%p1%{255}%&%dm,

``24-bit direct color mode is use'' with no further qualifications.
It doesn't say that emacs also needs to have special knowledge about
the terminal's model.

The page continues with examples for further developments but in
every case implies or outright says that certain capabilities are
the only requirement, not specific models.

> terminals, and that something comes from the lisp/term/ files.  Those
> files also set up Emacs for various non-standard function keys
> produced by the terminal, and support other features, like copy/paste
> etc.
>
> So these files are not a bug, they are the foundation of several
> important features in Emacs.

Their overriding what terminfo tells it is the buggy part. Terminfo
says ``terminal foo has RGB or setf24 and setb24 capabilities and
supports 16M colours'' but emacs disagrees.

Incidentally, both of the examples you gave are already supported
in some fashion by terminfo capabilities, for example:

        The kNNN capabilities map non-standard keys.

        bracketed+paste|xterm bracketed paste,
                BD=\E[?2004l,
                BE=\E[?2004h,
                PE=\E[201~,
                PS=\E[200~,

> > > X-style color names to escape sequences it should send to the terminal
> > > to produce a similar color.  That conversion is specific to the
> > > terminal, so Emacs relies on the name of the terminal to set up the
> > > conversion.
> > 
> > Hmm. If only there was some sort of database that described
> > terminal-specific capabilities and features and how to use them.
> > It could also ensure the programs were portable, even to terminals
> > that haven't been created yet!
>
> Yes.  But AFAIK there's no such database, so Emacs must use its own
> database.

I'm sorry, I was trying to use humour to point out that terminfo
is that database.

> So you now agree with me that a terminal file in lisp/term is needed
> for this situation?  Or are there issues that are still unaccounted
> for, as far as color support is concerned?

I don't agree. A new lisp/term file achieved the desired result,
or rather the appearance of it, but for all of the wrong reasons:
because the terminal lied about what it was.

Looking at it another way if a user uses one of the terminals emacs
supports but with certain features enabled or disabled by a custom
terminfo entry there's nothing to say that the name of that entry
has to follow any particular convention. A site prefix is a well
established naming scheme so something like mycompany-footerm does
not seem like an unreasonable name and nothing formally or informally
in terminfo or termcap, or in emacs' documentation for that matter,
suggests that it wouldn't work.

Indeed I'm quite surprised to find this behaviour: emacs, running
in xterm and informed it is running on a terminal identical to
xterm, does not act the same as it does when that description comes
under the *label* xterm. That's like the name of a variable influencing
its contents.

In short, if emacs only supports colour (or any other feature) on
specific terminals it knows about in advance, it should say so and
not imply that a terminfo or termcap capability is sufficient.

As it stands this paragraph:

        If you think that your terminal supports colors, but Emacs
        won't use them, check the termcap entry for your display
        type for color-related capabilities.

... should be amended to say ``check if your terminal or emulator is
one of these supported models: <list>'' but couldn't emacs believe
and use what terminfo has told it, otherwise why did it ask?

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> From: chohag@jtan.com
> cc: chohag@jtan.com, 69598@debbugs.gnu.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Fri, 08 Mar 2024 17:09:51 +0200."
> Date: Fri, 08 Mar 2024 18:52:28 +0000
> 
> > So you now agree with me that a terminal file in lisp/term is needed
> > for this situation?  Or are there issues that are still unaccounted
> > for, as far as color support is concerned?
> 
> I don't agree.

In that case, we will have to agree to disagree, sorry.

> In short, if emacs only supports colour (or any other feature) on
> specific terminals it knows about in advance, it should say so and
> not imply that a terminfo or termcap capability is sufficient.

Support for colors beyond the basic 8 is very specific to terminals.
Emacs solves this problem by relying on the name of the terminal.  So
any new terminal has to abide by this protocol.  That's how this stuff
works in Emacs.

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii writes:
> > From: chohag@jtan.com
> > cc: chohag@jtan.com, 69598@debbugs.gnu.org
> > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
> >    message dated "Fri, 08 Mar 2024 17:09:51 +0200."
> > Date: Fri, 08 Mar 2024 18:52:28 +0000
> > 
> > > So you now agree with me that a terminal file in lisp/term is needed
> > > for this situation?  Or are there issues that are still unaccounted
> > > for, as far as color support is concerned?
> > 
> > I don't agree.
>
> In that case, we will have to agree to disagree, sorry.

I don't, sorry.

> > In short, if emacs only supports colour (or any other feature) on
> > specific terminals it knows about in advance, it should say so and
> > not imply that a terminfo or termcap capability is sufficient.
>
> Support for colors beyond the basic 8 is very specific to terminals.

Hardware terminals which could not hope to display anything close
to individual full colour rendition for the background and foreground
of each character cell, without exhorbitant cost, did have myriad
ways of expressing their limited colour support, true, back when
such things were in the process of being invented, but out of all
the software terminal emulators and programs since which claim full
colour support, pretty much everything has followed the conventions
laid down by xterm and/or VTE which follow or try to follow ECMA-48,
T.416 and related formal and de-facto standards, such as describing
their capabilities with termcap (1978) and/or terminfo (1981).

> Emacs solves this problem by relying on the name of the terminal.  So

Everything except, apparently, Emacs.

> any new terminal has to abide by this protocol.  That's how this stuff
> works in Emacs.

That's what standards are for. Formal standards are those published
by some organisation, de-facto standards are the conventions followed
by a majority. Emacs' behaviour here is neither.

I should be clear that I'm not trying to get something fixed --- I
don't have a problem here. I can get along just fine but in my
travels I discovered that emacs did not work as I expected and, on
investigation, as it described itself and so I thought that GNU
would like know that their flagship product and/or its documentation
was deficient.

I have provided more information at your request which I hope you
find useful but I have no need or desire to pursue this matter.

Matthew

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> From: chohag@jtan.com
> cc: chohag@jtan.com, 69598@debbugs.gnu.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Fri, 08 Mar 2024 21:50:01 +0200."
> Date: Fri, 08 Mar 2024 21:13:27 +0000
> 
> I should be clear that I'm not trying to get something fixed --- I
> don't have a problem here. I can get along just fine but in my
> travels I discovered that emacs did not work as I expected and, on
> investigation, as it described itself and so I thought that GNU
> would like know that their flagship product and/or its documentation
> was deficient.
> 
> I have provided more information at your request which I hope you
> find useful but I have no need or desire to pursue this matter.

Thank you for your inputs.  You may wish reading the file
lisp/term/README, which attempts to explain this and other related
stuff.  Sorry I didn't remember we had this file and didn't point you
to it in the first place.

bug#69598: 29.2; colour support based on $TERM value not terminfo database

[Eli Zaretskii]

> Cc: 69598@debbugs.gnu.org
> Date: Sat, 09 Mar 2024 09:18:51 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: chohag@jtan.com
> > cc: chohag@jtan.com, 69598@debbugs.gnu.org
> > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
> >    message dated "Fri, 08 Mar 2024 21:50:01 +0200."
> > Date: Fri, 08 Mar 2024 21:13:27 +0000
> > 
> > I should be clear that I'm not trying to get something fixed --- I
> > don't have a problem here. I can get along just fine but in my
> > travels I discovered that emacs did not work as I expected and, on
> > investigation, as it described itself and so I thought that GNU
> > would like know that their flagship product and/or its documentation
> > was deficient.
> > 
> > I have provided more information at your request which I hope you
> > find useful but I have no need or desire to pursue this matter.
> 
> Thank you for your inputs.  You may wish reading the file
> lisp/term/README, which attempts to explain this and other related
> stuff.  Sorry I didn't remember we had this file and didn't point you
> to it in the first place.

No further comments within 2 weeks, so I'm now closing this bug.