bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag--- via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

Eli Zaretskii writes:
> > > Thanks.  These results AFAIU indicate that there's no problem with the
> > > terminfo level: the number of colors known to term.c (printed as max:)
> > > is the same no matter how you start Emacs.  Do you agree with this
> > > conclusion?
> >
> > That does appear to be the case. In fact the number of colours is
> > set by 'tty->TN_max_colors = tgetnum ("Co")' and then set again
> > when the RGB or other direct-colour capability is detected.
>
> That's correct, but why is that an issue?  What I saw in your
> debugging print-outs is that the value of tty->TN_max_colors was the

It's not an issue, just a description of where the value comes from.

> > Unfortunately the information from terminfo is sometimes ignored.
>
> Why "unfortunately", if the result is the correct number of colors?

It's unfortunate that emacs (apparently) discards what it's told
in favour of heuristics. Especially as the heuristics appear to
draw the wrong conclusion.

> > If we're at the point where the terminal's name matters one jot,
> > that is to say when the value of $TERM is used for _anything_ other
> > than selecting a terminfo entry, that is a bug.
>
> It is not necessarily a bug.  Emacs needs to be able to convert

It's a problem inasmuch as it is a violation of the terminfo protocol
and takes a sharp turn into non-standards territory. Perhaps bug
is not the best term. It's certainly a red flag.

> X-style color names to escape sequences it should send to the terminal
> to produce a similar color.  That conversion is specific to the
> terminal, so Emacs relies on the name of the terminal to set up the
> conversion.

Hmm. If only there was some sort of database that described
terminal-specific capabilities and features and how to use them.
It could also ensure the programs were portable, even to terminals
that haven't been created yet!

> We could have a new feature whereby, if terminfo redirected to a
> different entry, Emacs would load the lisp/term file of the redirected
> entry.  But that'd be a new feature that AFAIU never existed in Emacs.

My understanding is that the inclusion by terminfo is invisible to
programs which use it. Being xterm-specific leads you down the path
of a term/xxx.el file for each terminal which has non-standard
features that you wish to use.

You are correct to note that this part of the discussion is academic
until the fault is found.

> This is not enough.  You need also to change this line at the end of
> notworking.el:
>
>   (provide 'term/xterm)
>
> to say
>
>   (provide 'term/notworking)
>
> Sorry I didn't mention this before.

No worries. It still makes no difference though unfortunately.

But to be thorough I changed all instances of xterm in that
notworking.el to notworking and that did work. After a few minutes
I isolated the change to renaming terminal-init-xterm to
terminal-init-notworking. In fact that's the only change that needs
to be made: This diff is sufficient:
        
        --- lisp/term/xterm.el  Sat Jan  6 12:56:30 2024
        +++ lisp/term/notworking.el     Fri Mar  8 14:00:28 2024
        @@ -913,7 +913,7 @@
           ;; We likewise unconditionally enable support for focus tracking.
           (xterm--init-focus-tracking))

        -(defun terminal-init-xterm ()
        +(defun terminal-init-notworking ()
           "Terminal initialization function for xterm."
           (unwind-protect
               (progn

I was actually a little surprised when I changed the provide line
back to 'term/xterm and it continued to work but this agrees with
lisp/term/README.

That explains why the workaround of using a terminal name beginning
with xterm- works. It doesn't explain why that is necessary and
using setb24/setf24, the RGB capability and/or COLORTERM=truecolor
is insufficient (ie. it doesn't explain why the terminal name
_matters_). I didn't test Tc. I don't think it would make a difference.

> > Before correcting that, running with TERM=notworking reported 8
> > colours but with TERM=xterm-direct reported 16 (the extra 8 are all
> > black) and said in *Messages*: xterm-register-default-colors:
> > Unsupported number of xterm colors (16777214).
> >
> > ...
>
> Sorry, I don't think I understand the above.  In particular, I don't
> see the value 16777214 anywhere in term.c.  What did I miss?

There were two things. First that even though the first time I ran
the test last night my compiled test terminfo entries were incomplete,
re-running the test with complete entries made no difference to the
outcome of the test.

Second and separately from any other tests, I had changed the three
instances of 16777216 within the TERMINFO block to 16777213, 16777214
and 16777215 to see which path in that code was taken (tigetflag("RGB")
in all cases).

When I left the incorrect numbers in place, the *Messages* window
complained that the number was invalid but only when TERM=xterm-direct
or TERM=xterm-working.

(And just confirmed that with a working notworking.el, TERM=notworking
also complains about this problem)

> > Summary:
> >
> > My generated terminfo files were wrong. I needed to use tic -x.
> >
> > That doesn't matter though because whether or not emacs detects
> > "RGB", direct colour availability depends on the $TERM value beginning
> > with "xterm-" (and, curiously, that TN_max_colors is valid --- if
> > TERM=notworking an invalid TN_max_colors is ignored).
>
> Is that a fact or is it a conclusion from what you observed?  If the
> latter, I think we still have stuff to test, see above.

This is a description of the symptoms I have observed.

Matthew