* bug#69598: 29.2; colour support based on $TERM value not terminfo database @ 2024-03-06 23:01 chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 6:47 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-06 23:01 UTC (permalink / raw) To: 69598; +Cc: Jeremie Courreges-Anglas (CC'd OpenBSD maintainer in case it's a packaging issue somehow) Emacs with direct colour works only if $TERM is set to xterm-direct. Even if $TERM refers to a terminfo entry that is identical (confirmed by infocmp): turtle-direct|Test Emacs,use=xterm-direct, Within xterm, if TERM is set to turtle-direct and emacs launched with -nw, list-colors-display only lists 8 colours. I am on OpenBSD -current as of a day or so ago. I haven't tested any other values of $TERM and don't know of any with direct colour support anyway. --- auto report below In GNU Emacs 29.2 (build 1, x86_64-unknown-openbsd, GTK+ Version 3.24.41, cairo version 1.18.0) of 2024-03-04 built on amd64.ports.openbsd.org System Description: OpenBSD llama.datum 7.5 GENERIC.MP#54 amd64 Configured using: 'configure --build=amd64-unknown-openbsd --without-sound --with-x-toolkit=gtk3 --prefix=/usr/local --sysconfdir=/etc --mandir=/usr/local/man --infodir=/usr/local/info --localstatedir=/var --disable-silent-rules --disable-gtk-doc 'CFLAGS=-O2 -pipe -g' CPPFLAGS=-I/usr/local/include 'LDFLAGS=-L/usr/local/lib -g'' Configured features: CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON LCMS2 LIBOTF LIBXML2 M17N_FLT MODULES NOTIFY KQUEUE PDUMPER PNG RSVG SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB Important settings: value of $LANG: C.UTF-8 locale-coding-system: utf-8-unix Major mode: Fundamental Minor modes in effect: shell-dirtrack-mode: t windmove-mode: t tooltip-mode: t global-eldoc-mode: t show-paren-mode: t electric-indent-mode: t mouse-wheel-mode: t file-name-shadow-mode: t blink-cursor-mode: t column-number-mode: t line-number-mode: t transient-mark-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t Load-path shadows: None found. Features: (shadow sort mail-extr emacsbug message mailcap yank-media puny rfc822 mml mml-sec epa epg rfc6068 epg-config gnus-util mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils tabify macros cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs cua-base align tmm text-property-search sh-script smie treesit executable dired-aux dired dired-loaddefs misearch multi-isearch conf-mode vc-git diff-mode vc-dispatcher cperl-mode facemenu term/screen term/xterm xterm tramp tramp-loaddefs trampver tramp-integration files-x tramp-compat rx parse-time iso8601 time-date format-spec evil evil-keybindings evil-integration evil-maps evil-commands ffap url-parse auth-source eieio eieio-core password-cache json map byte-opt bytecomp byte-compile url-vars reveal flyspell ispell evil-jumps evil-command-window evil-search evil-ex shell subr-x pcomplete comint ansi-osc ansi-color evil-types evil-macros evil-repeat evil-states evil-core evil-common windmove calc calc-loaddefs calc-macs thingatpt rect evil-digraphs evil-vars ring edmacro kmacro undo-tree derived easy-mmode cl-extra help-mode cl-seq cl-macs gv diff cl-loaddefs cl-lib advice rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic indonesian philippine cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5 base64 format env code-pages mule custom widget keymap hashtable-print-readable backquote threads dbusbind kqueue lcms2 dynamic-setting system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process emacs) Memory information: ((conses 16 320087 46442) (symbols 48 18980 1) (strings 32 53946 2567) (string-bytes 1 1782815) (vectors 16 31174) (vector-slots 8 1018498 154632) (floats 8 64 251) (intervals 56 7633 2000) (buffers 984 37)) ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-06 23:01 bug#69598: 29.2; colour support based on $TERM value not terminfo database chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 6:47 ` Eli Zaretskii 2024-03-07 17:32 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-07 6:47 UTC (permalink / raw) To: chohag; +Cc: 69598, jca > Cc: Jeremie Courreges-Anglas <jca@wxcvbn.org> > Date: Wed, 06 Mar 2024 23:01:46 +0000 > From: chohag--- via "Bug reports for GNU Emacs, > the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> > > (CC'd OpenBSD maintainer in case it's a packaging issue somehow) > > Emacs with direct colour works only if $TERM is set to xterm-direct. > Even if $TERM refers to a terminfo entry that is identical (confirmed > by infocmp): > > turtle-direct|Test Emacs,use=xterm-direct, > > Within xterm, if TERM is set to turtle-direct and emacs launched with > -nw, list-colors-display only lists 8 colours. I believe this is some issue with terminfo, since I see no hits for the "xterm-direct" string in our sources. > I am on OpenBSD -current as of a day or so ago. > > I haven't tested any other values of $TERM and don't know of any > with direct colour support anyway. From NEWS: ** Emacs can support 24-bit color TTY without terminfo database. If your text-mode terminal supports 24-bit true color, but your system lacks the terminfo database, you can instruct Emacs to support 24-bit true color by setting 'COLORTERM=truecolor' in the environment. This is useful on systems such as FreeBSD which ships only with "etc/termcap". *** Emacs will now use 24-bit colors on terminals that support "Tc" capability. This is in addition to previously-supported ways of discovering 24-bit color support: either via the "RGB" or "setf24" capabilities, or if the 'COLORTERM' environment variable is set to the value "truecolor". Did you try the COLORTERM=truecolor setting? ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 6:47 ` Eli Zaretskii @ 2024-03-07 17:32 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 17:47 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 17:32 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 69598, chohag Eli Zaretskii writes: > > ** Emacs can support 24-bit color TTY without terminfo database. > If your text-mode terminal supports 24-bit true color, but your system > lacks the terminfo database, you can instruct Emacs to support 24-bit > true color by setting 'COLORTERM=truecolor' in the environment. This is > useful on systems such as FreeBSD which ships only with "etc/termcap". > > *** Emacs will now use 24-bit colors on terminals that support "Tc" capability. > This is in addition to previously-supported ways of discovering 24-bit > color support: either via the "RGB" or "setf24" capabilities, or if > the 'COLORTERM' environment variable is set to the value "truecolor". > > Did you try the COLORTERM=truecolor setting? I did now, in a new non-xterm terminal which does display all colours in emacs if $TERM is xterm-direct, and no joy (only lists 8 colours). I also performed this sequence on a linux (Debian 6.1.67-1 (2023-12-12)) box in xterm (379) now that my cat has vacated it: $ export TERM=xterm-direct $ emacs -nw # Version 28.2 $ echo 'fancy|Fancy Term,use=xterm-direct,' > fancy.info $ tic fancy.info $ export TERM=fancy $ emacs -nw In the first emacs, list-colors-display listed (I presume) 256 colours. Certainly a lot and with the X names (it does not name them nicely if the terminal reports 256 colours). In the second it listed 8. Emacs has no configuration on that box (it's for compiling). I recall testing the RGB capability late last night despite not seeing it documented anywhere and that did not work. I shall experiment with the Tc capability and RGB more carefully and see what effect they have nevertheless I think the above sequence now repeated on two distinct operating systems is quite telling. Both systems have little or no customisation beyond the base image apart from installed packages. Certainly nothing that should affect terminfo (apart from running tic). Cheers Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 17:32 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 17:47 ` Eli Zaretskii 2024-03-07 18:31 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-07 17:47 UTC (permalink / raw) To: chohag; +Cc: 69598 > From: chohag@jtan.com > cc: chohag@jtan.com, 69598@debbugs.gnu.org > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > message dated "Thu, 07 Mar 2024 08:47:37 +0200." > Date: Thu, 07 Mar 2024 17:32:38 +0000 > > Eli Zaretskii writes: > > > > ** Emacs can support 24-bit color TTY without terminfo database. > > If your text-mode terminal supports 24-bit true color, but your system > > lacks the terminfo database, you can instruct Emacs to support 24-bit > > true color by setting 'COLORTERM=truecolor' in the environment. This is > > useful on systems such as FreeBSD which ships only with "etc/termcap". > > > > *** Emacs will now use 24-bit colors on terminals that support "Tc" capability. > > This is in addition to previously-supported ways of discovering 24-bit > > color support: either via the "RGB" or "setf24" capabilities, or if > > the 'COLORTERM' environment variable is set to the value "truecolor". > > > > Did you try the COLORTERM=truecolor setting? > > I did now, in a new non-xterm terminal which does display all colours > in emacs if $TERM is xterm-direct, and no joy (only lists 8 colours). > > I also performed this sequence on a linux (Debian 6.1.67-1 (2023-12-12)) > box in xterm (379) now that my cat has vacated it: > > $ export TERM=xterm-direct > $ emacs -nw # Version 28.2 > $ echo 'fancy|Fancy Term,use=xterm-direct,' > fancy.info > $ tic fancy.info > $ export TERM=fancy > $ emacs -nw > > In the first emacs, list-colors-display listed (I presume) 256 > colours. Certainly a lot and with the X names (it does not name > them nicely if the terminal reports 256 colours). In the second it > listed 8. Emacs has no configuration on that box (it's for compiling). > > I recall testing the RGB capability late last night despite not > seeing it documented anywhere and that did not work. I shall > experiment with the Tc capability and RGB more carefully and see > what effect they have nevertheless I think the above sequence now > repeated on two distinct operating systems is quite telling. > > Both systems have little or no customisation beyond the base image > apart from installed packages. Certainly nothing that should affect > terminfo (apart from running tic). So I guess you will need to step with a debugger through the code in term.c which discovers and initializes the color-related capabilities, and see what's going on there on your system. Just from the information you provided, it is very hard to guess what could be the culprit. Thanks. ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 17:47 ` Eli Zaretskii @ 2024-03-07 18:31 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 19:26 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 18:31 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 69598, chohag Eli Zaretskii writes: > > I also performed this sequence on a linux (Debian 6.1.67-1 (2023-12-12)) > > box in xterm (379) now that my cat has vacated it: > > > > $ export TERM=xterm-direct > > $ emacs -nw # Version 28.2 > > $ echo 'fancy|Fancy Term,use=xterm-direct,' > fancy.info > > $ tic fancy.info > > $ export TERM=fancy > > $ emacs -nw > > > > In the first emacs, list-colors-display listed (I presume) 256 > > colours. Certainly a lot and with the X names (it does not name > > them nicely if the terminal reports 256 colours). In the second it > > listed 8. Emacs has no configuration on that box (it's for compiling). > > So I guess you will need to step with a debugger through the code in > term.c which discovers and initializes the color-related capabilities, If there's something you would like me to look for then I can but I am highly sceptical of the idea that it is isolated to something unique about my computers here. Besides you can see clearly in term.c that there are no references to xterm in the strings and hopefully none hidden behind macros (there is one to xterm+direct in a comment describing where hard-coded defaults are from, which is one of the xterm-direct building blocks), so whatever is going wrong is likely to be somewhere else, such as a misunderstanding deeper within the emacs runtime. > and see what's going on there on your system. Just from the Systems. Plural. Linux and BSD. I could probably open an xterm on something else somehow but the chance of the same problem on two distinct systems being caused by a local issue is very low. Did you confirm presence of the bug with the transcript I provided? > information you provided, it is very hard to guess what could be the > culprit. It's a lot harder for me. I only use emacs. What more information do you need? Here's another bit from a quick test while writing this email (in the unadulterated-Debian xterm that's still open): $ echo 'xterm-naughty|Naughty Term,use=xterm-direct,' >> fancy.info $ tic fancy.info $ export TERM=xterm-naughty $ emacs -nw That emacs displayed all the colours, so it's not 'xterm-direct' in $TERM that emacs is treating magically, it's 'xterm' (naughty-xterm lists 8 colours, so probably make that /^xterm/). Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 18:31 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 19:26 ` Eli Zaretskii 2024-03-07 19:59 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-07 19:26 UTC (permalink / raw) To: chohag; +Cc: 69598 > From: chohag@jtan.com > cc: chohag@jtan.com, 69598@debbugs.gnu.org > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > message dated "Thu, 07 Mar 2024 19:47:05 +0200." > Date: Thu, 07 Mar 2024 18:31:20 +0000 > > Eli Zaretskii writes: > > So I guess you will need to step with a debugger through the code in > > term.c which discovers and initializes the color-related capabilities, > > If there's something you would like me to look for then I can but > I am highly sceptical of the idea that it is isolated to something > unique about my computers here. I think there might be a misunderstanding. What meant was to see what happens in the function init_tty in this part of it: #ifdef TERMINFO { const char *fg = tigetstr ("setf24"); const char *bg = tigetstr ("setb24"); /* Non-standard support for 24-bit colors. */ if (fg && bg && fg != (char *) (intptr_t) -1 && bg != (char *) (intptr_t) -1) { tty->TS_set_foreground = fg; tty->TS_set_background = bg; tty->TN_max_colors = 16777216; } /* Standard support for 24-bit colors. */ else if (tigetflag ("RGB") > 0) { /* If the used Terminfo library supports only 16-bit signed values, tgetnum("Co") and tigetnum("colors") could return 32767. */ tty->TN_max_colors = 16777216; } /* Fall back to xterm+direct (semicolon version) if Tc is set (de-facto standard introduced by tmux) or if requested by the COLORTERM environment variable. */ else if ((tigetflag ("Tc") > 0) || ((bg = getenv ("COLORTERM")) != NULL && strcasecmp (bg, "truecolor") == 0)) { tty->TS_set_foreground = "\033[%?%p1%{8}%<%t3%p1%d%e38;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%d%;m"; tty->TS_set_background = "\033[%?%p1%{8}%<%t4%p1%d%e48;2;%p1%{65536}%/%d;%p1%{256}%/%{255}%&%d;%p1%{255}%&%d%;m"; tty->TN_max_colors = 16777216; } } #endif By contrast, it sounds like you are only talking about a terminfo entry that redirects to another entry. > Did you confirm presence of the bug with the transcript I provided? No, I cannot try that on the systems to which I have access. Maybe someone else could, preferably someone who knows more than I do about terminfo. ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 19:26 ` Eli Zaretskii @ 2024-03-07 19:59 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 20:10 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 19:59 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 69598, chohag Eli Zaretskii writes: > > From: chohag@jtan.com > > cc: chohag@jtan.com, 69598@debbugs.gnu.org > > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > > message dated "Thu, 07 Mar 2024 19:47:05 +0200." > > Date: Thu, 07 Mar 2024 18:31:20 +0000 > > > > Eli Zaretskii writes: > > > So I guess you will need to step with a debugger through the code in > > > term.c which discovers and initializes the color-related capabilities, > > > > If there's something you would like me to look for then I can but > > I am highly sceptical of the idea that it is isolated to something > > unique about my computers here. > > I think there might be a misunderstanding. What meant was to see what > happens in the function init_tty in this part of it: > > > #ifdef TERMINFO > { > ... // calls to terminfo database requests > } > #endif I understand. I haven't stepped through that piece of code in an emacs binary but modulo terminfo bugs (which I'm going to assume for now aren't happening) but you can see from it that it looks up strings in the database and records the result. Unless emacs has done anything unusual to load that database (very unlikely) it will 100% return the strings in the terminfo entry it's directed to look at. So it certainly (although I have not tried it) is not a problem in that piece of code. > By contrast, it sounds like you are only talking about a terminfo > entry that redirects to another entry. Yes, the problem can be isolated to creating a new terminfo entry and using it. But the problem is not with the terminfo entry itself. The new entry is an exact duplicate of a working terminfo entry (where working means that list-colors-display lists 256 named colours) and it only works if the new entry has a name which begins "xterm-". This means that somewhere between running the code above which does detect that 16M colours are available, emacs discards that information and instead (seems to) decide that support is there based on the name of the terminal in $TERM. > No, I cannot try that on the systems to which I have access. Maybe Any system with xterm and emacs should be able to reproduce the problem. I don't know how any other terminal in which emacs detects direct colour support would perform. > someone else could, preferably someone who knows more than I do about > terminfo. It is almost certainly related to either how terminfo is used, or because the information terminfo supplies is ignored at a critical point. Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 19:59 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 20:10 ` Eli Zaretskii 2024-03-07 21:45 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-07 20:10 UTC (permalink / raw) To: chohag; +Cc: 69598 > From: chohag@jtan.com > cc: chohag@jtan.com, 69598@debbugs.gnu.org > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > message dated "Thu, 07 Mar 2024 21:26:32 +0200." > Date: Thu, 07 Mar 2024 19:59:17 +0000 > > But the problem is not with the terminfo entry itself. The new entry > is an exact duplicate of a working terminfo entry (where working > means that list-colors-display lists 256 named colours) and it only > works if the new entry has a name which begins "xterm-". > > This means that somewhere between running the code above which does > detect that 16M colours are available, emacs discards that information > and instead (seems to) decide that support is there based on the > name of the terminal in $TERM. If you or someone else could establish that tty->TN_max_colors receives the correct value of the number of supported colors during initialization, and yet Emacs still doesn't realize those colors unless the TERM name begins with "xterm", we could make some progress, because it could point us to the code which is responsible for this lossage. Right now, I don't know where to look. ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 20:10 ` Eli Zaretskii @ 2024-03-07 21:45 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 21:50 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 7:11 ` Eli Zaretskii 0 siblings, 2 replies; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 21:45 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 69598, chohag Eli Zaretskii writes: > > From: chohag@jtan.com > > cc: chohag@jtan.com, 69598@debbugs.gnu.org > > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > > message dated "Thu, 07 Mar 2024 21:26:32 +0200." > > Date: Thu, 07 Mar 2024 19:59:17 +0000 > > > > But the problem is not with the terminfo entry itself. The new entry > > is an exact duplicate of a working terminfo entry (where working > > means that list-colors-display lists 256 named colours) and it only > > works if the new entry has a name which begins "xterm-". > > > > This means that somewhere between running the code above which does > > detect that 16M colours are available, emacs discards that information > > and instead (seems to) decide that support is there based on the > > name of the terminal in $TERM. > > If you or someone else could establish that tty->TN_max_colors > receives the correct value of the number of supported colors during > initialization, and yet Emacs still doesn't realize those colors > unless the TERM name begins with "xterm", we could make some progress, > because it could point us to the code which is responsible for this > lossage. Right now, I don't know where to look. I have (eventually, see below) followed this sequence of steps: $ cd ~/src/emacs-29.2 $ ./configure $ gmake launch xterm, and then within it: $ rm -fr ~/.terminfo $ echo 'notworking|Non-working clone,use=xterm-direct,' > new.info $ echo 'xterm-working|Working clone,use=xterm-direct,' >> new.info $ tic new.info # pre-patch compiled binary: $ export TERM=xterm-direct $ ~/src/emacs-29.2/src/emacs -nw $ M-x list-colors-display # confirm 256 named colours $ export TERM=notworking $ ~/src/emacs-29.2/src/emacs -nw $ M-x list-colors-display # confirm only 8 colours $ export TERM=xterm-working $ emacs -nw # system xterm $ M-x list-colors-display # confirm 256 named colours # patch: --- src/term.c~ Sat Jan 6 12:56:31 2024 +++ src/term.c Thu Mar 7 20:56:36 2024 @@ -4235,6 +4235,15 @@ tty->TN_no_color_video = 0; } + { + int bug = open("/tmp/debug", O_WRONLY | O_CREAT); + dprintf(bug, "pair: %s\n", tty->TS_orig_pair); + dprintf(bug, "bg: %s\n", tty->TS_set_background); + dprintf(bug, "fg: %s\n", tty->TS_set_foreground); + dprintf(bug, "max: %zu\n", tty->TN_max_colors); + close(bug); + } + tty_default_color_capabilities (tty, 1); MagicWrap (tty) = tgetflag ("xn"); $ gmake -C ~/src/emacs-29.2 $ export TERM=xterm-direct $ ~/src/emacs-29.2/src/emacs -nw $ M-x list-colors-display $ # confirm 256 named colours $ doas cat /tmp/debug # doas because umask 0 pair: bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m max: 16777216 $ rm -f /tmp/debug # just in case $ export TERM=notworking $ ~/src/emacs-29.2/src/emacs -nw $ M-x list-colors-display # confirm only 8 colours $ doas cat /tmp/debug pair: bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m max: 16777216 $ rm -f /tmp/debug $ export TERM=xterm-working $ ~/src/emacs-29.2/src/emacs -nw $ M-x list-colors-display # confirm 256 named colours $ doas cat /tmp/debug pair: bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m max: 16777216 Notably TS_orig_pair is printed here as an empty string in all cases (NULL would print as "(null)"? and there there are no extra spaces), created by requesting the op capability which *is* defined in xterm-direct. $ infocmp dumb xterm-direct | grep op: op: NULL, '\E[39;49m'. In fact before getting the patch to this stage I included the print statements within the block you suggested but of course this was not reached because of op not being detected. It didn't make any sense to me why tgetstr would return an empty string in that instance. address where tgetstr gets its data from is another name for tty->termcap_strings_buffer so changing the print statement to print that and setting $TERM to xterm-direct reveals: $ doas hexdump -C /tmp/debug 00000000 60 60 60 1b 5b 4c 27 27 27 0a |```.[L'''.| 0000000a I tried to replicate this on the Debian box but emacs wouldn't build despite the configure script being satisfied (not relevant here, but it couldn't find mpz_* despite libgmp-dev and then libgmp3-dev being installed). Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 21:45 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 21:50 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 7:11 ` Eli Zaretskii 1 sibling, 0 replies; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-07 21:50 UTC (permalink / raw) To: chohag; +Cc: 69598, Eli Zaretskii chohag@jtan.com writes: > > $ doas hexdump -C /tmp/debug > 00000000 60 60 60 1b 5b 4c 27 27 27 0a |```.[L'''.| > 0000000a > Sorry, but in case it wasn't obvious the ``` and ''' were added by me in the print statement: + dprintf(bug, "```%s'''\n", tty->termcap_strings_buffer); I was expecting cat to be enough. Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-07 21:45 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 21:50 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-08 7:11 ` Eli Zaretskii 2024-03-08 11:36 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 1 sibling, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-08 7:11 UTC (permalink / raw) To: chohag; +Cc: 69598 > From: chohag@jtan.com > cc: chohag@jtan.com, 69598@debbugs.gnu.org > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > message dated "Thu, 07 Mar 2024 22:10:30 +0200." > Date: Thu, 07 Mar 2024 21:45:44 +0000 > > I have (eventually, see below) followed this sequence of steps: > > $ cd ~/src/emacs-29.2 > $ ./configure > $ gmake > > launch xterm, and then within it: > > $ rm -fr ~/.terminfo > $ echo 'notworking|Non-working clone,use=xterm-direct,' > new.info > $ echo 'xterm-working|Working clone,use=xterm-direct,' >> new.info > $ tic new.info > > # pre-patch compiled binary: > > $ export TERM=xterm-direct > $ ~/src/emacs-29.2/src/emacs -nw > $ M-x list-colors-display # confirm 256 named colours > $ export TERM=notworking > $ ~/src/emacs-29.2/src/emacs -nw > $ M-x list-colors-display # confirm only 8 colours > $ export TERM=xterm-working > $ emacs -nw # system xterm > $ M-x list-colors-display # confirm 256 named colours > > # patch: > --- src/term.c~ Sat Jan 6 12:56:31 2024 > +++ src/term.c Thu Mar 7 20:56:36 2024 > @@ -4235,6 +4235,15 @@ > tty->TN_no_color_video = 0; > } > > + { > + int bug = open("/tmp/debug", O_WRONLY | O_CREAT); > + dprintf(bug, "pair: %s\n", tty->TS_orig_pair); > + dprintf(bug, "bg: %s\n", tty->TS_set_background); > + dprintf(bug, "fg: %s\n", tty->TS_set_foreground); > + dprintf(bug, "max: %zu\n", tty->TN_max_colors); > + close(bug); > + } > + > tty_default_color_capabilities (tty, 1); > > MagicWrap (tty) = tgetflag ("xn"); > > $ gmake -C ~/src/emacs-29.2 > $ export TERM=xterm-direct > $ ~/src/emacs-29.2/src/emacs -nw > $ M-x list-colors-display $ # confirm 256 named colours > $ doas cat /tmp/debug # doas because umask 0 > pair: > bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m > fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m > max: 16777216 > $ rm -f /tmp/debug # just in case > $ export TERM=notworking > $ ~/src/emacs-29.2/src/emacs -nw > $ M-x list-colors-display # confirm only 8 colours > $ doas cat /tmp/debug > pair: > bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m > fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m > max: 16777216 > $ rm -f /tmp/debug > $ export TERM=xterm-working > $ ~/src/emacs-29.2/src/emacs -nw > $ M-x list-colors-display # confirm 256 named colours > $ doas cat /tmp/debug > pair: > bg: 1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m > fg: 1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m > max: 16777216 > Thanks. These results AFAIU indicate that there's no problem with the terminfo level: the number of colors known to term.c (printed as max:) is the same no matter how you start Emacs. Do you agree with this conclusion? I think the problem is that if the terminal's name doesn't begin with "xterm", Emacs does not load lisp/term/xterm.el, which defines the colors that will be available. So please copy lisp/term/xterm.el to a file named lisp/term/notworking.el, and then start Emacs with TERM=notworking and see if that solves the problem. That is, arrange for a lisp/term/NAME.el file to be a copy of xterm.el when you invoke Emacs with TERM=NAME. > Notably TS_orig_pair is printed here as an empty string in all cases > (NULL would print as "(null)"? and there there are no extra spaces), > created by requesting the op capability which *is* defined in > xterm-direct. > > $ infocmp dumb xterm-direct | grep op: > op: NULL, '\E[39;49m'. > > In fact before getting the patch to this stage I included the print > statements within the block you suggested but of course this was > not reached because of op not being detected. > > It didn't make any sense to me why tgetstr would return an empty > string in that instance. address where tgetstr gets its data from > is another name for tty->termcap_strings_buffer so changing the > print statement to print that and setting $TERM to xterm-direct > reveals: > > $ doas hexdump -C /tmp/debug > 00000000 60 60 60 1b 5b 4c 27 27 27 0a |```.[L'''.| > 0000000a I don't think I see why TS_orig_pair's value is relevant to the issue at hand. Can you explain why you are interested in it? ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-08 7:11 ` Eli Zaretskii @ 2024-03-08 11:36 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 12:22 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-08 11:36 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 69598, chohag Eli Zaretskii writes: > > From: chohag@jtan.com > > cc: chohag@jtan.com, 69598@debbugs.gnu.org > > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > > message dated "Thu, 07 Mar 2024 22:10:30 +0200." > > Date: Thu, 07 Mar 2024 21:45:44 +0000 > > Thanks. These results AFAIU indicate that there's no problem with the > terminfo level: the number of colors known to term.c (printed as max:) > is the same no matter how you start Emacs. Do you agree with this > conclusion? That does appear to be the case. In fact the number of colours is set by 'tty->TN_max_colors = tgetnum ("Co")' and then set again when the RGB or other direct-colour capability is detected. Unfortunately the information from terminfo is sometimes ignored. > I think the problem is that if the terminal's name doesn't begin with > "xterm", Emacs does not load lisp/term/xterm.el, which defines the > colors that will be available. So please copy lisp/term/xterm.el to a > file named lisp/term/notworking.el, and then start Emacs with > TERM=notworking and see if that solves the problem. That is, arrange > for a lisp/term/NAME.el file to be a copy of xterm.el when you invoke > Emacs with TERM=NAME. If we're at the point where the terminal's name matters one jot, that is to say when the value of $TERM is used for _anything_ other than selecting a terminfo entry, that is a bug. I don't know how emacs finds files when it's not installed so I will be thorough: $ cd ~/src/emacs-29.2 $ cp lisp/term/xterm.el lisp/term/notworking.el $ gmake $ ls -l lisp/term/{xterm,notworking}* -rw-r--r-- 1 mking mking 46048 Mar 8 10:46 lisp/term/notworking.el -rw-r--r-- 1 mking mking 32368 Mar 8 10:46 lisp/term/notworking.elc -rw-r--r-- 1 mking mking 46048 Jan 6 12:56 lisp/term/xterm.el -rw-r--r-- 1 mking mking 32368 Jan 18 10:06 lisp/term/xterm.elc $ infocmp -x notworking xterm-direct # to confirm they're identical comparing notworking to xterm-direct. comparing booleans. comparing numbers. comparing strings. $ export TERM=notworking $ ./src/emacs -nw # confirm 8 colours $ export TERM=xterm-direct $ ./src/emacs -nw # confirm 256 named colours While I was setting up this test I'd forgotten that I changed the value in each of the three assignments to TN_max_colors in order to confirm which condition inside the TERMINFO block was satisfied. Before correcting that, running with TERM=notworking reported 8 colours but with TERM=xterm-direct reported 16 (the extra 8 are all black) and said in *Messages*: xterm-register-default-colors: Unsupported number of xterm colors (16777214). The value ending in 4 pointed to the tigetflag("RGB") block which reminded me of the -x option to tic telling it to accept non-standard capabilities (of which RGB is one) and I thought perhaps the option was also necessary when creating an entry which uses an entry which has non-standard capabilities. It turns out that it is and that notworking was lacking the extra capabilities. So I got excited that maybe I'd found the problem and it was in fact bad terminfo data. I did the whole round of tests again but this time using "tic -x new.info" instead of plain tic (confirmed with infocmp -x that the extended capabilities exist in the new terminfo entries notworking and xterm-working). The result was identical. Summary: My generated terminfo files were wrong. I needed to use tic -x. That doesn't matter though because whether or not emacs detects "RGB", direct colour availability depends on the $TERM value beginning with "xterm-" (and, curiously, that TN_max_colors is valid --- if TERM=notworking an invalid TN_max_colors is ignored). Creating the file lisp/term/notworking.el had no apparent effect. That's it. Nothing important follows. > > Notably TS_orig_pair is printed here as an empty string in all cases > > (NULL would print as "(null)"? and there there are no extra spaces), > > created by requesting the op capability which *is* defined in > > xterm-direct. > > > > $ infocmp dumb xterm-direct | grep op: > > op: NULL, '\E[39;49m'. > > I don't think I see why TS_orig_pair's value is relevant to the issue > at hand. Can you explain why you are interested in it? Immediately prior to the TERMINFO block is this code: /* SVr4/ANSI color support. If "op" isn't available, don't support color because we can't switch back to the default foreground and background. */ tty->TS_orig_pair = tgetstr ("op", address); if (tty->TS_orig_pair) { tty->TS_set_foreground = tgetstr ("AF", address); tty->TS_set_background = tgetstr ("AB", address); if (!tty->TS_set_foreground) { /* SVr4. */ tty->TS_set_foreground = tgetstr ("Sf", address); tty->TS_set_background = tgetstr ("Sb", address); } tty->TN_max_colors = tgetnum ("Co"); #ifdef TERMINFO { ... } #endif When I originally patched term.c (I've had enough of using one terminal to debug another terminal that is running its own terminal) I included the print statements at the end of the TERMINFO block and the debug file did not get created. I moved it down to above the next '}' line which appears to correspond to the test of tty->TS_orig_pair to no avail (the combination of C, CPP and indentation is hard to follow) then I moved it out of that block and began the test which got the results I sent last night. Probing it again with coffee it appears that TS_orig_pair is given a value and the block is entered so I expect I misread something when writing the patch. That reveals a misunderstanding, again because I assumed a cat would be enough. The pair line (printing tty->TS_orig_pair) does in fact have a value: 1b 5b 33 39 3b 34 39 6d which is the correct SGR control and would obviously not have an appearance when sent to a terminal. It is in the debug file. Focusing on tty->termcap_strings_buffer appears to be another red herring. I was trying to figure out where tgetstr is getting its data from and address née area née tty->termcap_strings_buffer was the obvious candidate. Cheers, Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-08 11:36 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-08 12:22 ` Eli Zaretskii 2024-03-08 14:23 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-08 12:22 UTC (permalink / raw) To: chohag; +Cc: 69598 > From: chohag@jtan.com > cc: chohag@jtan.com, 69598@debbugs.gnu.org > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > message dated "Fri, 08 Mar 2024 09:11:03 +0200." > Date: Fri, 08 Mar 2024 11:36:18 +0000 > > Eli Zaretskii writes: > > > From: chohag@jtan.com > > > cc: chohag@jtan.com, 69598@debbugs.gnu.org > > > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > > > message dated "Thu, 07 Mar 2024 22:10:30 +0200." > > > Date: Thu, 07 Mar 2024 21:45:44 +0000 > > > > Thanks. These results AFAIU indicate that there's no problem with the > > terminfo level: the number of colors known to term.c (printed as max:) > > is the same no matter how you start Emacs. Do you agree with this > > conclusion? > > That does appear to be the case. In fact the number of colours is > set by 'tty->TN_max_colors = tgetnum ("Co")' and then set again > when the RGB or other direct-colour capability is detected. That's correct, but why is that an issue? What I saw in your debugging print-outs is that the value of tty->TN_max_colors was the same in all of the 3 configurations you tried. Since your changes printed this value after the code that determines how many colors are supported, I concluded that terminal initialization computed the same number of colors in all 3 cases. Did I miss something? > Unfortunately the information from terminfo is sometimes ignored. Why "unfortunately", if the result is the correct number of colors? > > I think the problem is that if the terminal's name doesn't begin with > > "xterm", Emacs does not load lisp/term/xterm.el, which defines the > > colors that will be available. So please copy lisp/term/xterm.el to a > > file named lisp/term/notworking.el, and then start Emacs with > > TERM=notworking and see if that solves the problem. That is, arrange > > for a lisp/term/NAME.el file to be a copy of xterm.el when you invoke > > Emacs with TERM=NAME. > > If we're at the point where the terminal's name matters one jot, > that is to say when the value of $TERM is used for _anything_ other > than selecting a terminfo entry, that is a bug. It is not necessarily a bug. Emacs needs to be able to convert X-style color names to escape sequences it should send to the terminal to produce a similar color. That conversion is specific to the terminal, so Emacs relies on the name of the terminal to set up the conversion. We could have a new feature whereby, if terminfo redirected to a different entry, Emacs would load the lisp/term file of the redirected entry. But that'd be a new feature that AFAIU never existed in Emacs. In any case, I suggest to delay the discussion of what should be the solution to the issue until we understand the issue completely. AFAIU, we are not there yet, see above. > I don't know how emacs finds files when it's not installed so I > will be thorough: > > $ cd ~/src/emacs-29.2 > $ cp lisp/term/xterm.el lisp/term/notworking.el This is not enough. You need also to change this line at the end of notworking.el: (provide 'term/xterm) to say (provide 'term/notworking) Sorry I didn't mention this before. > While I was setting up this test I'd forgotten that I changed the > value in each of the three assignments to TN_max_colors in order > to confirm which condition inside the TERMINFO block was satisfied. > > Before correcting that, running with TERM=notworking reported 8 > colours but with TERM=xterm-direct reported 16 (the extra 8 are all > black) and said in *Messages*: xterm-register-default-colors: > Unsupported number of xterm colors (16777214). > > The value ending in 4 pointed to the tigetflag("RGB") block which > reminded me of the -x option to tic telling it to accept non-standard > capabilities (of which RGB is one) and I thought perhaps the option > was also necessary when creating an entry which uses an entry > which has non-standard capabilities. It turns out that it is and > that notworking was lacking the extra capabilities. > > So I got excited that maybe I'd found the problem and it was in > fact bad terminfo data. I did the whole round of tests again but > this time using "tic -x new.info" instead of plain tic (confirmed > with infocmp -x that the extended capabilities exist in the new > terminfo entries notworking and xterm-working). > > The result was identical. Sorry, I don't think I understand the above. In particular, I don't see the value 16777214 anywhere in term.c. What did I miss? > Summary: > > My generated terminfo files were wrong. I needed to use tic -x. > > That doesn't matter though because whether or not emacs detects > "RGB", direct colour availability depends on the $TERM value beginning > with "xterm-" (and, curiously, that TN_max_colors is valid --- if > TERM=notworking an invalid TN_max_colors is ignored). Is that a fact or is it a conclusion from what you observed? If the latter, I think we still have stuff to test, see above. > Creating the file lisp/term/notworking.el had no apparent effect. See above: the creation was incomplete. Thanks. ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-08 12:22 ` Eli Zaretskii @ 2024-03-08 14:23 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 15:09 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-08 14:23 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 69598, chohag Eli Zaretskii writes: > > > Thanks. These results AFAIU indicate that there's no problem with the > > > terminfo level: the number of colors known to term.c (printed as max:) > > > is the same no matter how you start Emacs. Do you agree with this > > > conclusion? > > > > That does appear to be the case. In fact the number of colours is > > set by 'tty->TN_max_colors = tgetnum ("Co")' and then set again > > when the RGB or other direct-colour capability is detected. > > That's correct, but why is that an issue? What I saw in your > debugging print-outs is that the value of tty->TN_max_colors was the It's not an issue, just a description of where the value comes from. > > Unfortunately the information from terminfo is sometimes ignored. > > Why "unfortunately", if the result is the correct number of colors? It's unfortunate that emacs (apparently) discards what it's told in favour of heuristics. Especially as the heuristics appear to draw the wrong conclusion. > > If we're at the point where the terminal's name matters one jot, > > that is to say when the value of $TERM is used for _anything_ other > > than selecting a terminfo entry, that is a bug. > > It is not necessarily a bug. Emacs needs to be able to convert It's a problem inasmuch as it is a violation of the terminfo protocol and takes a sharp turn into non-standards territory. Perhaps bug is not the best term. It's certainly a red flag. > X-style color names to escape sequences it should send to the terminal > to produce a similar color. That conversion is specific to the > terminal, so Emacs relies on the name of the terminal to set up the > conversion. Hmm. If only there was some sort of database that described terminal-specific capabilities and features and how to use them. It could also ensure the programs were portable, even to terminals that haven't been created yet! > We could have a new feature whereby, if terminfo redirected to a > different entry, Emacs would load the lisp/term file of the redirected > entry. But that'd be a new feature that AFAIU never existed in Emacs. My understanding is that the inclusion by terminfo is invisible to programs which use it. Being xterm-specific leads you down the path of a term/xxx.el file for each terminal which has non-standard features that you wish to use. You are correct to note that this part of the discussion is academic until the fault is found. > This is not enough. You need also to change this line at the end of > notworking.el: > > (provide 'term/xterm) > > to say > > (provide 'term/notworking) > > Sorry I didn't mention this before. No worries. It still makes no difference though unfortunately. But to be thorough I changed all instances of xterm in that notworking.el to notworking and that did work. After a few minutes I isolated the change to renaming terminal-init-xterm to terminal-init-notworking. In fact that's the only change that needs to be made: This diff is sufficient: --- lisp/term/xterm.el Sat Jan 6 12:56:30 2024 +++ lisp/term/notworking.el Fri Mar 8 14:00:28 2024 @@ -913,7 +913,7 @@ ;; We likewise unconditionally enable support for focus tracking. (xterm--init-focus-tracking)) -(defun terminal-init-xterm () +(defun terminal-init-notworking () "Terminal initialization function for xterm." (unwind-protect (progn I was actually a little surprised when I changed the provide line back to 'term/xterm and it continued to work but this agrees with lisp/term/README. That explains why the workaround of using a terminal name beginning with xterm- works. It doesn't explain why that is necessary and using setb24/setf24, the RGB capability and/or COLORTERM=truecolor is insufficient (ie. it doesn't explain why the terminal name _matters_). I didn't test Tc. I don't think it would make a difference. > > Before correcting that, running with TERM=notworking reported 8 > > colours but with TERM=xterm-direct reported 16 (the extra 8 are all > > black) and said in *Messages*: xterm-register-default-colors: > > Unsupported number of xterm colors (16777214). > > > > ... > > Sorry, I don't think I understand the above. In particular, I don't > see the value 16777214 anywhere in term.c. What did I miss? There were two things. First that even though the first time I ran the test last night my compiled test terminfo entries were incomplete, re-running the test with complete entries made no difference to the outcome of the test. Second and separately from any other tests, I had changed the three instances of 16777216 within the TERMINFO block to 16777213, 16777214 and 16777215 to see which path in that code was taken (tigetflag("RGB") in all cases). When I left the incorrect numbers in place, the *Messages* window complained that the number was invalid but only when TERM=xterm-direct or TERM=xterm-working. (And just confirmed that with a working notworking.el, TERM=notworking also complains about this problem) > > Summary: > > > > My generated terminfo files were wrong. I needed to use tic -x. > > > > That doesn't matter though because whether or not emacs detects > > "RGB", direct colour availability depends on the $TERM value beginning > > with "xterm-" (and, curiously, that TN_max_colors is valid --- if > > TERM=notworking an invalid TN_max_colors is ignored). > > Is that a fact or is it a conclusion from what you observed? If the > latter, I think we still have stuff to test, see above. This is a description of the symptoms I have observed. Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-08 14:23 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-08 15:09 ` Eli Zaretskii 2024-03-08 18:52 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-08 15:09 UTC (permalink / raw) To: chohag; +Cc: 69598 > From: chohag@jtan.com > cc: chohag@jtan.com, 69598@debbugs.gnu.org > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > message dated "Fri, 08 Mar 2024 14:22:52 +0200." > Date: Fri, 08 Mar 2024 14:23:26 +0000 > > > > If we're at the point where the terminal's name matters one jot, > > > that is to say when the value of $TERM is used for _anything_ other > > > than selecting a terminfo entry, that is a bug. > > > > It is not necessarily a bug. Emacs needs to be able to convert > > It's a problem inasmuch as it is a violation of the terminfo protocol > and takes a sharp turn into non-standards territory. Perhaps bug > is not the best term. It's certainly a red flag. We only do that for handling the colors, and I'm not sure I can see how terminfo can help us here. terminfo knows how many colors a terminal supports and which escape sequences to use for that, but it doesn't know which colors will be produced, in terms of the RGB triplets. So Emacs needs something else to support colors on text terminals, and that something comes from the lisp/term/ files. Those files also set up Emacs for various non-standard function keys produced by the terminal, and support other features, like copy/paste etc. So these files are not a bug, they are the foundation of several important features in Emacs. > > X-style color names to escape sequences it should send to the terminal > > to produce a similar color. That conversion is specific to the > > terminal, so Emacs relies on the name of the terminal to set up the > > conversion. > > Hmm. If only there was some sort of database that described > terminal-specific capabilities and features and how to use them. > It could also ensure the programs were portable, even to terminals > that haven't been created yet! Yes. But AFAIK there's no such database, so Emacs must use its own database. > > This is not enough. You need also to change this line at the end of > > notworking.el: > > > > (provide 'term/xterm) > > > > to say > > > > (provide 'term/notworking) > > > > Sorry I didn't mention this before. > > No worries. It still makes no difference though unfortunately. > > But to be thorough I changed all instances of xterm in that > notworking.el to notworking and that did work. After a few minutes > I isolated the change to renaming terminal-init-xterm to > terminal-init-notworking. In fact that's the only change that needs > to be made: This diff is sufficient: > > --- lisp/term/xterm.el Sat Jan 6 12:56:30 2024 > +++ lisp/term/notworking.el Fri Mar 8 14:00:28 2024 > @@ -913,7 +913,7 @@ > ;; We likewise unconditionally enable support for focus tracking. > (xterm--init-focus-tracking)) > > -(defun terminal-init-xterm () > +(defun terminal-init-notworking () > "Terminal initialization function for xterm." > (unwind-protect > (progn > > I was actually a little surprised when I changed the provide line > back to 'term/xterm and it continued to work but this agrees with > lisp/term/README. So you now agree with me that a terminal file in lisp/term is needed for this situation? Or are there issues that are still unaccounted for, as far as color support is concerned? > That explains why the workaround of using a terminal name beginning > with xterm- works. It doesn't explain why that is necessary and > using setb24/setf24, the RGB capability and/or COLORTERM=truecolor > is insufficient (ie. it doesn't explain why the terminal name > _matters_). I hope what I wrote above explains that. If you read the code in xterm.el, I think you will understand it. The other part of the color-related support is in lisp/term/tty-colors.el, btw. ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-08 15:09 ` Eli Zaretskii @ 2024-03-08 18:52 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 19:50 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-08 18:52 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 69598, chohag Eli Zaretskii writes: > > From: chohag@jtan.com > > cc: chohag@jtan.com, 69598@debbugs.gnu.org > > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > > message dated "Fri, 08 Mar 2024 14:22:52 +0200." > > Date: Fri, 08 Mar 2024 14:23:26 +0000 > > > > > > If we're at the point where the terminal's name matters one jot, > > > > that is to say when the value of $TERM is used for _anything_ other > > > > than selecting a terminfo entry, that is a bug. > > > > > > It is not necessarily a bug. Emacs needs to be able to convert > > > > It's a problem inasmuch as it is a violation of the terminfo protocol > > and takes a sharp turn into non-standards territory. Perhaps bug > > is not the best term. It's certainly a red flag. > > We only do that for handling the colors, and I'm not sure I can see > how terminfo can help us here. terminfo knows how many colors a > terminal supports and which escape sequences to use for that, but it > doesn't know which colors will be produced, in terms of the RGB > triplets. So Emacs needs something else to support colors on text Terminfo doesn't ``know'' what a terminal supports. Terminfo is the database and routines to look up capabality values. It's the applications which use terminfo that ``know'' what those capabilities do and, of course, many of them have a conventional use as described in terminfo(5). In order to describe capabilities of terminals which hadn't been made in 1980 it can be expanded with new capabilities. For example setb24 and setf24 are terminfo extensions which as near as I can tell originated in emacs to do exactly what you describe. https://www.gnu.org/software/emacs/manual/html_node/efaq/Colors-on-a-TTY.html mentions setb24 and setf24. In fact it's the only page on the web that I can find (apart those linking to it) with any record of these controls: Emacs 26.1 and later support direct color mode in terminals. If Emacs finds Terminfo capabilities `setb24' and `setf24', 24-bit direct color mode is used. The capability strings are expected to take one 24-bit pixel value as argument and transform the pixel to a string that can be used to send 24-bit colors to the terminal. Standard terminal definitions don't support these capabilities and therefore custom definition is needed. setb24=\E[48\:2\:\:%p1%{65536}%/%d\:%p1%{256}%/%{255}%&\ %d\:%p1%{255}%&%dm, setf24=\E[38\:2\:\:%p1%{65536}%/%d\:%p1%{256}%/%{255}%&\ %d\:%p1%{255}%&%dm, ``24-bit direct color mode is use'' with no further qualifications. It doesn't say that emacs also needs to have special knowledge about the terminal's model. The page continues with examples for further developments but in every case implies or outright says that certain capabilities are the only requirement, not specific models. > terminals, and that something comes from the lisp/term/ files. Those > files also set up Emacs for various non-standard function keys > produced by the terminal, and support other features, like copy/paste > etc. > > So these files are not a bug, they are the foundation of several > important features in Emacs. Their overriding what terminfo tells it is the buggy part. Terminfo says ``terminal foo has RGB or setf24 and setb24 capabilities and supports 16M colours'' but emacs disagrees. Incidentally, both of the examples you gave are already supported in some fashion by terminfo capabilities, for example: The kNNN capabilities map non-standard keys. bracketed+paste|xterm bracketed paste, BD=\E[?2004l, BE=\E[?2004h, PE=\E[201~, PS=\E[200~, > > > X-style color names to escape sequences it should send to the terminal > > > to produce a similar color. That conversion is specific to the > > > terminal, so Emacs relies on the name of the terminal to set up the > > > conversion. > > > > Hmm. If only there was some sort of database that described > > terminal-specific capabilities and features and how to use them. > > It could also ensure the programs were portable, even to terminals > > that haven't been created yet! > > Yes. But AFAIK there's no such database, so Emacs must use its own > database. I'm sorry, I was trying to use humour to point out that terminfo is that database. > So you now agree with me that a terminal file in lisp/term is needed > for this situation? Or are there issues that are still unaccounted > for, as far as color support is concerned? I don't agree. A new lisp/term file achieved the desired result, or rather the appearance of it, but for all of the wrong reasons: because the terminal lied about what it was. Looking at it another way if a user uses one of the terminals emacs supports but with certain features enabled or disabled by a custom terminfo entry there's nothing to say that the name of that entry has to follow any particular convention. A site prefix is a well established naming scheme so something like mycompany-footerm does not seem like an unreasonable name and nothing formally or informally in terminfo or termcap, or in emacs' documentation for that matter, suggests that it wouldn't work. Indeed I'm quite surprised to find this behaviour: emacs, running in xterm and informed it is running on a terminal identical to xterm, does not act the same as it does when that description comes under the *label* xterm. That's like the name of a variable influencing its contents. In short, if emacs only supports colour (or any other feature) on specific terminals it knows about in advance, it should say so and not imply that a terminfo or termcap capability is sufficient. As it stands this paragraph: If you think that your terminal supports colors, but Emacs won't use them, check the termcap entry for your display type for color-related capabilities. ... should be amended to say ``check if your terminal or emulator is one of these supported models: <list>'' but couldn't emacs believe and use what terminfo has told it, otherwise why did it ask? Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-08 18:52 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-08 19:50 ` Eli Zaretskii 2024-03-08 21:13 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 0 siblings, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-08 19:50 UTC (permalink / raw) To: chohag; +Cc: 69598, chohag > From: chohag@jtan.com > cc: chohag@jtan.com, 69598@debbugs.gnu.org > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > message dated "Fri, 08 Mar 2024 17:09:51 +0200." > Date: Fri, 08 Mar 2024 18:52:28 +0000 > > > So you now agree with me that a terminal file in lisp/term is needed > > for this situation? Or are there issues that are still unaccounted > > for, as far as color support is concerned? > > I don't agree. In that case, we will have to agree to disagree, sorry. > In short, if emacs only supports colour (or any other feature) on > specific terminals it knows about in advance, it should say so and > not imply that a terminfo or termcap capability is sufficient. Support for colors beyond the basic 8 is very specific to terminals. Emacs solves this problem by relying on the name of the terminal. So any new terminal has to abide by this protocol. That's how this stuff works in Emacs. ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-08 19:50 ` Eli Zaretskii @ 2024-03-08 21:13 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-09 7:18 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-08 21:13 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 69598, chohag Eli Zaretskii writes: > > From: chohag@jtan.com > > cc: chohag@jtan.com, 69598@debbugs.gnu.org > > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > > message dated "Fri, 08 Mar 2024 17:09:51 +0200." > > Date: Fri, 08 Mar 2024 18:52:28 +0000 > > > > > So you now agree with me that a terminal file in lisp/term is needed > > > for this situation? Or are there issues that are still unaccounted > > > for, as far as color support is concerned? > > > > I don't agree. > > In that case, we will have to agree to disagree, sorry. I don't, sorry. > > In short, if emacs only supports colour (or any other feature) on > > specific terminals it knows about in advance, it should say so and > > not imply that a terminfo or termcap capability is sufficient. > > Support for colors beyond the basic 8 is very specific to terminals. Hardware terminals which could not hope to display anything close to individual full colour rendition for the background and foreground of each character cell, without exhorbitant cost, did have myriad ways of expressing their limited colour support, true, back when such things were in the process of being invented, but out of all the software terminal emulators and programs since which claim full colour support, pretty much everything has followed the conventions laid down by xterm and/or VTE which follow or try to follow ECMA-48, T.416 and related formal and de-facto standards, such as describing their capabilities with termcap (1978) and/or terminfo (1981). > Emacs solves this problem by relying on the name of the terminal. So Everything except, apparently, Emacs. > any new terminal has to abide by this protocol. That's how this stuff > works in Emacs. That's what standards are for. Formal standards are those published by some organisation, de-facto standards are the conventions followed by a majority. Emacs' behaviour here is neither. I should be clear that I'm not trying to get something fixed --- I don't have a problem here. I can get along just fine but in my travels I discovered that emacs did not work as I expected and, on investigation, as it described itself and so I thought that GNU would like know that their flagship product and/or its documentation was deficient. I have provided more information at your request which I hope you find useful but I have no need or desire to pursue this matter. Matthew ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-08 21:13 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2024-03-09 7:18 ` Eli Zaretskii 2024-03-21 8:33 ` Eli Zaretskii 0 siblings, 1 reply; 20+ messages in thread From: Eli Zaretskii @ 2024-03-09 7:18 UTC (permalink / raw) To: chohag; +Cc: 69598 > From: chohag@jtan.com > cc: chohag@jtan.com, 69598@debbugs.gnu.org > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > message dated "Fri, 08 Mar 2024 21:50:01 +0200." > Date: Fri, 08 Mar 2024 21:13:27 +0000 > > I should be clear that I'm not trying to get something fixed --- I > don't have a problem here. I can get along just fine but in my > travels I discovered that emacs did not work as I expected and, on > investigation, as it described itself and so I thought that GNU > would like know that their flagship product and/or its documentation > was deficient. > > I have provided more information at your request which I hope you > find useful but I have no need or desire to pursue this matter. Thank you for your inputs. You may wish reading the file lisp/term/README, which attempts to explain this and other related stuff. Sorry I didn't remember we had this file and didn't point you to it in the first place. ^ permalink raw reply [flat|nested] 20+ messages in thread
* bug#69598: 29.2; colour support based on $TERM value not terminfo database 2024-03-09 7:18 ` Eli Zaretskii @ 2024-03-21 8:33 ` Eli Zaretskii 0 siblings, 0 replies; 20+ messages in thread From: Eli Zaretskii @ 2024-03-21 8:33 UTC (permalink / raw) To: chohag; +Cc: 69598-done > Cc: 69598@debbugs.gnu.org > Date: Sat, 09 Mar 2024 09:18:51 +0200 > From: Eli Zaretskii <eliz@gnu.org> > > > From: chohag@jtan.com > > cc: chohag@jtan.com, 69598@debbugs.gnu.org > > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org> > > message dated "Fri, 08 Mar 2024 21:50:01 +0200." > > Date: Fri, 08 Mar 2024 21:13:27 +0000 > > > > I should be clear that I'm not trying to get something fixed --- I > > don't have a problem here. I can get along just fine but in my > > travels I discovered that emacs did not work as I expected and, on > > investigation, as it described itself and so I thought that GNU > > would like know that their flagship product and/or its documentation > > was deficient. > > > > I have provided more information at your request which I hope you > > find useful but I have no need or desire to pursue this matter. > > Thank you for your inputs. You may wish reading the file > lisp/term/README, which attempts to explain this and other related > stuff. Sorry I didn't remember we had this file and didn't point you > to it in the first place. No further comments within 2 weeks, so I'm now closing this bug. ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2024-03-21 8:33 UTC | newest] Thread overview: 20+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-03-06 23:01 bug#69598: 29.2; colour support based on $TERM value not terminfo database chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 6:47 ` Eli Zaretskii 2024-03-07 17:32 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 17:47 ` Eli Zaretskii 2024-03-07 18:31 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 19:26 ` Eli Zaretskii 2024-03-07 19:59 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 20:10 ` Eli Zaretskii 2024-03-07 21:45 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-07 21:50 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 7:11 ` Eli Zaretskii 2024-03-08 11:36 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 12:22 ` Eli Zaretskii 2024-03-08 14:23 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 15:09 ` Eli Zaretskii 2024-03-08 18:52 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-08 19:50 ` Eli Zaretskii 2024-03-08 21:13 ` chohag--- via Bug reports for GNU Emacs, the Swiss army knife of text editors 2024-03-09 7:18 ` Eli Zaretskii 2024-03-21 8:33 ` Eli Zaretskii
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.