bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Hello there,

Summary: Support for 24-bit true colour, added in Emacs 28.1, breaks
         colours in Emacsen built without X support and running under
         GNU Screen (v4.08.00).

Steps to reproduce:

 1. Build Emacs >= 28.1 without X support.  The two configurations I have
    tested are below:

     ./configure\                          ./configure\
       --prefix=[…]\                          --prefix=[…]\
       --enable-check-lisp-object-type\       --enable-check-lisp-object-type\
       --disable-acl\                         --disable-acl\
       --without-dbus\                        --without-all\
       --without-gconf\                       --without-x\
       --without-gif\                         --with-file-notification=yes\
       --without-gsettings\                   --with-gnutls\
       --without-jpeg\                        --with-gpm\
       --without-modules\                     --with-json\
       --without-png\                         --with-mailutils\
       --without-rsvg\                        --with-modules\
       --without-selinux\                     --with-native-compilation\
       --without-sound\                       --with-libsystemd\
       --without-tiff\                        --with-small-ja-dic\
       --without-x\                           --with-sqlite3\
       --without-xpm\                         --with-threads\
                                              --with-xml2\
                                              --with-zlib\

 2. Launch a terminal program (e.g. GNOME Terminal) and run GNU Screen
    (bypassing any .screenrc):

     $ touch foo
     $ screen -c foo

 3. Run Emacs with 24-bit true colour active and observe the broken colours:

     $ COLORTERM=truecolor ./src/emacs -Q

    Screenshot: https://download.sebyte.me/misc/truecolor-active.png

 4. Run Emacs with 24-bit true colour inactive and observe the correct colours:

     $ COLORTERM= ./src/emacs -Q

    Screenshot: https://download.sebyte.me/misc/truecolor-inactive.png

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Sebastian Tennant <sdt@sebyte.me>
> Date: Fri, 17 Mar 2023 09:41:36 +0000
> 
> Summary: Support for 24-bit true colour, added in Emacs 28.1, breaks
>          colours in Emacsen built without X support and running under
>          GNU Screen (v4.08.00).
> 
> Steps to reproduce:
> 
>  1. Build Emacs >= 28.1 without X support.  The two configurations I have
>     tested are below:
> 
>      ./configure\                          ./configure\
>        --prefix=[…]\                          --prefix=[…]\
>        --enable-check-lisp-object-type\       --enable-check-lisp-object-type\
>        --disable-acl\                         --disable-acl\
>        --without-dbus\                        --without-all\
>        --without-gconf\                       --without-x\
>        --without-gif\                         --with-file-notification=yes\
>        --without-gsettings\                   --with-gnutls\
>        --without-jpeg\                        --with-gpm\
>        --without-modules\                     --with-json\
>        --without-png\                         --with-mailutils\
>        --without-rsvg\                        --with-modules\
>        --without-selinux\                     --with-native-compilation\
>        --without-sound\                       --with-libsystemd\
>        --without-tiff\                        --with-small-ja-dic\
>        --without-x\                           --with-sqlite3\
>        --without-xpm\                         --with-threads\
>                                               --with-xml2\
>                                               --with-zlib\
> 
>  2. Launch a terminal program (e.g. GNOME Terminal) and run GNU Screen
>     (bypassing any .screenrc):
> 
>      $ touch foo
>      $ screen -c foo
> 
>  3. Run Emacs with 24-bit true colour active and observe the broken colours:
> 
>      $ COLORTERM=truecolor ./src/emacs -Q
> 
>     Screenshot: https://download.sebyte.me/misc/truecolor-active.png
> 
>  4. Run Emacs with 24-bit true colour inactive and observe the correct colours:
> 
>      $ COLORTERM= ./src/emacs -Q
> 
>     Screenshot: https://download.sebyte.me/misc/truecolor-inactive.png

Thanks for the report, but it lacks several crucial details for us to
investigate the problem.

First, if you can build the latest emacs-29 branch of the Emacs Git
repository, please try that and tell whether the problem persists or
have been solved in the meantime.

If the problem persists in Emacs 29, then please tell why you need to
use COLORTERM=truecolor at all.  Emacs uses that as fallback, in case
all the other known methods of specifying true color via terminfo
didn't work.  This fallback relies on an assumption regarding the
commands to send to the terminal to turn on and off the colors, see
the file term.c in the Emacs source tree around line 4160.  The other
known methods of specifying support for true color use
'setf24'/'setb24' capabilities, or the 'RGB' flag is set by terminfo.

So setting COLORTERM=truecolor is the responsibility of the user: the
user should _only_ set it if the text-mode terminal actually supports
true color using the commands Emacs expects to work in that case, but
Emacs cannot detect that without COLORTERM=truecolor being set.  Is
that your case?

Finally, please show the display produced by "M-x list-colors-display"
in both cases: when COLORTERM=truecolor is and isn't set.  It is
important for us to know how many colors Emacs uses in each situation.

Thanks.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Fri, 17 Mar 2023 14:15:34 +0200, Eli Zaretskii <eliz@gnu.org> said:
    Eli> If the problem persists in Emacs 29, then please tell why you need to
    Eli> use COLORTERM=truecolor at all.  Emacs uses that as fallback, in case
    Eli> all the other known methods of specifying true color via terminfo
    Eli> didn't work.  This fallback relies on an assumption regarding the
    Eli> commands to send to the terminal to turn on and off the colors, see
    Eli> the file term.c in the Emacs source tree around line 4160.  The other
    Eli> known methods of specifying support for true color use
    Eli> 'setf24'/'setb24' capabilities, or the 'RGB' flag is set by terminfo.

    Eli> So setting COLORTERM=truecolor is the responsibility of the user: the
    Eli> user should _only_ set it if the text-mode terminal actually supports
    Eli> true color using the commands Emacs expects to work in that case, but
    Eli> Emacs cannot detect that without COLORTERM=truecolor being set.  Is
    Eli> that your case?

    Eli> Finally, please show the display produced by "M-x list-colors-display"
    Eli> in both cases: when COLORTERM=truecolor is and isn't set.  It is
    Eli> important for us to know how many colors Emacs uses in each situation.

For reasons unknown to me, I actually have COLORTERM=truecolor set in
my environment, which tickles this issue. But the only reason it does
so is because under screen by default I get
TERM=screen.xterm-256color. If I do

TERM=xterm-256color src/emacs -Q -nw

then I get 24bit colour (according to `display-color-cells')

I guess we could drop the 'screen.' prefix in
`init_display_interactive', although that does feel like a hack.

Robert
-- 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Sebastian Tennant <sdt@sebyte.me>,  62237@debbugs.gnu.org
> Date: Fri, 17 Mar 2023 16:39:19 +0100
> 
> For reasons unknown to me, I actually have COLORTERM=truecolor set in
> my environment, which tickles this issue. But the only reason it does
> so is because under screen by default I get
> TERM=screen.xterm-256color. If I do
> 
> TERM=xterm-256color src/emacs -Q -nw
> 
> then I get 24bit colour (according to `display-color-cells')
> 
> I guess we could drop the 'screen.' prefix in
> `init_display_interactive', although that does feel like a hack.

Why not in screen.el?

Is this screen.FOO format documented somewhere in screen's
documentation?

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Fri, 17 Mar 2023 18:30:23 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Sebastian Tennant <sdt@sebyte.me>,  62237@debbugs.gnu.org
    >> Date: Fri, 17 Mar 2023 16:39:19 +0100
    >> 
    >> For reasons unknown to me, I actually have COLORTERM=truecolor set in
    >> my environment, which tickles this issue. But the only reason it does
    >> so is because under screen by default I get
    >> TERM=screen.xterm-256color. If I do
    >> 
    >> TERM=xterm-256color src/emacs -Q -nw
    >> 
    >> then I get 24bit colour (according to `display-color-cells')
    >> 
    >> I guess we could drop the 'screen.' prefix in
    >> `init_display_interactive', although that does feel like a hack.

    Eli> Why not in screen.el?

Because the 24 bit colour detection is done in init_tty. By the time
we get to screen.el itʼs too late.

    Eli> Is this screen.FOO format documented somewhere in screen's
    Eli> documentation?

Yes:

       When screen tries to figure out a terminal name for  itself,  it  first
       looks  for an entry named "screen.<term>", where <term> is the contents
       of your $TERM variable.  If no such entry exists, screen tries "screen"
       (or  "screen-w"  if  the terminal is wide (132 cols or more)).  If even
       this entry cannot be found, "vt100" is used as a substitute.

       The idea is that if you have a terminal which doesn't support an impor‐
       tant  feature  (e.g.  delete  char or clear to EOS) you can build a new
       termcap/terminfo entry for screen (named "screen.<dumbterm>") in  which
       this  capability  has been disabled. If this entry is installed on your
       machines you are able to do a rlogin and still keep the  correct  term‐
-->    cap/terminfo  entry.  The terminal name is put in the $TERM variable of
       all new windows.  Screen also sets the $TERMCAP variable reflecting the
       capabilities of the virtual terminal emulated. Notice that, however, on
       machines using the terminfo database this variable has no effect.  Fur‐
       thermore, the variable $WINDOW is set to the window number of each win‐
       dow.

Robert
-- 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
> Date: Fri, 17 Mar 2023 18:44:19 +0100
> 
>     Eli> Is this screen.FOO format documented somewhere in screen's
>     Eli> documentation?
> 
> Yes:
> 
>        When screen tries to figure out a terminal name for  itself,  it  first
>        looks  for an entry named "screen.<term>", where <term> is the contents
>        of your $TERM variable.  If no such entry exists, screen tries "screen"
>        (or  "screen-w"  if  the terminal is wide (132 cols or more)).  If even
>        this entry cannot be found, "vt100" is used as a substitute.
> 
>        The idea is that if you have a terminal which doesn't support an impor‐
>        tant  feature  (e.g.  delete  char or clear to EOS) you can build a new
>        termcap/terminfo entry for screen (named "screen.<dumbterm>") in  which
>        this  capability  has been disabled. If this entry is installed on your
>        machines you are able to do a rlogin and still keep the  correct  term‐
> -->    cap/terminfo  entry.  The terminal name is put in the $TERM variable of
>        all new windows.  Screen also sets the $TERMCAP variable reflecting the
>        capabilities of the virtual terminal emulated. Notice that, however, on
>        machines using the terminfo database this variable has no effect.  Fur‐
>        thermore, the variable $WINDOW is set to the window number of each win‐
>        dow.

This seems to tell how 'screen' figures out the terminal name, not how
it sets TERM.  I asked who and why sets TERM to screen.SOMETHING.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Fri, 17 Mar 2023 20:55:23 +0200, Eli Zaretskii <eliz@gnu.org> said:
    >> machines you are able to do a rlogin and still keep the  correct  term‐
    --> cap/terminfo  entry.  The terminal name is put in the $TERM variable of
    >> all new windows.  Screen also sets the $TERMCAP variable reflecting the
    >> capabilities of the virtual terminal emulated. Notice that, however, on
    >> machines using the terminfo database this variable has no effect.  Fur‐
    >> thermore, the variable $WINDOW is set to the window number of each win‐
    >> dow.

    Eli> This seems to tell how 'screen' figures out the terminal name, not how
    Eli> it sets TERM.  I asked who and why sets TERM to screen.SOMETHING.

screen does: "The terminal name is put in the $TERM variable of all new windows."

Robert
-- 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
> Date: Sat, 18 Mar 2023 10:05:49 +0100
> 
> >>>>> On Fri, 17 Mar 2023 20:55:23 +0200, Eli Zaretskii <eliz@gnu.org> said:
>     >> machines you are able to do a rlogin and still keep the  correct  term‐
>     --> cap/terminfo  entry.  The terminal name is put in the $TERM variable of
>     >> all new windows.  Screen also sets the $TERMCAP variable reflecting the
>     >> capabilities of the virtual terminal emulated. Notice that, however, on
>     >> machines using the terminfo database this variable has no effect.  Fur‐
>     >> thermore, the variable $WINDOW is set to the window number of each win‐
>     >> dow.
> 
>     Eli> This seems to tell how 'screen' figures out the terminal name, not how
>     Eli> it sets TERM.  I asked who and why sets TERM to screen.SOMETHING.
> 
> screen does: "The terminal name is put in the $TERM variable of all new windows."

So how did Emacs ever succeed to work inside screen, then?  AFAIK, we
never supported this form of TERM's value.  Is this something
relatively new?

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Sat, 18 Mar 2023 11:09:10 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
    >> Date: Sat, 18 Mar 2023 10:05:49 +0100
    >> 
    >> >>>>> On Fri, 17 Mar 2023 20:55:23 +0200, Eli Zaretskii <eliz@gnu.org> said:
    >> >> machines you are able to do a rlogin and still keep the  correct  term‐
    --> cap/terminfo  entry.  The terminal name is put in the $TERM variable of
    >> >> all new windows.  Screen also sets the $TERMCAP variable reflecting the
    >> >> capabilities of the virtual terminal emulated. Notice that, however, on
    >> >> machines using the terminfo database this variable has no effect.  Fur‐
    >> >> thermore, the variable $WINDOW is set to the window number of each win‐
    >> >> dow.
    >> 
    Eli> This seems to tell how 'screen' figures out the terminal name, not how
    Eli> it sets TERM.  I asked who and why sets TERM to screen.SOMETHING.
    >> 
    >> screen does: "The terminal name is put in the $TERM variable of all new windows."

    Eli> So how did Emacs ever succeed to work inside screen, then?  AFAIK, we
    Eli> never supported this form of TERM's value.  Is this something
    Eli> relatively new?

I donʼt know, Iʼm a tmuxian ;-)

Note that emacs works fine, itʼs just the colours that are off, and
people sshʼing in to use screen would tend to set TERM themselves
anyway.

Robert
-- 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
> Date: Sat, 18 Mar 2023 11:02:16 +0100
> 
> >>>>> On Sat, 18 Mar 2023 11:09:10 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     >> From: Robert Pluim <rpluim@gmail.com>
>     >> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
>     >> Date: Sat, 18 Mar 2023 10:05:49 +0100
>     >> 
>     >> >>>>> On Fri, 17 Mar 2023 20:55:23 +0200, Eli Zaretskii <eliz@gnu.org> said:
>     >> >> machines you are able to do a rlogin and still keep the  correct  term‐
>     --> cap/terminfo  entry.  The terminal name is put in the $TERM variable of
>     >> >> all new windows.  Screen also sets the $TERMCAP variable reflecting the
>     >> >> capabilities of the virtual terminal emulated. Notice that, however, on
>     >> >> machines using the terminfo database this variable has no effect.  Fur‐
>     >> >> thermore, the variable $WINDOW is set to the window number of each win‐
>     >> >> dow.
>     >> 
>     Eli> This seems to tell how 'screen' figures out the terminal name, not how
>     Eli> it sets TERM.  I asked who and why sets TERM to screen.SOMETHING.
>     >> 
>     >> screen does: "The terminal name is put in the $TERM variable of all new windows."
> 
>     Eli> So how did Emacs ever succeed to work inside screen, then?  AFAIK, we
>     Eli> never supported this form of TERM's value.  Is this something
>     Eli> relatively new?
> 
> I donʼt know, Iʼm a tmuxian ;-)
> 
> Note that emacs works fine, itʼs just the colours that are off, and
> people sshʼing in to use screen would tend to set TERM themselves
> anyway.

Then I guess we should install your proposed fix in init_tty.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Sat, 18 Mar 2023 12:37:30 +0200, Eli Zaretskii <eliz@gnu.org> said:

    Eli> Then I guess we should install your proposed fix in init_tty.

In emacs-29? That seems a bit radical. Patch below in any case

I guess we could do something with not checking COLORTERM under screen
instead.

Robert
-- 

diff --git i/src/dispnew.c w/src/dispnew.c
index 87ec83acdf3..f165c604ae9 100644
--- i/src/dispnew.c
+++ w/src/dispnew.c
@@ -6586,6 +6586,21 @@ init_display_interactive (void)
       exit (1);
     }
 
+  if (!NILP (Vterm_strip_prefixes))
+    {
+      Lisp_Object prefixes = Vterm_strip_prefixes;
+      FOR_EACH_TAIL (prefixes)
+	{
+	  char *c_prefix = SSDATA (XCAR (prefixes));
+	  int len = strlen (c_prefix);
+	  if (strncmp (terminal_type, c_prefix, len) == 0)
+	    {
+	      terminal_type += len;
+	      break;
+	    }
+	}
+    }
+
   {
     struct terminal *t;
     struct frame *f = XFRAME (selected_frame);
@@ -6817,6 +6832,11 @@ syms_of_display (void)
 Possible values are t (below the tool bar), nil (above the tool bar).
 This option affects only builds where the tool bar is not external.  */);
 
+  DEFVAR_LISP ("term-strip-prefixes", Vterm_strip_prefixes,
+	       doc: /* List of prefixes to try to strip from the TERM environment variable.
+This will strip the first matching prefix only.  */);
+  Vterm_strip_prefixes = Fcons (build_string ("screen."), Qnil);
+
   pdumper_do_now_and_after_load (syms_of_display_for_pdumper);
 }
 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
> Date: Sat, 18 Mar 2023 12:44:45 +0100
> 
> >>>>> On Sat, 18 Mar 2023 12:37:30 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> Then I guess we should install your proposed fix in init_tty.
> 
> In emacs-29? That seems a bit radical. Patch below in any case

Yes, I think in emacs-29.  Why "radical"?

> I guess we could do something with not checking COLORTERM under screen
> instead.

That's a separate issue, from where I stand.  Users can unset
COLORTERM, but their true terminal type will still be "hidden" behind
the "screen." prefix, won't it?  The terminal type is about more than
just the colors.  Or does terminfo know about this "screen." business?

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Sat, 18 Mar 2023 15:29:52 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
    >> Date: Sat, 18 Mar 2023 12:44:45 +0100
    >> 
    >> >>>>> On Sat, 18 Mar 2023 12:37:30 +0200, Eli Zaretskii <eliz@gnu.org> said:
    >> 
    Eli> Then I guess we should install your proposed fix in init_tty.
    >> 
    >> In emacs-29? That seems a bit radical. Patch below in any case

    Eli> Yes, I think in emacs-29.  Why "radical"?

Changing the interpretation of the userʼs TERM seems pretty radical to
me, even if it will tend to improve usersʼ experience.

    >> I guess we could do something with not checking COLORTERM under screen
    >> instead.

    Eli> That's a separate issue, from where I stand.  Users can unset
    Eli> COLORTERM, but their true terminal type will still be "hidden" behind
    Eli> the "screen." prefix, won't it?  The terminal type is about more than
    Eli> just the colors.  Or does terminfo know about this "screen." business?

I have both a 'screen.xterm-256color' and a 'xterm-256color' terminfo
file. I donʼt think terminfo does any prefix stripping, as thereʼs a
whole bunch of screen.$TERM files, which would be unnecessary if
stripping were happening.

Perhaps the best thing to do is put an entry in etc/PROBLEMS?

Robert
-- 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Quoth Robert Pluim <rpluim@gmail.com>
on Mon, 20 Mar 2023 09:36:39 +0100:
>> […]
>> Yes, I think in emacs-29.  Why "radical"?
>
> Changing the interpretation of the userʼs TERM seems pretty radical
> to me, even if it will tend to improve usersʼ experience.
>
>>> I guess we could do something with not checking COLORTERM under screen
>>> instead.
>
>> That's a separate issue, from where I stand.  Users can unset
>> COLORTERM, but their true terminal type will still be "hidden"
>> behind the "screen." prefix, won't it?  The terminal type is about
>> more than just the colors.  Or does terminfo know about this
>> "screen." business?
>
> I have both a 'screen.xterm-256color' and a 'xterm-256color'
> terminfo file. I donʼt think terminfo does any prefix stripping, as
> thereʼs a whole bunch of screen.$TERM files, which would be
> unnecessary if stripping were happening.

FWIW, I agree that stripping the 'screen.' prefix isn’t the correct
thing to do.  (Let's not forget that the issue seems to have bitten
only one person in well over a year).

> Perhaps the best thing to do is put an entry in etc/PROBLEMS?

How about something like this:

 Emacs >= 28.1 respects the value of environment variable COLORTERM
 and if it is set to 'truecolor', expects the terminal to support
 24-bit true colour.

 GNOME Terminal supports true colour and, unnecessarily, sets
 COLORTERM=truecolor to make this clear.

 GNU Screen 4.X does ̲n̲o̲t support true colour.  If you start Emacs at
 Screen start time (via an entry in your .screenrc), Emacs will
 inherit the environment and, because of COLORTERM=truecolor, assume
 the terminal, i.e. the screen, supports true colour.  This will
 result in broken colours in Emacs.

 The simplest solution is to unset COLORTERM in the environment before
 launching screen.

Or words to that effect.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Sebastian Tennant <sdt@sebyte.me>
> Cc: Eli Zaretskii <eliz@gnu.org>,  62237@debbugs.gnu.org
> Date: Mon, 20 Mar 2023 08:57:45 +0000
> 
> > I have both a 'screen.xterm-256color' and a 'xterm-256color'
> > terminfo file. I donʼt think terminfo does any prefix stripping, as
> > thereʼs a whole bunch of screen.$TERM files, which would be
> > unnecessary if stripping were happening.
> 
> FWIW, I agree that stripping the 'screen.' prefix isn’t the correct
> thing to do.  (Let's not forget that the issue seems to have bitten
> only one person in well over a year).

Why did this bite you?  Don't you have those screen.FOO files?  If
not, why not?

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
> Date: Mon, 20 Mar 2023 09:36:39 +0100
> 
> >>>>> On Sat, 18 Mar 2023 15:29:52 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     >> From: Robert Pluim <rpluim@gmail.com>
>     >> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
>     >> Date: Sat, 18 Mar 2023 12:44:45 +0100
>     >> 
>     >> >>>>> On Sat, 18 Mar 2023 12:37:30 +0200, Eli Zaretskii <eliz@gnu.org> said:
>     >> 
>     Eli> Then I guess we should install your proposed fix in init_tty.
>     >> 
>     >> In emacs-29? That seems a bit radical. Patch below in any case
> 
>     Eli> Yes, I think in emacs-29.  Why "radical"?
> 
> Changing the interpretation of the userʼs TERM seems pretty radical to
> me, even if it will tend to improve usersʼ experience.

If screen.FOO is not recognizable while FOO is, then we cannot
possibly break anything by this change, however radical.  (But it
sounds like I misunderstood what is going on here, see below.)

>     >> I guess we could do something with not checking COLORTERM under screen
>     >> instead.
> 
>     Eli> That's a separate issue, from where I stand.  Users can unset
>     Eli> COLORTERM, but their true terminal type will still be "hidden" behind
>     Eli> the "screen." prefix, won't it?  The terminal type is about more than
>     Eli> just the colors.  Or does terminfo know about this "screen." business?
> 
> I have both a 'screen.xterm-256color' and a 'xterm-256color' terminfo
> file. I donʼt think terminfo does any prefix stripping, as thereʼs a
> whole bunch of screen.$TERM files, which would be unnecessary if
> stripping were happening.

Are the screen.$TERM files different from the corresponding $TERM
ones?  If so, what are the differences?

And if screen.$TERM files are available in the terminfo DB, then why
did you suggest to do something in init_tty in the first place?

Also, what about the lisp/term/ files -- are we loading the right
files when TERM -s set to screen.SOMETHING? should we?

> Perhaps the best thing to do is put an entry in etc/PROBLEMS?

About what?  If screen.FOO files are available, then everything should
already work correctly OOTB, no?  Or what am I missing here?

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Mon, 20 Mar 2023 14:15:35 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Changing the interpretation of the userʼs TERM seems pretty radical to
    >> me, even if it will tend to improve usersʼ experience.

    Eli> If screen.FOO is not recognizable while FOO is, then we cannot
    Eli> possibly break anything by this change, however radical.  (But it
    Eli> sounds like I misunderstood what is going on here, see below.)

Iʼm sure I donʼt entirely understand whatʼs going on either

    >> >> I guess we could do something with not checking COLORTERM under screen
    >> >> instead.
    >> 
    Eli> That's a separate issue, from where I stand.  Users can unset
    Eli> COLORTERM, but their true terminal type will still be "hidden" behind
    Eli> the "screen." prefix, won't it?  The terminal type is about more than
    Eli> just the colors.  Or does terminfo know about this "screen." business?
    >> 
    >> I have both a 'screen.xterm-256color' and a 'xterm-256color' terminfo
    >> file. I donʼt think terminfo does any prefix stripping, as thereʼs a
    >> whole bunch of screen.$TERM files, which would be unnecessary if
    >> stripping were happening.

    Eli> Are the screen.$TERM files different from the corresponding $TERM
    Eli> ones?  If so, what are the differences?

$ infocmp screen.xterm-256color xterm-256color

comparing screen.xterm-256color to xterm-256color.
    comparing booleans.
	bce: F:T.
	bw: T:F.
	ccc: F:T.
    comparing numbers.
    comparing strings.
	initc: NULL, '\E]4;%p1%d;rgb\:%p2%{255}%*%{1000}%/%2.2X/%p3%{255}%*%{1000}%/%2.2X/%p4%{255}%*%{1000}%/%2.2X\E\\'.
	invis: NULL, '\E[8m'.
	kIC: NULL, '\E[2;2~'.
	kNXT: NULL, '\E[6;2~'.
	kPRV: NULL, '\E[5;2~'.
	kend: '\E[4~', '\EOF'.
	khome: '\E[1~', '\EOH'.
	kmous: '\E[M', '\E[<'.
	oc: NULL, '\E]104\007'.
	ritm: NULL, '\E[23m'.
	rs1: '\Ec', '\Ec\E]104\007'.
	sgr: '%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p5%t;2%;m', '%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p5%t;2%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m'.
	sitm: NULL, '\E[3m'.

    Eli> And if screen.$TERM files are available in the terminfo DB, then why
    Eli> did you suggest to do something in init_tty in the first place?

Becasue the behaviour is different (all under 'screen' in gnome-terminal):

1.
COLORTERM=truecolor TERM=screen.xterm-256color src/emacs -nw -Q etc/NEWS

(display-color-cells) => 16777216

but the display looks black & white, and `list-colors-display' looks
really off.

2.
COLORTERM=truecolor TERM=xterm-256color src/emacs -nw -Q etc/NEWS

(display-color-cells) => 16777216

The display looks more colourful, but `list-colors-display' is still
off.

3.
COLORTERM= TERM=screen.xterm-256color src/emacs -nw -Q etc/NEWS

(display-color-cells) => 256
The display is now colourful, and `list-colors-display' has 256
entries, but:

4.
COLORTERM= TERM=xterm-256color src/emacs -nw -Q etc/NEWS

This is like [3], but the colours are different.

[2] is a definite improvement over [1], but it would be nice if we
could get to [3] or [4] (my gnome-terminal may be too old for proper
truecolor support, itʼs 3.83.3 with VTE 0.62.3)

    Eli> Also, what about the lisp/term/ files -- are we loading the right
    Eli> files when TERM -s set to screen.SOMETHING? should we?

Weʼre running term/screen.el (which then runs xterm.el)
 
    >> Perhaps the best thing to do is put an entry in etc/PROBLEMS?

    Eli> About what?  If screen.FOO files are available, then everything should
    Eli> already work correctly OOTB, no?  Or what am I missing here?

If everything worked OOTB, then yes, but our handling of COLORTERM is
still problematic. If we could delay the 24bit colour support decision
until weʼre in lisp/term I think that would help.

Robert
-- 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
> Date: Mon, 20 Mar 2023 15:08:14 +0100
> 
> >>>>> On Mon, 20 Mar 2023 14:15:35 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     >> Perhaps the best thing to do is put an entry in etc/PROBLEMS?
> 
>     Eli> About what?  If screen.FOO files are available, then everything should
>     Eli> already work correctly OOTB, no?  Or what am I missing here?
> 
> If everything worked OOTB, then yes, but our handling of COLORTERM is
> still problematic. If we could delay the 24bit colour support decision
> until weʼre in lisp/term I think that would help.

So the only real problem is COLORTERM=truecolor, and if it is not set,
then everything works reasonably well?  If so, why is COLORTERM set in
this case?  Is it GNOME which sets it, or is it something else?

COLORTERM support was added because it reportedly helped in some
real-life cases.  If it turns out it gets in the way in other cases,
we need either find a way of detecting those problematic cases where
we process COLORTERM, or ask users to unset the variable if it causes
trouble.  I don't see how we could defer processing COLORTERM to
later, as knowing how many colors Emacs can work with is necessary
very early into startup; too many things will break or work
incorrectly if we defer that to later.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Mon, 20 Mar 2023 16:23:28 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
    >> Date: Mon, 20 Mar 2023 15:08:14 +0100
    >> 
    >> >>>>> On Mon, 20 Mar 2023 14:15:35 +0200, Eli Zaretskii <eliz@gnu.org> said:
    >> 
    >> >> Perhaps the best thing to do is put an entry in etc/PROBLEMS?
    >> 
    Eli> About what?  If screen.FOO files are available, then everything should
    Eli> already work correctly OOTB, no?  Or what am I missing here?
    >> 
    >> If everything worked OOTB, then yes, but our handling of COLORTERM is
    >> still problematic. If we could delay the 24bit colour support decision
    >> until weʼre in lisp/term I think that would help.

    Eli> So the only real problem is COLORTERM=truecolor, and if it is not set,
    Eli> then everything works reasonably well?  If so, why is COLORTERM set in
    Eli> this case?  Is it GNOME which sets it, or is it something else?

Itʼs either GNOME or gnome-terminal. xterm doesnʼt get it set.

    Eli> COLORTERM support was added because it reportedly helped in some
    Eli> real-life cases.  If it turns out it gets in the way in other cases,
    Eli> we need either find a way of detecting those problematic cases where
    Eli> we process COLORTERM, or ask users to unset the variable if it causes
    Eli> trouble.  I don't see how we could defer processing COLORTERM to
    Eli> later, as knowing how many colors Emacs can work with is necessary
    Eli> very early into startup; too many things will break or work
    Eli> incorrectly if we defer that to later.

OK. Still sounds like etc/PROBLEMS to me :-)

Robert
-- 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Quoth Eli Zaretskii <eliz@gnu.org>
on Mon, 20 Mar 2023 14:17:03 +0200:
>>> I have both a 'screen.xterm-256color' and a 'xterm-256color'
>>> terminfo file. I donʼt think terminfo does any prefix stripping,
>>> as thereʼs a whole bunch of screen.$TERM files, which would be
>>> unnecessary if stripping were happening.
>>
>> FWIW, I agree that stripping the 'screen.' prefix isn’t the correct
>> thing to do.  (Let's not forget that the issue seems to have bitten
>> only one person in well over a year).
>
> Why did this bite you?

My suggestion for an entry in etc/PROBLEMS:

 https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62237#80

is the best explanation I can come up with.

> Don't you have those screen.FOO files?

These are installed by Debian package ncurses-term:

 $ ls -l /usr/share/terminfo/s/screen.x*
 lrwxrwxrwx 1 root root   20 Jan  1  2021 /usr/share/terminfo/s/screen.xterm-new -> screen.xterm-xfree86
 -rw-r--r-- 1 root root 1607 Jan  1  2021 /usr/share/terminfo/s/screen.xterm-r6
 -rw-r--r-- 1 root root 3675 Jan  1  2021 /usr/share/terminfo/s/screen.xterm-xfree86

 https://packages.debian.org/bullseye/all/ncurses-term/filelist

and these by package ncurses-base:

 $ ls -l /lib/terminfo/s/screen.x*
 -rw-r--r-- 1 root root 3573 Jan  1  2021 /lib/terminfo/s/screen.xterm-256color

 https://packages.debian.org/bullseye/all/ncurses-base/filelist

I've no idea why file screen.xterm-256color belongs to a different
package (and is installed in a different directory).

In case it helps, there's a lengthy discussion about
screen.xterm-256color here:

 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854414

Quoth Eli Zaretskii <eliz@gnu.org>
on Mon, 20 Mar 2023 16:23:28 +0200:
>> If everything worked OOTB, then yes, but our handling of COLORTERM
>> is still problematic. If we could delay the 24bit colour support
>> decision until weʼre in lisp/term I think that would help.
>
> So the only real problem is COLORTERM=truecolor, and if it is not
> set, then everything works reasonably well?  If so, why is COLORTERM
> set in this case?

Why it is set remains unclear.

> Is it GNOME which sets it, or is it something else?

It is set by VTE:

 https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62237#71

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: sdt@sebyte.me,  62237@debbugs.gnu.org
> Date: Mon, 20 Mar 2023 15:51:27 +0100
> 
> >>>>> On Mon, 20 Mar 2023 16:23:28 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> COLORTERM support was added because it reportedly helped in some
>     Eli> real-life cases.  If it turns out it gets in the way in other cases,
>     Eli> we need either find a way of detecting those problematic cases where
>     Eli> we process COLORTERM, or ask users to unset the variable if it causes
>     Eli> trouble.  I don't see how we could defer processing COLORTERM to
>     Eli> later, as knowing how many colors Emacs can work with is necessary
>     Eli> very early into startup; too many things will break or work
>     Eli> incorrectly if we defer that to later.
> 
> OK. Still sounds like etc/PROBLEMS to me :-)

I have now added a PROBLEMS entry about this issue on the emacs-29
branch, and I'm closing this bug.

Thank you both for all the useful information about the various
aspects of the issue.  I tried to use all of that in the PROBLEMS
entry.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Quoth Eli Zaretskii <eliz@gnu.org>
on Sat, 18 Mar 2023 11:09:10 +0200:
>>>> […] machines you are able to do a rlogin and still keep the
>>>> correct termcap/terminfo entry.  The terminal name is put in the
>>>> $TERM variable of all new windows.  Screen also sets the $TERMCAP
>>>> variable reflecting the capabilities of the virtual terminal
>>>> emulated. Notice that, however, on machines using the terminfo
>>>> database this variable has no effect.  Furthermore, the variable
>>>> $WINDOW is set to the window number of each window.
>>>>
>>> This seems to tell how 'screen' figures out the terminal name, not
>>> how it sets TERM.  I asked who and why sets TERM to
>>> screen.SOMETHING.
>>>
>> screen does: "The terminal name is put in the $TERM variable of all
>> new windows."
>>
> So how did Emacs ever succeed to work inside screen, then?  AFAIK,
> we never supported this form of TERM's value.

If you discard the ‘screen.’ prefix by explicitly setting TERM in your
.screenrc; for example:

 term xterm-256color

it makes no difference, i.e. colours are still broken (when COLORTERM
is set to ‘truecolour’).

> Is this something relatively new?

I started using screen in 2011 and I'm pretty sure it was prepending
‘.screen’ then.

I'm not convinced this is the issue.  Remember, the colours were fine
under screen in Emacs 27.2.  There's something in the truecolor
support added in Emacs 28.1, specifically the respect paid to the
vaule of COLORTERM, that doesn't agree with screen.


It seems truecolor support was added to screen in July 2015:

 https://git.savannah.gnu.org/cgit/screen.git/log/?qt=grep&q=truecolor

Debian bullseye ships screen 4.8.0 which, according to:

 https://git.savannah.gnu.org/cgit/screen.git/

was released in 2000 which would suggest my screen does supprt
truecolor.  Oddly, my screen(1) manpage makes no mention of truecolor
support, yet various online manpages, for example:

 https://www.man7.org/linux/man-pages/man1/screen.1.html

document a ‘truecolor on|off’ configuraion option:

 truecolor [on|off]

 Enables truecolor support. Currently autodetection of truecolor
 support cannot be done reliably, as such it's left to user to
 enable. Default is off.  Known terminals that may support it are:
 iTerm2, Konsole, st.  Xterm includes support for truecolor escapes
 but converts them back to indexed 256 color space.

I've added the line:

 truecolor on

to my .screenrc and it doesn't barf, but it makes no diffference
either.  I shall investigate further.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Robert Pluim]

>>>>> On Sat, 18 Mar 2023 10:34:10 +0000, Sebastian Tennant <sdt@sebyte.me> said:

    Sebastian> Quoth Eli Zaretskii <eliz@gnu.org>
    Sebastian> on Sat, 18 Mar 2023 11:09:10 +0200:
    >>>>> […] machines you are able to do a rlogin and still keep the
    >>>>> correct termcap/terminfo entry.  The terminal name is put in the
    >>>>> $TERM variable of all new windows.  Screen also sets the $TERMCAP
    >>>>> variable reflecting the capabilities of the virtual terminal
    >>>>> emulated. Notice that, however, on machines using the terminfo
    >>>>> database this variable has no effect.  Furthermore, the variable
    >>>>> $WINDOW is set to the window number of each window.
    >>>>> 
    >>>> This seems to tell how 'screen' figures out the terminal name, not
    >>>> how it sets TERM.  I asked who and why sets TERM to
    >>>> screen.SOMETHING.
    >>>> 
    >>> screen does: "The terminal name is put in the $TERM variable of all
    >>> new windows."
    >>> 
    >> So how did Emacs ever succeed to work inside screen, then?  AFAIK,
    >> we never supported this form of TERM's value.

    Sebastian> If you discard the ‘screen.’ prefix by explicitly setting TERM in your
    Sebastian> .screenrc; for example:

    Sebastian>  term xterm-256color

    Sebastian> it makes no difference, i.e. colours are still broken (when COLORTERM
    Sebastian> is set to ‘truecolour’).

Hmm, I get 24-bit colour in that case. Presumably because screen +
gnome-terminal supports it.

Robert
-- 

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Quoth Robert Pluim <rpluim@gmail.com>
on Sat, 18 Mar 2023 12:38:32 +0100:
>> […]  If you discard the ‘screen.’ prefix by explicitly setting TERM
>> in your .screenrc; for example:
>>
>>  term xterm-256color
>>
>> it makes no difference, i.e. colours are still broken (when
>> COLORTERM is set to ‘truecolour’).
>
> Hmm, I get 24-bit colour in that case. Presumably because screen +
> gnome-terminal supports it.

It seems my version of screen was too old and truecolor is not
supported.

I've now built and installed screen ‘master’ (4.99) and the problem is
resolved, i.e. provided my .screenrc includes the line ‘truecolor on’,
colours work fine with COLORTERM set to ‘truecolor’, or not set at
all.

Sorry about all this.  Many months ago, I built screen from master and
ran it to see if it made a difference.  I concluded then that it
didn't and I think this must have been because I didn't actually
install the version I built.  This meant that the relevant entries
under /usr/lib/terminfo/s were not written/updated.

In any case, we've established one thing for certain, the version of
screen shipped by Debian bullseye (stable) does not support truecolor.
Most strange is the fact that, according to dpkg, the version shipped
is 4.8.0:

 $ dpkg -l | grep screen
 ii  screen  4.8.0-6  amd64 terminal multiplexer with VT100/ANSI…
             ^^^^^

but screen itself reports:

 $ screen --version
 Screen version 4.08.00 (GNU) 05-Feb-20
                ^^^^

which makes no sense at all!

According to:

 https://packages.debian.org/search?keywords=screen

Debian bookworm (the next stable release) will ship screen version
4.9.0.  I've downloaded the deb and grepped the manpage and the string
"truecolor" is nowhere to be found so it looks as if screen shipped by
Debian will lack support for truecolor for some years to come.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Sebastian Tennant <sdt@sebyte.me>
> Cc: Eli Zaretskii <eliz@gnu.org>,  62237@debbugs.gnu.org
> Date: Sat, 18 Mar 2023 15:01:53 +0000
> 
> I've now built and installed screen ‘master’ (4.99) and the problem is
> resolved, i.e. provided my .screenrc includes the line ‘truecolor on’,
> colours work fine with COLORTERM set to ‘truecolor’, or not set at
> all.
> 
> Sorry about all this.  Many months ago, I built screen from master and
> ran it to see if it made a difference.  I concluded then that it
> didn't and I think this must have been because I didn't actually
> install the version I built.  This meant that the relevant entries
> under /usr/lib/terminfo/s were not written/updated.
> 
> In any case, we've established one thing for certain, the version of
> screen shipped by Debian bullseye (stable) does not support truecolor.
> Most strange is the fact that, according to dpkg, the version shipped
> is 4.8.0:
> 
>  $ dpkg -l | grep screen
>  ii  screen  4.8.0-6  amd64 terminal multiplexer with VT100/ANSI…
>              ^^^^^
> 
> but screen itself reports:
> 
>  $ screen --version
>  Screen version 4.08.00 (GNU) 05-Feb-20
>                 ^^^^
> 
> which makes no sense at all!
> 
> According to:
> 
>  https://packages.debian.org/search?keywords=screen
> 
> Debian bookworm (the next stable release) will ship screen version
> 4.9.0.  I've downloaded the deb and grepped the manpage and the string
> "truecolor" is nowhere to be found so it looks as if screen shipped by
> Debian will lack support for truecolor for some years to come.

According to this:

  https://bbs.archlinux.org/viewtopic.php?id=249670

true color support in 'screen' will be released in version 5, and the
4.x branch of 'screen' doesn't support it.  This is consistent with
the fact that you built 'screen' from their mast branch, and it
announces itself as version 4.99, i.e. the development version of
v5.x.  I guess an entry in etc/PROBLEMS about this is in order.

What is still a mystery to me is why does GNOME Terminal set
COLORTERM=truecolor in the environment.  Does it assume that 'screen'
will not be used or something?  Or does GNOME itself support true
color?

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Quoth Eli Zaretskii <eliz@gnu.org>
on Sat, 18 Mar 2023 17:13:45 +0200:
>> From: Sebastian Tennant <sdt@sebyte.me>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  62237@debbugs.gnu.org
>> Date: Sat, 18 Mar 2023 15:01:53 +0000
>>
>> […]
>>
>> According to:
>>
>>  https://packages.debian.org/search?keywords=screen
>>
>> Debian bookworm (the next stable release) will ship screen version
>> 4.9.0.  I've downloaded the deb and grepped the manpage and the
>> string "truecolor" is nowhere to be found so it looks as if screen
>> shipped by Debian will lack support for truecolor for some years to
>> come.
>
> According to this:
>
>   https://bbs.archlinux.org/viewtopic.php?id=249670
>
> true color support in 'screen' will be released in version 5, and the
> 4.x branch of 'screen' doesn't support it.  This is consistent with
> the fact that you built 'screen' from their mast branch, and it
> announces itself as version 4.99, i.e. the development version of
> v5.x.  I guess an entry in etc/PROBLEMS about this is in order.
>
> What is still a mystery to me is why does GNOME Terminal set
> COLORTERM=truecolor in the environment.  Does it assume that 'screen'
> will not be used or something?  Or does GNOME itself support true
> color?

Debian bullseye (stable) ships GNOME Terminal version 3.38.3,
tagged/released 4th Feb 2021:

 https://gitlab.gnome.org/GNOME/gnome-terminal/-/commits/3.38.3

Yet, this commit, made 26th April 2014:

 https://gitlab.gnome.org/GNOME/gnome-terminal/-/commits/1d5c1b6ca6373c1301494edbc9e43c3e6a9c9aaf/

reads:

 screen: Stop setting COLORTERM env var
 […]
 COLORTERM is a long-obsolete slang-only variable used to work around
 broken termcap/terminfo entries.

Hmm...


Turns out GNOME Terminal uses VTE (Virtual TErminal):

 https://gitlab.gnome.org/GNOME/vte

and COLORTERM is set to ‘truecolor’ in function merge_environ, on line
261 of file src/spawn.cc:

 https://gitlab.gnome.org/GNOME/vte/-/blob/master/src/spawn.cc

As to why, I've written to Christian Persch and I shall update this
bug report as soon as I receive a response.


-- 
Dorothy Annan Trevor Tennant database - DATTdb
http://dattdb.info

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Quoth Eli Zaretskii <eliz@gnu.org>
on Sat, 18 Mar 2023 17:13:45 +0200:
> […] Or does GNOME itself support true color?

GNOME Terminal/VTE does support true color, but why continue to employ
"a long-obsolete slang-only variable used to work around broken
termcap/terminfo entries"?

Another wrinkle in all this is the fact that I start Emacs at screen
start time (rather than a shell) via an entry in my .screenrc.  This
means the environment is effectively exported and explains why the
value of COLORTERM is picked up by Emacs.  If I allowed screen to fire
up a shell and then started Emacs from that shell, this problem would
never have reared its ugly head!

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Hello Eli,

Thanks for your prompt response.

Quoth Eli Zaretskii <eliz@gnu.org>
on Fri, 17 Mar 2023 14:15:34 +0200:
>> From: Sebastian Tennant <sdt@sebyte.me>
>> Date: Fri, 17 Mar 2023 09:41:36 +0000
>>
>> Summary: Support for 24-bit true colour, added in Emacs 28.1, breaks
>>          colours in Emacsen built without X support and running under
>>          GNU Screen (v4.08.00).
>> […]
>
> Thanks for the report, but it lacks several crucial details for us
> to investigate the problem.
>
> First, if you can build the latest emacs-29 branch of the Emacs Git
> repository, please try that and tell whether the problem persists or
> have been solved in the meantime.

The photos happen to show an Emacs I built from master a couple of
days ago, i.e. Emacs 30.0.50.  If you'd still like me to test branch
emacs-29, then let me know and I will happily do so.

> […]
>
> [T]he user should _only_ set it if the text-mode terminal actually
> supports true color using the commands Emacs expects to work in that
> case, but Emacs cannot detect that without COLORTERM=truecolor being
> set.  Is that your case?

Nope.  Let me explain.

I've been unable to move beyond 27.2 (on my laptops) for over a year
because of these broken colors.  I didn't file a bug report sooner
because I had no idea what the cause of the prolem was.

Yesterday, I read this entry in etc/NEWS.28:

 ** Emacs can support 24-bit color TTY without terminfo database.
 If your text-mode terminal supports 24-bit true color, but your system
 lacks the terminfo database, you can instruct Emacs to support 24-bit
 true color by setting 'COLORTERM=truecolor' in the environment.  This is
 useful on systems such as FreeBSD which ships only with "etc/termcap".

This lead me to investigate whether or not COLORTERM was set in my
environment.  I'm running Debian bullseye and terminfo is installed
but, nevertheless, GNOME Terminal sets COLORTERM=truecolor by default.
I discovered that if I unset COLORTERM, colors start working again!
(In Emacsen I've built without X, running in a screen).

To be clear, the colors are only broken in Emacsen running in a
screen.  Outside of screen, the colors are fine.

> Finally, please show the display produced by "M-x
> list-colors-display" in both cases: when COLORTERM=truecolor is and
> isn't set.  It is important for us to know how many colors Emacs
> uses in each situation.

I've updated the screenshots to show the output of
#'list-colors-display.  Here are the links again, for your
convenience.

 https://download.sebyte.me/misc/truecolor-active.png
 https://download.sebyte.me/misc/truecolor-inactive.png

Seb

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Sebastian Tennant <sdt@sebyte.me>
> Cc: 62237@debbugs.gnu.org
> Date: Fri, 17 Mar 2023 16:20:19 +0000
> 
> > First, if you can build the latest emacs-29 branch of the Emacs Git
> > repository, please try that and tell whether the problem persists or
> > have been solved in the meantime.
> 
> The photos happen to show an Emacs I built from master a couple of
> days ago, i.e. Emacs 30.0.50.  If you'd still like me to test branch
> emacs-29, then let me know and I will happily do so.

No, Emacs 30 is good enough.  (Your report says Emacs 28.1, which is
why I asked.)

>  ** Emacs can support 24-bit color TTY without terminfo database.
>  If your text-mode terminal supports 24-bit true color, but your system
>  lacks the terminfo database, you can instruct Emacs to support 24-bit
>  true color by setting 'COLORTERM=truecolor' in the environment.  This is
>  useful on systems such as FreeBSD which ships only with "etc/termcap".
> 
> This lead me to investigate whether or not COLORTERM was set in my
> environment.  I'm running Debian bullseye and terminfo is installed
> but, nevertheless, GNOME Terminal sets COLORTERM=truecolor by default.

What is TERM set to on that system?

> > Finally, please show the display produced by "M-x
> > list-colors-display" in both cases: when COLORTERM=truecolor is and
> > isn't set.  It is important for us to know how many colors Emacs
> > uses in each situation.
> 
> I've updated the screenshots to show the output of
> #'list-colors-display.  Here are the links again, for your
> convenience.
> 
>  https://download.sebyte.me/misc/truecolor-active.png
>  https://download.sebyte.me/misc/truecolor-inactive.png

This seems to say that your terminal isn't compatible with what Emacs
assumes under COLORTERM=truecolor.  It also says that your terminal
actually supports only 88 colors.  So you should unset COLORTERM in
your environment, or use another terminal emulator (like 256-color
Xterm or some really true-color emulator).

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Quoth Eli Zaretskii <eliz@gnu.org>
on Fri, 17 Mar 2023 18:42:43 +0200:
>> From: Sebastian Tennant <sdt@sebyte.me>
>> Cc: 62237@debbugs.gnu.org
>> Date: Fri, 17 Mar 2023 16:20:19 +0000
>>
>> […]
>> The photos happen to show an Emacs I built from master a couple of
>> days ago, i.e. Emacs 30.0.50.  If you'd still like me to test branch
>> emacs-29, then let me know and I will happily do so.
>
> No, Emacs 30 is good enough.  (Your report says Emacs 28.1, which is
> why I asked.)

It's easy to miss the "or higher".

>> […]
>> This lead me to investigate whether or not COLORTERM was set in my
>> environment.  I'm running Debian bullseye and terminfo is installed
>> but, nevertheless, GNOME Terminal sets COLORTERM=truecolor by default.
>
> What is TERM set to on that system?

In a plain GNOME Terminal, i.e. outside screen:

 xterm-256color

>> […]
>>
>>  https://download.sebyte.me/misc/truecolor-active.png
>>  https://download.sebyte.me/misc/truecolor-inactive.png
>
> This seems to say that your terminal isn't compatible with what
> Emacs assumes under COLORTERM=truecolor.

It would seem so.

> It also says that your terminal actually supports only 88 colors.

How do you infer that?

> So you should unset COLORTERM in your environment

That's precisely the fix I discovered yesterday.

> or use another terminal emulator (like 256-color Xterm or some
> really true-color emulator).

According to:

 https://github.com/termstandard/colors

command:

 $ printf "\x1b[38;2;255;100;0mTRUECOLOR\x1b[0m\n"

shows your terminal is capable of truecolor if the word TRUECOLOR is
printed in brown.  According to the result of this test, GNOME
Terminal (3.38.3) _is_ truecolor enabled.

I can observe the same broken colours in xterm (under the same
conditions) shipped by Debian bullseye.

I'm sure it has to do with screen and will investigate whther or not
my version of screen supports truecolor.

Seb

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> From: Sebastian Tennant <sdt@sebyte.me>
> Cc: 62237@debbugs.gnu.org
> Date: Fri, 17 Mar 2023 18:31:18 +0000
> 
> > No, Emacs 30 is good enough.  (Your report says Emacs 28.1, which is
> > why I asked.)
> 
> It's easy to miss the "or higher".

The "or higher" is not useful because it says nothing about the
versions you actually tested.  It could be Emacs 28.2, for example.
Please always tell all the details without assuming that our crystal
balls are clear enough to see them.

> > What is TERM set to on that system?
> 
> In a plain GNOME Terminal, i.e. outside screen:
> 
>  xterm-256color

That's 256 colors, not 64-bit true-color terminal, AFAIU.

> > It also says that your terminal actually supports only 88 colors.
> 
> How do you infer that?

By looking at the color names.

> > So you should unset COLORTERM in your environment
> 
> That's precisely the fix I discovered yesterday.

Then I guess this bug can be closed?

> According to:
> 
>  https://github.com/termstandard/colors
> 
> command:
> 
>  $ printf "\x1b[38;2;255;100;0mTRUECOLOR\x1b[0m\n"
> 
> shows your terminal is capable of truecolor if the word TRUECOLOR is
> printed in brown.  According to the result of this test, GNOME
> Terminal (3.38.3) _is_ truecolor enabled.

Which standard of true color does it support?

> I can observe the same broken colours in xterm (under the same
> conditions) shipped by Debian bullseye.
> 
> I'm sure it has to do with screen and will investigate whther or not
> my version of screen supports truecolor.

Thanks.

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Eli Zaretskii]

> Cc: 62237@debbugs.gnu.org
> Date: Fri, 17 Mar 2023 21:02:44 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > > What is TERM set to on that system?
> > 
> > In a plain GNOME Terminal, i.e. outside screen:
> > 
> >  xterm-256color

And what is it set to _inside_ screen?

bug#62237: 28.1 or higher: 24-bit true color breaks colours in Emacsen built without X under GNU Screen

[Sebastian Tennant]

Quoth Eli Zaretskii <eliz@gnu.org>
on Fri, 17 Mar 2023 21:02:44 +0200:
>>> No, Emacs 30 is good enough.  (Your report says Emacs 28.1, which
>>> is why I asked.)
>>
>> It's easy to miss the "or higher".
>
> The "or higher" is not useful because it says nothing about the
> versions you actually tested.  It could be Emacs 28.2, for example.
> Please always tell all the details without assuming that our crystal
> balls are clear enough to see them.

Fair enough.  Apologies for the lack of clarity.

>>> What is TERM set to on that system?
>>
>> In a plain GNOME Terminal, i.e. outside screen:
>>
>>  xterm-256color
>
> That's 256 colors, not 64-bit true-color terminal, AFAIU.

You may be right.  My understanding is very limited in this area.

>>> It also says that your terminal actually supports only 88 colors.
>>
>> How do you infer that?
>
> By looking at the color names.

Ah... OK.

>>> So you should unset COLORTERM in your environment
>>
>> That's precisely the fix I discovered yesterday.
>
> Then I guess this bug can be closed?

I suppose so.

It felt like an Emacs bug because I was happily using Emacs 27.2 and
when I upgraded to 28.1, the colours were broken all of a sudden, but
yes, the fault may be with screen.

>> According to:
>>
>>  https://github.com/termstandard/colors
>>
>> command:
>>
>>  $ printf "\x1b[38;2;255;100;0mTRUECOLOR\x1b[0m\n"
>>
>> shows your terminal is capable of truecolor if the word TRUECOLOR is
>> printed in brown.  According to the result of this test, GNOME
>> Terminal (3.38.3) _is_ truecolor enabled.
>
> Which standard of true color does it support?

I don't know how to answer this question.

>> I can observe the same broken colours in xterm (under the same
>> conditions) shipped by Debian bullseye.
>>
>> I'm sure it has to do with screen and will investigate whther or
>> not my version of screen supports truecolor.
>
> Thanks.

Haven't had time to investigate yet, but will do tomorrow.


Quoth Eli Zaretskii <eliz@gnu.org>
on Fri, 17 Mar 2023 21:50:16 +0200:
>>> What is TERM set to on that system?
>>
>> In a plain GNOME Terminal, i.e. outside screen:
>>
>>  xterm-256color
>
> And what is it set to _inside_ screen?

I set it to screen.gnome in my .screenrc, but this has no bearing on
the issue, i.e. if I comment out the ‘term screen.gnome’ line in my
.screenrc, TERM is set to screen.xterm-256color (as Robert pointed
out) and the colors are still broken.

Incidentally, I use screen.gnome because it limits the colors to eight
(and I prefer the simplicity of just eight colors).