bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Istvan Marko]

When an Emacs session has both an X11 frame and a tty frame on a 256
color capable terminal calling (clear-face-cache) in the X11 frame
results in the colors getting messed up in the existing tty frame. Newly
created tty frames are OK. Running (clear-face-cache) in the tty frame
fixes the affected frame.

Non-256-color tty frames don't seem to be affected.

I am running into this regularly because I often have Emacs sessions
with both X and tty frames, switching back and forth between them.
Sometimes when I switch back to the tty frame I find that the colors are
messed up. I suspect that this is triggered by redisplay_internal
periodically calling clear-face-cache.

I am able to reproduce this with the current master and emacs-26 using
the following steps:

Start Emacs in X11 mode and start Emacs server:

emacs -Q
M-x server-start

Then in a 256 color terminal with TERM=xterm-256color or similar:

emacsclient -t

Then switch back to the X frame and evaluate:

M-: (clear-face-cache) RET

This immediately corrupts the colors of the tty frame. Not sure if all
faces are affected but the mode-line face for example always turns black
or green for me so it's very obvious.

Running (clear-face-cache) in the tty frame restores the correct tty
faces. The X frame is not affected.

Looking at the git log e573d08ef15f0431ad8289b4242c49826f20efb6 ("Make
face realization be more frame-specific") seems like a potential suspect
but unfortunately I am no longer able to build that revision on my Linux
hosts (temacs loading gobbles up all available memory for some reason)

In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu)
 of 2018-07-06 built on somehost
Repository revision: 3bbd4ffc68bcc2b3e003a2179a508b82055ad770
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
System Description: Linux

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
user-error: Beginning of history; no preceding item

Configured using:
 'configure --without-x-toolkit --without-gpm --without-rsvg
 --without-gconf --without-xim --without-gsettings
 --with-file-notification=inotify --without-lcms2'

Configured features:
XPM JPEG TIFF GIF PNG IMAGEMAGICK SOUND DBUS NOTIFY ACL GNUTLS LIBXML2
FREETYPE XFT ZLIB OLDXMENU X11 THREADS

Important settings:
  value of $LANG: C
  locale-coding-system: nil

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
easymenu mml-sec password-cache epa derived epg epg-config gnus-util
rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils elec-pair time-date
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting font-render-setting x multi-tty make-network-process
emacs)

Memory information:
((conses 16 95891 6947)
 (symbols 48 20447 1)
 (miscs 40 40 85)
 (strings 32 29147 1543)
 (string-bytes 1 759273)
 (vectors 16 15030)
 (vector-slots 8 508852 6900)
 (floats 8 52 65)
 (intervals 56 243 142)
 (buffers 992 12))

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Eli Zaretskii]

> From: Istvan Marko <mi-ebugs@kismala.com>
> Date: Fri, 06 Jul 2018 10:18:39 -0700
> 
> When an Emacs session has both an X11 frame and a tty frame on a 256
> color capable terminal calling (clear-face-cache) in the X11 frame
> results in the colors getting messed up in the existing tty frame.

Can you tell more about what "messed-up" means?  E.g., what do you see
if you type "M-x list-colors-display RET" in a TTY frame after
invoking clear-face-cache on a GUI frame?

> Newly created tty frames are OK. Running (clear-face-cache) in the
> tty frame fixes the affected frame.

Strange: clear-face-cache does not work only on the selected frame, it
clears the face caches of _all_ the GUI frames, and is supposed to do
nothing for TTY frames, except forcing recreation of all faces on the
next redisplay.  So I'm not sure I understand what does "Running
(clear-face-cache) in the tty frame" mean.  Are you saying that if all
frames in a session are TTY frames, the problem doesn't happen?  Or do
you mean that a _second_ call to clear-face-cache fixes the TTY
frames?  If the latter, does it matter what frame is selected when you
issue that second call?

What if you invoke clear-face-cache, and then type "M-x redraw-display
RET" in a TTY frame -- does that still show "messed-up" colors?

> Start Emacs in X11 mode and start Emacs server:
> 
> emacs -Q
> M-x server-start
> 
> Then in a 256 color terminal with TERM=xterm-256color or similar:
> 
> emacsclient -t
> 
> Then switch back to the X frame and evaluate:
> 
> M-: (clear-face-cache) RET
> 
> This immediately corrupts the colors of the tty frame. Not sure if all
> faces are affected but the mode-line face for example always turns black
> or green for me so it's very obvious.

"Black or green"?  So the effect is not entirely deterministic?

Do you see this in Emacs 26 as well, or only on the master branch?

Thanks.

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Istvan Marko]

Eli Zaretskii <eliz@gnu.org> writes:

> Can you tell more about what "messed-up" means?  E.g., what do you see
> if you type "M-x list-colors-display RET" in a TTY frame after
> invoking clear-face-cache on a GUI frame?

Interestingly the colors of list-colors-display remain intact (except
for the modeline). list-faces-display however shows the problem.

Here are some screenshots with the M-x list-faces-display buffer from an
emacs-client -t frame on a 256 color terminal before and after running
clear-face-cache in the X11 frame.

Normal: https://i.imgur.com/QPUbARN.png

Bad: https://i.imgur.com/080IjKI.png

Note that some colors are retained, for example the background of
tty-menu-selected-face which is "red", a color that exists on 16 color
terminals too.

> Or do you mean that a _second_ call to clear-face-cache fixes the TTY
> frames?  

Yes, this. Calling clear-face-cache again, this time from the 256 color
TTY frame, fixes the TTY frames.

> If the latter, does it matter what frame is selected when you issue
> that second call?

Yes, the second call only helps if it's done while the 256 color TTY
frame is the active one.

> What if you invoke clear-face-cache, and then type "M-x redraw-display
> RET" in a TTY frame -- does that still show "messed-up" colors?

M-x redraw-display in the tty frame does not help.

> "Black or green"?  So the effect is not entirely deterministic?

I think it's deterministic but may have a couple of variations depending
on the state of my Emacs session or the terminal. I haven't figured
out

> Do you see this in Emacs 26 as well, or only on the master branch?

Yes, I see this with both 26 and master. I am fairly sure that this
wasn't happening about 2 years ago. Perhaps it started when I switched
from 25 to 26 but not sure. I am having trouble building an old enough
version on my current Linux system to verify but if we get stuck I will
find a way to build it and try to bisect.

-- 
	Istvan

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Noam Postavsky]

found 32072 26.1
tags 32072 + confirmed
quit

Istvan Marko <mi-ebugs@kismala.com> writes:

> I have done some more testing and found that 24-bit terminal frames are
> not affected, their colors remain intact after doing (clear-face-cache)
> in the X11 frame.

Hmm, interesting, I've bisected the problem to [1: e463e5762b].

[1: e463e5762b]: 2017-02-18 13:04:55 +0200
  Support 24-bit direct colors on text terminals
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=e463e5762bbe628be3d15da066a90f079a8468b3

> I have been planning to switch to 24-bit anyway so this is a good enough
> for me but let me know if it's worth trying to debug the 256 color case
> further.

It would be good not to break the 256 colour case, yes.  I've cc'd the
author of that 24 bit support, maybe they will have some more ideas.

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Date: Wed, 18 Jul 2018 21:39:45 -0400
> Cc: 32072@debbugs.gnu.org,
> 	Rami Ylimäki <rami.ylimaki@vincit.fi>
> 
> > I have done some more testing and found that 24-bit terminal frames are
> > not affected, their colors remain intact after doing (clear-face-cache)
> > in the X11 frame.
> 
> Hmm, interesting, I've bisected the problem to [1: e463e5762b].
> 
> [1: e463e5762b]: 2017-02-18 13:04:55 +0200
>   Support 24-bit direct colors on text terminals
>   https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=e463e5762bbe628be3d15da066a90f079a8468b3

Thanks.  Just stabbing in the dark here, but does the patch below
help?

diff --git a/lisp/term/tty-colors.el b/lisp/term/tty-colors.el
index ab9149e..a776c83 100644
--- a/lisp/term/tty-colors.el
+++ b/lisp/term/tty-colors.el
@@ -824,10 +824,12 @@ tty-color-canonicalize
 	(replace-regexp-in-string " +" "" (downcase color))
       color)))
 
-(defun tty-color-24bit (rgb)
-  "Return pixel value on 24-bit terminals. Return nil if RGB is
-nil or not on 24-bit terminal."
-  (when (and rgb (= (display-color-cells) 16777216))
+(defun tty-color-24bit (rgb &optional display)
+  "Return 24-bit color pixel value for RGB value on DISPLAY.
+DISPLAY can be a display name or a frame, and defaults to the
+selected frame's display.
+If DISPLAY is not on a 24-but TTY terminal, return nil."
+  (when (and rgb (= (display-color-cells display) 16777216))
     (let ((r (lsh (car rgb) -8))
 	  (g (lsh (cadr rgb) -8))
 	  (b (lsh (nth 2 rgb) -8)))
@@ -850,7 +852,7 @@ tty-color-define
       (error "Invalid specification for tty color \"%s\"" name))
   (tty-modify-color-alist
    (append (list (tty-color-canonicalize name)
-		 (or (tty-color-24bit rgb) index))
+		 (or (tty-color-24bit rgb frame) index))
 	   rgb)
    frame))
 
@@ -1026,7 +1028,7 @@ tty-color-desc
 	  (or (assoc color (tty-color-alist frame))
 	      (let ((rgb (tty-color-standard-values color)))
 		(and rgb
-		     (let ((pixel (tty-color-24bit rgb)))
+		     (let ((pixel (tty-color-24bit rgb frame)))
 		       (or (and pixel (cons color (cons pixel rgb)))
 			   (tty-color-approximate rgb frame)))))))))
 

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Rami Ylimäki]

[-- Attachment #1: Type: text/plain, Size: 242 bytes --]

2018-07-19 16:20 GMT+03:00 Eli Zaretskii <eliz@gnu.org>:
>
>
> Thanks.  Just stabbing in the dark here, but does the patch below
> help?
>
>
Yes. I tested the patch and can't reproduce the problem while I can
reproduce it without your patch.

[-- Attachment #2: Type: text/html, Size: 575 bytes --]

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Eli Zaretskii]

> From: Rami Ylimäki <rami.ylimaki@vincit.fi>
> Date: Thu, 19 Jul 2018 20:33:40 +0300
> Cc: Noam Postavsky <npostavs@gmail.com>, mi-ebugs@kismala.com, 32072@debbugs.gnu.org
> 
> 2018-07-19 16:20 GMT+03:00 Eli Zaretskii <eliz@gnu.org>:
> 
>  Thanks.  Just stabbing in the dark here, but does the patch below
>  help?
> 
> Yes. I tested the patch and can't reproduce the problem while I can reproduce it without your patch.

Thanks, pushed to emacs-26 branch.

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Istvan Marko]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, pushed to emacs-26 branch.

emacs-26 looks good here too now, thanks!

The fix will be needed in master too, right? It was broken in both 26
and master.

-- 
	Istvan

bug#32072: 27.0.50; clear-face-cache in an X frame breaks tty colors

[Eli Zaretskii]

> From: Istvan Marko <mi-ebugs@kismala.com>
> Cc: npostavs@gmail.com, 32072@debbugs.gnu.org
> Date: Thu, 19 Jul 2018 11:18:53 -0700
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks, pushed to emacs-26 branch.
> 
> emacs-26 looks good here too now, thanks!

Thanks, I'm therefore closing the bug.

> The fix will be needed in master too, right? It was broken in both 26
> and master.

The fix will be merged to master soon enough.