bug#62528: 28.2; Emacsclient doesn't use COLORTERM

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Vojtěch Balák]

[-- Attachment #1: Type: text/plain, Size: 11733 bytes --]

Hello,

When starting Emacs in a terminal emulator with 24 bit color support, 
I would like it to take advantage of this. According to the FAQ 
section 5.3, setting the variable COLORTERM to truecolor should 
result in Emacs using 24 bit colors. But according to my observations, 
this is not always true. 

Starting Emacs in these ways does result in it using true color: 
TERM=xterm-256color COLORTERM=truecolor emacs -nw 
TERM=xterm-direct COLORTERM= emacs -nw 
TERM=xterm-direct COLORTERM= emacsclient -nw 

Starting it in these ways does result in it NOT using true color: 
TERM=xterm-256color COLORTERM= emacs -nw 
TERM=xterm-256color COLORTERM= emacsclient -nw 
TERM=xterm-256color COLORTERM=truecolor emacsclient -nw 

The last one is the odd one out. According to my understanding, that
should result in 24 bit color being used, but in practice it does not. 

I verify the color mode by running M-x list-colors-display and seeing if 
the list of returned colors is longer than 256. 

Best regards,
Vojtěch Balák

In GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.36, cairo version 1.17.6) 
of 2023-01-03 built on 2 
Windowing system distributor 'The X.Org Foundation', version 11.0.12101007 
System Description: Arch Linux 

Configured using: 
'configure --sysconfdir=/etc --prefix=/usr --libexecdir=/usr/lib 
--localstatedir=/var --with-cairo --with-harfbuzz --with-libsystemd 
--with-modules --with-x-toolkit=gtk3 'CFLAGS=-march=x86-64 
-mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 
-Wformat -Werror=format-security -fstack-clash-protection 
-fcf-protection -g 
-ffile-prefix-map=/build/emacs/src=/usr/src/debug/emacs -flto=auto' 
'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto'' 

Configured features: 
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG 
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY 
PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE 
XIM XPM GTK3 ZLIB 

Important settings: 
value of $LANG: en_US.UTF-8 
locale-coding-system: utf-8-unix 

Major mode: Dashboard 

Minor modes in effect: 
global-tree-sitter-mode: t 
which-key-mode: t 
global-flycheck-mode: t 
company-posframe-mode: t 
global-company-mode: t 
company-mode: t 
global-git-gutter-mode: t 
global-git-commit-mode: t 
magit-auto-revert-mode: t 
projectile-mode: t 
helm-mode: t 
helm-minibuffer-history-mode: t 
shell-dirtrack-mode: t 
helm--remap-mouse-mode: t 
async-bytecomp-package-mode: t 
solaire-global-mode: t 
doom-modeline-mode: t 
recentf-mode: t 
global-evil-collection-unimpaired-mode: t 
evil-collection-unimpaired-mode: t 
evil-mode: t 
evil-local-mode: t 
global-undo-tree-mode: t 
undo-tree-mode: t 
override-global-mode: t 
tooltip-mode: t 
global-eldoc-mode: t 
show-paren-mode: t 
electric-indent-mode: t 
mouse-wheel-mode: t 
menu-bar-mode: t 
file-name-shadow-mode: t 
global-font-lock-mode: t 
font-lock-mode: t 
blink-cursor-mode: t 
auto-composition-mode: t 
auto-encryption-mode: t 
auto-compression-mode: t 
buffer-read-only: t 
column-number-mode: t 
line-number-mode: t 
transient-mark-mode: t 

Load-path shadows: 
/home/vojta/.config/emacs/elpa/transient-20230315.1520/transient hides /usr/share/emacs/28.2/lisp/transient 

Features: 
(facemenu shadow sort emacsbug sendmail goto-addr evil-collection-view 
view mule-util misearch multi-isearch image-file image-converter 
helm-external helm-net skeleton vc-mtn vc-hg vc-bzr vc-src vc-sccs 
vc-svn vc-cvs vc-rcs bug-reference winner mail-extr tramp-archive 
tramp-gvfs tramp-cache zeroconf helm-command helm-elisp helm-eval 
evil-collection-edebug edebug backtrace helm-info evil-collection-vc-git 
vc-git display-line-numbers rainbow-delimiters flycheck-rust org-appear 
flyspell ispell mixed-pitch org-indent org-element avl-tree ol-eww 
evil-collection-eww eww xdg url-queue mm-url ol-rmail ol-mhe ol-irc 
ol-info ol-gnus nnselect gnus-search eieio-opt evil-collection-speedbar 
speedbar ezimage dframe gnus-art mm-uu mml2015 mm-view mml-smime smime 
dig gnus-sum gnus-group gnus-undo gnus-start gnus-dbus dbus gnus-cloud 
nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range 
gnus-win evil-collection-gnus gnus nnheader ol-docview 
evil-collection-doc-view doc-view jka-compr ol-bibtex ol-bbdb ol-w3m 
ol-doi org-link-doi evil-collection-org org ob ob-tangle ob-ref ob-lob 
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete 
org-list org-faces org-entities org-version ob-emacs-lisp ob-core 
ob-eval org-table oc-basic bibtex ol org-keys oc org-compat org-macs 
org-loaddefs evil-collection-calendar cal-menu calendar cal-loaddefs 
time shr kinsoku svg xml dom evil-collection-lua-mode lua-mode 
terraform-mode hcl-mode dockerfile-mode evil-collection-sh-script 
sh-script smie executable haskell-mode haskell-cabal haskell-utils 
haskell-font-lock haskell-indentation haskell-string 
haskell-sort-imports haskell-lexeme haskell-align-imports 
haskell-complete-module haskell-ghc-support evil-collection-flymake 
flymake-proc flymake warnings dabbrev haskell-customize flycheck-joker 
evil-collection-cider cider tramp-sh cider-debug cider-browse-ns 
cider-mode cider-find cider-inspector cider-completion cider-profile 
cider-eval cider-jar cider-repl-history pulse cider-repl cider-resolve 
cider-test cider-overlays cider-stacktrace cider-doc cider-browse-spec 
cider-clojuredocs cider-eldoc cider-client cider-common cider-connection 
cider-util cider-popup sesman-browser nrepl-client nrepl-dict spinner 
sesman vc vc-dispatcher clojure-mode align parseedn parseclj-parser 
parseclj-lex parseclj-alist evil-collection-slime slime 
evil-collection-apropos apropos evil-collection-outline noutline outline 
hyperspec ranger hl-line evil-collection-vdiff vdiff hydra lv kubel-evil 
kubel evil-collection-eshell em-prompt esh-mode eshell esh-cmd esh-ext 
esh-opt esh-proc esh-io esh-arg esh-module esh-groups esh-util 
evil-collection-yaml-mode yaml-mode evil-collection-vterm vterm 
magit-bookmark evil-collection-bookmark bookmark color 
evil-collection-term term disp-table ehelp vterm-module term/xterm xterm 
tree-sitter-langs tree-sitter-langs-build evil-collection-tar-mode 
tar-mode evil-collection-arc-mode arc-mode archive-mode pp 
tree-sitter-hl tree-sitter tree-sitter-load tree-sitter-cli tsc tsc-dyn 
tsc-dyn-get dired-aux tsc-obsolete evil-collection-which-key which-key 
evil-collection-flycheck flycheck find-func company-posframe posframe 
company-oddmuse company-keywords company-etags etags fileloop generator 
evil-collection-xref xref company-gtags company-dabbrev-code 
company-dabbrev company-files company-clang company-capf company-cmake 
company-semantic company-template company-bbdb evil-collection-company 
company git-gutter evil-collection-magit magit-submodule magit-blame 
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch 
magit-clone magit-remote magit-commit magit-sequence magit-notes 
magit-worktree magit-tag magit-merge magit-branch magit-reset 
magit-files magit-refs magit-status magit magit-repos magit-apply 
magit-wip magit-log which-func evil-collection-imenu imenu magit-diff 
smerge-mode evil-collection-diff-mode diff-mode git-commit 
evil-collection-log-edit log-edit message rmc puny rfc822 mml mml-sec 
evil-collection-epa epa derived epg rfc6068 epg-config gnus-util rmail 
rmail-loaddefs mm-decode mm-bodies mm-encode mailabbrev mail-utils 
gmm-utils mailheader pcvs-util add-log magit-core magit-autorevert 
autorevert magit-margin magit-transient magit-process with-editor server 
magit-mode transient magit-git magit-base evil-collection-magit-section 
magit-section crm evil-collection-ripgrep ripgrep helm-projectile 
projectile project lisp-mnt mail-parse rfc2231 rfc2047 rfc2045 mm-util 
ietf-drums mail-prsvr evil-collection-grep grep evil-collection-compile 
compile ibuf-ext evil-collection-ibuffer ibuffer ibuffer-loaddefs 
helm-mode helm-misc evil-collection-helm helm-files 
evil-collection-image-dired image-dired filenotify tramp tramp-loaddefs 
trampver tramp-integration files-x tramp-compat shell pcomplete 
evil-collection-comint comint parse-time iso8601 time-date ls-lisp 
helm-buffers helm-occur helm-tags helm-locate helm-grep helm-regexp 
format-spec ansi-color helm-utils helm-help helm-types helm 
helm-global-bindings helm-easymenu helm-core async-bytecomp helm-source 
helm-multi-match helm-lib async evil-collection-neotree neotree 
solaire-mode face-remap doom-modeline doom-modeline-segments 
doom-modeline-env doom-modeline-core shrink-path rx f f-shortdoc 
evil-collection-shortdoc shortdoc text-property-search s dash compat 
compat-29 evil-collection-dashboard dashboard dashboard-widgets recentf 
tree-widget wid-edit ffap all-the-icons all-the-icons-faces 
data-material data-weathericons data-octicons data-fileicons 
data-faicons data-alltheicons evil-collection-unimpaired 
evil-collection-tabulated-list evil-collection-tab-bar 
evil-collection-simple evil-collection-replace 
evil-collection-process-menu evil-collection-package-menu 
evil-collection-info evil-collection-indent evil-collection-image 
image-mode evil-collection-dired dired dired-loaddefs exif 
evil-collection-help evil-collection-elisp-mode evil-collection-eldoc 
evil-collection-buff-menu evil-collection annalist evil evil-integration 
evil-maps evil-commands reveal evil-jumps evil-command-window 
evil-search evil-ex evil-types evil-macros evil-repeat evil-states 
evil-core advice evil-common thingatpt rect evil-digraphs evil-vars ring 
edmacro kmacro undo-tree diff queue doom-themes-ext-neotree 
doom-gruvbox-theme doom-themes doom-themes-base cl-extra help-mode 
use-package use-package-ensure use-package-delight use-package-diminish 
use-package-bind-key bind-key easy-mmode use-package-core finder-inf 
pcase slime-autoloads info package browse-url url url-proxy url-privacy 
url-expand url-methods url-history url-cookie url-domsuf url-util 
mailcap url-handlers url-parse auth-source cl-seq eieio eieio-core 
cl-macs eieio-loaddefs password-cache json subr-x map url-vars seq 
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl 
tooltip eldoc paren electric uniquify ediff-hook vc-hooks 
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd 
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace 
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar 
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock 
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic 
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao 
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech 
european ethiopic indian cyrillic chinese composite emoji-zwj charscript 
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray 
cl-preloaded nadvice button loaddefs faces cus-face macroexp files 
window text-properties overlay sha1 md5 base64 format env code-pages 
mule custom widget hashtable-print-readable backquote threads dbusbind 
inotify lcms2 dynamic-setting system-font-setting font-render-setting 
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs) 

Memory information: 
((conses 16 839697 87675) 
(symbols 48 58194 3) 
(strings 32 254832 18309) 
(string-bytes 1 7559626) 
(vectors 16 118605) 
(vector-slots 8 1538617 83734) 
(floats 8 1343 1501) 
(intervals 56 9681 4159) 
(buffers 992 29))

[-- Attachment #2: Type: text/html, Size: 15085 bytes --]

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Eli Zaretskii]

> Date: Wed, 29 Mar 2023 17:06:18 +0200
> From: Vojtěch Balák <vojtech@balak.me>
> 
> When starting Emacs in a terminal emulator with 24 bit color support, 
> I would like it to take advantage of this. According to the FAQ 
> section 5.3, setting the variable COLORTERM to truecolor should 
> result in Emacs using 24 bit colors. But according to my observations, 
> this is not always true. 
> 
> Starting Emacs in these ways does result in it using true color: 
> TERM=xterm-256color COLORTERM=truecolor emacs -nw 
> TERM=xterm-direct COLORTERM= emacs -nw 
> TERM=xterm-direct COLORTERM= emacsclient -nw 
> 
> Starting it in these ways does result in it NOT using true color: 
> TERM=xterm-256color COLORTERM= emacs -nw 
> TERM=xterm-256color COLORTERM= emacsclient -nw 
> TERM=xterm-256color COLORTERM=truecolor emacsclient -nw 
> 
> The last one is the odd one out. According to my understanding, that
> should result in 24 bit color being used, but in practice it does not. 

I think you are misinterpreting what the documentation says.
COLORTERM=truecolor can only affect Emacs, it cannot affect the
display triggered by emacsclient.

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Vojtěch Balák]

> I think you are misinterpreting what the documentation says.
> COLORTERM=truecolor can only affect Emacs, it cannot affect the
> display triggered by emacsclient.

Why is that the case, if you don't mind me asking? emacsclient does
support 24 bit color, since setting TERM=xterm-direct works. And
because both Emacs and emacsclient support 24 bit color, I would
expect COLORTERM to work for both.

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Eli Zaretskii]

> Date: Wed, 29 Mar 2023 20:42:53 +0200
> From: Vojtěch Balák <vojtech@balak.me>
> Cc: 62528@debbugs.gnu.org
> 
> > I think you are misinterpreting what the documentation says.
> > COLORTERM=truecolor can only affect Emacs, it cannot affect the
> > display triggered by emacsclient.
> 
> Why is that the case, if you don't mind me asking? emacsclient does
> support 24 bit color, since setting TERM=xterm-direct works. And
> because both Emacs and emacsclient support 24 bit color, I would
> expect COLORTERM to work for both.

COLORTERM should be in the environment of emacs when it starts, not in
the environment of emacsclient.

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Vojtěch Balák]

> COLORTERM should be in the environment of emacs when it starts, not in
> the environment of emacsclient.

Oh, that does work. You're right, I misunderstood the documentation.
Thank you for clarifying.

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Eli Zaretskii]

> Date: Thu, 30 Mar 2023 09:49:56 +0200
> From: Vojtěch Balák <vojtech@balak.me>
> Cc: 62528@debbugs.gnu.org
> 
> > COLORTERM should be in the environment of emacs when it starts, not in
> > the environment of emacsclient.
> 
> Oh, that does work. You're right, I misunderstood the documentation.
> Thank you for clarifying.

OK, I'm therefore closing this bug.

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Robert Pluim]

>>>>> On Thu, 30 Mar 2023 09:49:56 +0200, Vojtěch Balák <vojtech@balak.me> said:

    >> COLORTERM should be in the environment of emacs when it starts, not in
    >> the environment of emacsclient.

Thatʼs because we look it up using `getenv' in init_tty. If we used
`egetenv' instead, then we could honour the value of "COLORTERM" sent
by emacsclient.

Robert
-- 

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Vojtěch Balák <vojtech@balak.me>,
>   62528@debbugs.gnu.org
> Date: Thu, 30 Mar 2023 10:59:32 +0200
> 
> >>>>> On Thu, 30 Mar 2023 09:49:56 +0200, Vojtěch Balák <vojtech@balak.me> said:
> 
>     >> COLORTERM should be in the environment of emacs when it starts, not in
>     >> the environment of emacsclient.
> 
> Thatʼs because we look it up using `getenv' in init_tty. If we used
> `egetenv' instead, then we could honour the value of "COLORTERM" sent
> by emacsclient.

But egetenv would be wrong here, because it looks in
process-environment.

This is _exactly_ the issue here: emacsclient puts the environment of
the parent shell into process-environment, so that it could be
inherited by sub-processes, but Emacs itself should _not_ be sensitive
to the environment it prepares for sub-processes.

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Robert Pluim]

>>>>> On Thu, 30 Mar 2023 12:34:39 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Vojtěch Balák <vojtech@balak.me>,
    >> 62528@debbugs.gnu.org
    >> Date: Thu, 30 Mar 2023 10:59:32 +0200
    >> 
    >> >>>>> On Thu, 30 Mar 2023 09:49:56 +0200, Vojtěch Balák <vojtech@balak.me> said:
    >> 
    >> >> COLORTERM should be in the environment of emacs when it starts, not in
    >> >> the environment of emacsclient.
    >> 
    >> Thatʼs because we look it up using `getenv' in init_tty. If we used
    >> `egetenv' instead, then we could honour the value of "COLORTERM" sent
    >> by emacsclient.

    Eli> But egetenv would be wrong here, because it looks in
    Eli> process-environment.

Well yes, thatʼs the whole point

    Eli> This is _exactly_ the issue here: emacsclient puts the environment of
    Eli> the parent shell into process-environment, so that it could be
    Eli> inherited by sub-processes, but Emacs itself should _not_ be sensitive
    Eli> to the environment it prepares for sub-processes.

Normally Iʼd agree with you, but this a grey area: weʼre creating a
new frame, so having its characteristics depend on information sent by
emacsclient makes sense, especially since server.el already goes to
the trouble of ensuring that emacsclientʼs value of COLORTERM is
inherited.

Would you accept a compromise where we check `getenv', and if the
value is empty, check `egetenv'?

Robert
-- 

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: vojtech@balak.me,  62528@debbugs.gnu.org
> Date: Thu, 30 Mar 2023 11:41:42 +0200
> 
> Would you accept a compromise where we check `getenv', and if the
> value is empty, check `egetenv'?

Sorry, no.  First, because it will not behave differently from just
using egetenv to begin with.  More importantly, I don't think we want
to have such different color capabilities on different frames in the
same Emacs session, that will cause many problems we don't want to
touch with a 3-mile pole.

The fix is easy: set the variable before starting Emacs, so why should
we jump through hoops here?

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Robert Pluim]

>>>>> On Thu, 30 Mar 2023 13:36:06 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> More importantly, I don't think we want
    Eli> to have such different color capabilities on different frames in the
    Eli> same Emacs session, that will cause many problems we don't want to
    Eli> touch with a 3-mile pole.

Thatʼs a situation we already have, since those color capabilities are
derived from the terminal where emacsclient is invoked anyway.

Robert
-- 

bug#62528: 28.2; Emacsclient doesn't use COLORTERM

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: vojtech@balak.me,  62528@debbugs.gnu.org
> Date: Thu, 30 Mar 2023 14:11:29 +0200
> 
> >>>>> On Thu, 30 Mar 2023 13:36:06 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> More importantly, I don't think we want
>     Eli> to have such different color capabilities on different frames in the
>     Eli> same Emacs session, that will cause many problems we don't want to
>     Eli> touch with a 3-mile pole.
> 
> Thatʼs a situation we already have, since those color capabilities are
> derived from the terminal where emacsclient is invoked anyway.

But COLORTERM disregards terminfo, so it's much more dangerous.