bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Manuel Uberti]

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

 From 'emacs -Q', this is what I did:

- (add-hook 'compilation-filter-hook #'ansi-color-compilation-filter)
- M-x rgrep RET
- setq RET
- RET
- ~/emacs RET (this is where I keep the entire Emacs source code)

As you can see from the attached image, some of the results are red and some are 
not. This is not happening without ansi-color-compilation-filter.y

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.18, cairo 
version 1.16.0)
  of 2021-08-26 built on hathaway
Repository revision: 4ac29b943bdcc099f578660395b17b430551ff79
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12009000
System Description: Ubuntu 20.04 LTS

Configured using:
  'configure --with-harfbuzz --with-native-compilation'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
   value of $LC_MESSAGES: en_GB.UTF-8
   value of $LC_MONETARY: it_IT.UTF-8
   value of $LC_NUMERIC: it_IT.UTF-8
   value of $LC_TIME: it_IT.UTF-8
   value of $LANG: en_US.UTF-8
   value of $XMODIFIERS: @im=ibus
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny rfc822 mml mml-sec epa
derived epg epg-config gnus-util rmail rmail-loaddefs mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
time-date find-dired dired dired-loaddefs ffap url-parse auth-source
cl-seq eieio eieio-core cl-macs eieio-loaddefs password-cache json
subr-x map seq byte-opt gv bytecomp byte-compile cconv url-vars
thingatpt cl-loaddefs cl-lib files-x grep compile text-property-search
comint ansi-color ring iso-transl tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 5121823 158759)
  (symbols 48 7949 0)
  (strings 32 26036 7196)
  (string-bytes 1 889591)
  (vectors 16 79368)
  (vector-slots 8 680334 499758)
  (floats 8 28 57)
  (intervals 56 424949 1659)
  (buffers 992 12))

-- 
Manuel Uberti
www.manueluberti.eu

[-- Attachment #2: Screenshot from 2021-08-26 07-52-12.png --]
[-- Type: image/png, Size: 433123 bytes --]

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

found 50207 25.1
quit

Manuel Uberti [2021-08-26 07:57 +0200] wrote:

> From 'emacs -Q', this is what I did:
>
> - (add-hook 'compilation-filter-hook #'ansi-color-compilation-filter)
> - M-x rgrep RET
> - setq RET
> - RET
> - ~/emacs RET (this is where I keep the entire Emacs source code)
>
> As you can see from the attached image, some of the results are red and some are
> not. This is not happening without ansi-color-compilation-filter.y

I can reproduce the issue in Emacs 25.1 and newer, but not in 24.5
(configuration below signature), with the following modified recipe:

0. emacs -Q
1. (add-hook 'compilation-filter-hook
             (lambda ()
               (ansi-color-apply-on-region compilation-filter-start
                                           (point))))
2. C-x C-e
3. M-x rgrep RET
4. setq RET RET /dir/of/emacs/trunk RET
5. ...twiddle thumbs...
6. C-c C-k

Loading the 24.5 versions of ansi-color.el and compile.el in 25.1
between steps 0 and 1 does not seem to change anything.

Thanks,

-- 
Basil

In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2021-01-02 on tia
Windowing system distributor `The X.Org Foundation', version 11.0.12011000
System Description:	Debian GNU/Linux 11 (bullseye)

Configured using:
 `configure CC=clang 'CFLAGS=-O2 -march=native'
 --prefix=/home/blc/.local --program-suffix=-24 --with-x-toolkit=lucid
 --with-file-notification=yes --with-x'

Important settings:
  value of $LANG: en_IE.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Grep

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set
((lambda nil (ansi-color-apply-on-region compilation-filter-start (point))))
Grep interrupt

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns mail-prsvr mail-utils find-dired dired grep compile
comint ansi-color ring time-date tooltip electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
gfilenotify dynamic-setting system-font-setting font-render-setting
x-toolkit x multi-tty emacs)

In GNU Emacs 25.1.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2021-08-26 built on tia
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description:	Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure CC=clang 'CFLAGS=-O2 -march=native'
 --prefix=/home/blc/.local --program-suffix=-25.1 --with-x-toolkit=lucid
 --with-file-notification=yes --with-x --with-modules'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11 MODULES

Important settings:
  value of $LANG: en_IE.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Grep

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set
((lambda nil (ansi-color-apply-on-region compilation-filter-start (point))))
Mark set
Grep interrupt

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message format-spec rfc822 mml mml-sec
password-cache epg epg-config gnus-util mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util help-fns help-mode easymenu cl-loaddefs pcase
cl-lib mail-prsvr mail-utils find-dired dired thingatpt grep compile
comint ansi-color ring time-date mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting font-render-setting
x-toolkit x multi-tty make-network-process emacs)

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Jim Porter]

On 8/25/2021 10:57 PM, Manuel Uberti wrote:
>  From 'emacs -Q', this is what I did:
> 
> - (add-hook 'compilation-filter-hook #'ansi-color-compilation-filter)
> - M-x rgrep RET
> - setq RET
> - RET
> - ~/emacs RET (this is where I keep the entire Emacs source code)
> 
> As you can see from the attached image, some of the results are red and 
> some are not. This is not happening without ansi-color-compilation-filter.y

I encountered this a bit ago, and did a bit of diagnosis, but it ended 
up on my back-burner. I think the issue is due to how the 
compilation-filter-hooks for grep and ansi-color interact. `grep-filter' 
is fairly simple and wants to see both the start and end of an 
ANSI-colorized region, so it "rewinds" to the beginning of a line every 
time it's called. `ansi-color-compilation-filter', on the other hand, is 
smart enough to handle the case where it only sees the start of a 
colorized region in one call, and the end in the next call (see 
`ansi-color-context' for details).

These interact poorly, since normally `grep-filter' reads all the ANSI 
escapes, handles the ones it recognizes, and strips the rest, leaving 
`ansi-color-compilation-filter' with nothing to do. However, when grep's 
output currently shows the start of a colorized region but not the end, 
`grep-filter' doesn't touch that, assuming it can come back to it in the 
next call. By then `ansi-color-compilation-filter' has "stolen" that 
ANSI escape, confusing `grep-filter'.

To solve this, I just turn off `ansi-color-compilation-filter' for grep. 
When things are working right, it should be a no-op, and when things are 
working wrong, you see the behavior described in this bug. It might be 
nice to turn this off automatically though. Right now it's just a sharp 
corner people can cut themselves on.

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Basil L. Contovounesios [2021-08-26 12:10 +0100] wrote:

> I can reproduce the issue in Emacs 25.1 and newer, but not in 24.5
> (configuration below signature), with the following modified recipe:
>
> 0. emacs -Q
> 1. (add-hook 'compilation-filter-hook
>              (lambda ()
>                (ansi-color-apply-on-region compilation-filter-start
>                                            (point))))
> 2. C-x C-e
> 3. M-x rgrep RET
> 4. setq RET RET /dir/of/emacs/trunk RET
> 5. ...twiddle thumbs...
> 6. C-c C-k
>
> Loading the 24.5 versions of ansi-color.el and compile.el in 25.1
> between steps 0 and 1 does not seem to change anything.

Which makes sense, because the highlighting bisects to the following
change in grep.el:

Don't assume 'grep' supports GREP_OPTIONS.
2e4c2fe278 2014-09-16 17:07:12 -0700
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=2e4c2fe2787785a421f256541de642976e9bd90b

-- 
Basil

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Basil L. Contovounesios [2021-08-26 23:11 +0100] wrote:

> Basil L. Contovounesios [2021-08-26 12:10 +0100] wrote:
>
>> I can reproduce the issue in Emacs 25.1 and newer, but not in 24.5
>> (configuration below signature), with the following modified recipe:
>>
>> 0. emacs -Q
>> 1. (add-hook 'compilation-filter-hook
>>              (lambda ()
>>                (ansi-color-apply-on-region compilation-filter-start
>>                                            (point))))
>> 2. C-x C-e
>> 3. M-x rgrep RET
>> 4. setq RET RET /dir/of/emacs/trunk RET
>> 5. ...twiddle thumbs...
>> 6. C-c C-k
>>
>> Loading the 24.5 versions of ansi-color.el and compile.el in 25.1
>> between steps 0 and 1 does not seem to change anything.
>
> Which makes sense, because the highlighting bisects to the following
> change in grep.el:
>
> Don't assume 'grep' supports GREP_OPTIONS.
> 2e4c2fe278 2014-09-16 17:07:12 -0700
> https://git.sv.gnu.org/cgit/emacs.git/commit/?id=2e4c2fe2787785a421f256541de642976e9bd90b

That is with GNU grep 3.6, which no longer supports GREP_OPTIONS.

-- 
Basil

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Juri Linkov]

>> - (add-hook 'compilation-filter-hook #'ansi-color-compilation-filter)
>
> I encountered this a bit ago, and did a bit of diagnosis, but it ended up
> on my back-burner. I think the issue is due to how the
> compilation-filter-hooks for grep and ansi-color interact. `grep-filter' is
> fairly simple and wants to see both the start and end of an ANSI-colorized
> region, so it "rewinds" to the beginning of a line every time it's
> called. `ansi-color-compilation-filter', on the other hand, is smart enough
> to handle the case where it only sees the start of a colorized region in
> one call, and the end in the next call (see `ansi-color-context' for
> details).

Would it be possible to solve the problem by adding a new buffer-local
variable (disabled by default) that will enable line mode for
`ansi-color-compilation-filter' so that it will handle only complete lines
like grep mode does?

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Jim Porter]

On 8/26/2021 11:06 PM, Juri Linkov wrote:
> Would it be possible to solve the problem by adding a new buffer-local
> variable (disabled by default) that will enable line mode for
> `ansi-color-compilation-filter' so that it will handle only complete lines
> like grep mode does?

I think that would solve this bug. I don't know that it's necessary 
though: if `grep-filter' does everything correctly (or perhaps, "if it's 
allowed to do everything correctly"), there should be no ANSI escapes 
left for `ansi-color-compilation-filter' to process. `grep-mode' could 
probably just disable `ansi-color-compilation-filter' entirely.

Another, more-involved option would be for `grep-filter' to work more 
like `ansi-color-compilation-filter', i.e. keep track of the active 
escapes to use for the next hunk of text. I *think* this would make it 
more robust for searches spanning multiple lines, but admittedly I 
haven't tested this out to be sure.

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Augusto Stoffel]

For the record, I still see this bug in Emacs 29.0.60.

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Augusto Stoffel]

On Fri, 27 Aug 2021 at 09:06, Juri Linkov wrote:

>>> - (add-hook 'compilation-filter-hook #'ansi-color-compilation-filter)
>>
>> I encountered this a bit ago, and did a bit of diagnosis, but it ended up
>> on my back-burner. I think the issue is due to how the
>> compilation-filter-hooks for grep and ansi-color interact. `grep-filter' is
>> fairly simple and wants to see both the start and end of an ANSI-colorized
>> region, so it "rewinds" to the beginning of a line every time it's
>> called. `ansi-color-compilation-filter', on the other hand, is smart enough
>> to handle the case where it only sees the start of a colorized region in
>> one call, and the end in the next call (see `ansi-color-context' for
>> details).
>
> Would it be possible to solve the problem by adding a new buffer-local
> variable (disabled by default) that will enable line mode for
> `ansi-color-compilation-filter' so that it will handle only complete lines
> like grep mode does?

To complement this a bit, my hunch is that the problem is due to
`compilation-filter-start' being a number and not a marker (bug?).  When
`grep-filter' deletes some characters between `compilation-filter-start'
and the beginning of its line, it causes `ansi-color-compilation-filter'
to skip treating a portion of the buffer.

Or at least I was having this kind of issue in the `grep-heading-mode'
suggested in a recent ticket. :-)

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Juri Linkov]

>>>> - (add-hook 'compilation-filter-hook #'ansi-color-compilation-filter)
>>>
>>> I encountered this a bit ago, and did a bit of diagnosis, but it ended up
>>> on my back-burner. I think the issue is due to how the
>>> compilation-filter-hooks for grep and ansi-color interact. `grep-filter' is
>>> fairly simple and wants to see both the start and end of an ANSI-colorized
>>> region, so it "rewinds" to the beginning of a line every time it's
>>> called. `ansi-color-compilation-filter', on the other hand, is smart enough
>>> to handle the case where it only sees the start of a colorized region in
>>> one call, and the end in the next call (see `ansi-color-context' for
>>> details).
>>
>> Would it be possible to solve the problem by adding a new buffer-local
>> variable (disabled by default) that will enable line mode for
>> `ansi-color-compilation-filter' so that it will handle only complete lines
>> like grep mode does?
>
> To complement this a bit, my hunch is that the problem is due to
> `compilation-filter-start' being a number and not a marker (bug?).  When
> `grep-filter' deletes some characters between `compilation-filter-start'
> and the beginning of its line, it causes `ansi-color-compilation-filter'
> to skip treating a portion of the buffer.

Have you tried to replace a number with a marker?  Does it help to fix
this bug?

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Augusto Stoffel]

On Fri,  9 Dec 2022 at 09:20, Juri Linkov wrote:

>> To complement this a bit, my hunch is that the problem is due to
>> `compilation-filter-start' being a number and not a marker (bug?).  When
>> `grep-filter' deletes some characters between `compilation-filter-start'
>> and the beginning of its line, it causes `ansi-color-compilation-filter'
>> to skip treating a portion of the buffer.
>
> Have you tried to replace a number with a marker?  Does it help to fix
> this bug?

I got the chance to try this now and unfortunately making
compilation-filter-start into a marker doesn't help with this particular
bug.

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Augusto Stoffel]

On Fri,  9 Dec 2022 at 09:20, Juri Linkov wrote:

> Have you tried to replace a number with a marker?  Does it help to fix
> this bug?

I've tried this and, although I think it's sane and should be done
anyway, it didn't solve this bug in particular.

bug#50207: 28.0.50; ansi-color-compilation-filter and rgrep

[Juri Linkov]

>> Have you tried to replace a number with a marker?  Does it help to fix
>> this bug?
>
> I've tried this and, although I think it's sane and should be done
> anyway, it didn't solve this bug in particular.

Does this patch solve the bug?

```
diff --git a/lisp/ansi-color.el b/lisp/ansi-color.el
index 5e7015db549..d7508759702 100644
--- a/lisp/ansi-color.el
+++ b/lisp/ansi-color.el
@@ -688,6 +688,7 @@ ansi-color-filter-region
          (start (cadr context)))
     (save-excursion
       (goto-char start)
+      (forward-line 0)
       ;; Delete escape sequences.
       (while (re-search-forward ansi-color-control-seq-regexp end-marker t)
         (delete-region (match-beginning 0) (match-end 0)))
@@ -727,6 +728,7 @@ ansi-color-apply-on-region
          (end-marker (copy-marker end)))
     (save-excursion
       (goto-char start-marker)
+      (forward-line 0)
       ;; Find the next escape sequence.
       (while (re-search-forward ansi-color-control-seq-regexp end-marker t)
         ;; Extract escape sequence.
```