bug#41185: highlight-regexp not working properly

bug#41185: highlight-regexp not working properly

[jan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 6058 bytes --]

Hi,
was trying to pick out relevant preprocessor bits using highlight
regexp. Unexpected result. To reproduce

start emacs -Q

create a buffer, put it in C mode (M-x c-mode)

Yank in


#        ifdef VDW_CUTOFF_CHECK
    wco_vdw_S0 = (rsq_S0 < rcvdw2_S);
    wco_vdw_S1 = (rsq_S1 < rcvdw2_S);
#            ifndef HALF_LJ
    wco_vdw_S2 = (rsq_S2 < rcvdw2_S);
    wco_vdw_S3 = (rsq_S3 < rcvdw2_S);
#           endif
#        else
#            define wco_vdw_S0 wco_S0
#            define wco_vdw_S1 wco_S1
#            define wco_vdw_S2 wco_S2
#            define wco_vdw_S3 wco_S3
#        endif


Try to regexp-highlight particular preprocessor use:

M-s h r def HALF_LJ[^^]+?# endif RET

(the [^^] is because I want it to match multiline, and I don't expect
any carets in the text, so I negate that to allow newline matches and
make it lazy to pick out the shortest +?  It's sloppy but ok here)

Does not highlight anything (see
highlight-requested-but-no-highliting-occurred.PNG) - NOPE! MY
MISTAKE! Found out why while writing this - see note at end about lax
whitespace matching - my misunderstanding - the following still
applies though.

Now manually *type in* some text that should match above it:


def HALF_LJ
hello buglist
# endif


That doesn't get highlighted either.

Now put the cursor straight after the HALF_LJ and press delete to
remove the carriage return and get this:


def HALF_LJhello buglist
# endif


and highlight appears.
Press return to restore it to


def HALF_LJ
hello buglist
# endif


and the whole phrase stays highlighted (See
2.highlighted-after-delete-return-then-add-return.PNG)

It's possible to get it partially highlighted, which doesn't make
sense, see 3. improperly-partially-highlighted.PNG

Perhaps this is happening just in C-mode? Try it in scratch. It also
works intermittently, can also get incorrect partial highlight, see 4.
Similarly-in-scratch-buffer (note that the # endif is not yellow).


Note that the matching text further down (the lump of code I said to
yank in, above) *may* or *may not* get highlighted.
If I play with it, deleteing and adding back stuff, sometimes it will,
sometimes it gets partially highlighted - this is so even if I correct
for my misunderstanding - see next para.

(Note if you're playing around, I relied on lax whitespace matching
<https://www.emacswiki.org/emacs/RegularExpression>
"Idiosyncrasies of Emacs Regular Expressions - In an interactive
search, a space character stands for one or more whitespace
characters..."
however I got tripped up by the fact that lax whitespace matching does
not apply to highlight-regexp).

I can also get some degree of similar weirdness in ubuntu 19.10
(started normally, that is without -Q) in both *scratch* and c-mode.

jan


In GNU Emacs 26.3 (build 1, x86_64-w64-mingw32)
 of 2019-08-29 built on CIRROCUMULUS
Repository revision: 96dd0196c28bc36779584e47fffcca433c9309cd
Windowing system distributor 'Microsoft Corp.', version 6.1.7601
Recent messages:
Mark set [3 times]
Auto-saving...done
Mark set [4 times]
Auto-saving...done
Auto-saving...done
Auto-saving...done
Mark set [2 times]
Auto-saving...done
Saving file c:/Users/jan/Desktop/highlight-regexp-bug/bug.txt...
Wrote c:/Users/jan/Desktop/highlight-regexp-bug/bug.txt

Configured using:
 'configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS THREADS LCMS2

Important settings:
  value of $LANG: ENG
  locale-coding-system: cp1252

Major mode: Text

Minor modes in effect:
  desktop-save-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg gnus-util rmail rmail-loaddefs
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils misearch multi-isearch dabbrev browse-url url-util elec-pair
helm-gtags subr-x pulse which-func imenu helm-files helm-buffers
helm-occur helm-tags helm-locate helm-grep helm-regexp format-spec
helm-utils helm-help helm-types helm easy-mmode helm-source eieio-compat
helm-multi-match helm-lib async edmacro kmacro desktop frameset
cus-start cus-load finder-inf info package easymenu epg-config
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib time-date mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32
ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads w32notify w32
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 162243 12928)
 (symbols 48 26155 1)
 (miscs 40 61 209)
 (strings 32 49502 1493)
 (string-bytes 1 1501682)
 (vectors 16 23149)
 (vector-slots 8 594667 6206)
 (floats 8 89 268)
 (intervals 56 242 0)
 (buffers 992 12))

[-- Attachment #2: 1. highlight-requested-but-no-highliting-occurred.PNG --]
[-- Type: image/png, Size: 18847 bytes --]

[-- Attachment #3: 2.highlighted-after-delete-return-then-add-return.PNG --]
[-- Type: image/png, Size: 17923 bytes --]

[-- Attachment #4: 3. improperly-partially-highlighted.PNG --]
[-- Type: image/png, Size: 12195 bytes --]

[-- Attachment #5: 4. Similarly-in-scratch-buffer.PNG --]
[-- Type: image/png, Size: 8445 bytes --]

bug#41185: highlight-regexp not working properly

[Michael Heerdegen]

jan <rtm443x@googlemail.com> writes:

> Try to regexp-highlight particular preprocessor use:
>
> M-s h r def HALF_LJ[^^]+?# endif RET
>
> (the [^^] is because I want it to match multiline, and I don't expect
> any carets in the text, so I negate that to allow newline matches and
> make it lazy to pick out the shortest +?  It's sloppy but ok here)
>
> Does not highlight anything

Yes, thanks.

I can reproduce this.  I'm not a font-lock expert, but I suspect that
hi-lock doesn't support patterns matching multi-line text.  Font-lock
supports multi-line matching, but this needs some extra efforts to work,
and I don't think hi-lock is caring about this at all.

If this is correct, maybe this feature can be implemented, and until
then, this limitation should be explicitly documented.


Michael.

bug#41185: highlight-regexp not working properly for multiline-matching regexps

[Lars Ingebrigtsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> jan <rtm443x@googlemail.com> writes:
>
>> Try to regexp-highlight particular preprocessor use:
>>
>> M-s h r def HALF_LJ[^^]+?# endif RET
>>
>> (the [^^] is because I want it to match multiline, and I don't expect
>> any carets in the text, so I negate that to allow newline matches and
>> make it lazy to pick out the shortest +?  It's sloppy but ok here)
>>
>> Does not highlight anything
>
> Yes, thanks.
>
> I can reproduce this.  I'm not a font-lock expert, but I suspect that
> hi-lock doesn't support patterns matching multi-line text.

I can reproduce what the original bug report is seeing -- but that's
because the regexp seems to be wrong?  There's nothing in the test file
that matches "# endif".

However the following call does properly match the region I think was
intended, and works fine, as far as I can see (in Emacs 26.1 and 28):

(highlight-regexp "def HALF_LJ[^^]+?# +endif")

Or do I misunderstand what the original problem was?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41185: highlight-regexp not working properly for multiline-matching regexps

[Michael Heerdegen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I can reproduce what the original bug report is seeing -- but that's
> because the regexp seems to be wrong?  There's nothing in the test file
> that matches "# endif".
>
> However the following call does properly match the region I think was
> intended, and works fine, as far as I can see (in Emacs 26.1 and 28):
>
> (highlight-regexp "def HALF_LJ[^^]+?# +endif")
>
> Or do I misunderstand what the original problem was?

I think you must read the original report further, where there is a text
example with "endif" for that highlighting fails.  AFAICT the complaint
is about multi-line matches not being highlighted.

I think `highlight-regexp' is not able to highlight multiline
stuff, but the docstring doesn't tell anything about this.

Michael.

bug#41185: highlight-regexp not working properly for multiline-matching regexps

[Lars Ingebrigtsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> I think you must read the original report further, where there is a text
> example with "endif" for that highlighting fails.  AFAICT the complaint
> is about multi-line matches not being highlighted.
>
> I think `highlight-regexp' is not able to highlight multiline
> stuff, but the docstring doesn't tell anything about this.

It is being highlit, but not reliably, I see.  If you paste the test
cases in, they are highlit, but not if you type them in (or paste them
in in steps)...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#41185: highlight-regexp not working properly for multiline-matching regexps

[jan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

To reinforce Michael Heerdegen's point, now it seems to be failing on
trivial single-line strings too:

start emacs with -Q, open a new buffer, say 'asdf'

type in 'silly'

Highlight it; M-s h r silly ret

existing silly gets highligted.

Press return a couple of times then type in 'silly' again, does not
highlight (see silly_twice_only_one_got_highlighted.PNG)

Turn off highlighting (M-s h u) then back on again (M-s h r silly ret)
and both are now highlighted (see silly_both_highlited.PNG), showing
incremental highlighting is indeed failing.

regards

jan

On 13/06/2021, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
>> I think you must read the original report further, where there is a text
>> example with "endif" for that highlighting fails.  AFAICT the complaint
>> is about multi-line matches not being highlighted.
>>
>> I think `highlight-regexp' is not able to highlight multiline
>> stuff, but the docstring doesn't tell anything about this.
>
> It is being highlit, but not reliably, I see.  If you paste the test
> cases in, they are highlit, but not if you type them in (or paste them
> in in steps)...
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[-- Attachment #2: silly_twice_only_one_got_highlighted.PNG --]
[-- Type: image/png, Size: 7206 bytes --]

[-- Attachment #3: silly_both_highlited.PNG --]
[-- Type: image/png, Size: 7714 bytes --]

bug#41185: highlight-regexp not working properly for multiline-matching regexps

[Michael Heerdegen]

jan <rtm443x@googlemail.com> writes:

> start emacs with -Q, open a new buffer, say 'asdf'

> type in 'silly'
>
> Highlight it; M-s h r silly ret
>
> existing silly gets highligted.
>
> Press return a couple of times then type in 'silly' again, does not
> highlight (see silly_twice_only_one_got_highlighted.PNG)

Highlighting is not updated if overlays need to be used because the
buffer does not use font-lock-mode (see the docstring of
`highlight-regexp').

Were you aware of that fact, or does your experiment fail even for
"supported" buffers?


Regards,

Michael.

bug#41185: highlight-regexp not working properly for multiline-matching regexps

[jan via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ach, no, you're right. Tried it in C-mode, was ok.

I think Elil pointed this out in my original bug report about a year
ago and I forgot.

My mistake

jan

On 16/06/2021, Michael Heerdegen <michael_heerdegen@web.de> wrote:
> jan <rtm443x@googlemail.com> writes:
>
>> start emacs with -Q, open a new buffer, say 'asdf'
>
>> type in 'silly'
>>
>> Highlight it; M-s h r silly ret
>>
>> existing silly gets highligted.
>>
>> Press return a couple of times then type in 'silly' again, does not
>> highlight (see silly_twice_only_one_got_highlighted.PNG)
>
> Highlighting is not updated if overlays need to be used because the
> buffer does not use font-lock-mode (see the docstring of
> `highlight-regexp').
>
> Were you aware of that fact, or does your experiment fail even for
> "supported" buffers?
>
>
> Regards,
>
> Michael.
>