bug#28962: inserting newline at start of line in HERE doc breaks syntax highlighting

bug#28962: inserting newline at start of line in HERE doc breaks syntax highlighting

[Troy Hinckley]

[-- Attachment #1: Type: text/plain, Size: 5467 bytes --]

From: tjhinckl@sccj002243.i-did-not-set--mail-host-address--so-tickle-me
(troy.j.hinckley)
To: bug-gnu-emacs@gnu.org
Subject: 25.1; cperl syntax highlighting broken
--text follows this line--

When inside a cperl HERE doc, using any function that inserts a newline at
the start of a line will causes the rest of the file to highlighted with
font-lock-comment-face. The only way to fix this is to make changes to the
correctly highlighted part of the HERE doc which causes it to re-parse. My
guess is that cperl is getting mixed up with POD highlighting and trying to
apply cperl-pod-face since POD's and HERE documents are very closely
related in cperl-mode code.

To reproduce:
use the minimal cperl-mode file below
---------------------------------------------
my $here_doc = <<'_HERE_';
this is here doc line

_HERE_
---------------------------------------------
call `M-: (insert "\n")` when cursor is at the start of any line
inside the HERE doc.

You will see that the rest of the file is highlighted with the comment
face. Calling any function that finds the HERE doc region will be incorrect
(e.g. cperl-narrow-to-here-doc)



In GNU Emacs 25.1.1 (x86_64-suse-linux-gnu, GTK+ Version 2.18.6)
 of 2016-10-09 built on plxv1010
Windowing system distributor 'The X.Org Foundation', version 11.0.60900000
System Description: SUSE Linux Enterprise Server 11 (x86_64)

Configured using:
 'configure --prefix=/usr/intel/pkgs/emacs/25.1
 --prefix=/usr/intel/pkgs/emacs/25.1
 --libdir=/usr/intel/pkgs/emacs/25.1/lib64
 --x-includes=/usr/intel/pkgs/X11/R7.7/include
 --x-libraries=/usr/intel/pkgs/X11/R7.7/lib64 --with-x-toolkit=gtk2
 --without-sound --without-gif --build=x86_64-suse-linux
 --host=x86_64-suse-linux --target=x86_64-suse-linux CFLAGS= CPPFLAGS=
 'LDFLAGS=-O2 -fPIC -L/usr/intel/pkgs/emacs/25.1/lib64
 -L/usr/intel/pkgs/emacs/25.1/lib -Wl,--rpath
 -Wl,/usr/intel/pkgs/emacs/25.1/lib64:/usr/intel/pkgs/emacs/25.1/lib:/usr/intel/pkgs/emacs/25.1/lib64:/usr/intel/pkgs/emacs/25.1/lib:/usr/intel/pkgs/emacs/25.1/lib:/usr/intel/pkgs/emacs/25.1/lib64:/usr/intel/pkgs/gtk+/2.18.6-64/lib64:/usr/intel/pkgs/atk/1.28.0-64/lib64:/usr/intel/pkgs/pango/1.26.2-64/lib64:/usr/intel/pkgs/cairo/1.8.8-64/lib:/usr/intel/pkgs/poppler/0.12.1-64/lib:/usr/intel/pkgs/pixman/0.16.2-64/lib:/usr/intel/pkgs/X11/R7.5-64/lib:/usr/intel/pkgs/groff/1.20.1-64/:/usr/intel/pkgs/fontconfig/2.7.3-64/lib64:/usr/intel/pkgs/libexpat/2.0.1-64/lib:/usr/intel/pkgs/freetype/2.3.7-64/lib:/usr/intel/pkgs/zlib/1.2.x-64/lib64:/usr/intel/pkgs/libpng/1.2.40-64/lib:/usr/intel/pkgs/libxml2/2.7.6-64/lib64:/usr/intel/pkgs/lcms/1.19-64/lib64:/usr/intel/pkgs/tiff/3.9.1-64/lib64:/usr/intel/pkgs/zlib/1.2.x-64/lib64:/usr/intel/pkgs/jpeg/6b-64/lib:/usr/intel/pkgs/glib/2.22.3-64/lib64:/usr/intel/pkgs/libgettext/0.17-2-64/lib64:/usr/intel/pkgs/ncurses/5.6-64/lib64:/usr/intel/pkgs/libpng/1.2.40-64/lib:/usr/intel/pkgs/zlib/1.2.x-64/lib64:/usr/intel/pkgs/libiconv/1.13.1-64/lib:''

Configured features:
XPM JPEG TIFF PNG GPM NOTIFY ACL FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS
GTK2 X11

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LC_COLLATE: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: CPerl

Minor modes in effect:
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg gnus-util mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util help-fns mail-prsvr mail-utils cperl-mode
use-package diminish bind-key easy-mmode finder-inf advice edmacro
kmacro rx info package epg-config seq byte-opt gv bytecomp byte-compile
cl-extra help-mode easymenu cconv cl-loaddefs pcase cl-lib time-date
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote inotify dynamic-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 129555 5523)
 (symbols 48 24348 0)
 (miscs 40 142 120)
 (strings 32 30653 5834)
 (string-bytes 1 1166381)
 (vectors 16 17248)
 (vector-slots 8 524591 2239)
 (floats 8 198 61)
 (intervals 56 260 0)
 (buffers 976 17)
 (heap 1024 17165 1148))

[-- Attachment #2: Type: text/html, Size: 6692 bytes --]

bug#28962: A workaround to this issue is available in cperl-mode

[Harald Jörg]

The root cause is that within HERE documents and PODs there's no
syntax re-checking for every character inserted.

To recover from such a situation, cperl-mode offers a
command `cperl-find-pods-heres', also available from the menu
'Perl' -> 'Refresh "hard" constructions'.  A function which inserts
text in a POD or HERE document should call cperl-find-pods-heres
afterwards to avoid the issue.
-- 
Cheers,
haj

bug#28962: A workaround to this issue is available in cperl-mode

[Stefan Kangas]

Harald Jörg <haj@posteo.de> writes:

> The root cause is that within HERE documents and PODs there's no
> syntax re-checking for every character inserted.
>
> To recover from such a situation, cperl-mode offers a
> command `cperl-find-pods-heres', also available from the menu
> 'Perl' -> 'Refresh "hard" constructions'.  A function which inserts
> text in a POD or HERE document should call cperl-find-pods-heres
> afterwards to avoid the issue.

Is that something we should work on improving, or does that mean that we
should close this as wontfix?

bug#28962: A workaround to this issue is available in cperl-mode

[Harald Jörg]

On 8/26/20 10:21 PM, Stefan Kangas wrote:
> Harald Jörg writes:
> 
>> The root cause is that within HERE documents and PODs there's no
>> syntax re-checking for every character inserted.
>>
>> To recover from such a situation, cperl-mode offers a
>> command `cperl-find-pods-heres', also available from the menu
>> 'Perl' -> 'Refresh "hard" constructions'.  A function which inserts
>> text in a POD or HERE document should call cperl-find-pods-heres
>> afterwards to avoid the issue.
> 
> Is that something we should work on improving, or does that mean that we
> should close this as wontfix?

After some more digging (my experience with elisp is still limited)
I found that the problem can be avoided by using `insert-and-inherit'
instead of `insert'.  cperl-mode uses text properties to detect the
HEREiness of buffer contents - and `insert' just doesn't provide them.

So, the problem only occurs when text is inserted via elisp code. It
can be avoided by using `insert-and-inherit', or, if one can't modify
the function, recovered from by calling `cperl-find-pods-heres'.  In
my eyes this is good enough.  It seems that the reporter wasn't aware
of either possibility, so maybe they're happy if we tell them about
the workarounds - and then close as wontfix.
-- 
Cheers,
haj

bug#28962: A workaround to this issue is available in cperl-mode

[Stefan Kangas]

Harald Jörg <haj@posteo.de> writes:

> After some more digging (my experience with elisp is still limited)
> I found that the problem can be avoided by using `insert-and-inherit'
> instead of `insert'.  cperl-mode uses text properties to detect the
> HEREiness of buffer contents - and `insert' just doesn't provide them.
>
> So, the problem only occurs when text is inserted via elisp code. It
> can be avoided by using `insert-and-inherit', or, if one can't modify
> the function, recovered from by calling `cperl-find-pods-heres'.  In
> my eyes this is good enough.  It seems that the reporter wasn't aware
> of either possibility, so maybe they're happy if we tell them about
> the workarounds - and then close as wontfix.

Makes sense to me.  I think you should do that.

bug#28962: (no subject)

[Harald Jörg]

merge 28962 14343
thanks

After some more digging, I'm merging this bug with Bug#14343.

The root cause of both bugs is the same: Here-documents in CPerl mode
are marked with text properties.  Normal editing (in Emacs: via
`self-insert-command' will make new characters inherit from their
surroundings, so all is fine.

If, however, text is entered with another elisp function, then no
automatic inheritance takes place.  Examples are `insert' (this bug
report), `yank' (Bug#14343), but also `replace-string' if the first
character of the here-document gets replaced.

I am confident that this can be fixed if CPerl mode "steals" from Perl
mode and marks here-documents as c-style comments for fontification.
I am going to prepare a patch for this.

-- 
Cheers,
haj

bug#28962: cperl-mode: Inserting text into here-documents now works

[Harald Jörg]

The bug which made CPerl mode mistreat changes to here-documents has
been fixed in the repository, for the cases of (insert "\n") (Bug#28962)
and yanking (Bug#14343) in particular, and in general for changes by
elisp code.

cperl-mode.el from the repository can be used with Emacs 26.1 or newer.
-- 
Cheers,
haj

bug#28962: cperl-mode: Inserting text into here-documents now works

[Vincent Lefevre]

Hi,

On 2021-08-31 07:04:31 +0000, Harald Jörg wrote:
> The bug which made CPerl mode mistreat changes to here-documents has
> been fixed in the repository, for the cases of (insert "\n") (Bug#28962)
> and yanking (Bug#14343) in particular, and in general for changes by
> elisp code.
> 
> cperl-mode.el from the repository can be used with Emacs 26.1 or newer.

Concerning my bug #14343, I had noted 4 years ago in the associated
Debian bug that I could no longer reproduce the issue in Emacs 25:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=706701#17

So it seems that there had already been some fixes in the past...

(I could also check that there were no regressions in 26.1 and 27.1.)

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#14343: cperl-mode: Inserting text into here-documents now works

[Harald Jörg]

Hello,

> On 2021-08-31 07:04:31 +0000, Harald Jörg wrote:
>> The bug which made CPerl mode mistreat changes to here-documents has
>> been fixed in the repository, for the cases of (insert "\n") (Bug#28962)
>> and yanking (Bug#14343) in particular, and in general for changes by
>> elisp code.
>> 
>> cperl-mode.el from the repository can be used with Emacs 26.1 or newer.
>
> Concerning my bug #14343, I had noted 4 years ago in the associated
> Debian bug that I could no longer reproduce the issue in Emacs 25:
>
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=706701#17

Thanks for that info!  I admit that I haven't checked the fate of the
Debian bug report when I started testing, but I can confirm that with
your particular recipe, the bug does not appear.

> So it seems that there had already been some fixes in the past...

The bug was still open in the Emacs bugtracker, so I did some tests and
found that when I yank a region from regions _outside_ of a here-doc
text, then fontification still wasn't correct (without that fix).  For a
colorful example, triple-click the line "print <<EOF;" and paste
with the middle mouse button.

So, the fix in Emacs 25 was only partial.  As of now, whatever you paste
into the here-document, should be fontified correctly, so eventually
I've closed the bug in the Emacs bugtracker.
-- 
Cheers, and thanks for reporting back,
haj

bug#14343: cperl-mode: Inserting text into here-documents now works

[Vincent Lefevre]

On 2021-08-31 11:56:55 +0000, Harald Jörg wrote:
> The bug was still open in the Emacs bugtracker, so I did some tests and
> found that when I yank a region from regions _outside_ of a here-doc
> text, then fontification still wasn't correct (without that fix).  For a
> colorful example, triple-click the line "print <<EOF;" and paste
> with the middle mouse button.

Indeed, under Debian/unstable (with its GNU Emacs 27.1), I can confirm
that the bug was still occurring when pasting this after "foo" (but
not just before), and that everything is OK with the cperl-mode.el
from the repository.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)