bug#44465: 28.0.50; tcl-mode mismatched parenthesis for nested [strings]

bug#44465: 28.0.50; tcl-mode mismatched parenthesis for nested [strings]

[mvar]

After the applied fix in #39277, font locking fails in the following case:

set a "Testing: [split "192.168.1.1/24" "/"] address"

the last bracket ] reports "no matching parenthesis found"

this part:

192.168.1.1/24"

is not locked as a string, but "/" is.

Note that the whole argument is evaluated properly by tclsh as a
single contiguous string:

% set a "Testing: [split "192.168.1.1/24" "/"] address"
Testing: 192.168.1.1 24 address

i can't find a way around this behavior even with escaping the double
quotes or using single quotes, because doing that will change how the
argument is evaluated by tcl (array instead of string):

% set b "Testing: [split '192.168.1.1/24' '/'] address"
Testing: {} 192.168.1.1 24 {} address
% set a "Testing: [split \"192.168.1.1/24\" \"/\"] address"
Testing: {} 192.168.1.1 24 {} address
%

--

there's also another case of font locking failure, unrelated
to the #39277 changes:

puts {#test}

anything following # until end of line is locked as comment. But this
one is ok:

puts {a#test}

the inner a#test string is not locked as string but at least the
open-close brackets are marked properly.




In GNU Emacs 28.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.22.30, cairo version 1.15.12)
 of 2020-11-04 built on localhost
Repository revision: 2668bb47b03230400f94855a58b81a516b46acc5
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Red Hat Enterprise Linux Client 7.3 (Maipo)

Configured using:
 'configure --prefix=/opt/emacs-28_latest'

Configured features:
XPM JPEG TIFF GIF PNG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY INOTIFY
LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS
GTK3 X11 XDBE XIM MODULES THREADS JSON PDUMPER

Important settings:
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Major mode: Tcl

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
tcl easymenu comint ansi-color ring tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 56454 12331)
 (symbols 48 7150 1)
 (strings 32 20173 1496)
 (string-bytes 1 673817)
 (vectors 16 12950)
 (vector-slots 8 174548 13602)
 (floats 8 23 28)
 (intervals 56 216 0)
 (buffers 992 12)
 (heap 1024 14579 1030))

bug#44465: Still open after applying fix for #44834 and #44083

[Anticrisis]

This bug is still open after applying the patch in bug#44834. It would
be nice to add a failing test to test/lisp/progmodes/tcl-tests.el to
catch this, but I don't know how to automate tests for font lock
failures.

bug#44465: 28.0.50; tcl-mode mismatched parenthesis for nested [strings]

[mvar]

mvar <mvar.40k@gmail.com> writes:

> After the applied fix in #39277, font locking fails in the following case:
>
> set a "Testing: [split "192.168.1.1/24" "/"] address"
>
> the last bracket ] reports "no matching parenthesis found"
>
> this part:
>
> 192.168.1.1/24"
>
> is not locked as a string, but "/" is.
>
> Note that the whole argument is evaluated properly by tclsh as a
> single contiguous string:
>
> % set a "Testing: [split "192.168.1.1/24" "/"] address"
> Testing: 192.168.1.1 24 address
>
> i can't find a way around this behavior even with escaping the double
> quotes or using single quotes, because doing that will change how the
> argument is evaluated by tcl (array instead of string):
>
> % set b "Testing: [split '192.168.1.1/24' '/'] address"
> Testing: {} 192.168.1.1 24 {} address
> % set a "Testing: [split \"192.168.1.1/24\" \"/\"] address"
> Testing: {} 192.168.1.1 24 {} address
> %
>


i've been trying every once in a while to figure out a solution to this
bug but keep failing.

@Stefan i've added you to cc: in case you can help a bit and provide
some tips.
In the fix for #39277 you've added tcl--syntax-of-quote and also this comment:

(defconst tcl-syntax-propertize-function
  ;; FIXME: Handle the [...] commands nested inside "..." strings.

so my question, is the problem described in this bug report supposed to be fixed as
a regexp in the tcl-syntax-propertize-function rules list or should i try
to extend tcl--syntax-of-quote to cover this case too? I've failed in
both accounts so far, especially the regexp i can't find the proper
pattern to fix this and not break something else (which kind of proves the
saying "you fix 1 problem with regexp - you now have 2 problems" :)

thank you in advance,
Michalis

bug#44465: 28.0.50; tcl-mode mismatched parenthesis for nested [strings]

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> @Stefan i've added you to cc: in case you can help a bit and provide
> some tips.
> In the fix for #39277 you've added tcl--syntax-of-quote and also this comment:
>
> (defconst tcl-syntax-propertize-function
>   ;; FIXME: Handle the [...] commands nested inside "..." strings.
>
> so my question, is the problem described in this bug report supposed
> to be fixed as a regexp in the tcl-syntax-propertize-function rules
> list or should i try to extend tcl--syntax-of-quote to cover this case
> too?

Good question.
My gut tells me not to try and make the regexp more complex.


        Stefan