bug#41522: 26.3; Narrowing can prevent syntax propertization

bug#41522: 26.3; Narrowing can prevent syntax propertization

[Ikumi Keita]

[-- Attachment #1: Type: text/plain, Size: 7398 bytes --]

When syntax propertization occurs inside narrowed area of a buffer,
the outside area sometimes doesn't get syntax propertization even after
the narrowing is removed.

[How to reproduce]
1. Save the attached sample file as /tmp/sample.sh
2. emacs -Q -geometry 80x25 /tmp/sample.sh
3. Type C-u C-n C-e to go to the end of the (visible) fourth line.
4. Type C-x C-e as the comment there instructs.
5. Then the face of here-doc bleeds over the "EOF" token and continues
   to the end of the buffer.
   Expected result is that the here-doc face ends at "EOF" and the rest
   of the buffer is fontified with various suitable faces.

[Analysis]
In the above example, `narrow-to-region' restricts the buffer into the
last three lines of Local Variables section where syntax propertization
is not yet done. There, `syntax-ppss' triggers syntax propertization and
set `syntax-propertize--done' to (point-max). But actually that
`syntax-ppss' didn't syntax-properize outside the narrowing so it didn't
give the suitable `syntax-table' property to the newline following the
"EOF". Since `syntax-propertize--done' tells emacs that all portion of
the buffer has been propertized already, emacs does not give the
suitable `syntax-table' property to the newline following the "EOF" even
after `widen'. Thus the fontification of here-doc continues beyond "EOF".

In a similar way, if a lisp function goes to the point where syntax
propertization has not yet took place, makes narrowing and calls some
syntax-aware function which sets `syntax-propertize--done' to a value
inside the narrowing, a large portion of the buffer can be left without
suitable `syntax-table' property. This can lead to incorrect result
for not only font-lock but also syntax-aware functions such as
`forward-comment'.

Best regards,
Ikumi Keita

In GNU Emacs 26.3 (build 1, x86_64-unknown-freebsd12.0, GTK+ Version 3.24.9)
 of 2019-08-29 built on freebsd.vmware
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
Recent messages:
Indentation variables are now local.
Indentation setup for shell type sh
 [2 times]
Mark set
Saving file /home/keita/scr/sample.sh...
Wrote /home/keita/scr/sample.sh
Setting up indent for shell type sh
Indentation variables are now local.
Indentation setup for shell type sh


Configured using:
 'configure --with-canna --with-canna-includes=/usr/local/canna/include
--with-canna-libraries=/usr/local/canna/lib --without-xim
--disable-largefile --with-sound=yes --with-file-notification=yes
CFLAGS=-O3'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY ACL GNUTLS
LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS
LCMS2

Important settings:
  value of $EMACSLOADPATH: /home/keita/elisp:
  value of $LANG: ja_JP.eucJP
  locale-coding-system: japanese-iso-8bit-unix

Major mode: Info

Minor modes in effect:
  TeX-PDF-mode: t
  global-magit-file-mode: t
  diff-auto-refine-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
~/elisp/preview hides /usr/local/share/emacs/site-lisp/auctex/preview
~/elisp/reftex-parse hides /usr/local/share/emacs/26.3/lisp/textmodes/reftex-parse
/home/keita/.emacs.d/elpa/soap-client-3.1.5/soap-client hides /usr/local/share/emacs/26.3/lisp/net/soap-client
/home/keita/.emacs.d/elpa/soap-client-3.1.5/soap-inspect hides /usr/local/share/emacs/26.3/lisp/net/soap-inspect

Features:
(ibuf-ext ibuffer ibuffer-loaddefs shadow mh-mime mh-identity
mh-tool-bar mh-letter mh-show goto-addr gnus-cite gnus-art mm-uu mml2015
gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-win gnus-range gnus
nnheader wid-edit mh-xface mh-utils mh-comp mh-scan mh-gnus mm-view
mml-smime smime dig mh-e mh-compat mh-buffers mh-loaddefs emacsbug
sendmail sh-script smie executable perl-mode magit-extras vc-git pp
preview prv-emacs reftex-dcr reftex-auc reftex reftex-loaddefs
reftex-vars tex-bar tex-buf toolbar-x noutline outline font-latex latex
latex-flymake flymake-proc flymake warnings tex-ispell tex-style tex
dbus tex-mode compile latexenc misearch multi-isearch eieio-opt speedbar
sb-image ezimage dframe find-func cl-print thingatpt help-fns radix-tree
bug-reference edmacro kmacro magit-submodule magit-obsolete magit-blame
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch
magit-clone magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit magit-repos magit-apply
magit-wip magit-log which-func imenu magit-diff smerge-mode diff-mode
magit-core magit-autorevert autorevert filenotify magit-margin
magit-transient magit-process magit-mode git-commit transient magit-git
magit-section magit-utils crm log-edit easy-mmode message dired
dired-loaddefs format-spec rfc822 mml mml-sec gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mailabbrev mail-utils
gmm-utils mailheader pcvs-util add-log with-editor async-bytecomp async
shell pcomplete comint ansi-color ring server subr-x dash skeleton
cl-extra help-mode parse-time ucs-normalize json map epa derived epg
url-http tls gnutls url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr url-gw nsm rmc puny jka-compr twittering-mode
advice url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util mailcap xml elec-pair preview-latex
auto-loads tex-site canna-im finder-inf info package easymenu epg-config
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib time-date mule-util japan-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind kqueue lcms2 dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 813972 71846)
 (symbols 48 48314 42)
 (miscs 40 4992 2861)
 (strings 32 186223 9232)
 (string-bytes 1 5882079)
 (vectors 16 83667)
 (vector-slots 8 2273246 58558)
 (floats 8 330 584)
 (intervals 56 12890 4833)
 (buffers 992 41))

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: sample to reproduce bug --]
[-- Type: text/x-shellscript, Size: 3914 bytes --]

#!/bin/sh

# Type C-x C-e at the end of the next line:
# (progn (goto-char (point-max)) (narrow-to-region (point) (- (point) 50)) (syntax-ppss) (widen))

cat <<EOF
Nullam eu ante vel est convallis dignissim.  Fusce suscipit, wisi nec facilisis facilisis, est dui fermentum leo, quis tempor ligula erat quis odio.  Nunc porta vulputate tellus.  Nunc rutrum turpis sed pede.  Sed bibendum.  Aliquam posuere.  Nunc aliquet, augue nec adipiscing interdum, lacus tellus malesuada massa, quis varius mi purus non odio.  Pellentesque condimentum, magna ut suscipit hendrerit, ipsum augue ornare nulla, non luctus diam neque sit amet urna.  Curabitur vulputate vestibulum lorem.  Fusce sagittis, libero non molestie mollis, magna orci ultrices dolor, at vulputate neque nulla lacinia eros.  Sed id ligula quis est convallis tempor.  Curabitur lacinia pulvinar nibh.  Nam a sapien.

Pellentesque dapibus suscipit ligula.  Donec posuere augue in quam.  Etiam vel tortor sodales tellus ultricies commodo.  Suspendisse potenti.  Aenean in sem ac leo mollis blandit.  Donec neque quam, dignissim in, mollis nec, sagittis eu, wisi.  Phasellus lacus.  Etiam laoreet quam sed arcu.  Phasellus at dui in ligula mollis ultricies.  Integer placerat tristique nisl.  Praesent augue.  Fusce commodo.  Vestibulum convallis, lorem a tempus semper, dui dui euismod elit, vitae placerat urna tortor vitae lacus.  Nullam libero mauris, consequat quis, varius et, dictum id, arcu.  Mauris mollis tincidunt felis.  Aliquam feugiat tellus ut neque.  Nulla facilisis, risus a rhoncus fermentum, tellus tellus lacinia purus, et dictum nunc justo sit amet elit.

Pellentesque dapibus suscipit ligula.  Donec posuere augue in quam.  Etiam vel tortor sodales tellus ultricies commodo.  Suspendisse potenti.  Aenean in sem ac leo mollis blandit.  Donec neque quam, dignissim in, mollis nec, sagittis eu, wisi.  Phasellus lacus.  Etiam laoreet quam sed arcu.  Phasellus at dui in ligula mollis ultricies.  Integer placerat tristique nisl.  Praesent augue.  Fusce commodo.  Vestibulum convallis, lorem a tempus semper, dui dui euismod elit, vitae placerat urna tortor vitae lacus.  Nullam libero mauris, consequat quis, varius et, dictum id, arcu.  Mauris mollis tincidunt felis.  Aliquam feugiat tellus ut neque.  Nulla facilisis, risus a rhoncus fermentum, tellus tellus lacinia purus, et dictum nunc justo sit amet elit.

Nullam eu ante vel est convallis dignissim.  Fusce suscipit, wisi nec facilisis facilisis, est dui fermentum leo, quis tempor ligula erat quis odio.  Nunc porta vulputate tellus.  Nunc rutrum turpis sed pede.  Sed bibendum.  Aliquam posuere.  Nunc aliquet, augue nec adipiscing interdum, lacus tellus malesuada massa, quis varius mi purus non odio.  Pellentesque condimentum, magna ut suscipit hendrerit, ipsum augue ornare nulla, non luctus diam neque sit amet urna.  Curabitur vulputate vestibulum lorem.  Fusce sagittis, libero non molestie mollis, magna orci ultrices dolor, at vulputate neque nulla lacinia eros.  Sed id ligula quis est convallis tempor.  Curabitur lacinia pulvinar nibh.  Nam a sapien.

Pellentesque dapibus suscipit ligula.  Donec posuere augue in quam.  Etiam vel tortor sodales tellus ultricies commodo.  Suspendisse potenti.  Aenean in sem ac leo mollis blandit.  Donec neque quam, dignissim in, mollis nec, sagittis eu, wisi.  Phasellus lacus.  Etiam laoreet quam sed arcu.  Phasellus at dui in ligula mollis ultricies.  Integer placerat tristique nisl.  Praesent augue.  Fusce commodo.  Vestibulum convallis, lorem a tempus semper, dui dui euismod elit, vitae placerat urna tortor vitae lacus.  Nullam libero mauris, consequat quis, varius et, dictum id, arcu.  Mauris mollis tincidunt felis.  Aliquam feugiat tellus ut neque.  Nulla facilisis, risus a rhoncus fermentum, tellus tellus lacinia purus, et dictum nunc justo sit amet elit.
EOF

echo "abc"

### Local Variables:
### mode: sh
### End:

bug#41522: 26.3; Narrowing can prevent syntax propertization

[Dmitry Gutov]

Hi!

On 25.05.2020 11:42, Ikumi Keita wrote:
> In a similar way, if a lisp function goes to the point where syntax
> propertization has not yet took place, makes narrowing and calls some
> syntax-aware function which sets `syntax-propertize--done' to a value
> inside the narrowing, a large portion of the buffer can be left without
> suitable `syntax-table' property. This can lead to incorrect result
> for not only font-lock but also syntax-aware functions such as
> `forward-comment'.

Our current stance on the matter is that, in the simple case, code 
should call syntax-ppss from "widened" state. And the code which doesn't 
should be fixed. Or if it really need to do that (for some reason), it 
should take care to handle any undesired sife-effects itself.

See the definition of syntax-ppss-wide, which contains caches for both 
narrowed and "wide" syntax states.

bug#41522: 26.3; Narrowing can prevent syntax propertization

[Ikumi Keita]

Hi Dmitry, thanks for your reply.

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
> Our current stance on the matter is that, in the simple case, code
> should call syntax-ppss from "widened" state.

I suppose that this means that the code should call syntax-ppss (or
something like it) "before" narrowing is done, right? This code works
for me
----------------------------------------------------------------------
(progn (goto-char (point-max))
       (syntax-propertize (point)) ; <-
       (narrow-to-region (point) (- (point) 50))
       (syntax-ppss)
       (widen))
----------------------------------------------------------------------
while these don't:
----------------------------------------------------------------------
(progn (goto-char (point-max))
       (narrow-to-region (point) (- (point) 50))
       (syntax-ppss)
       (widen)
       (syntax-ppss) ; <-
)
----------------------------------------------------------------------
----------------------------------------------------------------------
(progn (goto-char (point-max))
       (narrow-to-region (point) (- (point) 50))
       (syntax-ppss)
       (widen)
       (syntax-propertize (point)) ; <-
)
----------------------------------------------------------------------

> And the code which doesn't should be fixed. Or if it really need to do
> that (for some reason), it should take care to handle any undesired
> sife-effects itself.

Hmm, then elisp programmers have to check whether each function used in
the code uses narrowing+syntactic functions inside it if this kind of
problem happens. I feel it a bit troublesome but at the same time I
understand it is very difficult to solve this problem cleanly.

OK, I try to be aware of this matter afterwards. Thanks for your
advice!

Regards,
Ikumi Keita

bug#41522: 26.3; Narrowing can prevent syntax propertization

[Dmitry Gutov]

On 25.05.2020 20:02, Ikumi Keita wrote:
> I suppose that this means that the code should call syntax-ppss (or
> something like it) "before" narrowing is done, right?

Or call (syntax-ppss-flush-cache (point-min)) afterward.

 > Hmm, then elisp programmers have to check whether each function used 
in the code uses narrowing+syntactic functions inside it if this kind of
problem happens. I feel it a bit troublesome but at the same time I 
understand it is very difficult to solve this problem cleanly.

One has to be careful with narrowing. It's an old problem.

 > OK, I try to be aware of this matter afterwards. Thanks for your
advice!

Welcome!