bug#8218: 23.2; scan-sexps sees inconsistent syntax

bug#8218: 23.2; scan-sexps sees inconsistent syntax

[Dima Kogan]

<#part type="text/x-perl" filename="/tmp/tst3.pm" disposition=inline description="bug demo">
<#/part>

I am seeing (scan-sexps) show inconsistent behavior depending on WHEN it
is evaluated. If the attached sample file is opened, the point placed at
the first '{', and (scan-sexps (point) 1) evaluated, the returned
position is at times of the first '}' (wrong) and at other times of the
second '}' (right). If the evaluation is done right when the file is
opened, I observe the wrong behavior. This is reliably triggered with

$ emacs -Q /tmp/tst3.pm --eval '(goto-char (scan-sexps 5 1))'

Any subsequent evaluation of the same form in that buffer produces the
correct behavior. If I turn off font-lock-mode, I consistently get
incorrect behavior. This has wide ramifications with occasional
incorrect indentation and hide-show operation, for instance.

I looked through the source a bit, and I see an inconsistency in the
syntax code returned by the
   code = SYNTAX_WITH_MULTIBYTE_CHECK (c)

line in scan_lists() in syntax.c . When things work, the '$' is
interpreted as punctuation and the following '{' as a block-open. When
things don't work, the '$' is a quote and the '{' is skipped
altogether:

| character | incorrect syntax | correct syntax |
|-----------+------------------+----------------|
| =         | Spunct           | Spunct         |
|           | Swhitespace      | Swhitespace    |
| $         | Scharquote       | Spunct         |
| {         |                  | Sopen          |
| P         | Sword            | Sword          |

syntax.c is a bit difficult to dive into, but it would be great if
someone who has experience with it would take a look.



In GNU Emacs 23.2.1 (x86_64-pc-linux-gnu, GTK+ Version 2.20.1)
 of 2010-12-11 on brahms, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.10707000
configured using `configure  '--build' 'x86_64-linux-gnu' '--build' 'x86_64-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.2/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/23.2/leim' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2' 'LDFLAGS=-g' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Perl

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<up> <up> M-: <help-echo> <down-mouse-2> <mouse-2> 
<return> <up> <up> <up> <up> <up> <up> <up> <down> 
M-x r e p o r t SPC b u g <return> s c a <backspace> 
a n - s e x p s SPC g e n e r a t e s SPC i n c o n 
s i s t e n t SPC s y n t a x C-SPC <home> M-w C-g 
C-g C-g <up> <up> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> M-: ( p o i n t ) <return> <up> M-x <up> <return> 
C-g M-x <up> <return>

Recent messages:
Loading /etc/emacs/site-start.d/50psvn.el (source)...done
Loading /etc/emacs23/site-start.d/50vm-init.el (source)...done
Loading /etc/emacs/site-start.d/50w3m-el.el (source)...done
Loading /etc/emacs/site-start.d/51semi.el (source)...done
Loading /etc/emacs/site-start.d/51speedbar.el (source)...done
Loading /etc/emacs/site-start.d/65wl-beta.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
212 (#o324, #xd4)
Quit [3 times]
92 (#o134, #x5c)
Quit

Load-path shadows:
/usr/share/emacs/23.2/site-lisp/auctex/tex-bar hides /usr/share/emacs/site-lisp/auctex/tex-bar
/usr/share/emacs/23.2/site-lisp/auctex/context-nl hides /usr/share/emacs/site-lisp/auctex/context-nl
/usr/share/emacs/23.2/site-lisp/auctex/tex-buf hides /usr/share/emacs/site-lisp/auctex/tex-buf
/usr/share/emacs/23.2/site-lisp/auctex/texmathp hides /usr/share/emacs/site-lisp/auctex/texmathp
/usr/share/emacs/23.2/site-lisp/auctex/tex hides /usr/share/emacs/site-lisp/auctex/tex
/usr/share/emacs/23.2/site-lisp/auctex/tex-mik hides /usr/share/emacs/site-lisp/auctex/tex-mik
/usr/share/emacs/23.2/site-lisp/auctex/tex-fptex hides /usr/share/emacs/site-lisp/auctex/tex-fptex
/usr/share/emacs/23.2/site-lisp/auctex/bib-cite hides /usr/share/emacs/site-lisp/auctex/bib-cite
/usr/share/emacs/23.2/site-lisp/auctex/tex-info hides /usr/share/emacs/site-lisp/auctex/tex-info
/usr/share/emacs/23.2/site-lisp/auctex/tex-jp hides /usr/share/emacs/site-lisp/auctex/tex-jp
/usr/share/emacs/23.2/site-lisp/auctex/tex-font hides /usr/share/emacs/site-lisp/auctex/tex-font
/usr/share/emacs/23.2/site-lisp/auctex/toolbar-x hides /usr/share/emacs/site-lisp/auctex/toolbar-x
/usr/share/emacs/23.2/site-lisp/auctex/latex hides /usr/share/emacs/site-lisp/auctex/latex
/usr/share/emacs/23.2/site-lisp/auctex/font-latex hides /usr/share/emacs/site-lisp/auctex/font-latex
/usr/share/emacs/23.2/site-lisp/auctex/tex-fold hides /usr/share/emacs/site-lisp/auctex/tex-fold
/usr/share/emacs/23.2/site-lisp/auctex/context hides /usr/share/emacs/site-lisp/auctex/context
/usr/share/emacs/23.2/site-lisp/auctex/context-en hides /usr/share/emacs/site-lisp/auctex/context-en
/usr/share/emacs/23.2/site-lisp/auctex/multi-prompt hides /usr/share/emacs/site-lisp/auctex/multi-prompt
/usr/share/emacs/23.2/site-lisp/auctex/tex-style hides /usr/share/emacs/site-lisp/auctex/tex-style
/usr/local/share/emacs/site-lisp/imaxima-autoconf-variables hides /usr/share/emacs/23.2/site-lisp/maxima/imaxima-autoconf-variables
/usr/local/share/emacs/site-lisp/imaxima hides /usr/share/emacs/23.2/site-lisp/maxima/imaxima
/usr/local/share/emacs/site-lisp/imath hides /usr/share/emacs/23.2/site-lisp/maxima/imath
/usr/share/emacs/site-lisp/autoconf/autotest-mode hides /usr/share/emacs/site-lisp/autotest-mode
/usr/share/emacs/23.2/site-lisp/magit hides /usr/share/emacs/site-lisp/magit
/usr/share/emacs/23.2/site-lisp/cscope/xcscope hides /usr/share/emacs/site-lisp/xcscope
/usr/share/emacs/23.2/site-lisp/cmake-data/cmake-mode hides /usr/share/emacs/site-lisp/cmake-mode
/usr/share/emacs/23.2/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs23/site-lisp/semi/pgg-pgp5 hides /usr/share/emacs/23.2/lisp/pgg-pgp5
/usr/share/emacs23/site-lisp/speedbar/sb-image hides /usr/share/emacs/23.2/lisp/sb-image
/usr/share/emacs23/site-lisp/semi/pgg hides /usr/share/emacs/23.2/lisp/pgg
/usr/share/emacs23/site-lisp/semi/pgg-parse hides /usr/share/emacs/23.2/lisp/pgg-parse
/usr/share/emacs23/site-lisp/cedet-common/ezimage hides /usr/share/emacs/23.2/lisp/ezimage
/usr/share/emacs23/site-lisp/flim/md4 hides /usr/share/emacs/23.2/lisp/md4
/usr/share/emacs23/site-lisp/semi/pgg-gpg hides /usr/share/emacs/23.2/lisp/pgg-gpg
/usr/share/emacs23/site-lisp/speedbar/speedbar hides /usr/share/emacs/23.2/lisp/speedbar
/usr/share/emacs23/site-lisp/flim/sha1 hides /usr/share/emacs/23.2/lisp/sha1
/usr/share/emacs23/site-lisp/semi/pgg-pgp hides /usr/share/emacs/23.2/lisp/pgg-pgp
/usr/share/emacs23/site-lisp/flim/hex-util hides /usr/share/emacs/23.2/lisp/hex-util
/usr/share/emacs23/site-lisp/speedbar/dframe hides /usr/share/emacs/23.2/lisp/dframe
/usr/share/emacs23/site-lisp/semi/pgg-def hides /usr/share/emacs/23.2/lisp/pgg-def
/usr/share/emacs23/site-lisp/dictionaries-common/flyspell hides /usr/share/emacs/23.2/lisp/textmodes/flyspell
/usr/share/emacs23/site-lisp/dictionaries-common/ispell hides /usr/share/emacs/23.2/lisp/textmodes/ispell
/usr/share/emacs/23.2/site-lisp/octave3.1-emacsen/octave-inf hides /usr/share/emacs/23.2/lisp/progmodes/octave-inf
/usr/share/emacs/23.2/site-lisp/octave3.1-emacsen/octave-mod hides /usr/share/emacs/23.2/lisp/progmodes/octave-mod
/usr/share/emacs23/site-lisp/org-mode/org-archive hides /usr/share/emacs/23.2/lisp/org/org-archive
/usr/share/emacs23/site-lisp/org-mode/org-mobile hides /usr/share/emacs/23.2/lisp/org/org-mobile
/usr/share/emacs23/site-lisp/org-mode/org-ascii hides /usr/share/emacs/23.2/lisp/org/org-ascii
/usr/share/emacs23/site-lisp/org-mode/org-clock hides /usr/share/emacs/23.2/lisp/org/org-clock
/usr/share/emacs23/site-lisp/org-mode/org-gnus hides /usr/share/emacs/23.2/lisp/org/org-gnus
/usr/share/emacs23/site-lisp/org-mode/org-indent hides /usr/share/emacs/23.2/lisp/org/org-indent
/usr/share/emacs23/site-lisp/org-mode/org-wl hides /usr/share/emacs/23.2/lisp/org/org-wl
/usr/share/emacs23/site-lisp/org-mode/org-exp-blocks hides /usr/share/emacs/23.2/lisp/org/org-exp-blocks
/usr/share/emacs23/site-lisp/org-mode/org-list hides /usr/share/emacs/23.2/lisp/org/org-list
/usr/share/emacs23/site-lisp/org-mode/org-w3m hides /usr/share/emacs/23.2/lisp/org/org-w3m
/usr/share/emacs23/site-lisp/org-mode/org-colview hides /usr/share/emacs/23.2/lisp/org/org-colview
/usr/share/emacs23/site-lisp/org-mode/org-freemind hides /usr/share/emacs/23.2/lisp/org/org-freemind
/usr/share/emacs23/site-lisp/org-mode/org-macs hides /usr/share/emacs/23.2/lisp/org/org-macs
/usr/share/emacs23/site-lisp/org-mode/org-crypt hides /usr/share/emacs/23.2/lisp/org/org-crypt
/usr/share/emacs23/site-lisp/org-mode/org-mhe hides /usr/share/emacs/23.2/lisp/org/org-mhe
/usr/share/emacs23/site-lisp/org-mode/org-table hides /usr/share/emacs/23.2/lisp/org/org-table
/usr/share/emacs23/site-lisp/org-mode/org hides /usr/share/emacs/23.2/lisp/org/org
/usr/share/emacs23/site-lisp/org-mode/org-bibtex hides /usr/share/emacs/23.2/lisp/org/org-bibtex
/usr/share/emacs23/site-lisp/org-mode/org-src hides /usr/share/emacs/23.2/lisp/org/org-src
/usr/share/emacs23/site-lisp/org-mode/org-compat hides /usr/share/emacs/23.2/lisp/org/org-compat
/usr/share/emacs23/site-lisp/org-mode/org-publish hides /usr/share/emacs/23.2/lisp/org/org-publish
/usr/share/emacs23/site-lisp/org-mode/org-id hides /usr/share/emacs/23.2/lisp/org/org-id
/usr/share/emacs23/site-lisp/org-mode/org-feed hides /usr/share/emacs/23.2/lisp/org/org-feed
/usr/share/emacs23/site-lisp/org-mode/org-habit hides /usr/share/emacs/23.2/lisp/org/org-habit
/usr/share/emacs23/site-lisp/org-mode/org-rmail hides /usr/share/emacs/23.2/lisp/org/org-rmail
/usr/share/emacs23/site-lisp/org-mode/org-plot hides /usr/share/emacs/23.2/lisp/org/org-plot
/usr/share/emacs23/site-lisp/org-mode/org-protocol hides /usr/share/emacs/23.2/lisp/org/org-protocol
/usr/share/emacs23/site-lisp/org-mode/org-faces hides /usr/share/emacs/23.2/lisp/org/org-faces
/usr/share/emacs23/site-lisp/org-mode/org-latex hides /usr/share/emacs/23.2/lisp/org/org-latex
/usr/share/emacs23/site-lisp/org-mode/org-jsinfo hides /usr/share/emacs/23.2/lisp/org/org-jsinfo
/usr/share/emacs23/site-lisp/org-mode/org-xoxo hides /usr/share/emacs/23.2/lisp/org/org-xoxo
/usr/share/emacs23/site-lisp/org-mode/org-vm hides /usr/share/emacs/23.2/lisp/org/org-vm
/usr/share/emacs23/site-lisp/org-mode/org-mouse hides /usr/share/emacs/23.2/lisp/org/org-mouse
/usr/share/emacs23/site-lisp/org-mode/org-timer hides /usr/share/emacs/23.2/lisp/org/org-timer
/usr/share/emacs23/site-lisp/org-mode/org-footnote hides /usr/share/emacs/23.2/lisp/org/org-footnote
/usr/share/emacs23/site-lisp/org-mode/org-html hides /usr/share/emacs/23.2/lisp/org/org-html
/usr/share/emacs23/site-lisp/org-mode/org-inlinetask hides /usr/share/emacs/23.2/lisp/org/org-inlinetask
/usr/share/emacs23/site-lisp/org-mode/org-attach hides /usr/share/emacs/23.2/lisp/org/org-attach
/usr/share/emacs23/site-lisp/org-mode/org-agenda hides /usr/share/emacs/23.2/lisp/org/org-agenda
/usr/share/emacs23/site-lisp/org-mode/org-datetree hides /usr/share/emacs/23.2/lisp/org/org-datetree
/usr/share/emacs23/site-lisp/org-mode/org-bbdb hides /usr/share/emacs/23.2/lisp/org/org-bbdb
/usr/share/emacs23/site-lisp/org-mode/org-irc hides /usr/share/emacs/23.2/lisp/org/org-irc
/usr/share/emacs23/site-lisp/org-mode/org-mew hides /usr/share/emacs/23.2/lisp/org/org-mew
/usr/share/emacs23/site-lisp/org-mode/org-mac-message hides /usr/share/emacs/23.2/lisp/org/org-mac-message
/usr/share/emacs23/site-lisp/org-mode/org-docbook hides /usr/share/emacs/23.2/lisp/org/org-docbook
/usr/share/emacs23/site-lisp/org-mode/org-info hides /usr/share/emacs/23.2/lisp/org/org-info
/usr/share/emacs23/site-lisp/org-mode/org-exp hides /usr/share/emacs/23.2/lisp/org/org-exp
/usr/share/emacs23/site-lisp/org-mode/org-install hides /usr/share/emacs/23.2/lisp/org/org-install
/usr/share/emacs23/site-lisp/org-mode/org-icalendar hides /usr/share/emacs/23.2/lisp/org/org-icalendar
/usr/share/emacs23/site-lisp/org-mode/org-remember hides /usr/share/emacs/23.2/lisp/org/org-remember
/usr/share/emacs23/site-lisp/flim/hmac-def hides /usr/share/emacs/23.2/lisp/net/hmac-def
/usr/share/emacs23/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/23.2/lisp/net/sasl-ntlm
/usr/share/emacs23/site-lisp/flim/sasl hides /usr/share/emacs/23.2/lisp/net/sasl
/usr/share/emacs23/site-lisp/flim/sasl-digest hides /usr/share/emacs/23.2/lisp/net/sasl-digest
/usr/share/emacs23/site-lisp/flim/hmac-md5 hides /usr/share/emacs/23.2/lisp/net/hmac-md5
/usr/share/emacs23/site-lisp/flim/ntlm hides /usr/share/emacs/23.2/lisp/net/ntlm
/usr/share/emacs23/site-lisp/flim/sasl-cram hides /usr/share/emacs/23.2/lisp/net/sasl-cram
/usr/share/emacs23/site-lisp/wl/rfc2368 hides /usr/share/emacs/23.2/lisp/mail/rfc2368
/usr/share/emacs23/site-lisp/cedet-common/cedet-files hides /usr/share/emacs/23.2/lisp/cedet/cedet-files
/usr/share/emacs23/site-lisp/cedet-common/mode-local hides /usr/share/emacs/23.2/lisp/cedet/mode-local
/usr/share/emacs23/site-lisp/cedet-common/inversion hides /usr/share/emacs/23.2/lisp/cedet/inversion
/usr/share/emacs23/site-lisp/cedet-common/cedet hides /usr/share/emacs/23.2/lisp/cedet/cedet

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
easymenu mml-sec password-cache mm-decode mm-bodies mm-encode mailcap
mail-parse rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader
gnus-util netrc time-date mm-util mail-prsvr gmm-utils wid-edit
mailheader canlock sha1 sha1-el hex-util hashcash mail-utils emacsbug
perl-mode w3m-load org-install muse-autoloads emacs-goodies-el
emacs-goodies-custom emacs-goodies-loaddefs easy-mmode cedet-autogen
advice advice-preload autoload help-fns preview-latex tex-site
auto-loads tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win
x-dnd font-setting tool-bar dnd fontset image fringe lisp-mode register
page menu-bar rfn-eshadow timer select scroll-bar mldrag mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind system-font-setting
font-render-setting gtk x-toolkit x multi-tty emacs)

bug#8218: corrected attachment

[Dima Kogan]

[-- Attachment #1: Type: text/plain, Size: 54 bytes --]

I didn't attach the sample file correctly. Here it is.

[-- Attachment #2: tst3.pm --]
[-- Type: application/x-perl, Size: 24 bytes --]

bug#8218: 23.2; scan-sexps sees inconsistent syntax

[Stefan Monnier]

> I am seeing (scan-sexps) show inconsistent behavior depending on WHEN it
> is evaluated. If the attached sample file is opened, the point placed at
> the first '{', and (scan-sexps (point) 1) evaluated, the returned
> position is at times of the first '}' (wrong) and at other times of the
> second '}' (right). If the evaluation is done right when the file is
> opened, I observe the wrong behavior. This is reliably triggered with

> $ emacs -Q /tmp/tst3.pm --eval '(goto-char (scan-sexps 5 1))'

> Any subsequent evaluation of the same form in that buffer produces the
> correct behavior. If I turn off font-lock-mode, I consistently get
> incorrect behavior. This has wide ramifications with occasional
> incorrect indentation and hide-show operation, for instance.

This is not a bug in syntax.c: the syntax of Perl is too complex for the
simple syntax-table mechanism, so it needs to use the escape hatch,
i.e. the `syntax-table' property.  This property is added to particular
parts of the text via font-lock-syntactic-keywords (in Emacs-23) and via
syntax-propertize (in Emacs-24).  So in Emacs-23, you will only get the
right answer if font-lock is enabled and has processed the relevant
parts of the buffer.  In Emacs-24, it can work right even if font-lock
is disabled, but you still need to make sure the relevant parts of the
buffer have been processed.  For that, you should call
(syntax-propertize POS) where POS is the upper bound of the chunk of
text you're interested.


        Stefan