bug#35119: 26.1; narrow-to-region loses word-start/symbol-start information at end

bug#35119: 26.1; narrow-to-region loses word-start/symbol-start information at end

[Sam Halliday]

If the function `narrow-to-region' (as it is in `looking-back') is used
to restrict the region prior to an invocation of re-search-forward or
looking-at, then zero length regexp patterns are lost at the boundaries.

It is unclear if this is a bug or an undocumented feature, but it has
consequences for lexers that must operate backwards (such as those used
by SMIE).


In GNU Emacs 26.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.7)
 of 2019-03-17 built on juergen
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description:	Arch Linux

Recent messages:

Mark set
You did not specify a variable
Type "q" to restore previous buffer. [2 times]
C-x <down> is undefined
funcall-interactively: Beginning of buffer [2 times]
Quit

Mark set
Quit

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 -fno-plt' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
GTK3 X11 MODULES THREADS LIBSYSTEMD LCMS2

Important settings:
  value of $LANG: en_GB.utf8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  auto-insert-mode: t
  global-magit-file-mode: t
  diff-auto-refine-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  whitespace-mode: t
  rainbow-delimiters-mode: t
  smartparens-strict-mode: t
  smartparens-mode: t
  company-mode: t
  yas-minor-mode: t
  rainbow-mode: t
  show-paren-mode: t
  flycheck-mode: t
  global-hl-todo-mode: t
  hl-todo-mode: t
  shell-dirtrack-mode: t
  projectile-mode: t
  flx-ido-mode: t
  ido-everywhere: t
  persistent-scratch-autosave-mode: t
  global-subword-mode: t
  subword-mode: t
  save-place-mode: t
  global-auto-revert-mode: t
  override-global-mode: t
  eldoc-mode: t
  mouse-wheel-mode: t
  prettify-symbols-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/fommil/.emacs.d/elpa/hydra-0.14.0/lv hides /home/fommil/.emacs.d/elpa/lv-0.14.0/lv
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch hides /usr/share/emacs/site-lisp/notmuch
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-wash hides /usr/share/emacs/site-lisp/notmuch-wash
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-tree hides /usr/share/emacs/site-lisp/notmuch-tree
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-tag hides /usr/share/emacs/site-lisp/notmuch-tag
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-show hides /usr/share/emacs/site-lisp/notmuch-show
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-query hides /usr/share/emacs/site-lisp/notmuch-query
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-print hides /usr/share/emacs/site-lisp/notmuch-print
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-parser hides /usr/share/emacs/site-lisp/notmuch-parser
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-mua hides /usr/share/emacs/site-lisp/notmuch-mua
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-message hides /usr/share/emacs/site-lisp/notmuch-message
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-maildir-fcc hides /usr/share/emacs/site-lisp/notmuch-maildir-fcc
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-lib hides /usr/share/emacs/site-lisp/notmuch-lib
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-jump hides /usr/share/emacs/site-lisp/notmuch-jump
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-hello hides /usr/share/emacs/site-lisp/notmuch-hello
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-draft hides /usr/share/emacs/site-lisp/notmuch-draft
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-crypto hides /usr/share/emacs/site-lisp/notmuch-crypto
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-compat hides /usr/share/emacs/site-lisp/notmuch-compat
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-company hides /usr/share/emacs/site-lisp/notmuch-company
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/notmuch-address hides /usr/share/emacs/site-lisp/notmuch-address
/home/fommil/.emacs.d/elpa/notmuch-0.28.3/coolj hides /usr/share/emacs/site-lisp/coolj

Features:
(shadow sort mail-extr emacsbug sendmail completion cl-print debug
hippie-exp tabify help-at-pt finder lisp-mnt jka-compr eieio-opt
speedbar sb-image ezimage dframe help-fns radix-tree pulse git-gutter
haskell-doc inf-haskell haskell-decl-scan haskell haskell-completions
haskell-load haskell-commands highlight-uses-mode haskell-modules
haskell-sandbox haskell-navigate-imports haskell-repl haskell-svg
haskell-collapse hideshow haskell-debug haskell-interactive-mode
haskell-presentation-mode haskell-hoogle haskell-process haskell-session
sql view ag vc-svn find-dired vc-git smartparens-javascript js
smartparens-html sgml-mode dom goto-chg misearch multi-isearch
network-stream starttls url-cache github-review s bug-reference
highlight-symbol magit-extras yatemplate autoinsert magit-submodule
magit-obsolete magit-blame magit-stash magit-bisect magit-push
magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-collab ghub-graphql treepy graphql ghub
url-http tls gnutls url-gw nsm url-auth url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
mailcap let-alist magit-files magit-refs magit-status magit magit-repos
magit-apply magit-wip magit-log which-func imenu magit-diff smerge-mode
diff-mode magit-core magit-autorevert magit-process magit-margin
magit-mode git-commit magit-git magit-section magit-utils magit-popup
crm log-edit message rmc puny rfc822 mml mml-sec epa derived epg
gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev
mail-utils gmm-utils mailheader pcvs-util add-log with-editor
async-bytecomp async server whitespace rainbow-delimiters
smartparens-config smartparens-text smartparens-haskell smartparens
company-oddmuse company-keywords company-etags company-gtags
company-dabbrev-code company-dabbrev company-files company-capf
company-cmake company-xcode company-clang company-semantic company-eclim
company-template company-bbdb company yasnippet rainbow-mode paren
elec-pair intellij-theme darcula-theme ox-gfm ox-md ox-odt rng-loc
rng-uri rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util ox-latex ox-icalendar ox-html table ox-ascii
ox-publish ox org-element avl-tree generator org org-macro org-footnote
org-pcomplete org-list org-faces org-entities org-version ob-emacs-lisp
ob ob-tangle org-src ob-ref ob-lob ob-table ob-keys ob-comint
org-loaddefs cal-menu calendar cal-loaddefs ob-exp ob-core org-compat
ob-eval org-macs graphviz-dot-mode package-utils pcase
fommil-manuscripts fommil-scala cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs fommil-haskell
hlint-refactor haskell-compile haskell-mode haskell-cabal haskell-utils
haskell-font-lock haskell-indentation haskell-string
haskell-sort-imports haskell-lexeme haskell-align-imports haskell-compat
haskell-complete-module haskell-ghc-support noutline outline
flymake-proc flymake warnings etags dabbrev haskell-customize
fommil-email erc-hl-nicks color erc-button erc-fill erc-stamp wid-edit
erc-goodies erc erc-backend erc-compat pp re-builder flycheck-cask
flycheck json map find-func rx subr-x dash cl hydra lv hl-todo
tramp-cache tramp-sh tramp tramp-compat tramp-loaddefs trampver
ucs-normalize shell pcomplete parse-time format-spec projectile grep
compile comint ansi-color ibuf-ext ibuffer ibuffer-loaddefs thingatpt
flx-ido advice flx ido persistent-scratch midnight xref project ring
dired dired-loaddefs diminish cap-words superword subword edmacro kmacro
cl-extra help-mode saveplace autorevert filenotify use-package
use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core finder-inf
ess-generics info package easymenu epg-config url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache url-vars seq byte-opt gv bytecomp byte-compile cconv
cl-loaddefs cl-lib time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 739335 331995)
 (symbols 48 55040 2)
 (miscs 40 697 5902)
 (strings 32 200805 34113)
 (string-bytes 1 5847037)
 (vectors 16 94852)
 (vector-slots 8 2205971 335526)
 (floats 8 388 2086)
 (intervals 56 9116 3117)
 (buffers 992 39))

-- 
Best regards,
Sam

bug#35119: 26.1; narrow-to-region loses word-start/symbol-start information at end

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Wed, 03 Apr 2019 12:19:08 +0100
> 
> If the function `narrow-to-region' (as it is in `looking-back') is used
> to restrict the region prior to an invocation of re-search-forward or
> looking-at, then zero length regexp patterns are lost at the boundaries.

Could you please provide a recipe to reproduce the issue?  I'm not
sure I understand what is the problem you are describing.

Thanks.

bug#35119: 26.1; narrow-to-region loses word-start/symbol-start information at end

[Sam Halliday]

Hi Eli,

Sorry that was a terrible bug report.

This impacts me in `looking-back'. Here's an interactive snippet to
demonstrate the problem (not minimised to`narrow-to-region'):

(defun look-for-35119 ()
  (interactive)
  (if (looking-back
       (rx (: word-end ":" word-start))
       ;;(rx (: word-end ":"))
       (- (point) 1) 't)
      (message "hit")
    (message "miss")))

in emacs-lisp-mode, which defines : as non-word, interactively
evaluate look-for-35119 when the point is just after the colon in this
example text

  wibble:wobble

I would expect to see "hit", but we get "miss". To demonstrate that
the word-start is the cause of the problem, try the commented regexp
and try again, you'll get "hit" but of course this regexp is not what
is intended. For example, it would also match in between :: in the
following:

  wibble::wobble

The cause is that the `narrow-to-region' call inside `looking-back' is
dropping the word-start zero length match at the beginning of wobble.
This may or may not be a bug in narrow-to-region, but I'm quite sure
it's a bug in `looking-back'. There is most likely a similar example
demonstrating that the zero lengths are missing at the start as well
as the end.

I've tried playing around with multiple alternative implementations of
`looking-back' but none are working for me. Probably the best
workaround I can think of is to extend the `narrow-to-region' call by
one more character at the start and the end. Dealing with the start is
easy, we just goto-char limit+1, but dealing with the end is difficult
as we need to put an anychar \\. matcher in the doctored regexp and
then the match-end is off-by-one from what the user expects, so then
we have to doctor that, and then all hell breaks loose.

Does that make sense?


On Wed, 3 Apr 2019 at 12:25, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Sam Halliday <sam.halliday@gmail.com>
> > Date: Wed, 03 Apr 2019 12:19:08 +0100
> >
> > If the function `narrow-to-region' (as it is in `looking-back') is used
> > to restrict the region prior to an invocation of re-search-forward or
> > looking-at, then zero length regexp patterns are lost at the boundaries.
>
> Could you please provide a recipe to reproduce the issue?  I'm not
> sure I understand what is the problem you are describing.
>
> Thanks.

bug#35119: 26.1; narrow-to-region loses word-start/symbol-start information at end

[Sam Halliday]

Hmm, on further investigation I think this may just be regexp behaviour.

I came up with this as an alternative to `looking-back'

(defun my-looking-back (regexp lower)
  (let ((upper (point))
        (start lower))
    (save-excursion
      (catch 'hit
        (while (< start upper)
          (goto-char start)
          (re-search-forward regexp upper 't)
          (when (= (point) upper)
            (throw 'hit 't))
          (setq start (+ 1 start)))
        nil))))

and it also fails to match the : in the example. So perhaps limit is
also excluding the zero-length implied by the subsequent character.

On Wed, 3 Apr 2019 at 13:30, Sam Halliday <sam.halliday@gmail.com> wrote:
>
> Hi Eli,
>
> Sorry that was a terrible bug report.
>
> This impacts me in `looking-back'. Here's an interactive snippet to
> demonstrate the problem (not minimised to`narrow-to-region'):
>
> (defun look-for-35119 ()
>   (interactive)
>   (if (looking-back
>        (rx (: word-end ":" word-start))
>        ;;(rx (: word-end ":"))
>        (- (point) 1) 't)
>       (message "hit")
>     (message "miss")))
>
> in emacs-lisp-mode, which defines : as non-word, interactively
> evaluate look-for-35119 when the point is just after the colon in this
> example text
>
>   wibble:wobble
>
> I would expect to see "hit", but we get "miss". To demonstrate that
> the word-start is the cause of the problem, try the commented regexp
> and try again, you'll get "hit" but of course this regexp is not what
> is intended. For example, it would also match in between :: in the
> following:
>
>   wibble::wobble
>
> The cause is that the `narrow-to-region' call inside `looking-back' is
> dropping the word-start zero length match at the beginning of wobble.
> This may or may not be a bug in narrow-to-region, but I'm quite sure
> it's a bug in `looking-back'. There is most likely a similar example
> demonstrating that the zero lengths are missing at the start as well
> as the end.
>
> I've tried playing around with multiple alternative implementations of
> `looking-back' but none are working for me. Probably the best
> workaround I can think of is to extend the `narrow-to-region' call by
> one more character at the start and the end. Dealing with the start is
> easy, we just goto-char limit+1, but dealing with the end is difficult
> as we need to put an anychar \\. matcher in the doctored regexp and
> then the match-end is off-by-one from what the user expects, so then
> we have to doctor that, and then all hell breaks loose.
>
> Does that make sense?
>
>
> On Wed, 3 Apr 2019 at 12:25, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Sam Halliday <sam.halliday@gmail.com>
> > > Date: Wed, 03 Apr 2019 12:19:08 +0100
> > >
> > > If the function `narrow-to-region' (as it is in `looking-back') is used
> > > to restrict the region prior to an invocation of re-search-forward or
> > > looking-at, then zero length regexp patterns are lost at the boundaries.
> >
> > Could you please provide a recipe to reproduce the issue?  I'm not
> > sure I understand what is the problem you are describing.
> >
> > Thanks.

bug#35119: 26.1; narrow-to-region loses word-start/symbol-start information at end

[Sam Halliday]

To be clear, I still think this is a bug... but I'm now thinking that
the bug is in re-search-forward.

This alternative looking-back works for the example but is broken in other ways

(defun haskell-tng-lexer:greedy-looking-back (regexp lower)
  (let ((upper (+ (point) 1)) ;; must be +1 to include zero-lengths
        (start lower))
    (save-excursion
      (catch 'hit
        (while (< start upper)
          (goto-char start)
          (re-search-forward regexp upper 't)
          (when (= (point) (- upper 1))
            (throw 'hit 't))
          (setq start (+ 1 start)))
        nil))))

On Wed, 3 Apr 2019 at 14:01, Sam Halliday <sam.halliday@gmail.com> wrote:
>
> Hmm, on further investigation I think this may just be regexp behaviour.
>
> I came up with this as an alternative to `looking-back'
>
> (defun my-looking-back (regexp lower)
>   (let ((upper (point))
>         (start lower))
>     (save-excursion
>       (catch 'hit
>         (while (< start upper)
>           (goto-char start)
>           (re-search-forward regexp upper 't)
>           (when (= (point) upper)
>             (throw 'hit 't))
>           (setq start (+ 1 start)))
>         nil))))
>
> and it also fails to match the : in the example. So perhaps limit is
> also excluding the zero-length implied by the subsequent character.
>
> On Wed, 3 Apr 2019 at 13:30, Sam Halliday <sam.halliday@gmail.com> wrote:
> >
> > Hi Eli,
> >
> > Sorry that was a terrible bug report.
> >
> > This impacts me in `looking-back'. Here's an interactive snippet to
> > demonstrate the problem (not minimised to`narrow-to-region'):
> >
> > (defun look-for-35119 ()
> >   (interactive)
> >   (if (looking-back
> >        (rx (: word-end ":" word-start))
> >        ;;(rx (: word-end ":"))
> >        (- (point) 1) 't)
> >       (message "hit")
> >     (message "miss")))
> >
> > in emacs-lisp-mode, which defines : as non-word, interactively
> > evaluate look-for-35119 when the point is just after the colon in this
> > example text
> >
> >   wibble:wobble
> >
> > I would expect to see "hit", but we get "miss". To demonstrate that
> > the word-start is the cause of the problem, try the commented regexp
> > and try again, you'll get "hit" but of course this regexp is not what
> > is intended. For example, it would also match in between :: in the
> > following:
> >
> >   wibble::wobble
> >
> > The cause is that the `narrow-to-region' call inside `looking-back' is
> > dropping the word-start zero length match at the beginning of wobble.
> > This may or may not be a bug in narrow-to-region, but I'm quite sure
> > it's a bug in `looking-back'. There is most likely a similar example
> > demonstrating that the zero lengths are missing at the start as well
> > as the end.
> >
> > I've tried playing around with multiple alternative implementations of
> > `looking-back' but none are working for me. Probably the best
> > workaround I can think of is to extend the `narrow-to-region' call by
> > one more character at the start and the end. Dealing with the start is
> > easy, we just goto-char limit+1, but dealing with the end is difficult
> > as we need to put an anychar \\. matcher in the doctored regexp and
> > then the match-end is off-by-one from what the user expects, so then
> > we have to doctor that, and then all hell breaks loose.
> >
> > Does that make sense?
> >
> >
> > On Wed, 3 Apr 2019 at 12:25, Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > > From: Sam Halliday <sam.halliday@gmail.com>
> > > > Date: Wed, 03 Apr 2019 12:19:08 +0100
> > > >
> > > > If the function `narrow-to-region' (as it is in `looking-back') is used
> > > > to restrict the region prior to an invocation of re-search-forward or
> > > > looking-at, then zero length regexp patterns are lost at the boundaries.
> > >
> > > Could you please provide a recipe to reproduce the issue?  I'm not
> > > sure I understand what is the problem you are describing.
> > >
> > > Thanks.

bug#35119: 26.1; narrow-to-region loses word-start/symbol-start information at end

[Lars Ingebrigtsen]

Sam Halliday <sam.halliday@gmail.com> writes:

> This impacts me in `looking-back'. Here's an interactive snippet to
> demonstrate the problem (not minimised to`narrow-to-region'):
>
> (defun look-for-35119 ()
>   (interactive)
>   (if (looking-back
>        (rx (: word-end ":" word-start))
>        ;;(rx (: word-end ":"))
>        (- (point) 1) 't)
>       (message "hit")
>     (message "miss")))
>
> in emacs-lisp-mode, which defines : as non-word, interactively
> evaluate look-for-35119 when the point is just after the colon in this
> example text
>
>   wibble:wobble

Here's a simpler test case:

(with-temp-buffer
  (emacs-lisp-mode)
  (insert "wibble:wobble")
  (goto-char (point-min))
  (search-forward ":")
  (if (looking-back "\\>:\\<" nil t)
      (message "hit")
    (message "miss")))

And, indeed, that fails because

(re-search-backward "\\(?:\\>:\\<\\)\\=" nil t)

fails.  There seems to be a general problem with re-search-backward and
these zero-length matches (as you said in a later message).

Here's the test case:

(with-temp-buffer
  (emacs-lisp-mode)
  (insert "wibble:wobble")
  (goto-char (point-min))
  (search-forward ":")
  (if (re-search-backward ":\\<" nil t)
      (message "hit")
    (message "miss")))

That is, if we're at the point where this zero-length match should
match, it doesn't.  This succeeds:

(with-temp-buffer
  (emacs-lisp-mode)
  (insert "wibble:wobble")
  (goto-char (point-min))
  (search-forward ":")
  (forward-char 1)
  (if (re-search-backward ":\\<" nil t)
      (message "hit")
    (message "miss")))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no