bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Shigeru Fukaya]

`forward-sexp', or `scan-sexps', doesn't seem to skip bool vectors correctly.

Bool vector's format is like #&N"XXX".
`forward-sexp' doesn't skip the whole bool vector, but stop before
double quotation.

And, maybe, just the same thing occurs for circular objects.
It skip only the label, and stop before the object.

Is the current behavior correct?

Regards,
Shigeru

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Stefan Monnier]

> `forward-sexp', or `scan-sexps', doesn't seem to skip bool vectors correctly.
> Bool vector's format is like #&N"XXX".
> `forward-sexp' doesn't skip the whole bool vector, but stop before
> double quotation.

Indeed.

> And, maybe, just the same thing occurs for circular objects.
> It skip only the label, and stop before the object.

Indeed.

> Is the current behavior correct?

No, it's not.  Patches welcome.


        Stefan

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Shigeru Fukaya]

I tried to fix the issue, and seemingly successful.
But I found more to do newly, possibly.  Those are,

1. #s(...)          hash table
2. ^[...] ^^[...]   char table
3. #@NUMBER         comments
4. $!               beginning of executable file
5. ##               empty symbol

I think doing for 1, 2, 5 are practically sufficient.

Shigeru

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Stefan Monnier]

> I tried to fix the issue, and seemingly successful.
> But I found more to do newly, possibly.  Those are,

> 1. #s(...)          hash table
> 2. ^[...] ^^[...]   char table
> 3. #@NUMBER         comments
> 4. $!               beginning of executable file
> 5. ##               empty symbol

3 should only appear in .elc files and the mode used in .elc files
(emacs-lisp-byte-code-mode) already sets up syntax-propertize-function
to handle them correctly.

I don't know what 4 is.  Or do you mean "#!" as the first two chars of
the file?  We could setup syntax-propertize-function to mark them as
comments, indeed.

> I think doing for 1, 2, 5 are practically sufficient.

Sounds right.


        Stefan

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Shigeru Fukaya]

[-- Attachment #1: Type: text/plain, Size: 2152 bytes --]

>> 2. ^[...] ^^[...]   char table

Sorry, but it's #^[...] #^^[...]


>> 4. $!               beginning of executable file

>I don't know what 4 is.  Or do you mean "#!" as the first two chars of
>the file?  We could setup syntax-propertize-function to mark them as
>comments, indeed.

It is just magic number of unix files.

(read-from-string "#!/usr/local/bin/emacs\n(pwd)") --> ((pwd) . 28)

see read1 in lread.c.


And, my changes to the latest emacs are,

1) add `scan-sexps' an optional argument, `elisp-syntax'.

2) pass `scan_lists' the optional argument.

3) `scan_lists' handles elisp specific handling when the argument is set.

4) `forward-sexp' calls `scan-sexps' with the optional argument non-nil
value when the current buffer's mode is elisp related mode.
Anso, do additional movement for labels of cyclic object, as is for
prefixes.

(Is a change in the last sentence above unnecessary?)


As a result, I seems doing expectedly on such an object as below.

(a  b  #&3"d"  #&99"ZZZ"  #1=a  #2= b  #1# #3= #40= (c) #40# #2# #10= #&10"A"
   #s(dummy 1 2 3)  #^^[dummy 1 2 3]  #^[dummy 1 2 3]  ##  xyz)


Regards,
Shigeru


---------------------
ChangeLog

	Add support for elisp syntax of bool vector, label of cyclic object,
	hash table, char-table and empty symbol.
	* lisp.el (forward-sexp)

	* syntax.c (scan_lists, Fscan_lists, Fscan_sexps)



lisp.el

(defun forward-sexp (&optional arg)
  "Move forward across one balanced expression (sexp).
With ARG, do it that many times.  Negative arg -N means
move backward across N balanced expressions.
This command assumes point is not in a string or comment.
Calls `forward-sexp-function' to do the work, if that is non-nil."
  (interactive "^p")
  (let ((elisp (memq major-mode '(emacs-lisp-mode
				  ;;eshell-mode
				  inferior-emacs-lisp-mode
				  lisp-interaction-mode))))
    (or arg (setq arg 1))
    (if forward-sexp-function
	(funcall forward-sexp-function arg)
      (goto-char (or (scan-sexps (point) arg elisp) (buffer-end arg)))
      (when (< arg 0)
	(when elisp
	  (save-match-data
	    (while (re-search-backward "#[0-9]+=\\s-*\\=" nil t))))
	(backward-prefix-chars)))))


[-- Attachment #2: syntax.patch --]
[-- Type: application/octet-stream, Size: 9577 bytes --]

*** syntax.c.orig	Mon Sep 23 15:37:54 2013
--- syntax.c	Tue Dec  3 07:45:17 2013
***************
*** 179,185 ****
  
  static Lisp_Object skip_chars (bool, Lisp_Object, Lisp_Object, bool);
  static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
! static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
  static void scan_sexps_forward (struct lisp_parse_state *,
                                  ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
                                  bool, Lisp_Object, int);
--- 179,185 ----
  
  static Lisp_Object skip_chars (bool, Lisp_Object, Lisp_Object, bool);
  static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
! static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool, bool);
  static void scan_sexps_forward (struct lisp_parse_state *,
                                  ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
                                  bool, Lisp_Object, int);
***************
*** 2562,2569 ****
    return ASCII_CHAR_P (c) || !multibyte_symbol_p ? SYNTAX (c) : Ssymbol;
  }
  
  static Lisp_Object
! scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
  {
    Lisp_Object val;
    ptrdiff_t stop = count > 0 ? ZV : BEGV;
--- 2562,2582 ----
    return ASCII_CHAR_P (c) || !multibyte_symbol_p ? SYNTAX (c) : Ssymbol;
  }
  
+ /* ELISPFLAG stands for emacs lisp syntax.
+    ELISPFLAG won't be true without SEXPFLAG is true.
+    When ELISPFLAG is true, special scan, over syntax table, is done.
+    They are:;
+     a. #N= -- label of cyclic object
+     b. #&N"S" -- bool vector
+     c. #s(...) -- hash table
+     d. #^[...] -- char-table
+     e. #^^[...] -- sub-char-table
+     f. ## -- empty symbol
+    where N is digits, "S" is string, ... is sexps.
+    Note label of cyclic object is not sexp, therefore skipped. */
+ 
  static Lisp_Object
! scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag, bool elispflag)
  {
    Lisp_Object val;
    ptrdiff_t stop = count > 0 ? ZV : BEGV;
***************
*** 2593,2598 ****
--- 2606,2614 ----
    immediate_quit = 1;
    QUIT;
  
+   /* for insurance. elispflag implies sexpflag below */
+   if (!sexpflag) elispflag = 0;
+ 
    SETUP_SYNTAX_TABLE (from, count);
    while (count > 0)
      {
***************
*** 2633,2638 ****
--- 2649,2756 ----
  	  if (prefix)
  	    continue;
  
+ 	  /* check elisp special syntax starting with '#'.
+ 	     need at least one char. */
+ 	  if (c == '#' && !depth && elispflag && from < stop)
+ 	    {
+ 	      EMACS_INT f = from;
+ 	      ptrdiff_t b = from_byte;
+ 	      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 	      switch (c)
+ 		{
+ 		case '0': case '1': case '2': case '3': case '4':
+ 		case '5': case '6': case '7': case '8': case '9':
+ 		  /* #N= */
+ 		  do
+ 		    {
+ 		      if (f == stop) break;
+ 		      INC_BOTH (f, b);
+ 		      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		    } while (c >= '0' && c <= '9');
+ 		  if (c == '=')
+ 		    {
+ 		      INC_BOTH (f, b);
+ 		      from = f;
+ 		      from_byte = b;
+ 		      continue;	/* skip this label */
+ 		    }
+ 		  break;
+ 
+ 		case '&':
+ 		  /* #&N"X" */
+ 		  if (f + 2 < stop)
+ 		    {
+ 		      INC_BOTH (f, b);
+ 		      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		      if (c >= '0' && c <= '9')
+ 			{
+ 			  do
+ 			    {
+ 			      if (f == stop) break;
+ 			      INC_BOTH (f, b);
+ 			      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 			    } while (c >= '0' && c <= '9');
+ 			  if (c == '"')
+ 			    {
+ 			      code = Sstring;
+ 			      INC_BOTH (f, b);
+ 			      from = f;
+ 			      from_byte = b;
+ 			      /* next in Sstring */
+ 			    }
+ 			}
+ 		    }
+ 		  break;
+ 
+ 		case 's':
+ 		  /* #s(...) */
+ 		  if (f + 1 < stop)
+ 		    {
+ 		      INC_BOTH (f, b);
+ 		      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		      if (c == '(')
+ 			{
+ 			  code = Sopen;
+ 			  INC_BOTH (f, b);
+ 			  from = f;
+ 			  from_byte = b;
+ 			  /* next in Sopen */
+ 			}
+ 		    }
+ 		  break;
+ 
+ 		case '^':
+ 		  /* #^[...] #^^[...]*/
+ 		  if (f + 1 < stop)
+ 		    {
+ 		      INC_BOTH (f, b);
+ 		      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		      if (c == '^' && f + 1 < stop)
+ 			{
+ 			  INC_BOTH (f, b);
+ 			  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 			}
+ 		      if (c == '[')
+ 			{
+ 			  code = Sopen;
+ 			  INC_BOTH (f, b);
+ 			  from = f;
+ 			  from_byte = b;
+ 			  /* next in Sopen */
+ 			}
+ 		    }
+ 		  break;
+ 
+ 		case '#':
+ 		  /* ## */
+ 		  INC_BOTH (f, b);
+ 		  from = f;
+ 		  from_byte = b;
+ 		  if (depth) continue;
+ 		  goto done;
+ 		}
+ 	    }
+ 
  	  switch (code)
  	    {
  	    case Sescape:
***************
*** 2805,2810 ****
--- 2923,2965 ----
  	    }
  	  else if (SYNTAX_FLAGS_PREFIX (syntax))
  	    continue;
+ 	  else if (!depth && elispflag && from > stop)
+ 	    {
+ 	      EMACS_INT f = from;
+ 	      ptrdiff_t b = from_byte;
+ 	      if (c == '=')
+ 		{
+ 		  /* #N= */
+ 		  DEC_BOTH (f, b);
+ 		  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		  if (c >= '0' && c <= '9')
+ 		    {
+ 		      do
+ 			{
+ 			  if (f == stop) break;
+ 			  DEC_BOTH (f, b);
+ 			  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 			} while (c >= '0' && c <= '9');
+ 		      if (c == '#')
+ 			{
+ 			  from = f;
+ 			  from_byte = b;
+ 			  continue; /* skip this label */
+ 			}
+ 		    }
+ 		}
+ 	      else if (c == '#')
+ 		{
+ 		  /* ## */
+ 		  DEC_BOTH (f, b);
+ 		  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		  if (c == '#') {
+ 		    from = f;
+ 		    from_byte = b;
+ 		    goto done2;
+ 		  }
+ 		}
+ 	    }
  
  	  switch (code)
  	    {
***************
*** 2864,2869 ****
--- 3019,3069 ----
  	      break;
  
  	    case Sopen:
+ 	      /* scan back #s(, #^[, #^^[ as if prefixes.
+ 		 need at least 2 chars. */
+ 	      if (depth == 1 && elispflag && from + 1 > stop)
+ 		{
+ 		  EMACS_INT f = from;
+ 		  ptrdiff_t b = from_byte;
+ 		  if (c == '(')
+ 		    {
+ 		      /* #s(...) */
+ 		      DEC_BOTH (f, b);
+ 		      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		      if (c == 's')
+ 			{
+ 			  DEC_BOTH (f, b);
+ 			  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 			  if (c == '#')
+ 			    {
+ 			      from = f;
+ 			      from_byte = b;
+ 			    }
+ 			}
+ 		    }
+ 		  else if (c == '[')
+ 		    {
+ 		      /* #^[...] or #^^[...] */
+ 		      DEC_BOTH (f, b);
+ 		      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		      if (c == '^')
+ 			{
+ 			  DEC_BOTH (f, b);
+ 			  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 			  if (c == '^' && f > stop)
+ 			    {
+ 			      /* #^^[...] */
+ 			      DEC_BOTH (f, b);
+ 			      c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 			    }
+ 			  if (c == '#')
+ 			    {
+ 			      from = f;
+ 			      from_byte = b;
+ 			    }
+ 			}
+ 		    }
+ 		}
  	    open2:
  	      if (!--depth) goto done2;
  	      if (depth < min_depth)
***************
*** 2922,2927 ****
--- 3122,3156 ----
  			break;
  		    }
  		}
+ 
+ 	      /* #&N"STRING" */
+ 	      if (!depth && elispflag && from + 2 > stop)
+ 		{
+ 		  EMACS_INT f = from;
+ 		  ptrdiff_t b = from_byte;
+ 		  DEC_BOTH (f, b);
+ 		  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 		  if (c >= '0' && c <= '9')
+ 		    {
+ 		      do
+ 			{
+ 			  if (f == stop) break;
+ 			  DEC_BOTH (f, b);
+ 			  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 			} while (c >= '0' && c <= '9');
+ 		      if (c == '&' && f > stop)
+ 			{
+ 			  DEC_BOTH (f, b);
+ 			  c = FETCH_CHAR_AS_MULTIBYTE (b);
+ 			  if (c == '#')
+ 			    {
+ 			      from = f;
+ 			      from_byte = b;
+ 			    }
+ 			}
+ 		    }
+ 		}
+ 
  	      if (!depth && sexpflag) goto done2;
  	      break;
  	    default:
***************
*** 2977,2988 ****
    CHECK_NUMBER (count);
    CHECK_NUMBER (depth);
  
!   return scan_lists (XINT (from), XINT (count), XINT (depth), 0);
  }
  
! DEFUN ("scan-sexps", Fscan_sexps, Sscan_sexps, 2, 2, 0,
         doc: /* Scan from character number FROM by COUNT balanced expressions.
  If COUNT is negative, scan backwards.
  Returns the character number of the position thus found.
  
  Comments are ignored if `parse-sexp-ignore-comments' is non-nil.
--- 3206,3219 ----
    CHECK_NUMBER (count);
    CHECK_NUMBER (depth);
  
!   return scan_lists (XINT (from), XINT (count), XINT (depth), 0, 0);
  }
  
! DEFUN ("scan-sexps", Fscan_sexps, Sscan_sexps, 2, 3, 0,
         doc: /* Scan from character number FROM by COUNT balanced expressions.
  If COUNT is negative, scan backwards.
+ If optional ELISP-SYNTAX is non-nil, handle elisp specific syntax as bool vector,
+ label of cyclic object, hash table, char-table, empty symbol.
  Returns the character number of the position thus found.
  
  Comments are ignored if `parse-sexp-ignore-comments' is non-nil.
***************
*** 2991,3002 ****
  in the middle of a parenthetical grouping, an error is signaled.
  If the beginning or end is reached between groupings
  but before count is used up, nil is returned.  */)
!   (Lisp_Object from, Lisp_Object count)
  {
    CHECK_NUMBER (from);
    CHECK_NUMBER (count);
  
!   return scan_lists (XINT (from), XINT (count), 0, 1);
  }
  
  DEFUN ("backward-prefix-chars", Fbackward_prefix_chars, Sbackward_prefix_chars,
--- 3222,3233 ----
  in the middle of a parenthetical grouping, an error is signaled.
  If the beginning or end is reached between groupings
  but before count is used up, nil is returned.  */)
!   (Lisp_Object from, Lisp_Object count, Lisp_Object elisp_syntax)
  {
    CHECK_NUMBER (from);
    CHECK_NUMBER (count);
  
!   return scan_lists (XINT (from), XINT (count), 0, 1, !NILP (elisp_syntax));
  }
  
  DEFUN ("backward-prefix-chars", Fbackward_prefix_chars, Sbackward_prefix_chars,

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Stefan Monnier]

> 1) add `scan-sexps' an optional argument, `elisp-syntax'.

I don't think I want to add such elisp-specific hacks to the
scan-sexps code.  Better would be to change emacs-lisp-mode to set
forward-sexp-function to a function that handles those cases.


        Stefan

bug#30132: 27.0.50; scan-sexps and ##

[Michael Heerdegen]

Hello,

Insert into *scratch*

  (list ##)

and put point at the beginning of "##".  Then eval M-:

  (goto-char (scan-sexps (point) 1))

and you get

| Debugger entered--Lisp error: (scan-error "Containing expression ends prematurely" 154 155)
|   scan-sexps(152 1)
|   (goto-char (scan-sexps (point) 1))
|   eval((goto-char (scan-sexps (point) 1)) t)
|   eval-expression((goto-char (scan-sexps (point) 1)) nil nil 127)
|   funcall-interactively(eval-expression (goto-char (scan-sexps (point) 1)) nil nil 127)
|   call-interactively(eval-expression nil nil)
|   command-execute(eval-expression)

Expected is that the scan doesn't fail.

## is the rarely used representation of an interned symbol named with
the empty string.  My problem is that where it is used (like in
"org-list.el"), el-search.el gets into trouble.


Thanks in advance,

Michael.


In GNU Emacs 27.0.50 (build 17, x86_64-pc-linux-gnu, GTK+ Version 3.22.26)
 of 2018-01-15 built on drachen
Repository revision: 55a83102629d597eb172e91698578930ff193c5d
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
System Description: Debian GNU/Linux testing (buster)

bug#30132: 27.0.50; scan-sexps and ##

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Hello,
>
> Insert into *scratch*
>
>   (list ##)
>
> and put point at the beginning of "##".

BTW, this is not the only kind of situation where `scan-sexps' can error
for expressions containing ##.  In particular, point doesn't necessarily
need to be directly before "##" for the problem to occur.


Michael.

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> 1) add `scan-sexps' an optional argument, `elisp-syntax'.
>
> I don't think I want to add such elisp-specific hacks to the
> scan-sexps code.  Better would be to change emacs-lisp-mode to set
> forward-sexp-function to a function that handles those cases.

But do we want to implement a brand new forward-sexp just for Emacs
Lisp?  I'm assuming that a Lisp-based implementation would be
uncomfortably slow...

To recap, the following doesn't forward-sexp correctly:

1. #s(...)          hash table
2. ^[...] ^^[...]   char table
5. ##               empty symbol
9. #&N"XXX"         bool vector

And especially 1) is rather annoying...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Stefan Monnier]

> But do we want to implement a brand new forward-sexp just for Emacs Lisp?

No, but I think syntax-propertize can take care of those things.

> I'm assuming that a Lisp-based implementation would be
> uncomfortably slow...
>
> To recap, the following doesn't forward-sexp correctly:
>
> 1. #s(...)          hash table
> 2. ^[...] ^^[...]   char table
> 5. ##               empty symbol
> 9. #&N"XXX"         bool vector

[ I believe for (2) there's a missing # in there somewhere, right?  ]

> And especially 1) is rather annoying...

I do wonder when/where these things show up, tho.
I'd expect those to appear in .elc files and in the output of M-: and
IELM buffers, but not in emacs-lisp-mode buffers.


        Stefan

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I do wonder when/where these things show up, tho.
> I'd expect those to appear in .elc files and in the output of M-: and
> IELM buffers, but not in emacs-lisp-mode buffers.

That's true.  Those things mostly show up when discussing Emacs Lisp and
not when writing Emacs Lisp -- so they're really more in text buffers
than in Lisp buffers...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 268 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But do we want to implement a brand new forward-sexp just for Emacs Lisp?
>
> No, but I think syntax-propertize can take care of those things.

The patch below seems to work, no new sexp movement commands needed?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 1977 bytes --]

From 463f98c55ff7d6bde576a9538d3ae3a33a576cf5 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Sat, 6 Jul 2019 21:55:03 -0400
Subject: [PATCH] Handle elisp #-syntax better (Bug#15998)

* elisp-mode.el (elisp-mode-syntax-propertize): New function.
(emacs-lisp-mode): Set it as syntax-propertize-function.
---
 lisp/progmodes/elisp-mode.el | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/lisp/progmodes/elisp-mode.el b/lisp/progmodes/elisp-mode.el
index cb1b17b447..c86277a309 100644
--- a/lisp/progmodes/elisp-mode.el
+++ b/lisp/progmodes/elisp-mode.el
@@ -213,6 +213,20 @@ emacs-lisp-macroexpand
       (if (bolp) (delete-char -1))
       (indent-region start (point)))))
 
+(defun elisp-mode-syntax-propertize (start end)
+  (goto-char start)
+  (funcall
+   (syntax-propertize-rules
+    ("##" (0 (unless (nth 8 (syntax-ppss)) ; Empty symbol.
+               (string-to-syntax "_"))))
+    ((rx "#" (or (seq (group-n 1 (+ digit) "="))     ; Object label.
+                 (seq (group-n 1 "&" (+ digit)) ?\") ; Bool-vector.
+                 (seq (group-n 1 "s") "(")           ; Record.
+                 (seq (group-n 1 (+ "^")) "[")))     ; Char-table.
+     (1 (unless (save-excursion (nth 8 (syntax-ppss (match-beginning 0))))
+          (string-to-syntax "'")))))
+   start end))
+
 (defcustom emacs-lisp-mode-hook nil
   "Hook run when entering Emacs Lisp mode."
   :options '(eldoc-mode imenu-add-menubar-index checkdoc-minor-mode)
@@ -242,6 +256,7 @@ emacs-lisp-mode
                 #'elisp-eldoc-documentation-function)
   (add-hook 'xref-backend-functions #'elisp--xref-backend nil t)
   (setq-local project-vc-external-roots-function #'elisp-load-path-roots)
+  (setq-local syntax-propertize-function #'elisp-mode-syntax-propertize)
   (add-hook 'completion-at-point-functions
             #'elisp-completion-at-point nil 'local)
   (add-hook 'flymake-diagnostic-functions #'elisp-flymake-checkdoc nil t)
-- 
2.11.0

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Stefan Monnier]

>> No, but I think syntax-propertize can take care of those things.
> The patch below seems to work, no new sexp movement commands needed?

It should work indeed, tho I'm not sure it'll fix the original problem,
because I don't know *where* the original problem happened and this
patch only fixes the case where we do such navigation in
emacs-lisp-mode, but things #s(...) and #&"..." are *very* rare in
those buffers.

So I'm not completely sure the cost imposed by this patch is worth
the benefit.  Maybe we could start by adding those rules to the
elisp-byte-code-mode?


        Stefan

bug#15998: 24.3; forward-sexp (scan-sexps) doesn't do well with some SEXPs

[Noam Postavsky]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> No, but I think syntax-propertize can take care of those things.
>> The patch below seems to work, no new sexp movement commands needed?
>
> It should work indeed, tho I'm not sure it'll fix the original problem,
> because I don't know *where* the original problem happened and this
> patch only fixes the case where we do such navigation in
> emacs-lisp-mode, but things #s(...) and #&"..." are *very* rare in
> those buffers.

I tend to end up with those things in my *scratch* buffer, after
evaluating various expressions (more so the #s(...) than #&"...", I
don't have much call for messing with bool vectors).  Michael bumped
into an instance of ## in org-list.el (see merged Bug#30132), although
that seems like a mistake in org.

> So I'm not completely sure the cost imposed by this patch is worth
> the benefit.

On the benefit side, it would allow removing the hack I put in for
indentation of #s(...) (that was for IELM, see Bug#31984).  On the other
hand, this seems to be breaking some edebug tests, so that needs more
investigation before it could be installed.

bug#30132: 27.0.50; scan-sexps and ##

[Lars Ingebrigtsen]

Noam Postavsky <npostavs@gmail.com> writes:

> * elisp-mode.el (elisp-mode-syntax-propertize): New function.
> (emacs-lisp-mode): Set it as syntax-propertize-function.

I've tried the patch, and it seems to work well.  However, it makes
edebug-tests-dotted-forms hang completely, and breaks other edebug
tests.

I've poked at this briefly, but I've been unable to see what makes
things hang here.  It would be nice to get this to work -- did you get
any further in debugging these test problems, Noam?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#15998: bug#30132: 27.0.50; scan-sexps and ##

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I've poked at this briefly, but I've been unable to see what makes
> things hang here.  It would be nice to get this to work -- did you get
> any further in debugging these test problems, Noam?

The offending bit was this:

(seq (group-n 1 (+ digit) "="))     ; Object label.

Which is something that should never appear in Lisp files anyway.  The
rest seem unproblematic (except possible performance problems), but
after testing it for a few minutes, I don't really see anything
noticeable.

So I'm pushing Noam's patch to Emacs 29 (with that object label thing
removed).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#30132: 27.0.50; scan-sexps and ##

[Michael Heerdegen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
> > I've poked at this briefly, but I've been unable to see what makes
> > things hang here.  It would be nice to get this to work -- did you get
> > any further in debugging these test problems, Noam?
>
> The offending bit was this:
>
> (seq (group-n 1 (+ digit) "="))     ; Object label.

I wonder why.

> Which is something that should never appear in Lisp files anyway.
> [...]
> So I'm pushing Noam's patch to Emacs 29 (with that object label thing
> removed).

Thanks.  My original use case seems to be fixed.

Michael.

bug#15998: bug#30132: 27.0.50; scan-sexps and ##

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> > The offending bit was this:
> >
> > (seq (group-n 1 (+ digit) "="))     ; Object label.
>
> I wonder why.

Dunno how the Edebug tests are supposed to run...:  When I load
edebug-tests.el and run those tests with M-x ert-run-tests-interactively
t RET, I always get lots of fails - also with what is installed now (and
even without the patch).

Michael.

bug#30132: 27.0.50; scan-sexps and ##

[Lars Ingebrigtsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Dunno how the Edebug tests are supposed to run...:  When I load
> edebug-tests.el and run those tests with M-x ert-run-tests-interactively
> t RET, I always get lots of fails - also with what is installed now (and
> even without the patch).

I run them with "make check" or "make edebug-tests".  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no