When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

Hi

I'm trying to modify the behaviour of erlang.el erlang-mode wrt. the
handling of
binaries in order to get forward-sexp and friends to handle << >> pair
matching.

I have been trying to add something like the following in the code and also
trying to put embedded message calls or even error calls to be able to see
when the function gets called.  So far I did not see it called.

``` lisp
 (defconst erlang-mode-syntax-propertize-function
    (syntax-propertize-rules
     ("\\(<\\)<" (1 "(>"))
     (">\\(>\\)" (2 ")<")))
    "Syntax properties to activate << >> pairing.")
  (setq-local parse-sexp-lookup-properties t)
  (setq-local syntax-propertize-function
              erlang-mode-syntax-propertize-function)
```
I most probably don't understand the mechanism.
I expanded the macro above to see the generated code
which makes sense, and read all I could find on Emacs syntax properties,
text properties, stickiness, etc..

What I'd like to understand is what calls the generated function, providing
it
the 2 arguments (start & end) and when should it be called.

Are these calls protected against error so I can't see them if they occur?

Thanks

/Pierre



-- 
/Pierre

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Pierre Rouleau wrote:

> I'm trying to modify the behaviour of erlang.el

Oh! Erlang, the 1986 functional programming language from
Ericsson for concurrent/distributed real-time
high-availability, or massively scalable soft real-time
systems ... *buzz*

> erlang-mode wrt. the handling of binaries in order to get
> forward-sexp and friends to handle << >> pair matching.

Maybe start with something simpler and see if you get that to
work, and then advance from there ...

E.g. from markdown-mode.el (GNU Emacs 29.0.50)

(defun markdown-syntax-propertize-blockquotes (start end)
  "Match blockquotes from START to END."
  (save-excursion
    (goto-char start)
    (while (and (re-search-forward markdown-regex-blockquote end t)
                (not (markdown-code-block-at-pos (match-beginning 0))))
      (put-text-property (match-beginning 0) (match-end 0)
                         'markdown-blockquote
                         (match-data t)))))

-- 
underground experts united
https://dataswamp.org/~incal

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

On Tue, Oct 5, 2021 at 12:16 AM Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> Pierre Rouleau wrote:
>
> > I'm trying to modify the behaviour of erlang.el
>
> Maybe start with something simpler and see if you get that to
> work, and then advance from there ...
>
> I have.  But actually I'd like to find out when the
syntax-propertize-function gets called.  Is this described somewhere?
-- 
/Pierre

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Stefan Monnier via Users list for the GNU Emacs text editor]

> What I'd like to understand is what calls the generated function, providing
> it the 2 arguments (start & end) and when should it be called.

It's called from various places in various cases.
Most importantly/directly it's called by `syntax-ppss` as well as
"internally" from the C-level syntax scanning code (e.g. the code
implementing `parse-partial-sexp`, `forward-sexp`, ...).

In shouldn't need/want to know the details, normally.

> Are these calls protected against error so I can't see them if they occur?

No, although some of them can occur in contexts where the error might be
silenced or at least not displayed in a prominent way.

Why do you ask?
Are you having an actual problem, or are you asking out of curiosity?


        Stefan

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

On Tue, Oct 5, 2021 at 8:56 AM Stefan Monnier via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> > What I'd like to understand is what calls the generated function,
> providing
> > it the 2 arguments (start & end) and when should it be called.
>
> > Are these calls protected against error so I can't see them if they
> occur?
>
> No, although some of them can occur in contexts where the error might be
> silenced or at least not displayed in a prominent way.
>
> Why do you ask?
> Are you having an actual problem, or are you asking out of curiosity?
>
>
Both actually.    I am trying to add binary pair matching in erlang-mode,
something that works only just after the << >> pair has just been typed
by the user, but later, the matching stops working as the syntax-table
attribute seems to disappear from the character.

I'm trying to make the syntax properties to 'stick' so I can use
forward-sexp, backward-sexp to navigate over Erlang code
code that uses these.  Like the following Erlang code:

Values = << <<(X+1)/integer>> || <<X>> <= <<3,7,5,4,7>> >>.

I tried to add the following after the syntax table is built by the
erlang mode function:

  (defconst erlang-mode-syntax-propertize-function
    (syntax-propertize-rules
     ("\\(<\\)<" (1 "(>"))
     (">\\(>\\)" (2 ")<")))
    "Syntax properties to activate << >> pairing.")
  (setq-local parse-sexp-lookup-properties t)
  (setq-local syntax-propertize-function
              erlang-mode-syntax-propertize-function)

But that does not seem to work.  So I am trying to understand why
it does not work.

I saw the caution note at the end of the syntax-propertize-function in
section 34.5 Syntax Properties that states:
"Caution: When this variable is non-nil, Emacs removes syntax-table
text properties arbitrarily and relies on syntax-propertize-function to
reapply them. Thus if this facility is used at all, the function must
apply all syntax-table text properties used by the major mode."

Since the erlang-mode modifies the text attribute of the << >> characters
when the user types them I was trying to see what I'm either not doing or
doing wrong.

So I wanted to see where the code was used because it
seems that my addition is just ignored or something in erlang.el prevents
it from working and/or my 'nirvana level' understanding of that mechanism is
obviously not high enough :-)

-- 
/Pierre

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Stefan Monnier]

>   (defconst erlang-mode-syntax-propertize-function
>     (syntax-propertize-rules
>      ("\\(<\\)<" (1 "(>"))
>      (">\\(>\\)" (2 ")<")))
>     "Syntax properties to activate << >> pairing.")
>   (setq-local parse-sexp-lookup-properties t)
>   (setq-local syntax-propertize-function
>               erlang-mode-syntax-propertize-function)

`setq-local` at the toplevel makes no sense.
Is it really the code you're using, or are you paraphrasing?


        Stefan

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

> `setq-local` at the toplevel makes no sense.
> Is it really the code you're using, or are you paraphrasing?
>
> Sorry. I did not provide enough context.

That code is not at top-level; it is being called by the erlang-mode
function.
At the moment I placed it right inside my local copy of the erlang.el
function
`erlang-electric-init' which gets called by what is defined by a
(define-derived-mode erlang-mode) form.

I won't leave that code there once I understand the behaviour.
I'm just modifying my local copy of erlang.el to
1) understand that code better
2) eliminate as many external sources of disruption while I investigate.

BTW, I'm trying to get it working on Emacs >= 26.1 if this matters.

-- 
/Pierre

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Pierre Rouleau wrote:

>> `setq-local` at the toplevel makes no sense. Is it really
>> the code you're using, or are you paraphrasing?
>>
>> Sorry. I did not provide enough context.
>
> That code is not at top-level; it is being called by the
> erlang-mode function. At the moment I placed it right inside
> my local copy of the erlang.el function
> `erlang-electric-init' which gets called by what is defined
> by a (define-derived-mode erlang-mode) form.
>
> I won't leave that code there once I understand the behaviour.
> I'm just modifying my local copy of erlang.el to
> 1) understand that code better
> 2) eliminate as many external sources of disruption while
> I investigate.
>
> BTW, I'm trying to get it working on Emacs >= 26.1 if
> this matters.

Well, it never hurts to get the latest Emacs! I'm on GNU Emacs
29.0.50 so that's quite a difference ... (Here are some
commands that may help you, I made them work on Debian 11 at
least but should be a no-op, almost:
https://dataswamp.org/~incal/conf/.zsh/install-emacs ) It is,
or they have made it I should say, _very easy_ to get/compile
the latest version!

As for your problem, relax, you/we will solve it ... j'ai
confiance
<https://www.youtube.com/watch?v=J43T8rEOg-I>

But ... I don't understand 100% what it is you'd like to
happen? Can you describe it in human terms rather? "When I do
... I'd like this to happen ... and not this ..." ?

-- 
underground experts united
https://dataswamp.org/~incal

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

Well, it never hurts to get the latest Emacs! I'm on GNU Emacs
> 29.0.50 so that's quite a difference ...
>

I want to make it work on all these platforms and 26.1 is a necessary target
imposed by non-technical constraints out of my control.


> But ... I don't understand 100% what it is you'd like to
> happen? Can you describe it in human terms rather? "When I do
> ... I'd like this to happen ... and not this ..." ?
>

No problem.
I know the following is long, but I think I need to give some context...

I’m trying to add navigation with forward-sexp and backward-sexp
inside erlang-mode buffers to support the Erlang Bit Syntax expressions
(
https://erlang.org/doc/reference_manual/expressions.html#bit-syntax-expressions
).

Currently, with the latest version of erlang.el, the erlang-mode activates
the ability to move to the >> matching the << just after point, but
*only* after you just write the Erlang code.

So let's say you just wrote the following Erlang statement inside an
erlang-mode buffer:

Abc = << <<Bin>> || Bin <- [<<3,7,5,4,7>>] >>

The statement is not complete, the full-stop has not yet been typed.
At this point you can use the backward-sexp command to move the point just
before the first < character on the line.  And you can also use
forward-sexp
to come back where you were.

Let's say you just type the full-stop (period) to terminate the statement
and then hit RET to move point to the next line.  From then on, if you
place
the point just before the first < or after the last >, forward-sexp and
backward-sexp do not work: they do not move the point to the matching
location.
If you save and re-open the file you also lose the matching ability.

My understanding of the reason why it stops working is because the
erlang.el syntax
table, modified by erlang.el `erlang-ensure-syntax-table-is-initialized'
does not identify
the < and > characters as parens that match another one.  The erlang.el
code does this:

(modify-syntax-entry ?< "." table)
(modify-syntax-entry ?> "." table)

It assigns punctuation syntax to these characters instead of doing the
following:

(modify-syntax-entry ?< "(>" table)
(modify-syntax-entry ?> ")<" table)

The reason the last 2 statements are *not* used is because, > and < are
also used, like
many other programming languages as comparison operators and in erlang you
also
have '->', '<-' and '<=' operators.  Giving open and closing delimiter
syntax to the < and >
characters would break operations and cause a lot of unbalanced issues.

So, I am trying to syntax propertize << and >> to provide ability to
navigate across the
bit syntax expression by setting the syntax-propertize-function to activate
syntax
properties to the outer characters of << and >>.

To do that I added the following at top level, just before the
(defun erlang-ensure-syntax-table-is-initialized  ... ) form:

 (defconst erlang-mode-syntax-propertize-function
    (syntax-propertize-rules
     ("\\(<\\)<" (1 "(>"))
     (">\\(>\\)" (2 ")<")))
    "Syntax properties to activate << >> pairing.")

And I added the following inside of
erlang-ensure-syntax-table-is-initialized
after the code that sets up the syntax table for Erlang :

  (setq-local parse-sexp-lookup-properties t)
  (setq-local syntax-propertize-function
              erlang-mode-syntax-propertize-function)

After byte compilation and testing I was expecting that I'd see the
syntax-table text
property of all outer characters of << and >> bit syntax inside the erlang
buffer to be
activated allowing me to navigate across their edges with forward-sexp and
backward-sexp.

But that does not work: I still cannot navigate with forward-sexp and
backward-sexp
and the outer << and >> characters do not have the text properties I
thought they would have.

I check inside the buffer if parse-sexp-lookup-properies is t and it is.
I also check in the buffer if syntax-propertize-function is set and it is
set to what I expect.
Yet I do not see a change of behaviour from what the non-modified erlang.el
code provides.

Therefore my question as to *when* the syntax-propertize-function should
take effect
and how I can see if it is applied.

-- 
/Pierre

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Pierre Rouleau wrote:

> I know the following is long, but I think I need to give some context...
>
> I’m trying to add navigation with forward-sexp and
> backward-sexp inside erlang-mode buffers to support the
> Erlang Bit Syntax expressions (
> https://erlang.org/doc/reference_manual/expressions.html#bit-syntax-expressions
> ).
>
> Currently, with the latest version of erlang.el, the
> erlang-mode activates the ability to move to the >> matching
> the << just after point, but *only* after you just write the
> Erlang code.
>
> So let's say you just wrote the following Erlang statement
> inside an erlang-mode buffer:
>
> Abc = << <<Bin>> || Bin <- [<<3,7,5,4,7>>] >> [...]

OK, I get it!

Did you read this

  Only single-character comment start and end sequences are represented thus.
  Two-character sequences are represented as described below.
  The second character of NEWENTRY is the matching parenthesis,
  used only if the first character is ‘(’ or ‘)’.
  Any additional characters are flags.
  Defined flags are the characters 1, 2, 3, 4, b, p, and n.
  1 means CHAR is the start of a two-char comment start sequence.
  2 means CHAR is the second character of such a sequence.
  3 means CHAR is the start of a two-char comment end sequence.
  4 means CHAR is the second character of such a sequence.

in `modify-syntax-entry'?

Weird/bad this isn't already in the mode BTW ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Stefan Monnier]

>> `setq-local` at the toplevel makes no sense.
>> Is it really the code you're using, or are you paraphrasing?
> Sorry. I did not provide enough context.
> That code is not at top-level; it is being called by the erlang-mode
> function.

OK, thanks, that makes sense (I just wanted to make sure the problem
wasn't some silly misunderstanding).

> I won't leave that code there once I understand the behaviour.
> I'm just modifying my local copy of erlang.el to
> 1) understand that code better
> 2) eliminate as many external sources of disruption while I investigate.

The code you show seems fine to me except for the `2` which should be
a `1` (there is only one subgroup in your second regexp).

To debug these things, you can try:

- `C-u C-x =` on the char(s) where you expect a syntax-table text property,
  to check whether it's present or not.
- Call `syntax-propertize` by hand in case of doubt.
- Trace your rules function.  E.g. use

      (defun my-syntax-propertize (beg end)
        (funcall
         (syntax-propertize-rules
           ("\\(<\\)<" (1 "(>"))
           (">\\(>\\)" (1 ")<")))
         beg end))

      [...]

      (setq-local syntax-propertize-function
                  #'my-syntax-propertize)

  and then `M-x trace-function RET my-syntax-propertize RET`.


        Stefan

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

> Did you read this
>
>   Only single-character comment start and end sequences are represented
> thus.
>   Two-character sequences are represented as described below.
>   The second character of NEWENTRY is the matching parenthesis,
>   used only if the first character is ‘(’ or ‘)’.
>

in `modify-syntax-entry'?
>
> I did.  That's why I wrote that I thought of using the following:

(modify-syntax-entry ?< "(>" table)
(modify-syntax-entry ?> ")<" table)

But I can't because that would cause unbalancing in code where the > and <
are used by themselves, not in << or >> pairs.


> Weird/bad this isn't already in the mode BTW ...
>
> I agree.  That's why I'm trying to improve it.

-- 
/Pierre

Fwd: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

(forgot to copy the list)

On Tue, Oct 5, 2021 at 3:42 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> OK, thanks, that makes sense (I just wanted to make sure the problem
> wasn't some silly misunderstanding).
>
No problem.

>
> The code you show seems fine to me except for the `2` which should be
> a `1` (there is only one subgroup in your second regexp).
>
> Ahh, a silly typo, when I copied from the wrong place.  I have the right
'1' in place in
erlang.el but it still does not behave as I would have expected.


> To debug these things, you can try:
>
> - `C-u C-x =` on the char(s) where you expect a syntax-table text property,
>   to check whether it's present or not.
>
This is had done to check the syntax and wrote something to list all text
properties
of the character at point because what-cursor-position does not show them
all.


> - Call `syntax-propertize` by hand in case of doubt.
> - Trace your rules function.  E.g. use
>
>       (defun my-syntax-propertize (beg end)
>         (funcall
>          (syntax-propertize-rules
>            ("\\(<\\)<" (1 "(>"))
>            (">\\(>\\)" (1 ")<")))
>          beg end))
>
>       [...]
>
>       (setq-local syntax-propertize-function
>                   #'my-syntax-propertize)


>   and then `M-x trace-function RET my-syntax-propertize RET`.
>
> Thanks! That's what I was looking for!  I'll read more on that topic.

Now I did experimentation with that and got the following trace
**when I complete a closing >> only**:

1 -> (my-syntax-propertize 980 1555)
1 <- my-syntax-propertize: nil
======================================================================
1 -> (my-syntax-propertize 980 1556)
1 <- my-syntax-propertize: nil
======================================================================
1 -> (my-syntax-propertize 1043 1616)
1 <- my-syntax-propertize: nil
======================================================================
1 -> (my-syntax-propertize 1043 1617)
1 <- my-syntax-propertize: nil

But I still can't use forward-sexp and backward-sexp to navigate from one
end of
<< to its matching >> or vice versa...


-- 
/Pierre

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

On Tue, Oct 5, 2021 at 3:42 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> OK, thanks, that makes sense (I just wanted to make sure the problem
> wasn't some silly misunderstanding).
>
No problem.

>
> The code you show seems fine to me except for the `2` which should be
> a `1` (there is only one subgroup in your second regexp).
>
> Ahh, a silly typo, when I copied from the wrong place.  I have the right
'1' in place in
erlang.el but it still does not behave as I would have expected.


> To debug these things, you can try:
>
> - `C-u C-x =` on the char(s) where you expect a syntax-table text property,
>   to check whether it's present or not.
>
This is had done to check the syntax and wrote something to list all text
properties
of the character at point because what-cursor-position does not show them
all.


> - Call `syntax-propertize` by hand in case of doubt.
> - Trace your rules function.  E.g. use
>
>       (defun my-syntax-propertize (beg end)
>         (funcall
>          (syntax-propertize-rules
>            ("\\(<\\)<" (1 "(>"))
>            (">\\(>\\)" (1 ")<")))
>          beg end))
>
>       [...]
>
>       (setq-local syntax-propertize-function
>                   #'my-syntax-propertize)
>
>   and then `M-x trace-function RET my-syntax-propertize RET`.
>
> Thanks! That's what I was looking for!  I'll read more on that topic.

Now I did experimentation with that and got the following trace
**when I complete a closing >> only**:

1 -> (my-syntax-propertize 980 1555)
1 <- my-syntax-propertize: nil
======================================================================
1 -> (my-syntax-propertize 980 1556)
1 <- my-syntax-propertize: nil
======================================================================
1 -> (my-syntax-propertize 1043 1616)
1 <- my-syntax-propertize: nil
======================================================================
1 -> (my-syntax-propertize 1043 1617)
1 <- my-syntax-propertize: nil

But I still can't use forward-sexp and backward-sexp to navigate from one
end of
<< to its matching >> or vice versa...

(Sorry for double posting, I did not send to the list the first time around)

-- 
/Pierre

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Stefan Monnier via Users list for the GNU Emacs text editor]

> 1 -> (my-syntax-propertize 980 1555)
> 1 <- my-syntax-propertize: nil
> ======================================================================
> 1 -> (my-syntax-propertize 980 1556)
> 1 <- my-syntax-propertize: nil
> ======================================================================
> 1 -> (my-syntax-propertize 1043 1616)
> 1 <- my-syntax-propertize: nil
> ======================================================================
> 1 -> (my-syntax-propertize 1043 1617)
> 1 <- my-syntax-propertize: nil

So, IIUC even though the function is called, the relevant char doesn't
get the corresponding `syntax-table` text property?

Maybe the major-mode is getting in the way?

You can try to use:

      (defun my-syntax-propertize (beg end)
        (funcall
         (syntax-propertize-rules
           ("\\(<\\)<"
            (1 (progn
                 (message "OPEN at %S" (match-beginning 1))
                 (string-to-syntax "(>")))))
           (">\\(>\\)"
            (1 (progn
                 (message "CLOSE at %S" (match-beginning 1))
                 (string-to-syntax "(>")))))
         beg end))

Then check your *Messages* buffer to see where the function placed the
property, then look at these positions (e.g. with `C-u C-x =`) to see if
the `syntax-table` text property is still present.

If the property is absent (i.e. has been removed after your function
added it), then you may want to check your `syntax-propertize-function`
and search for `syntax-table` in erlong-mode's code to see what might be
interfering (e.g. maybe this property is added to
`font-lock-extra-properties`?).


        Stefan

Re: When is a syntax-propertize-function called when parse-sexp-lookup-properties is t for a current buffer?

[Pierre Rouleau]

On Tue, Oct 5, 2021 at 9:55 PM Stefan Monnier via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

>
> So, IIUC even though the function is called, the relevant char doesn't
> get the corresponding `syntax-table` text property?
>
> Correct.

Maybe the major-mode is getting in the way?
>
> That's what I think.

You can try to use:
>
>       (defun my-syntax-propertize (beg end)
>
        (funcall
>          (syntax-propertize-rules
>            ("\\(<\\)<"
>             (1 (progn
>                  (message "OPEN at %S" (match-beginning 1))
>                  (string-to-syntax "(>")))))
>            (">\\(>\\)"
>             (1 (progn
>                  (message "CLOSE at %S" (match-beginning 1))
>                  (string-to-syntax "(>")))))
>          beg end))
>
> Then check your *Messages* buffer to see where the function placed the
> property, then look at these positions (e.g. with `C-u C-x =`) to see if
> the `syntax-table` text property is still present.
>
> If the property is absent (i.e. has been removed after your function
> added it), then you may want to check your `syntax-propertize-function`
> and search for `syntax-table` in erlong-mode's code to see what might be
> interfering (e.g. maybe this property is added to
> `font-lock-extra-properties`?).
>

I had done something similar and yes the syntax-table properties disappear
from the character.  The erlang.el erlang-mode sets up the syntax for < and
> as
punctuation character with:

(modify-syntax-entry ?< "." table)
(modify-syntax-entry ?> "." table)

It also sets up electric behaviour of >  for the > in '->' sequence to
automatically
insert newline and indent by doing:

;; Also, set up text properties for bit syntax handling.


  (put 'bitsyntax-open-outer  'syntax-table '(4 . ?>))
  (put 'bitsyntax-open-outer  'rear-nonsticky '(category))
  ;;


  (put 'bitsyntax-open-inner  'rear-nonsticky '(category))
  (put 'bitsyntax-close-inner 'rear-nonsticky '(category))
  ;;


  (put 'bitsyntax-close-outer 'syntax-table '(5 . ?<))
  (put 'bitsyntax-close-outer 'rear-nonsticky '(category))

I'm not very familiar yet with all the syntax handling mechanism and that's
why I wanted to know what could interfere and why.  I was looking for some
centralized info on the topic but I guess it's spread across text
properties, syntax , etc..

Thanks for the info, I will check some more things in the code.

-- 
/Pierre