syntax-after

syntax-after

[Stefan Monnier]

What was the reason for the following change?

2004-11-06  Richard M. Stallman  <rms@gnu.org>

	* subr.el (syntax-after): Return the syntax letter, not the raw code.

It seems really gratuitous.  There are already 2 ways to represent the
syntax info of a char: as a cons cell and as a string.  Both are documented
in the elisp manual.

The above change introduces yet a third representation, and an undocumented
one at that.  Was that really necessary?

I just saw that the above change caused Juri to install a fix to
descr-text.el that replaces a call to syntax-after with a copy of its
(previous) body.  I think this is moving backward, not forward.


        Stefan

Re: syntax-after

[Juri Linkov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> What was the reason for the following change?
>
> 2004-11-06  Richard M. Stallman  <rms@gnu.org>
>
> 	* subr.el (syntax-after): Return the syntax letter, not the raw code.
>
> It seems really gratuitous.  There are already 2 ways to represent the
> syntax info of a char: as a cons cell and as a string.  Both are documented
> in the elisp manual.
>
> The above change introduces yet a third representation, and an undocumented
> one at that.  Was that really necessary?
>
> I just saw that the above change caused Juri to install a fix to
> descr-text.el that replaces a call to syntax-after with a copy of its
> (previous) body.  I think this is moving backward, not forward.

FWIW, I grepped the whole source tree, and there were only 2 places
where `syntax-after' was used: `show-paren-function' and `describe-char'.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: syntax-after

[Richard Stallman]

    It seems really gratuitous.  There are already 2 ways to represent the
    syntax info of a char: as a cons cell and as a string.  Both are documented
    in the elisp manual.

    The above change introduces yet a third representation, and an undocumented
    one at that.  Was that really necessary?

It is not new.  My change makes it compatible with char-syntax, which
is why I did it.  It is also more comparible with the way
modify-syntax-entry works, since the same character is used now by both.

This change reduces the number of representations of the syntax by
one, as far as I know.  Or is there some other function that uses the
old representation of syntax-after?  If so, perhaps it should be
changed too.

    I just saw that the above change caused Juri to install a fix to
    descr-text.el that replaces a call to syntax-after with a copy of its
    (previous) body.

That is strange, I searched for all the calls and only found them in
paren.el.

Re: syntax-after

[Juri Linkov]

Richard Stallman <rms@gnu.org> writes:
>     It seems really gratuitous.  There are already 2 ways to represent the
>     syntax info of a char: as a cons cell and as a string.  Both are documented
>     in the elisp manual.
>
>     The above change introduces yet a third representation, and an undocumented
>     one at that.  Was that really necessary?
>
> It is not new.  My change makes it compatible with char-syntax, which
> is why I did it.  It is also more comparible with the way
> modify-syntax-entry works, since the same character is used now by both.
>
> This change reduces the number of representations of the syntax by
> one, as far as I know.  Or is there some other function that uses the
> old representation of syntax-after?  If so, perhaps it should be
> changed too.

Actually, with the change in `syntax-after' there are 4 representations now:

modify-syntax-entry  aref(syntax-table)  char-syntax  syntax-after      
"w"                  (2)                 ?w           (?w)
"w p"                (1048578)           ?w           args-out-of-range
". 124b"             (2818049)           ?.           args-out-of-range
"(]"                 (4 . ?\])           ?\(          (?\( . ?\])

(Note that `syntax-after' fails with args-out-of-range on some syntaxes.)

It seems you wanted to add syntax flags to the returned value which
are not available from `char-syntax'.

I think a better solution that would reduce the number of
representations is to implement a new function `syntax-to-string'
returning a syntax string (e.g. ". 124b").  All flags can be accessed
from this string easily either with a regexp or with `substring' or `aref'
when needed.

As I see in the Emacs source tree `char-syntax' is mostly used
in combination with `char-after':

    (eq (char-syntax (char-after point)) ?w)

where only syntax class character is needed, and not flags.

For such cases `syntax-after' would serve as a replacement to make
code simpler:

(defun syntax-after (point)
  (char-syntax (char-after point)))

>     I just saw that the above change caused Juri to install a fix to
>     descr-text.el that replaces a call to syntax-after with a copy of its
>     (previous) body.
>
> That is strange, I searched for all the calls and only found them in
> paren.el.

Since `describe-char' calls `internal-describe-syntax-value' which
accepts the internal syntax representation, I think no special
function for that code is needed, because there are too few cases
where the internal syntax representation is used (another one I know
is `describe-syntax' command).

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: syntax-after

[Stefan]

> It is not new.  My change makes it compatible with char-syntax, which
> is why I did it.

But the code you used is not compatible with `char-syntax'.  To be
compatible with `char-syntax' you'd have to drop the "matching paren"
(i.e. the cdr of the data) as well as the flags.
I don't think that hiding this "side" info would be a good idea either.

I.e. I think syntax-after should return the "raw" data as before (after all
that's why I wrote it like that originally).  If you find this raw data to
be intimidating, we should provide another function `syntax-class' such that

   (syntax-class (syntax-after)) == (char-syntax (char-after))

or something along these lines.

> It is also more comparible with the way modify-syntax-entry works, since
> the same character is used now by both.

> This change reduces the number of representations of the syntax by
> one, as far as I know.  Or is there some other function that uses the
> old representation of syntax-after?  If so, perhaps it should be
> changed too.

The raw representation (i.e. the cons cell representation used inside the
chartables used as syntax-tables) does appear at various places (it appears
if you do (aref <syntax-table> ?\[) or if you lookup the `syntax-table'
text-property, it is used in font-lock-syntactic-keywords for code that
needs to work on Emacs-20, ...) and it seems difficult to get rid of it now;
so no, you haven't reduced the number of representations.


        Stefan

Re: syntax-after

[Richard Stallman]

    modify-syntax-entry  aref(syntax-table)  char-syntax  syntax-after      
    "w"                  (2)                 ?w           (?w)
    "w p"                (1048578)           ?w           args-out-of-range
    ". 124b"             (2818049)           ?.           args-out-of-range

The args-out-of-range is simply a bug.  I intended syntax-after to
return (?w) and (?.)  in those cases, so that the car of the value
would match char-syntax.

    The raw representation (i.e. the cons cell representation used inside the
    chartables used as syntax-tables) does appear at various places (it appears
    if you do (aref <syntax-table> ?\[) or if you lookup the `syntax-table'
    text-property, it is used in font-lock-syntactic-keywords for code that
    needs to work on Emacs-20, ...) and it seems difficult to get rid of it now;
    so no, you haven't reduced the number of representations.

I intended this change as a step towards reducing the mess
of these different representations.  However, you've convinced
me the change should be undone.

I don't know if I will have net access this week--could you please
undo it?

      If you find this raw data to
    be intimidating, we should provide another function `syntax-class' such that

       (syntax-class (syntax-after)) == (char-syntax (char-after))

I guess we should do that.

Maybe we should make modify-syntax-entry accept these values,
as well as the strings it currently accepts.

Re: syntax-after

[Stefan Monnier]

>     The raw representation (i.e. the cons cell representation used inside
>     the chartables used as syntax-tables) does appear at various places
>     (it appears if you do (aref <syntax-table> ?\[) or if you lookup the
>     `syntax-table' text-property, it is used in
>     font-lock-syntactic-keywords for code that needs to work on Emacs-20,
>     ...) and it seems difficult to get rid of it now; so no, you haven't
>     reduced the number of representations.

> I intended this change as a step towards reducing the mess
> of these different representations.  However, you've convinced
> me the change should be undone.

> I don't know if I will have net access this week--could you please
> undo it?

Done,


        Stefan

Re: syntax-after

[Kim F. Storm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>     The raw representation (i.e. the cons cell representation used inside
>>     the chartables used as syntax-tables) does appear at various places
>>     (it appears if you do (aref <syntax-table> ?\[) or if you lookup the
>>     `syntax-table' text-property, it is used in
>>     font-lock-syntactic-keywords for code that needs to work on Emacs-20,
>>     ...) and it seems difficult to get rid of it now; so no, you haven't
>>     reduced the number of representations.
>
>> I intended this change as a step towards reducing the mess
>> of these different representations.  However, you've convinced
>> me the change should be undone.
>
>> I don't know if I will have net access this week--could you please
>> undo it?
>
> Done,
>

It seems you forgot to undo changes to etc/NEWS and syntax.texi...

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: syntax-after

[Stefan Monnier]

>>> I don't know if I will have net access this week--could you please
>>> undo it?
>> Done,
> It seems you forgot to undo changes to etc/NEWS and syntax.texi...

Oops, sorry.  I've fixed etc/NEWS and syntax.texi.
Any other place?


        Stefan

Re: syntax-after

[Juri Linkov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> I don't know if I will have net access this week--could you please
>>>> undo it?
>>> Done,
>> It seems you forgot to undo changes to etc/NEWS and syntax.texi...
>
> Oops, sorry.  I've fixed etc/NEWS and syntax.texi.
> Any other place?

You have reverted the changes in paren.el to old code:

    (dir (cond ((eq (car (syntax-after (1- (point)))) 5) -1)
               ((eq (car (syntax-after (point))) 4) 1)))

but this code is really bad, since it uses raw enum values 4 and 5
which are nowhere documented and too implementation specific.

You already proposed a new function `syntax-class'.  This would
improve the code.

I also think there should exist another function returning a complete
syntax string with all flags (e.g. `syntax-to-string').
Such C function could be created from `internal-describe-syntax-value'
(up to the line inserting "\twhich means:") with moving the rest of
the code in `internal-describe-syntax-value' (which just inserts
a readable syntax description) to Lisp side (e.g. to subr.el).

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: syntax-after

[Stefan]

> but this code is really bad, since it uses raw enum values 4 and 5
> which are nowhere documented and too implementation specific.

It's documented in the elisp manual.

> You already proposed a new function `syntax-class'.  This would
> improve the code.

Yes, but I have no time to devote to this for now.  I think focusing on
getting to a release is a better use of my time.

> I also think there should exist another function returning a complete
> syntax string with all flags (e.g. `syntax-to-string').

What for?

> Such C function could be created from `internal-describe-syntax-value'
> (up to the line inserting "\twhich means:") with moving the rest of
> the code in `internal-describe-syntax-value' (which just inserts
> a readable syntax description) to Lisp side (e.g. to subr.el).

I consciously decided to keep it in C when I introduced
internal-describe-syntax-value, for two reasons:
1 - the code was there and there didn't seem to be any strong need to move it.
2 - any addition/removal/change of syntax codes or syntax-flags would
    involve changes in syntax.c but pretty much nowhere else, and I think
    it's good to keep it that way.

Note also that given the cost of converting (from and) back to a string and
the inconvenience of manipulating this string, I recommend you try to avoid
writing code that uses such strings and instead try to work directly with
the internal representation.


        Stefan

Re: syntax-after

[Richard Stallman]

    It seems you forgot to undo changes to etc/NEWS and syntax.texi...

Could you do that?