Re: [Emacs-diffs] master 8dafacd: * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

Re: [Emacs-diffs] master 8dafacd: * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

[Dmitry Gutov]

On 01/16/2016 11:06 PM, Stefan Monnier wrote:
> branch: master
> commit 8dafacd0419ea890af461c9d42d4642155681eec
> Author: Stefan Monnier <monnier@iro.umontreal.ca>
> Commit: Stefan Monnier <monnier@iro.umontreal.ca>
>
>      * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

Hi Stefan,

Are there any particular cases where we should use it? You've added the 
var, but without any usages.

Re: [Emacs-diffs] master 8dafacd: * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

>> * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

> Are there any particular cases where we should use it? You've added the var,
> but without any usages.

I am also curious...

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] master 8dafacd: * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

[Stefan Monnier]

>>> * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var
>> Are there any particular cases where we should use it?

Any time where you want to use a different syntax-table.  E.g. e use it
in nxml-mode where the syntax-table to use normally should be
"text-like" (e.g. parens are matched), but where actual formal parsing
needs something very different (since nothing guarantees that parens
are matched, whereas matching <..> is a lot more useful).

I expect it to be handy as well in cases where the "normal" syntax table
may be changed at various times.  E.g. this allows the use of
syntax-ppss within with-syntax-table.  I actually imagine it may become
"normal and recommended" to set syntax-ppss-table in any buffer where
with-syntax-table may be used, since syntax-ppss can very easily be
triggered by accident (e.g. via forward-sexp -> syntax-propertize ->
syntax-ppss).

>> You've added the var, but without any usages.

Actually there is one usage in trunk (in nxml).  I just mistakenly
installed it before installing the corresponding change in syntax.el.


        Stefan

Re: [Emacs-diffs] master 8dafacd: * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

[Dmitry Gutov]

On 01/18/2016 12:32 AM, Stefan Monnier wrote:

> Any time where you want to use a different syntax-table.  E.g. e use it
> in nxml-mode where the syntax-table to use normally should be
> "text-like" (e.g. parens are matched), but where actual formal parsing
> needs something very different (since nothing guarantees that parens
> are matched, whereas matching <..> is a lot more useful).

Thanks. What is the purpose left for the "main" syntax table, though? 
Doesn't paren-matching code call syntax-ppss anyway?

I'm worried that we have _three_ different syntax tables now (including 
the font-lock- one), without clear documentation on how they should relate.

> I expect it to be handy as well in cases where the "normal" syntax table
> may be changed at various times.  E.g. this allows the use of
> syntax-ppss within with-syntax-table.  I actually imagine it may become
> "normal and recommended" to set syntax-ppss-table in any buffer where
> with-syntax-table may be used, since syntax-ppss can very easily be
> triggered by accident (e.g. via forward-sexp -> syntax-propertize ->
> syntax-ppss).

So, forward-sexp can trigger syntax-ppss via syntax-propertize, which 
will use syntax-ppss-table, but forward-sexp itself will use the main 
syntax table? At least when called with positive argument.

But up-list uses the return value of syntax-ppss, so it will be affected 
by syntax-ppss-table? And backward-sexp, too?

It doesn't seem easy to reason about.

>>> You've added the var, but without any usages.
>
> Actually there is one usage in trunk (in nxml).  I just mistakenly
> installed it before installing the corresponding change in syntax.el.

Yes, I missed that (that patch was long).

Re: [Emacs-diffs] master 8dafacd: * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

[Stefan Monnier]

> Doesn't paren-matching code call syntax-ppss anyway?

Not that I know, no.

> I'm worried that we have _three_ different syntax tables now (including the
> font-lock- one), without clear documentation on how they should relate.

Yes, this whole area is rather murky.  The font-lock syntax-table should
mostly disappear (90% of the uses were to turn "_" syntax into "w"
syntax so as to be able to use \< and \>, but this has been made
obsolete by the new \_< and \_>), I think.

But the interaction between the various syntax-tables as well as
forward-sexp-function deserves a serious rethink.  I think such a rethink
will need to start by distinguishing "user-level dwimish" behavior and
lower-level behavior which adheres to strict syntax rules.

E.g. in TeX and HTML, we want to match parentheses even though the formal
syntax does not give any special rule to parentheses.  Similar thing for
matching parens when the open and close parens are both inside comments,
but not inside the same one.

> So, forward-sexp can trigger syntax-ppss via syntax-propertize, which will
> use syntax-ppss-table, but forward-sexp itself will use the main syntax
> table? At least when called with positive argument.

Right.

> But up-list uses the return value of syntax-ppss, so it will be affected by
> syntax-ppss-table?

syntax-ppss is indeed used, but only to decide if we're strings or
comments, so it should be OK.

> It doesn't seem easy to reason about.

Indeed, it's not.


        Stefan

Re: [Emacs-diffs] master 8dafacd: * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

[Dmitry Gutov]

On 01/19/2016 01:12 AM, Stefan Monnier wrote:
>> Doesn't paren-matching code call syntax-ppss anyway?
>
> Not that I know, no.

electric-pair-syntax-info calls syntax-ppss. blink-matching-open calls 
(forward-sexp -1).

> Yes, this whole area is rather murky.  The font-lock syntax-table should
> mostly disappear (90% of the uses were to turn "_" syntax into "w"
> syntax so as to be able to use \< and \>, but this has been made
> obsolete by the new \_< and \_>), I think.

font-lock-syntax-table should go away, yes. But it's still there.

Are current-syntax-table and syntax-ppss-table allowed to differ more 
than that?

> E.g. in TeX and HTML, we want to match parentheses even though the formal
> syntax does not give any special rule to parentheses.  Similar thing for
> matching parens when the open and close parens are both inside comments,
> but not inside the same one.

Multiple major modes strike again! Or multiple syntactic regions, 
anyway. I'm not sure if having a different value of syntax-table (any 
variable) between different comments will help us any.

>> But up-list uses the return value of syntax-ppss, so it will be affected by
>> syntax-ppss-table?
>
> syntax-ppss is indeed used, but only to decide if we're strings or
> comments, so it should be OK.

Never for its 9th element?

>> It doesn't seem easy to reason about.
>
> Indeed, it's not.

So, I'm wondering if we could postpone introducing a new variable, and 
implement the same thing in nxml-mode somehow in a different way.

But you've probably already considered that.

Re: [Emacs-diffs] master 8dafacd: * lisp/emacs-lisp/syntax.el (syntax-ppss-table): New var

[Stefan Monnier]

>>> Doesn't paren-matching code call syntax-ppss anyway?
>> Not that I know, no.
> electric-pair-syntax-info calls syntax-ppss.

Yes, but again that's only to figure out if we're inside
a string/comment, so I think it's fine.

> Are current-syntax-table and syntax-ppss-table allowed to differ more
> than that?

In the one current use case, they differ significantly, yes.

>>> But up-list uses the return value of syntax-ppss, so it will be affected by
>>> syntax-ppss-table?
>> syntax-ppss is indeed used, but only to decide if we're strings or
>> comments, so it should be OK.
> Never for its 9th element?

AFAICT no (even though it could, indeed).

>>> It doesn't seem easy to reason about.
>> Indeed, it's not.
> So, I'm wondering if we could postpone introducing a new variable, and
> implement the same thing in nxml-mode somehow in a different way.

I think syntax-ppss-table is "the right way" in that it is the place
where we can define the "low-level, strict, formal syntax".

This new variable makes it possible to use with-syntax-table safely with
any random syntax table which happens to parse things the way we
currently need it.

There are indeed potential problems with interactions with other
functions, but I expect that the right fix for them would be to be more
careful about whether they use the output of syntax-ppss (and hence
the proper syntax of the major mode) or if they use something else
(e.g. forward-sexp) to get a more dwimish behavior.


        Stefan