bug#9148: Make `comment-normalize-vars' more syntax-aware

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Štěpán Němec]

Problem:
--------

Emacs currently provides relatively decent support for major modes with
multiple syntaxes, mainly via the `syntax-table' text property.

Unfortunately, even though there is a `comment-use-syntax' variable
which is apparently supposed to indicate that the comment functions
should rely on syntax information instead of `comment-prefix' and
friends, one piece is missing -- the function `comment-normalize-vars',
called at invocation of functions like {comment,uncomment}-region, still
requires `comment-start' to be set explicitly even though the syntax
information is sufficient and `comment-use-syntax' set.

Suggested solution:
-------------------

Modify `comment-normalize-vars' so that it uses the syntax information
(current syntax table, `syntax-table' text property) _only_, whenever
`comment-use-syntax' is set and this is possible.

Use case:
---------

Consider a major mode with two or more clearly separate syntaxes (Org
mode source blocks could be one example), including comments.

Even though it is possible to provide the different text extents with
different values of the `syntax-table' text property, the only way to
work around `comment-normalize-vars' not relying on them currently seems
something like the following:

(defadvice comment-normalize-vars (around use-syntax-info activate)
  (set (make-local-variable 'comment-start)
       (determine-comment-start-based-on-syntax-info))
  ad-do-it)

...which is evil, ugly and doesn't scale.

-- 
Štěpán

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Stefan Monnier]

> Unfortunately, even though there is a `comment-use-syntax' variable
> which is apparently supposed to indicate that the comment functions
> should rely on syntax information instead of `comment-prefix' and
> friends, one piece is missing -- the function `comment-normalize-vars',
> called at invocation of functions like {comment,uncomment}-region, still
> requires `comment-start' to be set explicitly even though the syntax
> information is sufficient and `comment-use-syntax' set.

I'm not sure I understand the problem.  Can you give a more concrete
example where you get undesirable behavior?
Setting comment-start is usually needed to tell Emacs *which* of the
major mode's comment syntaxes to use (so it's only theoretically
unneeded if the syntax-table only include a single comment syntax).


        Stefan

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Štěpán Němec]

On Tue, 02 Aug 2011 06:04:03 +0200
Stefan Monnier wrote:

>> Unfortunately, even though there is a `comment-use-syntax' variable
>> which is apparently supposed to indicate that the comment functions
>> should rely on syntax information instead of `comment-prefix' and
>> friends, one piece is missing -- the function `comment-normalize-vars',
>> called at invocation of functions like {comment,uncomment}-region, still
>> requires `comment-start' to be set explicitly even though the syntax
>> information is sufficient and `comment-use-syntax' set.
>
> I'm not sure I understand the problem.  Can you give a more concrete
> example where you get undesirable behavior?
> Setting comment-start is usually needed to tell Emacs *which* of the
> major mode's comment syntaxes to use (so it's only theoretically
> unneeded if the syntax-table only include a single comment syntax).

The point is that you can define a major mode with different syntax
tables in different parts of the text (using the syntax-table text
property), but instead of using that information to determine
comment-start and the other variables, `comment-normalize-vars' still
relies on `comment-start' being hard-coded (which doesn't really make
sense in a buffer with multiple comment syntaxes), even when
`comment-use-syntax' is set and the syntax-table information is
sufficient.

-- 
Štěpán

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Stefan Monnier]

> The point is that you can define a major mode with different syntax
> tables in different parts of the text (using the syntax-table text
> property), but instead of using that information to determine
> comment-start and the other variables, `comment-normalize-vars' still
> relies on `comment-start' being hard-coded (which doesn't really make
> sense in a buffer with multiple comment syntaxes), even when
> `comment-use-syntax' is set and the syntax-table information is
> sufficient.

But this is part of a much larger problem (multiple major modes in the
same buffer).

Basically you're suggesting to use the syntax-table property as a way to
specify the major mode used in each part and then teach newcomment.el to
never use buffer-local variables but only the syntax-table around point.

It might be a good idea, but it's a non-trivial change (and still
leaves open the question of how to specify that I want //...\n for the
C chunks and (*...*) for the Pascal chunks).


        Stefan

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Štěpán Němec]

On Tue, 02 Aug 2011 21:03:54 +0200
Stefan Monnier wrote:

>> The point is that you can define a major mode with different syntax
>> tables in different parts of the text (using the syntax-table text
>> property), but instead of using that information to determine
>> comment-start and the other variables, `comment-normalize-vars' still
>> relies on `comment-start' being hard-coded (which doesn't really make
>> sense in a buffer with multiple comment syntaxes), even when
>> `comment-use-syntax' is set and the syntax-table information is
>> sufficient.
>
> But this is part of a much larger problem (multiple major modes in the
> same buffer).

Maybe, but the specific change to `comment-normalize-vars' (gather the
appropriate value for `comment-start' from the syntax-table at point
instead of prompting the user when `comment-use-syntax' is set) I
proposed would seem easy enough and fix this very problem.

> Basically you're suggesting to use the syntax-table property as a way to
> specify the major mode used in each part and then teach newcomment.el to
> never use buffer-local variables but only the syntax-table around
> point.

More precisely and less ambitiously, I'm suggesting to do what
`comment-use-syntax' seems to be claiming already (that's also why I
still consider this a bug, not an enhancement request) -- to rely on
syntax-table instead of the regexp variables when manipulating comments.

> It might be a good idea, but it's a non-trivial change (and still
> leaves open the question of how to specify that I want //...\n for the
> C chunks and (*...*) for the Pascal chunks).

I don't understand. Again: the case I've described, the comment syntax
is unambiguously specified for every part of the buffer using the
syntax-table (either the buffer-local value, or the text property
value). The C chunks have the C syntax table as its value, the Pascal
chunks have the Pascal syntax table as its value (if there is such a
thing -- I don't use C or Pascal; in my specific case it was JavaScript
on one hand and a VimL[1]-like custom syntax with double quote comment
start and end-of-line comment end on the other).

The only thing that's needed to fix this very use case is make
`comment-normalize-vars' set `comment-start' from the syntax-table when
`comment-use-syntax' is set, instead of forcing me to advise it and do
it myself. It'd still be ugly, but not more ugly than it already is now,
and certainly better than using advice.


[1] The "language" used for Vim scripting.

-- 
Štěpán

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Stefan Monnier]

>>> The point is that you can define a major mode with different syntax
>>> tables in different parts of the text (using the syntax-table text
>>> property), but instead of using that information to determine
>>> comment-start and the other variables, `comment-normalize-vars' still
>>> relies on `comment-start' being hard-coded (which doesn't really make
>>> sense in a buffer with multiple comment syntaxes), even when
>>> `comment-use-syntax' is set and the syntax-table information is
>>> sufficient.
>> But this is part of a much larger problem (multiple major modes in the
>> same buffer).
> Maybe, but the specific change to `comment-normalize-vars' (gather the
> appropriate value for `comment-start' from the syntax-table at point
> instead of prompting the user when `comment-use-syntax' is set) I
> proposed would seem easy enough and fix this very problem.

We could do that, yes, but unless we regularly flush the comment-start
value we got, it will then keep using the first value found even on
chunks using another language.

>> Basically you're suggesting to use the syntax-table property as a way to
>> specify the major mode used in each part and then teach newcomment.el to
>> never use buffer-local variables but only the syntax-table around
>> point.
> More precisely and less ambitiously, I'm suggesting to do what
> `comment-use-syntax' seems to be claiming already (that's also why I
> still consider this a bug, not an enhancement request) -- to rely on
> syntax-table instead of the regexp variables when manipulating comments.

comment-use-syntax was less ambitious: it was mostly about using
forward-comment.  It does try to build some of the vars from the
syntax-table, but mostly because comment-end-skip was new in
newcomment.el and I needed the code to work with old packages ;-)

But yes, comment-start could sometimes also be set from the
syntax-table.  I might accept a patch for it, tho I'm not completely
sure how useful that would be.

>> It might be a good idea, but it's a non-trivial change (and still
>> leaves open the question of how to specify that I want //...\n for the
>> C chunks and (*...*) for the Pascal chunks).
> I don't understand. Again: the case I've described, the comment syntax
> is unambiguously specified for every part of the buffer using the
> syntax-table (either the buffer-local value, or the text property
> value). The C chunks have the C syntax table as its value, the Pascal

That's not unambiguous: the C syntax-table specifies several comment
syntaxes (/*..*/ and //..\n) and so does the Pascal syntax table (even
more combinations, IIRC).  So comment-start-skip can be built from the
syntax-table since it needs to match "any comment starter", but
comment-start is more problematic since it would need to choose whose
comment syntax to favor.

> The only thing that's needed to fix this very use case is make
> `comment-normalize-vars' set `comment-start' from the syntax-table when
> `comment-use-syntax' is set, instead of forcing me to advise it and do
> it myself. It'd still be ugly, but not more ugly than it already is now,
> and certainly better than using advice.

But if it's not a major mode used over the whole file, when do you flush
the comment-start value you computed?


        Stefan

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Lars Ingebrigtsen]

Štěpán Němec <stepnem@gmail.com> writes:

> Suggested solution:
> -------------------
>
> Modify `comment-normalize-vars' so that it uses the syntax information
> (current syntax table, `syntax-table' text property) _only_, whenever
> `comment-use-syntax' is set and this is possible.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

We now have some multi-mode modes in Emacs, so I'm not sure what's being
proposed here is necessary these days (or indeed it has been fixed in
the decade since this was reported).  Is this something that is still
relevant?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Štěpán Němec]

On Thu, 27 Jan 2022 18:57:09 +0100
Lars Ingebrigtsen wrote:

> Štěpán Němec <stepnem@gmail.com> writes:
>
>> Suggested solution:
>> -------------------
>>
>> Modify `comment-normalize-vars' so that it uses the syntax information
>> (current syntax table, `syntax-table' text property) _only_, whenever
>> `comment-use-syntax' is set and this is possible.
>
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
>
> We now have some multi-mode modes in Emacs, so I'm not sure what's being
> proposed here is necessary these days (or indeed it has been fixed in
> the decade since this was reported).  Is this something that is still
> relevant?

Honestly, I don't know: I ended up using the defadvice hack those 10
years ago, but a few years after that the major mode that needed it
became obsolete along with the application whose config files it
supported, and I haven't needed anything similar since then.

Given that nobody other than Stefan, who expressed reservations to my
suggestions, has chimed in during those 10 years and, as you say, the
landscape of possible solutions has likely changed, in its current state
this bug report will probably not be less useful closed than it is
opened, should you so prefer.

  Thanks,

  Štěpán

bug#9148: Make `comment-normalize-vars' more syntax-aware

[Lars Ingebrigtsen]

Štěpán Němec <stepnem@gmail.com> writes:

> Given that nobody other than Stefan, who expressed reservations to my
> suggestions, has chimed in during those 10 years and, as you say, the
> landscape of possible solutions has likely changed, in its current state
> this bug report will probably not be less useful closed than it is
> opened, should you so prefer.

OK; closing this bug report, then.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no