Re: master 5c70ff9: New user option 'font-lock-ignore'

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Eli Zaretskii]

Thanks for developing this feature.

Reading its documentation, I think it should be clarified to be more
useful.  See the questions/comments below:

> +@example
> +(@var{mode} @var{rule} @dots{})
> +@end example
> +
> +Here, @var{mode} is a symbol, say a major or minor mode.

Why "say"?  Can it usefully be something else? if so, what can it be?

>                                                            The
> +subsequent rules apply if the current major mode is derived from
> +@var{mode} or @var{mode} is bound and true as a variable.  Each
> +@var{rule} can be one of the following:
> +
> +@table @code
> +@cindex @var{font-lock-ignore} rules
> +@item @var{symbol}
> +A symbol, say a face name, matches any Font Lock keyword containing
> +the symbol in its definition.

I don't think I understand how can a keyword "contain" a symbol.
Please elaborate or suggest a different text that clarifies this.

> The symbol is interpreted as a glob
> +pattern; in particular, @code{*} matches everything.

Do you mean shell glob patterns?  AFAIK, we don't use them in Emacs
except for shell wildcards, so why are they used here? why not
regexps?  And in any case, those patterns need a thorough description,
since we don't use them elsewhere.  For example, the way to quote
meta-characters such as '*' must be documented, because it isn't
self-evident.

> +@item @var{string}
> +A string matches any font-lock keyword defined by a regexp that
> +matches the string.

I don't think I understand this sentence.  In particular, saying "a
regexp that matches the string" is at least unusual: we usually say it
the other way around: "a string that matches the regexp".  And what
does it mean for a keyword to be "defined by a regexp"?

> +In each buffer, Font Lock keywords that match at least one applicable
> +rule are disabled.

This should be _before_ the list of rules, not after.  And given the
fact that actions of the rules are implicit (i.e. they aren't present
in the rules themselves), I would suggest to call them "conditions",
not "rules".  A rule includes the action, not just the condition for
that action, and those "rules" don't.

> +@smallexample
> +(setq font-lock-ignore
> +      '((prog-mode font-lock-*-face
> +                   (except help-echo))
> +        (emacs-lisp-mode (except ";;;###autoload)")
> +        (whitespace-mode whitespace-empty-at-bob-regexp)
> +        (makefile-mode (except *))))
> +@end smallexample
> +
> +Line by line, this does the following:
> +
> +@enumerate
> +@item
> +In all programming modes, disable all font-lock keywords that apply
> +one of the standard font-lock faces (excluding strings and comments,
> +which are covered by syntactic Font Lock).
> +
> +@item
> +However, keep any keywords that add a @code{help-echo} text property.
> +
> +@item
> +In Emacs Lisp mode, also keep the highlighting of autoload cookies,
> +which would have been excluded by rule 1.
> +
> +@item
> +In @code{whitespace-mode} (a minor mode), don't highlight an empty
> +line at beginning of buffer.

Shouldn't you have "also" in this last description, like you did for
Emacs Lisp mode?

And why doesn't the example show all the possible forms of a "rule",
but just one of them?

> +*** New user option 'font-lock-ignore'.
> +This variable provides a mechanism to selectively disable font-lock
> +keywords.

I don't think it disables keywords, I think it disables
fontifications.  (How can one disable a keyword?)

> +
> ++++
>  *** New package vtable.el for formatting tabular data.
>  This package allows formatting data using variable-pitch fonts.
>  The resulting tables can display text in variable pitch fonts, text
> diff --git a/lisp/font-lock.el b/lisp/font-lock.el
> index d8a1fe3..8af3c30 100644
> --- a/lisp/font-lock.el
> +++ b/lisp/font-lock.el
> @@ -208,6 +208,7 @@
>  
>  (require 'syntax)
>  (eval-when-compile (require 'cl-lib))
> +(eval-when-compile (require 'subr-x))
>  
>  ;; Define core `font-lock' group.
>  (defgroup font-lock '((jit-lock custom-group))
> @@ -279,6 +280,42 @@ font-lock-maximum-decoration
>  				      (integer :tag "level" 1)))))
>    :group 'font-lock)
>  
> +(defcustom font-lock-ignore nil
> +  "Rules to selectively disable font-lock keywords.
> +This is a list of rule sets of the form
> +
> +  (MODE RULE ...)
> +
> +where:
> +
> + - MODE is a symbol, say a major or minor mode.  The subsequent
> +   rules apply if the current major mode is derived from MODE or
> +   MODE is bound and true as a variable.
> +
> + - Each RULE can be one of the following:
> +   - A symbol, say a face name.  It matches any font-lock keyword
> +     containing the symbol in its definition.  The symbol is
> +     interpreted as a glob pattern; in particular, `*' matches
> +     everything.
> +   - A string.  It matches any font-lock keyword defined by a regexp
> +     that matches the string.
> +   - A form (pred FUNCTION).  It matches if FUNCTION, which is called
> +     with the font-lock keyword as argument, returns non-nil.
> +   - A form (not RULE).  It matches if RULE doesn't.
> +   - A form (and RULE ...).  It matches if all the provided rules
> +     match.
> +   - A form (or RULE ...).  It matches if any of the provided rules
> +     match.
> +   - A form (except RULE ...).  This can be used only at top level or
> +     inside an `or' clause.  It undoes the effect of a previous
> +     matching rule.
> +
> +In each buffer, font lock keywords that match at least one
> +applicable rule are disabled."

Same comments apply to this doc string.

> +  :type '(alist :key-type symbol :value-type sexp)
> +  :group 'font-lock
> +  :version "29.1")
> +
>  (defcustom font-lock-verbose nil
>    "If non-nil, means show status messages for buffer fontification.
>  If a number, only buffers greater than this size have fontification messages."
> @@ -1810,9 +1847,8 @@ font-lock-compile-keywords
>        (error "Font-lock trying to use keywords before setting them up"))
>    (if (eq (car-safe keywords) t)
>        keywords
> -    (setq keywords
> -	  (cons t (cons keywords
> -			(mapcar #'font-lock-compile-keyword keywords))))
> +    (let ((compiled (mapcar #'font-lock-compile-keyword keywords)))
> +      (setq keywords `(t ,keywords ,@(font-lock--filter-keywords compiled))))
>      (if (and (not syntactic-keywords)
>  	     (let ((beg-function (with-no-warnings syntax-begin-function)))
>  	       (or (eq beg-function #'beginning-of-defun)
> @@ -1883,6 +1919,50 @@ font-lock-choose-keywords
>  	(t
>  	 (car keywords))))
>  
> +(defun font-lock--match-keyword (rule keyword)
> +  "Return non-nil if font-lock KEYWORD matches RULE.
> +See `font-lock-ignore' for the possible rules."
> +  (pcase-exhaustive rule
> +    ('* t)
> +    ((pred symbolp)
> +     (let ((regexp (when (string-match-p "[*?]" (symbol-name rule))
> +                     (wildcard-to-regexp (symbol-name rule)))))

So "[abcd]" isn't supported by these "glob patterns"?  This should be
documented.

I'm okay with clarifying the docs myself if you explain enough for me
to understand how to do that.

TIA

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Augusto Stoffel]

Hi Eli,

thanks for your feedback.  I've responded to your points below.

On Sat,  2 Apr 2022 at 09:36, Eli Zaretskii <eliz@gnu.org> wrote:

> Thanks for developing this feature.
>
> Reading its documentation, I think it should be clarified to be more
> useful.  See the questions/comments below:
>
>> +@example
>> +(@var{mode} @var{rule} @dots{})
>> +@end example
>> +
>> +Here, @var{mode} is a symbol, say a major or minor mode.
>
> Why "say"?  Can it usefully be something else? if so, what can it be?

This is easiest to explain with code.  A rule for a given xyz-mode
applies if

    (or (bound-and-true-p xyz-mode)
        (derived-mode-p xyz-mode))

Do you think it's better to just say “Here, @var{mode} a major or minor
mode”?  This would be slightly imprecise, because technically 'mode' can
be any variable.

>>                                                            The
>> +subsequent rules apply if the current major mode is derived from
>> +@var{mode} or @var{mode} is bound and true as a variable.  Each
>> +@var{rule} can be one of the following:
>> +
>> +@table @code
>> +@cindex @var{font-lock-ignore} rules
>> +@item @var{symbol}
>> +A symbol, say a face name, matches any Font Lock keyword containing
>> +the symbol in its definition.
>
> I don't think I understand how can a keyword "contain" a symbol.
> Please elaborate or suggest a different text that clarifies this.

Okay, as a general remark, this feature has been bolted on the existing
font lock mechanism, and it shows.  So the user has to interact to some
degree with the font-lock internals to make any meaningful use of it.

Specifically regarding your question, the user has to be aware that
'font-lock-keywords' is a list where (quoting the docstring):

    Each element [...] should have one of these forms:

     MATCHER
     (MATCHER . SUBEXP)
     (MATCHER . FACENAME)
     (MATCHER . HIGHLIGHT)
     (MATCHER HIGHLIGHT ...)
     (eval . FORM)

    where MATCHER can be either the regexp to search for, or the
    function name to call to make the search (called with one

So here I mean that 'symbol' equals the MATCHER, or the FACENAME, or is
a member of (flatten-tree FORM), and so on.

Unfortunately, I can't think of a better way to explain this.  Or, at
least, not in an annoyingly long-winded way.

(Coming back to the "bolted on" remark: If this feature had been planned
from the beginning, perhaps one would have forced the font-lock keyword
definitions to be named and/or include other metadata like "importance"
or "precedence".  But that is not the case, and the filtering options
provided here are IMO the best one can do at this point.)

>> The symbol is interpreted as a glob
>> +pattern; in particular, @code{*} matches everything.
>
> Do you mean shell glob patterns?  AFAIK, we don't use them in Emacs
> except for shell wildcards, so why are they used here? why not
> regexps?  And in any case, those patterns need a thorough description,
> since we don't use them elsewhere.  For example, the way to quote
> meta-characters such as '*' must be documented, because it isn't
> self-evident.

The intention is that the user can write, for instance

    (setq font-lock-ignore '((prog-mode font-lock-*-face)))

and this will be equivalent to

    (setq font-lock-ignore '((prog-mode (pred (lambda (obj)
                                                 (and (symbolp obj)
                                                      (string-match-p
                                                       "\\`font-lock-.*-face\\'"
                                                       (symbol-name obj))))))))

Do you agree this is a sensible and convenient mini-language?  I guess
Stefan liked the idea, at least :-).

>> +@item @var{string}
>> +A string matches any font-lock keyword defined by a regexp that
>> +matches the string.
>
> I don't think I understand this sentence.  In particular, saying "a
> regexp that matches the string" is at least unusual: we usually say it
> the other way around: "a string that matches the regexp".  And what
> does it mean for a keyword to be "defined by a regexp"?

Just to make sure we're on the same page, this refers, for instance, to
the rule

    (emacs-lisp-mode ";;;###autoload")

which would deactivate the highlighting of autoload cookies.

So, we could say "A string matches any font-lock rule which would
highlight that string".  It would be a slight lie, because this doesn't
work if the MATCHER is a function.  Should we trade precision for
clarity here?  I think we probably should.

>> +In each buffer, Font Lock keywords that match at least one applicable
>> +rule are disabled.
>
> This should be _before_ the list of rules, not after.  And given the
> fact that actions of the rules are implicit (i.e. they aren't present
> in the rules themselves), I would suggest to call them "conditions",
> not "rules".  A rule includes the action, not just the condition for
> that action, and those "rules" don't.

Okay, I'll move the text.  About rule vs. condition, let me think about
it.

>> +@smallexample
>> +(setq font-lock-ignore
>> +      '((prog-mode font-lock-*-face
>> +                   (except help-echo))
>> +        (emacs-lisp-mode (except ";;;###autoload)")
>> +        (whitespace-mode whitespace-empty-at-bob-regexp)
>> +        (makefile-mode (except *))))
>> +@end smallexample
>> +
>> +Line by line, this does the following:
>> +
>> +@enumerate
>> +@item
>> +In all programming modes, disable all font-lock keywords that apply
>> +one of the standard font-lock faces (excluding strings and comments,
>> +which are covered by syntactic Font Lock).
>> +
>> +@item
>> +However, keep any keywords that add a @code{help-echo} text property.
>> +
>> +@item
>> +In Emacs Lisp mode, also keep the highlighting of autoload cookies,
>> +which would have been excluded by rule 1.
>> +
>> +@item
>> +In @code{whitespace-mode} (a minor mode), don't highlight an empty
>> +line at beginning of buffer.
>
> Shouldn't you have "also" in this last description, like you did for
> Emacs Lisp mode?

Yes, or maybe "Additionally, if whitespace-mode (a minor-mode) is active,
don't highlight and empty line at the beginning of buffer."

> And why doesn't the example show all the possible forms of a "rule",
> but just one of them?

Well, I just wanted to keep true to this being a "@smallexample".  I
think it will be exceedingly rare that anyone needs any of the other
rules (and, pred, not, etc.).  It's so easy to implement those things
that it doesn't make sense not to, but mentioning them in the example
would perhaps distract from the main point.

>> +*** New user option 'font-lock-ignore'.
>> +This variable provides a mechanism to selectively disable font-lock
>> +keywords.
>
> I don't think it disables keywords, I think it disables
> fontifications.  (How can one disable a keyword?)

Hum, I'm not sure I understand the fine-grained distinction your are
making here.  font-lock.el speaks of "font lock keywords"; the thing
that fontifies variable names with font-lock-variable-face would be a
"font lock keyword" in that terminology.  I'm just following along the
terminology.

>> +
>> ++++
>>  *** New package vtable.el for formatting tabular data.
>>  This package allows formatting data using variable-pitch fonts.
>>  The resulting tables can display text in variable pitch fonts, text
>> diff --git a/lisp/font-lock.el b/lisp/font-lock.el
>> index d8a1fe3..8af3c30 100644
>> --- a/lisp/font-lock.el
>> +++ b/lisp/font-lock.el
>> @@ -208,6 +208,7 @@
>>  
>>  (require 'syntax)
>>  (eval-when-compile (require 'cl-lib))
>> +(eval-when-compile (require 'subr-x))
>>  
>>  ;; Define core `font-lock' group.
>>  (defgroup font-lock '((jit-lock custom-group))
>> @@ -279,6 +280,42 @@ font-lock-maximum-decoration
>>  				      (integer :tag "level" 1)))))
>>    :group 'font-lock)
>>  
>> +(defcustom font-lock-ignore nil
>> +  "Rules to selectively disable font-lock keywords.
>> +This is a list of rule sets of the form
>> +
>> +  (MODE RULE ...)
>> +
>> +where:
>> +
>> + - MODE is a symbol, say a major or minor mode.  The subsequent
>> +   rules apply if the current major mode is derived from MODE or
>> +   MODE is bound and true as a variable.
>> +
>> + - Each RULE can be one of the following:
>> +   - A symbol, say a face name.  It matches any font-lock keyword
>> +     containing the symbol in its definition.  The symbol is
>> +     interpreted as a glob pattern; in particular, `*' matches
>> +     everything.
>> +   - A string.  It matches any font-lock keyword defined by a regexp
>> +     that matches the string.
>> +   - A form (pred FUNCTION).  It matches if FUNCTION, which is called
>> +     with the font-lock keyword as argument, returns non-nil.
>> +   - A form (not RULE).  It matches if RULE doesn't.
>> +   - A form (and RULE ...).  It matches if all the provided rules
>> +     match.
>> +   - A form (or RULE ...).  It matches if any of the provided rules
>> +     match.
>> +   - A form (except RULE ...).  This can be used only at top level or
>> +     inside an `or' clause.  It undoes the effect of a previous
>> +     matching rule.
>> +
>> +In each buffer, font lock keywords that match at least one
>> +applicable rule are disabled."
>
> Same comments apply to this doc string.

Sure.

>> +  :type '(alist :key-type symbol :value-type sexp)
>> +  :group 'font-lock
>> +  :version "29.1")
>> +
>>  (defcustom font-lock-verbose nil
>>    "If non-nil, means show status messages for buffer fontification.
>>  If a number, only buffers greater than this size have fontification messages."
>> @@ -1810,9 +1847,8 @@ font-lock-compile-keywords
>>        (error "Font-lock trying to use keywords before setting them up"))
>>    (if (eq (car-safe keywords) t)
>>        keywords
>> -    (setq keywords
>> -	  (cons t (cons keywords
>> -			(mapcar #'font-lock-compile-keyword keywords))))
>> +    (let ((compiled (mapcar #'font-lock-compile-keyword keywords)))
>> +      (setq keywords `(t ,keywords ,@(font-lock--filter-keywords compiled))))
>>      (if (and (not syntactic-keywords)
>>  	     (let ((beg-function (with-no-warnings syntax-begin-function)))
>>  	       (or (eq beg-function #'beginning-of-defun)
>> @@ -1883,6 +1919,50 @@ font-lock-choose-keywords
>>  	(t
>>  	 (car keywords))))
>>  
>> +(defun font-lock--match-keyword (rule keyword)
>> +  "Return non-nil if font-lock KEYWORD matches RULE.
>> +See `font-lock-ignore' for the possible rules."
>> +  (pcase-exhaustive rule
>> +    ('* t)
>> +    ((pred symbolp)
>> +     (let ((regexp (when (string-match-p "[*?]" (symbol-name rule))
>> +                     (wildcard-to-regexp (symbol-name rule)))))
>
> So "[abcd]" isn't supported by these "glob patterns"?  This should be
> documented.

Ah, okay, I guess we should use the regexp "[][*?]" in that test.

> I'm okay with clarifying the docs myself if you explain enough for me
> to understand how to do that.

I guess yes, it would be great if we could have the perspective of a
different person incorporated in the documentation, so please go ahead!
I'll be glad to provide further clarifications.

> TIA

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
> Date: Sat, 02 Apr 2022 09:34:25 +0200
> 
> >> +@example
> >> +(@var{mode} @var{rule} @dots{})
> >> +@end example
> >> +
> >> +Here, @var{mode} is a symbol, say a major or minor mode.
> >
> > Why "say"?  Can it usefully be something else? if so, what can it be?
> 
> This is easiest to explain with code.  A rule for a given xyz-mode
> applies if
> 
>     (or (bound-and-true-p xyz-mode)
>         (derived-mode-p xyz-mode))
> 
> Do you think it's better to just say “Here, @var{mode} a major or minor
> mode”?  This would be slightly imprecise, because technically 'mode' can
> be any variable.

If MODE could be any symbol that is bound-and-true-p, does it mean
that just by defining a variable with a non-nil value will trigger
such a rule?

> >> +subsequent rules apply if the current major mode is derived from
> >> +@var{mode} or @var{mode} is bound and true as a variable.  Each
> >> +@var{rule} can be one of the following:
> >> +
> >> +@table @code
> >> +@cindex @var{font-lock-ignore} rules
> >> +@item @var{symbol}
> >> +A symbol, say a face name, matches any Font Lock keyword containing
> >> +the symbol in its definition.
> >
> > I don't think I understand how can a keyword "contain" a symbol.
> > Please elaborate or suggest a different text that clarifies this.
> 
> Okay, as a general remark, this feature has been bolted on the existing
> font lock mechanism, and it shows.  So the user has to interact to some
> degree with the font-lock internals to make any meaningful use of it.
> 
> Specifically regarding your question, the user has to be aware that
> 'font-lock-keywords' is a list where (quoting the docstring):
> 
>     Each element [...] should have one of these forms:
> 
>      MATCHER
>      (MATCHER . SUBEXP)
>      (MATCHER . FACENAME)
>      (MATCHER . HIGHLIGHT)
>      (MATCHER HIGHLIGHT ...)
>      (eval . FORM)
> 
>     where MATCHER can be either the regexp to search for, or the
>     function name to call to make the search (called with one
> 
> So here I mean that 'symbol' equals the MATCHER, or the FACENAME, or is
> a member of (flatten-tree FORM), and so on.

It can equal _any_ symbol in those places?  Even stuff like 'and'
etc.?  What a strange feature!  Why not enable just a test against the
MATCHER part?

> >> The symbol is interpreted as a glob
> >> +pattern; in particular, @code{*} matches everything.
> >
> > Do you mean shell glob patterns?  AFAIK, we don't use them in Emacs
> > except for shell wildcards, so why are they used here? why not
> > regexps?  And in any case, those patterns need a thorough description,
> > since we don't use them elsewhere.  For example, the way to quote
> > meta-characters such as '*' must be documented, because it isn't
> > self-evident.
> 
> The intention is that the user can write, for instance
> 
>     (setq font-lock-ignore '((prog-mode font-lock-*-face)))
> 
> and this will be equivalent to
> 
>     (setq font-lock-ignore '((prog-mode (pred (lambda (obj)
>                                                  (and (symbolp obj)
>                                                       (string-match-p
>                                                        "\\`font-lock-.*-face\\'"
>                                                        (symbol-name obj))))))))
> 
> Do you agree this is a sensible and convenient mini-language?  I guess
> Stefan liked the idea, at least :-).

I understand the intent.  I'm saying that we should document this
"mini-language" in sufficient detail, so that Lisp programmers knew
how to use it.

> >> +@item @var{string}
> >> +A string matches any font-lock keyword defined by a regexp that
> >> +matches the string.
> >
> > I don't think I understand this sentence.  In particular, saying "a
> > regexp that matches the string" is at least unusual: we usually say it
> > the other way around: "a string that matches the regexp".  And what
> > does it mean for a keyword to be "defined by a regexp"?
> 
> Just to make sure we're on the same page, this refers, for instance, to
> the rule
> 
>     (emacs-lisp-mode ";;;###autoload")
> 
> which would deactivate the highlighting of autoload cookies.
> 
> So, we could say "A string matches any font-lock rule which would
> highlight that string".

What do you mean by "font-lock rule" here?

> > And why doesn't the example show all the possible forms of a "rule",
> > but just one of them?
> 
> Well, I just wanted to keep true to this being a "@smallexample".

I see no problem with this: adding a couple of lines shouldn't make
the example too large.

> >> +*** New user option 'font-lock-ignore'.
> >> +This variable provides a mechanism to selectively disable font-lock
> >> +keywords.
> >
> > I don't think it disables keywords, I think it disables
> > fontifications.  (How can one disable a keyword?)
> 
> Hum, I'm not sure I understand the fine-grained distinction your are
> making here.  font-lock.el speaks of "font lock keywords"; the thing
> that fontifies variable names with font-lock-variable-face would be a
> "font lock keyword" in that terminology.  I'm just following along the
> terminology.

That terminology (if this is how you perceive it) is wrong, and I'm
about to change it to make that more clear.  font-lock-keywords are
not just keywords, they are rules for fontification.  This new feature
allows to selectively disable some of those fontifications without
editing the rules themselves.

> >> +(defun font-lock--match-keyword (rule keyword)
> >> +  "Return non-nil if font-lock KEYWORD matches RULE.
> >> +See `font-lock-ignore' for the possible rules."
> >> +  (pcase-exhaustive rule
> >> +    ('* t)
> >> +    ((pred symbolp)
> >> +     (let ((regexp (when (string-match-p "[*?]" (symbol-name rule))
> >> +                     (wildcard-to-regexp (symbol-name rule)))))
> >
> > So "[abcd]" isn't supported by these "glob patterns"?  This should be
> > documented.
> 
> Ah, okay, I guess we should use the regexp "[][*?]" in that test.

Why not allow a regexp to begin with?  Then 2 problems will be solved
in one blow: (1) the need to change the code, and (2) the need to
describe what is allowed and what isn't.

> > I'm okay with clarifying the docs myself if you explain enough for me
> > to understand how to do that.
> 
> I guess yes, it would be great if we could have the perspective of a
> different person incorporated in the documentation, so please go ahead!
> I'll be glad to provide further clarifications.

Will do, after you reply to this followup message.

Thanks.

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Augusto Stoffel]

On Sat,  2 Apr 2022 at 13:01, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Augusto Stoffel <arstoffel@gmail.com>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
>> Date: Sat, 02 Apr 2022 09:34:25 +0200
>> 
>> >> +@example
>> >> +(@var{mode} @var{rule} @dots{})
>> >> +@end example
>> >> +
>> >> +Here, @var{mode} is a symbol, say a major or minor mode.
>> >
>> > Why "say"?  Can it usefully be something else? if so, what can it be?
>> 
>> This is easiest to explain with code.  A rule for a given xyz-mode
>> applies if
>> 
>>     (or (bound-and-true-p xyz-mode)
>>         (derived-mode-p xyz-mode))
>> 
>> Do you think it's better to just say “Here, @var{mode} a major or minor
>> mode”?  This would be slightly imprecise, because technically 'mode' can
>> be any variable.
>
> If MODE could be any symbol that is bound-and-true-p, does it mean
> that just by defining a variable with a non-nil value will trigger
> such a rule?

That is correct.  Of course there could be a conflict if a major mode
name is also bound as a variable.  I haven't seen this happen in nature.
However, if you have a better way to test whether a given symbol is a
minor mode and is on, we should definitely use it.

>> >> +subsequent rules apply if the current major mode is derived from
>> >> +@var{mode} or @var{mode} is bound and true as a variable.  Each
>> >> +@var{rule} can be one of the following:
>> >> +
>> >> +@table @code
>> >> +@cindex @var{font-lock-ignore} rules
>> >> +@item @var{symbol}
>> >> +A symbol, say a face name, matches any Font Lock keyword containing
>> >> +the symbol in its definition.
>> >
>> > I don't think I understand how can a keyword "contain" a symbol.
>> > Please elaborate or suggest a different text that clarifies this.
>> 
>> Okay, as a general remark, this feature has been bolted on the existing
>> font lock mechanism, and it shows.  So the user has to interact to some
>> degree with the font-lock internals to make any meaningful use of it.
>> 
>> Specifically regarding your question, the user has to be aware that
>> 'font-lock-keywords' is a list where (quoting the docstring):
>> 
>>     Each element [...] should have one of these forms:
>> 
>>      MATCHER
>>      (MATCHER . SUBEXP)
>>      (MATCHER . FACENAME)
>>      (MATCHER . HIGHLIGHT)
>>      (MATCHER HIGHLIGHT ...)
>>      (eval . FORM)
>> 
>>     where MATCHER can be either the regexp to search for, or the
>>     function name to call to make the search (called with one
>> 
>> So here I mean that 'symbol' equals the MATCHER, or the FACENAME, or is
>> a member of (flatten-tree FORM), and so on.
>
> It can equal _any_ symbol in those places?  Even stuff like 'and'
> etc.?  What a strange feature!  Why not enable just a test against the
> MATCHER part?

The config example in the manual is not an academic one.  I personally
want to disable _most_ font-locking, therefore I use this as my very
first font-lock-ignore rule:

    (prog-mode font-lock-*-face)

But I will gladly take some warning fontifications if someone went the
trouble of implementing them.  So I'm using this rule as well:

   (prog-mode (except help-echo))

Does that catch only fontifications I care about, and not catch any
fontifications I don't care about?  Probably not, but it's a good enough
approximation.  It achieves something I find pretty good using only 1
very succinct heuristic.

Granted, 'and' is not a meaningful RULE.  But it's not disallowed
because there's not point in disallowing it.

>> >> The symbol is interpreted as a glob
>> >> +pattern; in particular, @code{*} matches everything.
>> >
>> > Do you mean shell glob patterns?  AFAIK, we don't use them in Emacs
>> > except for shell wildcards, so why are they used here? why not
>> > regexps?  And in any case, those patterns need a thorough description,
>> > since we don't use them elsewhere.  For example, the way to quote
>> > meta-characters such as '*' must be documented, because it isn't
>> > self-evident.
>> 
>> The intention is that the user can write, for instance
>> 
>>     (setq font-lock-ignore '((prog-mode font-lock-*-face)))
>> 
>> and this will be equivalent to
>> 
>>     (setq font-lock-ignore '((prog-mode (pred (lambda (obj)
>>                                                  (and (symbolp obj)
>>                                                       (string-match-p
>>                                                        "\\`font-lock-.*-face\\'"
>>                                                        (symbol-name obj))))))))
>> 
>> Do you agree this is a sensible and convenient mini-language?  I guess
>> Stefan liked the idea, at least :-).
>
> I understand the intent.  I'm saying that we should document this
> "mini-language" in sufficient detail, so that Lisp programmers knew
> how to use it.

Strictly speaking the docstring of 'font-lock-ignore' provides a
complete specification of the mini-language; if not, it's a bug in the
docstring.  But we can always expand the examples, of course.

>> >> +@item @var{string}
>> >> +A string matches any font-lock keyword defined by a regexp that
>> >> +matches the string.
>> >
>> > I don't think I understand this sentence.  In particular, saying "a
>> > regexp that matches the string" is at least unusual: we usually say it
>> > the other way around: "a string that matches the regexp".  And what
>> > does it mean for a keyword to be "defined by a regexp"?
>> 
>> Just to make sure we're on the same page, this refers, for instance, to
>> the rule
>> 
>>     (emacs-lisp-mode ";;;###autoload")
>> 
>> which would deactivate the highlighting of autoload cookies.
>> 
>> So, we could say "A string matches any font-lock rule which would
>> highlight that string".
>
> What do you mean by "font-lock rule" here?

Yeah, I meant "entry of font-lock-keywords".

>> > And why doesn't the example show all the possible forms of a "rule",
>> > but just one of them?
>> 
>> Well, I just wanted to keep true to this being a "@smallexample".
>
> I see no problem with this: adding a couple of lines shouldn't make
> the example too large.

Sounds good.

>> >> +*** New user option 'font-lock-ignore'.
>> >> +This variable provides a mechanism to selectively disable font-lock
>> >> +keywords.
>> >
>> > I don't think it disables keywords, I think it disables
>> > fontifications.  (How can one disable a keyword?)
>> 
>> Hum, I'm not sure I understand the fine-grained distinction your are
>> making here.  font-lock.el speaks of "font lock keywords"; the thing
>> that fontifies variable names with font-lock-variable-face would be a
>> "font lock keyword" in that terminology.  I'm just following along the
>> terminology.
>
> That terminology (if this is how you perceive it) is wrong, and I'm
> about to change it to make that more clear.  font-lock-keywords are
> not just keywords, they are rules for fontification.

I certainly agree that font-lock.el's use of "font-lock keyword" as a
term for "rules for fontification" is a bit odd.  OTOH it's an
established use.  Anyway, any changes that make things clearer are
welcome.

>  This new feature allows to selectively disable some of those
> fontifications without editing the rules themselves.

That's correct.  I see this as an end-user configuration, so providing
the ability to modify the fontifications would probably be going too
far.  If someone can come up with an expressive mini-language to modify
fontifications, that might be cool.  Other than that, the user can
always modify 'font-lock-keywords' manually.

>> >> +(defun font-lock--match-keyword (rule keyword)
>> >> +  "Return non-nil if font-lock KEYWORD matches RULE.
>> >> +See `font-lock-ignore' for the possible rules."
>> >> +  (pcase-exhaustive rule
>> >> +    ('* t)
>> >> +    ((pred symbolp)
>> >> +     (let ((regexp (when (string-match-p "[*?]" (symbol-name rule))
>> >> +                     (wildcard-to-regexp (symbol-name rule)))))
>> >
>> > So "[abcd]" isn't supported by these "glob patterns"?  This should be
>> > documented.
>> 
>> Ah, okay, I guess we should use the regexp "[][*?]" in that test.
>
> Why not allow a regexp to begin with?  Then 2 problems will be solved
> in one blow: (1) the need to change the code, and (2) the need to
> describe what is allowed and what isn't.

So your suggestion is that when RULE is a string, the string is treated
as a regexp and symbol names are matched against it, right?

IMO the current mini-langauge is more useful: the RULE "(defun" can be
used to match the following element of font-lock-keywords found in
emacs-lisp-mode:

```
("(\\(cl-def\\(?:generic\\|m\\(?:acro\\|ethod\\)\\|s\\(?:\\(?:truc\\|ubs\\)t\\)\\|type\\|un\\)\\|def\\(?:a\\(?:dvice\\|lias\\)\\|c\\(?:lass\\|onst\\|ustom\\)\\|face\\|g\\(?:eneric\\|roup\\)\\|ine-\\(?:advice\\|derived-mode\\|g\\(?:\\(?:eneric\\|lobal\\(?:\\(?:ized\\)?-minor\\)\\)-mode\\)\\|inline\\|minor-mode\\|skeleton\\|widget\\)\\|m\\(?:acro\\|ethod\\)\\|subst\\|theme\\|un\\|var\\(?:-local\\|alias\\)?\\)\\|ert-deftest\\)\\_>[ 	']*\\(([ 	']*\\)?\\(\\(setf\\)[ 	]+\\(?:\\sw\\|\\s_\\|\\\\.\\)+\\|\\(?:\\sw\\|\\s_\\|\\\\.\\)+\\)?"
 (1 font-lock-keyword-face)
 (3
  (let
      ((type
        (get
         (intern-soft
          (match-string 1))
         'lisp-define-type)))
    (cond
     ((eq type 'var)
      font-lock-variable-name-face)
     ((eq type 'type)
      font-lock-type-face)
     ((or
       (not
        (match-string 2))
       (and
        (match-string 2)
        (match-string 4)))
      font-lock-function-name-face)))
  nil t))
```

>> > I'm okay with clarifying the docs myself if you explain enough for me
>> > to understand how to do that.
>> 
>> I guess yes, it would be great if we could have the perspective of a
>> different person incorporated in the documentation, so please go ahead!
>> I'll be glad to provide further clarifications.
>
> Will do, after you reply to this followup message.

Great, thanks!

> Thanks.

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 02 Apr 2022 13:18:40 +0200
> 
> >> The intention is that the user can write, for instance
> >> 
> >>     (setq font-lock-ignore '((prog-mode font-lock-*-face)))
> >> 
> >> and this will be equivalent to
> >> 
> >>     (setq font-lock-ignore '((prog-mode (pred (lambda (obj)
> >>                                                  (and (symbolp obj)
> >>                                                       (string-match-p
> >>                                                        "\\`font-lock-.*-face\\'"
> >>                                                        (symbol-name obj))))))))
> >> 
> >> Do you agree this is a sensible and convenient mini-language?  I guess
> >> Stefan liked the idea, at least :-).
> >
> > I understand the intent.  I'm saying that we should document this
> > "mini-language" in sufficient detail, so that Lisp programmers knew
> > how to use it.
> 
> Strictly speaking the docstring of 'font-lock-ignore' provides a
> complete specification of the mini-language; if not, it's a bug in the
> docstring.

So you want to support only the "*" wildcard and nothing else?

> >> Hum, I'm not sure I understand the fine-grained distinction your are
> >> making here.  font-lock.el speaks of "font lock keywords"; the thing
> >> that fontifies variable names with font-lock-variable-face would be a
> >> "font lock keyword" in that terminology.  I'm just following along the
> >> terminology.
> >
> > That terminology (if this is how you perceive it) is wrong, and I'm
> > about to change it to make that more clear.  font-lock-keywords are
> > not just keywords, they are rules for fontification.
> 
> I certainly agree that font-lock.el's use of "font-lock keyword" as a
> term for "rules for fontification" is a bit odd.  OTOH it's an
> established use.

That's a historic accident, AFAICT: originally we didn't support there
anything but keywords.

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Augusto Stoffel]

On Sat,  2 Apr 2022 at 14:58, Eli Zaretskii <eliz@gnu.org> wrote:

>> Strictly speaking the docstring of 'font-lock-ignore' provides a
>> complete specification of the mini-language; if not, it's a bug in the
>> docstring.
>
> So you want to support only the "*" wildcard and nothing else?

Let's support all POSIX wildcards, since we have an implementation for
that at hand.  I think the docstring is complete, if terse:

   - A symbol, say a face name.  It matches any font-lock keyword
     containing the symbol in its definition.  The symbol is
     interpreted as a glob pattern; in particular, ‘*’ matches
     everything.

>> I certainly agree that font-lock.el's use of "font-lock keyword" as a
>> term for "rules for fontification" is a bit odd.  OTOH it's an
>> established use.
>
> That's a historic accident, AFAICT: originally we didn't support there
> anything but keywords.

I don't know the history but that's what I have been presuming.  I don't
have a strong opinion about whether the historic accident can or should
be corrected at this point.

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Eli Zaretskii]

I made changes to the documentation, please have a look.

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Stefan Monnier]

>> The symbol is interpreted as a glob
>> +pattern; in particular, @code{*} matches everything.
> Do you mean shell glob patterns?  AFAIK, we don't use them in Emacs
> except for shell wildcards, so why are they used here?

A bit of history: originally his code only supported `*` and I suggested
the use of `wildcard-to-regexp` to both simplify the code, be more
general, and simplify the doc since we can just say "glob pattern" which
is a known concept.

> why not regexps?

I did not suggest the use of regexps because I felt that regexps written
as symbols could be problematic (especially the `.` being a second-class
citizen among symbol constituents in ELisp).  Admittedly, in practice
`.` works fine (i.e. without backslash-escaping) in most places in
a symbol so maybe it wouldn't be that bad.

My gut tells me that globs are preferable here (slightly less verbose
and slightly less odd):

    (typer-mode foo-*-baz)

vs

    (typer-mode \\\`foo-.*-baz\\\')

But either way is fine by me (we can make the \\\` anchoring \\\' implicit).

> And in any case, those patterns need a thorough description,
> since we don't use them elsewhere.  For example, the way to quote
> meta-characters such as '*' must be documented, because it isn't
> self-evident.

AFAIK glob patterns come with their own standard way to escape,
e.g. `foo[*]bar`, tho admittedly to make it into a single symbol you need
to write it as `foo\[*\]bar` :-(


        Stefan

RE: [External] : Re: master 5c70ff9: New user option 'font-lock-ignore'

[Drew Adams]

[I haven't found the beginning of this thread,
even by searching the mailing list archive...]

FWIW, I proposed a related feature back in 2007,
and again in 2014 (both here and with a patch in
bug #18367 - see links below).

It lets you use text-property `font-lock-ignore'
(same name - surprise! ;-)) to make font-lock
ignore text with that property - "Hands off!".

This means, in particular, that font lock won't
erase or otherwise interfere with non-font-lock highlighting that you apply.  You can highlight
and unhighlight text independently (or not) of
`font-lock-mode'.
___

The code is here:

https://www.emacswiki.org/emacs/download/font-lock%2b.el

I've been using it since 2007.  It's used, e.g.,
in my libraries `highlight.el', Bookmark+, and 
`facemenu+.el'.
___

And before people reply with "Use `font-lock-face'
or `font-lock-extra-managed-props'": No - they
don't do the job at all, as was discussed here
at the time.

Quite the contrary: they just make `font-lock'
_also_ manage the text with the given property;
e.g., turning off `font-lock-mode' turns off any highlighting with that property.  The point is
not to give font-lock more control; it's to be
able to _remove_ control by font-lock from given
text.

See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18367#38
___

2014 emacs-devel thread:

https://lists.gnu.org/archive/html/emacs-devel/2014-08/msg00540.html

https://lists.gnu.org/archive/html/emacs-devel/2014-08/msg00583.html

2007 emacs-devel thread:

https://lists.gnu.org/archive/html/emacs-devel/2007-03/msg01459.html

Bug #18367 thread:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18367

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Augusto Stoffel]

On Sat,  2 Apr 2022 at 16:52, Eli Zaretskii <eliz@gnu.org> wrote:

> I made changes to the documentation, please have a look.

Looks good to me.  I just have one remark to make:

@@ -3337,23 +3350,23 @@ Line by line, this does the following:
 
 @enumerate
 @item
-In all programming modes, disable all font-lock keywords that apply
-one of the standard font-lock faces (excluding strings and comments,
-which are covered by syntactic Font Lock).
+In all programming modes, disable fontifications due to all font-lock
+keywords that apply one of the standard font-lock faces (excluding
 ^^^^^^^^
Here, we got one stray use of "keyword" in the funny sense of
font-lock.el


Thanks!

Re: [External] : Re: master 5c70ff9: New user option 'font-lock-ignore'

[Augusto Stoffel]

On Sat,  2 Apr 2022 at 15:28, Drew Adams <drew.adams@oracle.com> wrote:

> [I haven't found the beginning of this thread,
> even by searching the mailing list archive...]
>
> FWIW, I proposed a related feature back in 2007,
> and again in 2014 (both here and with a patch in
> bug #18367 - see links below).
>
> It lets you use text-property `font-lock-ignore'
> (same name - surprise! ;-)) to make font-lock
> ignore text with that property - "Hands off!".

From what I can tell this is only vaguely related.  The
'font-lock-ignore' user option is really meant for the end-user, and it
allows to remove some fontification rules that might hurt his or her
sensibilities.

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 02 Apr 2022 18:25:23 +0200
> 
> On Sat,  2 Apr 2022 at 16:52, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I made changes to the documentation, please have a look.
> 
> Looks good to me.  I just have one remark to make:
> 
> @@ -3337,23 +3350,23 @@ Line by line, this does the following:
>  
>  @enumerate
>  @item
> -In all programming modes, disable all font-lock keywords that apply
> -one of the standard font-lock faces (excluding strings and comments,
> -which are covered by syntactic Font Lock).
> +In all programming modes, disable fontifications due to all font-lock
> +keywords that apply one of the standard font-lock faces (excluding
>  ^^^^^^^^
> Here, we got one stray use of "keyword" in the funny sense of
> font-lock.el

I don't see anything wrong with the new text.  What would you use
instead that is more clear?

Re: master 5c70ff9: New user option 'font-lock-ignore'

[Augusto Stoffel]

On Sat,  2 Apr 2022 at 19:44, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Augusto Stoffel <arstoffel@gmail.com>
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sat, 02 Apr 2022 18:25:23 +0200
>> 
>> On Sat,  2 Apr 2022 at 16:52, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> > I made changes to the documentation, please have a look.
>> 
>> Looks good to me.  I just have one remark to make:
>> 
>> @@ -3337,23 +3350,23 @@ Line by line, this does the following:
>>  
>>  @enumerate
>>  @item
>> -In all programming modes, disable all font-lock keywords that apply
>> -one of the standard font-lock faces (excluding strings and comments,
>> -which are covered by syntactic Font Lock).
>> +In all programming modes, disable fontifications due to all font-lock
>> +keywords that apply one of the standard font-lock faces (excluding
>>  ^^^^^^^^
>> Here, we got one stray use of "keyword" in the funny sense of
>> font-lock.el
>
> I don't see anything wrong with the new text.  What would you use
> instead that is more clear?

One could say "In all programming modes, disable fontification rules
that apply one of the standard font-lock faces".

But what you wrote is a bit more specific, and the "font-lock keyword"
terminology is still used elsewhere.  So I guess I'll take back my
remark :-).

RE: [External] : Re: master 5c70ff9: New user option 'font-lock-ignore'

[Drew Adams]

> > FWIW, I proposed a related feature back in 2007,
> > and again in 2014 (both here and with a patch in
> > bug #18367 - see links below).
> >
> > It lets you use text-property `font-lock-ignore'
> > (same name - surprise! ;-)) to make font-lock
> > ignore text with that property - "Hands off!".
> 
> From what I can tell this is only vaguely related.  The
> 'font-lock-ignore' user option is really meant for the
> end-user, and it allows to remove some fontification
> rules that might hurt his or her sensibilities.

If you say so.  You also said:

  the user has to interact to some degree with the
  font-lock internals to make any meaningful use of it.

Your "end user" needs to know which font-lock rules
are causing the bother, in order to inactivate them.

To make use of what I proposed, a user doesn't need
to interact with any font-lock internals.  So yeah,
the two features are not related in that sense.

They're related in that each lets you tell font-lock
not to manage particular bits of text.  (IIUC)

Your feature (IIUC) lets you do that by inhibiting
particular font-lock rules.  So you need to identify
which rules are bothering you and know how to turn
only those off (and you have to do that with a user
option, for individual modes or sets of modes).

My feature lets you do it anywhere, anytime, in any
mode/context, by specifying the bits of text any way
you want.  And it applies to any and all font-lock
rules whatsoever.  It doesn't care, and you needn't
know, how or why font-lock is bothering that text.
It directly prevents font-lock from doing that.

As I see it, both features can be useful.

Your feature wouldn't help for the uses I make of
my feature.  Libraries that use my feature can, and
do, remain 100% ignorant of what font-lock tries to
do to the text in question - how or why it messes
with it.  They just stop it from doing so by making
that text off limits to font-lock.

My feature is particularly useful for ad hoc
highlighting that you want to be independent of
font-lock: not overridden by font-lock highlighting,
and not affected by turning font-lock off/on.
___

As a demonstration of direct "end-user" use, I just
added a simple command to toggle font-lock for text
you select with the mouse.  E.g., bind it to, say,
`M-down-mouse-2' or `C-x down-mouse-1', then select
some text to toggle font-lock on/off only there.

It doesn't matter how you apply text property
`font-lock-ignore', whether by code or interactively
in some way.  It's simple to define commands that
handle particular text (e.g. prompt for a regexp to
match, or apply to the active region).

https://www.emacswiki.org/emacs/download/font-lock%2b.el