minor mode for highlighting character classes such as non-ascii (bug 47455)

minor mode for highlighting character classes such as non-ascii (bug 47455)

[Roland Winkler]

In the context of bug 47455 the idea emerged that it can be useful
to have a minor mode that highlights non-ascii characters.  I want
to ask hear for feedback: does such a mode already exist?  Can it be
useful to have this minor mode more general such that the highlighted
character classes become configurable?  Are there possible use cases
for this beyond non-ascii characters?

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Eli Zaretskii]

> Date: Tue, 1 Jun 2021 11:16:53 -0500
> From: "Roland Winkler" <winkler@gnu.org>
> 
> In the context of bug 47455 the idea emerged that it can be useful
> to have a minor mode that highlights non-ascii characters.  I want
> to ask hear for feedback: does such a mode already exist?  Can it be
> useful to have this minor mode more general such that the highlighted
> character classes become configurable?  Are there possible use cases
> for this beyond non-ascii characters?

We already highlight some potentially confusing characters, see
no-break-char-display and faces nobreak-space and nobreak-hyphen.

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Óscar Fuentes]

"Roland Winkler" <winkler@gnu.org> writes:

> In the context of bug 47455 the idea emerged that it can be useful
> to have a minor mode that highlights non-ascii characters.  I want
> to ask hear for feedback: does such a mode already exist?  Can it be
> useful to have this minor mode more general such that the highlighted
> character classes become configurable?  Are there possible use cases
> for this beyond non-ascii characters?

Highlighting characters that does not belong to certain coding system
would be useful to me.

Sometimes I work on source code encoded as UTF-8 but they must not
contain characters not encodable by windows-1252. More precisely, string
literals must be composed of characters encodable by windows-1252.

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 01 Jun 2021 18:32:59 +0200
> 
> "Roland Winkler" <winkler@gnu.org> writes:
> 
> > In the context of bug 47455 the idea emerged that it can be useful
> > to have a minor mode that highlights non-ascii characters.  I want
> > to ask hear for feedback: does such a mode already exist?  Can it be
> > useful to have this minor mode more general such that the highlighted
> > character classes become configurable?  Are there possible use cases
> > for this beyond non-ascii characters?
> 
> Highlighting characters that does not belong to certain coding system
> would be useful to me.

Character cannot belong to a coding system.  I think you mean
"characters that cannot be safely encoded by a coding system".

> Sometimes I work on source code encoded as UTF-8 but they must not
> contain characters not encodable by windows-1252. More precisely, string
> literals must be composed of characters encodable by windows-1252.

We already make such tests when you save the buffer, so the code to do
that exists, and can be reused.  But I'm not sure this highlighting
couldn't turn out to be somewhat expensive.

RE: [External] : Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Drew Adams]

> > In the context of bug 47455 the idea emerged that it can be useful
> > to have a minor mode that highlights non-ascii characters.  I want
> > to ask hear for feedback: does such a mode already exist?  Can it be
> > useful to have this minor mode more general such that the highlighted
> > character classes become configurable?  Are there possible use cases
> > for this beyond non-ascii characters?

Eli> We already highlight some potentially confusing characters,
Eli> see no-break-char-display and faces nobreak-space and
Eli> nobreak-hyphen.

OF> Highlighting characters that does not belong to certain
OF> coding system would be useful to me.
OF>
OF> Sometimes I work on source code encoded as UTF-8 but they
OF> must not contain characters not encodable by windows-1252.
OF> More precisely, string literals must be composed of
OF> characters encodable by windows-1252.

That highlighting of no-break chars is limited.

1. You can't specify which no-break chars you want to highlight,
   and you can't highlight different such chars differently.

2. It is low-level highlighting, which doesn't use font-lock.

`whitespace.el' is similarly limited, e.g. coupling space and
hard-space chars, for highlighting.

More useful is something like what library `highlight-chars.el'
offers.  You can highlight any chars you like, in any way you
like.  You can highlight arbitrary chars, chars in char ranges,
char classes, or charsets.

https://www.emacswiki.org/emacs/ShowWhiteSpace#HighlightChars

https://www.emacswiki.org/emacs/download/highlight-chars.el

Re: [External] : Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> Date: Tue, 1 Jun 2021 16:49:21 +0000
> 
> More useful is something like what library `highlight-chars.el'
> offers.  You can highlight any chars you like, in any way you
> like.  You can highlight arbitrary chars, chars in char ranges,
> char classes, or charsets.

You mean, like highlight-regexp?

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Daniel Martín]

"Roland Winkler" <winkler@gnu.org> writes:

> In the context of bug 47455 the idea emerged that it can be useful
> to have a minor mode that highlights non-ascii characters.  I want
> to ask hear for feedback: does such a mode already exist?  Can it be
> useful to have this minor mode more general such that the highlighted
> character classes become configurable?  Are there possible use cases
> for this beyond non-ascii characters?

There's already a feature in the display engine that shows certain
non-ASCII characters with a special face.  For example, U+00A0 (no-break
space) is shown with the ‘nobreak-space’ face, U+00AD (soft hyphen),
‘U+2010’ (hyphen), and ‘U+2011’ (non-breaking hyphen) are shown with the
‘nobreak-hyphen’ face.  Perhaps your suggestion could be a
backwards-compatible extension to this hardcoded display mechanism.

Another option is to simply instruct users to use Hi Lock mode with
"[^[:ascii:]]".

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Roland Winkler]

On Tue Jun 1 2021 Daniel Martín wrote:
> There's already a feature in the display engine that shows certain
> non-ASCII characters with a special face.  For example, U+00A0 (no-break
> space) is shown with the ‘nobreak-space’ face, U+00AD (soft hyphen),
> ‘U+2010’ (hyphen), and ‘U+2011’ (non-breaking hyphen) are shown with the
> ‘nobreak-hyphen’ face.  Perhaps your suggestion could be a
> backwards-compatible extension to this hardcoded display mechanism.
> 
> Another option is to simply instruct users to use Hi Lock mode with
> "[^[:ascii:]]".

Thanks, Hi Lock mode seems to be already a close match.  I'll look
at this more closely.

The bug report is related to the fact that \(Bib\|La\)?TeX chokes on
any non-ascii characters.  Nonetheless, non-ascii characters can end
up in such files for all kinds of reasons.  So the idea is to have
something like a minor mode that highlights any non-ascii characters
so that this effectively warns the user about the presence of such
characters.

Personally, I have been fooled in particular by unicode 'ZERO WIDTH
SPACE' which probably requires special treatment.  Could it also
make sense to give this particular character a (possibly configurable)
treatment by the display engine?  (I know nothing about the display
engine.)

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Eli Zaretskii]

> Date: Tue, 1 Jun 2021 12:34:56 -0500
> From: "Roland Winkler" <winkler@gnu.org>
> Cc: emacs-devel@gnu.org, Eli Zaretskii <eliz@gnu.org>
> 
> Personally, I have been fooled in particular by unicode 'ZERO WIDTH
> SPACE' which probably requires special treatment.  Could it also
> make sense to give this particular character a (possibly configurable)
> treatment by the display engine?

It already does, see glyphless-char-display.  You can customize it to
your needs.

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Roland Winkler]

On Tue Jun 1 2021 Eli Zaretskii wrote:
> > Personally, I have been fooled in particular by unicode 'ZERO
> > WIDTH SPACE' which probably requires special treatment.  Could
> > it also make sense to give this particular character a (possibly
> > configurable) treatment by the display engine?
> 
> It already does, see glyphless-char-display.  You can customize it
> to your needs.

Excellent, I knew I may be reinventing the wheel with these ideas.

RE: [External] : Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Drew Adams]

> > More useful is something like what library `highlight-chars.el'
> > offers.  You can highlight any chars you like, in any way you
> > like.  You can highlight arbitrary chars, chars in char ranges,
> > char classes, or charsets.
> 
> You mean, like highlight-regexp?

No.  I mean like what I linked to.  There you'll
find a succinct description.  I think you'll be
able to see a real difference from
`highlight-regexp' (`hi-lock-face-buffer').

https://www.emacswiki.org/emacs/ShowWhiteSpace*HighlightChars

And the Commentary in highlight-chars.el offers
quite a bit more detail.

https://www.emacswiki.org/emacs/download/highlight-chars.el

But yes, hi-lock.el offers some improvement over
the hard-coded behavior of `nobreak-char-display'
and `whitespace-mode'.

Re: [External] : Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "ofv@wanadoo.es" <ofv@wanadoo.es>,
>         "emacs-devel@gnu.org"
> 	<emacs-devel@gnu.org>
> Date: Tue, 1 Jun 2021 18:07:33 +0000
> 
> But yes, hi-lock.el offers some improvement over
> the hard-coded behavior of `nobreak-char-display'
> and `whitespace-mode'.

How do you write a regexp or a character class that matches characters
which cannot be encoded by a given coding-system?

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Juri Linkov]

> In the context of bug 47455 the idea emerged that it can be useful
> to have a minor mode that highlights non-ascii characters.  I want
> to ask hear for feedback: does such a mode already exist?  Can it be
> useful to have this minor mode more general such that the highlighted
> character classes become configurable?  Are there possible use cases
> for this beyond non-ascii characters?

The GNU ELPA package 'markchars' has options to highlight
non-ASCII space chars, non-ASCII hyphen chars,
and other customizable patterns, e.g. "[[:nonascii:]]+".

Re: minor mode for highlighting character classes such as non-ascii (bug 47455)

[Roland Winkler]

On Tue Jun 1 2021 Juri Linkov wrote:
> The GNU ELPA package 'markchars' has options to highlight
> non-ASCII space chars, non-ASCII hyphen chars,
> and other customizable patterns, e.g. "[[:nonascii:]]+".

Wonderful, thanks for the feedback!  It seems that the only thing
that's missing is the proper advertisement of these existing
features and packages for the new use case.