Latin-1 non breaking space not highlighted

Latin-1 non breaking space not highlighted

[Ulrich Neumerkel]

--text follows this line--
This bug report will be sent to the Free Software Foundation,
not to your local site managers!
Please write in English, because the Emacs maintainers do not have
translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

In GNU Emacs 21.2.1 (i686-pc-linux-gnu, X toolkit)
 of 2002-04-08 on gupu.complang.tuwien.ac.at
configured using `configure  --without-toolkit-scroll-bars --prefix /usr'
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US
  locale-coding-system: iso-latin-1
  default-enable-multibyte-characters: nil

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

With show-trailing-whitespace being true in a Latin-1 buffer the
character "non breaking space" (C-x 8 SPACE) is not highlighted at the
end of line.


US
  locale-coding-system: iso-latin-1
  default-enable-multibyte-characters: nil

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

With show-trailing-whitespace being true in a Latin-1 buffer the
character "non breaking space" (C-x 8 SPACE) is not highlighted at the
end of line.

Re: Latin-1 non breaking space not highlighted

[Eli Zaretskii]

On Mon, 8 Apr 2002, Ulrich Neumerkel wrote:

> With show-trailing-whitespace being true in a Latin-1 buffer the
> character "non breaking space" (C-x 8 SPACE) is not highlighted at the
> end of line.

Currently, the definition of ``trailing whitespace'' includes only ASCII 
whitespace characters.  If we are to broaden this definition to non-ASCII 
characters, I think we need a more general definition; just adding a 
Latin-1 NBSP is not enough, IMHO.

Re: Latin-1 non breaking space not highlighted

[Richard Stallman]

    Currently, the definition of ``trailing whitespace'' includes only ASCII 
    whitespace characters.  If we are to broaden this definition to non-ASCII 
    characters, I think we need a more general definition; just adding a 
    Latin-1 NBSP is not enough, IMHO.

What is the syntax code of Latin-1 NBSP?  Is it "whitespace"?
If not, why not?

If it is, then why not make "trailing whitespace" include everything
with whitespace syntax?

Re: Latin-1 non breaking space not highlighted

[Eli Zaretskii]

On Tue, 9 Apr 2002, Richard Stallman wrote:

>     Currently, the definition of ``trailing whitespace'' includes only ASCII 
>     whitespace characters.  If we are to broaden this definition to non-ASCII 
>     characters, I think we need a more general definition; just adding a 
>     Latin-1 NBSP is not enough, IMHO.
> 
> What is the syntax code of Latin-1 NBSP?  Is it "whitespace"?

Yes.

> If it is, then why not make "trailing whitespace" include everything
> with whitespace syntax?

I didn't say we shouldn't do that.  I'm just not sure this is the right 
thing to do.

The current code simply doesn't consult the syntax tables, it has the 
``whitespace'' characters hardwired into it.

I cannot think of any particularly bad consequences of using the syntax 
tables, but I do see some issues that perhaps need to be considered:

  - the definition of whitespace will become mode-dependent;

  - Lisp programs that modify syntax tables could affect the 
    trailing-whitespace feature; in particular, if some unexpected 
    characters are defined to have whitespace syntax, users of trailing 
    whitespace might become surprised by the results.

Re: Latin-1 non breaking space not highlighted

[Richard Stallman]

    I cannot think of any particularly bad consequences of using the syntax 
    tables, but I do see some issues that perhaps need to be considered:

      - the definition of whitespace will become mode-dependent;

Really?  Which modes change this?

I don't think it would be a problem if this were affected
by some modes, I just am surprised that there are any modes that
would do so.

      - Lisp programs that modify syntax tables could affect the 
	trailing-whitespace feature; in particular, if some unexpected 
	characters are defined to have whitespace syntax, users of trailing 
	whitespace might become surprised by the results.

I think this would be a change for the better in such a case.

Could you make the change?

Re: Latin-1 non breaking space not highlighted

[Eli Zaretskii]

> Date: Wed, 10 Apr 2002 14:16:46 -0600 (MDT)
> From: Richard Stallman <rms@gnu.org>
> 
>       - the definition of whitespace will become mode-dependent;
> 
> Really?  Which modes change this?

I don't know whether there's a mode that actually changes the syntax
of whitespace characters.  But it's conceivable, no?

> I think this would be a change for the better in such a case.
> 
> Could you make the change?

I will add this to my todo.

Re: Latin-1 non breaking space not highlighted

[Eli Zaretskii]

> From: "Eli Zaretskii" <eliz@is.elta.co.il>
> Date: Thu, 11 Apr 2002 21:54:10 +0300
> 
> > Date: Wed, 10 Apr 2002 14:16:46 -0600 (MDT)
> > From: Richard Stallman <rms@gnu.org>
> > 
> >       - the definition of whitespace will become mode-dependent;
> > 
> > Really?  Which modes change this?
> 
> I don't know whether there's a mode that actually changes the syntax
> of whitespace characters.  But it's conceivable, no?

I see that the following packages modify the syntactic class of some
characters to be whitespace or redefine whitespace characters to have
some non-whitespace syntax:

  completion.el (where whitespace is the default class)
  font-lock.el (*may* do this if font-lock-defaults request that)
  gnus-score.el (makes digits have whitespace syntax)
  ietf-drums.el (makes punctuation characters have whitespace syntax)
  mml.el (ditto)
  imenu.el (*may* do so if imenu-syntax-alist requests that)
  some modes such as ada-mode and awk-mode modify the syntax of \f, \n, \r
  scheme.el defines ` and ' to have whitespace syntax
  speedbar defines many characters to have whitespace syntax
  bibtex (defines ~ to be whitespace)
  texinfmt (defines \ to be whitespace)

(I'm not sure I've seen all the possible cases).

So I think the conclusion is that it's rare, but does happen.

Re: Latin-1 non breaking space not highlighted

[Richard Stallman]

Given this info

      gnus-score.el (makes digits have whitespace syntax)
      ietf-drums.el (makes punctuation characters have whitespace syntax)
      mml.el (ditto)
      scheme.el defines ` and ' to have whitespace syntax
      speedbar defines many characters to have whitespace syntax
      bibtex (defines ~ to be whitespace)
      texinfmt (defines \ to be whitespace)

It looks like there is a real problem.  I guess we should not simply
change show-trailing-whitespace to use the whitespace syntax class.

Re: Latin-1 non breaking space not highlighted

[Eli Zaretskii]

> Date: Sat, 13 Apr 2002 13:06:22 -0600 (MDT)
> From: Richard Stallman <rms@gnu.org>
> 
> It looks like there is a real problem.  I guess we should not simply
> change show-trailing-whitespace to use the whitespace syntax class.

I see two possible solutions:

  - hardcode a few more special characters, like ISO8859-x NBSP and a
    few Unicode characters;
or
  - use syntax tables for non-ASCII characters only, assuming that no
    Lisp package does anything weird with their syntax class.

Re: Latin-1 non breaking space not highlighted

[Richard Stallman]

    I see two possible solutions:

      - hardcode a few more special characters, like ISO8859-x NBSP and a
	few Unicode characters;

For now, I think we should do it that way.

For the longer term, I think we should look at these modes
which change the definition of whitespace, because
I suspect they are kludges.  If there is no good clean way
to do what they are trying to do, perhaps we should make
a good clean way.  Perhaps it is reasonable for them to
make these modified syntax tables but they should use those
syntax tables only inside of specific Lisp code, never leave them
current so that they could affect commands in general.

Re: Latin-1 non breaking space not highlighted

[Eli Zaretskii]

On Mon, 15 Apr 2002, Richard Stallman wrote:

>     I see two possible solutions:
> 
>       - hardcode a few more special characters, like ISO8859-x NBSP and a
> 	few Unicode characters;
> 
> For now, I think we should do it that way.

Perhaps the char-table implementation suggested by Handa-san is better.  
It is certainly easier to customize by users, so it's more flexible.

Re: Latin-1 non breaking space not highlighted

[Stefan Monnier]

> 
> On Mon, 15 Apr 2002, Richard Stallman wrote:
> 
> >     I see two possible solutions:
> > 
> >       - hardcode a few more special characters, like ISO8859-x NBSP and a
> > 	few Unicode characters;
> > 
> > For now, I think we should do it that way.
> 
> Perhaps the char-table implementation suggested by Handa-san is better.
> It is certainly easier to customize by users, so it's more flexible.

I think using fundamental-mode's syntax-table (i.e. the standard syntax-table)
is the best choice.


	Stefan

Re: Latin-1 non breaking space not highlighted

[Eli Zaretskii]

On Thu, 18 Apr 2002, Stefan Monnier wrote:

> I think using fundamental-mode's syntax-table (i.e. the standard syntax-table)
> is the best choice.

But Handa-san listed a couple of problems with using the standard syntax 
tables.  For example, a newline has whitespace syntax, but we don't want 
it to be considered as trailing whitespace, do we?

Re: Latin-1 non breaking space not highlighted

[Stefan Monnier]

> But Handa-san listed a couple of problems with using the standard syntax 
> tables.

I seem to have missed his email.

>  For example, a newline has whitespace syntax, but we don't want 
> it to be considered as trailing whitespace, do we?

I assumed that newline was already treated specially so it wouldn't
even be looked up in the table.


	Stefan

Re: Latin-1 non breaking space not highlighted

[Richard Stallman]

    I think using fundamental-mode's syntax-table (i.e. the standard syntax-table)
    is the best choice.

That seems like a good idea, assuming the issue with newline is 
handled suitably.

Re: Latin-1 non breaking space not highlighted

[Ulrich Neumerkel]

> From: "Eli Zaretskii" <eliz@is.elta.co.il>
> I see two possible solutions:
>
>   - hardcode a few more special characters, like ISO8859-x NBSP and a
>     few Unicode characters;
> or
>   - use syntax tables for non-ASCII characters only, assuming that no
>     Lisp package does anything weird with their syntax class.

Both solutions may cause problems with other/new encoding systems.
Maybe better:
   - use the syntax table of the corresponding fundamental mode

Re: Latin-1 non breaking space not highlighted

[Richard Stallman]

    Maybe better:
       - use the syntax table of the corresponding fundamental mode

What is "the corresponding fundamental mode"?
That is not usual terminology and I am not sure what it means.

Re: Latin-1 non breaking space not highlighted

[Kenichi Handa]

"Eli Zaretskii" <eliz@is.elta.co.il> writes:
>>  It looks like there is a real problem.  I guess we should not simply
>>  change show-trailing-whitespace to use the whitespace syntax class.

> I see two possible solutions:

>   - hardcode a few more special characters, like ISO8859-x NBSP and a
>     few Unicode characters;
> or
>   - use syntax tables for non-ASCII characters only, assuming that no
>     Lisp package does anything weird with their syntax class.

I think show-trailing-whitespace should not be affected by
syntax of characters because what it should concern is a
glyph.  If a glyph of a character is just space, even if the
syntax is `symbol' or whatever else, it should be
highlighted.

So, what we need is a char-table of such characters that are
displayed by white glyph.  In addition to many NBSPs, that
char-table should also contain the following CJK white-space
characters:
 (make-char 'japanese-jisx0208-1978 33 33)
 (make-char 'chinese-gb2312 33 33)
 (make-char 'japanese-jisx0208 33 33)
 (make-char 'korean-ksc5601 33 33)
 (make-char 'chinese-cns11643-1 33 33)
 (make-char 'japanese-jisx0213-1 33 33)
 (make-char 'chinese-big5-1 33 33)

---
Ken'ichi HANDA
handa@etl.go.jp

Re: Latin-1 non breaking space not highlighted

[Kenichi Handa]

"Stefan Monnier" <monnier+gnu/emacs@RUM.cs.yale.edu> writes:
>>  But Handa-san listed a couple of problems with using the standard syntax 
>>  tables.

> I seem to have missed his email.

My point is:
show-trailing-whitespace should not mean highlighting such
charactes that have "whitespace" SYNTAX, but should mean
highlighting sucn characters that have "whitespace" GLYPH.

Both sets of characters mostly overlap but not necessarily
be the same.

For instance, even if I set syntax of NBSP to "word
constitute", show-trailing-whitespace should highlight it.

---
Ken'ichi HANDA
handa@etl.go.jp

Re: Latin-1 non breaking space not highlighted

[Stefan Monnier]

> "Stefan Monnier" <monnier+gnu/emacs@RUM.cs.yale.edu> writes:
> >>  But Handa-san listed a couple of problems with using the standard syntax 
> >>  tables.
> 
> > I seem to have missed his email.
> 
> My point is:
> show-trailing-whitespace should not mean highlighting such
> charactes that have "whitespace" SYNTAX, but should mean
> highlighting sucn characters that have "whitespace" GLYPH.

That's a separate issue.  Related to whether or not we should
take the display-table into account.

> Both sets of characters mostly overlap but not necessarily
> be the same.
> 
> For instance, even if I set syntax of NBSP to "word
> constitute",

When would you do that and why (considering that we're specifically
talking about the global standard-syntax-table) ?

> show-trailing-whitespace should highlight it.

Really ?  I think this very much depends on the answer to the
previous question.

We really just need a table somewhere that tells us what is whitespace
and what isn't.  Currently I think there are two such tables, one is
the standard-syntax-table, the other is the ` ' category.  Actually,
I [:space:] and [:blank:] are two more, but [:space:] relies on the
buffer-local syntax-table (i.e. not good, although it's not that bad
since it only uses it for non-ASCII chars, IIRC) and [:blank:] only
matches ASCII chars (i.e. not good either).


	Stefan

Re: Latin-1 non breaking space not highlighted

[Miles Bader]

"Stefan Monnier" <monnier+gnu/emacs@RUM.cs.yale.edu> writes:
> We really just need a table somewhere that tells us what is whitespace
> and what isn't.  Currently I think there are two such tables, one is
> the standard-syntax-table, the other is the ` ' category.  Actually,
> I [:space:] and [:blank:] are two more, but [:space:] relies on the
> buffer-local syntax-table (i.e. not good, although it's not that bad
> since it only uses it for non-ASCII chars, IIRC) and [:blank:] only
> matches ASCII chars (i.e. not good either).

To be honest, given my understanding of how `show-trailing-whitespace'
is intended to be used, Kenichi's suggestions seems much better -- it's
consistent, simple, and robust (thought I don't know how simple it would
be to implement).

Can you think of a (real) case where it _wouldn't_ suffice?

-Miles
-- 
`Life is a boundless sea of bitterness'

Re: Latin-1 non breaking space not highlighted

[Kenichi Handa]

"Stefan Monnier" <monnier+gnu/emacs@RUM.cs.yale.edu> writes:
>>  My point is:
>>  show-trailing-whitespace should not mean highlighting such
>>  charactes that have "whitespace" SYNTAX, but should mean
>>  highlighting sucn characters that have "whitespace" GLYPH.

> That's a separate issue.  Related to whether or not we should
> take the display-table into account.

In my understanding, the purpose of show-trailing-whitespace
is to tell users that there are characters that has space
glyphs (thus invisible) at end of line.  So, yes, if some
characters are displayed by SPCs because of display-table,
it is better that we take that info account (although it may
require a rather big change in the display engine).  Anyway,
we should not decide if a character should be highlighted or
not simply by its syntax.

>>  Both sets of characters mostly overlap but not necessarily
>>  be the same.
>>  
>>  For instance, even if I set syntax of NBSP to "word
>>  constitute",

> When would you do that and why (considering that we're specifically
> talking about the global standard-syntax-table) ?

The above is just an example.  I don't mean it's useful.
But, hmmm, perhaps, it may be useful for filling.

>>  show-trailing-whitespace should highlight it.

> Really ?  I think this very much depends on the answer to the
> previous question.

I really think NBSP should be highlighted also in such a
(hypothetical) situation.  What do the other people think?

> We really just need a table somewhere that tells us what is whitespace
> and what isn't.  Currently I think there are two such tables, one is
> the standard-syntax-table, the other is the ` ' category.  Actually,
> I [:space:] and [:blank:] are two more, but [:space:] relies on the
> buffer-local syntax-table (i.e. not good, although it's not that bad
> since it only uses it for non-ASCII chars, IIRC) and [:blank:] only
> matches ASCII chars (i.e. not good either).

I have forgotten about ` ' category.  Do you know which
character has this category and how is that category used?

---
Ken'ichi HANDA
handa@etl.go.jp

Re: Latin-1 non breaking space not highlighted

[Richard Stallman]

    In my understanding, the purpose of show-trailing-whitespace
    is to tell users that there are characters that has space
    glyphs (thus invisible) at end of line.  So, yes, if some
    characters are displayed by SPCs because of display-table,
    it is better that we take that info account (although it may
    require a rather big change in the display engine).

I wouldn't do this if it took more than 5 minutes.
It is too obscure a case to be worth more than that much
of any of our time.

    >>  For instance, even if I set syntax of NBSP to "word
    >>  constitute",

    I really think NBSP should be highlighted also in such a
    (hypothetical) situation.  What do the other people think?

I agree.