bug#12086: 24.1; character groups and `glyphless-char-display-control'

bug#12086: 24.1; character groups and `glyphless-char-display-control'

[Drew Adams]

1. Both the doc string and the doc in the Elisp manual have this
problem: Nothing is said about what the display is for a group that is
NOT included in the value of option `glyphless-char-display-control'.
 
IOW, this option apparently lets you specify, for one or more given
character groups, how its chars are to be displayed.  But nothing is
said about how the chars are displayed for a group that is not specified
using this option.

Presumably, some reference should be made here to
`glyphless-char-display'.  But even for that variable there is no
explanation of what normally sets that variable value, and based on what.
How is someone to know how a character group is displayed that is not
listed in `glyphless-char-display-control'?
 

2. I also wonder about the critter called a character "group", which is
apparently new with Emacs 24.  Where is the notion of character group
defined?  Although new with Emacs 24, I find it mentioned nowhere in NEWS.
 
And how does this critter relate, for instance, to a character set
(charset)?  For example, `c1-control' is a character group and
`control-1' is a charset.  But these seem to include the same set of
characters:

`describe-character-set control-1' says that it is the 8-bit control
codes (0x80..0x9F).  The doc for `glyphless-char-display-control' says
that group `c1-control' is the non-ASCII, non-printing characters
`U+0080' to `U+009F'.
 
Is that a coincidence or a deliberate correspondence?  Any rational
relation between char groups and charsets should be documented.  The
relation between char group `c1-control' and charset `control-1' seems
to be one such presumably rational relation.


3. How, if it is even possible, can a programmer turn off the effect of
the glyphless display, which uses face `glyphless-char'?
 
For example, for a non-breaking hyphen you can set
`nobreak-char-display' to nil and then use ordinary font-lock with any
face you want, to display the character other than with the default face
`escape-glyph'.
 
How to do the same thing for glyphless characters?  Is it even possible?
Or is the only control a programmer or user has is to set
`glyphless-char-display-control'?  That option controls the METHOD, but
not the face used.


4. If I do (insert-char #x80 4), I see four \200 chars in what appears to be
face `escape-glyph', and NOT in face `glyphless-char'.  Doesn't the doc say that
`glyphless-char' is used?  And `C-u C-x =' does not say anything about either of
those faces.  What is going on here?


In GNU Emacs 24.1.1 (i386-mingw-nt5.1.2600)
 of 2012-06-10 on MARVIN
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --with-gcc (4.6) --cflags
 -ID:/devel/emacs/libs/libXpm-3.5.8/include
 -ID:/devel/emacs/libs/libXpm-3.5.8/src
 -ID:/devel/emacs/libs/libpng-dev_1.4.3-1/include
 -ID:/devel/emacs/libs/zlib-dev_1.2.5-2/include
 -ID:/devel/emacs/libs/giflib-4.1.4-1/include
 -ID:/devel/emacs/libs/jpeg-6b-4/include
 -ID:/devel/emacs/libs/tiff-3.8.2-1/include
 -ID:/devel/emacs/libs/gnutls-3.0.9/include'
 

bug#12086: 24.1; character groups and `glyphless-char-display-control'

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sun, 29 Jul 2012 15:28:47 -0700
> 
> How is someone to know how a character group is displayed that is not
> listed in `glyphless-char-display-control'?

This is covered in the "Usual Display" node of the ELisp manual, which
is earlier in the same chapter.  Assuming that enough references to
that node are added to "Glyphless Chars", so that it's clear what
happens with the other characters, what else is missing?

> 2. I also wonder about the critter called a character "group", which is
> apparently new with Emacs 24.  Where is the notion of character group
> defined?

It's not really a notion, it comes from the '(GROUP . METHOD)' form of
glyphless-char-display-control.  There's nothing else to it.

> And how does this critter relate, for instance, to a character set
> (charset)?  For example, `c1-control' is a character group and
> `control-1' is a charset.  But these seem to include the same set of
> characters:
> 
> `describe-character-set control-1' says that it is the 8-bit control
> codes (0x80..0x9F).  The doc for `glyphless-char-display-control' says
> that group `c1-control' is the non-ASCII, non-printing characters
> `U+0080' to `U+009F'.
>  
> Is that a coincidence or a deliberate correspondence?  Any rational
> relation between char groups and charsets should be documented.  The
> relation between char group `c1-control' and charset `control-1' seems
> to be one such presumably rational relation.

It's a convenience device, nothing more.  The groups were chosen to
allow convenient reference to groups of characters that have similar
properties, as far as display is concerned.  c0-control and c1-control
are the names of these groups in old standards, see
http://en.wikipedia.org/wiki/C0_and_C1_control_codes, for example.  I
see no particular reason to delve into that in the manual; people who
know what these are will "Aha!" when they read; for others there's an
explicit definition of these groups in the manual.

> 3. How, if it is even possible, can a programmer turn off the effect of
> the glyphless display, which uses face `glyphless-char'?

As with every char-table: put nil in the character cell of the
character for which you want to disable this display.  If you want to
display it entirely, make the entire char-table full of nil.

> 4. If I do (insert-char #x80 4), I see four \200 chars in what appears to be
> face `escape-glyph', and NOT in face `glyphless-char'.  Doesn't the doc say that
> `glyphless-char' is used?

See "Usual Display" (although the face is not mentioned there).

> And `C-u C-x =' does not say anything about either of those faces.

Because it doesn't know about them.  These faces are produced by the
display engine directly, as part of rendering the non-printable
characters, not via text properties.  So "C-u C-=", which looks for
text properties and overlays, doesn't see them.

bug#12086: 24.1; character groups and `glyphless-char-display-control'

[Drew Adams]

> > How is someone to know how a character group is displayed 
> > that is not listed in `glyphless-char-display-control'?
> 
> This is covered in the "Usual Display" node of the ELisp manual, which
> is earlier in the same chapter.  Assuming that enough references to
> that node are added to "Glyphless Chars", so that it's clear what
> happens with the other characters, what else is missing?

Thanks for replying.

Yes, a reference to `Usual Display' would be a start.

But that will only tell readers (a) what a glyphless char is (good) and (b) how
it is displayed by default.  But that display is (IIUC) without regard to
`glyphless-char-display' (and `glyphless-char-display-control', which sets
`glyphless-char-display').

One question I have is whether anything besides `g-c-d-c' normally gives `g-c-d'
a value.  IOW, is `g-c-d' generally defined somewhere, so it becomes part of the
"default" display of glyphless chars (those which are not in an active display
table, that is)?  Or is `g-c-d' normally nil and gets set to non-nil pretty much
only by `g-c-d-c'?

Sorry, but this is not clear to me.

I was thinking that `g-c-d' was in general (i.e., often) defined somewhere by
default, as something non-nil.  So I asked the question of how to know how a
particular char group is displayed, if it is not listed in `g-c-d'.

I still have that question - about the _group_, after reading `Usual Display',
which tells how glyphless chars are displayed if (IIUC) `g-c-d' is nil.  But
what if it is non-nil and the group in question is not covered by it?

> > 2. I also wonder about the critter called a character 
> > "group", which is apparently new with Emacs 24.  Where is the notion of 
> > character group defined?
> 
> It's not really a notion, it comes from the '(GROUP . METHOD)' form of
> glyphless-char-display-control.  There's nothing else to it.

That's not too satisfying.  We introduce a new thingy, even if it is only in
order to describe the behavior of `glyphless-char-display-control'.  But we
don't say what it is.  We try to describe `glyphless-char-display-control' in
terms of it - seems a bit circular.  Replace "group" and "GROUP" with "jkjkljkl"
and what do you understand by reading the description?

Anyway, I understand that there is no formal meaning here, that all that is
meant by "character group" is some set of chars.

> > And how does this critter relate, for instance, to a character set
> > (charset)?  For example, `c1-control' is a character group and
> > `control-1' is a charset.  But these seem to include the same set of
> > characters:
> > 
> > `describe-character-set control-1' says that it is the 8-bit control
> > codes (0x80..0x9F).  The doc for `glyphless-char-display-control' says
> > that group `c1-control' is the non-ASCII, non-printing characters
> > `U+0080' to `U+009F'.
> >  
> > Is that a coincidence or a deliberate correspondence?  Any rational
> > relation between char groups and charsets should be documented.  The
> > relation between char group `c1-control' and charset 
> > `control-1' seems to be one such presumably rational relation.
> 
> It's a convenience device, nothing more.  The groups were chosen to
> allow convenient reference to groups of characters that have similar
> properties, as far as display is concerned.  c0-control and c1-control
> are the names of these groups in old standards, see
> http://en.wikipedia.org/wiki/C0_and_C1_control_codes, for example.  I
> see no particular reason to delve into that in the manual; people who
> know what these are will "Aha!" when they read; for others there's an
> explicit definition of these groups in the manual.

Sure, it's just a name.  But now we have both char sets and char groups, and
some of the thingies in the first are also the same set of chars as some
thingies in the second.

If there is no useful relation between these then the coincidence of names and
meanings (groups of chars vs char sets) is just confusing.  IOW, if similar
names are similar because of some useful relation then that relation should be
mentioned.  If they are similar for no special reason then that invites
confusion.
 
> > 3. How, if it is even possible, can a programmer turn off 
> > the effect of the glyphless display, which uses face `glyphless-char'?
> 
> As with every char-table: put nil in the character cell of the
> character for which you want to disable this display.  If you want to
> display it entirely, make the entire char-table full of nil.

So to display the char using a different face I need to put nil in the cell for
the char (and then do something to use the new face, such as font-lock)?  Seems
a bit contorted, but OK.  Maybe mention this?

What about the comparison with non-breaking hyphen and its control via
`nobreak-char-display'?  Why not have something similar here, to simplify things
(for users)?

> > 4. If I do (insert-char #x80 4), I see four \200 chars in 
> > what appears to be face `escape-glyph', and NOT in face `glyphless-char'.  
> > Doesn't the doc say that `glyphless-char' is used?
> 
> See "Usual Display" (although the face is not mentioned there).

It should be mentioned (somewhere), I think.

And it's not clear to me that that face is the right choice anyway.  Read the
doc string for that face: it does not apply here at all, AFAICT.  We should use
a new face for this.  See also bug #12048.

> > And `C-u C-x =' does not say anything about either of those faces.
> 
> Because it doesn't know about them.  These faces are produced by the
> display engine directly, as part of rendering the non-printable
> characters, not via text properties.  So "C-u C-=", which looks for
> text properties and overlays, doesn't see them.

Too bad.

bug#12086: 24.1; character groups and `glyphless-char-display-control'

[Lars Ingebrigtsen]

It's unclear what the conclusion is here.  Drew, is there anything else
in the documentation you think should be improved?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#12086: 24.1; character groups and `glyphless-char-display-control'

[Drew Adams]

> It's unclear what the conclusion is here.  Drew, is there anything else
> in the documentation you think should be improved?

Dunno.  I give up.  Eli read the bug report and did what he thought
was appropriate.  He knows much more about this than I do.