char-table-range

char-table-range

[Kenichi Handa]

Luc pointed out the current odd behaviour of
char-table-range.  What should we do if values of characters
in the specified range are different?

For the moment, I don't have a good idea.  Possible
behaviours are:

(1) return nil
(2) return the first non-nil value in the range.
(3) return the default value of the range (but ascii,
    eight-bit-XXXX doesn't have a default value).
(4) signal an error

By the way, I don't know why this function is necessary, in
what situation it is intended to use this function.

Currently, it is used only in cc-vars.el as below.

(defconst c-emacs-features
[...]
       ;; before and including Emacs 19.34
       ((and (fboundp 'char-table-p)
	     (char-table-p table))
	(setq entry (car (char-table-range table [?a]))))

but this is equivalent to 

	(setq entry (car (aref table ?a))))

---
Ken'ichi HANDA
handa@m17n.org

Re: char-table-range

[Luc Teirlinck]

Ken'ichi Handa wrote:

   Luc pointed out the current odd behaviour of
   char-table-range.  What should we do if values of characters
   in the specified range are different?

Maybe it should be clarified that this was pointed out in private
email while discussing a fix to another bug (so people do not start
searching the archives for a non-existing report).

The current behavior for ascii and eight-bit-{graphic,contol} is so
odd that it probably can be safely called a bug.  Note that in the
ielm run below (char-table-range cc 'ascii) actually returns the value
specified for eight-bit-control.  For other charsets (I took ipa as
example), things would seem to be OK.

My own opinion would be to have char-table-range return the default
value, except for ascii and eight-bit-{control,graphic} which do not
have a default value.  For those it would seem to me that the best
thing to do would be to throw an error.

===File ~/char-table-ielm===================================
*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (put 'five-slots 'char-table-extra-slots 5)
5
ELISP> (setq cc (make-char-table 'five-slots))
;; long output deleted.
ELISP> (set-char-table-range cc 'ascii 1)
1
ELISP> (set-char-table-range cc 'eight-bit-control 2)
2
ELISP> (set-char-table-range cc 'eight-bit-graphic 3)
3
ELISP> (set-char-table-range cc 'ipa 4)
4
ELISP> (char-table-range cc 'ascii)
2
ELISP> (char-table-range cc 'eight-bit-graphic)
nil
ELISP> (char-table-range cc 'eight-bit-control)
nil
ELISP> (char-table-range cc 'ipa)
4
ELISP> 
============================================================

Re: char-table-range

[Luc Teirlinck]

Ken'ichi Handa wrote:

   What should we do if values of characters
   in the specified range are different?

   For the moment, I don't have a good idea.  Possible
   behaviours are:

   (1) return nil
   (2) return the first non-nil value in the range.
   (3) return the default value of the range (but ascii,
       eight-bit-XXXX doesn't have a default value).
   (4) signal an error

I did not realize that currently, for charsets different from ascii and
eight-bit-{control,graphic}, char-table-range actually returns the
entire range of values if the value is not uniform.  So this could be
a fifth possibility, since it appears to be the current situation:

===File ~/char-table-ielm-b=================================
*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (put 'five-slots 'char-table-extra-slots 5)
5
ELISP> (setq cc (make-char-table 'five-slots))
ELISP> (make-char 'ipa)
6272
ELISP> (set-char-table-default cc 6272 1)
1
ELISP> (char-table-range cc 'ipa)
1
ELISP> (split-char 6320)
(ipa 48)

ELISP> (aset cc 6320 2)
2
ELISP> (char-table-range cc 'ipa)
#^^[nil 1 nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 2 nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil]

ELISP> (aset cc 32 1)
1
ELISP> (aset cc 33 2)
2
ELISP> (char-table-range cc 'ascii)
nil
ELISP> (aset cc 128 3)
3
ELISP> (char-table-range cc 'ascii)
3
ELISP> 
============================================================

Re: char-table-range

[Kenichi Handa]

In article <200402160212.i1G2Cbq04302@raven.dms.auburn.edu>, Luc Teirlinck <teirllm@dms.auburn.edu> writes:
> I did not realize that currently, for charsets different from ascii and
> eight-bit-{control,graphic}, char-table-range actually returns the
> entire range of values if the value is not uniform.  So this could be
> a fifth possibility, since it appears to be the current situation:

No.  I think it is a serious bug because that returned value
is a sub-char-table which should not be exposed to Lisp.

> My own opinion would be to have char-table-range return the default
> value, except for ascii and eight-bit-{control,graphic} which do not
> have a default value.  For those it would seem to me that the best
> thing to do would be to throw an error.

Actually, we can think that a char-table has a default value
for ascii and eight-bit-xxx.  It is the default value of the
char-table itself.  So, I now think that your idea of
returning the default value is good.

---
Ken'ichi HANDA
handa@m17n.org

Re: char-table-range

[Richard Stallman]

    Luc pointed out the current odd behaviour of
    char-table-range.  What should we do if values of characters
    in the specified range are different?

    For the moment, I don't have a good idea.  Possible
    behaviours are:

    (1) return nil
    (2) return the first non-nil value in the range.
    (3) return the default value of the range (but ascii,
	eight-bit-XXXX doesn't have a default value).
    (4) signal an error

I think signaling an error is the cleanest behavior.

    By the way, I don't know why this function is necessary, in
    what situation it is intended to use this function.

I think it may be included for completeness of access
to the data in a char table.

Re: char-table-range

[Richard Stallman]

    My own opinion would be to have char-table-range return the default
    value, except for ascii and eight-bit-{control,graphic} which do not
    have a default value.

Is there any other convenient way to get the default value and always
the default value?  For instance, by using a generic character?
If so, it would be better for char-table-range to signal an error
than to return the default value.

    I did not realize that currently, for charsets different from ascii and
    eight-bit-{control,graphic}, char-table-range actually returns the
    entire range of values if the value is not uniform.

That seems like a good thing for it to to.

Re: char-table-range

[Luc Teirlinck]

Richard Stallman wrote:

   Is there any other convenient way to get the default value and always
   the default value?  For instance, by using a generic character?

That seems to work.

       I did not realize that currently, for charsets different from ascii and
       eight-bit-{control,graphic}, char-table-range actually returns the
       entire range of values if the value is not uniform.

   That seems like a good thing for it to to.

Yes, but Handa remarked that the return value is given in a form
(sub-char-table) which is not supposed to be exported to Lisp.

I am currently reading the Elisp chapter on sequences.  In the section
on chartables several things are wrongly, incompletely or ambiguously
documented.  I will take care of that, but I do not want to "document"
bugs.

Apart from the problem we are discussing right now, there are two
other things that seem surprising to me about chartables.  I am not
necessarily claiming that they are bugs or misfeatures, I just want to
make sure that they are not before documenting them.

One seems very closely related to what we are currently discussing:

While it does seem possible to get the default value by specifying a
generic character, there is a strange subtle asymmetry between
_setting_ the default value for, say, ipa, and setting the
corresponding generic character (6272 in this case).

Setting the default value using `set-char-table-default' does _not_
override non-nil pre-existing values in the charset _except_ for the
generic character itself.  Setting the generic character itself
overrides the entire charset.  Is this intentional?  If yes, I will
clearly document it (because it is confusing).

The second thing that looks _somewhat_ strange (but maybe it is OK) is
that (make-char-table subtype init) will also set extra slots to INIT.
This is somewhat unexpected because a logical default for characters
would seldom seem to make sense for extra slots as well.  Maybe this
is intentional anyway.  In that case I will document it.

ielm run illustrating all of the above:

===File ~/chartable-ielm====================================
*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (put 'five-slots 'char-table-extra-slots 5)
5
ELISP> (setq cc (make-char-table 'five-slots 2))
#^[t 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 nil five-slots 2 2 2 2 2]

ELISP> (char-table-extra-slot cc 3)
2
ELISP> (make-char 'ipa)
6272
ELISP> (split-char 6320)
(ipa 48)

ELISP> (aset cc 6320 7)
7
ELISP> (aset cc 6272 1)
1
ELISP> (aref cc 6320)
1
ELISP> (aset cc 6320 7)
7
ELISP> (set-char-table-default cc 6272 8)
8
ELISP> (aref cc 6272)
8
ELISP> (aref cc 6320)
7
ELISP> (char-table-range cc 'ipa)
#^^[nil 8 nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil 7 nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil]

ELISP> 
============================================================

Re: char-table-range

[Luc Teirlinck]

Actually, there is still something else that I find somewhat
counter-intuitive about chartables and that is the identification of
invalid characters with generic characters.  Note how in the ielm run
below, the "real" generic character for ipa, 6272, and the invalid
character 6300 seem to be treated as identical.  Is this intentional?

===File ~/new-chartable-ielm================================
*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (put 'five-slots 'char-table-extra-slots 5)
5
ELISP> (setq cc (make-char-table 'five-slots 2))
ELISP> (make-char 'ipa)
6272
ELISP> (split-char 6320)
(ipa 48)

ELISP> (char-valid-p 6300)
nil
ELISP> (aset cc 6320 3)
3
ELISP> (aref cc 6272)
2
ELISP> (aref cc 6320)
3
ELISP> (aset cc 6300 "invalid char")
"invalid char"
ELISP> (aref cc 6272)
"invalid char"
ELISP> (aref cc 6320)
"invalid char"
ELISP> 
============================================================

Re: char-table-range

[Luc Teirlinck]

>From my previous message:

   While it does seem possible to get the default value by specifying a
   generic character, there is a strange subtle asymmetry between
   _setting_ the default value for, say, ipa, and setting the
   corresponding generic character (6272 in this case).

   Setting the default value using `set-char-table-default' does _not_
   override non-nil pre-existing values in the charset _except_ for the
   generic character itself.  Setting the generic character itself
   overrides the entire charset.  Is this intentional?  If yes, I will
   clearly document it (because it is confusing).

On second thought, this _must_ be intentional, otherwise the function
set-char-table-default would be completely unnecessary.  I will just
document the fact.

Sincerely,

Luc.

Re: char-table-range

[Richard Stallman]

	   I did not realize that currently, for charsets different from ascii and
	   eight-bit-{control,graphic}, char-table-range actually returns the
	   entire range of values if the value is not uniform.

       That seems like a good thing for it to to.

    Yes, but Handa remarked that the return value is given in a form
    (sub-char-table) which is not supposed to be exported to Lisp.

That is unfortunate.  The distinction between char-tables and
sub-char-tables is one that "sort of grew", and is sill being
clarified.

    I am currently reading the Elisp chapter on sequences.  In the section
    on chartables several things are wrongly, incompletely or ambiguously
    documented.  I will take care of that, but I do not want to "document"
    bugs.

Thank you.

    Setting the default value using `set-char-table-default' does _not_
    override non-nil pre-existing values in the charset _except_ for the
    generic character itself.  Setting the generic character itself
    overrides the entire charset.  Is this intentional?

I don't know.  But it is not very clean, so it would be nice to change it,
if that does not cause some other problem.

Handa, do you know if anything depends on this?

    The second thing that looks _somewhat_ strange (but maybe it is OK) is
    that (make-char-table subtype init) will also set extra slots to INIT.
    This is somewhat unexpected because a logical default for characters
    would seldom seem to make sense for extra slots as well.  Maybe this
    is intentional anyway.

It looks like a misfeature to me.  But before changing it, we had
better see if anything depends on it.  I think that means looking at
all the calls to make-char-table and checking each one.

Can you do that?

Re: char-table-range

[Richard Stallman]

      Note how in the ielm run
    below, the "real" generic character for ipa, 6272, and the invalid
    character 6300 seem to be treated as identical.  Is this intentional?

It is a little ugly, but it isn't very bad.
If nothing depends on this behavior, then it might be nicer
to get an error for an invalid character code, if that is not
hard to implement.

Re: char-table-range

[Luc Teirlinck]

Richard Stallman wrote:

       The second thing that looks _somewhat_ strange (but maybe it is OK) is
       that (make-char-table subtype init) will also set extra slots to INIT.
       This is somewhat unexpected because a logical default for characters
       would seldom seem to make sense for extra slots as well.  Maybe this
       is intentional anyway.

   It looks like a misfeature to me.  But before changing it, we had
   better see if anything depends on it.  I think that means looking at
   all the calls to make-char-table and checking each one.

   Can you do that?

It seems to be used (?) in exactly one place in the Emacs source code
(if I did not overlook anything): china-util.el

There we find the function `expand-euc-big5-alist' which is defined
inside an eval-when-compile and it is normally not defined at runtime.

It contains the line:

  (let ((chartable (make-char-table 'translation-table #xFF))

`chartable' is used in the return value. This sets all values,
including extra slots to #xFF = 255.  `expand-euc-big5-alist' is used
to put a translation-table property on the symbol `big5-to-cns' (whose
actual value as a variable is another translation table).  The value
of that translation-table property has its two extra slots set to 255.

Ielm run illustrating the above:

===File ~/china-util-ielm===================================
*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (load "china-util")
t
ELISP> (setq tt (get 'big5-to-cns 'translation-table))
;; Long ouput omitted
ELISP> (char-table-extra-slot tt 0)
255
ELISP> (char-table-extra-slot tt 1)
255
ELISP> (expand-euc-big5-alist nil)
*** Eval error ***  Symbol's function definition is void: expand-euc-big5-alist
ELISP> (load "china-util.el")
t
ELISP> (expand-euc-big5-alist nil)
#^[t 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
  255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 !
 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 nil translation-table 255 255]

ELISP> 
============================================================

Re: char-table-range

[Luc Teirlinck]

Ken 'ichi Handa wrote:

   (set-char-table-default TABLE IDX VAL)

     (1) If IDX is a symbol `ascii', set the default value for all
	 characters to VAL.

     (2) If IDX is GENERIC-CHAR, set the value for characters in
	 GENERIC-CHAR to nil, and set the default value for them to VAL.

     (3) If IDX is an individual character, set IDX to a generic
	 character that represents the smallest group of characters
	 containing IDX, and perform (2).

   I think (1) is a misfeature.  The default value for all characters
   should be set by specifying IDX nil.  (3) is also very questionable. I
   think we don't need it.

I do not know whether (2) is intended as description of the current
behavior of set-char-table-default or as a proposed new behavior.  As
a description of the current behavior it is not completely accurate.
Indeed, aset and set-char-table-range both behave exactly as described
in (2) for a generic character, but currently, set-char-table-default
for a generic character just sets the default value to VAL, _without_
first setting the value for characters in GENERIC-CHAR to nil (as aset
and set-char-table-range indeed both do).  See the ielm run below.

===File ~/chartable-stuff===================================
*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (setq cc (make-char-table 'translation-table))
ELISP> (aset cc (make-char 'ipa 48) 2)
2
ELISP> (aset cc (make-char 'ipa) 1)
1
ELISP> (aref cc (make-char 'ipa 48))
1
ELISP> (aset cc (make-char 'ipa 48) 2)
2
ELISP> (set-char-table-range cc (make-char 'ipa) 1)
1
ELISP> (aref cc (make-char 'ipa 48))
1
ELISP> (aset cc (make-char 'ipa 48) 2)
2
ELISP> (set-char-table-default cc (make-char 'ipa) 1)
1
ELISP> (aref cc (make-char 'ipa 48))
2
ELISP> 
============================================================

Re: char-table-range

[Kenichi Handa]

In article <200403030339.i233dJs02566@raven.dms.auburn.edu>, Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Ken 'ichi Handa wrote:
>    (set-char-table-default TABLE IDX VAL)

>      (1) If IDX is a symbol `ascii', set the default value for all
> 	 characters to VAL.

>      (2) If IDX is GENERIC-CHAR, set the value for characters in
> 	 GENERIC-CHAR to nil, and set the default value for them to VAL.

>      (3) If IDX is an individual character, set IDX to a generic
> 	 character that represents the smallest group of characters
> 	 containing IDX, and perform (2).

>    I think (1) is a misfeature.  The default value for all characters
>    should be set by specifying IDX nil.  (3) is also very questionable. I
>    think we don't need it.

> I do not know whether (2) is intended as description of the current
> behavior of set-char-table-default or as a proposed new behavior.  As
> a description of the current behavior it is not completely accurate.
> Indeed, aset and set-char-table-range both behave exactly as described
> in (2) for a generic character, but currently, set-char-table-default
> for a generic character just sets the default value to VAL, _without_
> first setting the value for characters in GENERIC-CHAR to nil (as aset
> and set-char-table-range indeed both do).  See the ielm run below.

Oops, you are right.  I myself was confused.  I wrote above
as description of the currently behavior, and should be
written as this:

  (2) If IDX is GENERIC-CHAR, set the default value for
      characters in GENERIC-CHAR to VAL.

My opininons/proposals were written without indentation.

---
Ken'ichi HANDA
handa@m17n.org

Re: char-table-range

[Kenichi Handa]

In article <200403030339.i233dJs02566@raven.dms.auburn.edu>, Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Ken 'ichi Handa wrote:
>    (set-char-table-default TABLE IDX VAL)

>      (1) If IDX is a symbol `ascii', set the default value for all
> 	 characters to VAL.

>      (2) If IDX is GENERIC-CHAR, set the value for characters in
> 	 GENERIC-CHAR to nil, and set the default value for them to VAL.

>      (3) If IDX is an individual character, set IDX to a generic
> 	 character that represents the smallest group of characters
> 	 containing IDX, and perform (2).

>    I think (1) is a misfeature.  The default value for all characters
>    should be set by specifying IDX nil.  (3) is also very questionable. I
>    think we don't need it.

> I do not know whether (2) is intended as description of the current
> behavior of set-char-table-default or as a proposed new behavior.  As
> a description of the current behavior it is not completely accurate.
> Indeed, aset and set-char-table-range both behave exactly as described
> in (2) for a generic character, but currently, set-char-table-default
> for a generic character just sets the default value to VAL, _without_
> first setting the value for characters in GENERIC-CHAR to nil (as aset
> and set-char-table-range indeed both do).  See the ielm run below.

Oops, you are right.  I myself was confused.  I wrote above
as description of the currently behavior, and should be
written as this:

  (2) If IDX is GENERIC-CHAR, set the default value for characters in
      GENERIC-CHAR to VAL.

My opininons/proposals were written without indentation.

---
Ken'ichi HANDA
handa@m17n.org

Re: char-table-range

[Richard Stallman]

    > in (2) for a generic character, but currently, set-char-table-default
    > for a generic character just sets the default value to VAL, _without_
    > first setting the value for characters in GENERIC-CHAR to nil

That seems correct to me, for set-char-table-default.