Question regarding Lisp_Object representation of symbols

Question regarding Lisp_Object representation of symbols

[Gerd Möllmann]

Symbols are currently encoded as words with tag = 0 (= Lisp_Symbol
enumerator), that when untagged are an offset from lispsym (the array of
built-in symbols), not a pointer to some structure like it is done for
other Lisp types. That makes the word 0x0 = Qnil, 0x1 = Qt, and so on.

My question comes from playing with another GC library, where this
unique encoding is, well, quite inconvenient. So, I'd like to ask two
questions:

- Does someone remember the reasons why this has been done?

- Is there something that relies on this? Maybe something that
  relies on 0x0 being a valid Lisp_Object?

Thanks!

Re: Question regarding Lisp_Object representation of symbols

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> - Does someone remember the reasons why this has been done?

To speed up comparisons between objects and Qnil, I think.

> - Is there something that relies on this? Maybe something that
>   relies on 0x0 being a valid Lisp_Object?

Yes, in the many places where memset/memclear is invoked to initialize
Lisp_Object fields or variables, for instance.  It's supposed to be the
case that these instances are marked with NIL_IS_ZERO for easy locating
if the representation of Qnil is ever to be changed, but that convention
is not observed in practice, or is actively circumvented, as by callers
of memclear, which defeat the purpose of that marker by calling this one
of its bearers to initialize a variety of structures that may or may not
comprise only Lisp_Objects, greatly inflating the number of callers that
must be examined before such a change in object representation.

Re: Question regarding Lisp_Object representation of symbols

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> - Does someone remember the reasons why this has been done?
>
> To speed up comparisons between objects and Qnil, I think.
>
>> - Is there something that relies on this? Maybe something that
>>   relies on 0x0 being a valid Lisp_Object?
>
> Yes, in the many places where memset/memclear is invoked to initialize
> Lisp_Object fields or variables, for instance.  It's supposed to be the
> case that these instances are marked with NIL_IS_ZERO for easy locating
> if the representation of Qnil is ever to be changed, but that convention
> is not observed in practice, or is actively circumvented, as by callers
> of memclear, which defeat the purpose of that marker by calling this one
> of its bearers to initialize a variety of structures that may or may not
> comprise only Lisp_Objects, greatly inflating the number of callers that
> must be examined before such a change in object representation.

Thank you very much, that was really helpful!

Re: Question regarding Lisp_Object representation of symbols

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Wed, 06 Mar 2024 07:48:31 +0100
> 
> Symbols are currently encoded as words with tag = 0 (= Lisp_Symbol
> enumerator), that when untagged are an offset from lispsym (the array of
> built-in symbols), not a pointer to some structure like it is done for
> other Lisp types. That makes the word 0x0 = Qnil, 0x1 = Qt, and so on.
> 
> My question comes from playing with another GC library, where this
> unique encoding is, well, quite inconvenient. So, I'd like to ask two
> questions:
> 
> - Does someone remember the reasons why this has been done?
> 
> - Is there something that relies on this? Maybe something that
>   relies on 0x0 being a valid Lisp_Object?

Paul, can you help us answer these questions?

Re: Question regarding Lisp_Object representation of symbols

[Paul Eggert]

On 2024-03-06 04:14, Eli Zaretskii wrote:
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>
>> - Does someone remember the reasons why this has been done?

Speed. It's significantly more efficient.


>> - Is there something that relies on this? Maybe something that
>>    relies on 0x0 being a valid Lisp_Object?

Yes, these assumptions are present in other places. I tried to mark all 
these places with "verify (NIL_IS_ZERO)" a decade ago, though I wouldn't 
be surprised if I missed one or two at the time, or if other 
dependencies have crept in since then (notably in src/comp.c, which I 
haven't audited for this).

Re: Question regarding Lisp_Object representation of symbols

[Gerd Möllmann]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 2024-03-06 04:14, Eli Zaretskii wrote:
>>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>>
>>> - Does someone remember the reasons why this has been done?
>
> Speed. It's significantly more efficient.
>
>>> - Is there something that relies on this? Maybe something that
>>>    relies on 0x0 being a valid Lisp_Object?
>
> Yes, these assumptions are present in other places. I tried to mark
> all these places with "verify (NIL_IS_ZERO)" a decade ago, though I
> wouldn't be surprised if I missed one or two at the time, or if other
> dependencies have crept in since then (notably in src/comp.c, which I
> haven't audited for this).

Thanks, Paul. I've also looked around a bit yesterday, and my impression
in the end was that making !NIL_IS_ZERO fly could easily be more work
than trying to somehow marry it with the library I'm evaluating.