Re: t and nil in pure memory?

Re: t and nil in pure memory?

[A. Soare]

>    7. Re: t and nil in pure memory? (Richard Stallman)
> 
> ------------------------------
> 
> Message: 7
> Date: Thu, 19 Nov 2009 23:12:55 -0500
> From: Richard Stallman <rms@gnu.org>
> Subject: Re: t and nil in pure memory?
> To: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: dak@gnu.org, emacs-devel@gnu.org
> Message-ID: <E1NBKrf-0005lW-OB@fencepost.gnu.org>
> Content-Type: text/plain; charset=ISO-8859-15
> 
>     As maintainer I care about maintainability of the code.  I will reject
>     such a change.
> 
> 2 lines whose meaning is totally obvious -- not much work to maintain.
> This looks like stubbornness to me.  I have no reason to be stubborn
> about it, so I will move on.
> 
> 

I did run

grep -riI "(put[[:blank:]\n]*t[^a-z]" *   and
grep -riI "(put[[:blank:]\n]*nil" *

in ./lisp directory, and I did find what I expected: nothing.

Could somebody give me an example in which adding properties to `t' and `nil' may be useful?





____________________________________________________

Gagnez un séjour au Maroc en découvrant les sketchs désopilants des ReVoila sur http://www.lesrevoila.fr/

Re: t and nil in pure memory?

[David Kastrup]

"A. Soare" <alinsoar@voila.fr> writes:

> I did run
>
> grep -riI "(put[[:blank:]\n]*t[^a-z]" *   and
> grep -riI "(put[[:blank:]\n]*nil" *
>
> in ./lisp directory, and I did find what I expected: nothing.
>
> Could somebody give me an example in which adding properties to `t'
> and `nil' may be useful?

Well, look in a running Emacs session:

(symbol-plist t) =>
(event-symbol-element-mask (t 0) event-symbol-elements (t) modifier-cache ((0 . t)))

(symbol-plist nil) =>
(event-symbol-element-mask (nil 0) event-symbol-elements (nil) modifier-cache ((0)))

So somebody _is_ tacking properties onto t and nil.

-- 
David Kastrup

t and nil in pure memory?

[Dan Nicolaescu]

The last GC before dumping generates 200K calls to mark_object, of those
20K have Qt or Qnil as an argument.

Would it make sense to put Qt and Qnil in pure memory?

We don't have any function that can create pure symbols.

Moving Qt and Qnil in pure memory would break things like:

(put t 'foo 'bar)

but maybe this works by accident rather than design.

How about other things that have SYMBOL_CONSTANT_P() set to 1?  Can they
go to pure memory?

More generally it would be very good to be able to put symbols in pure
memory, about 1/3 of the mark_object calls are for symbols, and it does
not seem that it's too useful to GC `car', `cdr', etc.  But it does not
seem to be too easy to do :-(

Re: t and nil in pure memory?

[Stefan Monnier]

> The last GC before dumping generates 200K calls to mark_object, of those
> 20K have Qt or Qnil as an argument.

> Would it make sense to put Qt and Qnil in pure memory?

No, because they may contain pointers to objects that aren't in pure
memory (via the plist).

> Moving Qt and Qnil in pure memory would break things like:
> (put t 'foo 'bar)

Exactly.

> but maybe this works by accident rather than design.

No, it's no accident.

> More generally it would be very good to be able to put symbols in pure
> memory, about 1/3 of the mark_object calls are for symbols, and it does
> not seem that it's too useful to GC `car', `cdr', etc.  But it does not
> seem to be too easy to do :-(

Since symbols have mutable fields that can contain pointers, it's
important to let the GC trace through them.

If we want to speed up the GC, then we should look at changing the
algorithm (e.g. make it generational).

Maybe we can also micro-optimize the code to let it run faster: we could
probably make `mark_object' into a macro that first checks the markbit
and only if that fails does it call really_mark_object recursively.
Of course, tests would be needed to try and figure out whether it really
makes things faster.


        Stefan

Re: t and nil in pure memory?

[Tobias C. Rittweiler]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > The last GC before dumping generates 200K calls to mark_object, of those
> > 20K have Qt or Qnil as an argument.
>
> > Would it make sense to put Qt and Qnil in pure memory?
>
> No, because they may contain pointers to objects that aren't in pure
> memory (via the plist).

In principle, you could scratch the plist slot from symbols themselves,
and instead make `symbol-plist' go over an EQ hash-table. (Such
externalizing of the symbol plist may make more sense in other dialects
of Lisp where the plist is used less often than in Elisp -- even though
I'm not aware of any implementation which does.)

  -T.

Re: t and nil in pure memory?

[Stefan Monnier]

>> > The last GC before dumping generates 200K calls to mark_object, of those
>> > 20K have Qt or Qnil as an argument.
>> 
>> > Would it make sense to put Qt and Qnil in pure memory?
>> 
>> No, because they may contain pointers to objects that aren't in pure
>> memory (via the plist).

> In principle, you could scratch the plist slot from symbols themselves,
> and instead make `symbol-plist' go over an EQ hash-table. (Such
> externalizing of the symbol plist may make more sense in other dialects
> of Lisp where the plist is used less often than in Elisp -- even though
> I'm not aware of any implementation which does.)

Actually, it's even worse than that: (interned) symbols have a `next'
field which points to the next symbol in the obarray (hash) bucket.
So that's yet another reason why (pure)symbols can have pointers to non
pure memory.


        Stefan

Re: t and nil in pure memory?

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

  > >> > The last GC before dumping generates 200K calls to mark_object, of those
  > >> > 20K have Qt or Qnil as an argument.
  > >> 
  > >> > Would it make sense to put Qt and Qnil in pure memory?
  > >> 
  > >> No, because they may contain pointers to objects that aren't in pure
  > >> memory (via the plist).
  > 
  > > In principle, you could scratch the plist slot from symbols themselves,
  > > and instead make `symbol-plist' go over an EQ hash-table. (Such
  > > externalizing of the symbol plist may make more sense in other dialects
  > > of Lisp where the plist is used less often than in Elisp -- even though
  > > I'm not aware of any implementation which does.)
  > 
  > Actually, it's even worse than that: (interned) symbols have a `next'
  > field which points to the next symbol in the obarray (hash) bucket.
  > So that's yet another reason why (pure)symbols can have pointers to non
  > pure memory.

Can't the next pointers could be NULL.
Is the next pointer used for anything else other than GC?
Because if it's not, then it seems that if the next pointer was stored
in the obarray the cache footprint of a Lisp_Symbol would be smaller.

Re: t and nil in pure memory?

[Stefan Monnier]

>> Actually, it's even worse than that: (interned) symbols have a `next'
>> field which points to the next symbol in the obarray (hash) bucket.
>> So that's yet another reason why (pure)symbols can have pointers to non
>> pure memory.
> Can't the next pointers could be NULL.

Of course: the last symbol in a bucket has a NULL there (as do
non-interned symbols).  Can't see why that would make a difference, tho.

> Is the next pointer used for anything else other than GC?

Yes, of course: it's used for the hash-lookup done in `intern' (and
`intern-soft').

> Because if it's not, then it seems that if the next pointer was stored
> in the obarray the cache footprint of a Lisp_Symbol would be smaller.

You can remove the `next' pointer to make symbols smaller (tho on 32bit
machines it would only make them smaller if you additionally remove
another field (like plist), because of the alignment constraint).
But then you need to link all interned symbols via an (external) list,
and since most symbols are interned (AFAIK), that would end up adding
more memory than is saved.  Admittedly, it could still result in
a smaller cache footprint.  It's just far from obvious that there's much
to gain here.

I still strongly believe there's more to be gained by optimizing the
tight mark_object (recursive) loop than by trying to squeeze a few more
bytes into the purespace: once your Emacs session grows sufficiently
large, the saving of not tracing through the purespace
becomes negligible.


        Stefan

Re: t and nil in pure memory?

[Richard Stallman]

    You can remove the `next' pointer to make symbols smaller (tho on 32bit
    machines

Do you mean 64-bit machines?

	     it would only make them smaller if you additionally remove
    another field (like plist), because of the alignment constraint).

That's because of 5 bits.  If we could get rid of them, or move them somewhere,
we could reduce the size by 64 bits.

One way to avoid the next pointer is by storing all the symbol
pointers directly in the obarray, and making the obarray larger when
it gets close to full.  That's not traditional for Lisp obarrays but
it might be ok.  That would replace N words in symbols with maybe
n*1.3 words in the obarray.

Then all that is needed is to move those 5 bits somewhere, perhaps
packing them into the pointers in a symbol in some kludgy way.

Anyway, it is easy to store the plists for t and nil in
a couple of C variables, without changing the format of symbols.
This requires a little special code in the plist access functions,
but maybe the time that costs will be less than what is saved in GC.

As for their next pointers, if they are the first symbols to be
interned, they are at the end of their obarray buckets, so their
next pointers never need to be changed.  (We should make `unintern'
check for t and nil and give an error.)

Re: t and nil in pure memory?

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Actually, it's even worse than that: (interned) symbols have a `next'
>>> field which points to the next symbol in the obarray (hash) bucket.
>>> So that's yet another reason why (pure)symbols can have pointers to non
>>> pure memory.
>> Can't the next pointers could be NULL.
>
> Of course: the last symbol in a bucket has a NULL there (as do
> non-interned symbols).  Can't see why that would make a difference, tho.
>
>> Is the next pointer used for anything else other than GC?
>
> Yes, of course: it's used for the hash-lookup done in `intern' (and
> `intern-soft').

If one sorts the pure symbols to the end of the obarray bucket chain
before dumping, they can't possibly be followed by non-pure symbols in
their life time since intern adds to the front of a bucket chain.

-- 
David Kastrup

Re: t and nil in pure memory?

[Stefan Monnier]

> If one sorts the pure symbols to the end of the obarray bucket chain
> before dumping, they can't possibly be followed by non-pure symbols in
> their life time since intern adds to the front of a bucket chain.

Except when you call unintern, of course.
No, I'm just not willing to go there.  Too much trouble, too little gain.


        Stefan

Re: t and nil in pure memory?

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> If one sorts the pure symbols to the end of the obarray bucket chain
>> before dumping, they can't possibly be followed by non-pure symbols in
>> their life time since intern adds to the front of a bucket chain.
>
> Except when you call unintern, of course.
> No, I'm just not willing to go there.  Too much trouble, too little gain.

I am skeptical that we have everything in place that is required to
properly support

(unintern nil)
(unintern t)

right now.

-- 
David Kastrup

Re: t and nil in pure memory?

[Richard Stallman]

    I am skeptical that we have everything in place that is required to
    properly support

    (unintern nil)
    (unintern t)

We should make them give errors -- that's much easier than
making them "work".  There's no reason they should be allowed.

Re: t and nil in pure memory?

[Stefan Monnier]

>     I am skeptical that we have everything in place that is required to
>     properly support

>     (unintern nil)
>     (unintern t)

> We should make them give errors -- that's much easier than
> making them "work".  There's no reason they should be allowed.

Rather I think there's no reason to spend any time either making them
work or making them signal an error.


        Stefan

Re: t and nil in pure memory?

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>     I am skeptical that we have everything in place that is required to
>>     properly support
>
>>     (unintern nil)
>>     (unintern t)
>
>> We should make them give errors -- that's much easier than
>> making them "work".  There's no reason they should be allowed.
>
> Rather I think there's no reason to spend any time either making them
> work or making them signal an error.

Well, you claimed that uninterning them was a concern for having the
obarray links end in pure memory.

-- 
David Kastrup

Re: t and nil in pure memory?

[Richard Stallman]

I will spend the time to make unintern give an error for t and nil.

Re: t and nil in pure memory?

[Stefan Monnier]

> I will spend the time to make unintern give an error for t and nil.

Please don't.  It's not just your time, it's the CPU time of everyone.
There are *many* ways for the user to shoot herself in the foot and make
her Emacs session completely unusable.  We're not going to start and try
and catch every one of them.  It's a waste of everyone's time and energy.


        Stefan

Re: t and nil in pure memory?

[Daniel Colascione]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefan Monnier wrote:
>> I will spend the time to make unintern give an error for t and nil.
> 
> Please don't.  It's not just your time, it's the CPU time of everyone.
> There are *many* ways for the user to shoot herself in the foot and make
> her Emacs session completely unusable.  We're not going to start and try
> and catch every one of them.  It's a waste of everyone's time and energy.

unintern isn't exactly on the fast path.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAksENIYACgkQ17c2LVA10VvpjgCg1p/JWAaXbxZyjTbOKoUVIkPB
nqkAn2rXjjRSbYmDpuCHfDJaqngTnwFG
=EYUk
-----END PGP SIGNATURE-----

Re: t and nil in pure memory?

[Stefan Monnier]

>>> I will spend the time to make unintern give an error for t and nil.
>> Please don't.  It's not just your time, it's the CPU time of everyone.
>> There are *many* ways for the user to shoot herself in the foot and make
>> her Emacs session completely unusable.  We're not going to start and try
>> and catch every one of them.  It's a waste of everyone's time and energy.
> unintern isn't exactly on the fast path.

I don't care if it is.  It's extra code to compile, load, run, and
maintain, and no extra feature for the user.  I.e. pure bloat.


        Stefan

Re: t and nil in pure memory?

[Richard Stallman]

    There are *many* ways for the user to shoot herself in the foot and make
    her Emacs session completely unusable.

We won't try all, but it is useful to catch some.  unintern is not
used a lot, so don't worry about the cpu time.

Re: t and nil in pure memory?

[Stefan Monnier]

>     There are *many* ways for the user to shoot herself in the foot and make
>     her Emacs session completely unusable.

> We won't try all, but it is useful to catch some.  unintern is not
> used a lot, so don't worry about the cpu time.

As maintainer I care about maintainability of the code.  I will reject
such a change.


        Stefan

Re: t and nil in pure memory?

[Richard Stallman]

    As maintainer I care about maintainability of the code.  I will reject
    such a change.

2 lines whose meaning is totally obvious -- not much work to maintain.
This looks like stubbornness to me.  I have no reason to be stubborn
about it, so I will move on.

Re: t and nil in pure memory?

[Chong Yidong]

Richard Stallman <rms@gnu.org> writes:

>     As maintainer I care about maintainability of the code.  I will reject
>     such a change.
>
> 2 lines whose meaning is totally obvious -- not much work to maintain.
> This looks like stubbornness to me.  I have no reason to be stubborn
> about it, so I will move on.

(I sent an earlier email about this, but it seems to have gotten lost.)

FWIW, in CLISP and CMUCL (unintern nil) and (unintern t) are no-ops
returning nil.  Presumably the other CL implementations do likewise.

It seems to me that it'd do no harm to do likewise, but since others
care strongly about this and I don't, I'll just shrug and go along...

Re: t and nil in pure memory?

[Sam Steingold]

> * Chong Yidong <plq@fghcvqpuvpxra.pbz> [2009-11-20 00:47:18 -0500]:
>
> FWIW, in CLISP and CMUCL (unintern nil) and (unintern t) are no-ops
> returning nil.  Presumably the other CL implementations do likewise.

This is not true about CLISP (and presumably other CL implementations).

[1]> (unintern cl:t)
NIL

This is indeed a noop because the default package is CL-USER and T lived
in CL.

[2]> (unintern cl:t :cl)

** - Continuable Error
UNINTERN(T): #<PACKAGE COMMON-LISP> is locked
If you continue (by typing 'continue'): Ignore the lock and proceed
The following restarts are also available:
ABORT          :R1      Abort main loop
Break 1 [3]> continue
#:T
USER[4]> (find-symbol "T" "CL")
NIL ;
NIL

i.e., it IS possible to remove the T symbol (with all the trouble this
entails).

As a CLISP maintainer, I don't think T & NIL are special enough.
We lock entire packages.
Please see <http://clisp.cons.org/impnotes/pack-lock.html>.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 9.04 (jaunty)
http://iris.org.il http://camera.org http://jihadwatch.org http://pmw.org.il
http://openvotingconsortium.org http://ffii.org http://mideasttruth.com
Computers are like air conditioners: they don't work with open windows!

Re: t and nil in pure memory?

[David Kastrup]

Sam Steingold <sds@gnu.org> writes:

>> * Chong Yidong <plq@fghcvqpuvpxra.pbz> [2009-11-20 00:47:18 -0500]:
>>
>> FWIW, in CLISP and CMUCL (unintern nil) and (unintern t) are no-ops
>> returning nil.  Presumably the other CL implementations do likewise.
>
> This is not true about CLISP (and presumably other CL implementations).
>
> [2]> (unintern cl:t :cl)
>
> ** - Continuable Error
>
> i.e., it IS possible to remove the T symbol (with all the trouble this
> entails).
>
> As a CLISP maintainer, I don't think T & NIL are special enough.

I am quite appalled to what my reductio ad absurdum argument has lead.
The original topic was in the thread title, and I pointed out that
obarray chains ending in pure memory do not need further modification.
Then it was pointed out that unintern may change the game in that
respect, and I replied that I did not think this much of a concern for t
and nil.

From then things went downhill.

-- 
David Kastrup

Re: t and nil in pure memory?

[Dan Nicolaescu]

Richard Stallman <rms@gnu.org> writes:

  >     There are *many* ways for the user to shoot herself in the foot and make
  >     her Emacs session completely unusable.
  > 
  > We won't try all, but it is useful to catch some.  unintern is not
  > used a lot, so don't worry about the cpu time.

Please don't do it, it does not help ANYTHING.  
This is just one minor way people can shoot themselves in a foot, but
compared to what you can do with, say, "fset" it's just a joke.

The original proposal was to make t and nil constant, with the thought
that this will provide some roadmap for being able to make a lot of
symbols constant, and that would have a real life speed benefit for 
speeding up GC.

But not being able to unintern t and nil is not useful to anyone.

Re: t and nil in pure memory?

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

  > I still strongly believe there's more to be gained by optimizing the
  > tight mark_object (recursive) 

Like this:

#define MAYBE_MARK_OBJECT(ARG)				\
  do {							\
    Lisp_Object OBJ = ARG;				\
    							\
    if ((XTYPE (OBJ) == Lisp_Symbol))			\
      {							\
	struct Lisp_Symbol *tptr = XSYMBOL (OBJ);	\
	if (!tptr->gcmarkbit)				\
	  mark_object (OBJ);				\
      }							\
    else						\
      mark_object (OBJ);				\
} while (0)

and replace all the recursive mark_object calls with MAYBE_MARK_OBJECT ?

The one testcase I tried (indenting config.h) does not benefit from this.

The big problem in GC is the trashing of the memory system due to
setting/resetting and reading the mark bits...

Re: t and nil in pure memory?

[Stefan Monnier]

> #define MAYBE_MARK_OBJECT(ARG)				\
>   do {							\
>     Lisp_Object OBJ = ARG;				\
>     							\
>     if ((XTYPE (OBJ) == Lisp_Symbol))			\

Please don't use this.  Always use SYMBOLP instead.

>       {							\
> 	struct Lisp_Symbol *tptr = XSYMBOL (OBJ);	\
> 	if (!tptr->gcmarkbit)				\
> 	  mark_object (OBJ);				\
>       }							\
>     else						\
>       mark_object (OBJ);				\
> } while (0)

This may save a few function calls for some symbols, but adds a test for
all other cases, so it's unlikely to win much if anything.

I think some of the needed changes to streamline the code would be to
remove the PUREP test (i.e. just always keep the mark bit set on pure
objects instead.  This will require changing the way we handle the
markbit on strings and vectors).

Maybe we'd also want to unify the code that extracts the markbit for the
various possible objects, so that we can check the markbit before we
dispatch on the type of object we're looking at.  That would be an even
more drastic change.

> The big problem in GC is the trashing of the memory system due to
> setting/resetting and reading the mark bits...

Could be as well.  In that case, maybe we should move all the markbits
to separate bitvectors like we have for cons and floats.


        Stefan

Re: t and nil in pure memory?

[Ken Raeburn]

On Nov 15, 2009, at 20:34, Stefan Monnier wrote:
>> The big problem in GC is the trashing of the memory system due to
>> setting/resetting and reading the mark bits...
>
> Could be as well.  In that case, maybe we should move all the markbits
> to separate bitvectors like we have for cons and floats.

There are lots of changes, major or minor, that might be worth  
exploring.  Prefetching, putting mark bits on different words or  
pages, tri-color schemes, incremental marking, generational GC, etc.,  
etc.  But maybe we should also consider a more drastic change: Use  
Hans Boehm's GC library.  (Guile has recently switched to using it.   
So if my Guile-Emacs work ever catches on, use of bdw-gc would happen  
as a side effect.)  If it doesn't simply make the performance someone  
else's problem to focus on, it at least lets us share the workload  
with a wider community.

Ken

Re: t and nil in pure memory?

[Daniel Colascione]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ken Raeburn wrote:
> Use Hans Boehm's GC

Conservative GCs scare the bejesus out of me. Their chief attraction is
that you can wedge them into a system not designed for GC and they'll
mostly work. But there's already a precise garbage collector, and I
don't think it's worthwhile (or safe) to get rid of it.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAksERPwACgkQ17c2LVA10VsUYQCdEY1zQrNgiVJxYP/IV8jbzS9K
cvoAoLNqE/oNkNV11QgPz8QV47yBFvOy
=0TVE
-----END PGP SIGNATURE-----

Re: t and nil in pure memory?

[Tom Tromey]

>>>>> "Daniel" == Daniel Colascione <daniel@censorshipresearch.org> writes:

Daniel> Ken Raeburn wrote:
>> Use Hans Boehm's GC

Daniel> Conservative GCs scare the bejesus out of me. Their chief attraction is
Daniel> that you can wedge them into a system not designed for GC and they'll
Daniel> mostly work. But there's already a precise garbage collector, and I
Daniel> don't think it's worthwhile (or safe) to get rid of it.

The Boehm GC can do precise marking of the heap.
On some platforms, Emacs already does conservative marking of the stack.
So, on these platforms, using the Boehm GC does not necessarily imply
any real change in conservatism.

Tom

Re: t and nil in pure memory?

[Tom Tromey]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

Ken> But maybe we should also consider a more drastic change: Use Hans
Ken> Boehm's GC library.

FWIW, there's already a branch for this.  Apparently the work was never
completed, though.  There's also a bit of traffic about it in the
emacs-devel archives.

Tom

Re: t and nil in pure memory?

[Ken Raeburn]

On Nov 18, 2009, at 14:13, Tom Tromey wrote:
>>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:
>
> Ken> But maybe we should also consider a more drastic change: Use Hans
> Ken> Boehm's GC library.
>
> FWIW, there's already a branch for this.  Apparently the work was  
> never
> completed, though.

Ah, yes, so there is...  I'd forgotten about that, thanks.

It occurs to me that -- again, if my guile-emacs work catches on --  
some of the remaining storage management code will probably have to  
change to be boehm-gc compatible.  Maybe I should see what it would  
take to update Dave's work to the current source base, as yet another  
project that may have its uses independent of the guile work.

>  There's also a bit of traffic about it in the
> emacs-devel archives.

I'm looking now... though most of it seems to be "maybe we should  
consider doing this" type traffic.  There's a report that a "very  
limited experiment" with XEmacs found it to be slower, but not much  
analysis of why.  And, it (the email report) was seven years ago;  
libgc may have improved.

Ken

Re: t and nil in pure memory?

[Stephen J. Turnbull]

Executive summary:

YMMV but my take is that the XEmacs experiment "failed" due to social
reasons, not technical ones.

Ken Raeburn writes:

 > I'm looking now... though most of it seems to be "maybe we should  
 > consider doing this" type traffic.  There's a report that a "very  
 > limited experiment" with XEmacs found it to be slower, but not much  
 > analysis of why.

That measurement was based on profiling and guessing at the actual
size of the slowdown.  The correct interpretation of "slowdown" is
that it was due to the overhead imposed by the precise collector,
which had not been removed, merely disabled.  I believe the reasons
that the experiment was not extended were

1.  It wasn't believed to be enough *faster* to justify the effort,
    believed to be substantial, to remove the existing collector's
    overhead code, and to analyze places where more precise collection
    would be apropriate (at that time the version of Boehm GC in use
    wasn't very helpful in that respect IIRC).
2.  The VM footprint of a long-running process increased by about 20%
    and of the RSS by about 40%, meaning more effort to tune Boehm GC;
    those were pretty significant numbers back then when XEmacs
    expanded to "Ten times Eight MB And Constantly Swapping".
3.  Mike Sperber had a student who wanted to work on a pluggable GC
    concept which would allow experimenting with different GC
    systems (which evolved into today's --kkcc and --newgc options to
    configure in XEmacs).

If you need more details I could go back and look it up, but I really
don't think there's any useful information to be had.  The experiment
was no joke, but it was very limited, just as you wrote.

Re: t and nil in pure memory?

[Ken Raeburn]

On Nov 23, 2009, at 00:31, Stephen J. Turnbull wrote:
> If you need more details I could go back and look it up, but I really
> don't think there's any useful information to be had.  The experiment
> was no joke, but it was very limited, just as you wrote.

Thanks for the info.  It sounds like maybe I should go ahead and  
investigate further...

Ken

Re: t and nil in pure memory?

[Stefan Monnier]

>> If you need more details I could go back and look it up, but I really
>> don't think there's any useful information to be had.  The experiment
>> was no joke, but it was very limited, just as you wrote.
> Thanks for the info.  It sounds like maybe I should go ahead and investigate
> further...

IIUC the Boehm GC has a fair bit of knobs and tricks to tune it for
particular uses, so I expect it can perform reasonably well for us, but
it will require tuning.  E.g. it'll require teaching the Boehm GC how to
precisely mark our objects.


        Stefan

Re: t and nil in pure memory?

[Stefan Monnier]

> There are lots of changes, major or minor, that might be worth exploring.
> Prefetching, putting mark bits on different words or  pages, tri-color
> schemes, incremental marking, generational GC, etc.,  etc.  But maybe we
> should also consider a more drastic change: Use  Hans Boehm's GC library.

It's definitely worth exploring.  IIRC the main problem with it was how
to get the necessary pointer-weakness where we expect it (hash-tables,
markers, undo-lists, and a couple others I can't remember), and of
course add the relevant hooks to get back the GC-precision we currently
enjoy.  Oh, and of course, we'd need to compare the performance (memory
use and CPU use, mostly) to make sure it doesn't get worse.  I'd expect
the memory use to go up slightly, not sure about the performance.
It likely would improve our vector-allocation performance (which
currently sucks).
Also we'd need to decide what to do about the platforms not supported by
the Boehm GC.


        Stefan

Re: t and nil in pure memory?

[Eli Zaretskii]

> Date: Wed, 11 Nov 2009 12:23:53 -0800 (PST)
> From: Dan Nicolaescu <dann@ics.uci.edu>
> 
> 
> The last GC before dumping generates 200K calls to mark_object, of those
> 20K have Qt or Qnil as an argument.

I don't think this matters enough to bother: even if we consider GC
too slow (and I don't think so), this is only 10% of it, so even
reducing it to zero will give only a marginal speedup.

Re: t and nil in pure memory?

[Richard Stallman]

    I don't think this matters enough to bother: even if we consider GC
    too slow (and I don't think so), this is only 10% of it, so even
    reducing it to zero will give only a marginal speedup.

Considering how many people run Emacs, I think even a 1% speedup
in GC is worth the effort if it is not much effort.

Re: t and nil in pure memory?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: dann@ics.uci.edu, emacs-devel@gnu.org
> Date: Thu, 12 Nov 2009 23:55:43 -0500
> 
>     I don't think this matters enough to bother: even if we consider GC
>     too slow (and I don't think so), this is only 10% of it, so even
>     reducing it to zero will give only a marginal speedup.
> 
> Considering how many people run Emacs, I think even a 1% speedup
> in GC is worth the effort if it is not much effort.

No matter how many people use Emacs, the 1% speedup will still save
only 1% of the time for each one of them.

Re: t and nil in pure memory?

[Richard Stallman]

    No matter how many people use Emacs, the 1% speedup will still save
    only 1% of the time for each one of them.

Sure, but even if it is only a small improvement for each person, it
is still worth doing if it is easy.  Suppose that this small speedup
transforms "annoying" into "not annoying" for just one user in a
thousand each day.  That would mean that, each day, hundreds or thousands
of people are happier.

Re: t and nil in pure memory?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: dann@ics.uci.edu, emacs-devel@gnu.org
> Date: Sat, 14 Nov 2009 06:23:16 -0500
> 
>     No matter how many people use Emacs, the 1% speedup will still save
>     only 1% of the time for each one of them.
> 
> Sure, but even if it is only a small improvement for each person, it
> is still worth doing if it is easy.  Suppose that this small speedup
> transforms "annoying" into "not annoying" for just one user in a
> thousand each day.  That would mean that, each day, hundreds or thousands
> of people are happier.

But a 1% speedup will not transform "annoying" into "not annoying",
and thus will make no one happier.

Re: t and nil in pure memory?

[Richard Stallman]

    But a 1% speedup will not transform "annoying" into "not annoying",
    and thus will make no one happier.

Once in a while, this small speedup will be enough to make that
difference for a particular person.  With so many users, it adds up.
Also, several such small speedups do add up.

Re: t and nil in pure memory?

[Richard Stallman]

    Would it make sense to put Qt and Qnil in pure memory?

That isn't valid.  The property lists of t and nil have to be writable.

We could store the property lists of t and nil in a special way.  Then
they could be pure, as long as we make sure their oblist links never
need to be changed.