CANNOT_DUMP support

CANNOT_DUMP support

[Chris Hall]

I know that GNUstep is a CANNOT_DUMP platform, and I have been 
informed that there aren't many others left.  Is there a list 
somewhere that I could view?

Also, what are the plans with regard to CANNOT_DUMP, if any, e.g. 
"continue to support indefinitely"?

Thanks,

Chris Hall

Re: CANNOT_DUMP support

[Dan Nicolaescu]

"Chris Hall" <chris@web.workinglinux.com> writes:

  > I know that GNUstep is a CANNOT_DUMP platform, and I have been
  > informed that there aren't many others left.  Is there a list
  > somewhere that I could view?

No list, just grep emacs/src/m/*

Re: CANNOT_DUMP support

[Chris Hall]

On 2008-02-10 07:07:14 -1000 Dan Nicolaescu <dann@ics.uci.edu> wrote:

> "Chris Hall" <chris@web.workinglinux.com> writes:
> 
>   > I know that GNUstep is a CANNOT_DUMP platform, and I have been
>   > informed that there aren't many others left.  Is there a list
>   > somewhere that I could view?
> 
> No list, just grep emacs/src/m/*
> 

Ah.  Thanks.

Hmm.  System/390, amdx86-64, ia-64, m68k.  So CANNOT_DUMP support 
appears likely for a while yet, I would think.

Re: CANNOT_DUMP support

[Dan Nicolaescu]

"Chris Hall" <chris@web.workinglinux.com> writes:

  > On 2008-02-10 07:07:14 -1000 Dan Nicolaescu <dann@ics.uci.edu> wrote:
  > 
  > > "Chris Hall" <chris@web.workinglinux.com> writes:
  > >
  > >   > I know that GNUstep is a CANNOT_DUMP platform, and I have been
  > >   > informed that there aren't many others left.  Is there a list
  > >   > somewhere that I could view?
  > >
  > > No list, just grep emacs/src/m/*
  > >
  > 
  > Ah.  Thanks.
  > 
  > Hmm.  System/390, amdx86-64, ia-64, m68k.  

Nope, look again, all those are either commented out or inside #if 0.

The only remaining one is in ibmrs6000.h when using sysV. I don't know
of any such system in current use ...

Re: CANNOT_DUMP support

[Stefan Monnier]

>> >   > I know that GNUstep is a CANNOT_DUMP platform, and I have been
>> >   > informed that there aren't many others left.  Is there a list
>> >   > somewhere that I could view?
>> >
>> > No list, just grep emacs/src/m/*
>> 
>> Ah.  Thanks.
>> 
>> Hmm.  System/390, amdx86-64, ia-64, m68k.  

> Nope, look again, all those are either commented out or inside #if 0.

> The only remaining one is in ibmrs6000.h when using sysV. I don't know
> of any such system in current use ...

Yup: CANNOT_DUMP is not on the way out, but it's only ever seriously
used for targets that are "in development".
We need to get dumping to work on GNUstep before we can consider it as
"ready for prime time" (tho I don't consider it as a prerequisite for
being on the trunk).


        Stefan

Re: CANNOT_DUMP support

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

  > >> >   > I know that GNUstep is a CANNOT_DUMP platform, and I have been
  > >> >   > informed that there aren't many others left.  Is there a list
  > >> >   > somewhere that I could view?
  > >> >
  > >> > No list, just grep emacs/src/m/*
  > >> 
  > >> Ah.  Thanks.
  > >> 
  > >> Hmm.  System/390, amdx86-64, ia-64, m68k.  
  > 
  > > Nope, look again, all those are either commented out or inside #if 0.
  > 
  > > The only remaining one is in ibmrs6000.h when using sysV. I don't know
  > > of any such system in current use ...
  > 
  > Yup: CANNOT_DUMP is not on the way out, but it's only ever seriously
  > used for targets that are "in development".
  > We need to get dumping to work on GNUstep before we can consider it as
  > "ready for prime time" (tho I don't consider it as a prerequisite for
  > being on the trunk).

Has there been any decision about merging that code?  It would probably
be better to merge it sooner rather than later...

Re: CANNOT_DUMP support

[Chris Hall]

On 2008-02-10 18:33:53 -1000 Dan Nicolaescu <dann@ics.uci.edu> wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
>   > >> >   > I know that GNUstep is a CANNOT_DUMP platform, and I 
> have been
>   > >> >   > informed that there aren't many others left.  Is there a 
> list
>   > >> >   > somewhere that I could view?
>   > >> >
>   > >> > No list, just grep emacs/src/m/*
>   > >>  > >> Ah.  Thanks.
>   > >>  > >> Hmm.  System/390, amdx86-64, ia-64, m68k.   >  > > Nope, 
> look 
> again, all those are either commented out or inside #if 0.
>   >  > > The only remaining one is in ibmrs6000.h when using sysV. I 
> don't know
>   > > of any such system in current use ...
>   >  > Yup: CANNOT_DUMP is not on the way out, but it's only ever 
> seriously
>   > used for targets that are "in development".
>   > We need to get dumping to work on GNUstep before we can consider 
> it as
>   > "ready for prime time" (tho I don't consider it as a prerequisite 
> for
>   > being on the trunk).
> 
> Has there been any decision about merging that code?  It would 
> probably
> be better to merge it sooner rather than later...
> 
Well, its been a long time since I've worked at the level unexec 
operates at, and even then it was on System/360 and Series/1, but it 
looks like an interesting problem.

At first glance I imagine dealing with late-binding in Objective C is 
one of the challenges in dumping on GNUstep platforms?

Anyway, at the moment I'm more interested in getting the GNUstep 
Emacs.app working a bit better, so I'm trying to get my mind around 
the multi-tty stuff.

I've found  http://lorentey.hu/project/emacs.html.hu very useful in 
getting started on that.

Thanks for all the useful responses, everybody.

Re: CANNOT_DUMP support

[Stephen J. Turnbull]

Chris Hall writes:

 > Well, its been a long time since I've worked at the level unexec 
 > operates at, and even then it was on System/360 and Series/1, but it 
 > looks like an interesting problem.

XEmacs has not had an unexec issue since 2001, because of the
introduction of the "portable dumper".  The portable dumper
transparently ignored the "Hannibal Lector" ld and (mostly) the "Chaos
Incarnate" SELinux loader.  It also has made preload support for new
platforms trivial.

The authoritative voice on that module is Olivier Galibert
<olivier.galibert@xemacs.org>, if the idea interests you.  The code
may even be legally[1] usable, as the number of contributors is very
small and all (except maybe Kyle Jones) would probably sign
assignments.  (It does depend on the layout and types of builtin Lisp
objects, so probably it wouldn't help all that much, though.)


Footnotes: 
[1]  By which I mean that it will satisfy the GNU Emacs project's
requirements for provenance and defensibility, not that it's under
some kind of non-GPL-compatible license.

Re: CANNOT_DUMP support

[Richard Stallman]

    XEmacs has not had an unexec issue since 2001, because of the
    introduction of the "portable dumper".

It sounds interesting.  

					    The portable dumper
    transparently ignored the "Hannibal Lector" ld and (mostly) the "Chaos
    Incarnate" SELinux loader.  It also has made preload support for new
    platforms trivial.

I am completely lost.  It sounds like a statement of opinion about ld
(I know what that does, at least) and about the SELinux loader (which
I know nothing about).  But it is stated in such a witty way that I
can't tell what opinion it is intended to express.

Can you tell us more about how this works and what it does?

    may even be legally[1] usable, as the number of contributors is very
    small and all (except maybe Kyle Jones) would probably sign
    assignments.

I don't think we even have contact with him any more.
Do you?  Supposing he won't sign, how hard is it to rip out
and replace his code?

      (It does depend on the layout and types of builtin Lisp
    objects, so probably it wouldn't help all that much, though.)

Do you mean that it would be hard to adapt it for the differences
in those layouts?

Could we adapt the idea, at least?

Re: CANNOT_DUMP support

[Stephen J. Turnbull]

Richard Stallman writes:

 >     XEmacs has not had an unexec issue since 2001, because of the
 >     introduction of the "portable dumper".
 > 
 > It sounds interesting.  
 > 
 >     The portable dumper transparently ignored the "Hannibal Lector"
 >     ld and (mostly) the "Chaos Incarnate" SELinux loader.  It also
 >     has made preload support for new platforms trivial.

The "Hannibal Lector" ld was an innovation that optimized the order of
ELF sections (?) in X?Emacs's executable, causing the unexec
computations to be incorrect (the end of code marker wasn't beginning
of data or something like that).  The SELinux loader randomly changes
the position of the programs sections at load time to make buffer
overrun exploits harder (or something like that, I'm not an SELinux
geek).  I'm not sure if we've addressed that yet, I think the portable
dumper did fail with that setting in SELinux.  ISTR that Olivier said
it could be handled though.

 > Can you tell us more about how this works and what it does?

The basic idea is that we load up Emacs with its Lisp, then start with
the GC roots and "wire up" (for technical details, ask Olivier) the
objects referenced with offsets, then write them to a file.  At
runtime, loading the file does the reverse.  I'm not sure how this is
done, whether the portable dumper actually traces all the offsets and
converts them to appropriate pointers at runtime, or whether the
system loader does this as part of its normal ELF relocation link
editing.  Whatever, it's not as fast as unexec, but it's real fast.

Of course you want to put the "file" into the emacs binary.  This has
been done but there were some minor issues.  I don't recall whether it
was lack of time by those who know about the feature or a real problem
that was solved.

 >     may even be legally[1] usable, as the number of contributors is
 >     very small and all (except maybe Kyle Jones) would probably
 >     sign assignments.
 > 
 > I don't think we even have contact with him any more.
 > Do you?

Not directly, but the VM maintainers do.

 > Supposing he won't sign, how hard is it to rip out and replace his
 > code?

You'll have to ask Olivier about issues that have to do with what is
where in the code.

 >       (It does depend on the layout and types of builtin Lisp
 >     objects, so probably it wouldn't help all that much, though.)
 > 
 > Do you mean that it would be hard to adapt it for the differences
 > in those layouts?

The portable dumper and runtime loader itself is a fairly small amount
of code.  Instead, each class of Lisp object gets its own marshalling
routines because of the various ways that data is incorporated.  Eg,
conses are trivial to save but you have to follow car and cdr.
Strings are a mess because they refer both to Lisp objects (any text
properties) and the string storage itself is not contiguous to the
string object and has to be described ad hoc.  And so on and so forth.

I don't think it's "hard", just tedious.

 > Could we adapt the idea, at least?

Yes; it's inherent in the pointer discipline in the Emacs Lisp
implementation, which (at this level) we share.  I'm sure it's been
proposed for Emacs, too.  What I'm contributing here more than
anything is (a) yes, it works, we've done it, and (b) a pointer to
Olivier who can tell you how much work is involved and probably will
consult on implementation (depending on his personal time
constraints).

Re: CANNOT_DUMP support

[Richard Stallman]

    The basic idea is that we load up Emacs with its Lisp, then start with
    the GC roots and "wire up" (for technical details, ask Olivier) the
    objects referenced with offsets, then write them to a file.  At
    runtime, loading the file does the reverse.  I'm not sure how this is
    done, whether the portable dumper actually traces all the offsets and
    converts them to appropriate pointers at runtime, or whether the
    system loader does this as part of its normal ELF relocation link
    editing.  Whatever, it's not as fast as unexec, but it's real fast.

Lisp objects are just part of what unexec dumps.  There is other
malloc data, and lots of global variables that don't point to Lisp
objects.  How do you handle them?

Re: CANNOT_DUMP support

[Stephen J. Turnbull]

Richard Stallman writes:

 > Lisp objects are just part of what unexec dumps.  There is other
 > malloc data, and lots of global variables that don't point to Lisp
 > objects.  How do you handle them?

True globals live in the executable itself ini the usual way, as far
as I know.  Other malloc data needs to be characterized to the dumper
in a way similar to the way that Lisp data is characterized to the GC.
Beyond that, I don't know; I just know that we do, successfully.

If somebody wants to follow up, Olivier Galibert knows how.

Re: CANNOT_DUMP support

[Chris Hall]

On 2008-02-13 20:47:46 -1000 Stephen J. Turnbull <stephen@xemacs.org> 
wrote:

> 
> If somebody wants to follow up, Olivier Galibert knows how.
> 

This page at the XEmacs web site:

http://www.xemacs.org/Releases/Public-21.2/projects/pdump.html

appears to have been written by one Olivier Galibert, and includes the 
following:

"The portable dumper executes Emacs as usual, then reloads the Lisp 
code and data very quickly from a specially formatted dump file. This 
is not as fast as the unexec method, but it requires much less 
knowledge of the operating system, and is thus more portable and 
maintainable.
... it is a distinct improvement over the traditional dump process on 
the NT platform.
... this feature is considered extremely important for the Windows 
platform, and truly reverting this feature to the status quo ante 
would require reimplementing pure space,"

Then under "Other open issues" (although it appears to be the only 
open issue):

"The pdumper has cost us ``pure space.'' this bothers Hrvoje, at 
least."

The implications of this are beyond my ken, but I assume pure space 
exists for a reason, so there exists a high probability of 
implications.

Re: CANNOT_DUMP support

[Kenichi Handa]

In article <87myq46nl9.fsf@uwakimon.sk.tsukuba.ac.jp>, "Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
> Lisp objects are just part of what unexec dumps.  There is other
> malloc data, and lots of global variables that don't point to Lisp
> objects.  How do you handle them?

> True globals live in the executable itself ini the usual way, as far
> as I know.  Other malloc data needs to be characterized to the dumper
> in a way similar to the way that Lisp data is characterized to the GC.
> Beyond that, I don't know; I just know that we do, successfully.

> If somebody wants to follow up, Olivier Galibert knows how.

As far as I know, Mr. Nagano wrote the code of portable
dumper for Emacs, and has already sent FSF the signed
ASSIGNMENT paper a few years ago.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: CANNOT_DUMP support

[Richard Stallman]

    As far as I know, Mr. Nagano wrote the code of portable
    dumper for Emacs, and has already sent FSF the signed
    ASSIGNMENT paper a few years ago.

By all means, let's install it and try it out.

Re: CANNOT_DUMP support

[Stefan Monnier]

>> Lisp objects are just part of what unexec dumps.  There is other
>> malloc data, and lots of global variables that don't point to Lisp
>> objects.  How do you handle them?

>> True globals live in the executable itself ini the usual way, as far
>> as I know.  Other malloc data needs to be characterized to the dumper
>> in a way similar to the way that Lisp data is characterized to the GC.
>> Beyond that, I don't know; I just know that we do, successfully.

>> If somebody wants to follow up, Olivier Galibert knows how.

> As far as I know, Mr. Nagano wrote the code of portable
> dumper for Emacs, and has already sent FSF the signed
> ASSIGNMENT paper a few years ago.

Is the code available somewhere?


        Stefan

Re: CANNOT_DUMP support

[Kenichi Handa]

In article <jwvfxvkl0ps.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > As far as I know, Mr. Nagano wrote the code of portable
> > dumper for Emacs, and has already sent FSF the signed
> > ASSIGNMENT paper a few years ago.

> Is the code available somewhere?

I dn't know.  According to this page:

  http://www.sodan.org/~knagano/emacs/pdump/

I sent a mail to him <knagano@sodan.org>, but I have not yet
got any reply.

Re: CANNOT_DUMP support

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@ics.uci.edu>
> Date: Sun, 10 Feb 2008 17:48:25 -0800
> Cc: Emacs devel <emacs-devel@gnu.org>
> 
>   > Hmm.  System/390, amdx86-64, ia-64, m68k.  
> 
> Nope, look again, all those are either commented out or inside #if 0.
> 
> The only remaining one is in ibmrs6000.h when using sysV. I don't know
> of any such system in current use ...

CANNOT_DUMP needs to stay, even if no platforms use it, because it is
important for new targets that are added to Emacs, before their
support for dumping is developed.

Re: CANNOT_DUMP support

[Richard Stallman]

    Also, what are the plans with regard to CANNOT_DUMP, if any, e.g. 
    "continue to support indefinitely"?

Yes.

But it would be nice if we could arrange for dumping with Gnustep.