Pure space

Pure space

[Pip Cet]

Hello everyone,

can we revisit the question of whether we still want pure space? I just
tracked down another pure space related bug, and I believe there's ample
evidence now that it's slowing down not just me but other developers:

* Gerd no longer uses purespace. He's added support for purespace to the
  MPS branch, but I imagine this took a while.
* Danny's recent regexp patch included some code for pure space, which I
  believe was unnecessary effort
* Adding headers to pure space objects in the MPS branch required some
  effort

I believe everyone who knows the source code can agree that pure space
adds significant complexity to the code, and I'm afraid that fixing the
remaining bugs (such as 'valid_lisp_object_p' returning 1 for all
objects that happen to point anywhere into pure space) would make it
even more complex.

I think it's particularly problematic to attempt to survive pure space
overflow: that means there are now "pure" objects for which PURE_P
returns false, and that's what caused this latest bug.

I keep running into pure space overflows that I think are caused by
going back and forth between configurations with and without native
compilation (without doing a full make bootstrap, which fixes
things). I'm willing to accept that's my fault, but I shouldn't have to
track down segfaults caused by that.

So, even if the decision is to keep pure space, can we at least make
pure space overflow a fatal condition?

I'd like some indication of how people see this before reopening
scratch/no-purespace...

Thanks
Pip

Re: Pure space

[Eli Zaretskii]

> Date: Fri, 16 Aug 2024 19:07:42 +0000
> From: Pip Cet <pipcet@protonmail.com>
> 
> can we revisit the question of whether we still want pure space?

We already decided in the past to remove the pure space only when we
remove the support for unexec builds.  So if we no longer want unexec
in Emacs 31 and beyond, we can remove pure space (and the unexec build
as well) on master.

Re: Pure space

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 16 Aug 2024 19:07:42 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>>
>> can we revisit the question of whether we still want pure space?
>
> We already decided in the past to remove the pure space only when we
> remove the support for unexec builds.  So if we no longer want unexec
> in Emacs 31 and beyond, we can remove pure space (and the unexec build
> as well) on master.

I think we can remove the unexec build in Emacs 31, and consequently
also purespace.  I was planning to propose doing that, but I'm happy
that Pip beat me to it.

Re: Pure space

[Po Lu]

Stefan Kangas <stefankangas@gmail.com> writes:

> I think we can remove the unexec build in Emacs 31, and consequently
> also purespace.  I was planning to propose doing that, but I'm happy
> that Pip beat me to it.

Portable dumping doesn't function on Windows 98 or DOS.  Please fix or
implement pdumper on these two platforms first, or hack their unexec
builds not to require pure space, which should not be terribly
difficult.  Moreover, whatever becomes of the portable ELF unexec should
not affect the Solaris unexec, which is provided by the operating system
and should function without the likes of gmalloc.

Otherwise this proposal won't fly.

Re: Pure space

[Stefan Kangas]

Po Lu <luangruo@yahoo.com> writes:

> Portable dumping doesn't function on Windows 98 or DOS.  Please fix or
> implement pdumper on these two platforms first, or hack their unexec
> builds not to require pure space, which should not be terribly
> difficult.

In my view, that's not a blocker for removing the unexec build.

We would take patches to fix MS-DOS and/or Windows 98, I think.

> Moreover, whatever becomes of the portable ELF unexec should
> not affect the Solaris unexec, which is provided by the operating system
> and should function without the likes of gmalloc.

AFAIK, the portable dumper is the default also on Solaris, so there is
no need to keep the unexec build around just for that platform.

Re: Pure space

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sat, 17 Aug 2024 05:10:29 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>, Pip Cet <pipcet@protonmail.com>, emacs-devel@gnu.org
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Portable dumping doesn't function on Windows 98 or DOS.  Please fix or
> > implement pdumper on these two platforms first, or hack their unexec
> > builds not to require pure space, which should not be terribly
> > difficult.
> 
> In my view, that's not a blocker for removing the unexec build.

Agreed.

> We would take patches to fix MS-DOS and/or Windows 98, I think.

On Windows 9X the pdumper build does work, it just crashes too much.
So it sounds like some problem that could be fixed.

If we want to keep the MSDOS port, the way forward is to switch it to
pdumper.  There's already an implementation in pdumper.c that uses
malloc and write instead of the missing mmap, it just was never tried
with the MSDOS port.  I don't see any reasons why it couldn't work.

Re: Pure space

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Kangas <stefankangas@gmail.com>
>> Date: Sat, 17 Aug 2024 05:10:29 -0700
>> Cc: Eli Zaretskii <eliz@gnu.org>, Pip Cet <pipcet@protonmail.com>, emacs-devel@gnu.org
>> 
>> Po Lu <luangruo@yahoo.com> writes:
>> 
>> > Portable dumping doesn't function on Windows 98 or DOS.  Please fix or
>> > implement pdumper on these two platforms first, or hack their unexec
>> > builds not to require pure space, which should not be terribly
>> > difficult.
>> 
>> In my view, that's not a blocker for removing the unexec build.
>
> Agreed.

The Windows 98 build is: it has many users and it is in a satisfactory
condition at present, so it would be a terrible regression.

>> We would take patches to fix MS-DOS and/or Windows 98, I think.
>
> On Windows 9X the pdumper build does work, it just crashes too much.
> So it sounds like some problem that could be fixed.

It crashes 100% of the time, actually, not merely "too much", which I
can't investigate without a functioning gdb, and which is very much
against the spirit of the word "portable".  Unless one of you are
willing to show me a GDB that is newer than Code::Blocks provides and
which groks recent DWARF debug info, I don't forsee any solution.

> If we want to keep the MSDOS port, the way forward is to switch it to
> pdumper.  There's already an implementation in pdumper.c that uses
> malloc and write instead of the missing mmap, it just was never tried
> with the MSDOS port.  I don't see any reasons why it couldn't work.

Why can't unexecoff be retained without pure space?  There's also no
reason against this enormously simpler solution, and the same is true of
the Solaris unexec, which is no more than 13 lines of trivial code.  You
need not concern yourself with them at all, while MS-DOS users receive a
new Emacs, and Solaris users, a binary that starts 110 ms faster, and
doesn't require juggling with dumper files whenever an installation or
its binaries are created or moved.

Re: Pure space

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Stefan Kangas <stefankangas@gmail.com>,  pipcet@protonmail.com,
>   emacs-devel@gnu.org
> Date: Sat, 17 Aug 2024 21:36:38 +0800
> 
> >> > Portable dumping doesn't function on Windows 98 or DOS.  Please fix or
> >> > implement pdumper on these two platforms first, or hack their unexec
> >> > builds not to require pure space, which should not be terribly
> >> > difficult.
> >> 
> >> In my view, that's not a blocker for removing the unexec build.
> >
> > Agreed.
> 
> The Windows 98 build is: it has many users and it is in a satisfactory
> condition at present, so it would be a terrible regression.

Then the interested users and developers should come forward and help
us find and fix whatever causes the crashes.  I have not seen any such
users, with you being the single exception, for a long time.  If you
can get someone motivated to work with us on fixing pdumper.c on
Windows 9X, please do, and let's take it from there.  Such an effort
will at least be an investment in the future.

> It crashes 100% of the time, actually, not merely "too much", which I
> can't investigate without a functioning gdb, and which is very much
> against the spirit of the word "portable".  Unless one of you are
> willing to show me a GDB that is newer than Code::Blocks provides and
> which groks recent DWARF debug info, I don't forsee any solution.

There's such a thing as printf debugging.  It is far from ideal, but
it's doable.

> > If we want to keep the MSDOS port, the way forward is to switch it to
> > pdumper.  There's already an implementation in pdumper.c that uses
> > malloc and write instead of the missing mmap, it just was never tried
> > with the MSDOS port.  I don't see any reasons why it couldn't work.
> 
> Why can't unexecoff be retained without pure space?

Because Emacs never used this combination, and because no one has time
or knowhow to debug the MSDOS port anymore (which still uses COFF
debug info, which makes the job of finding a proper GDB even harder).
I might be the last person who can do that, and I don't have time to
do that seriously, when significant problems pop up.

> There's also no reason against this enormously simpler solution

It is simple source code wise, but it is not a solution, it is a
problem.  We never used Emacs that way, and I'm quite sure this will
trigger strange bugs that I don't want us to need to debug.

Keeping the MSDOS build alive means keeping it in good health.  While
the current code base has some significant number of years under its
belt (although that, too, becomes smaller and smaller as years go by
and more significant code changes are installed) the version without
pure space was never used before.

Re: Pure space

[Pip Cet]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Date: Fri, 16 Aug 2024 19:07:42 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>>
>> can we revisit the question of whether we still want pure space?
>
> We already decided in the past to remove the pure space only when we
> remove the support for unexec builds.

IIRC, the situation was that unexec builds were broken on master at the
point I proposed the pure space removal.  I filed a bug about unexec and
things were fixed some time later.

Pure space and unexec aren't closely related at all, and the rebased
scratch/no-purespace branch, after some tedious fixes, starts fine when
configured with --with-dumping=unexec.

> So if we no longer want unexec
> in Emacs 31 and beyond, we can remove pure space (and the unexec build
> as well) on master.

What do you think about making pure space overflow fatal as a first
step?

Pip

Re: Pure space

[Eli Zaretskii]

> Date: Sat, 17 Aug 2024 08:45:16 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> 
> >> Date: Fri, 16 Aug 2024 19:07:42 +0000
> >> From: Pip Cet <pipcet@protonmail.com>
> >>
> >> can we revisit the question of whether we still want pure space?
> >
> > We already decided in the past to remove the pure space only when we
> > remove the support for unexec builds.
> 
> IIRC, the situation was that unexec builds were broken on master at the
> point I proposed the pure space removal.  I filed a bug about unexec and
> things were fixed some time later.
> 
> Pure space and unexec aren't closely related at all, and the rebased
> scratch/no-purespace branch, after some tedious fixes, starts fine when
> configured with --with-dumping=unexec.

See my other messages in this thread.

> > So if we no longer want unexec
> > in Emacs 31 and beyond, we can remove pure space (and the unexec build
> > as well) on master.
> 
> What do you think about making pure space overflow fatal as a first
> step?

How would that help?

Re: Pure space

[Pip Cet]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Date: Sat, 17 Aug 2024 08:45:16 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>> Cc: emacs-devel@gnu.org
>>
>> "Eli Zaretskii" <eliz@gnu.org> writes:
>>
>> >> Date: Fri, 16 Aug 2024 19:07:42 +0000
>> >> From: Pip Cet <pipcet@protonmail.com>
>> >>
>> >> can we revisit the question of whether we still want pure space?
>> >
>> > We already decided in the past to remove the pure space only when we
>> > remove the support for unexec builds.
>>
>> IIRC, the situation was that unexec builds were broken on master at the
>> point I proposed the pure space removal.  I filed a bug about unexec and
>> things were fixed some time later.
>>
>> Pure space and unexec aren't closely related at all, and the rebased
>> scratch/no-purespace branch, after some tedious fixes, starts fine when
>> configured with --with-dumping=unexec.

(Of course, those tedious fixes were completely unrelated to the dumping
method used).

> See my other messages in this thread.

I asked whether the decision to keep purespace (by artificially linking
it to another feature which we want to keep) could be revisited.  I take
it your answer is "no".  IMO, circumstances have changed considerably.
My guess is you disagree.

If the two issues are to be considered inextricably linked (which,
again, they're not; removing pure space does not require any changes to
unexec dumping), then any effort to maintain the purespace code must
count as "investing any development efforts in the unexec builds",
because that's the only reason maintaining purespace is still necessary.

IOW, if we count all the complexity and maintenance work that purespace
requires as costs of keeping the unexec builds, then we should drop
unexec ASAP.

>> > So if we no longer want unexec
>> > in Emacs 31 and beyond, we can remove pure space (and the unexec build
>> > as well) on master.
>>
>> What do you think about making pure space overflow fatal as a first
>> step?
>
> How would that help?

It would mean we could do without changes such as those in the patch
attached to the original message.

Pip

Re: Pure space

[Eli Zaretskii]

> Date: Sat, 17 Aug 2024 11:38:36 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> 
> > See my other messages in this thread.
> 
> I asked whether the decision to keep purespace (by artificially linking
> it to another feature which we want to keep) could be revisited.

The linking is not "artificial", and I explained why.  It is very much
real, at least for me.  If that is still not clear, let me be
completely clear: I'm opposed to any suggestion that will force us to
add significant maintenance burden for supporting obscure and
rarely-used configurations that almost no one builds and tests.  Maybe
you don't know, but I'm forced to build no less than 6 different
configurations every week, just to make sure they don't get broken.
For that reason, I WANT THE UNEXEC BUILD REMOVED!!  If what it takes
is to staunchly object removing the pure space unless unexec goes with
it, so be it.

> IMO, circumstances have changed considerably.  My guess is you
> disagree.

Your opinion is noted, but it has nothing to do with reality: I'm
required to build and sometimes fix all those configurations today the
same as I needed to do it several years ago, when we had the previous
"remove pure space" discussion.  I can sum all the hours I needed to
spend on this as result, if it will change your mind and perhaps cause
you to value my time and efforts more than you seem to do now.

> If the two issues are to be considered inextricably linked (which,
> again, they're not; removing pure space does not require any changes to
> unexec dumping), then any effort to maintain the purespace code must
> count as "investing any development efforts in the unexec builds",
> because that's the only reason maintaining purespace is still necessary.

From where I stand, it is clear that if we keep the unexec build, we
should keep it in good shape.  When the issue is looked at from this
perspective, rather than from the nonchalant one you propose (which
doesn't really care if unexec breaks or becomes less useful), then the
fallacy of your arguments should be crystal clear.

And I don't think I care for your hints about my alleged lies.  The
fact that we disagree doesn't mean that I intentionally tell lies, it
just means we have different perspectives, different experiences, and
different responsibilities here.

> IOW, if we count all the complexity and maintenance work that purespace
> requires as costs of keeping the unexec builds, then we should drop
> unexec ASAP.

I agree, and urge you to try to convince the single person who so far
objects to removing unexec, instead of wasting your time on trying to
change my mind on that.

> >> What do you think about making pure space overflow fatal as a first
> >> step?
> >
> > How would that help?
> 
> It would mean we could do without changes such as those in the patch
> attached to the original message.

I saw no patch attached to the original message (nor any other message
in this thread).  Am I lying again?

Re: Pure space

[Pip Cet]

"Eli Zaretskii" <eliz@gnu.org> writes:
> And I don't think I care for your hints about my alleged lies.  The
> fact that we disagree doesn't mean that I intentionally tell lies, it
> just means we have different perspectives, different experiences, and
> different responsibilities here.

I think that warrants a separate response: I'm sorry I caused offense
there. I think you're often inconsistent, but I don't believe that means
you're ever lying.

>> IOW, if we count all the complexity and maintenance work that purespace
>> requires as costs of keeping the unexec builds, then we should drop
>> unexec ASAP.
>
> I agree, and urge you to try to convince the single person who so far
> objects to removing unexec, instead of wasting your time on trying to
> change my mind on that.

Indeed, that seems the best way forward.

> I saw no patch attached to the original message (nor any other message
> in this thread).  Am I lying again?

No, this time it's me being inconsistent.  Patch attached (this time,
hopefully).


Pip

Re: Pure space

[Eli Zaretskii]

> Date: Sat, 17 Aug 2024 13:26:08 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> > And I don't think I care for your hints about my alleged lies.  The
> > fact that we disagree doesn't mean that I intentionally tell lies, it
> > just means we have different perspectives, different experiences, and
> > different responsibilities here.
> 
> I think that warrants a separate response: I'm sorry I caused offense
> there. I think you're often inconsistent, but I don't believe that means
> you're ever lying.

I'm not "often inconsistent".  You often disagree with my reasoning
and the relative weights I assign to some considerations, but that's
not "inconsistency".

> > I saw no patch attached to the original message (nor any other message
> > in this thread).  Am I lying again?
> 
> No, this time it's me being inconsistent.  Patch attached (this time,
> hopefully).

Ahm... ENPATCH

Re: Pure space

[Pip Cet]

[-- Attachment #1: Type: text/plain, Size: 563 bytes --]

On Saturday, August 17th, 2024 at 14:29, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Sat, 17 Aug 2024 13:26:08 +0000
> 
> > From: Pip Cet pipcet@protonmail.com
> > Cc: emacs-devel@gnu.org
> > 
> > "Eli Zaretskii" eliz@gnu.org writes:

> > > I saw no patch attached to the original message (nor any other message
> > > in this thread). Am I lying again?
> > 
> > No, this time it's me being inconsistent. Patch attached (this time,
> > hopefully).
> 
> Ahm... ENPATCH

Ah, it is my email client, then. Let me try again with the web client.

Pip

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Avoid-crashes-when-pure-space-overflows.patch --]
[-- Type: text/x-patch; name=0001-Avoid-crashes-when-pure-space-overflows.patch, Size: 1700 bytes --]

From 29b919dcce693135cc7f5d2952c551d2ac446602 Mon Sep 17 00:00:00 2001
From: Pip Cet <pipcet@protonmail.com>
Date: Fri, 16 Aug 2024 19:00:38 +0000
Subject: [PATCH] Avoid crashes when pure space overflows

When pure space has overflowed, all objects are potentially pure.  This
means we cannot use 'SDATA_OF_STRING' on them, or assume they're invalid
when printing them.

* src/alloc.c (string_bytes):
(pin_string): Do nothing after pure space overflow.
(valid_lisp_object_p): Never return 0 after pure space overflow.
---
 src/alloc.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index 06fe12cff3d..616ec3083b9 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -1797,7 +1797,7 @@ string_bytes (struct Lisp_String *s)
     (s->u.s.size_byte < 0 ? s->u.s.size & ~ARRAY_MARK_FLAG : s->u.s.size_byte);
 
   if (!PURE_P (s) && !pdumper_object_p (s) && s->u.s.data
-      && nbytes != SDATA_NBYTES (SDATA_OF_STRING (s)))
+      && !pure_bytes_used_before_overflow && nbytes != SDATA_NBYTES (SDATA_OF_STRING (s)))
     emacs_abort ();
   return nbytes;
 }
@@ -2644,7 +2644,7 @@ pin_string (Lisp_Object string)
 
   if (!(size > LARGE_STRING_BYTES
 	|| PURE_P (data) || pdumper_object_p (data)
-	|| s->u.s.size_byte == -3))
+	|| s->u.s.size_byte == -3 || pure_bytes_used_before_overflow))
     {
       eassert (s->u.s.size_byte == -1);
       sdata *old_sdata = SDATA_OF_STRING (s);
@@ -5606,6 +5606,9 @@ valid_lisp_object_p (Lisp_Object obj)
   if (pdumper_object_p (p))
     return pdumper_object_p_precise (p) ? 1 : 0;
 
+  if (pure_bytes_used_before_overflow)
+    return -1;
+
   struct mem_node *m = mem_find (p);
 
   if (m == MEM_NIL)
-- 
2.45.2

Re: Pure space

[Stefan Kangas]

Pip Cet <pipcet@protonmail.com> writes:

> Pure space and unexec aren't closely related at all, and the rebased
> scratch/no-purespace branch, after some tedious fixes, starts fine when
> configured with --with-dumping=unexec.

Could you please push the rebased branch to Savannah?

Re: Pure space

[Pip Cet]

"Stefan Kangas" <stefankangas@gmail.com> writes:

> Pip Cet <pipcet@protonmail.com> writes:
>
>> Pure space and unexec aren't closely related at all, and the rebased
>> scratch/no-purespace branch, after some tedious fixes, starts fine when
>> configured with --with-dumping=unexec.
>
> Could you please push the rebased branch to Savannah?

Sure, if someone could point me to the right way to do it.  "git push
--force-with-lease" refuses to update scratch/no-purespace, so should I
delete it and recreate it?

Thanks
Pip

Re: Pure space

[Andrea Corallo]

Pip Cet <pipcet@protonmail.com> writes:

> "Stefan Kangas" <stefankangas@gmail.com> writes:
>
>> Pip Cet <pipcet@protonmail.com> writes:
>>
>>> Pure space and unexec aren't closely related at all, and the rebased
>>> scratch/no-purespace branch, after some tedious fixes, starts fine when
>>> configured with --with-dumping=unexec.
>>
>> Could you please push the rebased branch to Savannah?
>
> Sure, if someone could point me to the right way to do it.  "git push
> --force-with-lease" refuses to update scratch/no-purespace, so should I
> delete it and recreate it?

Yes, you can delete scratch/no-purespace and push it again.

Re: Pure space

[Pip Cet]

"Andrea Corallo" <acorallo@gnu.org> writes:

> Pip Cet <pipcet@protonmail.com> writes:
>
>> "Stefan Kangas" <stefankangas@gmail.com> writes:
>>
>>> Pip Cet <pipcet@protonmail.com> writes:
>>>
>>>> Pure space and unexec aren't closely related at all, and the rebased
>>>> scratch/no-purespace branch, after some tedious fixes, starts fine when
>>>> configured with --with-dumping=unexec.
>>>
>>> Could you please push the rebased branch to Savannah?
>>
>> Sure, if someone could point me to the right way to do it.  "git push
>> --force-with-lease" refuses to update scratch/no-purespace, so should I
>> delete it and recreate it?
>
> Yes, you can delete scratch/no-purespace and push it again.

Done, thanks! I'll clean up the commit messages next, assuming we don't
reach a consensus to keep unexec and purespace for now.

Pip

Re: Pure space

[Po Lu]

Pip Cet <pipcet@protonmail.com> writes:

> So, even if the decision is to keep pure space, can we at least make
> pure space overflow a fatal condition?

Can't pure space be eliminated without affecting unexec?  It's not as
though every Lisp object in a dumped Emacs must be pure, and eliminating
read only objects would be a welcome change whether by itself or for
unexec builds.

Re: Pure space

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Pip Cet <pipcet@protonmail.com> writes:
>
>> So, even if the decision is to keep pure space, can we at least make
>> pure space overflow a fatal condition?
>
> Can't pure space be eliminated without affecting unexec?  It's not as
> though every Lisp object in a dumped Emacs must be pure, and eliminating
> read only objects would be a welcome change whether by itself or for
> unexec builds.

Btw, I get the impression that help facilities generate plenty of large
strings that are ultimately dumped, so it would be useful to measure the
size of pure space with and without the string deduplication
optimization enabled, and decide whether it would be worth preserving in
some form.

Re: Pure space

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Pip Cet <pipcet@protonmail.com> writes:
>>
>>> So, even if the decision is to keep pure space, can we at least make
>>> pure space overflow a fatal condition?
>>
>> Can't pure space be eliminated without affecting unexec?  It's not as
>> though every Lisp object in a dumped Emacs must be pure, and eliminating
>> read only objects would be a welcome change whether by itself or for
>> unexec builds.
>
> Btw, I get the impression that help facilities generate plenty of large
> strings that are ultimately dumped, so it would be useful to measure
> the

"large repetitive strings"

Re: Pure space

[Pip Cet]

"Po Lu" <luangruo@yahoo.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Pip Cet <pipcet@protonmail.com> writes:
>>
>>> So, even if the decision is to keep pure space, can we at least make
>>> pure space overflow a fatal condition?
>>
>> Can't pure space be eliminated without affecting unexec?  It's not as
>> though every Lisp object in a dumped Emacs must be pure, and eliminating
>> read only objects would be a welcome change whether by itself or for
>> unexec builds.
>
> Btw, I get the impression that help facilities generate plenty of large
> strings that are ultimately dumped, so it would be useful to measure the
> size of pure space with and without the string deduplication
> optimization enabled, and decide whether it would be worth preserving in
> some form.

It's already disabled in scratch/igc (when HAVE_MPS is set), and IIRC I
didn't have to increase the pure space size...

Keep in mind that the current implementation only works one way: if we
dump "c", then "abc", both strings will be copied to pure space
completely. If we dump "abc", then "c", the second string will be
deduplicated to point at "abc" + 2.  I suspect this is the opposite of
what we need.

So it might be fairer to fix that first, and compare then. However,
once we start implementing rudimentary data compression for dumps, where
do we stop?

Pip

Re: Pure space

[Eli Zaretskii]

> Date: Sat, 17 Aug 2024 08:57:46 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> > Btw, I get the impression that help facilities generate plenty of large
> > strings that are ultimately dumped, so it would be useful to measure the
> > size of pure space with and without the string deduplication
> > optimization enabled, and decide whether it would be worth preserving in
> > some form.
> 
> It's already disabled in scratch/igc (when HAVE_MPS is set), and IIRC I
> didn't have to increase the pure space size...

That's okay, because it isn't "removing pure space", and it is on a
feature branch which was not yet landed.  So we have enough time to
consider what to do with it until it becomes actually relevant.

> Keep in mind that the current implementation only works one way: if we
> dump "c", then "abc", both strings will be copied to pure space
> completely. If we dump "abc", then "c", the second string will be
> deduplicated to point at "abc" + 2.  I suspect this is the opposite of
> what we need.
> 
> So it might be fairer to fix that first, and compare then. However,
> once we start implementing rudimentary data compression for dumps, where
> do we stop?

Once again, the only way towards removing pure space is to decide that
we drop support for platforms where Emacs can only be built with
unexec.  If there's consensus about dropping the unexec-only
platforms, we can move towards removing pure; otherwise we cannot.

Re: Pure space

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 17 Aug 2024 16:16:49 +0800
> 
> Pip Cet <pipcet@protonmail.com> writes:
> 
> > So, even if the decision is to keep pure space, can we at least make
> > pure space overflow a fatal condition?
> 
> Can't pure space be eliminated without affecting unexec?

Maybe it could, but I'm not interested in investing any development
efforts in the unexec builds.  So, as already discussed and decided
long ago, if we want to remove the pure space, unexec will need to go
with it.

Re: Pure space

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe it could, but I'm not interested in investing any development
> efforts in the unexec builds.  So, as already discussed and decided
> long ago, if we want to remove the pure space, unexec will need to go
> with it.

What difficulties attend removing pure space without also removing
unexec?

Re: Pure space

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: pipcet@protonmail.com,  emacs-devel@gnu.org
> Date: Sat, 17 Aug 2024 19:46:54 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Maybe it could, but I'm not interested in investing any development
> > efforts in the unexec builds.  So, as already discussed and decided
> > long ago, if we want to remove the pure space, unexec will need to go
> > with it.
> 
> What difficulties attend removing pure space without also removing
> unexec?

They might make the unexec build less performant.  But my main worry
is that doing so will get us to where we never were before -- an
unexec build without pure space, and I'm unwilling to invest any
effort in that configuration.  It is enough that I need to routinely
build it on my machine to make sure it doesn't break; I don't want any
more time wasted on that.

Personally, I fail to understand why you want to keep unexec any
longer than we did.

Re: Pure space

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> They might make the unexec build less performant.  But my main worry
> is that doing so will get us to where we never were before -- an
> unexec build without pure space, and I'm unwilling to invest any
> effort in that configuration.  It is enough that I need to routinely
> build it on my machine to make sure it doesn't break; I don't want any
> more time wasted on that.
>
> Personally, I fail to understand why you want to keep unexec any
> longer than we did.

Chiefly because its removal is being considered for systems where unexec
never posed a problem: Solaris (where unexec is an OS feature and
virtually the recommended solution for such requirements as saving the
state of a program to disk), DJGPP, where you implemented specific
support for unexec in the C library, and suchlike.  I'm not so
implacably opposed to removing it for Windows, provided that its
successor is made to function on Windows 98, or the machines where it is
wanted disappear.

Re: Pure space

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: pipcet@protonmail.com,  emacs-devel@gnu.org
> Date: Sat, 17 Aug 2024 21:44:05 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > They might make the unexec build less performant.  But my main worry
> > is that doing so will get us to where we never were before -- an
> > unexec build without pure space, and I'm unwilling to invest any
> > effort in that configuration.  It is enough that I need to routinely
> > build it on my machine to make sure it doesn't break; I don't want any
> > more time wasted on that.
> >
> > Personally, I fail to understand why you want to keep unexec any
> > longer than we did.
> 
> Chiefly because its removal is being considered for systems where unexec
> never posed a problem: Solaris (where unexec is an OS feature and
> virtually the recommended solution for such requirements as saving the
> state of a program to disk), DJGPP, where you implemented specific
> support for unexec in the C library, and suchlike.  I'm not so
> implacably opposed to removing it for Windows, provided that its
> successor is made to function on Windows 98, or the machines where it is
> wanted disappear.

It should be quite clear that nowadays leaving unexec supported
becomes a burden for Emacs development, so it makes less and less
sense to insist on supporting it.