MPS experiment successful

MPS experiment successful

[Gerd Möllmann]

If someone doesn't remember what MPS is about: concurrent, incremental,
generational, mostly-copying, thread-safe GC.

I've now reached the point that I wanted to reach to consider my
experiment of using MPS in Emacs successful.

- Emacs builds from scratch
- Tests succeed to a sufficient degree
- Emacs starts in GUI mode

That's all. There is nothing more. And I'm currently undecided how to
proceed with this.

To make this credible :-), you can find the sources at

  https://github.com/gerd-moellmann/emacs-with-cl-packages/tree/igc

Please note that this is based on my local Emacs (see Github page). It
probably doesn't build/work on anything but macOS, using different Elisp
packages than I do, or not using the ones I do :-), and so on. I also
regularly force-push, and do other mean things.

Re: MPS experiment successful

[Björn Bidar]

I tried to built with branch.
Any reason that you used sys/_types/_ssize_t.h?

Re: MPS experiment successful

[Björn Bidar]

After including sys/types.h instead everything builds fine.
What' the benefit of using mps?

Re: MPS experiment successful

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Wed, 17 Apr 2024 12:46:25 +0200
> 
> If someone doesn't remember what MPS is about: concurrent, incremental,
> generational, mostly-copying, thread-safe GC.
> 
> I've now reached the point that I wanted to reach to consider my
> experiment of using MPS in Emacs successful.
> 
> - Emacs builds from scratch
> - Tests succeed to a sufficient degree
> - Emacs starts in GUI mode
> 
> That's all. There is nothing more. And I'm currently undecided how to
> proceed with this.

The next step would be to push a feature branch into the Emacs Git
repository and let people try the branch on the other supported
platforms.  If the code is not yet mature enough for that, please
suggest how to get from here to there.

Would you like to describe in a few words what would be the advantages
and disadvantages of this GC for Emacs?

Thanks.

Re: MPS experiment successful

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 2428 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> That's all. There is nothing more. And I'm currently undecided how to
>> proceed with this.
>
> The next step would be to push a feature branch into the Emacs Git
> repository and let people try the branch on the other supported
> platforms.  If the code is not yet mature enough for that, please
> suggest how to get from here to there.

I think technically it could be done, but there are some things that
would need to be done. From the top of my head:

- CL packages vs. obarrays (I do have packages but no obarrays)
- Handling of pure space
- Handling the new JSON stuff, which I don't have in my branch, I think.
- ...

And then of course the general work of getting a branch from a what is
basically a fork back to the forked repo, which I'm not so sure how to
go about that.

But my undecidedness comes more from what could come after that. I'm not
sure how willing I am to invest what can be years to get things
finished, esp. if no one else starts working on it. Had that one time
too often in the 90s, some might remember ;-).

> Would you like to describe in a few words what would be the advantages
> and disadvantages of this GC for Emacs?

Advantages for users:

For the user it means the final end to GC pauses. MPS runs in a
different thread wich can be in a different core.

What MPS means for the overall speed of Emacs I find impossible to say.
MPS doesn't seem to be slow at all though, although some slowdown can be
expected in the client because of thread-safe allocations.

Example for a confusing fact that I noticed today:

Full build without MPS, -O0, checking

real	26:20.01
user	1:32:15.54
sys	3:20.74

Same build with debug MPS (-lmps-debug)

real	14:07.90
user	44:17.94
sys	3:15.99

That's on an 2,3 GHz Quad-Core Intel Core i7, 16G RAM, SSD. 🤷

Possible advantages for developers of Emacs:

- MPS is thread-safe, i.e. it's a tiny step in that direction.
- I think igc.c is easier to understand that alloc.c.

Disadvantages:

As long as there are 2 GCs, there are 2 instead of 1 GC :-).

Will increase memory usage in its current state, part of which could be
optimized.

Some configurations/platform might not be possible to support. Hard to
say.

...

Let me also attach a badly written and curated Org file that I have in
that branch. Maybe that has some additional answers


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: igc.org --]
[-- Type: text/x-org, Size: 6450 bytes --]

#+title: MPS garbage collection for Emacs

* What is MPS?

The MPS (Memory Pool System) is a GC library developed by Ravenbrook
Ltd. MPS is available from [[https://github.com/Ravenbrook/mps?tab=readme-ov-file][Github]] under a BSD license. See the
[[https://memory-pool-system.readthedocs.io/en/latest/][documentation]].

In short, MPS implements incremental, generational, concurrent, copying,
thread-safe garbage collection on a large variety of platforms. It has
been around for a long time, is stable, and well documented.

* What is this branch?

This [[https://github.com/gerd-moellmann/emacs-with-cl-packages/tree/igc][branch]] is an experiment if Emacs can be made to use a GC based on
MPS. I'm doing this for my own entertainment, it's not in any form
"official".

* Caveats

This is my local Emacs, which is different from mainstream Emacs: It
uses CL packages, doesn't have obarrays, doesn't support pure space,
does not support shorthands and probably some other stuff.

In addition, I'm exclusively using macOS, so it's unlikely to compile or
run on other systems OOTB. It should not be too hard to port, though.

* Current state

Build succeeds up to and including =compile-first=, i.e. Emacs pdumps, and
compiles some =.elc= files.

* Things worth mentioning

** Configuration

There is a now configure switch =--with-mps= with values =no, yes, debug=.
If =debug= is given, Emacs links with the debug version of the MPS
library.

** Building MPS

I built MPS from its Git repo. I had to make two trivial fixes for macOS
for which I submitted issues upstream.

** Every object has a 1 word header

At the moment, every object has a one-word header, which is not visible
to the rest of Emacs. See ~struct igc_header~.

This means in particular that conses are 50% larger than they would
normally be. I did this because it is less work:

- All objects can be handled by one set of MPS callback functions.

- It simplifies the implementation of eq hash tables considerably by
  storing an address-independent hash in the header.

The header can be removed from conses (and other objects, if that's
worth it) by writing additional code using additional MPS pools with
their own object formats.

Note that doing this also means that one has to use MPS's location
dependency feature for implementing eq hash tables.

Also be aware that two calls to ~sxhash-eq~ can then return different
hashes when a concurrent GC happens between calls, unless something is
done to ensure that the hashed objects aren't moved by the GC for long
enough.

** MPS In-band headers

I have tried to use MPS in-band headers at first, but couldn't get it to
work. I don't claim they don't work, though. After all I was and still
am learning MPS.

** Weak hash tables

I didn't think that weak hash tables were important enough for my
experiment, so I didn't implement them to save work.

Weak tables can be implemented using the already present in =igc.c= AWL
pool and its allocation points, and then using MPS's dependent objects
in the hash table implementation. There are examples how to do this in
the MPS documentation, and in an example Scheme interpreter.

To prepare for that, keys and values of a hash table are already split
into two vectors. Two vectors are necessary because objects in an AWL
pool must either contain weak references only, or strong references
only. The currently malloc'd vectors would have to be replaced with
special vectors allocated from the AWL pool.

** Handling of a loaded pdump

The hot part of a loaded pdump (ca. 18 MB) is currently used as an
ambiguous root for MPS. A number of things could be investigated

- Use a root with barrier (~MPS_RM_PROT~)

- Copy objects from the dump to an MPS pool that uses ~MPS_KEY_GEN~ to
  allocate objects in an old generation.

  It is unclear to me from the docs if the AMC pool supports that, but
  one could use an AMS pool.

  After loading a dump we would copy the whole object graph to MPS,
  starting from static roots.  After that, the dump itself would no
  longer be used.

  Costs some load time, though.

There is also a slight problem currently that's a consequence of Emacs
mixing GC'd objects and malloc'd ones. The loaded dump is scanned
conservativly, but if such objects contain malloc'd data structures
holding references, these are invisble to MPS, so one has to jump
through hoops.

Examples:

- Hash tables hold keys and values in malloc'd vectors. If the hash
  table is in the dump, and the vectors are on the heap, keys and values
  won't be seen be MPS.

- Symbols in the dump may have a Lisp_Buffer_Local_Value that is on the
  heap.

- Buffers have a itree_tree that is malloc'd.

** Intervals and ~itree_node~

Problem with these two is that there are pointers from Lisp objects to
malloc'd memory and back. This is easier to handle if allocated
from MPS. Moving these to MPS makes things easier because MPS triggers
the scanning, and, not the least, makes an ambiguous scan of the loaded
dump keep things alive.

** Finalization

Is now implemented.

** Things old GC does except GC

The function ~garbage_collect~ does some things that are not directly
related to GC, simply because it is called every once in a while.

- compact buffers, undo-list.

This is currently not done, but could be done in another way, from a
timer, for instance.

** Not Considered

Some things are not implemented because they were out of scope. For
example,

- ~memory-report~ Could be done with MPS's pool walk functionality.

- profiler (~profiler-memory-start~...) No idea, haven't looked at it.

- Anything I don't currently use either because it doesn't exist on
  macOS (text conversions, for example), or because I didn't think it
  being essiential (xwidgets, for example).

** Knobs not tried

- Number of generations
- Size of generations
- Mortality probabilities
- Allocation policies, like ramp allocation
- ...

** Implementation

I think it's not too terrible, but some things should be improved

- Error handling. It currently aborts in many circumstances, but
  it is also not clear what else to do.

- Idle time use. It does something in this regard, but not much,
  and not always with a time constraint (handling MPS messages).

** Debugger

MPS uses memory barriers. In certain situations it is necessary to
remove these to be able to do certain things.  I've added a command
=xpostmortem= to the LLDB support for that. GDB will need something
similar.

Re: MPS experiment successful

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 17 Apr 2024 16:30:42 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> That's all. There is nothing more. And I'm currently undecided how to
> >> proceed with this.
> >
> > The next step would be to push a feature branch into the Emacs Git
> > repository and let people try the branch on the other supported
> > platforms.  If the code is not yet mature enough for that, please
> > suggest how to get from here to there.
> 
> I think technically it could be done, but there are some things that
> would need to be done. From the top of my head:
> 
> - CL packages vs. obarrays (I do have packages but no obarrays)
> - Handling of pure space
> - Handling the new JSON stuff, which I don't have in my branch, I think.
> - ...

The packages code will need to be moved aside, yes.  But for the rest,
why shouldn't a simple merge do the job?  They seem to be orthogonal,
by and large.

What is the issue with handling pure space?

> Will increase memory usage in its current state, part of which could be
> optimized.

How much larger is the memory footprint?

> Some configurations/platform might not be possible to support.

Which ones?

Re: MPS experiment successful

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 17 Apr 2024 16:30:42 +0200
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> >> That's all. There is nothing more. And I'm currently undecided how to
>> >> proceed with this.
>> >
>> > The next step would be to push a feature branch into the Emacs Git
>> > repository and let people try the branch on the other supported
>> > platforms.  If the code is not yet mature enough for that, please
>> > suggest how to get from here to there.
>>
>> I think technically it could be done, but there are some things that
>> would need to be done. From the top of my head:
>>
>> - CL packages vs. obarrays (I do have packages but no obarrays)
>> - Handling of pure space
>> - Handling the new JSON stuff, which I don't have in my branch, I think.
>> - ...
>
> The packages code will need to be moved aside, yes.  But for the rest,
> why shouldn't a simple merge do the job?  They seem to be orthogonal,
> by and large.

Could be, one has to try. Obarrays could of course be added.

> What is the issue with handling pure space?

I would have to investigate how pure space works, and how to handle it
with MPS. Noting that is impossible in principle, I think.

>> Will increase memory usage in its current state, part of which could be
>> optimized.
>
> How much larger is the memory footprint?

I don't have numbers, but what I can say is that, ATM, every object is 1
word larger, including conses. I did that to siimplify things, and it
could be changed in principle. I think it described in the Org file I
sent.

>> Some configurations/platform might not be possible to support.
>
> Which ones?

MPS must be supported on a platform, of course. MPS' INSTALL file says

  Platforms
  .........

  The MPS uses a six-character platform code to express a combination of
  operating system, CPU architecture, and compiler toolchain.  Each
  six-character code breaks down into three pairs of characters, like
  this::

      OSARCT

  Where ``OS`` denotes the operating system, ``AR`` the CPU
  architecture, and ``CT`` the compiler toolchain.  Here are the
  platforms that we have regular access to and on which the MPS works
  well:

  ==========   =========   =============   ============  =================
  Platform     OS          Architecture    Compiler      Makefile
  ==========   =========   =============   ============  =================
  ``fri3gc``   FreeBSD     IA-32           GCC           ``fri3gc.gmk``
  ``fri3ll``   FreeBSD     IA-32           Clang         ``fri3ll.gmk``
  ``fri6gc``   FreeBSD     x86-64          GCC           ``fri6gc.gmk``
  ``fri6ll``   FreeBSD     x86-64          Clang         ``fri6ll.gmk``
  ``lia6gc``   Linux       ARM64           GCC           ``lia6gc.gmk``
  ``lia6ll``   Linux       ARM64           Clang         ``lia6ll.gmk``
  ``lii3gc``   Linux       IA-32           GCC           ``lii3gc.gmk``
  ``lii6gc``   Linux       x86-64          GCC           ``lii6gc.gmk``
  ``lii6ll``   Linux       x86-64          Clang         ``lii6ll.gmk``
  ``w3i3mv``   Windows     IA-32           Microsoft C   ``w3i3mv.nmk``
  ``w3i6mv``   Windows     x86-64          Microsoft C   ``w3i6mv.nmk``
  ``xca6ll``   macOS       ARM64           Clang         ``mps.xcodeproj``
  ``xci6ll``   macOS       x86-64          Clang         ``mps.xcodeproj``
  ==========   =========   =============   ============  =================

  Historically, the MPS worked on a much wider variety of platforms, and
  still could: IRIX, OSF/1 (Tru64), Solaris, SunOS, Classic Mac OS;
  MIPS, PowerPC, ALPHA, SPARC v8, SPARC v9; Metrowerks Codewarrior,
  SunPro C, Digital C, EGCS, Pelles C. If you are interested in support
  on any of these platforms or any new platforms, please contact
  Ravenbrook at `mps-questions@ravenbrook.com <mailto:mps-questions@ravenbrook.com>`_.

As far as Emacs is is concerned, I don't have a clear picture what all
is currently supported in which combinations. For example, wasn't there
something that made Lisp_Object twice as large as a word? Not sure, but
I think that would make things more difficult.

I'm also currently supporting USE_LSB_TAG only. Could maybe be
supported, but unless someone tries...

And so on.

Re: MPS experiment successful

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 17 Apr 2024 18:29:53 +0200
> 
> As far as Emacs is is concerned, I don't have a clear picture what all
> is currently supported in which combinations. For example, wasn't there
> something that made Lisp_Object twice as large as a word? Not sure, but
> I think that would make things more difficult.

You mean --with-wide-int?

> I'm also currently supporting USE_LSB_TAG only. Could maybe be
> supported, but unless someone tries...

That and the previous item is a huge turn-off for me personally, FWIW,
because I use exactly these two features.

Re: MPS experiment successful

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 17 Apr 2024 18:29:53 +0200
>> 
>> As far as Emacs is is concerned, I don't have a clear picture what all
>> is currently supported in which combinations. For example, wasn't there
>> something that made Lisp_Object twice as large as a word? Not sure, but
>> I think that would make things more difficult.
>
> You mean --with-wide-int?

Yes, I think that's it. With 64 bit words, this makes Lisp_Object 128
bits, right? If that's the case, MPS won't work because if demands that
references are of word/pointer size, AFAIU.

OTOH, hm, pointers will still be of word size, so maybe one could come
up with something clever. Don't know what ATM, though, especially when
you say you are using !USE_LSB_TAG. Maybe one could make use of the GC
header somehow, if all objects have one. Or maybe one doesn't need the
tag bits at all... Except that symbols are a special case again.
Whatever... I think the least one can say is that this will not be easy.

>> I'm also currently supporting USE_LSB_TAG only. Could maybe be
>> supported, but unless someone tries...
>
> That and the previous item is a huge turn-off for me personally, FWIW,
> because I use exactly these two features.

If you want !USE_LSB_TAG, someone (tm) could implement that :-). But
it's not there now, because I didn't want to get lost in the lisp.h
macro jungle while still learning MPS. Keep it simple, and so on...

Re: MPS experiment successful

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 17 Apr 2024 20:34:47 +0200
> 
> > You mean --with-wide-int?
> 
> Yes, I think that's it. With 64 bit words, this makes Lisp_Object 128
> bits, right?

Possibly, except that no one uses that on a 64-bit platform, AFAIK.  I
use it for a 32-bit build, where it makes Lisp objects 64-bit.

> If that's the case, MPS won't work because if demands that
> references are of word/pointer size, AFAIU.

Ouch!  That's a real turn-off!

> OTOH, hm, pointers will still be of word size, so maybe one could come
> up with something clever. Don't know what ATM, though, especially when
> you say you are using !USE_LSB_TAG.

The --with-wide-int build doesn't need to use USE_LSB_TAG because
there's more than enough bits at the MSB end.

> > That and the previous item is a huge turn-off for me personally, FWIW,
> > because I use exactly these two features.
> 
> If you want !USE_LSB_TAG, someone (tm) could implement that :-). But
> it's not there now, because I didn't want to get lost in the lisp.h
> macro jungle while still learning MPS. Keep it simple, and so on...

OK, but I cannot do anything with a branch I cannot build on my main
development machine.  So here we are.

Re: MPS experiment successful

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> The packages code will need to be moved aside, yes.  But for the rest,
> why shouldn't a simple merge do the job?  They seem to be orthogonal,
> by and large.

Coming to think of it again - how would I tell git merge the difference
between the package related changes and GC related changes?

Re: MPS experiment successful

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 17 Apr 2024 20:41:12 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The packages code will need to be moved aside, yes.  But for the rest,
> > why shouldn't a simple merge do the job?  They seem to be orthogonal,
> > by and large.
> 
> Coming to think of it again - how would I tell git merge the difference
> between the package related changes and GC related changes?

Assuming you know at what commit you started working on MPS, make the
diffs relative to that commit.  If you didn't in parallel work on
packages as well, there should be no problem.  If you did make changes
to packages while working on MPS, tough.

(I'm actually surprised you didn't start a local branch for the MPS
stuff.  If you did, just make the diffs from that branch.)

Re: MPS experiment successful

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 17 Apr 2024 20:41:12 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > The packages code will need to be moved aside, yes.  But for the rest,
>> > why shouldn't a simple merge do the job?  They seem to be orthogonal,
>> > by and large.
>> 
>> Coming to think of it again - how would I tell git merge the difference
>> between the package related changes and GC related changes?
>
> Assuming you know at what commit you started working on MPS, make the
> diffs relative to that commit.  If you didn't in parallel work on
> packages as well, there should be no problem.  If you did make changes
> to packages while working on MPS, tough.

Thanks, I thought you meant git merge. Patch formatting could work, but
loses the history, of course. Which wouldn't be very interesting, unless
someone wants to watch me learning MPS. And Emacs internals :-).

> (I'm actually surprised you didn't start a local branch for the MPS
> stuff.  If you did, just make the diffs from that branch.)

Not sure what you mean.

I branched cl-packages from master a longer while ago, on Github. That
branch has the packages stuff, and I merged master into it once in a
while.

Then I branched the igc branch from cl-packages, so it has both the
package stuff and the MPS stuff. And I think I merged cl-packages into
igc once or twice.

And yes, a diff cl-package...igc could work, or what the syntax is.

Re: MPS experiment successful

[Dmitry Gutov]

On 17/04/2024 22:51, Gerd Möllmann wrote:
> And yes, a diff cl-package...igc could work, or what the syntax is.

Either git diff x...y or git log x...y. The latter should show you the 
list of your commits that are in y but not in x.

Re: MPS experiment successful

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 17/04/2024 22:51, Gerd Möllmann wrote:
>> And yes, a diff cl-package...igc could work, or what the syntax is.
>
> Either git diff x...y or git log x...y. The latter should show you the
> list of your commits that are in y but not in x.

Thanks.

Re: MPS experiment successful

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 17 Apr 2024 21:51:07 +0200
> 
> I branched cl-packages from master a longer while ago, on Github. That
> branch has the packages stuff, and I merged master into it once in a
> while.
> 
> Then I branched the igc branch from cl-packages, so it has both the
> package stuff and the MPS stuff. And I think I merged cl-packages into
> igc once or twice.

If you have a special branch for MPS, then the diffs from the point
you forked that branch is all you need.

Re: MPS experiment successful

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 17 Apr 2024 21:51:07 +0200
>> 
>> I branched cl-packages from master a longer while ago, on Github. That
>> branch has the packages stuff, and I merged master into it once in a
>> while.
>> 
>> Then I branched the igc branch from cl-packages, so it has both the
>> package stuff and the MPS stuff. And I think I merged cl-packages into
>> igc once or twice.
>
> If you have a special branch for MPS, then the diffs from the point
> you forked that branch is all you need.

Ok, there should now be a new branch scratch/igc containing the result
of my attempt to transfer the changes to master.

The branch builds for me without MPS. It cannot possibly compile with
MPS because that part uses CL packages, lacks support for obarrays etc.

The transfer was not entirely easy, and it's likely that I broke
something, even in the non-MPS case.

If someone has the time, it would be helpful to check that this is not
the case. I'm currently too ruthless to do that. Good candidates are
sort.c, fns.c, alloc.c, lisp.h.

Re: MPS experiment successful

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> Ok, there should now be a new branch scratch/igc containing the result
> of my attempt to transfer the changes to master.
>
> The branch builds for me without MPS. It cannot possibly compile with
> MPS because that part uses CL packages, lacks support for obarrays etc.
>
> The transfer was not entirely easy, and it's likely that I broke
> something, even in the non-MPS case.
>
> If someone has the time, it would be helpful to check that this is not
> the case. I'm currently too ruthless to do that. Good candidates are
> sort.c, fns.c, alloc.c, lisp.h.

I think the branch scratch/igc has now reached the same state as in my
local Emacs, unless I'm overlooking something of course. To quote myself
up-thread:

  If someone doesn't remember what MPS is about: concurrent, incremental,
  generational, mostly-copying, thread-safe GC.

  I've now reached the point that I wanted to reach to consider my
  experiment of using MPS in Emacs successful.

  - Emacs builds from scratch
  - Tests succeed to a sufficient degree
  - Emacs starts in GUI mode

  That's all. There is nothing more. And I'm currently undecided how to
  proceed with this.

Please note that I'm using macOS only. Can't say anything about other
platforms.

Re: MPS experiment successful

[Dmitry Gutov]

On 17/04/2024 17:30, Gerd Möllmann wrote:
> What MPS means for the overall speed of Emacs I find impossible to say.
> MPS doesn't seem to be slow at all though, although some slowdown can be
> expected in the client because of thread-safe allocations.
> 
> Example for a confusing fact that I noticed today:
> 
> Full build without MPS, -O0, checking
> 
> real	26:20.01
> user	1:32:15.54
> sys	3:20.74
> 
> Same build with debug MPS (-lmps-debug)
> 
> real	14:07.90
> user	44:17.94
> sys	3:15.99

That sounds pretty great, though.

One might have worried that a new GC would help exchange better (lower) 
latencies for lower throughput, but that doesn't seem to be the case.

Re: MPS experiment successful

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 17/04/2024 17:30, Gerd Möllmann wrote:
>> What MPS means for the overall speed of Emacs I find impossible to say.
>> MPS doesn't seem to be slow at all though, although some slowdown can be
>> expected in the client because of thread-safe allocations.
>> Example for a confusing fact that I noticed today:
>> Full build without MPS, -O0, checking
>> real	26:20.01
>> user	1:32:15.54
>> sys	3:20.74
>> Same build with debug MPS (-lmps-debug)
>> real	14:07.90
>> user	44:17.94
>> sys	3:15.99
>
> That sounds pretty great, though.
>
> One might have worried that a new GC would help exchange better
> (lower) latencies for lower throughput, but that doesn't seem to be
> the case.

Or it's a fluke of some kind I'm too dumb to understand. (Trying to damp
expectations a bit :-)).

Re: MPS experiment successful

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> That's all. There is nothing more. And I'm currently undecided how to
>>> proceed with this.
>>
>> The next step would be to push a feature branch into the Emacs Git
>> repository and let people try the branch on the other supported
>> platforms.  If the code is not yet mature enough for that, please
>> suggest how to get from here to there.
>
> I think technically it could be done, but there are some things that
> would need to be done. From the top of my head:
>
> - CL packages vs. obarrays (I do have packages but no obarrays)
> - Handling of pure space
> - Handling the new JSON stuff, which I don't have in my branch, I think.
> - ...
>
> And then of course the general work of getting a branch from a what is
> basically a fork back to the forked repo, which I'm not so sure how to
> go about that.
>
> But my undecidedness comes more from what could come after that. I'm not
> sure how willing I am to invest what can be years to get things
> finished, esp. if no one else starts working on it. Had that one time
> too often in the 90s, some might remember ;-).
>
>> Would you like to describe in a few words what would be the advantages
>> and disadvantages of this GC for Emacs?
>
> Advantages for users:
>
> For the user it means the final end to GC pauses. MPS runs in a
> different thread wich can be in a different core.
>
> What MPS means for the overall speed of Emacs I find impossible to say.
> MPS doesn't seem to be slow at all though, although some slowdown can be
> expected in the client because of thread-safe allocations.
>
> Example for a confusing fact that I noticed today:
>
> Full build without MPS, -O0, checking
>
> real	26:20.01
> user	1:32:15.54
> sys	3:20.74
>
> Same build with debug MPS (-lmps-debug)
>
> real	14:07.90
> user	44:17.94
> sys	3:15.99
>
> That's on an 2,3 GHz Quad-Core Intel Core i7, 16G RAM, SSD. 🤷

Hi Gerd,

sounds very interesting.  I think we'll need more performance figures
especially on non debug builds in the future to better undesrtand this.

OTOH even in case it proves to be a tiny bit slower having no pauses is
a big plus.

  Andrea

Re: MPS experiment successful

[Gerd Möllmann]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> I tried to built with branch.
> Any reason that you used sys/_types/_ssize_t.h?

Eglot/clangd sometimes inserts #includes. I guess that happened here,
too.

Re: MPS experiment successful

[Gerd Möllmann]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> After including sys/types.h instead everything builds fine.

I would guess that you didn't use --with-mps when configuring Emacs.
Sorry for not mentioning that. IOW, you can build that branch with and
without MPS.

To install MPS, the only way I know ATM on macOS is to build it from its
Github repo. But see some (very trivial) issues I submitted to the MPS
project on Github.

> What' the benefit of using mps?

For the user it means the end of GC pauses because GC runs concurrently
(in a different thread, on a different core).

Re: MPS experiment successful

[Björn Bidar]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>> After including sys/types.h instead everything builds fine.
>
> I would guess that you didn't use --with-mps when configuring Emacs.
> Sorry for not mentioning that. IOW, you can build that branch with and
> without MPS.

Even --with-mps works fine after the change.

Re: MPS experiment successful

[Gerd Möllmann]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>>> After including sys/types.h instead everything builds fine.
>>
>> I would guess that you didn't use --with-mps when configuring Emacs.
>> Sorry for not mentioning that. IOW, you can build that branch with and
>> without MPS.
>
> Even --with-mps works fine after the change.

Could you please check that src/Makefile has LIBMPS=-lmps in it, and
src/config.h defines HAVE_MPS?

Maybe something in configure doesn't work.

BTW, configure should print "Does Emacs use -lmps? yes", or something like
that at the end, in the summary output.