STatus of MPS branch

STatus of MPS branch

[Gerd Möllmann]

I've now fixed all aborts/bus errors and so on in scratch/igc that were
caused by things related to GC, and that I could identify from running
the tests.

  SUMMARY OF TEST RESULTS
  -----------------------
  Files examined: 513
  Ran 7602 tests, 7327 results as expected, 7 unexpected, 268 skipped
  3 files contained unexpected results:
    src/emacs-module-tests.log
    lisp/progmodes/eglot-tests.log
    lisp/net/tramp-tests.log

There are many things other people could help with at this point. Some
require no knowledge of MPS, or C. For example:

- Make it work on anything not macOS.

- Run tests with native compilation.

- Determine what the reason for the remaining test failure is.

- Figure out what is done in the old GC in garbage_collect that is not
  directly related to GC, and how to do that when GC is concurrent.
  Examples are buffer undo_list, window prev_buffers/next_buffers
  handling (which I don't understand at all, at present), and whatever
  else there is in garbage_collect/sweep.

- Figure out which Elisp functionality can't be supported with igc and
  what to do with them. A trivial example would be post-gc-hook.

- Or something that requires some MPS knowleage: implement weak hash
  tables.

- and so on, and so on...

If someone wants to help with something, I think it would be a good idea
to say so here.

Re: STatus of MPS branch

[Michael Albinus]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

Hi Gerd,

>   SUMMARY OF TEST RESULTS
>   -----------------------
>   Files examined: 513
>   Ran 7602 tests, 7327 results as expected, 7 unexpected, 268 skipped
>   3 files contained unexpected results:
>     src/emacs-module-tests.log
>     lisp/progmodes/eglot-tests.log
>     lisp/net/tramp-tests.log
>
> If someone wants to help with something, I think it would be a good idea
> to say so here.

I volunteer to check the Tramp errors. Could you please provide me the
tramp-tests.log file?

Best regards, Michael.

Re: STatus of MPS branch

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> Hi Gerd,

Hi Michael

>
>>   SUMMARY OF TEST RESULTS
>>   -----------------------
>>   Files examined: 513
>>   Ran 7602 tests, 7327 results as expected, 7 unexpected, 268 skipped
>>   3 files contained unexpected results:
>>     src/emacs-module-tests.log
>>     lisp/progmodes/eglot-tests.log
>>     lisp/net/tramp-tests.log
>>
>> If someone wants to help with something, I think it would be a good idea
>> to say so here.
>
> I volunteer to check the Tramp errors. Could you please provide me the
> tramp-tests.log file?

Thanks! Please find attached


[-- Attachment #2: tramp-tests.log --]
[-- Type: application/octet-stream, Size: 7049 bytes --]

Re: STatus of MPS branch

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Hi Michael

Hi Gerd,

Could you please instrument tramp-tests.el with the attached patch. Then
rerun the tests with

--8<---------------cut here---------------start------------->8---
# make -C test tramp-tests.log
--8<---------------cut here---------------end--------------->8---

and show the log file.

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1540 bytes --]

diff --git a/test/lisp/net/tramp-tests.el b/test/lisp/net/tramp-tests.el
index cdd2a1efdb2..560a52be946 100644
--- a/test/lisp/net/tramp-tests.el
+++ b/test/lisp/net/tramp-tests.el
@@ -3425,20 +3425,24 @@ tramp-test17-insert-directory
 	    (should (file-exists-p tmp-name2))
 	    (with-temp-buffer
 	      (insert-directory tmp-name1 nil)
+	      (message "Hallo 1\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should (looking-at-p (rx (literal tmp-name1)))))
 	    (with-temp-buffer
 	      (insert-directory (file-name-as-directory tmp-name1) nil)
+	      (message "Hallo 2\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should
                (looking-at-p (rx (literal (file-name-as-directory tmp-name1))))))
 	    (with-temp-buffer
 	      (insert-directory tmp-name1 "-al")
+	      (message "Hallo 3\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should
 	       (looking-at-p (rx bol (+ nonl) blank (literal tmp-name1) eol))))
 	    (with-temp-buffer
 	      (insert-directory (file-name-as-directory tmp-name1) "-al")
+	      (message "Hallo 4\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should
 	       (looking-at-p
@@ -3446,6 +3450,7 @@ tramp-test17-insert-directory
 	    (with-temp-buffer
 	      (insert-directory
 	       (file-name-as-directory tmp-name1) "-al" nil 'full-directory-p)
+	      (message "Hallo 5\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should
 	       (looking-at-p

Re: STatus of MPS branch

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 378 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Hi Michael
>
> Hi Gerd,
>
> Could you please instrument tramp-tests.el with the attached patch. Then
> rerun the tests with
>
> # make -C test tramp-tests.log
>
> and show the log file.
>
> Best regards, Michael.

Danke, Michael. Please find attached


[-- Attachment #2: tramp-tests.log --]
[-- Type: application/octet-stream, Size: 23734 bytes --]

Re: STatus of MPS branch

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I've now fixed all aborts/bus errors and so on in scratch/igc that were
> caused by things related to GC, and that I could identify from running
> the tests.
>
>   SUMMARY OF TEST RESULTS
>   -----------------------
>   Files examined: 513
>   Ran 7602 tests, 7327 results as expected, 7 unexpected, 268 skipped
>   3 files contained unexpected results:
>     src/emacs-module-tests.log
>     lisp/progmodes/eglot-tests.log
>     lisp/net/tramp-tests.log

In module-tests, the test module/function-finalizer fails.

The test expects that something is finalized after calling
garbage-collect. That is simply not the case with MPS. Finalization
happens at an unspecified time, MPS puts a finalization message on a
message queue which the client can process when it wants to.

So, this is not a problem.

I think I'll disable this test somehow.

Re: STatus of MPS branch

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Sun, 21 Apr 2024 06:38:58 +0200
> 
> In module-tests, the test module/function-finalizer fails.
> 
> The test expects that something is finalized after calling
> garbage-collect. That is simply not the case with MPS. Finalization
> happens at an unspecified time, MPS puts a finalization message on a
> message queue which the client can process when it wants to.
> 
> So, this is not a problem.
> 
> I think I'll disable this test somehow.

Instead of disabling it, it would be better to have some way of making
sure the finalizer works, even if that happens at some unspecified
time.  Maybe make the test wait for that, or maybe MPS has a way of
forcing GC to happen or something?

Re: STatus of MPS branch

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Instead of disabling it, it would be better to have some way of making
> sure the finalizer works, even if that happens at some unspecified
> time.  Maybe make the test wait for that, or maybe MPS has a way of
> forcing GC to happen or something?

You're right, but there is a ton of other stuff to do, and I have to
prioritize somehow.

(BTW, instead of disabling the test, I provided a new feature "mps" and
gave the test an :expected result accordingly.)

Re: STatus of MPS branch

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I've now fixed all aborts/bus errors and so on in scratch/igc that were
> caused by things related to GC, and that I could identify from running
> the tests.
>
>   SUMMARY OF TEST RESULTS
>   -----------------------
>   Files examined: 513
>   Ran 7602 tests, 7327 results as expected, 7 unexpected, 268 skipped
>   3 files contained unexpected results:
>     src/emacs-module-tests.log
>     lisp/progmodes/eglot-tests.log
>     lisp/net/tramp-tests.log
>
> If someone wants to help with something, I think it would be a good idea
> to say so here.

In eglot-tests, the log ends with

     passed  20/46  eglot-test-non-unique-completions (2.539077 sec)
    skipped  21/46  eglot-test-path-to-uri-windows (0.000435 sec)
    skipped  22/46  eglot-test-project-wide-diagnostics-rust-analyzer (0.004449 sec)
    skipped  23/46  eglot-test-project-wide-diagnostics-typescript (0.005207 sec)
  [eglot-tests] [eglot-test-rename-a-symbol]: test start
  [eglot] Connected! Server `clangd' now managing `(c-mode c-ts-mode c++-mode c++-ts-mode objc-mode)' buffers in project `rename-project'.
  [eglot] Server wants to edit:
    /var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/eglot--fixture-3puiY5/rename-project/main.c (2 changes)
    /private/var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/eglot--fixture-3puiY5/rename-project/main.c (2 changes)
  Proceed? (y or n)

I don't understand the test, and won't do a deep dive into Eglot at this
point, so I'll ignore this. It's not obvious to me how that relates to
the new GC, if it does.

If someone could debug this, that weould be nice.

Re: STatus of MPS branch

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 1358 bytes --]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

Hi Gerd,

>> Could you please instrument tramp-tests.el with the attached patch. Then
>> rerun the tests with
>>
>> # make -C test tramp-tests.log
>>
>> and show the log file.
>
> Danke, Michael. Please find attached

Well, this is weird. "Hallo 1" shows the invocation of
'(insert-directory tmp-name1 nil)', which should just return the file
name. In my case, it is

--8<---------------cut here---------------start------------->8---
Hallo 1
/mock:gandalf:/tmp/tramp-test9LSbRf
--8<---------------cut here---------------end--------------->8---

But your log shows the long listing

--8<---------------cut here---------------start------------->8---
Hallo 1
  drwxr-xr-x 3 gerd staff 96 Apr 20 18:40 /mock:Pro.fritz.box:/var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/tramp-testT6CzoZ
--8<---------------cut here---------------end--------------->8---

Hmm, we must write more traces. Could you, pls, apply the extended
patch?. Running the test could be restricted to the test case in
question via

--8<---------------cut here---------------start------------->8---
# make -C test tramp-tests.log SELECTOR=tramp-test17-insert-directory
--8<---------------cut here---------------end--------------->8---

Let's see what tramp-tests.log tells us.

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2258 bytes --]

diff --git a/test/lisp/net/tramp-tests.el b/test/lisp/net/tramp-tests.el
index cdd2a1efdb2..a575b63b78e 100644
--- a/test/lisp/net/tramp-tests.el
+++ b/test/lisp/net/tramp-tests.el
@@ -3410,6 +3410,7 @@ tramp-test17-insert-directory
   ;; one file.  So we refrain from testing.
   (skip-unless (not (tramp--test-ange-ftp-p)))

+  (tramp--test-instrument-test-case 6
   (dolist (quoted (if (tramp--test-expensive-test-p) '(nil t) '(nil)))
     (let* ((tmp-name1
             (expand-file-name (tramp--test-make-temp-name nil quoted)))
@@ -3425,20 +3426,24 @@ tramp-test17-insert-directory
 	    (should (file-exists-p tmp-name2))
 	    (with-temp-buffer
 	      (insert-directory tmp-name1 nil)
+	      (message "Hallo 1\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should (looking-at-p (rx (literal tmp-name1)))))
 	    (with-temp-buffer
 	      (insert-directory (file-name-as-directory tmp-name1) nil)
+	      (message "Hallo 2\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should
                (looking-at-p (rx (literal (file-name-as-directory tmp-name1))))))
 	    (with-temp-buffer
 	      (insert-directory tmp-name1 "-al")
+	      (message "Hallo 3\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should
 	       (looking-at-p (rx bol (+ nonl) blank (literal tmp-name1) eol))))
 	    (with-temp-buffer
 	      (insert-directory (file-name-as-directory tmp-name1) "-al")
+	      (message "Hallo 4\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should
 	       (looking-at-p
@@ -3446,6 +3451,7 @@ tramp-test17-insert-directory
 	    (with-temp-buffer
 	      (insert-directory
 	       (file-name-as-directory tmp-name1) "-al" nil 'full-directory-p)
+	      (message "Hallo 5\n%s" (tramp-get-buffer-string))
 	      (goto-char (point-min))
 	      (should
 	       (looking-at-p
@@ -3481,7 +3487,7 @@ tramp-test17-insert-directory
 	       :type 'file-missing)))

 	;; Cleanup.
-	(ignore-errors (delete-directory tmp-name1 'recursive))))))
+	(ignore-errors (delete-directory tmp-name1 'recursive)))))))

 (ert-deftest tramp-test17-dired-with-wildcards ()
   "Check `dired' with wildcards."

Re: STatus of MPS branch

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> - Run tests with native compilation.

I'm oding that. It currently does not build with native compilation.

Error: native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers")
  signal(native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers"))
  comp--native-compile("emacs-lisp/bytecomp.el")

which sounds like it could have to do with moving pointers.

Re: STatus of MPS branch

[Gerd Möllmann]

Michael Albinus <michael.albinus@gmx.de> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> Hi Gerd,
>
>>> Could you please instrument tramp-tests.el with the attached patch. Then
>>> rerun the tests with
>>>
>>> # make -C test tramp-tests.log
>>>
>>> and show the log file.
>>
>> Danke, Michael. Please find attached
>
> Well, this is weird. "Hallo 1" shows the invocation of
> '(insert-directory tmp-name1 nil)', which should just return the file
> name. In my case, it is
>
> Hallo 1
> /mock:gandalf:/tmp/tramp-test9LSbRf
>
>
> But your log shows the long listing
>
> Hallo 1
>   drwxr-xr-x 3 gerd staff 96 Apr 20 18:40 /mock:Pro.fritz.box:/var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/tramp-testT6CzoZ
>
>
> Hmm, we must write more traces. Could you, pls, apply the extended
> patch?. Running the test could be restricted to the test case in
> question via
>
> # make -C test tramp-tests.log SELECTOR=tramp-test17-insert-directory
>
> Let's see what tramp-tests.log tells us.

Thanks. Before I do that, could this be caused by macOS' ls vs. gls?

  % gls --version
  ls (GNU coreutils) 9.5

while ls is the macOS one (probably from FreeBSD, but the man page
doesn't say so).

Re: STatus of MPS branch

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>> Hi Gerd,
>>
>>>> Could you please instrument tramp-tests.el with the attached patch. Then
>>>> rerun the tests with
>>>>
>>>> # make -C test tramp-tests.log
>>>>
>>>> and show the log file.
>>>
>>> Danke, Michael. Please find attached
>>
>> Well, this is weird. "Hallo 1" shows the invocation of
>> '(insert-directory tmp-name1 nil)', which should just return the file
>> name. In my case, it is
>>
>> Hallo 1
>> /mock:gandalf:/tmp/tramp-test9LSbRf
>>
>>
>> But your log shows the long listing
>>
>> Hallo 1
>>   drwxr-xr-x 3 gerd staff 96 Apr 20 18:40 /mock:Pro.fritz.box:/var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/tramp-testT6CzoZ
>>
>>
>> Hmm, we must write more traces. Could you, pls, apply the extended
>> patch?. Running the test could be restricted to the test case in
>> question via
>>
>> # make -C test tramp-tests.log SELECTOR=tramp-test17-insert-directory
>>
>> Let's see what tramp-tests.log tells us.
>
> Thanks. Before I do that, could this be caused by macOS' ls vs. gls?
>
>   % gls --version
>   ls (GNU coreutils) 9.5
>
> while ls is the macOS one (probably from FreeBSD, but the man page
> doesn't say so).

I found this in mit init.el

  (if (string-match "Pro" system-name)
      (setq insert-directory-program "/usr/local/bin/gls")
    (setq insert-directory-program "/opt/homebrew/bin/gls"))

meaning that the tests wouldn't do that, right?

Re: STatus of MPS branch

[Michael Albinus]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

Hi Gerd,

>> Thanks. Before I do that, could this be caused by macOS' ls vs. gls?
>>
>>   % gls --version
>>   ls (GNU coreutils) 9.5
>>
>> while ls is the macOS one (probably from FreeBSD, but the man page
>> doesn't say so).
>
> I found this in mit init.el
>
>   (if (string-match "Pro" system-name)
>       (setq insert-directory-program "/usr/local/bin/gls")
>     (setq insert-directory-program "/opt/homebrew/bin/gls"))
>
> meaning that the tests wouldn't do that, right?

Perhaps. What does '/usr/local/bin/gls <filename>' return?

Best regards, Michael.

Re: STatus of MPS branch

[Gerd Möllmann]

Michael Albinus <michael.albinus@gmx.de> writes:

>> I found this in mit init.el
>>
>>   (if (string-match "Pro" system-name)
>>       (setq insert-directory-program "/usr/local/bin/gls")
>>     (setq insert-directory-program "/opt/homebrew/bin/gls"))
>>
>> meaning that the tests wouldn't do that, right?
>
> Perhaps. What does '/usr/local/bin/gls <filename>' return?

/Users/gerd/emacs/github/cl-packages % gls ~/.zshenv 
/Users/gerd/.zshenv
/Users/gerd/emacs/github/cl-packages % ls ~/.zshenv 
/Users/gerd/.zshenv

But the ls output is in purple, on the terminal.

Re: STatus of MPS branch

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> Hi Gerd,
>
>>> Could you please instrument tramp-tests.el with the attached patch. Then
>>> rerun the tests with
>>>
>>> # make -C test tramp-tests.log
>>>
>>> and show the log file.
>>
>> Danke, Michael. Please find attached
>
> Well, this is weird. "Hallo 1" shows the invocation of
> '(insert-directory tmp-name1 nil)', which should just return the file
> name. In my case, it is
>
> Hallo 1
> /mock:gandalf:/tmp/tramp-test9LSbRf
>
>
> But your log shows the long listing
>
> Hallo 1
>   drwxr-xr-x 3 gerd staff 96 Apr 20 18:40 /mock:Pro.fritz.box:/var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/tramp-testT6CzoZ
>
>
> Hmm, we must write more traces. Could you, pls, apply the extended
> patch?. Running the test could be restricted to the test case in
> question via
>
> # make -C test tramp-tests.log SELECTOR=tramp-test17-insert-directory
>
> Let's see what tramp-tests.log tells us.

Please find attached


[-- Attachment #2: tramp-test.log --]
[-- Type: application/octet-stream, Size: 41005 bytes --]

Re: STatus of MPS branch

[Michael Albinus]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>>> I found this in mit init.el
>>>
>>>   (if (string-match "Pro" system-name)
>>>       (setq insert-directory-program "/usr/local/bin/gls")
>>>     (setq insert-directory-program "/opt/homebrew/bin/gls"))
>>>
>>> meaning that the tests wouldn't do that, right?
>>
>> Perhaps. What does '/usr/local/bin/gls <filename>' return?
>
> /Users/gerd/emacs/github/cl-packages % gls ~/.zshenv 
> /Users/gerd/.zshenv
> /Users/gerd/emacs/github/cl-packages % ls ~/.zshenv 
> /Users/gerd/.zshenv

Looks good.

> But the ls output is in purple, on the terminal.

No problem, Tramp uses terminal type "dumb". And it filters remaining
ANSI control escape sequences.

Please run now the instrumented test, hopefully, we know more then.

Best regards, Michael.

Re: STatus of MPS branch

[Gerd Möllmann]

Michael Albinus <michael.albinus@gmx.de> writes:

> Please run now the instrumented test, hopefully, we know more then.

Should be in another mail. Soory, had to build w/o native-comp first.

Re: STatus of MPS branch

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> - Run tests with native compilation.
>
> I'm oding that. It currently does not build with native compilation.
>
> Error: native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers")
>   signal(native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers"))
>   comp--native-compile("emacs-lisp/bytecomp.el")
>
> which sounds like it could have to do with moving pointers.

Ok, as a starter, there are some Lisp_Object members of comp_t that are
not staticpro'd, so something has to be done for a moving collector.

@Andrea: Don't know, should things like comp_t::driver_options be
staticpro'd for the "normal" GC? If so, things would automatically work
with MPS OOTB.

Re: STatus of MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> - Run tests with native compilation.
>
> I'm oding that. It currently does not build with native compilation.
>
> Error: native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers")
>   signal(native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers"))
>   comp--native-compile("emacs-lisp/bytecomp.el")
>
> which sounds like it could have to do with moving pointers.

Hi Gerd,

does scratch/igc support hash table with custom defined test?  Is
`sxhash-equal-including-properties` supposed to work on the branch?

Thanks

  Andrea

Re: STatus of MPS branch

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 21 Apr 2024 07:39:33 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Instead of disabling it, it would be better to have some way of making
> > sure the finalizer works, even if that happens at some unspecified
> > time.  Maybe make the test wait for that, or maybe MPS has a way of
> > forcing GC to happen or something?
> 
> You're right, but there is a ton of other stuff to do, and I have to
> prioritize somehow.

I think making sure that finalizers work is important, so its priority
should not low.  It's basically one of the important traits of our GC.

Re: STatus of MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>>> - Run tests with native compilation.
>>
>> I'm oding that. It currently does not build with native compilation.
>>
>> Error: native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers")
>>   signal(native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers"))
>>   comp--native-compile("emacs-lisp/bytecomp.el")
>>
>> which sounds like it could have to do with moving pointers.
>
> Ok, as a starter, there are some Lisp_Object members of comp_t that are
> not staticpro'd, so something has to be done for a moving collector.
>
> @Andrea: Don't know, should things like comp_t::driver_options be
> staticpro'd for the "normal" GC? If so, things would automatically work
> with MPS OOTB.

I can't recall any reason why they should not be staticpro'd, so yes I
think you can static protect them and see if it works further.

From the changes I see in the branch I think should not be too hard to
have native comp working on it (last famous words).

Thanks

  Andrea

Re: Eglot tests on MPS branch (was: STatus of MPS branch)

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Sun, 21 Apr 2024 08:15:10 +0200
> 
> In eglot-tests, the log ends with
> 
>      passed  20/46  eglot-test-non-unique-completions (2.539077 sec)
>     skipped  21/46  eglot-test-path-to-uri-windows (0.000435 sec)
>     skipped  22/46  eglot-test-project-wide-diagnostics-rust-analyzer (0.004449 sec)
>     skipped  23/46  eglot-test-project-wide-diagnostics-typescript (0.005207 sec)
>   [eglot-tests] [eglot-test-rename-a-symbol]: test start
>   [eglot] Connected! Server `clangd' now managing `(c-mode c-ts-mode c++-mode c++-ts-mode objc-mode)' buffers in project `rename-project'.
>   [eglot] Server wants to edit:
>     /var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/eglot--fixture-3puiY5/rename-project/main.c (2 changes)
>     /private/var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/eglot--fixture-3puiY5/rename-project/main.c (2 changes)
>   Proceed? (y or n)
> 
> I don't understand the test, and won't do a deep dive into Eglot at this
> point, so I'll ignore this. It's not obvious to me how that relates to
> the new GC, if it does.
> 
> If someone could debug this, that weould be nice.

I added João, but please start a new thread for each group of related
tests that fail, so as not to lump all of them in this single thread.
This thread is already too large and includes too many unrelated
problems.

I redirected this discussion to a separate thread just now.

Re: STatus of MPS branch

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 21 Apr 2024 07:39:33 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Instead of disabling it, it would be better to have some way of making
>> > sure the finalizer works, even if that happens at some unspecified
>> > time.  Maybe make the test wait for that, or maybe MPS has a way of
>> > forcing GC to happen or something?
>> 
>> You're right, but there is a ton of other stuff to do, and I have to
>> prioritize somehow.
>
> I think making sure that finalizers work is important, so its priority
> should not low.  It's basically one of the important traits of our GC.

Sure, but not now, sorry.

Re: STatus of MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> does scratch/igc support hash table with custom defined test?  Is
> `sxhash-equal-including-properties` supposed to work on the branch?

It should support everything as before, except weak tables which aren't
implemented yet.

Re: STatus of MPS branch

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: emacs-devel <emacs-devel@gnu.org>
> Date: Sun, 21 Apr 2024 09:39:25 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> Hi Gerd,
> 
> >> Could you please instrument tramp-tests.el with the attached patch. Then
> >> rerun the tests with
> >>
> >> # make -C test tramp-tests.log
> >>
> >> and show the log file.
> >
> > Danke, Michael. Please find attached
> 
> Well, this is weird. "Hallo 1" shows the invocation of
> '(insert-directory tmp-name1 nil)', which should just return the file
> name. In my case, it is
> 
> --8<---------------cut here---------------start------------->8---
> Hallo 1
> /mock:gandalf:/tmp/tramp-test9LSbRf
> --8<---------------cut here---------------end--------------->8---

Please move this discussion to a separate thread, with a suitable
Subject line.  And please in the future start a new thread for
discussing each new problem with the test suite on that branch.

TIA

Re: Native compilation on MPS branch (was: STatus of MPS branch)

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Sun, 21 Apr 2024 09:49:50 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > - Run tests with native compilation.
> 
> I'm oding that. It currently does not build with native compilation.
> 
> Error: native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers")
>   signal(native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers"))
>   comp--native-compile("emacs-lisp/bytecomp.el")
> 
> which sounds like it could have to do with moving pointers.

I added Andrea and moved this to a separate thread.

Re: Native compilation on MPS branch

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Andrea Corallo <acorallo@gnu.org>
> Date: Sun, 21 Apr 2024 10:45:44 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> >
> >> - Run tests with native compilation.
> >
> > I'm oding that. It currently does not build with native compilation.
> >
> > Error: native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers")
> >   signal(native-ice ("emacs-lisp/bytecomp.el" "can't find data in relocation containers"))
> >   comp--native-compile("emacs-lisp/bytecomp.el")
> >
> > which sounds like it could have to do with moving pointers.
> 
> Ok, as a starter, there are some Lisp_Object members of comp_t that are
> not staticpro'd, so something has to be done for a moving collector.
> 
> @Andrea: Don't know, should things like comp_t::driver_options be
> staticpro'd for the "normal" GC? If so, things would automatically work
> with MPS OOTB.

Andrea, please reply in this new thread, dedicated to this particular
issue.

Thanks.

Re: STatus of MPS branch

[Michael Albinus]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

Hi Gerd,

>> Hmm, we must write more traces. Could you, pls, apply the extended
>> patch?. Running the test could be restricted to the test case in
>> question via
>>
>> # make -C test tramp-tests.log SELECTOR=tramp-test17-insert-directory
>>
>> Let's see what tramp-tests.log tells us.
>
> Please find attached

And it tells us the truth™.

--8<---------------cut here---------------start------------->8---
10:15:35.807389 tramp-send-command (6) # /usr/local/bin/gls --color=never --quoting-style\=literal --show-control-chars --dired -d tramp-test9vvifz 2>/dev/null
10:15:35.815699 tramp-wait-for-regexp (6) #
  drwxr-xr-x 3 gerd staff 96 Apr 21 10:15 tramp-test9vvifz
//DIRED// 42 58
//DIRED-OPTIONS// --quoting-style=literal
///3ce82232b01ca3404a92011c73a2fd60#$
--8<---------------cut here---------------end--------------->8---

Your gls command returns the long listing of the file, although not
instructed via an "-l" option. I've compared it with gls on a FreeBSD
machine, which behaves as Tramp expects:

--8<---------------cut here---------------start------------->8---
% gls --version
ls (GNU coreutils) 9.1
% gls --color=never --quoting-style\=literal --show-control-chars --dired -d lisp
lisp
--8<---------------cut here---------------end--------------->8---

This seems to be a recent change.
<https://www.gnu.org/software/coreutils/manual/coreutils.html#What-information-is-listed>,
which is about coreutils 9.5, says:

--8<---------------cut here---------------start------------->8---
The --dired (-D) option implies long format output with hyperlinks
disabled, and takes precedence over previously specified output formats
or hyperlink mode.
--8<---------------cut here---------------end--------------->8---

OTOH, the coreutils 9.1 info manual on that machine claims

--8<---------------cut here---------------start------------->8---
The ‘--dired’ (‘-D’) option has well-defined behavior only when
long format is in effect and hyperlinks are disabled (e.g.,
‘--hyperlink=none’).
--8<---------------cut here---------------end--------------->8---

So I have to fix it in Tramp.

Best regards, Michael.

Re: Eglot tests on MPS branch (was: STatus of MPS branch)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2300 bytes --]

On Sun, Apr 21, 2024, 10:09 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Gerd Möllmann <gerd.moellmann@gmail.com>
> > Date: Sun, 21 Apr 2024 08:15:10 +0200
> >
> > In eglot-tests, the log ends with
> >
> >      passed  20/46  eglot-test-non-unique-completions (2.539077 sec)
> >     skipped  21/46  eglot-test-path-to-uri-windows (0.000435 sec)
> >     skipped  22/46  eglot-test-project-wide-diagnostics-rust-analyzer
> (0.004449 sec)
> >     skipped  23/46  eglot-test-project-wide-diagnostics-typescript
> (0.005207 sec)
> >   [eglot-tests] [eglot-test-rename-a-symbol]: test start
> >   [eglot] Connected! Server `clangd' now managing `(c-mode c-ts-mode
> c++-mode c++-ts-mode objc-mode)' buffers in project `rename-project'.
> >   [eglot] Server wants to edit:
> >
>  /var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/eglot--fixture-3puiY5/rename-project/main.c
> (2 changes)
> >
>  /private/var/folders/1d/k_6t25f94sl83szqbf8gpkrh0000gn/T/eglot--fixture-3puiY5/rename-project/main.c
> (2 changes)
> >   Proceed? (y or n)
> >
> > I don't understand the test, and won't do a deep dive into Eglot at this
> > point, so I'll ignore this. It's not obvious to me how that relates to
> > the new GC, if it does.
> >
> > If someone could debug this, that weould be nice.
>
> I added João


I'm afraid I don't know what the MPS branch is. Eglot tests were
passing  today on master after a period of instability that was solved
recently (thursday/friday)".  I'm only skipping 5 tests here (but the CI
will likely skip more if it doesn't have more than clangd installed.

Ran 49 tests, 44 results as expected, 0 unexpected, 5 skipped (2024-04-21
10:48:14+0100, 23.782101 sec)

5 skipped results:
  SKIPPED  eglot-test-eclipse-connect
  SKIPPED  eglot-test-json-basic
  SKIPPED  eglot-test-path-to-uri-windows
  SKIPPED  eglot-test-snippet-completions
  SKIPPED  eglot-test-snippet-completions-with-company

The eglot-test-rename-a-symbol test asks the server to rename a symbol in
a source file in a project.  It checks that the rename is done correctly
and automatically.  It should not be interactive -- as the transcript
seems to indicate -- unless the eglot-confirm-server-edits user option
has been set up to explicitly request this.

João

[-- Attachment #2: Type: text/html, Size: 3461 bytes --]

Re: STatus of MPS branch

[Gerd Möllmann]

Michael Albinus <michael.albinus@gmx.de> writes:

> So I have to fix it in Tramp.

Good news ;-). C-u 1000 M-x thanks.

Re: Eglot tests on MPS branch

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> I'm afraid I don't know what the MPS branch is.

It's a GC based on Ravenbrook MPS (concurrent, incremental,
generational, mostly-copying, thread-safe).

Early stage (almost exactly 2 months since inception), most tests pass,
and so on.

I'm currently ignoring the test failure because of the early stage.

Re: Native compilation on MPS branch

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Ok, as a starter, there are some Lisp_Object members of comp_t that are
>> not staticpro'd, so something has to be done for a moving collector.
>> 
>> @Andrea: Don't know, should things like comp_t::driver_options be
>> staticpro'd for the "normal" GC? If so, things would automatically work
>> with MPS OOTB.
>
> Andrea, please reply in this new thread, dedicated to this particular
> issue.

The staticpro's alas didn't suffice. I'll debug that, but probably not
today. Thanks!

Re: Eglot tests on MPS branch (was: STatus of MPS branch)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 21 Apr 2024 10:53:25 +0100
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> I'm afraid I don't know what the MPS branch is.

It's the scratch/igc branch in Git.  It uses a different GC
implementation.  See

  https://lists.gnu.org/archive/html/emacs-devel/2024-04/msg00322.html

for more details.

Tramp tests in MPS branch (was: STatus of MPS branch)

[Michael Albinus]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>> So I have to fix it in Tramp.
>
> Good news ;-). C-u 1000 M-x thanks.

I've pushed a fix to the master branch. Could you, pls, check?

Best regards, Michael.

Re: Eglot tests on MPS branch

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

Maybe that new GC is corrupting the list that variable points to, dunno. If
it tracks master, that test should pass. It isn't even one of Eglot's
unstable tests. I'm afraid I can't be of more help.



On Sun, Apr 21, 2024, 11:13 Gerd Möllmann <gerd.moellmann@gmail.com> wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> > I'm afraid I don't know what the MPS branch is.
>
> It's a GC based on Ravenbrook MPS (concurrent, incremental,
> generational, mostly-copying, thread-safe).
>
> Early stage (almost exactly 2 months since inception), most tests pass,
> and so on.
>
> I'm currently ignoring the test failure because of the early stage.
>

[-- Attachment #2: Type: text/html, Size: 1120 bytes --]

Re: Tramp tests in MPS branch

[Gerd Möllmann]

Michael Albinus <michael.albinus@gmx.de> writes:

> I've pushed a fix to the master branch. Could you, pls, check?

Verified in master

Ran 97 tests, 82 results as expected, 0 unexpected, 15 skipped (2024-04-21 14:42:49+0200, 138.543946 sec)

Re: Native compilation on MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>> Ok, as a starter, there are some Lisp_Object members of comp_t that are
>>> not staticpro'd, so something has to be done for a moving collector.
>>> 
>>> @Andrea: Don't know, should things like comp_t::driver_options be
>>> staticpro'd for the "normal" GC? If so, things would automatically work
>>> with MPS OOTB.
>>
>> Andrea, please reply in this new thread, dedicated to this particular
>> issue.
>
> The staticpro's alas didn't suffice. I'll debug that, but probably not
> today. Thanks!

Keep us updated, I hope I can find some time to dedicate to the MPS
effor the upcoming week.

Thanks

  Andrea

Re: STatus of MPS branch

[Helmut Eller]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

On Sat, Apr 20 2024, Gerd Möllmann wrote:

> There are many things other people could help with at this point. Some
> require no knowledge of MPS, or C. For example:
>
> - Make it work on anything not macOS.

I'm trying to build the igc branch on Debian.  But I get a SIGSEGV when
loaddefs-gen.el gets loaded.

I configured with:
  CFLAGS='-g -O0 -I/scratch/emacs/mps-install/include -L/scratch/emacs/mps-install/lib' configure --enable-checking=yes --without-all --without-x --with-mps=debug

and made some changes to igc.c as seen in the diff below; but those seem
rather innocent.

GCC still emits this warning:

igc.c:229:18: warning: ‘pvec_type’ is narrower than values of its type
  229 |   enum pvec_type pvec_type : IGC_PVEC_BITS;

Is this something to worry about?

Helmut


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: igc.diff --]
[-- Type: text/x-diff, Size: 2117 bytes --]

diff --git a/src/igc.c b/src/igc.c
index dfb25d6c824..5de83980863 100644
--- a/src/igc.c
+++ b/src/igc.c
@@ -73,7 +73,7 @@
 static void
 igc_assert_fail (const char *file, unsigned line, const char *msg)
 {
-  extern void die (const char *, const char *, int);
+  //extern void die (const char *, const char *, int);
   die (msg, file, line);
 }
 
@@ -81,7 +81,7 @@ igc_assert_fail (const char *file, unsigned line, const char *msg)
 # define igc_assert(expr)                         \
    if (!(expr))                                   \
      igc_assert_fail (__FILE__, __LINE__, #expr); \
-   else
+   /* else */
 #else
 # define igc_assert(expr) (void) 9
 #endif
@@ -122,7 +122,7 @@ is_aligned (const mps_addr_t addr)
 #define IGC_CHECK_RES(res) \
   if ((res) != MPS_RES_OK) \
     emacs_abort ();        \
-  else
+  /* else */
 
 #define IGC_WITH_PARKED(gc)                        \
   for (int i = (mps_arena_park (gc->arena), 1); i; \
@@ -801,6 +801,7 @@ scan_pure (mps_ss_t ss, void *start, void *end, void *closure)
   MPS_SCAN_BEGIN (ss)
   {
     igc_assert (start == (void *) pure);
+    extern ptrdiff_t pure_bytes_used_lisp;
     end = (char *) pure + pure_bytes_used_lisp;
     if (end > start)
       IGC_FIX_CALL (ss, scan_ambig (ss, start, end, NULL));
@@ -954,6 +955,7 @@ fix_itree_node (mps_ss_t ss, struct itree_node *n)
 static mps_res_t
 fix_image (mps_ss_t ss, struct image *i)
 {
+#ifdef HAVE_WINDOW_SYSTEM
   MPS_SCAN_BEGIN (ss)
   {
     IGC_FIX12_OBJ (ss, &i->spec);
@@ -964,6 +966,7 @@ fix_image (mps_ss_t ss, struct image *i)
   }
   MPS_SCAN_END (ss);
   return MPS_RES_OK;
+#endif
 }
 
 static mps_res_t
@@ -2342,6 +2345,7 @@ thread_ap (enum igc_obj_type type)
     case IGC_OBJ_FLOAT:
       return t->d.leaf_ap;
     }
+  emacs_abort ();
 }
 
 /* Conditional breakpoints can be so slow that it is often more
@@ -2572,12 +2576,14 @@ igc_make_itree_node (void)
   return n;
 }
 
+#ifdef HAVE_WINDOW_SYSTEM
 struct image *
 igc_make_image (void)
 {
   struct image *img = alloc (sizeof *img, IGC_OBJ_IMAGE, PVEC_FREE);
   return img;
 }
+#endif
 
 struct face *
 igc_make_face (void)

[-- Attachment #3: gdb-session.txt --]
[-- Type: text/plain, Size: 7875 bytes --]

(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
`/scratch/emacs/emacs-igc/src/bootstrap-emacs' has changed; re-reading symbols.
Starting program: /scratch/emacs/emacs-igc/src/bootstrap-emacs -batch --no-site-file --no-site-lisp -l ./emacs-lisp/loaddefs-gen.elc -f loaddefs-generate--emacs-batch . ./calc ./calendar ./cedet ./cedet/ede ./cedet/semantic ./cedet/semantic/analyze ./cedet/semantic/bovine ./cedet/semantic/decorate ./cedet/semantic/symref ./cedet/semantic/wisent ./cedet/srecode ./emacs-lisp ./emulation ./erc ./eshell ./gnus ./image ./international ./language ./leim ./leim/quail ./mail ./mh-e ./net ./nxml ./org ./play ./progmodes ./textmodes ./url ./use-package ./vc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
  INFO     Scraping files for loaddefs... 
  INFO     Scraping files for loaddefs...10% 
  INFO     Scraping files for loaddefs...20% 
  INFO     Scraping files for loaddefs...30% 
  INFO     Scraping files for loaddefs...40% 
  INFO     Scraping files for loaddefs...50% 
  INFO     Scraping files for loaddefs...60% 
  INFO     Scraping files for loaddefs...70% 
  INFO     Scraping files for loaddefs...80% 

Program received signal SIGSEGV, Segmentation fault.

Breakpoint 1, handle_sigsegv (sig=11, 
    siginfo=0x555555f6ed70 <sigsegv_stack+64784>, 
    arg=0x555555f6ec40 <sigsegv_stack+64480>) at sysdep.c:1930
1930      bool fatal = gc_in_progress;
(gdb) xbacktrace 
"help-add-fundoc-usage" (0xe5f03550)
0xe8406ec0 PVEC_COMPILED
0xe8406fd0 PVEC_COMPILED
"loaddefs-generate--make-autoload" (0xe5f03280)
"loaddefs-generate--make-autoload" (0xe5f03200)
"loaddefs-generate--parse-file" (0xe5f03140)
"loaddefs-generate" (0xe5f03048)
"loaddefs-generate--emacs-batch" (0xffffc058)
"funcall" (0xffffc050)
"if" (0xffffc280)
"cond" (0xffffc430)
"let*" (0xffffc630)
"while" (0xffffc800)
"let*" (0xffffca00)
"progn" (0xffffcb70)
"if" (0xffffcce0)
"let" (0xffffcf00)
"let" (0xffffd120)
"command-line-1" (0xffffd2b0)
"let" (0xffffd5e0)
"command-line" (0xffffd770)
"unwind-protect" (0xffffd9e0)
"let" (0xffffdc00)
"if" (0xffffdda0)
"normal-top-level" (0xffffdf30)
(gdb) backtrace 
#0  handle_sigsegv (sig=11, siginfo=0x555555f6ed70 <sigsegv_stack+64784>, 
    arg=0x555555f6ec40 <sigsegv_stack+64480>) at sysdep.c:1930
#1  <signal handler called>
#2  string_intervals (s=XIL(0x7ffff0698edc))
    at /scratch/emacs/emacs-igc/src/lisp.h:4063
#3  0x0000555555767d97 in concat_to_string (nargs=3, args=0x7fffe5f035b8)
    at fns.c:957
#4  0x0000555555767373 in Fconcat (nargs=3, args=0x7fffe5f035b8) at fns.c:746
#5  0x00005555557b466a in exec_byte_code (fun=XIL(0x7fffe64689a5), 
    args_template=257, nargs=1, args=0x7fffe5f03628) at bytecode.c:1229
#6  0x000055555575e1fc in funcall_lambda (fun=XIL(0x7fffe840faad), nargs=0, 
    arg_vector=0x7fffffffc058) at eval.c:3200
#7  0x000055555575d4e7 in funcall_general (fun=XIL(0x7fffe840faad), numargs=0, 
    args=0x7fffffffc058) at eval.c:2982
#8  0x000055555575d7c4 in Ffuncall (nargs=1, args=0x7fffffffc050)
    at eval.c:3032
#9  0x000055555575c088 in eval_sub (form=XIL(0x7fffe7036d63)) at eval.c:2507
#10 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#11 0x0000555555756eae in Fif (args=XIL(0x7fffe7036d33)) at eval.c:395
#12 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7036d23)) at eval.c:2486
#13 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#14 0x0000555555756f45 in Fcond (args=XIL(0x7fffe7031013)) at eval.c:419
#15 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7030fe3)) at eval.c:2486
#16 0x0000555555756fcd in Fprogn (body=XIL(0x7fffe7030f43)) at eval.c:439
#17 0x000055555575855c in FletX (args=XIL(0x7fffe7030f03)) at eval.c:979
#18 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7030ef3)) at eval.c:2486
#19 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#20 0x0000555555756ffd in prog_ignore (body=XIL(0x7fffe7030ee3)) at eval.c:450
#21 0x00005555557589a6 in Fwhile (args=XIL(0x7fffe7030ed3)) at eval.c:1068
#22 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7030ec3)) at eval.c:2486
#23 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#24 0x000055555575855c in FletX (args=XIL(0x7fffe7030e83)) at eval.c:979
#25 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7030e73)) at eval.c:2486
#26 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#27 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7030e53)) at eval.c:2486
#28 0x0000555555756e8c in Fif (args=XIL(0x7fffe7030e33)) at eval.c:394
#29 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7030e23)) at eval.c:2486
#30 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#31 0x000055555575893e in Flet (args=XIL(0x7fffe7030e03)) at eval.c:1047
#32 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7030df3)) at eval.c:2486
#33 0x0000555555756fcd in Fprogn (body=XIL(0x7fffe702ade3)) at eval.c:439
#34 0x000055555575893e in Flet (args=XIL(0x7fffe702adc3)) at eval.c:1047
#35 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe702adb3)) at eval.c:2486
#36 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#37 0x000055555575e607 in funcall_lambda (fun=XIL(0x7fffe702ad53), nargs=1, 
    arg_vector=0x7fffffffd2b0) at eval.c:3283
#38 0x000055555575e004 in apply_lambda (fun=XIL(0x7fffe702ad43), 
    args=XIL(0x7fffe703a6a3), count=...) at eval.c:3154
#39 0x000055555575c735 in eval_sub (form=XIL(0x7fffe703a693)) at eval.c:2625
#40 0x0000555555756fcd in Fprogn (body=XIL(0x7fffe70399d3)) at eval.c:439
#41 0x000055555575893e in Flet (args=XIL(0x7fffe70397e3)) at eval.c:1047
#42 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe70397d3)) at eval.c:2486
#43 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#44 0x000055555575e607 in funcall_lambda (fun=XIL(0x7fffe7039793), nargs=0, 
    arg_vector=0x7fffffffd770) at eval.c:3283
#45 0x000055555575e004 in apply_lambda (fun=XIL(0x7fffe7039783), args=XIL(0), 
    count=...) at eval.c:3154
#46 0x000055555575c735 in eval_sub (form=XIL(0x7fffe70471f3)) at eval.c:2625
#47 0x000055555575907b in Funwind_protect (args=XIL(0x7fffe70462e3))
    at eval.c:1328
#48 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe70462d3)) at eval.c:2486
#49 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#50 0x000055555575893e in Flet (args=XIL(0x7fffe70462b3)) at eval.c:1047
#51 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe70462a3)) at eval.c:2486
#52 0x0000555555756fcd in Fprogn (body=XIL(0x7fffe7045cf3)) at eval.c:439
#53 0x0000555555756eae in Fif (args=XIL(0x7fffe7045b83)) at eval.c:395
#54 0x000055555575beb6 in eval_sub (form=XIL(0x7fffe7045b73)) at eval.c:2486
#55 0x0000555555756fcd in Fprogn (body=XIL(0)) at eval.c:439
#56 0x000055555575e607 in funcall_lambda (fun=XIL(0x7fffe7045b23), nargs=0, 
    arg_vector=0x7fffffffdf30) at eval.c:3283
#57 0x000055555575e004 in apply_lambda (fun=XIL(0x7fffe7045b13), args=XIL(0), 
    count=...) at eval.c:3154
--Type <RET> for more, q to quit, c to continue without paging--
#58 0x000055555575c735 in eval_sub (form=XIL(0x7fffe72674cb)) at eval.c:2625
#59 0x000055555575b931 in Feval (form=XIL(0x7fffe72674cb), lexical=XIL(0x30))
    at eval.c:2396
#60 0x00005555556875aa in top_level_2 () at keyboard.c:1184
#61 0x000055555575991b in internal_condition_case (
    bfun=0x555555687543 <top_level_2>, handlers=XIL(0x90), 
    hfun=0x555555686de8 <cmd_error>) at eval.c:1544
#62 0x0000555555687612 in top_level_1 (ignore=XIL(0)) at keyboard.c:1196
#63 0x0000555555758d5e in internal_catch (tag=XIL(0xe3a0), 
    func=0x5555556875c3 <top_level_1>, arg=XIL(0)) at eval.c:1224
#64 0x000055555568747e in command_loop () at keyboard.c:1145
#65 0x000055555568688a in recursive_edit_1 () at keyboard.c:754
#66 0x0000555555686ab6 in Frecursive_edit () at keyboard.c:837
#67 0x00005555556825d0 in main (argc=41, argv=0x7fffffffe528) at emacs.c:2626
(gdb) 

Re: STatus of MPS branch

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Sat, Apr 20 2024, Gerd Möllmann wrote:
>
>> There are many things other people could help with at this point. Some
>> require no knowledge of MPS, or C. For example:
>>
>> - Make it work on anything not macOS.
>
> I'm trying to build the igc branch on Debian.  But I get a SIGSEGV when
> loaddefs-gen.el gets loaded.
>
> I configured with:
>   CFLAGS='-g -O0 -I/scratch/emacs/mps-install/include -L/scratch/emacs/mps-install/lib' configure --enable-checking=yes --without-all --without-x --with-mps=debug
>
> and made some changes to igc.c as seen in the diff below; but those seem
> rather innocent.
>
> GCC still emits this warning:
>
> igc.c:229:18: warning: ‘pvec_type’ is narrower than values of its type
>   229 |   enum pvec_type pvec_type : IGC_PVEC_BITS;

Hi Helmut!

Good catch, clang didn't find that. You could increase the bit field
size by 1, and the one for the hash by 1, for example.

> Is this something to worry about?

I don't think so. I'm using the pvec_type for debugging, mainly.

>
> Helmut
>
> diff --git a/src/igc.c b/src/igc.c
> @@ -81,7 +81,7 @@ igc_assert_fail (const char *file, unsigned line, const char *msg)
>  # define igc_assert(expr)                         \
>     if (!(expr))                                   \
>       igc_assert_fail (__FILE__, __LINE__, #expr); \
> -   else
> +   /* else */
>  #else
>  # define igc_assert(expr) (void) 9
>  #endif
> @@ -122,7 +122,7 @@ is_aligned (const mps_addr_t addr)
>  #define IGC_CHECK_RES(res) \
>    if ((res) != MPS_RES_OK) \
>      emacs_abort ();        \
> -  else
> +  /* else */

Don't know, this looks a bit problemantic, but if it's the problem I
don't know. Removing the else means something like

  if (x)
    igc_assert (...);
  else
    do something else

expand to something one doesn't want. The else is swallowed by the if in
the macro...

>  #define IGC_WITH_PARKED(gc)                        \
>    for (int i = (mps_arena_park (gc->arena), 1); i; \
> @@ -801,6 +801,7 @@ scan_pure (mps_ss_t ss, void *start, void *end, void *closure)
>    MPS_SCAN_BEGIN (ss)
>    {
>      igc_assert (start == (void *) pure);
> +    extern ptrdiff_t pure_bytes_used_lisp;
>      end = (char *) pure + pure_bytes_used_lisp;
>      if (end > start)
>        IGC_FIX_CALL (ss, scan_ambig (ss, start, end, NULL));
> @@ -954,6 +955,7 @@ fix_itree_node (mps_ss_t ss, struct itree_node *n)
>  static mps_res_t
>  fix_image (mps_ss_t ss, struct image *i)
>  {
> +#ifdef HAVE_WINDOW_SYSTEM
>    MPS_SCAN_BEGIN (ss)
>    {
>      IGC_FIX12_OBJ (ss, &i->spec);
> @@ -964,6 +966,7 @@ fix_image (mps_ss_t ss, struct image *i)
>    }
>    MPS_SCAN_END (ss);
>    return MPS_RES_OK;
> +#endif
>  }

If I'm reading this one right, the function doesn't return a value.
But maybe this doesn't get compiled?

Otherwise, I don't see something suspicious.

>   INFO     Scraping files for loaddefs...80% 
>
> Program received signal SIGSEGV, Segmentation fault.
>
> Breakpoint 1, handle_sigsegv (sig=11, 
>     siginfo=0x555555f6ed70 <sigsegv_stack+64784>, 
>     arg=0x555555f6ec40 <sigsegv_stack+64480>) at sysdep.c:1930
> 1930      bool fatal = gc_in_progress;
> (gdb) backtrace 
> #0  handle_sigsegv (sig=11, siginfo=0x555555f6ed70 <sigsegv_stack+64784>, 
>     arg=0x555555f6ec40 <sigsegv_stack+64480>) at sysdep.c:1930
> #1  <signal handler called>
> #2  string_intervals (s=XIL(0x7ffff0698edc))
>     at /scratch/emacs/emacs-igc/src/lisp.h:4063
> #3  0x0000555555767d97 in concat_to_string (nargs=3, args=0x7fffe5f035b8)
>     at fns.c:957

I think I'd start in #2 here, by looking at S. First question would be
if the string itself it ok, and what kind of string it is. I haven't
used GDB with Emacs here for a very long time (no GDB on macOS), so I'm
a bit out of my comfort zone.

# We need the pointer to the Lisp_String that's in S, let's say it's P.
# Don't know how, sorry
???

# Is P in the dump?
p pdumper_object_p (P) 

# Is P potentially under MPS control?
p is_mps (P)

# If it is, what does its header look like (assuming 64 bit words)
p *(struct igc_header *) ((char *) P - 8)

And then, in general, we would need to know where that string comes
from. It could for example be that the string has been free'd by MPS
because it didn't see a reference to it. That's admittedly not entirely
easy. It memans going up the stack, and look around, maybe infering from
the Lisp backtrace what Emacs was trying to do, and such.

Re: STatus of MPS branch

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> Date: Sun, 21 Apr 2024 17:37:09 +0200
> 
> > Breakpoint 1, handle_sigsegv (sig=11, 
> >     siginfo=0x555555f6ed70 <sigsegv_stack+64784>, 
> >     arg=0x555555f6ec40 <sigsegv_stack+64480>) at sysdep.c:1930
> > 1930      bool fatal = gc_in_progress;
> > (gdb) backtrace 
> > #0  handle_sigsegv (sig=11, siginfo=0x555555f6ed70 <sigsegv_stack+64784>, 
> >     arg=0x555555f6ec40 <sigsegv_stack+64480>) at sysdep.c:1930
> > #1  <signal handler called>
> > #2  string_intervals (s=XIL(0x7ffff0698edc))
> >     at /scratch/emacs/emacs-igc/src/lisp.h:4063
> > #3  0x0000555555767d97 in concat_to_string (nargs=3, args=0x7fffe5f035b8)
> >     at fns.c:957
> 
> I think I'd start in #2 here, by looking at S. First question would be
> if the string itself it ok, and what kind of string it is. I haven't
> used GDB with Emacs here for a very long time (no GDB on macOS), so I'm
> a bit out of my comfort zone.
> 
> # We need the pointer to the Lisp_String that's in S, let's say it's P.
> # Don't know how, sorry

Like this:

  (gdb) source /path/to/emacs/src/.gdbinit
  (gdb) p s
  (gdb) xtype
  (gdb) xstring

The last command should only be issued if "xtype" before this says S
is a Lisp string.  Otherwise, S is an object of the wrong type, and
one should try figuring out how a non-string object ended up in a call
to string_intervals.

Re: STatus of MPS branch

[Helmut Eller]

On Sun, Apr 21 2024, Gerd Möllmann wrote:

[..]
>> igc.c:229:18: warning: ‘pvec_type’ is narrower than values of its type
>>   229 |   enum pvec_type pvec_type : IGC_PVEC_BITS;
>
> Good catch, clang didn't find that. You could increase the bit field
> size by 1, and the one for the hash by 1, for example.

I did this; GCC stopped complaining but otherwise it made no observable
difference.

[..]
>>  # define igc_assert(expr)                         \
>>     if (!(expr))                                   \
>>       igc_assert_fail (__FILE__, __LINE__, #expr); \
>> -   else
>> +   /* else */
[..]
> Don't know, this looks a bit problemantic, but if it's the problem I
> don't know. Removing the else means something like
>
>   if (x)
>     igc_assert (...);
>   else
>     do something else
>
> expand to something one doesn't want. The else is swallowed by the if in
> the macro...

Without some change GCC emits lots of warnings of this kind:
igc.c: In function ‘to_words’:
igc.c:246:49: warning: suggest braces around empty body in an ‘else’ statement [-Wempty-body]
  246 |   igc_assert (nbytes % sizeof (mps_word_t) == 0);
      |                                                 ^

So maybe the macro should end with "else {}".

[..]
>>  static mps_res_t
>>  fix_image (mps_ss_t ss, struct image *i)
>>  {
>> +#ifdef HAVE_WINDOW_SYSTEM
>>    MPS_SCAN_BEGIN (ss)
>>    {
>>      IGC_FIX12_OBJ (ss, &i->spec);
>> @@ -964,6 +966,7 @@ fix_image (mps_ss_t ss, struct image *i)
>>    }
>>    MPS_SCAN_END (ss);
>>    return MPS_RES_OK;
>> +#endif
>>  }
>
> If I'm reading this one right, the function doesn't return a value.
> But maybe this doesn't get compiled?

Yes. it's just to get it compiled; struct image isn't defined with my
config.

>>   INFO     Scraping files for loaddefs...80% 
>>
>> Program received signal SIGSEGV, Segmentation fault.
>>
>> Breakpoint 1, handle_sigsegv (sig=11, 
>>     siginfo=0x555555f6ed70 <sigsegv_stack+64784>, 
>>     arg=0x555555f6ec40 <sigsegv_stack+64480>) at sysdep.c:1930
>> 1930      bool fatal = gc_in_progress;
>> (gdb) backtrace 
>> #0  handle_sigsegv (sig=11, siginfo=0x555555f6ed70 <sigsegv_stack+64784>, 
>>     arg=0x555555f6ec40 <sigsegv_stack+64480>) at sysdep.c:1930
>> #1  <signal handler called>
>> #2  string_intervals (s=XIL(0x7ffff0698edc))
>>     at /scratch/emacs/emacs-igc/src/lisp.h:4063
>> #3  0x0000555555767d97 in concat_to_string (nargs=3, args=0x7fffe5f035b8)
>>     at fns.c:957
>
> I think I'd start in #2 here, by looking at S. First question would be
> if the string itself it ok, and what kind of string it is. I haven't
> used GDB with Emacs here for a very long time (no GDB on macOS), so I'm
> a bit out of my comfort zone.
>
> # We need the pointer to the Lisp_String that's in S, let's say it's P.
> # Don't know how, sorry
> ???

I changed some things and now I can no longer reproduce the exact same
problem.  But it looked to me like a perfectly normal string with
intervals=0x0.  Strangely, accessing the intervals field from GDB didn't
cause any SIGSEGV.

However, after setting breakpoints in handle_sigsegv and sigHandle (from
MPS) I discovered that sigHandle wasn't called.  So my hypothesis is
that the signal handler isn't initialized properly.  In particular, it
seems problematic that ProtSetup (from MPS) is called before
init_signals.

Then I moved the call to init_signals in emacs.c up before the call to
init_igc and voilà: the build completed.  With an apparently working
(tty-only) Emacs.

Helmut

Re: STatus of MPS branch

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Sun, Apr 21 2024, Gerd Möllmann wrote:
>
> I did this; GCC stopped complaining but otherwise it made no observable
> difference.

Thanks. I've done that now on the branch, too + a compile-time assertion
for clang.

> Without some change GCC emits lots of warnings of this kind:
> igc.c: In function ‘to_words’:
> igc.c:246:49: warning: suggest braces around empty body in an ‘else’ statement [-Wempty-body]
>   246 |   igc_assert (nbytes % sizeof (mps_word_t) == 0);
>       |                                                 ^
>
> So maybe the macro should end with "else {}".

That would also not work. Something like

  if (x)
    igc_assert (y);
  else

would then be a syntax error. It would expand to

  if (x)
    if (...)
      ...
    else
     {
     }
 ;
 else

and the last else hangs in the air.

This is a situation where the code is definitely correct, and I find GCC
a bit over-eager, or maybe it's the compiler switches Emacs uses.

You could put a

#pragma GCC diagnostic ignored "-Wempty-body"

Or put "do { ... } while (0)" around what's currently the macro body,
i.e.

#define igc_assert(e) do { \
... original macro body here.
} while (0)

I'll reply to the rest in a different mail..

Re: STatus of MPS branch

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> I changed some things and now I can no longer reproduce the exact same
> problem.  But it looked to me like a perfectly normal string with
> intervals=0x0.  Strangely, accessing the intervals field from GDB didn't
> cause any SIGSEGV.
>
> However, after setting breakpoints in handle_sigsegv and sigHandle (from
> MPS) I discovered that sigHandle wasn't called.  So my hypothesis is
> that the signal handler isn't initialized properly.  In particular, it
> seems problematic that ProtSetup (from MPS) is called before
> init_signals.
>
> Then I moved the call to init_signals in emacs.c up before the call to
> init_igc and voilà: the build completed.  With an apparently working
> (tty-only) Emacs.

Oh, I have an idea! Hopefully I can explain it. Let me try...

MPS is a copying GC, and it uses read-barriers for object forwarding.

Let's say O is an object managed by MPS, and it decides during GC that O
shall be moved in memory to the new location N. This is done by copying
O to N, storing at O the information that it has moved to N, and putting
a read-barrier on O.

The last step means that any access to O by the client (Emacs) leads to
MPS being involved, which updates the reference to O to a reference to
N.

Plus...

MacOS has a kernel that is, let's not get into details, a conglomerate
of Mach and FreeBSD kernel. This means MPS can use Mach mechanisms for
implementing read/write barriers, and I know it does because I landed in
that code in LLDB at some point.

Linux is not Mach, so MPS must use VM protection and signals.

And finally, the idea now would be that MPS's signal handling (probably
SIGSEGV and SIGBUS) and Emacs' signal handling interfere with one
another.

For example, MPS has done its thing and read-protected O from above.
Emacs accesses O, a SEGV is signaled, Emacs gets the signal and thinks
it's bit the dust. And so on.

Or, IOW, we need some Emacs signal handling support personal.

Does that make sense?

Re: STatus of MPS branch

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> Then I moved the call to init_signals in emacs.c up before the call to
> init_igc and voilà: the build completed.  With an apparently working
> (tty-only) Emacs.

:-) :-) :-)

Thanks!

Re: STatus of MPS branch

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Helmut Eller <eller.helmut@gmail.com> writes:
>
>> Then I moved the call to init_signals in emacs.c up before the call to
>> init_igc and voilà: the build completed.  With an apparently working
>> (tty-only) Emacs.
>
> :-) :-) :-)
>
> Thanks!

Forgot to say it explicitly, but if you feel like it, please feel free
to commit to the branch, of course :-). Or send me patches, or as you
like.

A good day, today :-).

Re: MPS signals and Emacs (was: STatus of MPS branch)

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> Date: Mon, 22 Apr 2024 07:28:40 +0200
> 
> Helmut Eller <eller.helmut@gmail.com> writes:
> 
> > I changed some things and now I can no longer reproduce the exact same
> > problem.  But it looked to me like a perfectly normal string with
> > intervals=0x0.  Strangely, accessing the intervals field from GDB didn't
> > cause any SIGSEGV.
> >
> > However, after setting breakpoints in handle_sigsegv and sigHandle (from
> > MPS) I discovered that sigHandle wasn't called.  So my hypothesis is
> > that the signal handler isn't initialized properly.  In particular, it
> > seems problematic that ProtSetup (from MPS) is called before
> > init_signals.
> >
> > Then I moved the call to init_signals in emacs.c up before the call to
> > init_igc and voilà: the build completed.  With an apparently working
> > (tty-only) Emacs.
> 
> Oh, I have an idea! Hopefully I can explain it. Let me try...
> 
> MPS is a copying GC, and it uses read-barriers for object forwarding.
> 
> Let's say O is an object managed by MPS, and it decides during GC that O
> shall be moved in memory to the new location N. This is done by copying
> O to N, storing at O the information that it has moved to N, and putting
> a read-barrier on O.
> 
> The last step means that any access to O by the client (Emacs) leads to
> MPS being involved, which updates the reference to O to a reference to
> N.
> 
> Plus...
> 
> MacOS has a kernel that is, let's not get into details, a conglomerate
> of Mach and FreeBSD kernel. This means MPS can use Mach mechanisms for
> implementing read/write barriers, and I know it does because I landed in
> that code in LLDB at some point.
> 
> Linux is not Mach, so MPS must use VM protection and signals.
> 
> And finally, the idea now would be that MPS's signal handling (probably
> SIGSEGV and SIGBUS) and Emacs' signal handling interfere with one
> another.
> 
> For example, MPS has done its thing and read-protected O from above.
> Emacs accesses O, a SEGV is signaled, Emacs gets the signal and thinks
> it's bit the dust. And so on.
> 
> Or, IOW, we need some Emacs signal handling support personal.

Paul, can you please help by suggesting how to handle this situation?

Re: MPS signals and Emacs

[Paul Eggert]

On 2024-04-21 23:15, Eli Zaretskii wrote:

> Paul, can you please help by suggesting how to handle this situation?

I don't know MPS or the MPS branch. However, it sounds like the MPS 
branch should indeed call init_signals (to let Emacs set up signal 
handlers the way it likes) before calling ProtSetup (to let MPS override 
Emacs's handling of SIGBUS and SIGSEGV).

Emacs's SIGBUS and SIGSEGV handlers are used only deal with serious 
internal fatal errors, which should never happen, and if they do happen 
I hope MPS will do the right thing with them by passing them to Emacs's 
thread_fatal_action. You should test that by injecting a serious 
internal fatal error (usage of a bad pointer) into Emacs and seeing that 
MPS does the right thing with it.

I assume that MPS is not being used to allocate any of Emacs's own 
objects, as copying collectors are incompatible with Emacs's habit of 
squirreling away pointers inside integers.

Re: Native compilation on MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> Keep us updated, I hope I can find some time to dedicate to the MPS
> effor the upcoming week.

Hi Andrea!

I'm currently on a different problem that's the first one occurring when
I don't do a build from scratch, but try to pdump with an existing eln.

   Loading emacs-lisp/macroexp (native compiled elisp)...
   Loading /Users/gerd/emacs/github/igc/lisp/cus-face.el (source)...
   Loading /Users/gerd/emacs/github/igc/lisp/faces.el (source)...
   Loading /Users/gerd/emacs/github/igc/lisp/ldefs-boot.el (source)...
   Loading /Users/gerd/emacs/github/igc/lisp/button.el (source)...
   Loading /Users/gerd/emacs/github/igc/lisp/emacs-lisp/cl-preloaded.el (source)...

   ./character.h:381: Emacs fatal error: assertion failed: 0xC0 <= c
   Fatal error 6: Aborted

That's in string_char_and_len, and the reason is that the string is
apparently kaputt. The actual problem seems to appear much earlier and
2km away. A symbol is created with name

(struct Lisp_String) {
  u = {
    s = {
      size = 19
      size_byte = 19
      intervals = NULL
      data = 0x000000017961c6b0 "macroexp--backtrace"
    }
    next = 0x0000000000000013
    gcaligned = '\x13'
  }
}

And that name/string isn't seen by MPS. Which should mean the symbol
isn't seen, which should mean it's stored somewhere where I don't look.

Since macroexp is dumped native-compiled, I thought I'd rather ask the
expert, where that might be, than digging around in pdumper, or where
else.

Re: MPS signals and Emacs

[Gerd Möllmann]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 2024-04-21 23:15, Eli Zaretskii wrote:
>
>> Paul, can you please help by suggesting how to handle this situation?
>
> I don't know MPS or the MPS branch. However, it sounds like the MPS
> branch should indeed call init_signals (to let Emacs set up signal
> handlers the way it likes) before calling ProtSetup (to let MPS
> override Emacs's handling of SIGBUS and SIGSEGV).
>
> Emacs's SIGBUS and SIGSEGV handlers are used only deal with serious
> internal fatal errors, which should never happen, and if they do
> happen I hope MPS will do the right thing with them by passing them to
> Emacs's thread_fatal_action. You should test that by injecting a
> serious internal fatal error (usage of a bad pointer) into Emacs and
> seeing that MPS does the right thing with it.

Thanks!

> I assume that MPS is not being used to allocate any of Emacs's own
> objects, as copying collectors are incompatible with Emacs's habit of
> squirreling away pointers inside integers.

It's excactly that, a (mostly) copying collector for Emacs :-). Also
other stuff, like concurrent, incremental, generational, thread-safe.

The branch is young (8 weeks), but it's looking good so far, I think.

Re: MPS signals and Emacs

[Paul Eggert]

On 2024-04-22 00:19, Gerd Möllmann wrote:
>> I assume that MPS is not being used to allocate any of Emacs's own
>> objects, as copying collectors are incompatible with Emacs's habit of
>> squirreling away pointers inside integers.
> It's excactly that, a (mostly) copying collector for Emacs :-).

Since Emacs uses a conservative collector, how can MPS move Emacs 
objects? MPS can't know which words are pointers and which are not, so 
it can't update pointers when Emacs objects move.

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 09:19:22 +0200
> 
> Paul Eggert <eggert@cs.ucla.edu> writes:
> 
> > I assume that MPS is not being used to allocate any of Emacs's own
> > objects, as copying collectors are incompatible with Emacs's habit of
> > squirreling away pointers inside integers.
> 
> It's excactly that, a (mostly) copying collector for Emacs :-).

Does this mean it's incompatible with Emacs, for the reasons explained
by Paul above?  Or does it mean MPS is not used to allocate any of
Emacs's own objects (and if so, what is used to allocate those)?

Re: MPS signals and Emacs

[Gerd Möllmann]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 2024-04-22 00:19, Gerd Möllmann wrote:
>>> I assume that MPS is not being used to allocate any of Emacs's own
>>> objects, as copying collectors are incompatible with Emacs's habit of
>>> squirreling away pointers inside integers.
>> It's excactly that, a (mostly) copying collector for Emacs :-).
>
> Since Emacs uses a conservative collector, how can MPS move Emacs
> objects? MPS can't know which words are pointers and which are not, so
> it can't update pointers when Emacs objects move.

There are ambigupus and there are exact referenes. An object references
from at least one ambiguos reference cannot be moved, others can.

Where exact and where ambigous references are (or areas of these), one
has to tell MPS, of course. It's not like Boehm's collector where
everything is ambiguous.

Re: MPS signals and Emacs

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> It's excactly that, a (mostly) copying collector for Emacs :-).
>
> Does this mean it's incompatible with Emacs, for the reasons explained
> by Paul above?  Or does it mean MPS is not used to allocate any of
> Emacs's own objects (and if so, what is used to allocate those)?

Neither. I think Paul mistakes it with something.

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 09:49:29 +0200
> 
> Paul Eggert <eggert@cs.ucla.edu> writes:
> 
> > On 2024-04-22 00:19, Gerd Möllmann wrote:
> >>> I assume that MPS is not being used to allocate any of Emacs's own
> >>> objects, as copying collectors are incompatible with Emacs's habit of
> >>> squirreling away pointers inside integers.
> >> It's excactly that, a (mostly) copying collector for Emacs :-).
> >
> > Since Emacs uses a conservative collector, how can MPS move Emacs
> > objects? MPS can't know which words are pointers and which are not, so
> > it can't update pointers when Emacs objects move.
> 
> There are ambigupus and there are exact referenes. An object references
> from at least one ambiguos reference cannot be moved, others can.
> 
> Where exact and where ambigous references are (or areas of these), one
> has to tell MPS, of course. It's not like Boehm's collector where
> everything is ambiguous.

What do "exact reference" and "ambiguous reference" mean, in Emacs
terms?

Re: MPS signals and Emacs

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> There are ambigupus and there are exact referenes. An object references
>> from at least one ambiguos reference cannot be moved, others can.
>> 
>> Where exact and where ambigous references are (or areas of these), one
>> has to tell MPS, of course. It's not like Boehm's collector where
>> everything is ambiguous.
>
> What do "exact reference" and "ambiguous reference" mean, in Emacs
> terms?

Ambiguour references are ones that could be a reference or a could as
well be a random bit pattern.

An example in Emacs is the control stack marking. I haven't looked at it
since I added that, so I don't know if that's still the same, but back
then it worked like this: Take a word W from the stack. See if W is
something that looks like a pointer into Lisp memeory (the red-black
tree, mem_xxx). See if W is a valid Lisp object start if that's the
case. If that's also the case, mark W as live. That means W might be
marked live, in principle, even if it isn't really.

Exact references are all references that aren't ambiguous. For example,
the name of a symbol had better be always a valid Lisp_Object.

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: eggert@cs.ucla.edu,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 10:27:42 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What do "exact reference" and "ambiguous reference" mean, in Emacs
> > terms?
> 
> Ambiguour references are ones that could be a reference or a could as
> well be a random bit pattern.
> 
> An example in Emacs is the control stack marking.

Is that just one example, or is that the _only_ example?  If the
former, what other examples of ambiguous references are there in
Emacs?

Re: MPS signals and Emacs

[Mattias Engdegård]

22 apr. 2024 kl. 10.55 skrev Eli Zaretskii <eliz@gnu.org>:

> Is that just one example, or is that the _only_ example?  If the
> former, what other examples of ambiguous references are there in
> Emacs?

We currently treat part of the bytecode stack as conservative GC roots as well. This isn't really required but simplifies the execution engine. (There are plans to change this.)

Re: MPS signals and Emacs

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> An example in Emacs is the control stack marking.
>
> Is that just one example, or is that the _only_ example?  If the
> former, what other examples of ambiguous references are there in
> Emacs?

I think I've also seen at least one other example, but I forget where
that was (maybe specpdl, or bytecode?). I think the function called
is mark_mem or so.

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: eggert@cs.ucla.edu,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 11:02:56 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> An example in Emacs is the control stack marking.
> >
> > Is that just one example, or is that the _only_ example?  If the
> > former, what other examples of ambiguous references are there in
> > Emacs?
> 
> I think I've also seen at least one other example, but I forget where
> that was (maybe specpdl, or bytecode?). I think the function called
> is mark_mem or so.

So any Lisp object referenced from the C stack or from byte-code will
not be copied by MPS?  What are the implications of that for Emacs?
AFAIU, the current GC considers any such object as live, marks it, and
doesn't collect it as garbage during the sweep step, but being unable
to copy such an object means something very different for a copying
GC, doesn't it?  Because AFAIU live objects _are_ copied.

Re: MPS signals and Emacs

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> I think I've also seen at least one other example, but I forget where
>> that was (maybe specpdl, or bytecode?). I think the function called
>> is mark_mem or so.
>
> So any Lisp object referenced from the C stack or from byte-code will
> not be copied by MPS?  

Yes.

> What are the implications of that for Emacs? AFAIU, the current GC
> considers any such object as live, marks it, and doesn't collect it as
> garbage during the sweep step, but being unable to copy such an object
> means something very different for a copying GC, doesn't it? Because
> AFAIU live objects _are_ copied.

No, it's a bit different. Being able to copy and objects being live are
orthogonal. At the heart of the mostly-copying idea is simply that
ambigupusly references aren't moved in memory (so they aren't copied).
That doesn't mean these objects aren't live. They are.

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: eggert@cs.ucla.edu,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 11:29:30 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I think I've also seen at least one other example, but I forget where
> >> that was (maybe specpdl, or bytecode?). I think the function called
> >> is mark_mem or so.
> >
> > So any Lisp object referenced from the C stack or from byte-code will
> > not be copied by MPS?  
> 
> Yes.
> 
> > What are the implications of that for Emacs? AFAIU, the current GC
> > considers any such object as live, marks it, and doesn't collect it as
> > garbage during the sweep step, but being unable to copy such an object
> > means something very different for a copying GC, doesn't it? Because
> > AFAIU live objects _are_ copied.
> 
> No, it's a bit different. Being able to copy and objects being live are
> orthogonal. At the heart of the mostly-copying idea is simply that
> ambigupusly references aren't moved in memory (so they aren't copied).
> That doesn't mean these objects aren't live. They are.

Doesn't copying GC work by copying live objects to a new memory
region, and then freeing the old region?  If so, if some objects
cannot be moved, how can the old region be freed?

Re: MPS signals and Emacs

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Doesn't copying GC work by copying live objects to a new memory
> region, and then freeing the old region?  If so, if some objects
> cannot be moved, how can the old region be freed?

Yes. A simple precise GC (no support for ambiguous references) might
work with 2 regions, which are usually called to-space and froms-space.
When a GC happens, everything live is copied from from-space to
to-space. At the end, to-space has all live objects, and both spaces can
switch roles in the next GC.

That doesn't work with ambigupus references, of course.

Instead imagine a memory area that is divided into a number of
subregions, let's say into VM pages, I think that's a good example. Add
to that some administrative data structures holding information about
pages, say if it's part of to-space or from-space. Just as an example,
there are so many variations, of this...

Now a GC happens. We can copy objects from a pages in from-space to a
page in to-soace, and everything works as in the copying collector, in
principle. At the end, we just switch what we consider to-space and
from-space. Maybe a 1 in the page info data structure formerly stood for
to-space and 0 for to-space. Now exchange the meaning of 0 and 1.

Ok, I hope. Now say we have an ambiguous reference to an object O in
page P. We can copy the objects as above, but not O, which we leave
where it is. At the end of the GC, we move P to to-space by simply
changing its number in the page info data. Mission accomplished.

Hm, I have to add that this is such a drastically oversimplifying
horrible description that I slmost get a bad conscience. Almost ;-).

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: eggert@cs.ucla.edu,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 12:22:25 +0200
> 
> Now a GC happens. We can copy objects from a pages in from-space to a
> page in to-soace, and everything works as in the copying collector, in
> principle. At the end, we just switch what we consider to-space and
> from-space. Maybe a 1 in the page info data structure formerly stood for
> to-space and 0 for to-space. Now exchange the meaning of 0 and 1.
> 
> Ok, I hope. Now say we have an ambiguous reference to an object O in
> page P. We can copy the objects as above, but not O, which we leave
> where it is. At the end of the GC, we move P to to-space by simply
> changing its number in the page info data. Mission accomplished.

I don't understand this part:

  At the end of the GC, we move P to to-space by simply changing its
  number in the page info data.

What is "its number" here?  Since all the live objects that could be
copied were copied from P to to-space, we now have a single object O
in P (which is in from-space) and all the rest in to-space.  Now comes
that magic sentence where I got lost, and somehow "moves P to
to-space".  How does it do that?  And why move P to to-space in the
first place, when all of its objects except O are already copied to
to-space?

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: eggert@cs.ucla.edu,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 12:22:25 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Doesn't copying GC work by copying live objects to a new memory
> > region, and then freeing the old region?  If so, if some objects
> > cannot be moved, how can the old region be freed?
> 
> Yes. A simple precise GC (no support for ambiguous references) might
> work with 2 regions, which are usually called to-space and froms-space.
> When a GC happens, everything live is copied from from-space to
> to-space. At the end, to-space has all live objects, and both spaces can
> switch roles in the next GC.

Btw, does this mean that a copying GC will make the memory used for
live objects roughly doubled in size?

Re: MPS signals and Emacs

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't understand this part:
>
>   At the end of the GC, we move P to to-space by simply changing its
>   number in the page info data.
>
> What is "its number" here?  

Assume the page info data structure looks like this

struct info
{
  int number;
} infos[...];

somehow indexed by VM page. And say we have

int to_space, from_space;

then info[x].number == to_space would mean page X is in to-space, and
analogously for from-space.

> Since all the live objects that could be copied were copied from P to
> to-space, we now have a single object O in P (which is in from-space)
> and all the rest in to-space. 

Right. The important part is that P's entry in the info array above would
be changed to to_space. By that P gets part of to-space.

> Now comes that magic sentence where I got lost, and somehow "moves P
> to to-space". How does it do that? 

Ah, okay. It's what I wrote above. (I said it was horrible ;-)).

> And why move P to to-space in the first place, when all of its objects
> except O are already copied to to-space?

Not sure if I understand... Because of O? O is live, and if P would
remain in from-space, O's memory could be overwritten by the next GC,
where P would be considered part of to-space if we don't prevent it.

Re: MPS signals and Emacs

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: eggert@cs.ucla.edu,  eller.helmut@gmail.com,  emacs-devel@gnu.org
>> Date: Mon, 22 Apr 2024 12:22:25 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Doesn't copying GC work by copying live objects to a new memory
>> > region, and then freeing the old region?  If so, if some objects
>> > cannot be moved, how can the old region be freed?
>> 
>> Yes. A simple precise GC (no support for ambiguous references) might
>> work with 2 regions, which are usually called to-space and froms-space.
>> When a GC happens, everything live is copied from from-space to
>> to-space. At the end, to-space has all live objects, and both spaces can
>> switch roles in the next GC.
>
> Btw, does this mean that a copying GC will make the memory used for
> live objects roughly doubled in size?

Yes, exactly. It's a well-known problem.

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: eggert@cs.ucla.edu,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 13:00:37 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And why move P to to-space in the first place, when all of its objects
> > except O are already copied to to-space?
> 
> Not sure if I understand... Because of O? O is live, and if P would
> remain in from-space, O's memory could be overwritten by the next GC,
> where P would be considered part of to-space if we don't prevent it.

So we will now have in to-space 2 copies of all the objects that were
in P and could be copied to to-space?  IOW, instead of a single page
P, we now have 2 pages: one that has all the objects in P except O,
and another with O?

Re: MPS signals and Emacs

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> Not sure if I understand... Because of O? O is live, and if P would
>> remain in from-space, O's memory could be overwritten by the next GC,
>> where P would be considered part of to-space if we don't prevent it.
>
> So we will now have in to-space 2 copies of all the objects that were
> in P and could be copied to to-space?  IOW, instead of a single page
> P, we now have 2 pages: one that has all the objects in P except O,
> and another with O?

Yes!

Maybe with the remark that everything in P except O is actually not in
use.

Re: MPS signals and Emacs

[Helmut Eller]

On Sun, Apr 21 2024, Paul Eggert wrote:

[..]
> Emacs's SIGBUS and SIGSEGV handlers are used only deal with serious
> internal fatal errors, which should never happen, and if they do
> happen I hope MPS will do the right thing with them by passing them to
> Emacs's thread_fatal_action. You should test that by injecting a
> serious internal fatal error (usage of a bad pointer) into Emacs and
> seeing that MPS does the right thing with it.

The code and the comment around line 100 in protsigx.c[*] suggests that
MPS is trying to do the right thing by invoking the previously installed
handler with:

  kill(getpid(), PROT_SIGNAL);

Where PROT_SIGNAL=SIGSEGV.  As the comment there explains, this does not
pass along the siginfo_t.  That of course means, that the previously
installed handler can't do much other than aborting (si_code will be 0).

I verified that this works, i.e. handle_sigsegv is actually called after
sigHandle when accessing address 0x4.

Whether this makes any sense in a multi-threaded setup; well, I'm not so
sure.

For the moment it's probably good enough but it will likely make
debugging bugs that aren't GC-related a bit harder.  In GDB it's already
necessary to say something like

  handle SIGSEGV nostop pass print

so that the process doesn't stop every time a memory barrier is hit.

I was also puzzled by the fact that during pdump, sigHandle worked fine
but later during loaddefs-gen not.  I then found this comment in
init_signals:

  /* Don't alter signal handlers if dumping.  On some machines,
     changing signal handlers sets static data that would make signals
     fail to work right when the dumped Emacs is run.  */
  if (will_dump_p ())
    return;

That explains the puzzling behavior but it makes me wonder what kind of
machine that could possibly be.  Also MPS clearly has to install memory
barriers, will_dump_p or not.  Will that affect those machines?

Helmut

[*] https://github.com/Ravenbrook/mps/blob/master/code/protsgix.c

Re: MPS signals and Emacs

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Gerd Möllmann
>  <gerd.moellmann@gmail.com>,
>   emacs-devel@gnu.org
> Date: Mon, 22 Apr 2024 16:10:00 +0200
> 
> I was also puzzled by the fact that during pdump, sigHandle worked fine
> but later during loaddefs-gen not.  I then found this comment in
> init_signals:
> 
>   /* Don't alter signal handlers if dumping.  On some machines,
>      changing signal handlers sets static data that would make signals
>      fail to work right when the dumped Emacs is run.  */
>   if (will_dump_p ())
>     return;
> 
> That explains the puzzling behavior but it makes me wonder what kind of
> machine that could possibly be.  Also MPS clearly has to install memory
> barriers, will_dump_p or not.  Will that affect those machines?

I think we only need to avoid installing signal handlers when dumping
using unexec, because with pdumper we don't dump the data of signal
handlers.  Paul, am I right?

If I'm right, the will_dump_p call should be replaced with
will_dump_with_unexec_p (which is not relevant to the MPS branch
anyway).

Re: STatus of MPS branch

[Helmut Eller]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

On Mon, Apr 22 2024, Gerd Möllmann wrote:

> Forgot to say it explicitly, but if you feel like it, please feel free
> to commit to the branch, of course :-). Or send me patches, or as you
> like.

Below are is the (rather small) patch that I used for compiling the
branch.

Helmut


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Small-changes-need-for-GCC-Linux.patch --]
[-- Type: text/x-diff, Size: 4080 bytes --]

From 85a40f3def8dbe3d549652c0b6d47fee19b07785 Mon Sep 17 00:00:00 2001
From: Helmut Eller <eller.helmut@gmail.com>
Date: Mon, 22 Apr 2024 16:59:28 +0200
Subject: [PATCH] Small changes need for GCC/Linux.

* emacs.c (main): Call init_signals before init_igc; this avoids
overwriting MPS's SIGSEGV handler.
* fns.c: Remove spurios #include <sys/_types/_size_t.h>
* igc.c: Minor changes to avoid GCC warnings.
(igc_assert_fail): die() is already declared in lisp.h
(igc_assert, IGC_CHECK_RES): Use the do { ... } while(0) idiom to
avoid GCC warnings about empty statements.
(scan_pure): Tell GCC that pure_bytes_used_lisp exists.
(thread_ap): Abort for unhandled cases.
(fix_image, igc_make_image): Add #ifdef HAVE_WINDOW_SYSTEM in
places that need struct image.
---
 src/emacs.c |  4 ++--
 src/fns.c   |  1 -
 src/igc.c   | 31 ++++++++++++++++++++++---------
 3 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/src/emacs.c b/src/emacs.c
index 795d43e4d67..885708750da 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1425,6 +1425,8 @@ main (int argc, char **argv)
   ns_init_pool ();
 #endif
 
+  init_signals ();
+
 #ifdef HAVE_MPS
   init_igc ();
 #endif
@@ -1946,8 +1948,6 @@ main (int argc, char **argv)
     malloc_enable_thread ();
 #endif
 
-  init_signals ();
-
   noninteractive1 = noninteractive;
 
   /* Perform basic initializations (not merely interning symbols).  */
diff --git a/src/fns.c b/src/fns.c
index 634f67e15ac..65994f45dcc 100644
--- a/src/fns.c
+++ b/src/fns.c
@@ -21,7 +21,6 @@ Copyright (C) 1985-2024 Free Software Foundation, Inc.
 
 #include <stddef.h>
 #include <stdlib.h>
-#include <sys/_types/_size_t.h>
 #include <sys/random.h>
 #include <unistd.h>
 #include <filevercmp.h>
diff --git a/src/igc.c b/src/igc.c
index ceddd66abdd..680f1a5ccb6 100644
--- a/src/igc.c
+++ b/src/igc.c
@@ -73,15 +73,19 @@
 static void
 igc_assert_fail (const char *file, unsigned line, const char *msg)
 {
-  extern void die (const char *, const char *, int);
   die (msg, file, line);
 }
 
 #ifdef IGC_DEBUG
-# define igc_assert(expr)                         \
-   if (!(expr))                                   \
-     igc_assert_fail (__FILE__, __LINE__, #expr); \
-   else
+
+#define igc_assert(expr)				\
+  do							\
+    {							\
+      if (!(expr))					\
+	igc_assert_fail (__FILE__, __LINE__, #expr);	\
+    }							\
+  while (0)						\
+
 #else
 # define igc_assert(expr) (void) 9
 #endif
@@ -119,10 +123,13 @@ is_aligned (const mps_addr_t addr)
   return ((mps_word_t) addr & IGC_TAG_MASK) == 0;
 }
 
-#define IGC_CHECK_RES(res) \
-  if ((res) != MPS_RES_OK) \
-    emacs_abort ();        \
-  else
+#define IGC_CHECK_RES(res)			\
+  do						\
+    {						\
+      if ((res) != MPS_RES_OK)			\
+	emacs_abort ();				\
+    }						\
+  while (0)					\
 
 #define IGC_WITH_PARKED(gc)                        \
   for (int i = (mps_arena_park (gc->arena), 1); i; \
@@ -802,6 +809,7 @@ scan_pure (mps_ss_t ss, void *start, void *end, void *closure)
   MPS_SCAN_BEGIN (ss)
   {
     igc_assert (start == (void *) pure);
+    extern ptrdiff_t pure_bytes_used_lisp;
     end = (char *) pure + pure_bytes_used_lisp;
     if (end > start)
       IGC_FIX_CALL (ss, scan_ambig (ss, start, end, NULL));
@@ -957,11 +965,13 @@ fix_image (mps_ss_t ss, struct image *i)
 {
   MPS_SCAN_BEGIN (ss)
   {
+#ifdef HAVE_WINDOW_SYSTEM
     IGC_FIX12_OBJ (ss, &i->spec);
     IGC_FIX12_OBJ (ss, &i->dependencies);
     IGC_FIX12_OBJ (ss, &i->lisp_data);
     IGC_FIX12_RAW (ss, &i->next);
     IGC_FIX12_RAW (ss, &i->prev);
+#endif
   }
   MPS_SCAN_END (ss);
   return MPS_RES_OK;
@@ -2343,6 +2353,7 @@ thread_ap (enum igc_obj_type type)
     case IGC_OBJ_FLOAT:
       return t->d.leaf_ap;
     }
+  emacs_abort ();
 }
 
 /* Conditional breakpoints can be so slow that it is often more
@@ -2573,12 +2584,14 @@ igc_make_itree_node (void)
   return n;
 }
 
+#ifdef HAVE_WINDOW_SYSTEM
 struct image *
 igc_make_image (void)
 {
   struct image *img = alloc (sizeof *img, IGC_OBJ_IMAGE, PVEC_FREE);
   return img;
 }
+#endif
 
 struct face *
 igc_make_face (void)
-- 
2.39.2

Re: STatus of MPS branch

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> Below are is the (rather small) patch that I used for compiling the
> branch.

Thanks Helmut, it's pushed.

Re: MPS signals and Emacs

[Paul Eggert]

On 2024-04-22 00:49, Gerd Möllmann wrote:
> There are ambigupus and there are exact referenes. An object references
> from at least one ambiguos reference cannot be moved, others can.

It's not merely a question of ambiguous vs exact references. It's also a 
question of hashing.

For example, the Emacs Lisp internals hash references with low-level 
functions like sxhash_eq's call to XHASH, to implement functions like 
make-hash-table. If exact references are moved, hashing them won't work 
because their XHASH values will change. Does the MPS branch use exact 
references for any Lisp object visible to the Elisp programmer? If so, 
won't there be subtle bugs with make-hash-table and the like? Or does 
the MPS branch work around this issue by rebuilding every hash table 
after a GC?

Also, does the MPS branch have similar workarounds for every other place 
the C code uses XHASH, XLI, XPNTR, etc.? For example, suppose GC occurs 
during sort_overlays, so that compare_overlays (which uses XLI) is no 
longer a total ordering, which means qsort has undefined behavior?

Re: MPS signals and Emacs

[Gerd Möllmann]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 2024-04-22 00:49, Gerd Möllmann wrote:
>> There are ambigupus and there are exact referenes. An object references
>> from at least one ambiguos reference cannot be moved, others can.
>
> It's not merely a question of ambiguous vs exact references. It's also
> a question of hashing.

There are several possibilites. Something MPS supports is described here:

  https://memory-pool-system.readthedocs.io/en/latest/topic/location.html

In the branch, I've used an address-independent hash for now, to keep
things simple and make fast progress.

(Don't know if you read that in some thread on emacs-devel, perhaps it's
useful as background info: The whole effort started as an experiment if
MPS could potentially be used in Emacs. It took me 8 weeks so far, which
includes learning MPS.

As of now, it can build Emacs w/o native compilation from scratch, and
tests succeed except 3. One of which was caused by a change in
coreutils, for which Michael Albinus added a fix to master. The second
was a test that expected a certain behavior of finalizers, which I
didn't give priority now. The 3rd test is one of Eglot's which I don't
understand, and don't have the nerves to investigate.

Initially it was macOS only, but Helmut showed that it works on Debian,
too.

Not to bad :-). But naturally, a lot of things will still be needed to
be done.)

> For example, the Emacs Lisp internals hash references with low-level
> functions like sxhash_eq's call to XHASH, to implement functions like
> make-hash-table. If exact references are moved, hashing them won't
> work because their XHASH values will change. Does the MPS branch use
> exact references for any Lisp object visible to the Elisp programmer?
> If so, won't there be subtle bugs with make-hash-table and the like?
> Or does the MPS branch work around this issue by rebuilding every hash
> table after a GC?

The problem with eq hash tables is pretty well known in CL/Scheme
implementations. CMUCL, for example, keep track which keys move during
GC, and get/puthash rehash these. What MPS came up with is described
under the link above.

> Also, does the MPS branch have similar workarounds for every other
> place the C code uses XHASH, XLI, XPNTR, etc.? For example, suppose GC
> occurs during sort_overlays, so that compare_overlays (which uses XLI)
> is no longer a total ordering, which means qsort has undefined
> behavior?

Of course, places depending on addresses not changing will need to be
delt with.

What the problem with XPMTR is I didn't understand. In the branch, I'm
using both pointers and tagged values as references, if you meant that.

Re: Native compilation on MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Keep us updated, I hope I can find some time to dedicate to the MPS
>> effor the upcoming week.
>
> Hi Andrea!
>
> I'm currently on a different problem that's the first one occurring when
> I don't do a build from scratch, but try to pdump with an existing eln.
>
>    Loading emacs-lisp/macroexp (native compiled elisp)...
>    Loading /Users/gerd/emacs/github/igc/lisp/cus-face.el (source)...
>    Loading /Users/gerd/emacs/github/igc/lisp/faces.el (source)...
>    Loading /Users/gerd/emacs/github/igc/lisp/ldefs-boot.el (source)...
>    Loading /Users/gerd/emacs/github/igc/lisp/button.el (source)...
>    Loading /Users/gerd/emacs/github/igc/lisp/emacs-lisp/cl-preloaded.el (source)...
>
>    ./character.h:381: Emacs fatal error: assertion failed: 0xC0 <= c
>    Fatal error 6: Aborted
>
> That's in string_char_and_len, and the reason is that the string is
> apparently kaputt. The actual problem seems to appear much earlier and
> 2km away. A symbol is created with name
>
> (struct Lisp_String) {
>   u = {
>     s = {
>       size = 19
>       size_byte = 19
>       intervals = NULL
>       data = 0x000000017961c6b0 "macroexp--backtrace"
>     }
>     next = 0x0000000000000013
>     gcaligned = '\x13'
>   }
> }
>
> And that name/string isn't seen by MPS. Which should mean the symbol
> isn't seen, which should mean it's stored somewhere where I don't look.
>
> Since macroexp is dumped native-compiled, I thought I'd rather ask the
> expert, where that might be, than digging around in pdumper, or where
> else.

Hi Gerd!

I'm not sure the pdumper plays a role here, IIUC the string is the name
of a native compiled function that was recently loaded.

AFAIR the names of functions being registered are keep in
'data_ephemeral_vec' referenced inside 'load_comp_unit'.  The comment in
comp.c says:

   /* Note: data_ephemeral_vec is not GC protected except than by
      this function frame.  After this functions will be
      deactivated GC will be free to collect it, but it MUST
      survive till 'top_level_run' has finished his job.  We store
      into the ephemeral allocation class only objects that we know
      are necessary exclusively during the first load.  Once these
      are collected we don't have to maintain them in the heap
      forever.  */

Is MPS scanning the stack for Lisp like objects like our current GC?

Also other immediate objects kept in the eln are reachable by the GC
through the compilation unit object (see Lisp_Native_Comp_Unit).

I guess this string belongs to one of the two cases.

Hope it helps

  Andrea

Re: MPS signals and Emacs

[Paul Eggert]

On 4/22/24 07:42, Eli Zaretskii wrote:
> I think we only need to avoid installing signal handlers when dumping
> using unexec, because with pdumper we don't dump the data of signal
> handlers.  Paul, am I right?
> 
> If I'm right, the will_dump_p call should be replaced with
> will_dump_with_unexec_p (which is not relevant to the MPS branch
> anyway).

I think you're right here (though I haven't read the code to double check).

Re: MPS signals and Emacs

[Paul Eggert]

On 4/22/24 13:55, Gerd Möllmann wrote:
> What the problem with XPMTR is I didn't understand. In the branch, I'm
> using both pointers and tagged values as references, if you meant that.

Oh, with XPNTR I was merely referring to the low-level primitives that 
are like XHASH in that they look directly at the underlying pointer. I 
have not looked at the MPS branch. If you've already 
reviewed/altered/replaced all the uses of XPNTR then you should be OK.

Re: Native compilation on MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> I'm not sure the pdumper plays a role here, IIUC the string is the name
> of a native compiled function that was recently loaded.

Right.

> AFAIR the names of functions being registered are keep in
> 'data_ephemeral_vec' referenced inside 'load_comp_unit'.  The comment in
> comp.c says:
>
>    /* Note: data_ephemeral_vec is not GC protected except than by
>       this function frame.  After this functions will be
>       deactivated GC will be free to collect it, but it MUST
>       survive till 'top_level_run' has finished his job.  We store
>       into the ephemeral allocation class only objects that we know
>       are necessary exclusively during the first load.  Once these
>       are collected we don't have to maintain them in the heap
>       forever.  */
>
> Is MPS scanning the stack for Lisp like objects like our current GC?

Yes, it is.

> Also other immediate objects kept in the eln are reachable by the GC
> through the compilation unit object (see Lisp_Native_Comp_Unit).
> some
> I guess this string belongs to one of the two cases.

Thanks Andrea! Maybe I have overlooked that.

(Could take a bit because I get a new MBPro soon, and then I have to
play with it first, of course ;-). And I have to make transfering stuff
from/to the branch in my fork easier somehow.)

Re: MPS signals and Emacs

[Gerd Möllmann]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 4/22/24 13:55, Gerd Möllmann wrote:
>> What the problem with XPMTR is I didn't understand. In the branch, I'm
>> using both pointers and tagged values as references, if you meant that.
>
> Oh, with XPNTR I was merely referring to the low-level primitives that
> are like XHASH in that they look directly at the underlying pointer. I
> have not looked at the MPS branch. 

Ah, ok, I understand. Thanks!

> If you've already reviewed/altered/replaced all the uses of XPNTR then
> you should be OK.

One thing I can promise is that I won't review all occurrences of
anything ever ;-).

Re: MPS signals and Emacs

[Gerd Möllmann]

dick <dick.r.chiang@gmail.com> writes:

> GM> Initially it was macOS only, but Helmut showed that it works on
> GM> Debian, too.
>
> The name "Helmut" sounds like a guy who smokes the good herb.  It's a
> non-starter if MPS doesn't jive under x11 (which turns on
> HAVE_TEXT_CONVERSION).

Thanks. Can't help with that though, I'm afraid, I'm not a Linux user,
never have been, privately. Feel like sending patches?

Re: MPS signals and Emacs

[Helmut Eller]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

On Tue, Apr 23 2024, Gerd Möllmann wrote:

> dick <dick.r.chiang@gmail.com> writes:
>
>> GM> Initially it was macOS only, but Helmut showed that it works on
>> GM> Debian, too.
>>
>> The name "Helmut" sounds like a guy who smokes the good herb.  It's a
>> non-starter if MPS doesn't jive under x11 (which turns on
>> HAVE_TEXT_CONVERSION).
>
> Thanks. Can't help with that though, I'm afraid, I'm not a Linux user,
> never have been, privately. Feel like sending patches?

The patch below would allow compilation with X.  It starts and I don't
see any obvious breakage.  Maybe good enough to keep the kids quiet.

Helmut


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-compilation-with-X.patch --]
[-- Type: text/x-diff, Size: 2548 bytes --]

From 0c5683728fffef2e3a74c72f1ff9e7964b35e6ec Mon Sep 17 00:00:00 2001
From: Helmut Eller <eller.helmut@gmail.com>
Date: Tue, 23 Apr 2024 08:23:04 +0200
Subject: [PATCH] Allow compilation with X.

* src/igc: Turn the error about text conversion into a warning.
  (Figc_roots): Initialize the result.

* src/gtkutil.c (xg_mark_data)
* src/xselect.c (mark_xselect)
* src/xterm.c (mark_xterm): Don't define marking functions when
  compiling with MPS.
---
 src/gtkutil.c | 2 ++
 src/igc.c     | 5 +++--
 src/xselect.c | 2 ++
 src/xterm.c   | 2 ++
 4 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/gtkutil.c b/src/gtkutil.c
index c067f7b53ac..a01b8a41d5d 100644
--- a/src/gtkutil.c
+++ b/src/gtkutil.c
@@ -3025,6 +3025,7 @@ unref_cl_data (xg_menu_cb_data *cl_data)
 
 /* Function that marks all lisp data during GC.  */
 
+#ifndef HAVE_MPS
 void
 xg_mark_data (void)
 {
@@ -3069,6 +3070,7 @@ xg_mark_data (void)
     }
 #endif
 }
+#endif
 
 /* Callback called when a menu item is destroyed.  Used to free data.
    W is the widget that is being destroyed (not used).
diff --git a/src/igc.c b/src/igc.c
index 680f1a5ccb6..8c250eb57e6 100644
--- a/src/igc.c
+++ b/src/igc.c
@@ -58,7 +58,8 @@
 # error "HAVE_PDUMPER required"
 #endif
 #ifdef HAVE_TEXT_CONVERSION
-# error "HAVE_TEXT_CONVERSION not supported"
+//# error "HAVE_TEXT_CONVERSION not supported"
+# warning "HAVE_TEXT_CONVERSION not supported"
 #endif
 
 /* Note: Emacs will call allocation functions whlle aborting. This leads
@@ -2647,7 +2648,7 @@ DEFUN ("igc-roots", Figc_roots, Sigc_roots, 0, 0, 0, doc : /* */)
 (void)
 {
   struct igc *gc = global_igc;
-  Lisp_Object roots;
+  Lisp_Object roots = Qnil;
 
   for (igc_root_list *r = gc->roots; r; r = r->next)
     {
diff --git a/src/xselect.c b/src/xselect.c
index fd0f06eeed9..b93c2423f0e 100644
--- a/src/xselect.c
+++ b/src/xselect.c
@@ -3488,6 +3488,7 @@ syms_of_xselect_for_pdumper (void)
   property_change_reply = Fcons (Qnil, Qnil);
 }
 
+#ifndef HAVE_MPS
 void
 mark_xselect (void)
 {
@@ -3518,3 +3519,4 @@ mark_xselect (void)
 	mark_object (cs->string);
     }
 }
+#endif
diff --git a/src/xterm.c b/src/xterm.c
index e08ffd15b18..36f83a33244 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -32212,6 +32212,7 @@ init_xterm (void)
 #endif
 }
 
+#ifndef HAVE_MPS
 void
 mark_xterm (void)
 {
@@ -32263,6 +32264,7 @@ mark_xterm (void)
     }
 #endif
 }
+#endif
 
 /* Error handling functions for Lisp functions that expose X protocol
    requests.  They are mostly like `x_catch_errors' and friends, but
-- 
2.39.2

Re: MPS signals and Emacs

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> The patch below would allow compilation with X.  It starts and I don't
> see any obvious breakage.  Maybe good enough to keep the kids quiet.

:-).

Could you perhaps push it? I _must_ now play with my new MacBook ;-).

Re: MPS signals and Emacs

[Helmut Eller]

On Tue, Apr 23 2024, Gerd Möllmann wrote:

> Could you perhaps push it? I _must_ now play with my new MacBook ;-).

I don't have commit rights and happy that I don't have the
responsibilities that would with them.  Anyway, no hurry.

Helmut

Re: MPS signals and Emacs

[Eli Zaretskii]

> Date: Mon, 22 Apr 2024 15:06:16 -0700
> Cc: gerd.moellmann@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 4/22/24 07:42, Eli Zaretskii wrote:
> > I think we only need to avoid installing signal handlers when dumping
> > using unexec, because with pdumper we don't dump the data of signal
> > handlers.  Paul, am I right?
> > 
> > If I'm right, the will_dump_p call should be replaced with
> > will_dump_with_unexec_p (which is not relevant to the MPS branch
> > anyway).
> 
> I think you're right here (though I haven't read the code to double check).

Thanks, I installed the change on master.  Let's see if anything
breaks.

Re: MPS signals and Emacs

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Tue, Apr 23 2024, Gerd Möllmann wrote:
>
>> Could you perhaps push it? I _must_ now play with my new MacBook ;-).
>
> I don't have commit rights and happy that I don't have the
> responsibilities that would with them.  Anyway, no hurry.

Thanks, and pushed.

(And flabbergasted that _everything_ seems to work on my new MB. Very
suspicious :-).

Re: Native compilation on MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> Also other immediate objects kept in the eln are reachable by the GC
> through the compilation unit object (see Lisp_Native_Comp_Unit).

Hi Andrea - just to keep you up to date...

That was a good tip, thanks! I had overlooked the * in the
Lisp_Object * at the end completely, and something has to be done for
that, that's sure.

I've added that now, but alas, it doesn't seem to be sufficient yet. I
get a similar error as before. I'll continue to debug this in the next
days.

Re: Native compilation on MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Also other immediate objects kept in the eln are reachable by the GC
>> through the compilation unit object (see Lisp_Native_Comp_Unit).
>
> Hi Andrea - just to keep you up to date...
>
> That was a good tip, thanks! I had overlooked the * in the
> Lisp_Object * at the end completely, and something has to be done for
> that, that's sure.
>
> I've added that now, but alas, it doesn't seem to be sufficient yet. I
> get a similar error as before. I'll continue to debug this in the next
> days.

Cool, keep us updated.

Thanks

  Andrea

Re: MPS signals and Emacs

[Gerd Möllmann]

dick <dick.r.chiang@gmail.com> writes:

> How do you afford Apple products on an emacs developer's salary?

Not being an Emacs developer helps because that means you
don't have to flip burgers.

Collecting markers with MPS (was: STatus of MPS branch)

[Helmut Eller]

On Sat, Apr 20 2024, Gerd Möllmann wrote:

> - Figure out what is done in the old GC in garbage_collect that is not
>   directly related to GC, and how to do that when GC is concurrent.
>   Examples are buffer undo_list, window prev_buffers/next_buffers

In the igc branch, markers are currently not collected because they are
referenced from the buffer's chain of markers.  The old GC has special
code to break this link.  There is also special code to break links from
the undo-list to markers.

With the MPS, we could introduce explicit weak reference objects,
similar to WeakRef in Javascript[*].  Buffers would then, instead of the
current chain of markers, have a list of weak references to markers.
(Markers would no longer have the embedded "next" field.)  The same for
the undo-list: instead of markers, it would contain weak references to
markers.

This would allow markers to be collected, but the empty weak-ref objects
would still be in the undo-list and the buffer's markers-list.  A timer
could periodically go through those lists and remove the empty
weak-refs.

Does this sound like a reasonable plan?  To me, it sounds like it will
require relatively big changes to the buffer code.  Much bigger than the
200 or so lines of special code in the old GC.  Also, the undo-list is
visible to Lisp code, so those weak reference objects in there must be
new types of Lisp objects.

On the plus side, explicit weak reference objects could make some things
easier to understand and might be more widely useful to Lisp
programmers.

What do you think?  Is there an more elegant alternative?

Helmut

[*] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakRef

Re: Collecting markers with MPS (was: STatus of MPS branch)

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> Date: Wed, 24 Apr 2024 09:26:08 +0200
> 
> On Sat, Apr 20 2024, Gerd Möllmann wrote:
> 
> > - Figure out what is done in the old GC in garbage_collect that is not
> >   directly related to GC, and how to do that when GC is concurrent.
> >   Examples are buffer undo_list, window prev_buffers/next_buffers
> 
> In the igc branch, markers are currently not collected because they are
> referenced from the buffer's chain of markers.  The old GC has special
> code to break this link.  There is also special code to break links from
> the undo-list to markers.
> 
> With the MPS, we could introduce explicit weak reference objects,
> similar to WeakRef in Javascript[*].  Buffers would then, instead of the
> current chain of markers, have a list of weak references to markers.
> (Markers would no longer have the embedded "next" field.)  The same for
> the undo-list: instead of markers, it would contain weak references to
> markers.
> 
> This would allow markers to be collected, but the empty weak-ref objects
> would still be in the undo-list and the buffer's markers-list.  A timer
> could periodically go through those lists and remove the empty
> weak-refs.
> 
> Does this sound like a reasonable plan?  To me, it sounds like it will
> require relatively big changes to the buffer code.  Much bigger than the
> 200 or so lines of special code in the old GC.  Also, the undo-list is
> visible to Lisp code, so those weak reference objects in there must be
> new types of Lisp objects.
> 
> On the plus side, explicit weak reference objects could make some things
> easier to understand and might be more widely useful to Lisp
> programmers.
> 
> What do you think?  Is there an more elegant alternative?

My opinions here mean very little for now, as I don't yet have a good
view of the issues, but in general: why do we have to do everything
the old GC did in the new GC?  Can't we leave some of the stuff to be
done in the main thread?  For example, the old GC would compact buffer
text, Lisp strings, and font caches -- this cannot be done from a
different thread, AFAIU.  Why not keep doing this from the main
thread, like we do now?  They are not really directly relevant to GC,
just some housekeeping tasks that need to be done from time to time.
Yes, it would mean the main thread still needs to stop from time to
time, but for much shorter periods of time.  And it will allow us to
sidestep the significant changes like those mentioned above, some of
which would mean Lisp-level changes that will affect Lisp programs.

If leaving some of this stuff in the main thread is reasonable, we
could add the above two jobs to that part, which would allow us to
leave the existing related code almost intact.

Does this make sense?

Re: Collecting markers with MPS

[Helmut Eller]

On Wed, Apr 24 2024, Eli Zaretskii wrote:

[...]
>> What do you think?  Is there an more elegant alternative?
>
> My opinions here mean very little for now, as I don't yet have a good
> view of the issues, but in general: why do we have to do everything
> the old GC did in the new GC?

And I was hoping you would say: "We don't need to collect markers.  It's
a nicety that was easy to do in the old GC.  If it's not easy in the new
GC, leave it out".

> Can't we leave some of the stuff to be
> done in the main thread?  For example, the old GC would compact buffer
> text, Lisp strings, and font caches -- this cannot be done from a
> different thread, AFAIU.  Why not keep doing this from the main
> thread, like we do now? They are not really directly relevant to GC,
> just some housekeeping tasks that need to be done from time to time.

How would the main thread know that a marker is not referenced from
anywhere else than from the undo-list?  Are you saying the main thread
should still do some marking as the old GC did, so that it can make the
decisions on the same mark bits?

> Yes, it would mean the main thread still needs to stop from time to
> time, but for much shorter periods of time.  And it will allow us to
> sidestep the significant changes like those mentioned above, some of
> which would mean Lisp-level changes that will affect Lisp programs.
>
> If leaving some of this stuff in the main thread is reasonable, we
> could add the above two jobs to that part, which would allow us to
> leave the existing related code almost intact.
>
> Does this make sense?

I think, moving some things from the old GC code to timers certainly
would make sense.  But for the marker issue in particular, I don't see
how it could be done without help from the GC.

Helmut

Re: Collecting markers with MPS

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Helmut Eller <eller.helmut@gmail.com>
>> Cc: emacs-devel <emacs-devel@gnu.org>
>> Date: Wed, 24 Apr 2024 09:26:08 +0200
>> 
>> On Sat, Apr 20 2024, Gerd Möllmann wrote:
>> 
>> > - Figure out what is done in the old GC in garbage_collect that is not
>> >   directly related to GC, and how to do that when GC is concurrent.
>> >   Examples are buffer undo_list, window prev_buffers/next_buffers
>> 
>> In the igc branch, markers are currently not collected because they are
>> referenced from the buffer's chain of markers.  The old GC has special
>> code to break this link.  There is also special code to break links from
>> the undo-list to markers.

Exactly! Glad that you look at this!

>> With the MPS, we could introduce explicit weak reference objects,
>> similar to WeakRef in Javascript[*].  Buffers would then, instead of the
>> current chain of markers, have a list of weak references to markers.
>> (Markers would no longer have the embedded "next" field.)  The same for
>> the undo-list: instead of markers, it would contain weak references to
>> markers.
>> 
>> This would allow markers to be collected, but the empty weak-ref objects
>> would still be in the undo-list and the buffer's markers-list.  A timer
>> could periodically go through those lists and remove the empty
>> weak-refs.
>> 
>> Does this sound like a reasonable plan?  To me, it sounds like it will
>> require relatively big changes to the buffer code.  Much bigger than the
>> 200 or so lines of special code in the old GC.  Also, the undo-list is
>> visible to Lisp code, so those weak reference objects in there must be
>> new types of Lisp objects.

Makes sense to me, but I must say that I have 0 experience with that
using MPS. It might require some experimentation how to about
implementing such weak references, but that could be fun.

Let me give Eli a brief overview what MPS has.

MPS support weak references. Objects containing weak references must be
allocated from a special type of memory pool, which MPS calls AWL pools
(= Automantic Weak Linked pool or so). Such a pool, and MPS allocatioin
points are already there in igc.

An object in AWL can contain references, but these references must
either all be weak or all be strong.

(That's okay for weak hash tables, where all keys are weak/strong or all
values are weak/strong references. Which, BTW, is the reason why I split
key_and_value into two vectors in the branch.)

>> On the plus side, explicit weak reference objects could make some things
>> easier to understand and might be more widely useful to Lisp
>> programmers.

Yes, I'd find that a big plus, too.

>> What do you think?  Is there an more elegant alternative?

Can't think of one. 

> My opinions here mean very little for now, as I don't yet have a good
> view of the issues, but in general: why do we have to do everything
> the old GC did in the new GC?  Can't we leave some of the stuff to be
> done in the main thread?  For example, the old GC would compact buffer
> text, Lisp strings, and font caches -- this cannot be done from a
> different thread, AFAIU.  Why not keep doing this from the main
> thread, like we do now?  They are not really directly relevant to GC,
> just some housekeeping tasks that need to be done from time to time.
> Yes, it would mean the main thread still needs to stop from time to
> time, but for much shorter periods of time.  And it will allow us to
> sidestep the significant changes like those mentioned above, some of
> which would mean Lisp-level changes that will affect Lisp programs.
>
> If leaving some of this stuff in the main thread is reasonable, we
> could add the above two jobs to that part, which would allow us to
> leave the existing related code almost intact.
>
> Does this make sense?

Yes, it makes sense. One could of course use a timer. Or maybe, don't
know, run something when we handle an MPS message. That just occurred to
me a second ago, so it might be nonsense.

But Helmut's idea _is_ pretty elegant and appealing I must say. And it
relieves the client of some more work (with the idea of improved
responsiveness...).

Re: Collecting markers with MPS

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> How would the main thread know that a marker is not referenced from
> anywhere else than from the undo-list?  Are you saying the main thread
> should still do some marking as the old GC did, so that it can make the
> decisions on the same mark bits?

Oh, that's right! Didn't think of it.

Re: Collecting markers with MPS

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: gerd.moellmann@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 24 Apr 2024 10:56:27 +0200
> 
> On Wed, Apr 24 2024, Eli Zaretskii wrote:
> 
> And I was hoping you would say: "We don't need to collect markers.  It's
> a nicety that was easy to do in the old GC.  If it's not easy in the new
> GC, leave it out".

Can we really do that?  I don't think so, but maybe I'm missing
something.

> How would the main thread know that a marker is not referenced from
> anywhere else than from the undo-list?  Are you saying the main thread
> should still do some marking as the old GC did, so that it can make the
> decisions on the same mark bits?

Can't the new GC tell us which markers are unreferenced?

Re: Collecting markers with MPS

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Can't the new GC tell us which markers are unreferenced?

I'm afraid no. At least I haven't seen it mentioned in the docs.

Re: Collecting markers with MPS

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Can't the new GC tell us which markers are unreferenced?
>
> I'm afraid no. At least I haven't seen it mentioned in the docs.

Err, and the marker would be references from the list anyway...

Re: Native compilation on MPS branch

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

Andrea Corallo <acorallo@gnu.org> writes:

> Cool, keep us updated.

Daily update ;-).

Haven't got much further, and can use a little break now.

I don't know if it makes sense to attach this, but maybe it does... I'm
sometimes keeping notes when debugging (which I find useful when
debugging more than 1 thing at a time).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: debugging --]
[-- Type: text/x-org, Size: 3900 bytes --]

:PROPERTIES:
:ID:       544E65F2-9EED-4D0E-812F-79205DA0040F
:END:
#+title: Debugging Ideas
* Assertion in string_char_and_length

In read_make_string a string is made that later leads to the assertion
in string_char_and_len.

#+begin_src sh
(lldb) p s
(const char *) 0x000000016fdf8588 "macroexp--cons"
#+end_src

Symbol whose name is that string is made in read:

#+begin_src sh
result = dwim-print result
(Lisp_Object) 0x00000000080935f8 (struct Lisp_Symbol *) $7 = 0x0000000108cc7568
#+end_src

Whole thing read looks like

#+begin_src sh
(lldb) xdebug_print retval
[let byte-compile-form-stack cons (byte-compile-form-stack) nil make-symbol "shared" "unshared" "tail" "new-el" let* (nil) while...
#+end_src

The read is in from load_static_obj, which is called from
load_comp_unit.

#+begin_src c
      if (!loading_dump)
	{
	  comp_u->optimize_qualities =
	    load_static_obj (comp_u, TEXT_OPTIM_QLY_SYM);
	  comp_u->data_vec = load_static_obj (comp_u, TEXT_DATA_RELOC_SYM);
	  comp_u->data_impure_vec =
	    load_static_obj (comp_u, TEXT_DATA_RELOC_IMPURE_SYM);

	  if (!NILP (Vpurify_flag))
	    /* Non impure can be copied into pure space.  */
	    comp_u->data_vec = Fpurecopy (comp_u->data_vec);

	  if (!NILP (Vpurify_flag))
	    /* Non impure can be copied into pure space.  */
	    comp_u->data_vec = Fpurecopy (comp_u->data_vec);
#+end_src

The cu is allocated in MPS, purify is nil

#+begin_src sh
(lldb) p comp_u
(Lisp_Native_Comp_Unit *) 0x0000000108cc51f0
(lldb) p is_mps (comp_u)
(bool) true
(lldb) p globals.f_Vpurify_flag
(Lisp_Object) NULL
#+end_src

Objects are copied

#+begin_src c
    Lisp_Object *data_relocs = dynlib_sym (handle, DATA_RELOC_SYM);
    Lisp_Object *data_imp_relocs = comp_u->data_imp_relocs;
    ...
    EMACS_INT d_vec_len = XFIXNUM (Flength (comp_u->data_vec));
    for (EMACS_INT i = 0; i < d_vec_len; i++)
      data_relocs[i] = AREF (comp_u->data_vec, i);
#+end_src

data_relocs points somewhere into the loaded dylib, I'd say

#+begin_src sh
  (lldb) p data_relocs
  (Lisp_Object *) 0x0000000100ebcfd8 (struct Lisp_Symbol *) $26 = 0x0000000201af0f48
  (lldb) p is_mps (data_relocs)
  (bool) false
  (lldb) p pdumper_object_p (data_relocs)
  (bool) false
  (lldb) p is_pure (data_relocs)
  (bool) false
#+end_src

data_imp_relocs too.

#+begin_src sh
  (lldb) p *comp_u
(Lisp_Native_Comp_Unit) {
  header = (size = 4611686018863607815)
  file = 0x0000000108cc50d4 (struct Lisp_String *) $28 = 0x0000000108cc50d0
  optimize_qualities = 0x0000000108cc53bb (struct Lisp_Cons *) $29 = 0x0000000108cc53b8
  lambda_gc_guard_h = 0x0000000108cc5255 (struct Lisp_Vector *) $30 = 0x0000000108cc5250
  lambda_c_name_idx_h = 0x0000000108cc52ad (struct Lisp_Vector *) $31 = 0x0000000108cc52a8
  data_fdoc_v = NULL
  data_vec = 0x0000000108cd000d (struct Lisp_Vector *) $32 = 0x0000000108cd0008
  data_impure_vec = NULL
  data_imp_relocs = 0x0000000100ebda00 (struct Lisp_Symbol *) $33 = 0x0000000201af1970
  loaded_once = false
  load_ongoing = true
  handle = 0x0000000088eb8210
}
#+end_src

That all looks like I would have expected.
(Note to self: remember register_comp_unit...)
That all happens in (require macroexp)

Continue. Landing in copy_sequence with invalid string.

#+begin_src sh
(struct Lisp_String) {
  u = {
    s = {
      size = 4546911032
      size_byte = -1
      intervals = NULL
      data = 0x0000000108cca740 "H\xdd\U00000003\U0000000f\U00000001"
    }
    next = 0x000000010f045b38
    gcaligned = '8'
  }
}
#+end_src

(Note to self: xbacktrace broken. By LLVM 18?)

Coming from print_prune_charset <- print_object <- print <-
Fprin1 of that string, ..., backtrace_frame_apply <- mapbacktrace ....
<- signal_or_quit.

From (error "Not enough aruments in format string") in styled_format.
args[0] is the kaputte string.

From readevalloop_eager_expand_eval, call of macroexpand.

And finito by calling xdebug_print, process was killed.

Re: Collecting markers with MPS

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Helmut Eller <eller.helmut@gmail.com>,  emacs-devel@gnu.org
> Date: Wed, 24 Apr 2024 12:58:23 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Can't the new GC tell us which markers are unreferenced?
> >
> > I'm afraid no. At least I haven't seen it mentioned in the docs.

Does this mean MPS doesn't allow a user-defined function to be called
when some object is found unreferenced?  I'd assume such callbacks are
available, since some objects need custom code to dispose of.

> Err, and the marker would be references from the list anyway...

Maybe I don't understand the problem we are talking about.  Is this
about unchain_dead_markers, which is called from sweep_buffers?  If
so, how are non-dead markers currently identified during the mark
phase?

Re: Collecting markers with MPS

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Helmut Eller <eller.helmut@gmail.com>,  emacs-devel@gnu.org
>> Date: Wed, 24 Apr 2024 12:58:23 +0200
>> 
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>> 
>> > Eli Zaretskii <eliz@gnu.org> writes:
>> >
>> >> Can't the new GC tell us which markers are unreferenced?
>> >
>> > I'm afraid no. At least I haven't seen it mentioned in the docs.
>
> Does this mean MPS doesn't allow a user-defined function to be called
> when some object is found unreferenced?  I'd assume such callbacks are
> available, since some objects need custom code to dispose of.

I think you mean finalization, which is supported, and implemented in
igc.

(MPS puts a finalization message on a message queue, which the client
processes when it wants. It can then call xfree malloc'd memory,
,unload compilation units etc.)

>> Err, and the marker would be references from the list anyway...
>
> Maybe I don't understand the problem we are talking about.  Is this
> about unchain_dead_markers, which is called from sweep_buffers?  If
> so, how are non-dead markers currently identified during the mark
> phase?

I'll better leave that Helmut to answer, he probably knows that better
than me, at the moment. Finalization of markers would not happen,
because the list it is in keeps it alive.

Re: Collecting markers with MPS

[Helmut Eller]

On Wed, Apr 24 2024, Eli Zaretskii wrote:

>> And I was hoping you would say: "We don't need to collect markers.  It's
>> a nicety that was easy to do in the old GC.  If it's not easy in the new
>> GC, leave it out".
>
> Can we really do that?  I don't think so, but maybe I'm missing
> something.

We don't need it for correctness.  We would unlikely run out of memory.
However, a buffer with an unreasonably long marker-chain will make some
operations slower.  How much slowness can we accept?  I don't know;
maybe I should measure this first.

We could also provide functions to manually unlink markers from the
buffer that programmers could use to keep the slowness in check.

>> How would the main thread know that a marker is not referenced from
>> anywhere else than from the undo-list?  Are you saying the main thread
>> should still do some marking as the old GC did, so that it can make the
>> decisions on the same mark bits?
>
> Can't the new GC tell us which markers are unreferenced?

The MPS supports finalization and weak references.  Finalization could
tell us which markers are unreferenced.  The problem is that markers
stay referenced as long as they are part of the buffer's marker-chain or
the undo-list.

That's why we could use weak references here.  If a marker is referenced
only through weak references then MPS can collect it.  MPS will also set
the weak reference to 0 when the marker has been collected.  So that we
know when we no longer can/need to update the marker.

The old GC knows which references are weak; that's what the special code
there is for.  For MPS, we somehow need to tell it which references are
weak.

Helmut

Re: Collecting markers with MPS

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: gerd.moellmann@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 24 Apr 2024 17:03:10 +0200
> 
> On Wed, Apr 24 2024, Eli Zaretskii wrote:
> 
> >> And I was hoping you would say: "We don't need to collect markers.  It's
> >> a nicety that was easy to do in the old GC.  If it's not easy in the new
> >> GC, leave it out".
> >
> > Can we really do that?  I don't think so, but maybe I'm missing
> > something.
> 
> We don't need it for correctness.  We would unlikely run out of memory.

Are you sure?  Emacs could easily create gobs of markers.  For
example, if there are buffers with lost of non-ASCII characters, each
time we need to compute character position from byte position or vice
versa, we create more markers in the buffer.  In a session that runs
for weeks, there could be a lot of them, I think.

> However, a buffer with an unreasonably long marker-chain will make some
> operations slower.  How much slowness can we accept?  I don't know;
> maybe I should measure this first.

I think we had in the past complaints from Ihor that some buffers with
a lot of markers slowed down things.

> We could also provide functions to manually unlink markers from the
> buffer that programmers could use to keep the slowness in check.

If we can do this manually, we should be able to do it from a timer,
no?

> The MPS supports finalization and weak references.  Finalization could
> tell us which markers are unreferenced.  The problem is that markers
> stay referenced as long as they are part of the buffer's marker-chain or
> the undo-list.
> 
> That's why we could use weak references here.  If a marker is referenced
> only through weak references then MPS can collect it.  MPS will also set
> the weak reference to 0 when the marker has been collected.  So that we
> know when we no longer can/need to update the marker.
> 
> The old GC knows which references are weak; that's what the special code
> there is for.  For MPS, we somehow need to tell it which references are
> weak.

So why did you say this will take much more code than in the old GC?

Re: Collecting markers with MPS

[Helmut Eller]

On Wed, Apr 24 2024, Eli Zaretskii wrote:

>> We don't need it for correctness.  We would unlikely run out of memory.
>
> Are you sure?  Emacs could easily create gobs of markers.  For
> example, if there are buffers with lost of non-ASCII characters, each
> time we need to compute character position from byte position or vice
> versa, we create more markers in the buffer.  In a session that runs
> for weeks, there could be a lot of them, I think.

Maybe those people who run Emacs for weeks can afford new Macbooks so
that they can it run for months :-)

>> We could also provide functions to manually unlink markers from the
>> buffer that programmers could use to keep the slowness in check.
>
> If we can do this manually, we should be able to do it from a timer,
> no?

Yes, but only the programmers know which markers they no longer need.

>> The MPS supports finalization and weak references.  Finalization could
>> tell us which markers are unreferenced.  The problem is that markers
>> stay referenced as long as they are part of the buffer's marker-chain or
>> the undo-list.
>> 
>> That's why we could use weak references here.  If a marker is referenced
>> only through weak references then MPS can collect it.  MPS will also set
>> the weak reference to 0 when the marker has been collected.  So that we
>> know when we no longer can/need to update the marker.
>> 
>> The old GC knows which references are weak; that's what the special code
>> there is for.  For MPS, we somehow need to tell it which references are
>> weak.
>
> So why did you say this will take much more code than in the old GC?

Because I think introducing those weak reference objects in the
marker-chain needs quite a number of changes.  It's like changing a foo*
to a foo**.  Simple conceptually, but many places need to be touched.
Most of the changes should be straightforward, I hope.

Helmut

Re: Collecting markers with MPS

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: gerd.moellmann@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 24 Apr 2024 18:21:22 +0200
> 
> On Wed, Apr 24 2024, Eli Zaretskii wrote:
> 
> >> We could also provide functions to manually unlink markers from the
> >> buffer that programmers could use to keep the slowness in check.
> >
> > If we can do this manually, we should be able to do it from a timer,
> > no?
> 
> Yes, but only the programmers know which markers they no longer need.

Sorry, I don't follow: how does the old GC detect dead markers?  Why
cannot a function we write do the same?

I don't believe programmers will know which markers are no longer
needed, primarily because most markers are completely unknown to
programmers, Emacs creates them for its own purposes and never exposes
them to Lisp.

Re: Collecting markers with MPS

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> Yes, but only the programmers know which markers they no longer need.
>
> Sorry, I don't follow: how does the old GC detect dead markers?  Why
> cannot a function we write do the same?

IIRC, the existing GC doesn't process the lists until all other marking
has been done. If a marker is not GC-marked then, we know that there are
no references than from the lists.

Re: Native compilation on MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Cool, keep us updated.
>
> Daily update ;-).

:)

> Haven't got much further, and can use a little break now.
>
> I don't know if it makes sense to attach this, but maybe it does... I'm
> sometimes keeping notes when debugging (which I find useful when
> debugging more than 1 thing at a time).

[...]

>
> The read is in from load_static_obj, which is called from
> load_comp_unit.
>
> #+begin_src c
>       if (!loading_dump)
> 	{
> 	  comp_u->optimize_qualities =
> 	    load_static_obj (comp_u, TEXT_OPTIM_QLY_SYM);
> 	  comp_u->data_vec = load_static_obj (comp_u, TEXT_DATA_RELOC_SYM);
> 	  comp_u->data_impure_vec =
> 	    load_static_obj (comp_u, TEXT_DATA_RELOC_IMPURE_SYM);
>
> 	  if (!NILP (Vpurify_flag))
> 	    /* Non impure can be copied into pure space.  */
> 	    comp_u->data_vec = Fpurecopy (comp_u->data_vec);
>
> 	  if (!NILP (Vpurify_flag))
> 	    /* Non impure can be copied into pure space.  */
> 	    comp_u->data_vec = Fpurecopy (comp_u->data_vec);
> #+end_src
>
> The cu is allocated in MPS, purify is nil
>
> #+begin_src sh
> (lldb) p comp_u
> (Lisp_Native_Comp_Unit *) 0x0000000108cc51f0
> (lldb) p is_mps (comp_u)
> (bool) true
> (lldb) p globals.f_Vpurify_flag
> (Lisp_Object) NULL
> #+end_src
>
> Objects are copied
>
> #+begin_src c
>     Lisp_Object *data_relocs = dynlib_sym (handle, DATA_RELOC_SYM);
>     Lisp_Object *data_imp_relocs = comp_u->data_imp_relocs;
>     ...
>     EMACS_INT d_vec_len = XFIXNUM (Flength (comp_u->data_vec));
>     for (EMACS_INT i = 0; i < d_vec_len; i++)
>       data_relocs[i] = AREF (comp_u->data_vec, i);
> #+end_src
>
> data_relocs points somewhere into the loaded dylib, I'd say

That's correct

> #+begin_src sh
>   (lldb) p data_relocs
>   (Lisp_Object *) 0x0000000100ebcfd8 (struct Lisp_Symbol *) $26 = 0x0000000201af0f48
>   (lldb) p is_mps (data_relocs)
>   (bool) false
>   (lldb) p pdumper_object_p (data_relocs)
>   (bool) false
>   (lldb) p is_pure (data_relocs)
>   (bool) false
> #+end_src
>
> data_imp_relocs too.

Yep

> #+begin_src sh
>   (lldb) p *comp_u
> (Lisp_Native_Comp_Unit) {
>   header = (size = 4611686018863607815)
>   file = 0x0000000108cc50d4 (struct Lisp_String *) $28 = 0x0000000108cc50d0
>   optimize_qualities = 0x0000000108cc53bb (struct Lisp_Cons *) $29 = 0x0000000108cc53b8
>   lambda_gc_guard_h = 0x0000000108cc5255 (struct Lisp_Vector *) $30 = 0x0000000108cc5250
>   lambda_c_name_idx_h = 0x0000000108cc52ad (struct Lisp_Vector *) $31 = 0x0000000108cc52a8
>   data_fdoc_v = NULL
>   data_vec = 0x0000000108cd000d (struct Lisp_Vector *) $32 = 0x0000000108cd0008
>   data_impure_vec = NULL
>   data_imp_relocs = 0x0000000100ebda00 (struct Lisp_Symbol *) $33 = 0x0000000201af1970
>   loaded_once = false
>   load_ongoing = true
>   handle = 0x0000000088eb8210
> }
> #+end_src
>
> That all looks like I would have expected.

To me as well.

> (Note to self: remember register_comp_unit...)
> That all happens in (require macroexp)
>
> Continue. Landing in copy_sequence with invalid string.

:/

Dumb question: are you usnug rr to debug this?

Thanks

  Andrea

Re: Native compilation on MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> Dumb question: are you usnug rr to debug this?

Not dumb at all :-). Last time I looked, which is now 10s ago, Homebrew
still didn't offer rr. That's on arm, in case it matters.

If someone reading this knows how to do reversible debugging with LLDB -
I'm all ears :-)

Re: Collecting markers with MPS

[Helmut Eller]

[-- Attachment #1: Type: text/plain, Size: 375 bytes --]

On Wed, Apr 24 2024, Gerd Möllmann wrote:

> Makes sense to me, but I must say that I have 0 experience with that
> using MPS. It might require some experimentation how to about
> implementing such weak references, but that could be fun.

I implemented such weak references in the patch below.  Do I use the
IGC_OBJ_WEAK tag roughly how you intended it?

Helmut


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Experimental-weak-ref-objects.patch --]
[-- Type: text/x-diff, Size: 9677 bytes --]

From 4b0d85a06e6cec4a8c77f271e25cd87abaa52fdc Mon Sep 17 00:00:00 2001
From: Helmut Eller <eller.helmut@gmail.com>
Date: Wed, 24 Apr 2024 20:54:16 +0200
Subject: [PATCH] Experimental weak-ref objects

---
 src/data.c            |   5 +-
 src/igc.c             | 158 ++++++++++++++++++++++++++++++++++++++++--
 src/igc.h             |  16 +++++
 src/lisp.h            |   1 +
 src/pdumper.c         |   2 +
 src/print.c           |  10 +++
 test/src/igc-tests.el |  16 +++++
 7 files changed, 200 insertions(+), 8 deletions(-)
 create mode 100644 test/src/igc-tests.el

diff --git a/src/data.c b/src/data.c
index d52b73b8681..49b33727d75 100644
--- a/src/data.c
+++ b/src/data.c
@@ -296,7 +296,10 @@ DEFUN ("cl-type-of", Fcl_type_of, Scl_type_of, 1, 1, 0,
           return Qsqlite;
         case PVEC_SUB_CHAR_TABLE:
           return Qsub_char_table;
-        /* "Impossible" cases.  */
+	case PVEC_WEAK_REF:
+	  return Qweak_ref;
+
+	/* "Impossible" cases.  */
 	case PVEC_MISC_PTR:
         case PVEC_OTHER:
         case PVEC_FREE: ;
diff --git a/src/igc.c b/src/igc.c
index d0b3e8e3fa6..e64e0621c77 100644
--- a/src/igc.c
+++ b/src/igc.c
@@ -63,6 +63,12 @@
 # warning "HAVE_TEXT_CONVERSION not supported"
 #endif
 
+struct Lisp_Weak_Ref
+{
+  union vectorlike_header header;
+  Lisp_Object ref;
+} GCALIGNED_STRUCT;
+
 /* Note: Emacs will call allocation functions whlle aborting. This leads
    to all sorts of interesting phenomena when an assertion fails inside
    a function called from MPS.
@@ -1024,9 +1030,92 @@ fix_face_cache (mps_ss_t ss, struct face_cache *c)
 }
 
 static mps_res_t
-fix_weak (mps_ss_t ss, mps_addr_t base)
+fix_weak_ref (mps_ss_t ss, struct Lisp_Weak_Ref *wref)
+{
+  MPS_SCAN_BEGIN (ss)
+  {
+    const mps_word_t tagged_word = *(mps_word_t *)&wref->ref;
+    const enum Lisp_Type tag = tagged_word & IGC_TAG_MASK;
+
+    switch (tag)
+      {
+      case Lisp_Int0:
+      case Lisp_Int1:
+	return MPS_RES_OK;
+
+      case Lisp_Type_Unused0:
+	emacs_abort ();
+
+      case Lisp_Symbol:
+	{
+	  ptrdiff_t off = tagged_word ^ Lisp_Symbol;
+	  mps_addr_t client = (mps_addr_t)((char *)lispsym + off);
+	  if (is_mps (client))
+	    {
+	      mps_addr_t base = client_to_base (client);
+	      if (MPS_FIX1 (ss, base))
+		{
+		  mps_res_t res = MPS_FIX2 (ss, &base);
+		  if (res != MPS_RES_OK)
+		    return res;
+		  if (base == NULL)
+		    {
+		      wref->ref = Qnil;
+		    }
+		  else
+		    {
+		      client = base_to_client (base);
+		      ptrdiff_t new_off = (char *)client - (char *)lispsym;
+		      wref->ref = (Lisp_Object)(new_off | tag);
+		    }
+		}
+	    }
+	}
+	break;
+
+      default:
+	{
+	  const mps_addr_t client = (mps_addr_t)(tagged_word ^ tag);
+	  if (is_mps (client))
+	    {
+	      mps_addr_t base = client_to_base (client);
+	      if (MPS_FIX1 (ss, base))
+		{
+		  const mps_res_t res = MPS_FIX2 (ss, &base);
+		  if (res != MPS_RES_OK)
+		    return res;
+		  if (base == NULL)
+		    {
+		      wref->ref = Qnil;
+		    }
+		  else
+		    {
+		      const mps_addr_t client2 = base_to_client (base);
+		      wref->ref = (Lisp_Object)((mps_word_t)client2 | tag);
+		    }
+		}
+	    }
+	}
+      }
+  }
+  MPS_SCAN_END (ss);
+  return MPS_RES_OK;
+}
+
+static mps_res_t
+fix_weak (mps_ss_t ss, struct igc_header* base)
 {
-  MPS_SCAN_BEGIN (ss) { igc_assert (!"fix_weak"); }
+  MPS_SCAN_BEGIN (ss) {
+    const mps_addr_t client = base_to_client(base);
+    switch (base->pvec_type)
+      {
+      case PVEC_WEAK_REF:
+	IGC_FIX_CALL_FN (ss, struct Lisp_Weak_Ref, client, fix_weak_ref);
+	break;
+      default:
+	igc_assert (!"fix_weak");
+      }
+  }
   MPS_SCAN_END (ss);
   return MPS_RES_OK;
 }
@@ -1139,7 +1228,7 @@ dflt_scanx (mps_ss_t ss, mps_addr_t base_start, mps_addr_t base_limit,
 	    break;
 
 	  case IGC_OBJ_WEAK:
-	    IGC_FIX_CALL_FN (ss, mps_word_t, client, fix_weak);
+	    IGC_FIX_CALL_FN (ss, struct igc_header, base, fix_weak);
 	    break;
 	  }
       }
@@ -1633,6 +1722,9 @@ fix_vector (mps_ss_t ss, struct Lisp_Vector *v)
 #endif
 	IGC_FIX_CALL_FN (ss, struct Lisp_Vector, v, fix_vectorlike);
 	break;
+
+      case PVEC_WEAK_REF:
+	emacs_abort ();
       }
   }
   MPS_SCAN_END (ss);
@@ -2231,6 +2323,7 @@ finalize_vector (mps_addr_t v)
     case PVEC_XWIDGET_VIEW:
     case PVEC_TERMINAL:
     case PVEC_MARKER:
+    case PVEC_WEAK_REF:
       igc_assert (!"not implemented");
       break;
     }
@@ -2321,6 +2414,7 @@ maybe_finalize (mps_addr_t client, enum pvec_type tag)
 #ifdef IN_MY_FORK
     case PVEC_PACKAGE:
 #endif
+    case PVEC_WEAK_REF:
       break;
     }
 }
@@ -2387,9 +2481,11 @@ thread_ap (enum igc_obj_type type)
     case IGC_OBJ_PAD:
     case IGC_OBJ_FWD:
     case IGC_OBJ_LAST:
-    case IGC_OBJ_WEAK:
       emacs_abort ();
 
+    case IGC_OBJ_WEAK:
+      return t->d.weak_weak_ap;
+
     case IGC_OBJ_CONS:
     case IGC_OBJ_SYMBOL:
     case IGC_OBJ_INTERVAL:
@@ -2425,6 +2521,14 @@ igc_collect (void)
   mps_arena_release (gc->arena);
 }
 
+DEFUN ("igc--collect", Figc__collect, Sigc__collect, 0, 0, 0, doc
+       : /* */)
+(void)
+{
+  igc_collect ();
+  return Qnil;
+}
+
 static unsigned
 obj_hash (void)
 {
@@ -2665,11 +2769,46 @@ igc_make_face_cache (void)
   return c;
 }
 
-struct Lisp_Buffer_Local_Value *
-igc_alloc_blv (void)
+DEFUN ("igc-make-weak-ref", Figc_make_weak_ref, Sigc_make_weak_ref, 1, 1, 0,
+       doc
+       : /* todo */)
+(Lisp_Object target)
+{
+  const enum pvec_type type = PVEC_WEAK_REF;
+  struct Lisp_Weak_Ref *wref = alloc (sizeof *wref, IGC_OBJ_WEAK, type);
+  int nwords_lisp = VECSIZE(struct Lisp_Weak_Ref);
+  XSETPVECTYPESIZE (wref, type, nwords_lisp, 0);
+  maybe_finalize (wref, type);
+  wref->ref = target;
+  Lisp_Object obj = make_lisp_ptr (wref, Lisp_Vectorlike);
+  return obj;
+}
+
+static void
+CHECK_WEAK_REF_P (Lisp_Object x)
+{
+  CHECK_TYPE (WEAK_REF_P (x), Qweak_ref_p, x);
+}
+
+Lisp_Object
+igc_weak_ref_deref (struct Lisp_Weak_Ref *wref)
+{
+  return wref->ref;
+}
+
+DEFUN ("igc-weak-ref-deref", Figc_weak_reaf_deref, Sigc_weak_ref_deref, 1, 1,
+       0, doc
+       : /* todo */)
+(Lisp_Object obj)
+{
+  CHECK_WEAK_REF_P (obj);
+  return igc_weak_ref_deref (XWEAK_REF (obj));
+}
+
+struct Lisp_Buffer_Local_Value *igc_alloc_blv (void)
 {
   struct Lisp_Buffer_Local_Value *blv
-    = alloc (sizeof *blv, IGC_OBJ_BLV, PVEC_FREE);
+      = alloc (sizeof *blv, IGC_OBJ_BLV, PVEC_FREE);
   return blv;
 }
 
@@ -2871,7 +3010,12 @@ syms_of_igc (void)
 {
   defsubr (&Sigc_info);
   defsubr (&Sigc_roots);
+  defsubr (&Sigc_make_weak_ref);
+  defsubr (&Sigc_weak_ref_deref);
+  defsubr (&Sigc__collect);
   DEFSYM (Qambig, "ambig");
   DEFSYM (Qexact, "exact");
+  DEFSYM (Qweak_ref_p, "weak-ref-p");
+  DEFSYM (Qweak_ref, "weak-ref");
   Fprovide (intern_c_string ("mps"), Qnil);
 }
diff --git a/src/igc.h b/src/igc.h
index 0183b8da71f..e29e8f7cecc 100644
--- a/src/igc.h
+++ b/src/igc.h
@@ -93,6 +93,22 @@ #define EMACS_IGC_H
 void igc_root_create_exact (Lisp_Object *start, Lisp_Object *end);
 void igc_root_create_exact_ptr (void *var_addr);
 
+struct Lisp_Weak_Ref;
+Lisp_Object igc_weak_ref_deref (struct Lisp_Weak_Ref *);
+
+INLINE bool
+WEAK_REF_P (Lisp_Object x)
+{
+  return PSEUDOVECTORP (x, PVEC_WEAK_REF);
+}
+
+INLINE struct Lisp_Weak_Ref *
+XWEAK_REF (Lisp_Object a)
+{
+  eassert (WEAK_REF_P (a));
+  return XUNTAG (a, Lisp_Vectorlike, struct Lisp_Weak_Ref);
+}
+
 # define eassert_not_mps() eassert (false)
 #else
 # define igc_break() (void) 0
diff --git a/src/lisp.h b/src/lisp.h
index ce1516e9673..32b28921c44 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -1047,6 +1047,7 @@ DEFINE_GDB_SYMBOL_END (PSEUDOVECTOR_FLAG)
   PVEC_TS_NODE,
   PVEC_TS_COMPILED_QUERY,
   PVEC_SQLITE,
+  PVEC_WEAK_REF,
 
   /* These should be last, for internal_equal and sxhash_obj.  */
   PVEC_COMPILED,
diff --git a/src/pdumper.c b/src/pdumper.c
index 8b179247db0..cca773a41bf 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -3170,6 +3170,8 @@ dump_vectorlike (struct dump_context *ctx,
     case PVEC_TS_NODE:
     case PVEC_TS_COMPILED_QUERY:
       break;
+    case PVEC_WEAK_REF:
+      emacs_abort();
     }
   char msg[60];
   snprintf (msg, sizeof msg, "pseudovector type %d", (int) ptype);
diff --git a/src/print.c b/src/print.c
index 81b6439fd7b..a2bd8ef0b19 100644
--- a/src/print.c
+++ b/src/print.c
@@ -33,6 +33,7 @@ Copyright (C) 1985-2024 Free Software Foundation, Inc.
 #include "blockinput.h"
 #include "xwidget.h"
 #include "dynlib.h"
+#include "igc.h"
 
 #include <c-ctype.h>
 #include <float.h>
@@ -2089,6 +2090,15 @@ print_vectorlike_unreadable (Lisp_Object obj, Lisp_Object printcharfun,
 	return;
       }
 
+    case PVEC_WEAK_REF:
+      {
+	Lisp_Object target = igc_weak_ref_deref (XWEAK_REF(obj));
+	print_c_string ("#<weak-ref ", printcharfun);
+	print_object (target, printcharfun, escapeflag);
+	print_c_string (">", printcharfun);
+	return;
+      }
+
     /* Types handled earlier.  */
     case PVEC_NORMAL_VECTOR:
     case PVEC_RECORD:
diff --git a/test/src/igc-tests.el b/test/src/igc-tests.el
new file mode 100644
index 00000000000..02caf6661f5
--- /dev/null
+++ b/test/src/igc-tests.el
@@ -0,0 +1,16 @@
+;;; igc-tests.el --- Tests for igc.c                -*- lexical-binding: t -*-
+
+(require 'ert)
+
+(defvar igc-test-v0 (igc-make-weak-ref (list 1 2)))
+(defvar igc-test-v1 (igc-make-weak-ref (make-symbol "foo")))
+
+(ert-deftest igc-test-weak-refs ()
+  (igc--collect)
+  (garbage-collect)
+  (should (equal (igc-weak-ref-deref igc-test-v0) nil))
+  (should (equal (igc-weak-ref-deref igc-test-v1) nil))
+  (let ((wref (igc-make-weak-ref (list 3 4))))
+    (should (equal (igc-weak-ref-deref wref) '(3 4)))))
+
+(provide 'igc-tests)
-- 
2.39.2

Re: Collecting markers with MPS

[Helmut Eller]

On Wed, Apr 24 2024, Eli Zaretskii wrote:

>> Yes, but only the programmers know which markers they no longer need.
>
> Sorry, I don't follow: how does the old GC detect dead markers?

I don't know the details. You'd have to read the code.  It's wizard
level.

> Why cannot a function we write do the same?

Because we would probably have to scan the heap for all potentially live
objects.

> I don't believe programmers will know which markers are no longer
> needed, primarily because most markers are completely unknown to
> programmers, Emacs creates them for its own purposes and never exposes
> them to Lisp.

Not all of course.  But those programmers who create markers like crazy
probably know it.  If they assume that markers are expensive then they
could better keep track of them (or not create them in the first place).

Helmut

Re: Collecting markers with MPS

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Wed, Apr 24 2024, Eli Zaretskii wrote:
>
>>> Yes, but only the programmers know which markers they no longer need.
>>
>> Sorry, I don't follow: how does the old GC detect dead markers?
>
> I don't know the details. You'd have to read the code.  It's wizard
> level.

Not really. In the mark phase, process everything except what's weak.
When that is done, we know what's referenced non-weak, and can process
the weak pqrts. Everything that is marked now is referenced strongly.
Plus some minor details with interdependencies between objects
containing weak references.

>> Why cannot a function we write do the same?
>
> Because we would probably have to scan the heap for all potentially live
> objects.

Yup. The complete mark-phase of the old GC. That's would be very
expensive.

Re: Collecting markers with MPS

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Wed, Apr 24 2024, Gerd Möllmann wrote:
>
>> Makes sense to me, but I must say that I have 0 experience with that
>> using MPS. It might require some experimentation how to about
>> implementing such weak references, but that could be fun.
>
> I implemented such weak references in the patch below.  Do I use the
> IGC_OBJ_WEAK tag roughly how you intended it?

👍

Can I push that? We could make fixes, if any are necessary afterwards.
And I find that it's certainly a useful addition, independent of markers.

Re: Collecting markers with MPS

[Helmut Eller]

On Wed, Apr 24 2024, Gerd Möllmann wrote:

> Can I push that? We could make fixes, if any are necessary afterwards.
> And I find that it's certainly a useful addition, independent of markers.

Sure.  We could also remove it later.

Helmut

Re: Collecting markers with MPS

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Wed, Apr 24 2024, Gerd Möllmann wrote:
>
>> Can I push that? We could make fixes, if any are necessary afterwards.
>> And I find that it's certainly a useful addition, independent of markers.
>
> Sure.  We could also remove it later.

Thanks! It's in.

Re: Native compilation on MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Dumb question: are you usnug rr to debug this?
>
> Not dumb at all :-). Last time I looked, which is now 10s ago, Homebrew
> still didn't offer rr. That's on arm, in case it matters.

IME rr is invaluable for debugging GC bugs, one can do things like
placing watch-points while doing reverse execution and discover
immediately where something was modified the last time.  You did a very
good job reaching this point of the development without rr.

Anyway I'll mention a trick I find extremely usefull to discover where
something was modified last time before a certain event without rr, I
apologize if you know it already but in case it might help some other
reader :)

1 When we crash we set a watch point on the memory we see got
unexpectedly modified
2 We set the ignore number of the watch-point to a very large number
3 We restart the execution of the program
4 When the program crashes again we can see how many time the watch point
was actually ignored
5 We subtract 1 to this number and we set it again as the ignore number
of the watch-point
6 We restart again and tada!! We are where that piece of memory was
modified for the last time :)

Hope this helps someone

  Andrea

Re: Native compilation on MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Andrea Corallo <acorallo@gnu.org> writes:
>>
>>> Dumb question: are you usnug rr to debug this?
>>
>> Not dumb at all :-). Last time I looked, which is now 10s ago, Homebrew
>> still didn't offer rr. That's on arm, in case it matters.
>
> IME rr is invaluable for debugging GC bugs, one can do things like
> placing watch-points while doing reverse execution and discover
> immediately where something was modified the last time.  

Yeah, I agree. It's sort of the holy grail of debugging.

(I remember an effort to add reversible debugging to LLDB some years
ago, don't know what became of that. I should probably investigate that
a bit.)

> You did a very good job reaching this point of the development without
> rr.

Thanks. See - nobody really needs this new-fangled stuff that young
people are using. It's like with these tictac videos. No, I'm not
envious.

> Anyway I'll mention a trick I find extremely usefull to discover where
> something was modified last time before a certain event without rr, I
> apologize if you know it already but in case it might help some other
> reader :)
>
> 1 When we crash we set a watch point on the memory we see got
> unexpectedly modified
> 2 We set the ignore number of the watch-point to a very large number
> 3 We restart the execution of the program
> 4 When the program crashes again we can see how many time the watch point
> was actually ignored
> 5 We subtract 1 to this number and we set it again as the ignore number
> of the watch-point
> 6 We restart again and tada!! We are where that piece of memory was
> modified for the last time :)
>
> Hope this helps someone

Nice trick, thanks for sharing!

Re: Native compilation on MPS branch

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 24 Apr 2024 16:54:30 -0400
> 
> IME rr is invaluable for debugging GC bugs, one can do things like
> placing watch-points while doing reverse execution and discover
> immediately where something was modified the last time.  You did a very
> good job reaching this point of the development without rr.

We have the last_marked[] array to facilitate debugging of GC without
rr.  I hope we will be able to do something similar with MPS, because
rr is not universally available (e.g., not on Windows).

> Anyway I'll mention a trick I find extremely usefull to discover where
> something was modified last time before a certain event without rr, I
> apologize if you know it already but in case it might help some other
> reader :)
> 
> 1 When we crash we set a watch point on the memory we see got
> unexpectedly modified
> 2 We set the ignore number of the watch-point to a very large number
> 3 We restart the execution of the program
> 4 When the program crashes again we can see how many time the watch point
> was actually ignored
> 5 We subtract 1 to this number and we set it again as the ignore number
> of the watch-point
> 6 We restart again and tada!! We are where that piece of memory was
> modified for the last time :)

This only works if you disable ASLR, AFAIU.

Re: Native compilation on MPS branch

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Wed, 24 Apr 2024 16:54:30 -0400
>> 
>> IME rr is invaluable for debugging GC bugs, one can do things like
>> placing watch-points while doing reverse execution and discover
>> immediately where something was modified the last time.  You did a very
>> good job reaching this point of the development without rr.
>
> We have the last_marked[] array to facilitate debugging of GC without
> rr.  I hope we will be able to do something similar with MPS, because
> rr is not universally available (e.g., not on Windows).
>
>> Anyway I'll mention a trick I find extremely usefull to discover where
>> something was modified last time before a certain event without rr, I
>> apologize if you know it already but in case it might help some other
>> reader :)
>> 
>> 1 When we crash we set a watch point on the memory we see got
>> unexpectedly modified
>> 2 We set the ignore number of the watch-point to a very large number
>> 3 We restart the execution of the program
>> 4 When the program crashes again we can see how many time the watch point
>> was actually ignored
>> 5 We subtract 1 to this number and we set it again as the ignore number
>> of the watch-point
>> 6 We restart again and tada!! We are where that piece of memory was
>> modified for the last time :)
>
> This only works if you disable ASLR, AFAIU.

Correct, on GNU/Linux gdb does that by default, no idea on other
systems.

  Andrea

Re: Native compilation on MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>>> Andrea Corallo <acorallo@gnu.org> writes:
>>>
>>>> Dumb question: are you usnug rr to debug this?
>>>
>>> Not dumb at all :-). Last time I looked, which is now 10s ago, Homebrew
>>> still didn't offer rr. That's on arm, in case it matters.
>>
>> IME rr is invaluable for debugging GC bugs, one can do things like
>> placing watch-points while doing reverse execution and discover
>> immediately where something was modified the last time.  
>
> Yeah, I agree. It's sort of the holy grail of debugging.
>
> (I remember an effort to add reversible debugging to LLDB some years
> ago, don't know what became of that. I should probably investigate that
> a bit.)
>
>> You did a very good job reaching this point of the development without
>> rr.
>
> Thanks. See - nobody really needs this new-fangled stuff that young
> people are using.

Agree, the trick I shared actually covers for me the vast majority of
the cases I'd use rr for :)

  Andrea

Re: Native compilation on MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> Agree, the trick I shared actually covers for me the vast majority of
> the cases I'd use rr for :)

BTW, maybe it's not obvious why I didn't use watchpoints so far...

My theory ATM is that this is a caused by a reference that escapes my
scanning/fixing because I don't know it exists. The reference therefore
does not notice what is happening around it and that the object it
pointed to has moved.

So, the problem is that something _doesn't_ change, which is kind of
hard to watch.

Re: Collecting markers with MPS

[Mattias Engdegård]

24 apr. 2024 kl. 21.18 skrev Helmut Eller <eller.helmut@gmail.com>:

> Not all of course.  But those programmers who create markers like crazy
> probably know it.

No they don't. Speaking from experience.

Re: Native compilation on MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Agree, the trick I shared actually covers for me the vast majority of
>> the cases I'd use rr for :)
>
> BTW, maybe it's not obvious why I didn't use watchpoints so far...
>
> My theory ATM is that this is a caused by a reference that escapes my
> scanning/fixing because I don't know it exists. The reference therefore
> does not notice what is happening around it and that the object it
> pointed to has moved.
>
> So, the problem is that something _doesn't_ change, which is kind of
> hard to watch.

I see, if you see is necessary we can try with a simpler case before
full native bootstrap, like a non native build but with native-comp
support.  Than we can compile a single file and load it.

PS I'm trying to compile the branch to give it a try :)

Re: Native compilation on MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> I see, if you see is necessary we can try with a simpler case before
> full native bootstrap, like a non native build but with native-comp
> support.  Than we can compile a single file and load it.

I think that's basically what I'm already doing: I'm building from git
clean -xdf with native-compile, which fails with some error. Then I'm
building again, not clean. This time macroexp.eln gets loaded when
dumping, as the only one being native-compiled, and I land in
string_char_and_length's assertion (a different error than with the
clean build),

> PS I'm trying to compile the branch to give it a try :)

👍

Re: Native compilation on MPS branch

[Gerd Möllmann]

Just FYI:

LLDB 18.4.1 no longer works with my etc/emacs_lldb.py. Need to find out
what's going on first.

Re: Native compilation on MPS branch

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> I see, if you see is necessary we can try with a simpler case before
>> full native bootstrap, like a non native build but with native-comp
>> support.  Than we can compile a single file and load it.
>
> I think that's basically what I'm already doing: I'm building from git
> clean -xdf with native-compile, which fails with some error. Then I'm
> building again, not clean. This time macroexp.eln gets loaded when
> dumping, as the only one being native-compiled, and I land in
> string_char_and_length's assertion (a different error than with the
> clean build),

I see, just remember in case you need an even simpler testcase where we
load from eln a single function foo we can make it 👍

  Andrea

Re: Collecting markers with MPS

[Helmut Eller]

On Wed, Apr 24 2024, Gerd Möllmann wrote:

> Yes, it makes sense. One could of course use a timer. Or maybe, don't
> know, run something when we handle an MPS message. That just occurred to
> me a second ago, so it might be nonsense.

A variation could be this: we split the marker type up into three parts.
One part is only handled by the C code (malloc-ed, non-moving,
reference-counted), the second part is handed out to Lisp, and the third
part is specifically for the undo-list.

The Lisp part is basically a pointer to the C part, but it has a
finalizer that decrements the reference count in the C part.

The third part is also a pointer to the C part, but the C part also
keeps a (traced) reference to this part.  When the reference count drops
to zero, we can set the pointer from the undo-list to the C part so 0.
Ie. this is basically weak reference, implemented manually.

An advantage of this scheme is that the GC only needs to support
finalization.  Also, the involved changes would likely work for both the
old GC and the new GC.  Which might be an issue if the new GC remains a
purely optional feature.

Helmut

Re: Native compilation on MPS branch

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Andrea Corallo <acorallo@gnu.org> writes:
>>
>>> I see, if you see is necessary we can try with a simpler case before
>>> full native bootstrap, like a non native build but with native-comp
>>> support.  Than we can compile a single file and load it.
>>
>> I think that's basically what I'm already doing: I'm building from git
>> clean -xdf with native-compile, which fails with some error. Then I'm
>> building again, not clean. This time macroexp.eln gets loaded when
>> dumping, as the only one being native-compiled, and I land in
>> string_char_and_length's assertion (a different error than with the
>> clean build),
>
> I see, just remember in case you need an even simpler testcase where we
> load from eln a single function foo we can make it 👍

Thanks, good to know 👍

Re: Collecting markers with MPS

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Wed, Apr 24 2024, Gerd Möllmann wrote:
>
>> Yes, it makes sense. One could of course use a timer. Or maybe, don't
>> know, run something when we handle an MPS message. That just occurred to
>> me a second ago, so it might be nonsense.
>
> A variation could be this: we split the marker type up into three parts.
> One part is only handled by the C code (malloc-ed, non-moving,
> reference-counted), the second part is handed out to Lisp, and the third
> part is specifically for the undo-list.
>
> The Lisp part is basically a pointer to the C part, but it has a
> finalizer that decrements the reference count in the C part.
>
> The third part is also a pointer to the C part, but the C part also
> keeps a (traced) reference to this part.  When the reference count drops
> to zero, we can set the pointer from the undo-list to the C part so 0.
> Ie. this is basically weak reference, implemented manually.
>
> An advantage of this scheme is that the GC only needs to support
> finalization.  Also, the involved changes would likely work for both the
> old GC and the new GC.  Which might be an issue if the new GC remains a
> purely optional feature.

Hm, I must say that I like your weak-refs a _lot_ better. I mean a LOT.
(Google translation of weak ref = schwacher Schiedsrichter, BTW :-)).
And I think making a weak-ref for the old GC is also be relatively easy.

Re: Native compilation on MPS branch

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: gerd.moellmann@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 25 Apr 2024 01:48:43 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andrea Corallo <acorallo@gnu.org>
> >> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> >> Date: Wed, 24 Apr 2024 16:54:30 -0400
> >> 
> >> IME rr is invaluable for debugging GC bugs, one can do things like
> >> placing watch-points while doing reverse execution and discover
> >> immediately where something was modified the last time.  You did a very
> >> good job reaching this point of the development without rr.
> >
> > We have the last_marked[] array to facilitate debugging of GC without
> > rr.  I hope we will be able to do something similar with MPS, because
> > rr is not universally available (e.g., not on Windows).
> >
> >> Anyway I'll mention a trick I find extremely usefull to discover where
> >> something was modified last time before a certain event without rr, I
> >> apologize if you know it already but in case it might help some other
> >> reader :)
> >> 
> >> 1 When we crash we set a watch point on the memory we see got
> >> unexpectedly modified
> >> 2 We set the ignore number of the watch-point to a very large number
> >> 3 We restart the execution of the program
> >> 4 When the program crashes again we can see how many time the watch point
> >> was actually ignored
> >> 5 We subtract 1 to this number and we set it again as the ignore number
> >> of the watch-point
> >> 6 We restart again and tada!! We are where that piece of memory was
> >> modified for the last time :)
> >
> > This only works if you disable ASLR, AFAIU.
> 
> Correct, on GNU/Linux gdb does that by default, no idea on other
> systems.

It does it also on Windows, if the version of the OS is new enough to
support that.

basic questions on MPS

[Andrea Corallo]

Because I was expecting to just install MPS through my distro but I then
discovered it's not packaged, a couple of questions came to my mind
today while trying to compile it.

Do we know if MPS is distributed with any GNU/Linux distribution?

Also which are the main users (in terms of software) of MPS?

Thanks

  Andrea

Re: basic questions on MPS

[Helmut Eller]

I'd also have some MPS related questions.  Just curiosity.

1) Does somebody have experience with this MMTK [https://www.mmtk.io/]?
How does it compare to MPS?

2) When I learned that MPS uses hardware based read barriers, I was
surprised.  I had somewhere in the back of my mind some vague
memory/opinion that those are too slow to be practical.  But I actually
don't know why that should be the case.  Can somebody explain what the
problem with hardware based read barriers could be?

3) MPS uses a mostly-copying strategy.  Copying works well for
short-lived objects.  For long-lived objects it's not so great.  The big
guys (OpenJDK, V8 etc.) switch to mark-compact or something else for
older generations.  Does MPS do something special for long-lived
objects?

And I have to say that MPS looks great to me.  Especially the
documentation.

Helmut

Re: basic questions on MPS

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Thu, 25 Apr 2024 12:04:33 -0400
> 
> Because I was expecting to just install MPS through my distro but I then
> discovered it's not packaged, a couple of questions came to my mind
> today while trying to compile it.
> 
> Do we know if MPS is distributed with any GNU/Linux distribution?

No idea.  But it should be a matter of a simple search of the relevant
sites, no?

> Also which are the main users (in terms of software) of MPS?

You can find a few by searching the Internet.  I found these:

  Harlequin RIP (Raster Image Processor) -- not a surprise, since MPS
       was originally developed by Harlequin

  Open Dylan (also from Harlequin, so no surprises) offers MPS as a
       build option (https://opendylan.org/release-notes/2019.1.html)

  Clasp (https://github.com/clasp-developers/clasp), a Common Lisp
       implementation that seamlessly interoperates with C++ libraries

  Configura Magic (CM),
       https://support.configura.com/hc/en-us/sections/360008682434-CM-Language,
       an object-oriented language

Re: basic questions on MPS

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>,  Gerd Möllmann
>  <gerd.moellmann@gmail.com>
> Date: Thu, 25 Apr 2024 19:51:43 +0200
> 
> 1) Does somebody have experience with this MMTK [https://www.mmtk.io/]?

This one seems to be implemented in Rust, and I don't see any C
bindings mentioned in their docs, only Java bindings.

Re: basic questions on MPS

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Date: Thu, 25 Apr 2024 12:04:33 -0400
>> 
>> Because I was expecting to just install MPS through my distro but I then
>> discovered it's not packaged, a couple of questions came to my mind
>> today while trying to compile it.
>> 
>> Do we know if MPS is distributed with any GNU/Linux distribution?
>
> No idea.  But it should be a matter of a simple search of the relevant
> sites, no?

I could find any, that's why I asked, maybe someone has more accurate
info.

>> Also which are the main users (in terms of software) of MPS?
>
> You can find a few by searching the Internet.  I found these:
>
>   Harlequin RIP (Raster Image Processor) -- not a surprise, since MPS
>        was originally developed by Harlequin
>
>   Open Dylan (also from Harlequin, so no surprises) offers MPS as a
>        build option (https://opendylan.org/release-notes/2019.1.html)
>
>   Clasp (https://github.com/clasp-developers/clasp), a Common Lisp
>        implementation that seamlessly interoperates with C++ libraries
>
>   Configura Magic (CM),
>        https://support.configura.com/hc/en-us/sections/360008682434-CM-Language,
>        an object-oriented language

I found similar results but hoped to find some more popular production
SW relaying on it.

  Andrea

Re: basic questions on MPS

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> I'd also have some MPS related questions.  Just curiosity.
>
> 1) Does somebody have experience with this MMTK [https://www.mmtk.io/]?
> How does it compare to MPS?

Can't say much, I'm afraid. ISTR that it is relatively young, and they
said somewhere that they are not for production use yet (ca. half a year
ago). That's about what I recall. And it's Rust, if that matters.

> 2) When I learned that MPS uses hardware based read barriers, I was
> surprised.  I had somewhere in the back of my mind some vague
> memory/opinion that those are too slow to be practical.  But I actually
> don't know why that should be the case.  Can somebody explain what the
> problem with hardware based read barriers could be?

I pass.

> 3) MPS uses a mostly-copying strategy.  Copying works well for
> short-lived objects.  For long-lived objects it's not so great.  The big
> guys (OpenJDK, V8 etc.) switch to mark-compact or something else for
> older generations.  Does MPS do something special for long-lived
> objects?

I haven't seen something mentioned in the docs.

Re: basic questions on MPS

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> I found similar results but hoped to find some more popular production
> SW relaying on it.

Maybe you could ask Ravenbrook Ltd?

Re: basic questions on MPS

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: emacs-devel@gnu.org,  gerd.moellmann@gmail.com
> Date: Thu, 25 Apr 2024 14:53:28 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Also which are the main users (in terms of software) of MPS?
> >
> > You can find a few by searching the Internet.  I found these:
> >
> >   Harlequin RIP (Raster Image Processor) -- not a surprise, since MPS
> >        was originally developed by Harlequin
> >
> >   Open Dylan (also from Harlequin, so no surprises) offers MPS as a
> >        build option (https://opendylan.org/release-notes/2019.1.html)
> >
> >   Clasp (https://github.com/clasp-developers/clasp), a Common Lisp
> >        implementation that seamlessly interoperates with C++ libraries
> >
> >   Configura Magic (CM),
> >        https://support.configura.com/hc/en-us/sections/360008682434-CM-Language,
> >        an object-oriented language
> 
> I found similar results but hoped to find some more popular production
> SW relaying on it.

Well, Emacs is popular...

Re: basic questions on MPS

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: emacs-devel@gnu.org,  gerd.moellmann@gmail.com
>> Date: Thu, 25 Apr 2024 14:53:28 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Also which are the main users (in terms of software) of MPS?
>> >
>> > You can find a few by searching the Internet.  I found these:
>> >
>> >   Harlequin RIP (Raster Image Processor) -- not a surprise, since MPS
>> >        was originally developed by Harlequin
>> >
>> >   Open Dylan (also from Harlequin, so no surprises) offers MPS as a
>> >        build option (https://opendylan.org/release-notes/2019.1.html)
>> >
>> >   Clasp (https://github.com/clasp-developers/clasp), a Common Lisp
>> >        implementation that seamlessly interoperates with C++ libraries
>> >
>> >   Configura Magic (CM),
>> >        https://support.configura.com/hc/en-us/sections/360008682434-CM-Language,
>> >        an object-oriented language
>> 
>> I found similar results but hoped to find some more popular production
>> SW relaying on it.
>
> Well, Emacs is popular...

Absolutely, and I'm not saying that having no popular SW relying on it
alone would be no go.  Emacs was the first popular user of libgccjit as
well and I think I played a role with that :)

I'm just just trying to gather info so we have a better picture to
reason on.

  Andrea

Re: basic questions on MPS

[Vibhav Pant]

On Thu, Apr 25, 2024 at 11:23 PM Helmut Eller <eller.helmut@gmail.com> wrote:
>
> I'd also have some MPS related questions.  Just curiosity.
>
> 1) Does somebody have experience with this MMTK [https://www.mmtk.io/]?
> How does it compare to MPS?
>

From my understanding, MMTk is a *framework* for writing a memory
manager + an API for allocating and working with allocated memory. A
language runtime being "ported" to MMTk seems to involve these two
things:
* The language's VM and memory management model needs to be thoroughly
known to MMTk. It should know how objects addresses in the language
look like, where and how their metadata gets stored and loaded, how
copy semantics are implemented, etc
(https://docs.rs/mmtk/latest/mmtk/vm/trait.VMBinding.html).
* The language runtime calls *into* MMTk's language-agnostic memory
management API for allocating memory, setting metadata, attach
finalizers, etc. Even stuff like pointer arithmetic needs to be done
through MMTk, as the language runtime code cannot make any assumptions
about the memory model.

Once that's done, a program can choose which GC algorithm it would
like to use during runtime
(https://docs.mmtk.io/api/mmtk/util/options/enum.PlanSelector.html).
MMTk/mmtk-core by itself does not "support" any languages, as one goal
of the project seems to be to provide a way to write garbage
collection algorithms for an abstract virtual machine.

Perhaps not the same thing, but a similar idea is LLVM's support for
GC (https://llvm.org/docs/GarbageCollection.html), as LLVM IR code
generated by a compiler in such a case uses special instrinsics for
fiddling with pointer metadata, declaring write and load barriers,
etc, with the basic idea being that LLVM knows everything about
managed objects in the language runtime. You can then write a garbage
collector as an LLVM plugin, which theoretically can be loaded at
runtime.

Vibhav
---
Vibhav Pant
vibhavp@gmail.com
GPG: 7ED1 D48C 513C A024 BE3A  785F E3FB 28CB 6AB5 9598

Re: basic questions on MPS

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> I found similar results but hoped to find some more popular production
>> SW relaying on it.
>
> Maybe you could ask Ravenbrook Ltd?

Good point I'll drop them a mail and report.

  Andrea

Re: basic questions on MPS

[Helmut Eller]

On Fri, Apr 26 2024, Vibhav Pant wrote:

> From my understanding, MMTk is a *framework* for writing a memory
> manager + an API for allocating and working with allocated memory. A
> language runtime being "ported" to MMTk seems to involve these two
> things:
> * The language's VM and memory management model needs to be thoroughly
> known to MMTk. It should know how objects addresses in the language
> look like, where and how their metadata gets stored and loaded, how
> copy semantics are implemented, etc
> (https://docs.rs/mmtk/latest/mmtk/vm/trait.VMBinding.html).
> * The language runtime calls *into* MMTk's language-agnostic memory
> management API for allocating memory, setting metadata, attach
> finalizers, etc. Even stuff like pointer arithmetic needs to be done
> through MMTk, as the language runtime code cannot make any assumptions
> about the memory model.
>
> Once that's done, a program can choose which GC algorithm it would
> like to use during runtime
> (https://docs.mmtk.io/api/mmtk/util/options/enum.PlanSelector.html).
> MMTk/mmtk-core by itself does not "support" any languages, as one goal
> of the project seems to be to provide a way to write garbage
> collection algorithms for an abstract virtual machine.

Thank you for the explanation.  It's interesting that they still have
acceptable performance with so many layers of abstraction.  And I guess
it's nice that everything is strongly typed instead of a bunch macros in
a header file.

> Perhaps not the same thing, but a similar idea is LLVM's support for
> GC (https://llvm.org/docs/GarbageCollection.html), as LLVM IR code
> generated by a compiler in such a case uses special instrinsics for
> fiddling with pointer metadata, declaring write and load barriers,
> etc, with the basic idea being that LLVM knows everything about
> managed objects in the language runtime. You can then write a garbage
> collector as an LLVM plugin, which theoretically can be loaded at
> runtime.

I think this is a rather theoretical feature.  In practice, LLVM has the
same problem as GCC: they are designed for C/C++ and tracking pointers,
as needed for a precise GC, is just not part of the design.

Helmut

Re: basic questions on MPS

[Gerd Möllmann]

Vibhav Pant <vibhavp@gmail.com> writes:

> On Thu, Apr 25, 2024 at 11:23 PM Helmut Eller <eller.helmut@gmail.com> wrote:
>>
>> I'd also have some MPS related questions.  Just curiosity.
>>
>> 1) Does somebody have experience with this MMTK [https://www.mmtk.io/]?
>> How does it compare to MPS?
>>
>
> From my understanding, MMTk is a *framework* for writing a memory
> manager + an API for allocating and working with allocated memory. A
> language runtime being "ported" to MMTk seems to involve these two
> things:
> * The language's VM and memory management model needs to be thoroughly
> known to MMTk. It should know how objects addresses in the language
> look like, where and how their metadata gets stored and loaded, how
> copy semantics are implemented, etc
> (https://docs.rs/mmtk/latest/mmtk/vm/trait.VMBinding.html).
> * The language runtime calls *into* MMTk's language-agnostic memory
> management API for allocating memory, setting metadata, attach
> finalizers, etc. Even stuff like pointer arithmetic needs to be done
> through MMTk, as the language runtime code cannot make any assumptions
> about the memory model.

That sounds expensive, to say the least, both performance-wise and
implementation effort involved to make Emacs use the MMTk API. If it
could technically be done (Rust <-> C?).

> Once that's done, a program can choose which GC algorithm it would
> like to use during runtime
> (https://docs.mmtk.io/api/mmtk/util/options/enum.PlanSelector.html).
> MMTk/mmtk-core by itself does not "support" any languages, as one goal
> of the project seems to be to provide a way to write garbage
> collection algorithms for an abstract virtual machine.

Thanks for the overview!

> Perhaps not the same thing, but a similar idea is LLVM's support for
> GC (https://llvm.org/docs/GarbageCollection.html), as LLVM IR code
> generated by a compiler in such a case uses special instrinsics for
> fiddling with pointer metadata, declaring write and load barriers,
> etc, with the basic idea being that LLVM knows everything about
> managed objects in the language runtime. You can then write a garbage
> collector as an LLVM plugin, which theoretically can be loaded at
> runtime.

Yes, I've seen that too. Very interesting, but I guess it's out of reach
for GNU.

Re: basic questions on MPS

[Gerd Möllmann]

Helmut Eller <eller.helmut@gmail.com> writes:

> And I guess it's nice that everything is strongly typed instead of a
> bunch macros in a header file.

C-u 1000 M-x thumbs-up for the type safety!

Re: Native compilation on MPS branch

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: gerd.moellmann@gmail.com,  emacs-devel@gnu.org
>> Date: Thu, 25 Apr 2024 01:48:43 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Andrea Corallo <acorallo@gnu.org>
>> >> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> >> Date: Wed, 24 Apr 2024 16:54:30 -0400
>> >> 
>> >> IME rr is invaluable for debugging GC bugs, one can do things like
>> >> placing watch-points while doing reverse execution and discover
>> >> immediately where something was modified the last time.  You did a very
>> >> good job reaching this point of the development without rr.
>> >
>> > We have the last_marked[] array to facilitate debugging of GC without
>> > rr.  I hope we will be able to do something similar with MPS, because
>> > rr is not universally available (e.g., not on Windows).
>> >
>> >> Anyway I'll mention a trick I find extremely usefull to discover where
>> >> something was modified last time before a certain event without rr, I
>> >> apologize if you know it already but in case it might help some other
>> >> reader :)
>> >> 
>> >> 1 When we crash we set a watch point on the memory we see got
>> >> unexpectedly modified
>> >> 2 We set the ignore number of the watch-point to a very large number
>> >> 3 We restart the execution of the program
>> >> 4 When the program crashes again we can see how many time the watch point
>> >> was actually ignored
>> >> 5 We subtract 1 to this number and we set it again as the ignore number
>> >> of the watch-point
>> >> 6 We restart again and tada!! We are where that piece of memory was
>> >> modified for the last time :)
>> >
>> > This only works if you disable ASLR, AFAIU.
>> 
>> Correct, on GNU/Linux gdb does that by default, no idea on other
>> systems.
>
> It does it also on Windows, if the version of the OS is new enough to
> support that.

LLDB does it too on macOS.

I'm BTW one step further, in that emacs_lldb.py works unchanged with
LLVM/HEAD. So, they apparently have broken something in in 18. 

And I'm now using the fixes that Eli and Helmut made. I thought it
didn't matter to change the baseline because I got a new clang with LLVM
too, so how much more could I possibly rock the boat?

Result is that the same scenario as before (clean build, then build with
macroexp native-compiled) leads to a different outcode

Loading bindings (source)...
Loading window (source)...
Loading /Users/gerd/emacs/savannah/igc/lisp/files.el (source)...
Loading emacs-lisp/macroexp (native compiled elisp)...
Loading /Users/gerd/emacs/savannah/igc/lisp/cus-face.el (source)...
Loading /Users/gerd/emacs/savannah/igc/lisp/faces.el (source)...
Loading /Users/gerd/emacs/savannah/igc/lisp/ldefs-boot.el (source)...
Loading /Users/gerd/emacs/savannah/igc/lisp/button.el (source)...
Loading /Users/gerd/emacs/savannah/igc/lisp/emacs-lisp/cl-preloaded.el (source)...

./lisp.h:1777: Emacs fatal error: assertion failed: VECTORLIKEP (a)
Fatal error 6: Aborted

Just for info.

Re: basic questions on MPS

[vibhavp]

On Fri, 2024-04-26 at 08:36 +0200, Helmut Eller wrote:
> 
> Thank you for the explanation.  It's interesting that they still have
> acceptable performance with so many layers of abstraction.  And I
> guess
> it's nice that everything is strongly typed instead of a bunch macros
> in
> a header file.
> 
> 
Indeed, which is why the MMTk documentation recommends that final
production builds be compiled with LTO enabled. If everything's getting
built using LLVM, then cross-language LTO (Rust <-> C) also gets
utilized, ideally.

> > Perhaps not the same thing, but a similar idea is LLVM's support
> > for
> > GC (https://llvm.org/docs/GarbageCollection.html), as LLVM IR code
> > generated by a compiler in such a case uses special instrinsics for
> > fiddling with pointer metadata, declaring write and load barriers,
> > etc, with the basic idea being that LLVM knows everything about
> > managed objects in the language runtime. You can then write a
> > garbage
> > collector as an LLVM plugin, which theoretically can be loaded at
> > runtime.
> 
> I think this is a rather theoretical feature.  In practice, LLVM has
> the
> same problem as GCC: they are designed for C/C++ and tracking
> pointers,
> as needed for a precise GC, is just not part of the design.

Pointer tracking in LLVM is still WIP from what I understand, but the
documentation for statepoint tracking intrinsics
(https://llvm.org/docs/Statepoints.html) does claim that "these
mechanisms are well proven with commercial java implementation with a
fully relocating collector having shipped using them." (I believe
they're referring to Azul).

Vibhav

Re: basic questions on MPS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Can somebody explain what the
  > problem with hardware based read barriers could be?

What is a "read barrier"?

  > 3) MPS uses a mostly-copying strategy.  Copying works well for
  > short-lived objects.  For long-lived objects it's not so great.  The big
  > guys (OpenJDK, V8 etc.) switch to mark-compact or something else for
  > older generations.

Could you explain that statement a little?  "Older generations" of
what?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: basic questions on MPS

[Helmut Eller]

On Fri, Apr 26 2024, Richard Stallman wrote:

>   >   Can somebody explain what the
>   > problem with hardware based read barriers could be?
>
> What is a "read barrier"?

The definition from the MPS documentation is this:

 A read barrier is a block on reading from certain memory locations by
 certain threads or processes. Read barriers are used for incremental or
 concurrent garbage collection.

For a GC algorithm it is often useful to know the relatively small
subset of objects that are accessed (or written to) in a certain time
interval.  Memory barriers are the mechanism to identify this subset; at
least approximately.

This can be implemented with hardware or software (or a combination).
Hardware barriers use the MMU via syscalls.  Software barriers are a
data structure that must be updated on every read (or write) operation.
This requires compiler assistance or disciplined programmers.

>   > 3) MPS uses a mostly-copying strategy.  Copying works well for
>   > short-lived objects.  For long-lived objects it's not so great.  The big
>   > guys (OpenJDK, V8 etc.) switch to mark-compact or something else for
>   > older generations.
>
> Could you explain that statement a little?  "Older generations" of
> what?

A generational GC algorithm partitions objects by age.  The subsets are
called generations.  New objects are allocated in the first generation.
After a while, they are moved the second generation or, if they are no
longer referenced, they are reclaimed.  Those from the second generation
are moved to the third and so on.  The same idea can be expressed with
more colorful words: objects are born in the youngest generation, they
live for a while, they are promoted to the next older generation until
they die.

Generational GCs focus the attention on the young generation.  The older
generations are scanned much less frequently.

A copying GC identifies live objects, copies them to a new region and
those that stay behind are reclaimed.  This works extremely well for the
first generation because many objects are only needed temporarily and
don't need to be copied to the second generation.

However, some objects are moved to the older generations.  E.g. a data
structure that starts small but accumulates useful stuff incrementally.
Over time it will be come bigger and older.  Now, copying this big data
structure around for eternity could be problematic.  It would be better
just to keep it in place.  That's what the mark-compact algorithm does.

Helmut

Re: basic questions on MPS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks for the explanation?

  > This can be implemented with hardware or software (or a combination).
  > Hardware barriers use the MMU via syscalls.

Do the CPUs we use nowadays have hardware features for that?
What aout the CPUs in the newest non=backdoored machines, such
as the Thinkpad t800 and x200s?

                                                 Software barriers are a
  > data structure that must be updated on every read (or write) operation.
  > This requires compiler assistance or disciplined programmers.

That sounds rather slow.  But maybe you have practical experience to
measure it.  Can GCC insert this cod eautomatically?  Or does Emacs do
it manually?  Either way, how much slowdown does it actually cause?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: basic questions on MPS

[Helmut Eller]

On Sun, Apr 28 2024, Richard Stallman wrote:

> Do the CPUs we use nowadays have hardware features for that?
> What aout the CPUs in the newest non=backdoored machines, such
> as the Thinkpad t800 and x200s?

Yes, of course.  An MMU is needed to implement virtual memory.  Without
MMUs, operating systems like Unix would not be possible. 

>
>                                                  Software barriers are a
>   > data structure that must be updated on every read (or write) operation.
>   > This requires compiler assistance or disciplined programmers.
>
> That sounds rather slow.  But maybe you have practical experience to
> measure it.  Can GCC insert this cod eautomatically?  Or does Emacs do
> it manually?  Either way, how much slowdown does it actually cause?

There are clever ways to do it.  Like putting part of the data structure
in the lower bits of pointers or if an object is accessed in a loop, the
update only needs to be done once.  It's more practical for higher level
languages, like Java or Haskell, that don't expose addresses directly to
programmers.  For GCC and Emacs (with MPS) a hardware based solution is
probably better.

I'm no expert on this matter.  That's why I asked the question in the
first place.

Helmut

Re: basic questions on MPS

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Do the CPUs we use nowadays have hardware features for that?
  > > What aout the CPUs in the newest non=backdoored machines, such
  > > as the Thinkpad t800 and x200s?

  > Yes, of course.  An MMU is needed to implement virtual memory.  Without
  > MMUs, operating systems like Unix would not be possible. 

Thanks.  I didn't know that an ordinary MMU, designed for kernels
to do paging, would also have the features for an application program
to set up its own read barriers.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)