Unexec dumping results in "Segmentation fault" on Windows Msys2

Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

Hello. Whenever I try to build emacs-27 branch using msys2 
--with-dumping=unexec I get:

make[2]: *** [Makefile:296: 
/d/Emacs/source/repo/lisp/emacs-lisp/macroexp.elc] Segmentation fault
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:296: 
/d/Emacs/source/repo/lisp/emacs-lisp/byte-opt.elc] Segmentation fault
make[2]: *** [Makefile:296: 
/d/Emacs/source/repo/lisp/emacs-lisp/autoload.elc] Segmentation fault
make[2]: *** [Makefile:296: 
/d/Emacs/source/repo/lisp/emacs-lisp/cconv.elc] Segmentation fault
make[2]: *** [Makefile:296: 
/d/Emacs/source/repo/lisp/emacs-lisp/bytecomp.elc] Segmentation fault

This only happens with the repository version. I've tried "make 
extraclean" and other clean targets, they did not help.

With the source archive it's different in that Emacs compiles fine, but 
it immediately crashes silently when you run it. That's actually the 
other problem I'm trying to debug, since I'm trying to find a way to get 
Emacs 26 to compile on Msys2 again and I'm observing the same 
Segmentation faults for any unexec Emacs versions from the repo. So this 
seems like a problem with some newer Msys2 package.

Could not test the master branch since trying to build it results in:

D:/Emacs/source/repo/src/pdumper.c: In function 'thaw_hash_tables':
D:/Emacs/source/repo/src/pdumper.c:5472:30: error: 'pdumper_hashes' 
undeclared (first use in this function); did you mean 'pdumper_hook'?
  5472 |   Lisp_Object hash_tables = *pdumper_hashes;
       |                              ^~~~~~~~~~~~~~
       |                              pdumper_hook

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Date: Sat, 3 Apr 2021 23:20:17 +0300
> 
> Hello. Whenever I try to build emacs-27 branch using msys2 
> --with-dumping=unexec I get:
> 
> make[2]: *** [Makefile:296: 
> /d/Emacs/source/repo/lisp/emacs-lisp/macroexp.elc] Segmentation fault
> make[2]: *** Waiting for unfinished jobs....
> make[2]: *** [Makefile:296: 
> /d/Emacs/source/repo/lisp/emacs-lisp/byte-opt.elc] Segmentation fault
> make[2]: *** [Makefile:296: 
> /d/Emacs/source/repo/lisp/emacs-lisp/autoload.elc] Segmentation fault
> make[2]: *** [Makefile:296: 
> /d/Emacs/source/repo/lisp/emacs-lisp/cconv.elc] Segmentation fault
> make[2]: *** [Makefile:296: 
> /d/Emacs/source/repo/lisp/emacs-lisp/bytecomp.elc] Segmentation fault
> 
> This only happens with the repository version. I've tried "make 
> extraclean" and other clean targets, they did not help.
> 
> With the source archive it's different in that Emacs compiles fine, but 
> it immediately crashes silently when you run it. That's actually the 
> other problem I'm trying to debug, since I'm trying to find a way to get 
> Emacs 26 to compile on Msys2 again and I'm observing the same 
> Segmentation faults for any unexec Emacs versions from the repo. So this 
> seems like a problem with some newer Msys2 package.

The only reasonable way of debugging this kind of problems is to run
the crashing command under GDB, and when it crashes look for the
reason(s) using the debugger.  It's impossible to investigate the
problem just by describing the crashes.

> Could not test the master branch since trying to build it results in:

Is this with unexec or with pdumper?  If the latter, I just fixed that
on master.

P.S. Why are you trying to build the unexec version of Emacs?  It is
basically unmaintained, left to bitrot, so you are likely to meet
problems no one is especially interested in fixing.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> Date: Sun, 04 Apr 2021 10:11:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Could not test the master branch since trying to build it results in:
> 
> Is this with unexec or with pdumper?  If the latter, I just fixed that
> on master.                            ^^^^^^^^^^^^^

Sorry, it should have been "If the former".  There's no problem I
could spot with the pdumper build.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

Yeah, I know about GDB, but was thinking that someone with more 
experience than me at debugging such issues would be able to 
independently confirm this. I'll try GDB eventually and report my findings.

The second crash happened for unexec, it seems like (some) pdumper code 
was getting built there. I can confirm that your commit fixed unexec at 
least to the point that I'm getting the same segfault now. Another issue 
with master I just ran into is:

D:/Emacs/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: 
image.o:image.c:(.text+0x7307): undefined reference to 
`rsvg_handle_set_stylesheet'
D:/Emacs/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: 
image.o:image.c:(.text+0x73d7): undefined reference to 
`rsvg_handle_set_stylesheet'

Doing --without-rsvg works. The current msys2 rsvg version is 
librsvg-2.50.3-1. Looks like possible fallout from #44065, since a 
similar result was reported there.

As for why am I even bothering with unexec - I started at noticing that 
compiling 26.3 no longer works with the newer Msys2 versions, though I 
was sure able to compile it before. Then I eventually tracked that it's 
unexec that's the problem and the reason 27 works is due to pdumper.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 4 Apr 2021 11:41:33 +0300
> 
> Yeah, I know about GDB, but was thinking that someone with more 
> experience than me at debugging such issues would be able to 
> independently confirm this. I'll try GDB eventually and report my findings.

It's very inefficient (read: impractical) to debug this kind of
problems remotely.  And I wouldn't hold my breath expecting people to
build the unexec version.

> The second crash happened for unexec, it seems like (some) pdumper code 
> was getting built there. I can confirm that your commit fixed unexec at 
> least to the point that I'm getting the same segfault now. Another issue 
> with master I just ran into is:
> 
> D:/Emacs/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: 
> image.o:image.c:(.text+0x7307): undefined reference to 
> `rsvg_handle_set_stylesheet'
> D:/Emacs/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: 
> image.o:image.c:(.text+0x73d7): undefined reference to 
> `rsvg_handle_set_stylesheet'

This should be fixed now.  Alan, please remember the WINDOWSNT aspect
when you add calls to RSVG functions never used before.  We had this
discussion in the past, I believe.

> Doing --without-rsvg works. The current msys2 rsvg version is 
> librsvg-2.50.3-1. Looks like possible fallout from #44065, since a 
> similar result was reported there.

No, because that change was never committed.  It's a new problem,
since yesterday.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

Here's what I was able to gather in regards to this issue.

Whenever I'm trying to build master I get:
dumped_data_commit: memory exhausted.
Enlarge dumped_data[]!

Due to this I have to set DUMPED_HEAP_SIZE (24*1024*1024). Just 
mentioning this for completeness sake, that's not the issue here.

Segfaults are triggered by msys2 binutils version 2.36. If we try to 
debug the segfault with GDB we get this:

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007ff7c7514862 in main (argc=9, argv=0x1f815ad1860)
     at D:/Emacs/source/repo/src/emacs.c:960
960       stack_bottom = (char *) &stack_bottom_variable;

It seems like the initial bootstrap-emacs.exe is so broken that it fails 
at even the simplest things. The question I keep asking myself is, if we 
assume that it's our build environment that has some problem, why is 
unexec the only place that is harmed by it? Would it be possible to come 
up with some isolated test case that would demonstrate the problem? Then 
maybe it can be submitted to msys2 or mingw64 developers.

Now, lets downgrade binutils to version 2.35(or older), you can do that 
by grabbing the archive from here( https://repo.msys2.org/mingw/x86_64/ 
) and then calling "pacman -U  
mingw-w64-x86_64-binutils-2.35.1-3-any.pkg.tar.zst" on it. Emacs 
versions 27.2, 27.1, 26.3 and older would compile fine. But the master 
still won't. Instead of a segfault we get a more helpful Emacs crash. If 
we attach gdb to it, here's what we get:

#10 0x00007ff911ef0a9e in ntdll!KiUserExceptionDispatcher ()
    from /c/WINDOWS/SYSTEM32/ntdll.dll
#11 0x00007ff9103c43d7 in msvcrt!memmove () from 
/c/WINDOWS/System32/msvcrt.dll
#12 0x0000000400191e25 in insert_1_both (
     string=0x4506840 "(fn FILENAME)\377\377\377", nchars=13, nbytes=13,
     inherit=false, prepare=true, before_markers=false)
     at D:/Emacs/source/repo/src/insdel.c:915
#13 0x000000040024f689 in Fprin1_to_string (object=..., noescape=...)
     at D:/Emacs/source/repo/src/print.c:685
#14 0x000000040020be9a in styled_format (nargs=2, args=0xbf0720, 
message=false)
     at D:/Emacs/source/repo/src/editfns.c:3322
#15 0x000000040020b69f in Fformat (nargs=2, args=0xbf0720)
     at D:/Emacs/source/repo/src/editfns.c:3059
#16 0x000000040021b946 in eval_sub (form=...)
     at D:/Emacs/source/repo/src/eval.c:2363
#17 0x000000040021dbd0 in apply_lambda (fun=..., args=..., count=228)
     at D:/Emacs/source/repo/src/eval.c:3056

Again, memory related. Since we know that unexec works in emacs26 and 
emacs27 branches, I went for another row of bisecting and traced the 
offending commit to cddf85d256. Now this is interesting in that if 
unexec triggers a crash here, is there a way to get this to crash Emacs 
during the normal usage? Also, I have an old msys2 backup from 2017 that 
I've used for testing and I'm getting the same kind of exceptions with 
it, so I don't think we can write this master branch issue off on the 
build environment.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 15 Apr 2021 01:11:53 +0300
> 
> Segfaults are triggered by msys2 binutils version 2.36. If we try to 
> debug the segfault with GDB we get this:
> 
> Thread 1 received signal SIGSEGV, Segmentation fault.
> 0x00007ff7c7514862 in main (argc=9, argv=0x1f815ad1860)
>      at D:/Emacs/source/repo/src/emacs.c:960
> 960       stack_bottom = (char *) &stack_bottom_variable;

This is very strange.  Is this in temacs.exe or in the dumped
emacs.exe?  What happens when you start temacs from the directory
where it was built, like this:

  temacs -Q

Does it crash then?

What does this show in GDB at the point of the crash?

  (gdb) p &stack_bottom_variable

> It seems like the initial bootstrap-emacs.exe is so broken that it fails 
> at even the simplest things. The question I keep asking myself is, if we 
> assume that it's our build environment that has some problem, why is 
> unexec the only place that is harmed by it?

Because unexec produces a new binary executable from the memory of a
running Emacs process, and evidently that executable is broken in
fundamental ways, for some reason.  The mere fact that using an older
version of Binutils produces different results points to some change
in the assembler/linker area.  Did MSYS2 folks change anything in how
MinGW64 Binutils are configured? e.g., what about LTO usage?

> #10 0x00007ff911ef0a9e in ntdll!KiUserExceptionDispatcher ()
>     from /c/WINDOWS/SYSTEM32/ntdll.dll
> #11 0x00007ff9103c43d7 in msvcrt!memmove () from 
> /c/WINDOWS/System32/msvcrt.dll
> #12 0x0000000400191e25 in insert_1_both (
>      string=0x4506840 "(fn FILENAME)\377\377\377", nchars=13, nbytes=13,
>      inherit=false, prepare=true, before_markers=false)
>      at D:/Emacs/source/repo/src/insdel.c:915

You don't show enough data to come up with ideas.  All I can say is
that insert_1_both tried to access memory in some invalid way.  The
source line is this:

  memcpy (GPT_ADDR, string, nbytes);

which is intended to insert text of the STRING argument (13 bytes of
it) into the gap.  Why this segfaults I have no idea.  You didn't
event show the entire C backtrace, so I don't know if this is the
original crash or a secondary one, which happened when processing the
original exception.  I also don't understand why we see msvcrt!memmove
in the backtrace: Emacs calls memcpy, not memmove, and on my system if
I put a breakpoint at that source line and step into the call, I find
myself in msvcrt!memcpy, as expected.  Maybe it's something that
MinGW64 runtime or your version of GCC do differently, I don't know.

> #13 0x000000040024f689 in Fprin1_to_string (object=..., noescape=...)
>      at D:/Emacs/source/repo/src/print.c:685
> #14 0x000000040020be9a in styled_format (nargs=2, args=0xbf0720, 
> message=false)
>      at D:/Emacs/source/repo/src/editfns.c:3322
> #15 0x000000040020b69f in Fformat (nargs=2, args=0xbf0720)
>      at D:/Emacs/source/repo/src/editfns.c:3059
> #16 0x000000040021b946 in eval_sub (form=...)
>      at D:/Emacs/source/repo/src/eval.c:2363
> #17 0x000000040021dbd0 in apply_lambda (fun=..., args=..., count=228)
>      at D:/Emacs/source/repo/src/eval.c:3056
> 
> Again, memory related. Since we know that unexec works in emacs26 and 
> emacs27 branches, I went for another row of bisecting and traced the 
> offending commit to cddf85d256. Now this is interesting in that if 
> unexec triggers a crash here, is there a way to get this to crash Emacs 
> during the normal usage? Also, I have an old msys2 backup from 2017 that 
> I've used for testing and I'm getting the same kind of exceptions with 
> it, so I don't think we can write this master branch issue off on the 
> build environment.

This is not an efficient method of investigating the problem, IME.
Bisecting is not going to help you unless you find a change that is
simple and localized enough to give the "eureka!" moment, and the one
you found isn't.

The way to debug this is to use the debugger and try to understand
what exactly causes the crash.  For example, in the above case, what's
wrong with the memcpy call? is GPT_ADDR invalid, per chance?  More
generally, what did Emacs try to do when it crashed?  The full
backtrace would help us understand that; it could be that the real
problem is elsewhere, way up the callstack, and this is just the
fallout.

IOW, you need to actively debug the problem where it happens and find
the root cause of the crashes.  Only then we can start thinking about
which change broke it and how to repair it.

You could also keep teasing me until I find the time to debug this
myself, but I don't promise this will happen any time soon, given what
I have on my plate.  And even if I find the time, there's no guarantee
I will see the problem: I use a different version of GCC (9.2.0) a
different runtime and headers (mingw.org's MinGW, not MinGW64), and I
build my own Binutils from sources, configuring them as I see fit,
which is different from what the MSYS2 folks do.

Sorry I couldn't be of more help.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Camm Maguire]

Greetings!  This is just a general note to observe that gcl and its
compiled products (maxima, acl2, axiom, fricas, hol88...) have been
using and fixing unexec for years across multiple platforms.  Its now
stable for us across the debian universe.  It would
be great if we could synchronize and avoid duplication of effort with
emacs.  (I don't have much time for this myself, but could port to an
unexec library, etc.)

Take care,
-- 
Camm Maguire			     		    camm@maguirefamily.org
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Camm Maguire <camm@maguirefamily.org>
> Date: Thu, 15 Apr 2021 09:07:40 -0400
> Cc: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>, emacs-devel@gnu.org
> 
> Greetings!  This is just a general note to observe that gcl and its
> compiled products (maxima, acl2, axiom, fricas, hol88...) have been
> using and fixing unexec for years across multiple platforms.  Its now
> stable for us across the debian universe.  It would
> be great if we could synchronize and avoid duplication of effort with
> emacs.  (I don't have much time for this myself, but could port to an
> unexec library, etc.)

Emacs moves away of unexec since v27.1, and we will probably deprecate
and remove the code in some future release.

Thanks.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

Binutils triggered segfault:

$ temacs.exe -Q
Warning: Lisp directory 
'd:/Emacs/configurations/msys2-brake-test/src/lisp': No such file or 
directory
Cannot open load file: No such file or directory, loadup.el

(gdb) p &stack_bottom_variable
$1 = (void **) 0xbd3cbff6c8

Emacs configure logs using binutils 2.35 and 2.36 both have --enable-lto 
and nothing in them seems to be any different between each other...

Master crash:

I went for bisecting hoping to find that simple localized change. It was 
not my intention to tease you, at least not since the second letter. ;-) 
I can grab as many full backtraces or do other tests as needed. It's 
just that the GDB session looked completely fine to me in my admittedly 
limited understanding of low level Emacs internals. The crash happens 
when bootstrap-emacs is doing dumping(same place as the other problem). 
Here's a couple backtraces:

https://gist.githubusercontent.com/sg2002/32ea64634a89e7b407f50e29b4ab5f7e/raw/aa0bcc517e1a4de733a9dc2678f8c45daefb95f9/gistfile1.txt

https://gist.githubusercontent.com/sg2002/d9cfaf1268973b20d66d79f5df575498/raw/06091019c1d30b863e6000fb642e09316d82a344/gistfile1.txt

As for whether this is even repeatable with the original MinGW, I'd say 
probably not, but I'm going to test that and report results later.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 15 Apr 2021 18:47:31 +0300
> 
> Binutils triggered segfault:
> 
> $ temacs.exe -Q
> Warning: Lisp directory 
> 'd:/Emacs/configurations/msys2-brake-test/src/lisp': No such file or 
> directory
> Cannot open load file: No such file or directory, loadup.el

Does the directory d:/Emacs/configurations/msys2-brake-test/src/lisp
exist?  If so, please try running temacs.exe from cmd.exe, not from
MSYS2 Bash.  If that fails with the same error message, then please
step into the call to file_accessible_directory_p in this function
(from lread.c):

  static void
  load_path_check (Lisp_Object lpath)
  {
    Lisp_Object path_tail;

    /* The only elements that might not exist are those from
       PATH_LOADSEARCH, EMACSLOADPATH.  Anything else is only added if
       it exists.  */
    for (path_tail = lpath; !NILP (path_tail); path_tail = XCDR (path_tail))
      {
	Lisp_Object dirfile;
	dirfile = Fcar (path_tail);
	if (STRINGP (dirfile))
	  {
	    dirfile = Fdirectory_file_name (dirfile);
	    if (! file_accessible_directory_p (dirfile)) <<<<<<<<<<<<<<<<<
	      dir_warning ("Lisp directory", XCAR (path_tail));
	  }
      }
  }

and see why it fails, including the code in w32_accessible_directory_p
that it calls.

> (gdb) p &stack_bottom_variable
> $1 = (void **) 0xbd3cbff6c8

Nothing wrong with that.  And what do these produce:

  (gdb) p current_thread
  (gdb) p current_thread->m_stack_bottom

> I went for bisecting hoping to find that simple localized change. It was 
> not my intention to tease you, at least not since the second letter. ;-) 
> I can grab as many full backtraces or do other tests as needed. It's 
> just that the GDB session looked completely fine to me in my admittedly 
> limited understanding of low level Emacs internals. The crash happens 
> when bootstrap-emacs is doing dumping(same place as the other problem). 
> Here's a couple backtraces:
> 
> https://gist.githubusercontent.com/sg2002/32ea64634a89e7b407f50e29b4ab5f7e/raw/aa0bcc517e1a4de733a9dc2678f8c45daefb95f9/gistfile1.txt
> 
> https://gist.githubusercontent.com/sg2002/d9cfaf1268973b20d66d79f5df575498/raw/06091019c1d30b863e6000fb642e09316d82a344/gistfile1.txt

These look like some kind of infinite recursion that causes stack
overflow.  So the final segfault is not interesting; what's
interesting is why there's infinite recursion.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

Folder d:/Emacs/configurations/msys2-brake-test/src/lisp does not exist.

(gdb) p current_thread
$1 = (struct thread_state *) 0xffe868d4df20
(gdb) p current_thread->m_stack_bottom
Cannot access memory at address 0xffe868d4df68

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 15 Apr 2021 22:17:30 +0300
> 
> Folder d:/Emacs/configurations/msys2-brake-test/src/lisp does not exist.

Then please put a breakpoint in load_path_check, where it shows this
message, and when it breaks, show the backtrace and the arguments
(including the value of 'lpath', the argument).

> (gdb) p current_thread
> $1 = (struct thread_state *) 0xffe868d4df20
> (gdb) p current_thread->m_stack_bottom
> Cannot access memory at address 0xffe868d4df68

Is this Emacs 28 or Emacs 27?  Built with Binutils 2.36 or 2.35?

What are the values of current_thread in temacs and in the dumped
emacs?  Here:

  gdb ./temacs.exe
  ...
  (gdb) start -Q
  (gdb) p current_thread
  (gdb) q

  gdb ./emacs.exe
  ...
  (gdb) start -Q
  (gdb) p current_thread
  (gdb) q

Are the values the same?  And if you start temacs several times like
the above, do you always see the same values?

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

All of the following was done on the main branch, revision cddf85d256. 
With binutils 2.36, so the segfault is triggered here. I can redo all of 
this on any other revision if you want.

load_path_check:

Thread 1 hit Breakpoint 1, load_path_check (lpath=...)
     at D:/Emacs/source/repo/src/lread.c:4469
4469                dir_warning ("Lisp directory", XCAR (path_tail));
(gdb) p lpath
$1 = {i = 0x1f8b61c1b43}
(gdb) backtrace
#0  load_path_check (lpath=...) at D:/Emacs/source/repo/src/lread.c:4469
#1  0x00007ff654870215 in init_lread ()
     at D:/Emacs/source/repo/src/lread.c:4668
#2  0x00007ff6546a5a06 in main (argc=2, argv=0x1f8b62517b0)
     at D:/Emacs/source/repo/src/emacs.c:1778

Temacs p current_thread: 0x40095df20.

Emacs.exe does not exist("mv -f emacs.exe bootstrap-emacs.exe" gets done 
just before it tries to compile those elisp files and crashes), but for 
bootstrap-emacs.exe it's 0xffea01a5df20p.

I could not find any change in those values after running that a few times.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 16 Apr 2021 19:57:57 +0300
> 
> Temacs p current_thread: 0x40095df20.
> 
> Emacs.exe does not exist("mv -f emacs.exe bootstrap-emacs.exe" gets done 
> just before it tries to compile those elisp files and crashes), but for 
> bootstrap-emacs.exe it's 0xffea01a5df20p.

This seems to confirm my guess that ASLR is at play here.  So please
try those two linker switches I mentioned in my previous message, they
could fix the problem.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

Eli, you were correct. Msys2 developers enabled ASLR by default in 
binutils 2.36 ( 
https://www.msys2.org/news/#2021-01-31-aslr-enabled-by-default ).

I was able to avoid the segfault by adding those switches to 
LD_SWITCH_SYSTEM_TEMACS in src/Makefile. Adding them as generic LDFLAGS 
works too.

Maybe it's worth documenting this problem in nt/INSTALL.W64, but since 
unexec is kinda deprecated I'm not entirely sure...

This leaves only the main branch unexec problem. I was able to reproduce 
it with MinGW. I'll report later if my debugging results in any other 
useful info.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Date: Wed, 21 Apr 2021 19:33:11 +0300
> Cc: emacs-devel@gnu.org
> 
> Eli, you were correct. Msys2 developers enabled ASLR by default in 
> binutils 2.36 ( 
> https://www.msys2.org/news/#2021-01-31-aslr-enabled-by-default ).
> 
> I was able to avoid the segfault by adding those switches to 
> LD_SWITCH_SYSTEM_TEMACS in src/Makefile. Adding them as generic LDFLAGS 
> works too.
> 
> Maybe it's worth documenting this problem in nt/INSTALL.W64, but since 
> unexec is kinda deprecated I'm not entirely sure...

It's much easier to teach the build process to use those switches when
building for unexec, so I'm going to do that soon.

> This leaves only the main branch unexec problem.

Could you please reiterate what is the problem on master that is left?
I understand that it is not resolved by using those 2 linker switches?

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

The problem that's left is that the main branch crashes with unexec 
dumping on first lisp file compilation ever since commit cddf85d256. The 
backtraces for that I grabbed earlier:

https://gist.githubusercontent.com/sg2002/32ea64634a89e7b407f50e29b4ab5f7e/raw/aa0bcc517e1a4de733a9dc2678f8c45daefb95f9/gistfile1.txt

https://gist.githubusercontent.com/sg2002/d9cfaf1268973b20d66d79f5df575498/raw/06091019c1d30b863e6000fb642e09316d82a344/gistfile1.txt

I also was able to confirm that this bug happens with MinGW.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 21 Apr 2021 21:19:45 +0300
> 
> The problem that's left is that the main branch crashes with unexec 
> dumping on first lisp file compilation ever since commit cddf85d256. The 
> backtraces for that I grabbed earlier:
> 
> https://gist.githubusercontent.com/sg2002/32ea64634a89e7b407f50e29b4ab5f7e/raw/aa0bcc517e1a4de733a9dc2678f8c45daefb95f9/gistfile1.txt
> 
> https://gist.githubusercontent.com/sg2002/d9cfaf1268973b20d66d79f5df575498/raw/06091019c1d30b863e6000fb642e09316d82a344/gistfile1.txt

Like I said before: this looks like infinite recursion that causes
stack overflow.  A Lisp-level backtrace could help.  Does "xbacktrace"
report anything interesting?  If not, the way to get the Lisp
backtrace is to go to each frame that calls Ffuncall and do this:

  (gdb) p args[0]
  (gdb) xsymbol

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

I now know what's going on there.

buffer.c/init_buffer has some unexec-specific code that remaps memory 
for the special buffers inherited from temacs. We fail at 
Fprin1_to_string which uses the " prin1" special buffer. cddf85d256 
removed FOR_EACH_BUFFER macro and replaced it everywhere with the use of 
FOR_EACH_LIVE_BUFFER. Because FOR_EACH_LIVE_BUFFER does not iterate over 
the " prin1" buffer, it does not get its memory remapped and this breaks 
print functionality and elisp compilation, which depends on it.

FOR_EACH_BUFFER worked before due to buffer structure operating as a 
linked list and since it's kind of an ugly way of doing things, make 
sense why Stefan removed it. But I'm not if adding those special buffers 
to Vbuffer_alist(FOR_EACH_LIVE_BUFFER uses it) would not break anything.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 29 Apr 2021 22:17:10 +0300
> 
> I now know what's going on there.
> 
> buffer.c/init_buffer has some unexec-specific code that remaps memory 
> for the special buffers inherited from temacs. We fail at 
> Fprin1_to_string which uses the " prin1" special buffer. cddf85d256 
> removed FOR_EACH_BUFFER macro and replaced it everywhere with the use of 
> FOR_EACH_LIVE_BUFFER. Because FOR_EACH_LIVE_BUFFER does not iterate over 
> the " prin1" buffer, it does not get its memory remapped and this breaks 
> print functionality and elisp compilation, which depends on it.
> 
> FOR_EACH_BUFFER worked before due to buffer structure operating as a 
> linked list and since it's kind of an ugly way of doing things, make 
> sense why Stefan removed it. But I'm not if adding those special buffers 
> to Vbuffer_alist(FOR_EACH_LIVE_BUFFER uses it) would not break anything.

Thanks.  The " prin1" buffer is not in Vbuffer_alist intentionally.  I
installed a possible fix for this on master (100% untested), please
see if it fixes the problem.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

Thank you. It finally works.

While testing I had another problem related to pdumper stuff getting 
used without being defined. So I wrote a trivial patch for it. But then 
I retested with a clean bootstrap and learned that it normally only 
gives a warning for it:

make[2]: Entering directory '/d/Emacs/configurations/master-unexec/lisp'
   ELC      /d/Emacs/source/repo/lisp/emacs-lisp/bytecomp.elc

In end of data:
D:/Emacs/source/repo/lisp/emacs-lisp/bytecomp.el:5314:39: Warning: the
     function `pdumper-stats' is not known to be defined.

Anyway, I've attached that patch, feel free to apply or not apply it as 
you see fit, since it only removes that one warning, and since unexec is 
deprecated anyway...


[-- Attachment #2: 0001-Don-t-use-pdumper-stats-with-unexec.patch --]
[-- Type: text/plain, Size: 1160 bytes --]

From 9abf8228f55894d62621edc1e5467467e667e1bc Mon Sep 17 00:00:00 2001
From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
Date: Sat, 1 May 2021 15:27:21 +0300
Subject: [PATCH] Don't use pdumper-stats with unexec

* lisp/emacs-lisp/bytecomp.el (byte-compile-refresh-preloaded): Check if
pdumper-stats is bound before using it.
---
 lisp/emacs-lisp/bytecomp.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index 9be54ca4f5..e93cee9924 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -5311,7 +5311,8 @@ byte-compile-refresh-preloaded
   "Reload any Lisp file that was changed since Emacs was dumped.
 Use with caution."
   (let* ((argv0 (car command-line-args))
-         (emacs-file (or (cdr (nth 2 (pdumper-stats)))
+         (emacs-file (or (and (fboundp 'pdumper-stats)
+                              (cdr (nth 2 (pdumper-stats))))
                          (executable-find argv0))))
     (if (not (and emacs-file (file-exists-p emacs-file)))
         (message "Can't find %s to refresh preloaded Lisp files" argv0)
-- 
2.31.1.windows.1

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 2 May 2021 12:43:16 +0300
> 
> While testing I had another problem related to pdumper stuff getting 
> used without being defined. So I wrote a trivial patch for it. But then 
> I retested with a clean bootstrap and learned that it normally only 
> gives a warning for it:
> 
> make[2]: Entering directory '/d/Emacs/configurations/master-unexec/lisp'
>    ELC      /d/Emacs/source/repo/lisp/emacs-lisp/bytecomp.elc
> 
> In end of data:
> D:/Emacs/source/repo/lisp/emacs-lisp/bytecomp.el:5314:39: Warning: the
>      function `pdumper-stats' is not known to be defined.
> 
> Anyway, I've attached that patch, feel free to apply or not apply it as 
> you see fit, since it only removes that one warning, and since unexec is 
> deprecated anyway...

Thanks, I installed that.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> Date: Wed, 21 Apr 2021 20:41:03 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> > Date: Wed, 21 Apr 2021 19:33:11 +0300
> > Cc: emacs-devel@gnu.org
> > 
> > Eli, you were correct. Msys2 developers enabled ASLR by default in 
> > binutils 2.36 ( 
> > https://www.msys2.org/news/#2021-01-31-aslr-enabled-by-default ).
> > 
> > I was able to avoid the segfault by adding those switches to 
> > LD_SWITCH_SYSTEM_TEMACS in src/Makefile. Adding them as generic LDFLAGS 
> > works too.
> > 
> > Maybe it's worth documenting this problem in nt/INSTALL.W64, but since 
> > unexec is kinda deprecated I'm not entirely sure...
> 
> It's much easier to teach the build process to use those switches when
> building for unexec, so I'm going to do that soon.

Now done on the master branch, please test.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

This breaks 32 bit mingw64 build due to ld complaining:

D:/Emacs/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/10.2.0/../../../../i686-w64-mingw32/bin/ld.exe: 
unrecognised option: -disable-high-entropy-va

So you also have to check whether we're doing a 64 bit build or not.

Also, while the Msys2 website is suggesting we use those particular 
forms, ld warns us:

D:/Emacs/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/10.2.0/../../../../i686-w64-mingw32/bin/ld.exe: 
Warning: grouped short command line options are deprecated: 
-disable-high-entropy-va

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 22 Apr 2021 21:59:39 +0300
> 
> This breaks 32 bit mingw64 build due to ld complaining:
> 
> D:/Emacs/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/10.2.0/../../../../i686-w64-mingw32/bin/ld.exe: 
> unrecognised option: -disable-high-entropy-va

Thanks, should be fixed now.

> Also, while the Msys2 website is suggesting we use those particular 
> forms, ld warns us:
> 
> D:/Emacs/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/10.2.0/../../../../i686-w64-mingw32/bin/ld.exe: 
> Warning: grouped short command line options are deprecated: 
> -disable-high-entropy-va

I don't understand this warning.  And since this is a 32-bit linker,
the same one that said this option is not recognized, I'm guessing
this is because it didn't recognize it, i.e. not a real warning.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

Yeah, you're probably right, since you actually use the longer form. 
I'll retest and report if I get this warning again.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 15 Apr 2021 22:17:30 +0300
> 
> Folder d:/Emacs/configurations/msys2-brake-test/src/lisp does not exist.
> 
> (gdb) p current_thread
> $1 = (struct thread_state *) 0xffe868d4df20
> (gdb) p current_thread->m_stack_bottom
> Cannot access memory at address 0xffe868d4df68

A stub in the dark: add the following switches to the temacs link
command line, then rebuild temacs, redump emacs, and see if the
problems go away:

    -Wl,-disable-high-entropy-va -Wl,-disable-reloc-section

These disable the Windows ASLR, which could interfere with the
assumption that addresses that are recorded at dump time stay the same
when the dumped Emacs is restarted.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 15 Apr 2021 18:47:31 +0300
> 
> Emacs configure logs using binutils 2.35 and 2.36 both have --enable-lto 
> and nothing in them seems to be any different between each other...

What does config.log say about link-time-optimizations?  If it enables
them, I suggest to try configuring with --disable-link-time-optimization.

Re: Unexec dumping results in "Segmentation fault" on Windows Msys2

[Nikolay Kudryavtsev]

Link time optimizations are not mentioned in the config, I've tried 
explicitly disabling them just in case, no change.