bug#65727: 30.0.50; Build failure in MSYS2 when --with-native-compilation

bug#65727: 30.0.50; Build failure in MSYS2 when --with-native-compilation

[voi dfoo]

[-- Attachment #1: Type: text/plain, Size: 2137 bytes --]

Occasionally I use appveyor/GitHub Actions to build Emacs for Windows
using MSYS environment. At some point the build started to fail when
making .elc files.

I suspect that newer version of dependencies (libgccjit?) caused the
crash because a previous passing commit now also failed.

I don't have a local environment to investigate further. I don't know
whether the following information is actionable but here they are

- The appveyor build history:
https://ci.appveyor.com/project/voidfoo/emacs-w64/history

- A previous passing AppVeyor build on top of commit 4b3de74:
  https://ci.appveyor.com/project/voidfoo/emacs-w64/builds/47147210

- GitHub action build on top of 4b3de74 that now is failing:
  https://github.com/voidfoo/emacs/actions/runs/6068869155/job/16462420999

- I tried to do a debug build and that is passing

https://github.com/voidfoo/emacs/actions/runs/6063066665/job/16449969955#step:6:3

In failing builds, it seems that Emacs crashed

  ELC      arc-mode.elc
  ELC      array.elc
Backtrace:
00007ff6afeeb38e
00007ff6afdb91c1
00007ff6afdd9d61
00007ff6aff4fafa
00007ffb47848060
00007ffb48265097
00007ffb481c4ce7
00007ffb48263e06
00007ff6afe3eb58
00007ff6afe48aea
00007ff6afe490ba
00007ffb1dfa77f1
00007ff6afe48aea
00007ff6afe490ba
00007ffb1dfa7910
00007ff6afe48aea
00007ffb1b05516f
00007ff6afe48aea
00007ffb1b0556db
00007ff6afe48aea
00007ffb1b055632
00007ff6afe48aea
00007ffb1dfb6902
00007ff6afe4c27e
00007ff6afe48aea
00007ffb1dfb59ca
00007ff6afe48aea
00007ffb1dfb24eb
00007ff6afe4c27e
00007ff6afe94818
00007ff6afe48aea
00007ffb1dfc8dcb
00007ffb1dfc901f
00007ffb1dfc93ba
00007ff6afe48aea
00007ffb1dfb0ddc
00007ff6afe48aea
00007ffb1dfb0c6e
00007ff6afe48aea
00007ffb1dfa1fbb
00007ff6afe48aea
00007ffb1dfb0d06
00007ff6afe48aea
00007ffb1dfaed7c
00007ff6afe48aea
00007ffb1dfaf79a
00007ff6afe48aea
00007ffb1dfad8c4
00007ff6afe48aea
00007ffb1dfcc9a5
00007ff6afe48aea
00007ffb1dfcc41f
00007ff6afe48aea
00007ffb19f72422
00007ff6afe48aea
00007ffb2d90f0bb
00007ff6afe48aea
00007ffb2d9072cf
00007ff6afe48aea
00007ffb2d903190
00007ff6afe4cc9a
00007ff6afe4f22a
...
make[3]: *** [Makefile:328: array.elc] Error 3


--
Voi dFoo

[-- Attachment #2: Type: text/html, Size: 2822 bytes --]

bug#65727: 30.0.50; Build failure in MSYS2 when --with-native-compilation

[Eli Zaretskii]

> From: voi dfoo <void.foo@gmail.com>
> Date: Sun, 3 Sep 2023 22:06:41 -0700
> 
> Occasionally I use appveyor/GitHub Actions to build Emacs for Windows
> using MSYS environment. At some point the build started to fail when
> making .elc files.
> 
> I suspect that newer version of dependencies (libgccjit?) caused the
> crash because a previous passing commit now also failed.
> 
> I don't have a local environment to investigate further. I don't know
> whether the following information is actionable but here they are

Did you per chance upgrade GCC and/or libgccjit and/or Binutils
between the last successful build and the first failed one?

bug#65727: 30.0.50; Build failure in MSYS2 when --with-native-compilation

[voi dfoo]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

The dependency versions are not explicitly specified in my installation
script

"pacman -S --needed --noconfirm git zip base-devel
mingw-w64-x86_64-autotools mingw-w64-x86_64-toolchain
mingw-w64-x86_64-xpm-nox mingw-w64-x86_64-libtiff mingw-w64-x86_64-giflib
mingw-w64-x86_64-libpng mingw-w64-x86_64-libjpeg-turbo
mingw-w64-x86_64-librsvg mingw-w64-x86_64-libwebp mingw-w64-x86_64-lcms2
mingw-w64-x86_64-gnutls mingw-w64-x86_64-jansson mingw-w64-x86_64-libgccjit
mingw-w64-x86_64-libxml2 mingw-w64-x86_64-gnutls mingw-w64-x86_64-zlib
mingw-w64-x86_64-harfbuzz mingw-w64-x86_64-tree-sitter
mingw-w64-x86_64-sqlite3"


There is a update of the package to gcc 13.2.0 that may be related:
https://github.com/msys2/MINGW-packages/commits/master/mingw-w64-gcc

On Mon, Sep 4, 2023, 5:29 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: voi dfoo <void.foo@gmail.com>
> > Date: Sun, 3 Sep 2023 22:06:41 -0700
> >
> > Occasionally I use appveyor/GitHub Actions to build Emacs for Windows
> > using MSYS environment. At some point the build started to fail when
> > making .elc files.
> >
> > I suspect that newer version of dependencies (libgccjit?) caused the
> > crash because a previous passing commit now also failed.
> >
> > I don't have a local environment to investigate further. I don't know
> > whether the following information is actionable but here they are
>
> Did you per chance upgrade GCC and/or libgccjit and/or Binutils
> between the last successful build and the first failed one?
>

[-- Attachment #2: Type: text/html, Size: 2130 bytes --]

bug#65727: Acknowledgement (30.0.50; Build failure in MSYS2 when --with-native-compilation)

[voi dfoo]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

I found out that it is a dupe of
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63365

Not sure why I didn't find it when searching the bug database before
reporting. This issue can be closed now.

On Mon, Sep 4, 2023, 2:20 AM GNU bug Tracking System <help-debbugs@gnu.org>
wrote:

> Thank you for filing a new bug report with debbugs.gnu.org.
>
> This is an automatically generated reply to let you know your message
> has been received.
>
> Your message is being forwarded to the package maintainers and other
> interested parties for their attention; they will reply in due course.
>
> Your message has been sent to the package maintainer(s):
>  bug-gnu-emacs@gnu.org
>
> If you wish to submit further information on this problem, please
> send it to 65727@debbugs.gnu.org.
>
> Please do not send mail to help-debbugs@gnu.org unless you wish
> to report a problem with the Bug-tracking system.
>
> --
> 65727: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=65727
> GNU Bug Tracking System
> Contact help-debbugs@gnu.org with problems
>

[-- Attachment #2: Type: text/html, Size: 1980 bytes --]

bug#65727: Acknowledgement (30.0.50; Build failure in MSYS2 when --with-native-compilation)

[Stefan Kangas]

forcemerge 63365 65727
thanks

voi dfoo <void.foo@gmail.com> writes:

> I found out that it is a dupe of
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63365

Thanks, I'm therefore merging the bugs.

bug#65727: 30.0.50; Build failure in MSYS2 when --with-native-compilation

[Cyril Arnould]

 > Interesting, would you mind instead of using 'printf' to try using
 > '__attribute__((optimize("O0")))' on the function to verify that the
 > issue is really an optimization?

With the following, the build fails:


diff --git a/src/thread.c b/src/thread.c
index 040ca39511e..ffdcf420af0 100644
--- a/src/thread.c
+++ b/src/thread.c
@@ -692,6 +692,7 @@ mark_threads_callback (void *ignore)
      }
  }

+__attribute__((optimize("O0")))
  void
  mark_threads (void)
  {


I hope this was correct? If I replace it with
'__attribute__((optimize("no-optimize-sibling-calls")))', the build
succeeds again.

I therefore tried to compile the entirety of thread.c with -O0
instead, upon which the build succeeded again. I'll try to add the
'__attribute__((optimize("O0")))' in more thread.c functions to see
which ones need it for a successful build (unless you have a better
suggestion?).

To summarize the current state:

thread.c compiled with -O2: build fails
thread.c compiled with -O2 -fno-optimize-sibling-calls: build succeeds
thread.c compiled with -O0: build succeeds

thread.c compiled with -O2 and
- mark_threads with printf: build succeeds
- mark_threads with 
__attribute__((optimize("no-optimize-sibling-calls"))): build succeeds
- mark_threads with __attribute__((optimize("O0"))): build fails

bug#65727: 30.0.50; Build failure in MSYS2 when --with-native-compilation

[Andrea Corallo]

Cyril Arnould <cyril.arnould@outlook.com> writes:

>> Interesting, would you mind instead of using 'printf' to try using
>> '__attribute__((optimize("O0")))' on the function to verify that the
>> issue is really an optimization?
>
> With the following, the build fails:
>
>
> diff --git a/src/thread.c b/src/thread.c
> index 040ca39511e..ffdcf420af0 100644
> --- a/src/thread.c
> +++ b/src/thread.c
> @@ -692,6 +692,7 @@ mark_threads_callback (void *ignore)
>      }
>  }
>
> +__attribute__((optimize("O0")))
>  void
>  mark_threads (void)
>  {
>
>
> I hope this was correct? If I replace it with
> '__attribute__((optimize("no-optimize-sibling-calls")))', the build
> succeeds again.
>
> I therefore tried to compile the entirety of thread.c with -O0
> instead, upon which the build succeeded again. I'll try to add the
> '__attribute__((optimize("O0")))' in more thread.c functions to see
> which ones need it for a successful build (unless you have a better
> suggestion?).
>
> To summarize the current state:
>
> thread.c compiled with -O2: build fails
> thread.c compiled with -O2 -fno-optimize-sibling-calls: build succeeds
> thread.c compiled with -O0: build succeeds
>
> thread.c compiled with -O2 and
> - mark_threads with printf: build succeeds
> - mark_threads with
>   __attribute__((optimize("no-optimize-sibling-calls"))): build
>  succeeds
> - mark_threads with __attribute__((optimize("O0"))): build fails

Mmmh 🧐, must say it does not make much sense to me.  Are we sure these
results are reliably reproducible and we are not looking at some
statistic fluctuation of a non very reproducible issue?

Anyway if marking 'mark_threads' with
__attribute__((optimize("no-optimize-sibling-calls"))) solves the issue
in a stable way I think we should compare the assembly output of the two
functions.

Thanks!

  Andrea

bug#65727: 30.0.50; Build failure in MSYS2 when --with-native-compilation

[Cyril Arnould]

[-- Attachment #1: Type: text/plain, Size: 8399 bytes --]

 > Mmmh 🧐, must say it does not make much sense to me.  Are we sure these
 > results are reliably reproducible and we are not looking at some
 > statistic fluctuation of a non very reproducible issue?

I understand your scepticism. However, I've ran every build 2-4 times,
and so far my reproducibility is at 100%. As in; with the same
configuration the build (as a whole) either always fails or is always
successful. It does seem random however which of the byte-compile
steps fail in the end. Maybe the -fno-optimize-sibling-calls just
makes the failure much more unlikely.

 > Anyway if marking 'mark_threads' with
 > __attribute__((optimize("no-optimize-sibling-calls"))) solves the issue
 > in a stable way I think we should compare the assembly output of the two
 > functions.

I've attached the corresponding objdump -S -d (including the original
.o files). This time I've modified thread.c so that 'mark_threads'
comes dead last. I ran the builds another 4 times each to make
sure. Moving the function in the source also moves it to the end of
the dumps, makes for easier diffing:

diff -ubBw thread.txt thread-attr-no-optimize-sibling-calls.txt
--- thread.txt  2024-05-15 17:59:49.003434900 +0200
+++ thread-attr-no-optimize-sibling-calls.txt   2024-05-15 
17:38:35.029292300 +0200
@@ -1,5 +1,5 @@

-../thread-last.o:     file format pe-x86-64
+../thread-attr-last.o:     file format pe-x86-64


  Disassembly of section .text:
@@ -2305,6 +2305,7 @@

  00000000000017d0 <mark_threads>:

+__attribute__((optimize("no-optimize-sibling-calls")))
  void
  mark_threads (void)
  {
@@ -2316,50 +2317,49 @@
      17d9:      57                      push   %rdi
      17da:      56                      push   %rsi
      17db:      53                      push   %rbx
-    17dc:      48 81 ec a8 00 00 00    sub    $0xa8,%rsp
-    17e3:      48 8d 2c 24             lea    (%rsp),%rbp
-    17e7:      0f 11 75 00             movups %xmm6,0x0(%rbp)
-    17eb:      0f 11 7d 10             movups %xmm7,0x10(%rbp)
-    17ef:      44 0f 11 45 20          movups %xmm8,0x20(%rbp)
-    17f4:      44 0f 11 4d 30          movups %xmm9,0x30(%rbp)
-    17f9:      44 0f 11 55 40          movups %xmm10,0x40(%rbp)
-    17fe:      44 0f 11 5d 50          movups %xmm11,0x50(%rbp)
-    1803:      44 0f 11 65 60          movups %xmm12,0x60(%rbp)
-    1808:      44 0f 11 6d 70          movups %xmm13,0x70(%rbp)
-    180d:      44 0f 11 b5 80 00 00    movups %xmm14,0x80(%rbp)
-    1814:      00
-    1815:      44 0f 11 bd 90 00 00    movups %xmm15,0x90(%rbp)
-    181c:      00
+    17dc:      48 81 ec c8 00 00 00    sub    $0xc8,%rsp
+    17e3:      48 8d 6c 24 20          lea    0x20(%rsp),%rbp
+    17e8:      0f 11 75 00             movups %xmm6,0x0(%rbp)
+    17ec:      0f 11 7d 10             movups %xmm7,0x10(%rbp)
+    17f0:      44 0f 11 45 20          movups %xmm8,0x20(%rbp)
+    17f5:      44 0f 11 4d 30          movups %xmm9,0x30(%rbp)
+    17fa:      44 0f 11 55 40          movups %xmm10,0x40(%rbp)
+    17ff:      44 0f 11 5d 50          movups %xmm11,0x50(%rbp)
+    1804:      44 0f 11 65 60          movups %xmm12,0x60(%rbp)
+    1809:      44 0f 11 6d 70          movups %xmm13,0x70(%rbp)
+    180e:      44 0f 11 b5 80 00 00    movups %xmm14,0x80(%rbp)
+    1815:      00
+    1816:      44 0f 11 bd 90 00 00    movups %xmm15,0x90(%rbp)
+    181d:      00
    flush_stack_call_func1 (func, arg);
-    181d:      31 d2                   xor    %edx,%edx
-    181f:      48 8d 0d da eb ff ff    lea -0x1426(%rip),%rcx        # 
400 <mark_threads_callback>
+    181e:      31 d2                   xor    %edx,%edx
+    1820:      48 8d 0d d9 eb ff ff    lea -0x1427(%rip),%rcx        # 
400 <mark_threads_callback>
+    1827:      e8 00 00 00 00          call   182c <mark_threads+0x5c>
+    182c:      90                      nop
    flush_stack_call_func (mark_threads_callback, NULL);
  }
-    1826:      0f 10 75 00             movups 0x0(%rbp),%xmm6
-    182a:      0f 10 7d 10             movups 0x10(%rbp),%xmm7
-    182e:      44 0f 10 45 20          movups 0x20(%rbp),%xmm8
-    1833:      44 0f 10 4d 30          movups 0x30(%rbp),%xmm9
-    1838:      44 0f 10 55 40          movups 0x40(%rbp),%xmm10
-    183d:      44 0f 10 5d 50          movups 0x50(%rbp),%xmm11
-    1842:      44 0f 10 65 60          movups 0x60(%rbp),%xmm12
-    1847:      44 0f 10 6d 70          movups 0x70(%rbp),%xmm13
-    184c:      44 0f 10 b5 80 00 00    movups 0x80(%rbp),%xmm14
-    1853:      00
-    1854:      44 0f 10 bd 90 00 00    movups 0x90(%rbp),%xmm15
-    185b:      00
-    185c:      48 81 c4 a8 00 00 00    add    $0xa8,%rsp
-    1863:      5b                      pop    %rbx
-    1864:      5e                      pop    %rsi
-    1865:      5f                      pop    %rdi
-    1866:      41 5c                   pop    %r12
-    1868:      41 5d                   pop    %r13
-    186a:      41 5e                   pop    %r14
-    186c:      41 5f                   pop    %r15
-    186e:      5d                      pop    %rbp
-    186f:      e9 00 00 00 00          jmp    1874 <mark_threads+0xa4>
-    1874:      90                      nop
-    1875:      90                      nop
-    1876:      90                      nop
+    182d:      0f 10 75 00             movups 0x0(%rbp),%xmm6
+    1831:      0f 10 7d 10             movups 0x10(%rbp),%xmm7
+    1835:      44 0f 10 45 20          movups 0x20(%rbp),%xmm8
+    183a:      44 0f 10 4d 30          movups 0x30(%rbp),%xmm9
+    183f:      44 0f 10 55 40          movups 0x40(%rbp),%xmm10
+    1844:      44 0f 10 5d 50          movups 0x50(%rbp),%xmm11
+    1849:      44 0f 10 65 60          movups 0x60(%rbp),%xmm12
+    184e:      44 0f 10 6d 70          movups 0x70(%rbp),%xmm13
+    1853:      44 0f 10 b5 80 00 00    movups 0x80(%rbp),%xmm14
+    185a:      00
+    185b:      44 0f 10 bd 90 00 00    movups 0x90(%rbp),%xmm15
+    1862:      00
+    1863:      48 81 c4 c8 00 00 00    add    $0xc8,%rsp
+    186a:      5b                      pop    %rbx
+    186b:      5e                      pop    %rsi
+    186c:      5f                      pop    %rdi
+    186d:      41 5c                   pop    %r12
+    186f:      41 5d                   pop    %r13
+    1871:      41 5e                   pop    %r14
+    1873:      41 5f                   pop    %r15
+    1875:      5d                      pop    %rbp
+    1876:      c3                      ret
      1877:      90                      nop
      1878:      90                      nop
      1879:      90                      nop


[-- Attachment #2: objdump.tar.gz --]
[-- Type: application/x-gzip, Size: 743990 bytes --]

bug#65727: 30.0.50; Build failure in MSYS2 when --with-native-compilation

[Cyril Arnould]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

 > If you could also share the two preprocessed versions of thread.c would
 > be of great help.  You should be able to do it adding to your original
 > GCC invokation like "-E -o thread.i".

What I ended up doing was:

cd src
make thread.o -W thread.c CFLAGS='-g3 -O2 -gdwarf-2 
-Wno-error=implicit-function-declaration -E -o thread.i'

I hope this is what you were after. The addition of
'-Wno-error=implicit-function-declaration' comes from issue #70889
btw. I've also edited the "E:/Git/emacs-debug-gcc14/emacs" paths in
the thread.i files by hand so they don't show up in the diff; they
were originally compiled in two separate folders. The resulting files
are almost exactly the same:


diff -ubBw thread.i thread-attr-no-optimize-sibling-calls.i
--- thread.i    2024-05-15 20:34:08 +0000
+++ thread-attr-no-optimize-sibling-calls.i     2024-05-15 20:33:44 +0000
@@ -152355,12 +152355,13 @@

  }

+__attribute__((optimize("no-optimize-sibling-calls")))
  void
  mark_threads (void)
  {
    flush_stack_call_func (mark_threads_callback,
-# 1181 "thread.c" 3 4
+# 1182 "thread.c" 3 4
                                                 ((void *)0)
-# 1181 "thread.c"
+# 1182 "thread.c"
                                                     );
  }


[-- Attachment #2: preproc.tar.gz --]
[-- Type: application/x-gzip, Size: 1320429 bytes --]