bug#71292: 30.0.50; macOS: .elns have no debug info

bug#71292: 30.0.50; macOS: .elns have no debug info

[Gerd Möllmann]

On macOS, executables (including .dylibs (=.eln)) never not contain
debug info themselves. Debug info is instead contained in .o files, and
executables reference the .o files used to build them in a debug map.
This can be seen with, e.g.

  dsymutil --dump-debug-map

This means no .o files, no debug info.

libgccjit deletes the .o files if debug > 0, gcc-14 -g does not delete
them, the .o files are kept in some temporary directory.

I found that the following change makes libgccjit behave like gcc:

modified   lisp/emacs-lisp/comp.el
@@ -99,7 +99,7 @@ native-comp-compiler-options
   :version "28.1")

 (defcustom native-comp-driver-options
-  (cond ((eq system-type 'darwin) '("-Wl,-w"))
+  (cond ((eq system-type 'darwin) '("-Wl,-w" "-save-temps=obj"))
         ((eq system-type 'cygwin) '("-Wl,-dynamicbase")))
   "Options passed verbatim to the native compiler's back-end driver.

This does not address the generation of .dSYM bundles. The .o files in
the temp dirs will be deleted at some point by which the debug info of
an executable is lost. This can be prevented by producing a .dSYM
bundle for the executable with

  dsymutil <executable>

and keeping the .dSYM around with the executable. .dSYM bundles can be
loaded in LLDB with

  process symbols add <dsym>.

To produce .dSYMs for native lisp:

  #! /usr/bin/env zsh
  rm -rf native-lisp/**/*.dSYM
  for f in native-lisp/**/*.eln; do
    dsymutil $f
  done

I've made jit@gcc.gnu.org aware of this. Maybe they will change
something in the future.

Example in LLDB with native-comp-debug == 2

  frame #7: 0x000000010350db68 comp-7672a6ed-3a3b5dd7.eln`F636f6d702d2d6c696d706c6966792d626c6f636b_comp__limplify_block_0 + 1632 at comp-7672a6ed-3a3b5dd7.c:101440

and one can look at the C file.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Fri, 31 May 2024 15:55:43 +0200
> 
> Example in LLDB with native-comp-debug == 2
> 
>   frame #7: 0x000000010350db68 comp-7672a6ed-3a3b5dd7.eln`F636f6d702d2d6c696d706c6966792d626c6f636b_comp__limplify_block_0 + 1632 at comp-7672a6ed-3a3b5dd7.c:101440
> 
> and one can look at the C file.

But the C file is also generally unavailable, and not only on macOS.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Date: Fri, 31 May 2024 15:55:43 +0200
>> 
>> Example in LLDB with native-comp-debug == 2
>> 
>>   frame #7: 0x000000010350db68 comp-7672a6ed-3a3b5dd7.eln`F636f6d702d2d6c696d706c6966792d626c6f636b_comp__limplify_block_0 + 1632 at comp-7672a6ed-3a3b5dd7.c:101440
>> 
>> and one can look at the C file.
>
> But the C file is also generally unavailable, and not only on macOS.

The C file is generated by libgccjit when setting native-comp-debug to 2.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 71292@debbugs.gnu.org
> Date: Fri, 31 May 2024 18:31:35 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> >> Date: Fri, 31 May 2024 15:55:43 +0200
> >> 
> >> Example in LLDB with native-comp-debug == 2
> >> 
> >>   frame #7: 0x000000010350db68 comp-7672a6ed-3a3b5dd7.eln`F636f6d702d2d6c696d706c6966792d626c6f636b_comp__limplify_block_0 + 1632 at comp-7672a6ed-3a3b5dd7.c:101440
> >> 
> >> and one can look at the C file.
> >
> > But the C file is also generally unavailable, and not only on macOS.
> 
> The C file is generated by libgccjit when setting native-comp-debug to 2.

Then how about using -keep-temps as part of native-comp-debug = 3?

bug#71292: 30.0.50; macOS: .elns have no debug info

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: 71292@debbugs.gnu.org
>> Date: Fri, 31 May 2024 18:31:35 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> >> Date: Fri, 31 May 2024 15:55:43 +0200
>> >> 
>> >> Example in LLDB with native-comp-debug == 2
>> >> 
>> >>   frame #7: 0x000000010350db68 comp-7672a6ed-3a3b5dd7.eln`F636f6d702d2d6c696d706c6966792d626c6f636b_comp__limplify_block_0 + 1632 at comp-7672a6ed-3a3b5dd7.c:101440
>> >> 
>> >> and one can look at the C file.
>> >
>> > But the C file is also generally unavailable, and not only on macOS.
>> 
>> The C file is generated by libgccjit when setting native-comp-debug to 2.
>
> Then how about using -keep-temps as part of native-comp-debug = 3?

I think you mean debug >= 2?

Anyway, debug info is also valuable when you don't have the C file, in
the disassembly. That is for debug > 0. Otherwise you see only bare hex
addresses and such.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 71292@debbugs.gnu.org
> Date: Fri, 31 May 2024 20:12:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then how about using -keep-temps as part of native-comp-debug = 3?
> 
> I think you mean debug >= 2?

Mo, I meant > 2.

> Anyway, debug info is also valuable when you don't have the C file, in
> the disassembly. That is for debug > 0. Otherwise you see only bare hex
> addresses and such.

Debug info and keeping the *.o files is not the same.  Keeping the *.o
files is really gross, which is why I suggested debug > 2 (as it keeps
other intermediate files).

Doesn't macOS support separate debug info, which can be created with
objcopy?  If it does, you could copy the debug info from the *.o
files, and then deleting them would not hamper debugging.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: 71292@debbugs.gnu.org
>> Date: Fri, 31 May 2024 20:12:45 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Then how about using -keep-temps as part of native-comp-debug = 3?
>> 
>> I think you mean debug >= 2?
>
> Mo, I meant > 2.

No .o files = no debug info on macOS. So you are proposing that one
can get debug info on macOS only with debug == 3, which produces a host
of files that are not useful at all. That makes no sense to me.

>> Anyway, debug info is also valuable when you don't have the C file, in
>> the disassembly. That is for debug > 0. Otherwise you see only bare hex
>> addresses and such.
>
> Debug info and keeping the *.o files is not the same.  Keeping the *.o
> files is really gross, which is why I suggested debug > 2 (as it keeps
> other intermediate files).

Keeping the .o files is normal on macOS. Gcc-14 -g does that, for
example. Clang produces produce bundles directly, if one wants. GCC
doesn't have an option for doing that.

> Doesn't macOS support separate debug info, which can be created with
> objcopy?  If it does, you could copy the debug info from the *.o
> files, and then deleting them would not hamper debugging.

MacOS has .dSYM bundles, but how would you convince libgccjit/gcc to
invoke dsymutil to produce them before it deletes the .o files? I don't
see how that can be done.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 71292@debbugs.gnu.org
> Date: Fri, 31 May 2024 20:52:16 +0200
> 
> > Doesn't macOS support separate debug info, which can be created with
> > objcopy?  If it does, you could copy the debug info from the *.o
> > files, and then deleting them would not hamper debugging.
> 
> MacOS has .dSYM bundles, but how would you convince libgccjit/gcc to
> invoke dsymutil to produce them before it deletes the .o files? I don't
> see how that can be done.

So let me be sure I understand: there's no was on macOS to produce a
shared library with debug info, except by keeping the *.o files
around?  IOW, when the linker produces a shared library, it doesn't
copy the debug info from the *.o files to the shared library, not even
given some optional command-line switch?

bug#71292: 30.0.50; macOS: .elns have no debug info

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: 71292@debbugs.gnu.org
>> Date: Fri, 31 May 2024 20:52:16 +0200
>> 
>> > Doesn't macOS support separate debug info, which can be created with
>> > objcopy?  If it does, you could copy the debug info from the *.o
>> > files, and then deleting them would not hamper debugging.
>> 
>> MacOS has .dSYM bundles, but how would you convince libgccjit/gcc to
>> invoke dsymutil to produce them before it deletes the .o files? I don't
>> see how that can be done.
>
> So let me be sure I understand: there's no was on macOS to produce a
> shared library with debug info, except by keeping the *.o files
> around?  IOW, when the linker produces a shared library, it doesn't
> copy the debug info from the *.o files to the shared library, not even
> given some optional command-line switch?

That's correct.

An executables on macOS cannot contain debug info as part of its own
file. It contains only a "debug map" referencing the .o files from which
it was built, and these contain the debug info.

Alternatively, if the .o files are not present, or have changed, tools
like debuggers can use .dSYM bundles which are produced by dsymutil from
the .o files in an executable's debug map when it was built. It's acting
like a sort of debug info linker in that case.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>> Cc: 71292@debbugs.gnu.org
>>> Date: Fri, 31 May 2024 20:52:16 +0200
>>> 
>>> > Doesn't macOS support separate debug info, which can be created with
>>> > objcopy?  If it does, you could copy the debug info from the *.o
>>> > files, and then deleting them would not hamper debugging.
>>> 
>>> MacOS has .dSYM bundles, but how would you convince libgccjit/gcc to
>>> invoke dsymutil to produce them before it deletes the .o files? I don't
>>> see how that can be done.
>>
>> So let me be sure I understand: there's no was on macOS to produce a
>> shared library with debug info, except by keeping the *.o files
>> around?  IOW, when the linker produces a shared library, it doesn't
>> copy the debug info from the *.o files to the shared library, not even
>> given some optional command-line switch?
>
> That's correct.
>
> An executables on macOS cannot contain debug info as part of its own
> file. It contains only a "debug map" referencing the .o files from which
> it was built, and these contain the debug info.
>
> Alternatively, if the .o files are not present, or have changed, tools
> like debuggers can use .dSYM bundles which are produced by dsymutil from
> the .o files in an executable's debug map when it was built. It's acting
> like a sort of debug info linker in that case.

Just to mention it...

When debugging a native-compiled function (speed = 1) an macOS/arm64, I
find that the debug info that libgccjit 14.1 generates is apparently
incorrect.

For example, I see car_safe being called with some argument, but when I
look in the caller in the generated C file and print what it purportedly
passes as an argumebnt, it's something completely different.

That is of course pretty useless.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> When debugging a native-compiled function (speed = 1) an macOS/arm64, I
> find that the debug info that libgccjit 14.1 generates is apparently
> incorrect.

Or let me rather say that debug info and code generated don't agree. It
could of course also be a code generation problem.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 71292@debbugs.gnu.org
> Date: Sat, 01 Jun 2024 08:15:39 +0200
> 
> When debugging a native-compiled function (speed = 1) an macOS/arm64, I
> find that the debug info that libgccjit 14.1 generates is apparently
> incorrect.
> 
> For example, I see car_safe being called with some argument, but when I
> look in the caller in the generated C file and print what it purportedly
> passes as an argumebnt, it's something completely different.

This should be reported to the GCC and libgccjit folks.  The most
probable cause is incorrect DWARF debug info emitted by the compiler.
Alternatively, it could be that LLDB interprets the debug info
incorrectly.  I sometimes see similar problems when debugging Emacs,
especially in optimized versions: arguments in function calls are
shown in reverse order, which of course is bogus, since the code
works as expected.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: 71292@debbugs.gnu.org
>> Date: Sat, 01 Jun 2024 08:15:39 +0200
>> 
>> When debugging a native-compiled function (speed = 1) an macOS/arm64, I
>> find that the debug info that libgccjit 14.1 generates is apparently
>> incorrect.
>> 
>> For example, I see car_safe being called with some argument, but when I
>> look in the caller in the generated C file and print what it purportedly
>> passes as an argumebnt, it's something completely different.
>
> This should be reported to the GCC and libgccjit folks.  The most
> probable cause is incorrect DWARF debug info emitted by the compiler.
> Alternatively, it could be that LLDB interprets the debug info
> incorrectly.  I sometimes see similar problems when debugging Emacs,
> especially in optimized versions: arguments in function calls are
> shown in reverse order, which of course is bogus, since the code
> works as expected.

After reading the GCC "how to report a bug" advice, I don't think I can
produce something useful for them in any reasonable time frame.

The jit people will also not be of great help I'm afraid. There seems to
be no one in the know using macOS :-).

But there have been some arm64 related commits in GCC during the last
weeks, so there's hope.

BTW, I'm now using the attached change in my Emacs, which allows me to
set native-comp-debug, -speed, and -driver-options via environment
variables. Maybe that's something for Andrea (CC'd).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: comp settings --]
[-- Type: text/x-patch, Size: 2847 bytes --]

From ec514e5e81034f5524bba552c56f7f8caa9fff71 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Gerd=20M=C3=B6llmann?= <gerd@gnu.org>
Date: Sun, 2 Jun 2024 07:25:19 +0200
Subject: [PATCH] Allow overriding comp settings for speed, debug, driver

---
 lisp/emacs-lisp/comp.el | 41 ++++++++++++++++++++++++++++++-----------
 1 file changed, 30 insertions(+), 11 deletions(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 4c76f95a0e9..2cf9b43dddb 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -3585,17 +3585,7 @@ native-compile
 return the compiled function."
   (comp--native-compile function-or-file nil output))
 
-;;;###autoload
-(defun batch-native-compile (&optional for-tarball)
-  "Perform batch native compilation of remaining command-line arguments.
-
-Native compilation equivalent of `batch-byte-compile'.
-Use this from the command line, with `-batch'; it won't work
-in an interactive Emacs session.
-Optional argument FOR-TARBALL non-nil means the file being compiled
-as part of building the source tarball, in which case the .eln file
-will be placed under the native-lisp/ directory (actually, in the
-last directory in `native-comp-eln-load-path')."
+(defun batch-native-compile-1 (&optional for-tarball)
   (comp-ensure-native-compiler)
   (let ((comp-running-batch-compilation t)
         (native-compile-target-directory
@@ -3610,6 +3600,35 @@ batch-native-compile
              else
              collect (byte-compile-file file))))
 
+;;;###autoload
+(cl-defun batch-native-compile (&optional for-tarball)
+  "Perform batch native compilation of remaining command-line arguments.
+
+Native compilation equivalent of `batch-byte-compile'.
+Use this from the command line, with `-batch'; it won't work
+in an interactive Emacs session.
+Optional argument FOR-TARBALL non-nil means the file being compiled
+as part of building the source tarball, in which case the .eln file
+will be placed under the native-lisp/ directory (actually, in the
+last directory in `native-comp-eln-load-path')."
+  (cl-macrolet
+      ((with-env ((&rest vars) &body body)
+         (let (clauses)
+           (cl-with-gensyms (val)
+             (dolist (var vars)
+	       (let ((env (string-replace
+		           "-" "_"
+                           (upcase (symbol-name var)))))
+	         (push `(,var (let ((,val (getenv ,env)))
+			        (if ,val
+			            (car (read-from-string ,val))
+			          ,var)))
+		       clauses))))
+           `(let (,@(nreverse clauses)) ,@body))))
+    (with-env (native-comp-speed native-comp-debug
+                                 native-comp-driver-options)
+      (batch-native-compile-1 for-tarball))))
+
 ;; In use by elisp-mode.el
 (defun comp--write-bytecode-file (eln-file)
   "After native compilation write the bytecode file for ELN-FILE.
-- 
2.45.2

bug#71292: 30.0.50; macOS: .elns have no debug info

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 71292@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>
> Date: Sun, 02 Jun 2024 07:41:13 +0200
> 
> > This should be reported to the GCC and libgccjit folks.  The most
> > probable cause is incorrect DWARF debug info emitted by the compiler.
> > Alternatively, it could be that LLDB interprets the debug info
> > incorrectly.  I sometimes see similar problems when debugging Emacs,
> > especially in optimized versions: arguments in function calls are
> > shown in reverse order, which of course is bogus, since the code
> > works as expected.
> 
> After reading the GCC "how to report a bug" advice, I don't think I can
> produce something useful for them in any reasonable time frame.
> 
> The jit people will also not be of great help I'm afraid. There seems to
> be no one in the know using macOS :-).

I think you should still report that, as best as you can and could
afford to.  It is way better than keeping the information here or to
yourself.  Maybe someone will pick up the gauntlet.  Or not.

> BTW, I'm now using the attached change in my Emacs, which allows me to
> set native-comp-debug, -speed, and -driver-options via environment
> variables. Maybe that's something for Andrea (CC'd).

In general, I don't like using environment variables for this, because
they are passed to sub-processes as well, something you don't
necessarily want.

bug#71292: 30.0.50; macOS: .elns have no debug info

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: 71292@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>
>> Date: Sun, 02 Jun 2024 07:41:13 +0200
>> 
>> > This should be reported to the GCC and libgccjit folks.  The most
>> > probable cause is incorrect DWARF debug info emitted by the compiler.
>> > Alternatively, it could be that LLDB interprets the debug info
>> > incorrectly.  I sometimes see similar problems when debugging Emacs,
>> > especially in optimized versions: arguments in function calls are
>> > shown in reverse order, which of course is bogus, since the code
>> > works as expected.
>> 
>> After reading the GCC "how to report a bug" advice, I don't think I can
>> produce something useful for them in any reasonable time frame.
>> 
>> The jit people will also not be of great help I'm afraid. There seems to
>> be no one in the know using macOS :-).
>
> I think you should still report that, as best as you can and could
> afford to.  It is way better than keeping the information here or to
> yourself.  Maybe someone will pick up the gauntlet.  Or not.

+1

>> BTW, I'm now using the attached change in my Emacs, which allows me to
>> set native-comp-debug, -speed, and -driver-options via environment
>> variables. Maybe that's something for Andrea (CC'd).
>
> In general, I don't like using environment variables for this, because
> they are passed to sub-processes as well, something you don't
> necessarily want.

+1 as well

  Andrea