bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

[Eli Zaretskii <eliz@gnu.org>]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, 46256@debbugs.gnu.org,
>         andrewjmoreton@gmail.com
> Date: Thu, 04 Mar 2021 14:49:47 +0000
> 
> I've a reproducer that is most luckily due to the same issue you are
> observing:
> 
> emacs -batch -l comp -f batch-native-compile .../emacs/lisp/progmodes/cc-engine.el
> 
> GC kicks-in and we end-up marking #<subr c-string-list-p>, we try then
> to mark its compilation unit but we segfault (backtrace below).

AFAICT, the crash I see here, while compiling subr-x.el, is not inside
GC: gc_in_progress is zero when Emacs crashes.

To make debugging easier, I started Emacs like this:

  emacs -batch -l comp -f batch-byte-native-compile-for-bootstrap ../lisp/emacs-lisp/subr-x.el

(AFAIU, using batch-byte-native-compile-for-bootstrap is currently the
only way of invoking the native compilation in the same Emacs process,
not in an async subprocess, is that right?)

It crashes inside comp--compile-ctxt-to-file, and when it does, the C
stack seems to be smashed:

  Thread 1 received signal SIGSEGV, Segmentation fault.
  0x06acac3e in ?? ()
  (gdb) bt
  #0  0x06acac3e in ?? ()
  #1  0x00010101 in ?? ()
  Backtrace stopped: previous frame inner to this frame (corrupt stack?)

  Lisp Backtrace:
  "comp--compile-ctxt-to-file" (0x82ca78)
  "comp-compile-ctxt-to-file" (0x82cc88)
  "comp-final1" (0x82cfb0)
  "comp-final" (0x82d238)
  "comp--native-compile" (0x82d468)
  "batch-native-compile" (0x82d6a0)
  "batch-byte-native-compile-for-bootstrap" (0x82d908)
  "command-line-1" (0x82e360)
  "command-line" (0x82ef08)
  "normal-top-level" (0x82f630)

I then put a breakpoint in comp--compile-ctxt-to-file and stepped
through it.  This behaves erratically: if I just step with "next", it
seems to crash inside the call to gcc_jit_context_set_int_option,
here:

  gcc_jit_context_set_int_option (comp.ctxt,
				  GCC_JIT_INT_OPTION_OPTIMIZATION_LEVEL,
				  comp.speed < 0 ? 0
				  : (comp.speed > 3 ? 3 : comp.speed));

But if I "stepi" inside gcc_jit_int_option_optimization_level, it
somehow seems to return, proceeds to this line:

  gcc_jit_context_compile_to_file (comp.ctxt,
				   GCC_JIT_OUTPUT_KIND_DYNAMIC_LIBRARY,
				   SSDATA (tmp_file));

Then something goes wrong inside it: the backtrace shows bogus
addresses (note the "0xbaadf00d" thingies):

  0x0766f180 in ?? ()
  (gdb) bt
  #0  0x0766f180 in ?? ()
  #1  0x672f756e in ?? ()
  #2  0x652f7469 in ?? ()
  #3  0x7363616d in ?? ()
  #4  0x74616e2f in ?? ()
  #5  0x2d657669 in ?? ()
  #6  0x706d6f63 in ?? ()
  #7  0x74616e2f in ?? ()
  #8  0x2d657669 in ?? ()
  #9  0x7073696c in ?? ()
  #10 0x2e38322f in ?? ()
  #11 0x30352e30 in ?? ()
  #12 0x3238312d in ?? ()
  #13 0x65306335 in ?? ()
  #14 0x75732f32 in ?? ()
  #15 0x782d7262 in ?? ()
  #16 0x6432302d in ?? ()
  #17 0x33666566 in ?? ()
  #18 0x37312d32 in ?? ()
  #19 0x62656166 in ?? ()
  #20 0x47736431 in ?? ()
  #21 0x47305561 in ?? ()
  #22 0x6e6c652e in ?? ()
  #23 0x706d742e in ?? ()
  #24 0xbaadf000 in ?? ()
  #25 0xbaadf00d in ?? ()
  #26 0xbaadf00d in ?? ()
  #27 0xbaadf00d in ?? ()
  #28 0xbaadf00d in ?? ()
  #29 0xbaadf00d in ?? ()
  #30 0xbaadf00d in ?? ()
  #31 0xbaadf00d in ?? ()
  #32 0xbaadf00d in ?? ()
  #33 0xbaadf00d in ?? ()
  #34 0xbaadf00d in ?? ()
  #35 0xbaadf00d in ?? ()
  #36 0xbaadf00d in ?? ()
  #37 0xbaadf00d in ?? ()
  #38 0xbaadf00d in ?? ()
  #39 0xbaadf00d in ?? ()
  #40 0xbaadf00d in ?? ()
  #41 0xbaadf00d in ?? ()
  #42 0xbaadf00d in ?? ()
  #43 0xbaadf00d in ?? ()
  #44 0xbaadf00d in ?? ()
  #45 0xbaadf00d in ?? ()
  #46 0xbaadf00d in ?? ()
  #47 0xbaadf00d in ?? ()
  #48 0xbaadf00d in ?? ()
  #49 0xbaadf00d in ?? ()
  #50 0xbaadf00d in ?? ()
  #51 0xbaadf00d in ?? ()
  #52 0xbaadf00d in ?? ()
  #53 0xbaadf00d in ?? ()
  #54 0xbaadf00d in ?? ()
  #55 0xbaadf00d in ?? ()
  #56 0xbaadf00d in ?? ()
  #57 0xbaadf00d in ?? ()
  #58 0xbaadf00d in ?? ()
  #59 0xbaadf00d in ?? ()
  #60 0xbaadf00d in ?? ()
  #61 0xbaadf00d in ?? ()
  #62 0xbaadf00d in ?? ()
  #63 0xbaadf00d in ?? ()
  #64 0xbaadf00d in ?? ()
  #65 0xbaadf00d in ?? ()
  #66 0xbaadf00d in ?? ()
  #67 0xbaadf00d in ?? ()
  #68 0xbaadf00d in ?? ()
  #69 0xbaadf00d in ?? ()
  #70 0xbaadf00d in ?? ()
  #71 0xbaadf00d in ?? ()
  #72 0xbaadf00d in ?? ()
  #73 0xbaadf00d in ?? ()
  #74 0xbaadf00d in ?? ()
  #75 0xbaadf00d in ?? ()
  #76 0xbaadf00d in ?? ()
  #77 0xbaadf00d in ?? ()
  #78 0xbaadf00d in ?? ()
  #79 0xbaadf00d in ?? ()
  #80 0xbaadf00d in ?? ()
  #81 0xbaadf00d in ?? ()
  #82 0xbaadf00d in ?? ()
  #83 0xbaadf00d in ?? ()
  #84 0xbaadf00d in ?? ()
  #85 0xbaadf00d in ?? ()
  #86 0xbaadf00d in ?? ()
  #87 0xbaadf00d in ?? ()
  #88 0xbaadf00d in ?? ()
  #89 0xbaadf00d in ?? ()
  #90 0xbaadf00d in ?? ()
  #91 0xbaadf00d in ?? ()
  #92 0xbaadf00d in ?? ()
  #93 0xbaadf00d in ?? ()
  #94 0xbaadf00d in ?? ()
  #95 0xbaadf00d in ?? ()
  #96 0xbaadf00d in ?? ()
  #97 0xbaadf00d in ?? ()
  #98 0xbaadf00d in ?? ()
  #99 0xbaadf00d in ?? ()
  #100 0xbaadf00d in ?? ()
  #101 0xbaadf00d in ?? ()
  #102 0xbaadf00d in ?? ()
  #103 0xbaadf00d in ?? ()
  #104 0xbaadf00d in ?? ()
  #105 0xbaadf00d in ?? ()
  #106 0xbaadf00d in ?? ()
  #107 0xbaadf00d in ?? ()
  #108 0xbaadf00d in ?? ()
  #109 0xbaadf00d in ?? ()
  #110 0xbaadf00d in ?? ()
  #111 0xbaadf00d in ?? ()
  #112 0xbaadf00d in ?? ()
  #113 0xbaadf00d in ?? ()
  #114 0xbaadf00d in ?? ()
  #115 0xbaadf00d in ?? ()
  #116 0xbaadf00d in ?? ()
  #117 0xbaadf00d in ?? ()
  #118 0xbaadf00d in ?? ()
  #119 0xbaadf00d in ?? ()
  #120 0xbaadf00d in ?? ()
  #121 0xbaadf00d in ?? ()
  #122 0xbaadf00d in ?? ()
  #123 0xbaadf00d in ?? ()
  #124 0xbaadf00d in ?? ()
  #125 0xbaadf00d in ?? ()
  #126 0xbaadf00d in ?? ()
  #127 0xbaadf00d in ?? ()
  #128 0xbaadf00d in ?? ()
  #129 0xbaadf00d in ?? ()
  #130 0xbaadf00d in ?? ()
  #131 0xbaadf00d in ?? ()
  #132 0xbaadf00d in ?? ()
  #133 0xbaadf00d in ?? ()
  #134 0xbaadf00d in ?? ()
  #135 0xbaadf00d in ?? ()
  #136 0xbaadf00d in ?? ()
  #137 0xbaadf00d in ?? ()
  #138 0xbaadf00d in ?? ()
  #139 0xbaadf00d in ?? ()
  #140 0xbaadf00d in ?? ()
  #141 0xbaadf00d in ?? ()
  #142 0xbaadf00d in ?? ()
  #143 0xbaadf00d in ?? ()
  #144 0xbaadf00d in ?? ()
  #145 0xbaadf00d in ?? ()
  #146 0xbaadf00d in ?? ()
  #147 0xbaadf00d in ?? ()
  #148 0xbaadf00d in ?? ()
  #149 0xbaadf00d in ?? ()
  #150 0xbaadf00d in ?? ()
  #151 0xbaadf00d in ?? ()
  #152 0xbaadf00d in ?? ()
  #153 0xbaadf00d in ?? ()
  #154 0xbaadf00d in ?? ()
  #155 0xbaadf00d in ?? ()
  #156 0xbaadf00d in ?? ()
  #157 0xbaadf00d in ?? ()
  #158 0xbaadf00d in ?? ()
  #159 0xbaadf00d in ?? ()
  #160 0xbaadf00d in ?? ()
  #161 0xbaadf00d in ?? ()
  #162 0xbaadf00d in ?? ()
  #163 0xbaadf00d in ?? ()
  #164 0xbaadf00d in ?? ()
  #165 0xbaadf00d in ?? ()
  #166 0xbaadf00d in ?? ()
  #167 0xbaadf00d in ?? ()
  #168 0xbaadf00d in ?? ()
  #169 0xbaadf00d in ?? ()
  #170 0xbaadf00d in ?? ()
  #171 0xbaadf00d in ?? ()
  #172 0xbaadf00d in ?? ()
  #173 0xbaadf00d in ?? ()
  #174 0xbaadf00d in ?? ()
  #175 0xbaadf00d in ?? ()
  #176 0xbaadf00d in ?? ()
  #177 0xbaadf00d in ?? ()
  #178 0xbaadf00d in ?? ()
  #179 0xbaadf00d in ?? ()
  #180 0xbaadf00d in ?? ()
  #181 0xbaadf00d in ?? ()
  #182 0xbaadf00d in ?? ()
  #183 0xbaadf00d in ?? ()
  #184 0xbaadf00d in ?? ()
  #185 0xbaadf00d in ?? ()
  #186 0xbaadf00d in ?? ()
  #187 0xbaadf00d in ?? ()
  #188 0xbaadf00d in ?? ()
  #189 0xbaadf00d in ?? ()
  #190 0xbaadf00d in ?? ()
  #191 0xbaadf00d in ?? ()
  #192 0xbaadf00d in ?? ()
  #193 0xbaadf00d in ?? ()
  #194 0xbaadf00d in ?? ()
  #195 0xbaadf00d in ?? ()
  #196 0xbaadf00d in ?? ()
  #197 0xbaadf00d in ?? ()
  #198 0xbaadf00d in ?? ()
  #199 0xbaadf00d in ?? ()
  #200 0xbaadf00d in ?? ()
  #201 0xbaadf00d in ?? ()
  #202 0xbaadf00d in ?? ()
  #203 0xbaadf00d in ?? ()
  #204 0xbaadf00d in ?? ()
  #205 0xbaadf00d in ?? ()
  #206 0xbaadf00d in ?? ()
  #207 0xbaadf00d in ?? ()
  #208 0xbaadf00d in ?? ()
  #209 0xbaadf00d in ?? ()
  #210 0xbaadf00d in ?? ()
  #211 0xbaadf00d in ?? ()
  #212 0xbaadf00d in ?? ()
  #213 0xbaadf00d in ?? ()
  #214 0xbaadf00d in ?? ()
  #215 0xbaadf00d in ?? ()
  #216 0xbaadf00d in ?? ()
  #217 0xbaadf00d in ?? ()
  #218 0xbaadf00d in ?? ()
  #219 0xbaadf00d in ?? ()
  #220 0xbaadf00d in ?? ()
  #221 0xbaadf00d in ?? ()
  #222 0xbaadf00d in ?? ()
  #223 0xbaadf00d in ?? ()
  #224 0xbaadf00d in ?? ()
  #225 0xbaadf00d in ?? ()
  #226 0xbaadf00d in ?? ()
  #227 0xbaadf00d in ?? ()
  #228 0xbaadf00d in ?? ()
  #229 0xbaadf00d in ?? ()
  #230 0xbaadf00d in ?? ()
  #231 0xbaadf00d in ?? ()
  #232 0xabababab in ?? ()
  #233 0xabababab in ?? ()
  #234 0xfeeefeee in ?? ()
  #235 0x00000000 in ?? ()

Maybe if I "stepi" inside that last libgccjit function, I will be able
to advance more.  But something is definitely fishy here, and I'm not
sure what that is.  Ideas for further debugging are welcome.