32-bit MinGW build with JIT

32-bit MinGW build with JIT

[Eli Zaretskii]

I tried to build the JIT branch with 32-bit MinGW tools.  Since Tom
said wide ints are not yet supported, I didn't configure with that
option.

The build failed half way through bootstrap.  Specifically,
bootstrap-emacs crashes the first time it attempts to JIT-compile a
byte-compiled function, and then invoke the resulting JIT.

The crash is in this snippet from eval.c:

#ifdef HAVE_LIBJIT
         if (initialized)
           {
             struct Lisp_Vector *vec = XVECTOR (fun);

             if (vec->contents[COMPILED_JIT_CODE] == NULL)
               emacs_jit_compile (fun);

             if (vec->contents[COMPILED_JIT_CODE] != NULL)
               return funcall_subr (fun,
                                    (struct subr_function *) vec->contents[COMPILED_JIT_CODE],
                                    nargs, arg_vector);
           }
#endif /* HAVE_LIBJIT */

funcall_subr calls the JIT function, which attempts to call Ffuncall,
and the latter segfaults because its arguments are not valid Lisp
objects.

I'm not yet sure what causes this, but since I presume that the branch
was successfully built on a 64-bit host, I suspect some snafu with
pointers that are narrower than 64-bit integers.  I couldn't yet spot
any code that would be suspect, although some portions of the code
work differently on 64-bit and 32-bit hosts, for example:

  if (sizeof (ptrdiff_t) == 8)
    ptrdiff_t_type = jit_type_ulong;
  else
    {
      eassert (sizeof (ptrdiff_t) == 4);
      ptrdiff_t_type = jit_type_uint;
    }

(This is btw wrong, I think, in a 32-bit build --with-wide-int.)

Also, I'm not sure why we are making a "long constant" (i.e., a 64-bit
data type) here:

         mandatory_val
           = jit_value_create_long_constant (func, ptrdiff_t_type, mandatory);

Why shouldn't this be of the same type as ptrdiff_t_type?

Anyway, what I wanted to tell is that as of now, it is very hard to
debug these problems.  All the libjit types are "incomplete" as far as
GDB is concerned, there are no facilities to display libjit types and
values in human-readable form, C-level backtraces show ?? when a JIT
compiled function is called, etc.  I didn't yet succeed in
establishing whether the invalid Lisp arguments for Ffuncall are
already invalid when we compile the function, or get corrupted as part
of compilation.

I think we need to upgrade our debugging facilities, including adding
commands to .gdbinit, as part of working on this branch.

Tom, if you didn't try to build on a 32-bit host, could you please try
that, so I could decide if what I report is specific to a 32-bit build
or to MS-Windows?

Thanks.

Re: 32-bit MinGW build with JIT

[Eli Zaretskii]

> Date: Thu, 16 Aug 2018 17:09:15 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Anyway, what I wanted to tell is that as of now, it is very hard to
> debug these problems.  All the libjit types are "incomplete" as far as
> GDB is concerned, there are no facilities to display libjit types and
> values in human-readable form, C-level backtraces show ?? when a JIT
> compiled function is called, etc.  I didn't yet succeed in
> establishing whether the invalid Lisp arguments for Ffuncall are
> already invalid when we compile the function, or get corrupted as part
> of compilation.
> 
> I think we need to upgrade our debugging facilities, including adding
> commands to .gdbinit, as part of working on this branch.

Oh, and one more thing: I think we need a Lisp variable to disable JIT
even if it has been compiled in.  This should be important for
debugging and testing, if nothing else.

Re: 32-bit MinGW build with JIT

[Eli Zaretskii]

Tried the same 32-bit MinGW build on another machine.  Got a similar
crash while compiling macroexp.el:

       ELC      emacs-lisp/macroexp.elc

     lisp.h:2019: Emacs fatal error: assertion failed: sym->u.s.redirect == SYMBOL_FORWARDED && sym->u.s.val.fwd

Trying to reproduce this under GDB, I don't get the assertion
violation, but sym->u.s.val.fwd looks bogus, because
do_symval_forwarding aborts:

     #0  0x7c90120f in ntdll!DbgBreakPoint () from C:\WINDOWS\system32\ntdll.dll
     #1  0x0124ae24 in emacs_abort () at w32fns.c:10775
     #2  0x011878c6 in do_symval_forwarding (
	 valcontents=0x2032a6b <dumped_data+10676555>) at data.c:1032
     #3  0x0118866e in find_symbol_value (symbol=XIL(0xa493b0)) at data.c:1274
     #4  0x011ab727 in specbind (symbol=XIL(0xa493b0), value=XIL(0xa492e0))
	 at eval.c:3362
     #5  0x060100c4 in ?? ()
     #6  0x011aa08f in funcall_subr (error_obj=XIL(0x1634b35), subr=0xfa5218,
	 numargs=5, args=0x827840) at eval.c:2909
     #7  0x011aa672 in funcall_lambda (fun=XIL(0x1634b35), nargs=5,
	 arg_vector=0x827840) at eval.c:3033

     (gdb) p sym.u.s.val.fwd
     $8 = (union Lisp_Fwd *) 0x2032a6b <dumped_data+10676555>
     (gdb) p *$
     $9 = {
       u_intfwd = {
	 type = 53117696,
	 intvar = 0x32a7b02
       },
       u_boolfwd = {
	 type = 53117696,
	 boolvar = 0x32a7b02
       },
       u_objfwd = {
	 type = 53117696,
	 objvar = 0x32a7b02
       },
       u_buffer_objfwd = {
	 type = 53117696,
	 offset = 53115650,
	 predicate = make_number(13279936)
       },
       u_kboard_objfwd = {
	 type = 53117696,
	 offset = 53115650
       }

Any ideas are welcome.

Re: 32-bit MinGW build with JIT

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli> I'm not yet sure what causes this, but since I presume that the branch
Eli> was successfully built on a 64-bit host, I suspect some snafu with
Eli> pointers that are narrower than 64-bit integers.

Thanks for trying this.  I'm very ill so I haven't been able to look
into this code.  I will when I'm able.

Also I'm about 1/4 of the way through converting it to use gcc-jit, so
we can see how well that works.

Eli> All the libjit types are "incomplete" as far as
Eli> GDB is concerned, there are no facilities to display libjit types and
Eli> values in human-readable form, C-level backtraces show ?? when a JIT
Eli> compiled function is called, etc.

I built my own libjit with debuginfo and IIRC that was good enough for
normal things.

For stack traces through JIT code, you need the gdb unwinder I wrote.
It hasn't been checked in to libjit yet:

    http://lists.gnu.org/archive/html/libjit/2018-02/msg00014.html

I only made this work for x86-64 but porting it shouldn't be too hard.

Tom

Re: 32-bit MinGW build with JIT

[Eli Zaretskii]

> From: Tom Tromey <tom@tromey.com>
> Cc: Tom Tromey <tom@tromey.com>,  emacs-devel@gnu.org
> Date: Sun, 19 Aug 2018 09:24:37 -0600
> 
> >>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> Eli> I'm not yet sure what causes this, but since I presume that the branch
> Eli> was successfully built on a 64-bit host, I suspect some snafu with
> Eli> pointers that are narrower than 64-bit integers.
> 
> Thanks for trying this.  I'm very ill so I haven't been able to look
> into this code.  I will when I'm able.

Sorry to hear you are ill, and hoping you will get well soon.

> Also I'm about 1/4 of the way through converting it to use gcc-jit, so
> we can see how well that works.

Does that mean any further effort on the libjit branch is a waste of
time?  Also, it isn't clear to me whether gcc-jit works on non-ELF
platforms.

> I only made this work for x86-64 but porting it shouldn't be too hard.

OK, thanks.

Re: 32-bit MinGW build with JIT

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

>> Also I'm about 1/4 of the way through converting it to use gcc-jit, so
>> we can see how well that works.

Eli> Does that mean any further effort on the libjit branch is a waste of
Eli> time?  Also, it isn't clear to me whether gcc-jit works on non-ELF
Eli> platforms.

I thought it would be interesting to compare them.
I don't think further work on the libjit branch is a waste of time.
On the whole I tend to think that libjit, due to having few
dependencies and a slimmer runtime model, is the better JIT solution.

Tom

Re: 32-bit MinGW build with JIT

[Eli Zaretskii]

> From: Tom Tromey <tom@tromey.com>
> Cc: Tom Tromey <tom@tromey.com>,  emacs-devel@gnu.org
> Date: Sun, 19 Aug 2018 09:50:14 -0600
> 
> I don't think further work on the libjit branch is a waste of time.
> On the whole I tend to think that libjit, due to having few
> dependencies and a slimmer runtime model, is the better JIT solution.

Thanks, that's good to know, as after spending a day and a half on
debugging, I finally know the root cause for the crashes.

Re: 32-bit MinGW build with JIT

[Paul Eggert]

Eli Zaretskii wrote:
> Sorry to hear you are ill, and hoping you will get well soon.

Likewise! Please focus on your health; it's more important than Emacs. (Though 
we're very much looking forward to hearing more from you about Emacs....)

Re: 32-bit MinGW build with JIT

[Ken Brown]

On 8/19/2018 12:15 PM, Eli Zaretskii wrote:
>> From: Tom Tromey <tom@tromey.com>
>> Cc: Tom Tromey <tom@tromey.com>,  emacs-devel@gnu.org
>> Date: Sun, 19 Aug 2018 09:50:14 -0600
>>
>> I don't think further work on the libjit branch is a waste of time.
>> On the whole I tend to think that libjit, due to having few
>> dependencies and a slimmer runtime model, is the better JIT solution.
> 
> Thanks, that's good to know, as after spending a day and a half on
> debugging, I finally know the root cause for the crashes.

It now builds on 32-bit Cygwin also, thanks to your work.  I tested both 
the gtk3 build and the w32 build.  I can't test 64-bit Cygwin because 
libjit doesn't build there.  (My understanding from the libjit TODO file 
and the libjit mailing list archives is that it doesn't build on 64-bit 
MS-Windows either.)

Ken

Re: 32-bit MinGW build with JIT

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Mon, 27 Aug 2018 15:52:24 -0400
> 
> It now builds on 32-bit Cygwin also, thanks to your work.  I tested both 
> the gtk3 build and the w32 build.

Good to know, thanks.

> I can't test 64-bit Cygwin because libjit doesn't build there.  (My
> understanding from the libjit TODO file and the libjit mailing list
> archives is that it doesn't build on 64-bit MS-Windows either.)

Can you please point me to the relevant messages in the mailing-list
archives?

Re: 32-bit MinGW build with JIT

[Ken Brown]

On 8/28/2018 1:25 AM, Eli Zaretskii wrote:
>> I can't test 64-bit Cygwin because libjit doesn't build there.  (My
>> understanding from the libjit TODO file and the libjit mailing list
>> archives is that it doesn't build on 64-bit MS-Windows either.)
> 
> Can you please point me to the relevant messages in the mailing-list
> archives?

https://lists.gnu.org/archive/html/dotgnu-libjit/2009-10/msg00012.html
https://lists.gnu.org/archive/html/dotgnu-libjit/2009-10/msg00011.html
https://lists.gnu.org/archive/html/dotgnu-libjit/2012-08/msg00003.html

Ken

Re: 32-bit MinGW build with JIT

[Eli Zaretskii]

> Cc: tom@tromey.com, emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Tue, 28 Aug 2018 09:12:40 -0400
> 
> On 8/28/2018 1:25 AM, Eli Zaretskii wrote:
> >> I can't test 64-bit Cygwin because libjit doesn't build there.  (My
> >> understanding from the libjit TODO file and the libjit mailing list
> >> archives is that it doesn't build on 64-bit MS-Windows either.)
> > 
> > Can you please point me to the relevant messages in the mailing-list
> > archives?
> 
> https://lists.gnu.org/archive/html/dotgnu-libjit/2009-10/msg00012.html
> https://lists.gnu.org/archive/html/dotgnu-libjit/2009-10/msg00011.html
> https://lists.gnu.org/archive/html/dotgnu-libjit/2012-08/msg00003.html

Thanks.  One of these:

  https://lists.gnu.org/archive/html/dotgnu-libjit/2012-08/msg00000.html

points to the bug tracker:

  https://savannah.gnu.org/patch/?group=dotgnu-pnet

where one can find relevant patches.  Perhaps they can be rebased on
the current libjit trunk and re-submitted to the current maintainer?