bug#41242: Port feature/native-comp to Windows

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

The attached patches contain changes to make the feature/native-comp branch work
on Windows.

There are a few remaining issues:

* The loading process is very slow. This is especially annoying when coupled
  with autoloading. For example, autoloading Helm may stall Emacs for 5 seconds
  in my machine.

  I have thought a possible solution to this problem: load the byte-compiled
  file and put the native-compiled version in a list. Then load that list one by
  one on an idle-timer, that way the UI freezes should be minimized. This could
  reuse the "late loading" machinery implemented for deferred compilation.

* `package-delete` fails because it tries to delete the .eln file via
  `delete-file`. This is impossible in Windows because it's generally impossible
  to delete files that have an open HANDLE in the system.

  Three solutions have crossed my mind:

  - Edit `package-delete` to put the eln on a list of files that should be
    deleted when Emacs is closed.

  - Implement an unloading mechanism for native-compiled files.

  - Copy eln files to temporary files and load those temporary files instead.
    This means that deleting the original eln file is possible.

  I'd prefer the second option, but I have a semi-working patch for the third
  option.

* The `emacs_dir` environment variable needs to be set when loading the dump
  file. It is necessary for `expand-file-name`, which calls emacs_root_dir(). I
  haven't investigated why this is necessary yet. One workaround would be to use
  GetModuleFileName() to get the path to emacs.exe when `emacs_dir` is not set.

[-- Attachment #2: 0001-HACK-Ensure-the-emacs_root_dir-function-does-not-cra.patch --]
[-- Type: application/octet-stream, Size: 1042 bytes --]

From 68ad10da9ff538ab21da1a4f40ee4b556d6a3d76 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 16:30:11 -0300
Subject: [PATCH 1/8] HACK: Ensure the `emacs_root_dir` function does not
 crash.

That function is called when calling `load_pdump`. The problem is that
that happens too early in the initialization process and the
`emacs_dir` environment variable will not have been initialized yet.

The proper fix would be to initialize `emacs_dir` early enough. I do
not know enough of the Emacs internals to do that safely.
---
 src/w32.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/w32.c b/src/w32.c
index 0f69e652a5..a8c763f23e 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -3154,7 +3154,7 @@ emacs_root_dir (void)
 
   p = getenv ("emacs_dir");
   if (p == NULL)
-    emacs_abort ();
+    p = "C:/Users";
   filename_from_ansi (p, root_dir);
   root_dir[parse_root (root_dir, NULL)] = '\0';
   dostounix_filename (root_dir);
-- 
2.25.1.windows.1


[-- Attachment #3: 0002-Do-not-block-SIGIO-in-platforms-that-don-t-have-it.patch --]
[-- Type: application/octet-stream, Size: 909 bytes --]

From 1ce7fe9d2fa66bbce45523b6275071cad6853248 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 16:02:58 -0300
Subject: [PATCH 2/8] Do not block SIGIO in platforms that don't have it.

* src/comp.c (comp--compile-ctxt-to-file): Add a preprocessor check to
avoid blocking SIGIO in platforms that don't have it.
---
 src/comp.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/comp.c b/src/comp.c
index e3a80adfa9..00f9d4b74a 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -3345,7 +3345,9 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
       sigemptyset (&blocked);
       sigaddset (&blocked, SIGALRM);
       sigaddset (&blocked, SIGINT);
+#ifdef USABLE_SIGIO
       sigaddset (&blocked, SIGIO);
+#endif
       pthread_sigmask (SIG_BLOCK, &blocked, &oldset);
     }
   emit_ctxt_code ();
-- 
2.25.1.windows.1


[-- Attachment #4: 0003-Handle-setjmp-taking-two-arguments-in-Windows.patch --]
[-- Type: application/octet-stream, Size: 4674 bytes --]

From 8ea5ceda6389c227f82e8d698b644344f17f52b9 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 15:56:09 -0300
Subject: [PATCH 3/8] Handle setjmp() taking two arguments in Windows.

* src/comp.c: Add `define_setjmp_deps()` and `emit_setjmp()` which
abstract over this difference in behavior between operating systems.

WARNING: Not all cases are handled by this patch. The Mingw-64
setjmp.h header deals with many other combinations. I don't think it
is a good idea to replicate the logic of that header inside
emacs. (Maybe a few lines in the configure script could be added to
handle this problem?)
---
 src/comp.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 52 insertions(+), 6 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 00f9d4b74a..0d46628e6a 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -22,6 +22,7 @@
 
 #ifdef HAVE_NATIVE_COMP
 
+#include <setjmp.h>
 #include <stdlib.h>
 #include <stdio.h>
 #include <signal.h>
@@ -74,10 +75,15 @@ #define DECL_BLOCK(name, func)				\
   gcc_jit_block *(name) =				\
     gcc_jit_function_new_block ((func), STR (name))
 
-#ifdef HAVE__SETJMP
-#define SETJMP _setjmp
+#ifndef _WIN32
+# ifdef HAVE__SETJMP
+#  define SETJMP _setjmp
+# else
+#  define SETJMP setjmp
+# endif
 #else
-#define SETJMP setjmp
+/* snippet from MINGW-64 setjmp.h */
+# define SETJMP _setjmp
 #endif
 #define SETJMP_NAME SETJMP
 
@@ -174,6 +180,9 @@ #define F_RELOC_MAX_SIZE 1500
   gcc_jit_function *setcdr;
   gcc_jit_function *check_type;
   gcc_jit_function *check_impure;
+#ifdef _WIN32
+  gcc_jit_function *setjmp_ctx_func;
+#endif
   Lisp_Object func_blocks_h; /* blk_name -> gcc_block.  */
   Lisp_Object exported_funcs_h; /* c-func-name -> gcc_jit_function *.  */
   Lisp_Object imported_funcs_h; /* subr_name -> gcc_jit_field *reloc_field.  */
@@ -1474,6 +1483,29 @@ emit_limple_call_ref (Lisp_Object insn, bool direct)
 			direct);
 }
 
+static gcc_jit_rvalue *
+emit_setjmp (gcc_jit_rvalue *buf)
+{
+#ifndef _WIN32
+  gcc_jit_rvalue *args[] = {buf};
+  return emit_call (intern_c_string (STR (SETJMP_NAME)), comp.int_type, 1, args,
+                   false);
+#else
+  /* _setjmp (buf, __builtin_frame_address (0)) */
+  gcc_jit_rvalue *args[2];
+
+  args[0] = gcc_jit_context_new_rvalue_from_int (comp.ctxt, comp.unsigned_type, 0);
+
+  args[1] = gcc_jit_context_new_call(comp.ctxt,
+                                     NULL,
+                                     comp.setjmp_ctx_func,
+                                     1, args);
+  args[0] = buf;
+  return emit_call (intern_c_string (STR (SETJMP_NAME)), comp.int_type, 2, args,
+                    false);
+#endif
+}
+
 /* Register an handler for a non local exit.  */
 
 static void
@@ -1500,8 +1532,7 @@ emit_limple_push_handler (gcc_jit_rvalue *handler, gcc_jit_rvalue *handler_type,
 	NULL);
 
   gcc_jit_rvalue *res;
-  res =
-    emit_call (intern_c_string (STR (SETJMP_NAME)), comp.int_type, 1, args, false);
+  res = emit_setjmp(args[0]);
   emit_cond_jump (res, handler_bb, guarded_bb);
 }
 
@@ -2060,8 +2091,14 @@ #define ADD_IMPORTED(f_name, ret_type, nargs, args)			       \
   args[1] = comp.int_type;
   ADD_IMPORTED (push_handler, comp.handler_ptr_type, 2, args);
 
+#ifndef _WIN32
   args[0] = gcc_jit_type_get_pointer (gcc_jit_struct_as_type (comp.jmp_buf_s));
   ADD_IMPORTED (SETJMP_NAME, comp.int_type, 1, args);
+#else
+  args[0] = gcc_jit_type_get_pointer (gcc_jit_struct_as_type (comp.jmp_buf_s));
+  args[1] = comp.void_ptr_type;
+  ADD_IMPORTED (SETJMP_NAME, comp.int_type, 2, args);
+#endif
 
   ADD_IMPORTED (record_unwind_protect_excursion, comp.void_type, 0, NULL);
 
@@ -2301,7 +2338,7 @@ define_jmp_buf (void)
       gcc_jit_context_new_array_type (comp.ctxt,
 				      NULL,
 				      comp.char_type,
-				      sizeof (jmp_buf)),
+				      sizeof (sys_jmp_buf)),
       "stuff");
   comp.jmp_buf_s =
     gcc_jit_context_new_struct_type (comp.ctxt,
@@ -2969,6 +3006,14 @@ define_CHECK_IMPURE (void)
     gcc_jit_block_end_with_void_return (err_block, NULL);
 }
 
+static void
+define_setjmp_deps (void)
+{
+  comp.setjmp_ctx_func
+    = gcc_jit_context_get_builtin_function (comp.ctxt,
+                                            "__builtin_frame_address");
+}
+
 /* Define a function to convert boolean into t or nil */
 
 static void
@@ -3357,6 +3402,7 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
   define_PSEUDOVECTORP ();
   define_CHECK_TYPE ();
   define_CHECK_IMPURE ();
+  define_setjmp_deps ();
   define_bool_to_lisp_obj ();
   define_setcar_setcdr ();
   define_add1_sub1 ();
-- 
2.25.1.windows.1


[-- Attachment #5: 0004-Handle-LISP_WORDS_ARE_POINTERS-and-CHECK_LISP_OBJECT.patch --]
[-- Type: application/octet-stream, Size: 18158 bytes --]

From 5b8d529baa8b3c51a9220aa149362c2d7c5fa160 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 14:30:14 -0300
Subject: [PATCH 4/8] Handle LISP_WORDS_ARE_POINTERS and
 CHECK_LISP_OBJECT_TYPE.

* src/comp.c: Introduce the Lisp_X, Lisp_Word, and Lisp_Word_tag
types. These types are used instead of long or long long. Use
emacs_int_type and emacs_uint_types where appropriate.
(emit_coerce): Add special logic that handles the case when
Lisp_Object is a struct. This is necessary for handling the
--enable-check-lisp-object-type configure option.

* src/lisp.h: Since libgccjit does not support opaque unions, change
Lisp_X to be struct. This is done to ensure that the same types are
used in the same binary. It is probably unnecessary since only a
pointer to it is used.
---
 src/comp.c | 328 ++++++++++++++++++++++++++++++++++++-----------------
 src/lisp.h |   5 +-
 2 files changed, 228 insertions(+), 105 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 0d46628e6a..0d5880ad3c 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -116,6 +116,16 @@ #define F_RELOC_MAX_SIZE 1500
   gcc_jit_type *char_ptr_type;
   gcc_jit_type *ptrdiff_type;
   gcc_jit_type *uintptr_type;
+#if LISP_WORDS_ARE_POINTERS
+  gcc_jit_struct *lisp_X_s;
+  gcc_jit_type *lisp_X;
+#endif
+  gcc_jit_type *lisp_word_type;
+  gcc_jit_type *lisp_word_tag_type;
+#ifdef LISP_OBJECT_IS_STRUCT
+  gcc_jit_field *lisp_obj_i;
+  gcc_jit_struct *lisp_obj_s;
+#endif
   gcc_jit_type *lisp_obj_type;
   gcc_jit_type *lisp_obj_ptr_type;
   /* struct Lisp_Cons */
@@ -158,7 +168,8 @@ #define F_RELOC_MAX_SIZE 1500
   gcc_jit_field *cast_union_as_c_p;
   gcc_jit_field *cast_union_as_v_p;
   gcc_jit_field *cast_union_as_lisp_cons_ptr;
-  gcc_jit_field *cast_union_as_lisp_obj;
+  gcc_jit_field *cast_union_as_lisp_word;
+  gcc_jit_field *cast_union_as_lisp_word_tag;
   gcc_jit_field *cast_union_as_lisp_obj_ptr;
   gcc_jit_function *func; /* Current function being compiled.  */
   bool func_has_non_local; /* From comp-func has-non-local slot.  */
@@ -347,8 +358,10 @@ type_to_cast_field (gcc_jit_type *type)
     field = comp.cast_union_as_c_p;
   else if (type == comp.lisp_cons_ptr_type)
     field = comp.cast_union_as_lisp_cons_ptr;
-  else if (type == comp.lisp_obj_type)
-    field = comp.cast_union_as_lisp_obj;
+  else if (type == comp.lisp_word_type)
+    field = comp.cast_union_as_lisp_word;
+  else if (type == comp.lisp_word_tag_type)
+    field = comp.cast_union_as_lisp_word_tag;
   else if (type == comp.lisp_obj_ptr_type)
     field = comp.cast_union_as_lisp_obj_ptr;
   else
@@ -627,6 +640,31 @@ emit_coerce (gcc_jit_type *new_type, gcc_jit_rvalue *obj)
   if (new_type == old_type)
     return obj;
 
+#ifdef LISP_OBJECT_IS_STRUCT
+  if (old_type == comp.lisp_obj_type)
+    {
+      gcc_jit_rvalue *lwordobj =
+        gcc_jit_rvalue_access_field (obj, NULL, comp.lisp_obj_i);
+      return emit_coerce (new_type, lwordobj);
+    }
+
+  if (new_type == comp.lisp_obj_type)
+    {
+      gcc_jit_rvalue *lwordobj =
+        emit_coerce (comp.lisp_word_type, obj);
+
+      gcc_jit_lvalue *tmp_s
+        = gcc_jit_function_new_local (comp.func, NULL, comp.lisp_obj_type,
+                                      format_string ("lisp_obj_%td", i++));
+
+      gcc_jit_block_add_assignment (comp.block, NULL,
+                                    gcc_jit_lvalue_access_field (tmp_s, NULL,
+                                                                 comp.lisp_obj_i),
+                                    lwordobj);
+      return gcc_jit_lvalue_as_rvalue (tmp_s);
+    }
+#endif
+
   gcc_jit_field *orig_field =
     type_to_cast_field (old_type);
   gcc_jit_field *dest_field = type_to_cast_field (new_type);
@@ -664,14 +702,8 @@ emit_binary_op (enum gcc_jit_binary_op op,
 /* Should come with libgccjit.  */
 
 static gcc_jit_rvalue *
-emit_rvalue_from_long_long (long long n)
+emit_rvalue_from_long_long (gcc_jit_type *type, long long n)
 {
-#ifndef WIDE_EMACS_INT
-  xsignal1 (Qnative_ice,
-	    build_string ("emit_rvalue_from_long_long called in non wide int"
-			  " configuration"));
-#endif
-
   emit_comment (format_string ("emit long long: %lld", n));
 
   gcc_jit_rvalue *high =
@@ -697,7 +729,7 @@ emit_rvalue_from_long_long (long long n)
 		      32));
 
   return
-    emit_coerce (comp.long_long_type,
+    emit_coerce (type,
       emit_binary_op (
 	GCC_JIT_BINARY_OP_BITWISE_OR,
 	comp.unsigned_long_long_type,
@@ -712,29 +744,135 @@ emit_rvalue_from_long_long (long long n)
 }
 
 static gcc_jit_rvalue *
-emit_most_positive_fixnum (void)
+emit_rvalue_from_unsigned_long_long (gcc_jit_type *type, unsigned long long n)
+{
+  emit_comment (format_string ("emit unsigned long long: %llu", n));
+
+  gcc_jit_rvalue *high =
+    gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+					  comp.unsigned_long_long_type,
+					  n >> 32);
+  gcc_jit_rvalue *low =
+    emit_binary_op (GCC_JIT_BINARY_OP_RSHIFT,
+		    comp.unsigned_long_long_type,
+		    emit_binary_op (GCC_JIT_BINARY_OP_LSHIFT,
+				    comp.unsigned_long_long_type,
+				    gcc_jit_context_new_rvalue_from_long (
+				      comp.ctxt,
+				      comp.unsigned_long_long_type,
+				      n),
+				    gcc_jit_context_new_rvalue_from_int (
+				      comp.ctxt,
+				      comp.unsigned_long_long_type,
+				      32)),
+		    gcc_jit_context_new_rvalue_from_int (
+		      comp.ctxt,
+		      comp.unsigned_long_long_type,
+		      32));
+
+  return emit_coerce (
+           type,
+           emit_binary_op (
+             GCC_JIT_BINARY_OP_BITWISE_OR,
+             comp.unsigned_long_long_type,
+             emit_binary_op (
+               GCC_JIT_BINARY_OP_LSHIFT,
+               comp.unsigned_long_long_type,
+               high,
+               gcc_jit_context_new_rvalue_from_int (comp.ctxt,
+                                                    comp.unsigned_long_long_type,
+                                                    32)),
+             low));
+}
+
+static gcc_jit_rvalue *
+emit_rvalue_from_emacs_uint (EMACS_UINT val)
+{
+  if (val != (long) val)
+    {
+      return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+                                                   comp.emacs_uint_type,
+                                                   val);
+    }
+  else
+    {
+      return emit_rvalue_from_unsigned_long_long (comp.emacs_uint_type, val);
+    }
+}
+
+static gcc_jit_rvalue *
+emit_rvalue_from_emacs_int (EMACS_INT val)
+{
+  if (val != (long) val)
+    {
+      return emit_rvalue_from_long_long (comp.emacs_int_type, val);
+    }
+  else
+    {
+      return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+                                                   comp.emacs_int_type, val);
+    }
+}
+
+static gcc_jit_rvalue *
+emit_rvalue_from_lisp_word_tag (Lisp_Word_tag val)
 {
-#if EMACS_INT_MAX > LONG_MAX
-  return emit_rvalue_from_long_long (MOST_POSITIVE_FIXNUM);
+  if (val != (long) val)
+    {
+      return emit_rvalue_from_unsigned_long_long (comp.lisp_word_tag_type, val);
+    }
+  else
+    {
+      return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+                                                   comp.lisp_word_tag_type,
+                                                   val);
+    }
+}
+
+static gcc_jit_rvalue *
+emit_rvalue_from_lisp_word (Lisp_Word val)
+{
+#if LISP_WORDS_ARE_POINTERS
+  return gcc_jit_context_new_rvalue_from_ptr (comp.ctxt,
+                                              comp.lisp_word_type,
+                                              val);
 #else
-  return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
-					       comp.emacs_int_type,
-					       MOST_POSITIVE_FIXNUM);
+  if (val != (long) val)
+    {
+      return emit_rvalue_from_unsigned_long_long (comp.lisp_word_type, val);
+    }
+  else
+    {
+      return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+                                                   comp.lisp_word_type,
+                                                   val);
+    }
 #endif
 }
 
 static gcc_jit_rvalue *
-emit_most_negative_fixnum (void)
+emit_rvalue_from_lisp_obj (Lisp_Object obj)
 {
-#if EMACS_INT_MAX > LONG_MAX
-  return emit_rvalue_from_long_long (MOST_NEGATIVE_FIXNUM);
+#ifdef LISP_OBJECT_IS_STRUCT
+  return emit_coerce(comp.lisp_obj_type,
+                     emit_rvalue_from_lisp_word (obj.i));
 #else
-  return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
-					       comp.emacs_int_type,
-					       MOST_NEGATIVE_FIXNUM);
+  return emit_rvalue_from_lisp_word (obj);
 #endif
 }
 
+static gcc_jit_rvalue *
+emit_most_positive_fixnum (void)
+{
+  return emit_rvalue_from_emacs_int(MOST_POSITIVE_FIXNUM);
+}
+
+static gcc_jit_rvalue *
+emit_most_negative_fixnum (void)
+{
+  return emit_rvalue_from_emacs_int (MOST_NEGATIVE_FIXNUM);
+}
+
 /*
    Emit the equivalent of:
    (typeof_ptr) ((uintptr) ptr + size_of_ptr_ref * i)
@@ -769,7 +907,7 @@ emit_ptr_arithmetic (gcc_jit_rvalue *ptr, gcc_jit_type *ptr_type,
 emit_XLI (gcc_jit_rvalue *obj)
 {
   emit_comment ("XLI");
-  return obj;
+  return emit_coerce (comp.emacs_int_type, obj);
 }
 
 static gcc_jit_lvalue *
@@ -779,54 +917,40 @@ emit_lval_XLI (gcc_jit_lvalue *obj)
   return obj;
 }
 
-/*
+
 static gcc_jit_rvalue *
 emit_XLP (gcc_jit_rvalue *obj)
 {
   emit_comment ("XLP");
 
-  return gcc_jit_rvalue_access_field (obj,
-				      NULL,
-				      comp.lisp_obj_as_ptr);
+  return emit_coerce(comp.void_ptr_type, obj);
 }
 
-static gcc_jit_lvalue *
-emit_lval_XLP (gcc_jit_lvalue *obj)
-{
-  emit_comment ("lval_XLP");
+/* TODO */
+/* static gcc_jit_lvalue * */
+/* emit_lval_XLP (gcc_jit_lvalue *obj) */
+/* { */
+/*   emit_comment ("lval_XLP"); */
+
+/*   return gcc_jit_lvalue_access_field (obj, */
+/* 				      NULL, */
+/* 				      comp.lisp_obj_as_ptr); */
+/* } */
 
-  return gcc_jit_lvalue_access_field (obj,
-				      NULL,
-				      comp.lisp_obj_as_ptr);
-} */
 static gcc_jit_rvalue *
-emit_XUNTAG (gcc_jit_rvalue *a, gcc_jit_type *type, long long lisp_word_tag)
+emit_XUNTAG (gcc_jit_rvalue *a, gcc_jit_type *type, Lisp_Word_tag lisp_word_tag)
 {
   /* #define XUNTAG(a, type, ctype) ((ctype *)
      ((char *) XLP (a) - LISP_WORD_TAG (type))) */
   emit_comment ("XUNTAG");
 
-#ifndef WIDE_EMACS_INT
   return emit_coerce (
 	   gcc_jit_type_get_pointer (type),
 	   emit_binary_op (
 	     GCC_JIT_BINARY_OP_MINUS,
-	     comp.emacs_int_type,
-	     emit_XLI (a),
-	     gcc_jit_context_new_rvalue_from_int (
-	       comp.ctxt,
-	       comp.emacs_int_type,
-	       lisp_word_tag)));
-#else
-  return emit_coerce (
-	   gcc_jit_type_get_pointer (type),
-	   emit_binary_op (
-	     GCC_JIT_BINARY_OP_MINUS,
-	     comp.unsigned_long_long_type,
-	     /* FIXME Should be XLP.  */
-	     emit_XLI (a),
-	     emit_rvalue_from_long_long (lisp_word_tag)));
-#endif
+	     comp.uintptr_type,
+	     emit_XLP (a),
+	     emit_rvalue_from_lisp_word_tag(lisp_word_tag)));
 }
 
 static gcc_jit_rvalue *
@@ -853,7 +977,7 @@ emit_EQ (gcc_jit_rvalue *x, gcc_jit_rvalue *y)
 }
 
 static gcc_jit_rvalue *
-emit_TAGGEDP (gcc_jit_rvalue *obj, ptrdiff_t tag)
+emit_TAGGEDP (gcc_jit_rvalue *obj, Lisp_Word_tag tag)
 {
    /* (! (((unsigned) (XLI (a) >> (USE_LSB_TAG ? 0 : VALBITS)) \
 	- (unsigned) (tag)) \
@@ -1054,17 +1178,7 @@ emit_make_fixnum_LSB_TAG (gcc_jit_rvalue *n)
 			comp.emacs_int_type,
 			tmp, comp.lisp_int0);
 
-  gcc_jit_lvalue *res = gcc_jit_function_new_local (comp.func,
-						    NULL,
-						    comp.lisp_obj_type,
-						    "lisp_obj_fixnum");
-
-  gcc_jit_block_add_assignment (comp.block,
-				NULL,
-				emit_lval_XLI (res),
-				tmp);
-
-  return gcc_jit_lvalue_as_rvalue (res);
+  return emit_coerce (comp.lisp_obj_type, tmp);
 }
 
 static gcc_jit_rvalue *
@@ -1076,10 +1190,8 @@ emit_make_fixnum_MSB_TAG (gcc_jit_rvalue *n)
     return XIL (n);
   */
 
-  gcc_jit_rvalue *intmask =
-    emit_coerce (comp.emacs_uint_type,
-		 emit_rvalue_from_long_long ((EMACS_INT_MAX
-					      >> (INTTYPEBITS - 1))));
+  gcc_jit_rvalue *intmask = emit_rvalue_from_emacs_uint (INTMASK);
+
   n = emit_binary_op (GCC_JIT_BINARY_OP_BITWISE_AND,
 		      comp.emacs_uint_type,
 		      intmask, n);
@@ -1090,12 +1202,10 @@ emit_make_fixnum_MSB_TAG (gcc_jit_rvalue *n)
 		    emit_binary_op (GCC_JIT_BINARY_OP_LSHIFT,
 				    comp.emacs_uint_type,
 				    comp.lisp_int0,
-				    gcc_jit_context_new_rvalue_from_int (
-				      comp.ctxt,
-				      comp.emacs_uint_type,
-				      VALBITS)),
+                                    emit_rvalue_from_emacs_uint(VALBITS)),
 		    n);
-  return emit_XLI (emit_coerce (comp.emacs_int_type, n));
+
+  return emit_coerce (comp.lisp_obj_type, n);
 }
 
 
@@ -1114,17 +1224,10 @@ emit_const_lisp_obj (Lisp_Object obj)
   emit_comment (format_string ("const lisp obj: %s",
 			       SSDATA (Fprin1_to_string (obj, Qnil))));
 
-  if (NIL_IS_ZERO && EQ (obj, Qnil))
+  if (EQ (obj, Qnil))
     {
       gcc_jit_rvalue *n;
-#ifdef WIDE_EMACS_INT
-      eassert (NIL_IS_ZERO);
-      n = emit_rvalue_from_long_long (0);
-#else
-      n = gcc_jit_context_new_rvalue_from_ptr (comp.ctxt,
-					       comp.void_ptr_type,
-					       NULL);
-#endif
+      n = emit_rvalue_from_lisp_word ((Lisp_Word) iQnil);
       return emit_coerce (comp.lisp_obj_type, n);
     }
 
@@ -1345,16 +1448,7 @@ emit_mvar_val (Lisp_Object mvar)
 	  /* We can still emit directly objects that are self-contained in a
 	     word (read fixnums).  */
 	  emit_comment (SSDATA (Fprin1_to_string (constant, Qnil)));
-	  gcc_jit_rvalue *word;
-#ifdef WIDE_EMACS_INT
-	  word = emit_rvalue_from_long_long (constant);
-#else
-	  word =
-	    gcc_jit_context_new_rvalue_from_ptr (comp.ctxt,
-						 comp.void_ptr_type,
-						 XLP (constant));
-#endif
-	  return emit_coerce (comp.lisp_obj_type, word);
+          return emit_rvalue_from_lisp_obj(constant);
 	}
       /* Other const objects are fetched from the reloc array.  */
       return emit_const_lisp_obj (constant);
@@ -2518,11 +2612,16 @@ define_cast_union (void)
 			       NULL,
 			       comp.lisp_cons_ptr_type,
 			       "cons_ptr");
-  comp.cast_union_as_lisp_obj =
+  comp.cast_union_as_lisp_word =
     gcc_jit_context_new_field (comp.ctxt,
 			       NULL,
-			       comp.lisp_obj_type,
-			       "lisp_obj");
+			       comp.lisp_word_type,
+			       "lisp_word");
+  comp.cast_union_as_lisp_word_tag =
+    gcc_jit_context_new_field (comp.ctxt,
+                               NULL,
+                               comp.lisp_word_tag_type,
+                               "lisp_word_tag");
   comp.cast_union_as_lisp_obj_ptr =
     gcc_jit_context_new_field (comp.ctxt,
 			       NULL,
@@ -2543,7 +2642,8 @@ define_cast_union (void)
       comp.cast_union_as_c_p,
       comp.cast_union_as_v_p,
       comp.cast_union_as_lisp_cons_ptr,
-      comp.cast_union_as_lisp_obj,
+      comp.cast_union_as_lisp_word,
+      comp.cast_union_as_lisp_word_tag,
       comp.cast_union_as_lisp_obj_ptr };
   comp.cast_union_type =
     gcc_jit_context_new_union_type (comp.ctxt,
@@ -2810,8 +2910,8 @@ define_add1_sub1 (void)
 					  GCC_JIT_COMPARISON_NE,
 					  n_fixnum,
 					  i == 0
-					  ? emit_most_positive_fixnum ()
-					  : emit_most_negative_fixnum ())),
+					  ? emit_rvalue_from_emacs_int (MOST_POSITIVE_FIXNUM)
+					  : emit_rvalue_from_emacs_int (MOST_NEGATIVE_FIXNUM))),
 	inline_block,
 	fcall_block);
 
@@ -3307,9 +3407,31 @@ DEFUN ("comp--init-ctxt", Fcomp__init_ctxt, Scomp__init_ctxt,
   comp.emacs_uint_type = gcc_jit_context_get_int_type (comp.ctxt,
 						       sizeof (EMACS_UINT),
 						       false);
-  /* No XLP is emitted for now so lets define this always as integer
-     disregarding LISP_WORDS_ARE_POINTERS value.  */
-  comp.lisp_obj_type = comp.emacs_int_type;
+#if LISP_WORDS_ARE_POINTERS
+  comp.lisp_X_s = gcc_jit_context_new_opaque_struct (comp.ctxt,
+                                                     NULL,
+                                                     "Lisp_X");
+  comp.lisp_X = gcc_jit_struct_as_type(comp.lisp_X_s);
+  comp.lisp_word_type = gcc_jit_type_get_pointer(comp.lisp_X);
+#else
+  comp.lisp_word_type = comp.emacs_int_type;
+#endif
+  comp.lisp_word_tag_type
+    = gcc_jit_context_get_int_type (comp.ctxt, sizeof (Lisp_Word_tag), false);
+#ifdef LISP_OBJECT_IS_STRUCT
+  comp.lisp_obj_i = gcc_jit_context_new_field (comp.ctxt,
+                                               NULL,
+                                               comp.lisp_word_type,
+                                               "i");
+  comp.lisp_obj_s = gcc_jit_context_new_struct_type (comp.ctxt,
+                                                     NULL,
+                                                     "Lisp_Object",
+                                                     1,
+                                                     &comp.lisp_obj_i);
+  comp.lisp_obj_type = gcc_jit_struct_as_type (comp.lisp_obj_s);
+#else
+  comp.lisp_obj_type = comp.lisp_word_type;
+#endif
   comp.lisp_obj_ptr_type = gcc_jit_type_get_pointer (comp.lisp_obj_type);
   comp.one =
     gcc_jit_context_new_rvalue_from_int (comp.ctxt,
diff --git a/src/lisp.h b/src/lisp.h
index 3d082911f5..e3d196ef9b 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -299,12 +299,12 @@ #define GCALIGNED(type) (alignof (type) % GCALIGNMENT == 0)
 
 /* Lisp_Word is a scalar word suitable for holding a tagged pointer or
    integer.  Usually it is a pointer to a deliberately-incomplete type
-   'union Lisp_X'.  However, it is EMACS_INT when Lisp_Objects and
+   'struct Lisp_X'.  However, it is EMACS_INT when Lisp_Objects and
    pointers differ in width.  */
 
 #define LISP_WORDS_ARE_POINTERS (EMACS_INT_MAX == INTPTR_MAX)
 #if LISP_WORDS_ARE_POINTERS
-typedef union Lisp_X *Lisp_Word;
+typedef struct Lisp_X *Lisp_Word;
 #else
 typedef EMACS_INT Lisp_Word;
 #endif
@@ -573,6 +573,7 @@ #define ENUM_BF(TYPE) enum TYPE
 
 #ifdef CHECK_LISP_OBJECT_TYPE
 typedef struct Lisp_Object { Lisp_Word i; } Lisp_Object;
+# define LISP_OBJECT_IS_STRUCT
 # define LISP_INITIALLY(w) {w}
 # undef CHECK_LISP_OBJECT_TYPE
 enum CHECK_LISP_OBJECT_TYPE { CHECK_LISP_OBJECT_TYPE = true };
-- 
2.25.1.windows.1


[-- Attachment #6: 0005-Remove-a-layer-of-indirection-for-access-to-pure-sto.patch --]
[-- Type: application/octet-stream, Size: 3345 bytes --]

From 717301bf3f1f73569765ccd88e161c73c33c7001 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 16:23:10 -0300
Subject: [PATCH 5/8] Remove a layer of indirection for access to pure storage.

* src/comp.c: Taking the address of an array is the same as casting it
to a pointer. Therefore, the C expression `(EMACS_INT **) &pure` is in
fact adding a layer of indirection that is not necessary. The fix is
to cast the `pure` array to a pointer and store that in a void pointer
that is part of the compiled shared library.
---
 src/comp.c | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 0d5880ad3c..69525acfc0 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -38,7 +38,7 @@
 
 /* C symbols emitted for the load relocation mechanism.  */
 #define CURRENT_THREAD_RELOC_SYM "current_thread_reloc"
-#define PURE_RELOC_SYM "pure_reloc"
+#define PURE_PTR_SYM "pure_ptr"
 #define DATA_RELOC_SYM "d_reloc"
 #define DATA_RELOC_IMPURE_SYM "d_reloc_imp"
 #define DATA_RELOC_EPHEMERAL_SYM "d_reloc_eph"
@@ -152,7 +152,7 @@ #define F_RELOC_MAX_SIZE 1500
   gcc_jit_type *thread_state_ptr_type;
   gcc_jit_rvalue *current_thread_ref;
   /* Other globals.  */
-  gcc_jit_rvalue *pure_ref;
+  gcc_jit_rvalue *pure_ptr;
   /* libgccjit has really limited support for casting therefore this union will
      be used for the scope.  */
   gcc_jit_type *cast_union_type;
@@ -1419,8 +1419,7 @@ emit_PURE_P (gcc_jit_rvalue *ptr)
 	GCC_JIT_BINARY_OP_MINUS,
 	comp.uintptr_type,
 	ptr,
-	gcc_jit_lvalue_as_rvalue (
-	  gcc_jit_rvalue_dereference (comp.pure_ref, NULL))),
+        comp.pure_ptr),
       gcc_jit_context_new_rvalue_from_int (comp.ctxt,
 					   comp.uintptr_type,
 					   PURESIZE));
@@ -2244,14 +2243,14 @@ emit_ctxt_code (void)
 	gcc_jit_type_get_pointer (comp.thread_state_ptr_type),
 	CURRENT_THREAD_RELOC_SYM));
 
-  comp.pure_ref =
+  comp.pure_ptr =
     gcc_jit_lvalue_as_rvalue (
       gcc_jit_context_new_global (
 	comp.ctxt,
 	NULL,
 	GCC_JIT_GLOBAL_EXPORTED,
-	gcc_jit_type_get_pointer (comp.void_ptr_type),
-	PURE_RELOC_SYM));
+        comp.void_ptr_type,
+	PURE_PTR_SYM));
 
   gcc_jit_context_new_global (
 	comp.ctxt,
@@ -3767,13 +3766,13 @@ load_comp_unit (struct Lisp_Native_Comp_Unit *comp_u, bool loading_dump,
     {
       struct thread_state ***current_thread_reloc =
 	dynlib_sym (handle, CURRENT_THREAD_RELOC_SYM);
-      EMACS_INT ***pure_reloc = dynlib_sym (handle, PURE_RELOC_SYM);
+      void **pure_ptr = dynlib_sym (handle, PURE_PTR_SYM);
       Lisp_Object *data_relocs = dynlib_sym (handle, DATA_RELOC_SYM);
       Lisp_Object *data_imp_relocs = dynlib_sym (handle, DATA_RELOC_IMPURE_SYM);
       void **freloc_link_table = dynlib_sym (handle, FUNC_LINK_TABLE_SYM);
 
       if (!(current_thread_reloc
-	    && pure_reloc
+	    && pure_ptr
 	    && data_relocs
 	    && data_imp_relocs
 	    && data_eph_relocs
@@ -3784,7 +3783,7 @@ load_comp_unit (struct Lisp_Native_Comp_Unit *comp_u, bool loading_dump,
 	xsignal1 (Qnative_lisp_file_inconsistent, comp_u->file);
 
       *current_thread_reloc = &current_thread;
-      *pure_reloc = (EMACS_INT **)&pure;
+      *pure_ptr = pure;
 
       /* Imported functions.  */
       *freloc_link_table = freloc.link_table;
-- 
2.25.1.windows.1


[-- Attachment #7: 0006-Workaround-the-32768-chars-command-line-limit-in-Win.patch --]
[-- Type: application/octet-stream, Size: 2422 bytes --]

From 567594a35d5730783c94b28fdbd6a25104488120 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 14:04:06 -0300
Subject: [PATCH 6/8] Workaround the 32768 chars command line limit in Windows.

* lisp/emacs-lisp/comp.el (comp-run-async-workers): Pass the
compilation commands through a temporary file that is loaded by the
child process. This is also done all other operating systems, even
those that support long command lines. It should not be a problem
since libgccjit uses temporary files too.
---
 lisp/emacs-lisp/comp.el | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index c2a95feec1..d32f93a1e1 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -2239,6 +2239,9 @@ comp-run-async-workers
                             (message "Compiling %s..." ,source-file)
                             (native-compile ,source-file ,(and load t))))
                    (source-file1 source-file) ;; Make the closure works :/
+                   (temp-file (make-temp-file
+                               (concat "emacs-async-comp-" (file-name-base source-file) "-")
+                               nil ".el" (prin1-to-string expr)))
                    (load1 load)
                    (process (make-process
                              :name (concat "Compiling: " source-file)
@@ -2246,13 +2249,14 @@ comp-run-async-workers
                              :command (list
                                        (expand-file-name invocation-name
                                                          invocation-directory)
-                                       "--batch" "--eval" (prin1-to-string expr))
+                                       "--batch" "-l" temp-file)
                              :sentinel
                              (lambda (process _event)
                                (run-hook-with-args
                                 'comp-async-cu-done-hook
                                 source-file)
                                (accept-process-output process)
+                               (ignore-errors (delete-file temp-file))
                                (when (and load1
                                           (zerop (process-exit-status process)))
                                  (native-elisp-load
-- 
2.25.1.windows.1


[-- Attachment #8: 0007-Load-libgccjit-dynamically-in-Windows.patch --]
[-- Type: application/octet-stream, Size: 26333 bytes --]

From 5119898fbd347752c46b09deecb2365d6c7fce7f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 11 May 2020 20:43:06 -0300
Subject: [PATCH 7/8] Load libgccjit dynamically in Windows.

* configure.ac: don't add linker flags if compiling on
Windows. Compile dynlib.c if modules or native compilation are
enabled. Always compile comp.c
* lisp/term/w32-win.el: Map 'gccjit to "libgccjit.dll" in
`dynamic-library-alist`.
* src/Makefile.in: Update comments. Update to handle changes in
configure.ac.
* src/comp.c: Add declarations of used libgccjit functions using
DEF_DLL_FN. Add calls to load_gccjit_if_necessary() where necessary.
Add `native-comp-available-p`
* src/comp.h: Remove Fnative_elisp_load. Add syms_of_comp().
* src/emacs.c (main): Always call syms_of_comp()
* src/w32.c (globals_of_w32): Clear Vlibrary_cache when starting
because the libraries loaded when dumping will not be loaded when
starting.
* src/w32fns.c: Add Qgccjit symbol.
---
 configure.ac         |  19 ++-
 lisp/term/w32-win.el |   3 +-
 src/Makefile.in      |   9 +-
 src/comp.c           | 374 ++++++++++++++++++++++++++++++++++++++++++-
 src/comp.h           |   6 +-
 src/emacs.c          |   2 -
 src/w32.c            |   4 +
 src/w32fns.c         |   1 +
 8 files changed, 398 insertions(+), 20 deletions(-)

diff --git a/configure.ac b/configure.ac
index 23b94cf6ca..ea0144f404 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3666,6 +3666,7 @@ AC_DEFUN
 LIBMODULES=
 HAVE_MODULES=no
 MODULES_OBJ=
+NEED_DYNLIB=no
 case $opsys in
   cygwin|mingw32) MODULES_SUFFIX=".dll" ;;
   darwin) MODULES_SUFFIX=".dylib" ;;
@@ -3701,7 +3702,8 @@ AC_DEFUN
 fi
 
 if test "${HAVE_MODULES}" = yes; then
-   MODULES_OBJ="dynlib.o emacs-module.o"
+   MODULES_OBJ="emacs-module.o"
+   NEED_DYNLIB=yes
    AC_DEFINE(HAVE_MODULES, 1, [Define to 1 if dynamic modules are enabled])
    AC_DEFINE_UNQUOTED(MODULES_SUFFIX, "$MODULES_SUFFIX",
      [System extension for dynamic libraries])
@@ -3785,7 +3787,6 @@ AC_DEFUN
 
 HAVE_NATIVE_COMP=no
 LIBGCCJIT_LIB=
-COMP_OBJ=
 if test "${with_nativecomp}" != "no"; then
     emacs_save_LIBS=$LIBS
     LIBS="-lgccjit"
@@ -3793,8 +3794,11 @@ AC_DEFUN
       [AC_LINK_IFELSE([libgccjit_smoke_test], [], [libgccjit_not_found])])
     LIBS=$emacs_save_LIBS
     HAVE_NATIVE_COMP=yes
-    LIBGCCJIT_LIB="-lgccjit -ldl"
-    COMP_OBJ="comp.o"
+    # mingw32 loads the library dynamically.
+    if test "${opsys}" != "mingw32"; then
+      LIBGCCJIT_LIB="-lgccjit -ldl"
+    fi
+    NEED_DYNLIB=yes
     AC_DEFINE(HAVE_NATIVE_COMP, 1, [Define to 1 if you have the libgccjit library (-lgccjit).])
 fi
 if test "${HAVE_NATIVE_COMP}" = yes && test "${HAVE_PDUMPER}" = no; then
@@ -3804,7 +3808,12 @@ AC_DEFUN
   [System extension for native compiled elisp])
 AC_SUBST(HAVE_NATIVE_COMP)
 AC_SUBST(LIBGCCJIT_LIB)
-AC_SUBST(COMP_OBJ)
+
+DYNLIB_OBJ=
+if test "${NEED_DYNLIB}" = yes; then
+  DYNLIB_OBJ="dynlib.o"
+fi
+AC_SUBST(DYNLIB_OBJ)
 
 ### Use -lpng if available, unless '--with-png=no'.
 HAVE_PNG=no
diff --git a/lisp/term/w32-win.el b/lisp/term/w32-win.el
index 5901e0295e..6b9716ca30 100644
--- a/lisp/term/w32-win.el
+++ b/lisp/term/w32-win.el
@@ -289,7 +289,8 @@ libgnutls-version
        '(libxml2 "libxml2-2.dll" "libxml2.dll")
        '(zlib "zlib1.dll" "libz-1.dll")
        '(lcms2 "liblcms2-2.dll")
-       '(json "libjansson-4.dll")))
+       '(json "libjansson-4.dll")
+       '(gccjit "libgccjit.dll")))
 
 ;;; multi-tty support
 (defvar w32-initialized nil
diff --git a/src/Makefile.in b/src/Makefile.in
index 63f909ae14..85709184da 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -241,7 +241,7 @@ LIBZ =
 
 ## system-specific libs for dynamic modules, else empty
 LIBMODULES = @LIBMODULES@
-## dynlib.o emacs-module.o if modules enabled, else empty
+## emacs-module.o if modules enabled, else empty
 MODULES_OBJ = @MODULES_OBJ@
 
 XRANDR_LIBS = @XRANDR_LIBS@
@@ -327,8 +327,9 @@ GMP_LIB =
 GMP_OBJ = @GMP_OBJ@
 
 LIBGCCJIT = @LIBGCCJIT_LIB@
-## dynlib.o comp.o if native compiler is enabled, otherwise empty.
-COMP_OBJ = @COMP_OBJ@
+
+## dynlib.o if necessary, else empty
+DYNLIB_OBJ = @DYNLIB_OBJ@
 
 RUN_TEMACS = ./temacs
 
@@ -418,7 +419,7 @@ base_obj =
 	cmds.o casetab.o casefiddle.o indent.o search.o regex-emacs.o undo.o \
 	alloc.o pdumper.o data.o doc.o editfns.o callint.o \
 	eval.o floatfns.o fns.o font.o print.o lread.o $(MODULES_OBJ) \
-	syntax.o $(UNEXEC_OBJ) bytecode.o $(COMP_OBJ) \
+	syntax.o $(UNEXEC_OBJ) bytecode.o comp.o $(DYNLIB_OBJ) \
 	process.o gnutls.o callproc.o \
 	region-cache.o sound.o timefns.o atimer.o \
 	doprnt.o intervals.o textprop.o composite.o xml.o lcms.o $(NOTIFY_OBJ) \
diff --git a/src/comp.c b/src/comp.c
index 69525acfc0..b43d3eddb3 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -20,6 +20,8 @@
 
 #include <config.h>
 
+#include "lisp.h"
+
 #ifdef HAVE_NATIVE_COMP
 
 #include <setjmp.h>
@@ -28,7 +30,6 @@
 #include <signal.h>
 #include <libgccjit.h>
 
-#include "lisp.h"
 #include "puresize.h"
 #include "window.h"
 #include "dynlib.h"
@@ -36,6 +37,347 @@
 #include "blockinput.h"
 #include "sha512.h"
 
+\f
+/********************************/
+/* Dynamic loading of libgccjit */
+/********************************/
+
+#ifdef WINDOWSNT
+# include "w32common.h"
+
+#undef gcc_jit_block_add_assignment
+#undef gcc_jit_block_add_comment
+#undef gcc_jit_block_add_eval
+#undef gcc_jit_block_end_with_conditional
+#undef gcc_jit_block_end_with_jump
+#undef gcc_jit_block_end_with_return
+#undef gcc_jit_block_end_with_void_return
+#undef gcc_jit_context_acquire
+#undef gcc_jit_context_compile_to_file
+#undef gcc_jit_context_dump_reproducer_to_file
+#undef gcc_jit_context_dump_to_file
+#undef gcc_jit_context_get_builtin_function
+#undef gcc_jit_context_get_first_error
+#undef gcc_jit_context_get_int_type
+#undef gcc_jit_context_get_type
+#undef gcc_jit_context_new_array_access
+#undef gcc_jit_context_new_array_type
+#undef gcc_jit_context_new_binary_op
+#undef gcc_jit_context_new_call
+#undef gcc_jit_context_new_call_through_ptr
+#undef gcc_jit_context_new_comparison
+#undef gcc_jit_context_new_field
+#undef gcc_jit_context_new_function
+#undef gcc_jit_context_new_function_ptr_type
+#undef gcc_jit_context_new_global
+#undef gcc_jit_context_new_opaque_struct
+#undef gcc_jit_context_new_param
+#undef gcc_jit_context_new_rvalue_from_int
+#undef gcc_jit_context_new_rvalue_from_long
+#undef gcc_jit_context_new_rvalue_from_ptr
+#undef gcc_jit_context_new_struct_type
+#undef gcc_jit_context_new_unary_op
+#undef gcc_jit_context_new_union_type
+#undef gcc_jit_context_release
+#undef gcc_jit_context_set_bool_option
+#undef gcc_jit_context_set_int_option
+#undef gcc_jit_context_set_logfile
+#undef gcc_jit_function_get_param
+#undef gcc_jit_function_new_block
+#undef gcc_jit_function_new_local
+#undef gcc_jit_lvalue_access_field
+#undef gcc_jit_lvalue_as_rvalue
+#undef gcc_jit_lvalue_get_address
+#undef gcc_jit_param_as_lvalue
+#undef gcc_jit_param_as_rvalue
+#undef gcc_jit_rvalue_access_field
+#undef gcc_jit_rvalue_dereference
+#undef gcc_jit_rvalue_dereference_field
+#undef gcc_jit_rvalue_get_type
+#undef gcc_jit_struct_as_type
+#undef gcc_jit_struct_set_fields
+#undef gcc_jit_type_get_pointer
+
+/* In alphabetical order */
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_rvalue_from_int,
+            (gcc_jit_context *ctxt, gcc_jit_type *numeric_type, int value));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_lvalue_as_rvalue,
+            (gcc_jit_lvalue *lvalue));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_rvalue_access_field,
+            (gcc_jit_rvalue *struct_or_union, gcc_jit_location *loc,
+             gcc_jit_field *field));
+DEF_DLL_FN (void, gcc_jit_block_add_comment,
+            (gcc_jit_block *block, gcc_jit_location *loc, const char *text));
+DEF_DLL_FN (void, gcc_jit_context_release, (gcc_jit_context *ctxt));
+DEF_DLL_FN (const char *, gcc_jit_context_get_first_error,
+            (gcc_jit_context *ctxt));
+DEF_DLL_FN (gcc_jit_block *, gcc_jit_function_new_block,
+            (gcc_jit_function *func, const char *name));
+DEF_DLL_FN (gcc_jit_context *, gcc_jit_context_acquire, (void));
+DEF_DLL_FN (gcc_jit_field *, gcc_jit_context_new_field,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, gcc_jit_type *type,
+             const char *name));
+DEF_DLL_FN (gcc_jit_function *, gcc_jit_context_get_builtin_function,
+            (gcc_jit_context *ctxt, const char *name));
+DEF_DLL_FN (gcc_jit_function *, gcc_jit_context_new_function,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_function_kind kind, gcc_jit_type *return_type,
+             const char *name, int num_params, gcc_jit_param **params,
+             int is_variadic));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_context_new_array_access,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, gcc_jit_rvalue *ptr,
+             gcc_jit_rvalue *index));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_context_new_global,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_global_kind kind, gcc_jit_type *type,
+             const char *name));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_function_new_local,
+            (gcc_jit_function *func, gcc_jit_location *loc, gcc_jit_type *type,
+             const char *name));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_lvalue_access_field,
+            (gcc_jit_lvalue *struct_or_union, gcc_jit_location *loc,
+             gcc_jit_field *field));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_param_as_lvalue, (gcc_jit_param *param));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_rvalue_dereference,
+            (gcc_jit_rvalue *rvalue, gcc_jit_location *loc));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_rvalue_dereference_field,
+            (gcc_jit_rvalue *ptr, gcc_jit_location *loc, gcc_jit_field *field));
+DEF_DLL_FN (gcc_jit_param *, gcc_jit_context_new_param,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, gcc_jit_type *type,
+             const char *name));
+DEF_DLL_FN (gcc_jit_param *, gcc_jit_function_get_param,
+            (gcc_jit_function *func, int index));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_binary_op,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_binary_op op, gcc_jit_type *result_type,
+             gcc_jit_rvalue *a, gcc_jit_rvalue *b));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_call,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             gcc_jit_function *func, int numargs , gcc_jit_rvalue **args));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_call_through_ptr,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             gcc_jit_rvalue *fn_ptr, int numargs, gcc_jit_rvalue **args));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_comparison,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_comparison op, gcc_jit_rvalue *a, gcc_jit_rvalue *b));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_rvalue_from_long,
+            (gcc_jit_context *ctxt, gcc_jit_type *numeric_type, long value));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_rvalue_from_ptr,
+            (gcc_jit_context *ctxt, gcc_jit_type *pointer_type, void *value));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_unary_op,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_unary_op op, gcc_jit_type *result_type,
+             gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_lvalue_get_address,
+            (gcc_jit_lvalue *lvalue, gcc_jit_location *loc));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_param_as_rvalue, (gcc_jit_param *param));
+DEF_DLL_FN (gcc_jit_struct *, gcc_jit_context_new_opaque_struct,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, const char *name));
+DEF_DLL_FN (gcc_jit_struct *, gcc_jit_context_new_struct_type,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, const char *name,
+             int num_fields, gcc_jit_field **fields));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_get_int_type,
+            (gcc_jit_context *ctxt, int num_bytes, int is_signed));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_get_type,
+            (gcc_jit_context *ctxt, enum gcc_jit_types type_));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_new_array_type,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             gcc_jit_type *element_type, int num_elements));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_new_function_ptr_type,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             gcc_jit_type *return_type, int num_params,
+             gcc_jit_type **param_types, int is_variadic));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_new_union_type,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, const char *name,
+             int num_fields, gcc_jit_field **fields));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_rvalue_get_type, (gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_struct_as_type,
+            (gcc_jit_struct *struct_type));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_type_get_pointer, (gcc_jit_type *type));
+DEF_DLL_FN (void, gcc_jit_block_add_assignment,
+            (gcc_jit_block *block, gcc_jit_location *loc, gcc_jit_lvalue *lvalue,
+             gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (void, gcc_jit_block_add_eval,
+            (gcc_jit_block *block, gcc_jit_location *loc,
+             gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (void, gcc_jit_block_end_with_conditional,
+            (gcc_jit_block *block, gcc_jit_location *loc,
+             gcc_jit_rvalue *boolval, gcc_jit_block *on_true,
+             gcc_jit_block *on_false));
+DEF_DLL_FN (void, gcc_jit_block_end_with_jump,
+            (gcc_jit_block *block, gcc_jit_location *loc,
+             gcc_jit_block *target));
+DEF_DLL_FN (void, gcc_jit_block_end_with_return,
+            (gcc_jit_block *block, gcc_jit_location *loc,
+             gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (void, gcc_jit_block_end_with_void_return,
+            (gcc_jit_block *block, gcc_jit_location *loc));
+DEF_DLL_FN (void, gcc_jit_context_compile_to_file,
+            (gcc_jit_context *ctxt, enum gcc_jit_output_kind output_kind,
+             const char *output_path));
+DEF_DLL_FN (void, gcc_jit_context_dump_reproducer_to_file,
+            (gcc_jit_context *ctxt, const char *path));
+DEF_DLL_FN (void, gcc_jit_context_dump_to_file,
+            (gcc_jit_context *ctxt, const char *path, int update_locations));
+DEF_DLL_FN (void, gcc_jit_context_set_bool_option,
+            (gcc_jit_context *ctxt, enum gcc_jit_bool_option opt, int value));
+DEF_DLL_FN (void, gcc_jit_context_set_int_option,
+            (gcc_jit_context *ctxt, enum gcc_jit_int_option opt, int value));
+DEF_DLL_FN (void, gcc_jit_context_set_logfile,
+            (gcc_jit_context *ctxt, FILE *logfile, int flags, int verbosity));
+DEF_DLL_FN (void, gcc_jit_struct_set_fields,
+            (gcc_jit_struct *struct_type, gcc_jit_location *loc, int num_fields,
+             gcc_jit_field **fields));
+
+static bool
+init_gccjit_functions (void)
+{
+  HMODULE library;
+
+  if (!(library = w32_delayed_load (Qgccjit)))
+    {
+      return false;
+    }
+
+  /* In alphabetical order */
+  LOAD_DLL_FN(library, gcc_jit_block_add_assignment);
+  LOAD_DLL_FN(library, gcc_jit_block_add_comment);
+  LOAD_DLL_FN(library, gcc_jit_block_add_eval);
+  LOAD_DLL_FN(library, gcc_jit_block_end_with_conditional);
+  LOAD_DLL_FN(library, gcc_jit_block_end_with_jump);
+  LOAD_DLL_FN(library, gcc_jit_block_end_with_return);
+  LOAD_DLL_FN(library, gcc_jit_block_end_with_void_return);
+  LOAD_DLL_FN(library, gcc_jit_context_acquire);
+  LOAD_DLL_FN(library, gcc_jit_context_compile_to_file);
+  LOAD_DLL_FN(library, gcc_jit_context_dump_reproducer_to_file);
+  LOAD_DLL_FN(library, gcc_jit_context_dump_to_file);
+  LOAD_DLL_FN(library, gcc_jit_context_get_builtin_function);
+  LOAD_DLL_FN(library, gcc_jit_context_get_first_error);
+  LOAD_DLL_FN(library, gcc_jit_context_get_int_type);
+  LOAD_DLL_FN(library, gcc_jit_context_get_type);
+  LOAD_DLL_FN(library, gcc_jit_context_new_array_access);
+  LOAD_DLL_FN(library, gcc_jit_context_new_array_type);
+  LOAD_DLL_FN(library, gcc_jit_context_new_binary_op);
+  LOAD_DLL_FN(library, gcc_jit_context_new_call);
+  LOAD_DLL_FN(library, gcc_jit_context_new_call_through_ptr);
+  LOAD_DLL_FN(library, gcc_jit_context_new_comparison);
+  LOAD_DLL_FN(library, gcc_jit_context_new_field);
+  LOAD_DLL_FN(library, gcc_jit_context_new_function);
+  LOAD_DLL_FN(library, gcc_jit_context_new_function_ptr_type);
+  LOAD_DLL_FN(library, gcc_jit_context_new_global);
+  LOAD_DLL_FN(library, gcc_jit_context_new_opaque_struct);
+  LOAD_DLL_FN(library, gcc_jit_context_new_param);
+  LOAD_DLL_FN(library, gcc_jit_context_new_rvalue_from_int);
+  LOAD_DLL_FN(library, gcc_jit_context_new_rvalue_from_long);
+  LOAD_DLL_FN(library, gcc_jit_context_new_rvalue_from_ptr);
+  LOAD_DLL_FN(library, gcc_jit_context_new_struct_type);
+  LOAD_DLL_FN(library, gcc_jit_context_new_unary_op);
+  LOAD_DLL_FN(library, gcc_jit_context_new_union_type);
+  LOAD_DLL_FN(library, gcc_jit_context_release);
+  LOAD_DLL_FN(library, gcc_jit_context_set_bool_option);
+  LOAD_DLL_FN(library, gcc_jit_context_set_int_option);
+  LOAD_DLL_FN(library, gcc_jit_context_set_logfile);
+  LOAD_DLL_FN(library, gcc_jit_function_get_param);
+  LOAD_DLL_FN(library, gcc_jit_function_new_block);
+  LOAD_DLL_FN(library, gcc_jit_function_new_local);
+  LOAD_DLL_FN(library, gcc_jit_lvalue_access_field);
+  LOAD_DLL_FN(library, gcc_jit_lvalue_as_rvalue);
+  LOAD_DLL_FN(library, gcc_jit_lvalue_get_address);
+  LOAD_DLL_FN(library, gcc_jit_param_as_lvalue);
+  LOAD_DLL_FN(library, gcc_jit_param_as_rvalue);
+  LOAD_DLL_FN(library, gcc_jit_rvalue_access_field);
+  LOAD_DLL_FN(library, gcc_jit_rvalue_dereference);
+  LOAD_DLL_FN(library, gcc_jit_rvalue_dereference_field);
+  LOAD_DLL_FN(library, gcc_jit_rvalue_get_type);
+  LOAD_DLL_FN(library, gcc_jit_struct_as_type);
+  LOAD_DLL_FN(library, gcc_jit_struct_set_fields);
+  LOAD_DLL_FN(library, gcc_jit_type_get_pointer);
+
+  return true;
+}
+
+/* In alphabetical order */
+#define gcc_jit_block_add_assignment fn_gcc_jit_block_add_assignment
+#define gcc_jit_block_add_comment fn_gcc_jit_block_add_comment
+#define gcc_jit_block_add_eval fn_gcc_jit_block_add_eval
+#define gcc_jit_block_end_with_conditional fn_gcc_jit_block_end_with_conditional
+#define gcc_jit_block_end_with_jump fn_gcc_jit_block_end_with_jump
+#define gcc_jit_block_end_with_return fn_gcc_jit_block_end_with_return
+#define gcc_jit_block_end_with_void_return fn_gcc_jit_block_end_with_void_return
+#define gcc_jit_context_acquire fn_gcc_jit_context_acquire
+#define gcc_jit_context_compile_to_file fn_gcc_jit_context_compile_to_file
+#define gcc_jit_context_dump_reproducer_to_file fn_gcc_jit_context_dump_reproducer_to_file
+#define gcc_jit_context_dump_to_file fn_gcc_jit_context_dump_to_file
+#define gcc_jit_context_get_builtin_function fn_gcc_jit_context_get_builtin_function
+#define gcc_jit_context_get_first_error fn_gcc_jit_context_get_first_error
+#define gcc_jit_context_get_int_type fn_gcc_jit_context_get_int_type
+#define gcc_jit_context_get_type fn_gcc_jit_context_get_type
+#define gcc_jit_context_new_array_access fn_gcc_jit_context_new_array_access
+#define gcc_jit_context_new_array_type fn_gcc_jit_context_new_array_type
+#define gcc_jit_context_new_binary_op fn_gcc_jit_context_new_binary_op
+#define gcc_jit_context_new_call fn_gcc_jit_context_new_call
+#define gcc_jit_context_new_call_through_ptr fn_gcc_jit_context_new_call_through_ptr
+#define gcc_jit_context_new_comparison fn_gcc_jit_context_new_comparison
+#define gcc_jit_context_new_field fn_gcc_jit_context_new_field
+#define gcc_jit_context_new_function fn_gcc_jit_context_new_function
+#define gcc_jit_context_new_function_ptr_type fn_gcc_jit_context_new_function_ptr_type
+#define gcc_jit_context_new_global fn_gcc_jit_context_new_global
+#define gcc_jit_context_new_opaque_struct fn_gcc_jit_context_new_opaque_struct
+#define gcc_jit_context_new_param fn_gcc_jit_context_new_param
+#define gcc_jit_context_new_rvalue_from_int fn_gcc_jit_context_new_rvalue_from_int
+#define gcc_jit_context_new_rvalue_from_long fn_gcc_jit_context_new_rvalue_from_long
+#define gcc_jit_context_new_rvalue_from_ptr fn_gcc_jit_context_new_rvalue_from_ptr
+#define gcc_jit_context_new_struct_type fn_gcc_jit_context_new_struct_type
+#define gcc_jit_context_new_unary_op fn_gcc_jit_context_new_unary_op
+#define gcc_jit_context_new_union_type fn_gcc_jit_context_new_union_type
+#define gcc_jit_context_release fn_gcc_jit_context_release
+#define gcc_jit_context_set_bool_option fn_gcc_jit_context_set_bool_option
+#define gcc_jit_context_set_int_option fn_gcc_jit_context_set_int_option
+#define gcc_jit_context_set_logfile fn_gcc_jit_context_set_logfile
+#define gcc_jit_function_get_param fn_gcc_jit_function_get_param
+#define gcc_jit_function_new_block fn_gcc_jit_function_new_block
+#define gcc_jit_function_new_local fn_gcc_jit_function_new_local
+#define gcc_jit_lvalue_access_field fn_gcc_jit_lvalue_access_field
+#define gcc_jit_lvalue_as_rvalue fn_gcc_jit_lvalue_as_rvalue
+#define gcc_jit_lvalue_get_address fn_gcc_jit_lvalue_get_address
+#define gcc_jit_param_as_lvalue fn_gcc_jit_param_as_lvalue
+#define gcc_jit_param_as_rvalue fn_gcc_jit_param_as_rvalue
+#define gcc_jit_rvalue_access_field fn_gcc_jit_rvalue_access_field
+#define gcc_jit_rvalue_dereference fn_gcc_jit_rvalue_dereference
+#define gcc_jit_rvalue_dereference_field fn_gcc_jit_rvalue_dereference_field
+#define gcc_jit_rvalue_get_type fn_gcc_jit_rvalue_get_type
+#define gcc_jit_struct_as_type fn_gcc_jit_struct_as_type
+#define gcc_jit_struct_set_fields fn_gcc_jit_struct_set_fields
+#define gcc_jit_type_get_pointer fn_gcc_jit_type_get_pointer
+
+#endif
+
+static bool
+load_gccjit_if_necessary (bool mandatory)
+{
+#ifdef WINDOWSNT
+  static bool tried_to_initialize_once;
+  static bool gccjit_initialized;
+
+  if (!tried_to_initialize_once)
+    {
+      tried_to_initialize_once = true;
+      Lisp_Object status;
+      gccjit_initialized = init_gccjit_functions ();
+      status = gccjit_initialized ? Qt : Qnil;
+      Vlibrary_cache = Fcons (Fcons (Qgccjit, status), Vlibrary_cache);
+    }
+
+  if (mandatory && !gccjit_initialized)
+    xsignal1(Qnative_compiler_error, build_string("libgccjit not found"));
+
+  return gccjit_initialized;
+#else
+  return true;
+#endif
+}
+
+\f
 /* C symbols emitted for the load relocation mechanism.  */
 #define CURRENT_THREAD_RELOC_SYM "current_thread_reloc"
 #define PURE_PTR_SYM "pure_ptr"
@@ -3328,6 +3670,8 @@ DEFUN ("comp--init-ctxt", Fcomp__init_ctxt, Scomp__init_ctxt,
        doc: /* Initialize the native compiler context. Return t on success.  */)
      (void)
 {
+  load_gccjit_if_necessary(true);
+
   if (comp.ctxt)
     {
       xsignal1 (Qnative_ice,
@@ -3474,6 +3818,8 @@ DEFUN ("comp--release-ctxt", Fcomp__release_ctxt, Scomp__release_ctxt,
        doc: /* Release the native compiler context.  */)
      (void)
 {
+  load_gccjit_if_necessary(true);
+
   if (comp.ctxt)
     gcc_jit_context_release (comp.ctxt);
 
@@ -3490,6 +3836,8 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
        doc: /* Compile as native code the current context to file.  */)
      (Lisp_Object base_name)
 {
+  load_gccjit_if_necessary(true);
+
   CHECK_STRING (base_name);
 
   gcc_jit_context_set_int_option (comp.ctxt,
@@ -3660,6 +4008,9 @@ maybe_defer_native_compilation (Lisp_Object function_name,
       fflush (f);
     }
 #endif
+  if (!load_gccjit_if_necessary(false))
+    return;
+
   if (!comp_deferred_compilation
       || noninteractive
       || !NILP (Vpurify_flag)
@@ -3928,10 +4279,26 @@ DEFUN ("native-elisp-load", Fnative_elisp_load, Snative_elisp_load, 1, 2, 0,
   return Qt;
 }
 
+#endif /* HAVE_NATIVE_COMP */
+
+DEFUN ("native-comp-available-p", Fnative_comp_available_p,
+       Snative_comp_available_p, 0, 0, 0,
+       doc: /* Returns t if native compilation of Lisp files is available in
+this instance of Emacs. */)
+  (void)
+{
+#ifdef HAVE_NATIVE_COMP
+  return load_gccjit_if_necessary(false) ? Qt : Qnil;
+#else
+  return Qnil;
+#endif
+}
+
 \f
 void
 syms_of_comp (void)
 {
+#ifdef HAVE_NATIVE_COMP
   /* Compiler control customizes.  */
   DEFVAR_BOOL ("comp-deferred-compilation", comp_deferred_compilation,
 	       doc: /* If t compile asyncronously every .elc file loaded.  */);
@@ -4073,6 +4440,7 @@ syms_of_comp (void)
 	       doc: /* Hash table symbol-name -> function-value.  For
 		       internal use during  */);
   Vcomp_deferred_pending_h = CALLN (Fmake_hash_table, QCtest, Qeq);
-}
+#endif
 
-#endif /* HAVE_NATIVE_COMP */
+  defsubr (&Snative_comp_available_p);
+}
diff --git a/src/comp.h b/src/comp.h
index cbdcaccd5f..e6ab32ff8e 100644
--- a/src/comp.h
+++ b/src/comp.h
@@ -82,11 +82,7 @@ maybe_defer_native_compilation (Lisp_Object function_name,
 				Lisp_Object definition)
 {}
 
-static inline Lisp_Object
-Fnative_elisp_load (Lisp_Object file, Lisp_Object late_load)
-{
-  eassume (false);
-}
+extern void syms_of_comp (void);
 
 #endif
 
diff --git a/src/emacs.c b/src/emacs.c
index 2c90825742..e75cb58834 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1606,10 +1606,8 @@ main (int argc, char **argv)
   init_json ();
 #endif
 
-#ifdef HAVE_NATIVE_COMP
   if (!initialized)
     syms_of_comp ();
-#endif
 
   no_loadup
     = argmatch (argv, argc, "-nl", "--no-loadup", 6, NULL, &skip_args);
diff --git a/src/w32.c b/src/w32.c
index a8c763f23e..ceb8f7ef66 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -10586,6 +10586,10 @@ globals_of_w32 (void)
 #endif
 
   w32_crypto_hprov = (HCRYPTPROV)0;
+
+  /* We need to forget about libraries that were loaded during the
+     dumping process (e.g. libgccjit) */
+  Vlibrary_cache = Qnil;
 }
 
 /* For make-serial-process  */
diff --git a/src/w32fns.c b/src/w32fns.c
index e595b0285a..eeb73489dd 100644
--- a/src/w32fns.c
+++ b/src/w32fns.c
@@ -10462,6 +10462,7 @@ syms_of_w32fns (void)
   DEFSYM (Qzlib, "zlib");
   DEFSYM (Qlcms2, "lcms2");
   DEFSYM (Qjson, "json");
+  DEFSYM (Qgccjit, "gccjit");
 
   Fput (Qundefined_color, Qerror_conditions,
 	pure_list (Qundefined_color, Qerror));
-- 
2.25.1.windows.1


[-- Attachment #9: 0008-Windows-Use-NUMBER_OF_PROCESSORS-environment-variabl.patch --]
[-- Type: application/octet-stream, Size: 1433 bytes --]

From 2bc054317756f2e81bee766ad3888fbad4cecc46 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Wed, 13 May 2020 16:22:17 -0300
Subject: [PATCH 8/8] Windows: Use NUMBER_OF_PROCESSORS environment variable.

* lisp/emacs-lisp/comp.el (comp-effective-async-max-jobs): Use
NUMBER_OF_PROCESSORS environment variable if system is Windows NT,
"nproc" if it is in PATH or a default of 1.
---
 lisp/emacs-lisp/comp.el | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index d32f93a1e1..26bb79fcd1 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -2208,9 +2208,11 @@ comp-async-runnings
     (if (zerop comp-async-jobs-number)
         (or num-cpus
             (setf num-cpus
-                  ;; Half of the CPUs or at least one.
-                  ;; FIXME portable?
-                  (max 1 (/ (string-to-number (shell-command-to-string "nproc"))
+                  (max 1 (/ (cond ((eq 'windows-nt system-type)
+                                   (string-to-number (getenv "NUMBER_OF_PROCESSORS")))
+                                  ((executable-find "nproc")
+                                   (string-to-number (shell-command-to-string "nproc")))
+                                  (t 1))
                             2))))
       comp-async-jobs-number)))
 
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Wed, 13 May 2020 16:26:57 -0300
> 
> * The loading process is very slow. This is especially annoying when coupled
>   with autoloading. For example, autoloading Helm may stall Emacs for 5 seconds
>   in my machine.

Did you manage to understand why?  How many bytes of *.eln files does
Emacs load while autoloading Helm, for example?

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Wed, 13 May 2020 16:26:57 -0300
> 
> * `package-delete` fails because it tries to delete the .eln file via
>   `delete-file`. This is impossible in Windows because it's generally impossible
>   to delete files that have an open HANDLE in the system.

Is that the handle from LoadLibrary?  If so, cannot we close the
handle once the .eln file is loaded?

Thanks.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> Did you manage to understand why?  How many bytes of *.eln files does
> Emacs load while autoloading Helm, for example?

I don't know how to measure that, sorry. AFAIK Emacs just maps many subr to the
correct function pointers and the OS takes care of loading the appropriate code
on page faults.

My guess is that autoloading triggers a long series of eln loading operations,
that, as a group, are very expensive.

Do you know what profiler I could use to check what Emacs is doing?

> Is that the handle from LoadLibrary?  If so, cannot we close the
> handle once the .eln file is loaded?

Exactly. No, calling FreeLibrary would unload the file from the address space.

El mié., 13 may. 2020 a las 16:40, Eli Zaretskii (<eliz@gnu.org>) escribió:
>
> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Wed, 13 May 2020 16:26:57 -0300
> >
> > * `package-delete` fails because it tries to delete the .eln file via
> >   `delete-file`. This is impossible in Windows because it's generally impossible
> >   to delete files that have an open HANDLE in the system.
>
> Is that the handle from LoadLibrary?  If so, cannot we close the
> handle once the .eln file is loaded?
>
> Thanks.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Did you manage to understand why?  How many bytes of *.eln files does
>> Emacs load while autoloading Helm, for example?
>
> I don't know how to measure that, sorry. AFAIK Emacs just maps many subr to the
> correct function pointers and the OS takes care of loading the appropriate code
> on page faults.

Loading elns quite stress also the reader that is still used to
deserialize all objects except functions.

The fact that the load is that slower on Windows seems to indicate that
the equivalent dlopen dlsym are less performant than the other systems
we have tried (the reader should be quite the same).

That said there must be a way to profile on Windows so we get a picture
of what is going on.

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Wed, 13 May 2020 17:01:59 -0300
> Cc: 41242@debbugs.gnu.org
> 
> > Did you manage to understand why?  How many bytes of *.eln files does
> > Emacs load while autoloading Helm, for example?
> 
> I don't know how to measure that, sorry. AFAIK Emacs just maps many subr to the
> correct function pointers and the OS takes care of loading the appropriate code
> on page faults.
> 
> My guess is that autoloading triggers a long series of eln loading operations,
> that, as a group, are very expensive.
> 
> Do you know what profiler I could use to check what Emacs is doing?

I'd begin with Emacs's built-in "M-x profiler-start".  After invoking
that, start the Helm loading command, and when it ends, invoke
profiler-report.  Post the resulting profile fully expanded, and maybe
that will give some clues about what to examine next.

It will also be useful to have a comparable measurement from
GNU/Linux, so that we could compare the profiles and the elapsed
times.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
>> Date: Wed, 13 May 2020 16:26:57 -0300
>> 
>> * `package-delete` fails because it tries to delete the .eln file via
>>   `delete-file`. This is impossible in Windows because it's generally impossible
>>   to delete files that have an open HANDLE in the system.
>
> Is that the handle from LoadLibrary?  If so, cannot we close the
> handle once the .eln file is loaded?

AFAIU we cannot close the handle till we have code around (compiled Lisp
function) coming from the eln.

Andrea

PS BTW I see now we never close the handle but this is a bug, I'll fix now.

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> PS BTW I see now we never close the handle but this is a bug, I'll fix now.

Ah no we do it already.  Missed that sorry I'm fused.

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> The attached patches contain changes to make the feature/native-comp branch work
> on Windows.
>
> There are a few remaining issues:
>
> * The loading process is very slow. This is especially annoying when coupled
>   with autoloading. For example, autoloading Helm may stall Emacs for 5 seconds
>   in my machine.
>
>   I have thought a possible solution to this problem: load the byte-compiled
>   file and put the native-compiled version in a list. Then load that list one by
>   one on an idle-timer, that way the UI freezes should be minimized. This could
>   reuse the "late loading" machinery implemented for deferred compilation.

That's an option, but before implementing any machinery base on a timers
we need to profile to understand where we are loosing the time.

The timer is a mitigation but will still freeze the editor so it's a
last resource.

What is the state of your paperwork?  I guess still ongoing am I
correct?

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> What is the state of your paperwork?  I guess still ongoing am I
> correct?

I have already sent the signed form. I haven't got a response yet.

El mié., 13 may. 2020 a las 16:56, Andrea Corallo (<akrl@sdf.org>) escribió:
>
> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>
> > The attached patches contain changes to make the feature/native-comp branch work
> > on Windows.
> >
> > There are a few remaining issues:
> >
> > * The loading process is very slow. This is especially annoying when coupled
> >   with autoloading. For example, autoloading Helm may stall Emacs for 5 seconds
> >   in my machine.
> >
> >   I have thought a possible solution to this problem: load the byte-compiled
> >   file and put the native-compiled version in a list. Then load that list one by
> >   one on an idle-timer, that way the UI freezes should be minimized. This could
> >   reuse the "late loading" machinery implemented for deferred compilation.
>
> That's an option, but before implementing any machinery base on a timers
> we need to profile to understand where we are loosing the time.
>
> The timer is a mitigation but will still freeze the editor so it's a
> last resource.
>
> What is the state of your paperwork?  I guess still ongoing am I
> correct?
>
> Thanks
>
>   Andrea
>
> --
> akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> * `package-delete` fails because it tries to delete the .eln file via
>   `delete-file`. This is impossible in Windows because it's generally impossible
>   to delete files that have an open HANDLE in the system.
>
>   Three solutions have crossed my mind:
>
>   - Edit `package-delete` to put the eln on a list of files that should be
>     deleted when Emacs is closed.
>
>   - Implement an unloading mechanism for native-compiled files.
>
>   - Copy eln files to temporary files and load those temporary files instead.
>     This means that deleting the original eln file is possible.
>
>   I'd prefer the second option, but I have a semi-working patch for the third
>   option.

I don't think the second is an option.  The unload machanism is already
in place but is GC driven, I don't see how else it could work.

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

On May 14, 2020 1:18:26 PM GMT+03:00, Andrea Corallo <akrl@sdf.org> wrote:
> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
> 
> > * `package-delete` fails because it tries to delete the .eln file
> via
> >   `delete-file`. This is impossible in Windows because it's
> generally impossible
> >   to delete files that have an open HANDLE in the system.
> >
> >   Three solutions have crossed my mind:
> >
> >   - Edit `package-delete` to put the eln on a list of files that
> should be
> >     deleted when Emacs is closed.
> >
> >   - Implement an unloading mechanism for native-compiled files.
> >
> >   - Copy eln files to temporary files and load those temporary files
> instead.
> >     This means that deleting the original eln file is possible.
> >
> >   I'd prefer the second option, but I have a semi-working patch for
> the third
> >   option.
> 
> I don't think the second is an option.  The unload machanism is
> already
> in place but is GC driven, I don't see how else it could work.

Windows doesn't let you delete a shared library that's loaded by a process, but it does let you rename it.  So I think the solution would be to rename the .eln file to something like .eln.old, and then let the GC driven unload machinery to delete that old file.

Btw, what happens if more than one Emacs session have the same .eln file loaded, and one of them wants to recompile it?

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think the second is an option.  The unload machanism is
>> already
>> in place but is GC driven, I don't see how else it could work.
>
> Windows doesn't let you delete a shared library that's loaded by a
> process, but it does let you rename it.  So I think the solution would
> be to rename the .eln file to something like .eln.old, and then let
> the GC driven unload machinery to delete that old file.

Do we have guarantees that each object is collected before Emacs is
eventually closed?  I thought is not the case.

> Btw, what happens if more than one Emacs session have the same .eln file loaded, and one of them wants to recompile it?

Now to avoid this problem we always delete the file before recompiling.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: bug-gnu-emacs@gnu.org,
>         Nicolas Bértolo
>  <nicolasbertolo@gmail.com>,
>         41242@debbugs.gnu.org
> Date: Thu, 14 May 2020 11:17:11 +0000
> 
> > Windows doesn't let you delete a shared library that's loaded by a
> > process, but it does let you rename it.  So I think the solution would
> > be to rename the .eln file to something like .eln.old, and then let
> > the GC driven unload machinery to delete that old file.
> 
> Do we have guarantees that each object is collected before Emacs is
> eventually closed?  I thought is not the case.

I don't know enough about the "GC driven unload" you mentioned, but if
that is not bulletproof enough, we could add a kill-emacs hook to take
care of this.  And if push comes to shove, we could use a Windows API
that causes a file to be deleted when the last handle on it is closed,
but that would add complexity, so I think we should try easier ways
first.

> > Btw, what happens if more than one Emacs session have the same .eln file loaded, and one of them wants to recompile it?
> 
> Now to avoid this problem we always delete the file before recompiling.

But that's unportable, and won't work on Windows, for the same reasons
as the issue we are discussing here.  Or am I missing something?

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: bug-gnu-emacs@gnu.org,
>>         Nicolas Bértolo
>>  <nicolasbertolo@gmail.com>,
>>         41242@debbugs.gnu.org
>> Date: Thu, 14 May 2020 11:17:11 +0000
>>
>> > Windows doesn't let you delete a shared library that's loaded by a
>> > process, but it does let you rename it.  So I think the solution would
>> > be to rename the .eln file to something like .eln.old, and then let
>> > the GC driven unload machinery to delete that old file.
>>
>> Do we have guarantees that each object is collected before Emacs is
>> eventually closed?  I thought is not the case.
>
> I don't know enough about the "GC driven unload" you mentioned, but if
> that is not bulletproof enough, we could add a kill-emacs hook to take
> care of this.  And if push comes to shove, we could use a Windows API
> that causes a file to be deleted when the last handle on it is closed,
> but that would add complexity, so I think we should try easier ways
> first.

Now simply when the compilation unit object is collected, the handle is
closed.  The compilation unit is collected when is not reachable by any
of the functions defined into.

Loading a new compilation unit B that redefines the same functions
defined by A does not guarantees much, some of the old definitions of A
could be still in use somewhere, in that case A will not be collected.

So yeah I think we need a specific mechanism (kill-emacs hook as you
suggest) to do the cleanup when Emacs goes down.

>> > Btw, what happens if more than one Emacs session have the same .eln file loaded, and one of them wants to recompile it?
>>
>> Now to avoid this problem we always delete the file before recompiling.
>
> But that's unportable, and won't work on Windows, for the same reasons
> as the issue we are discussing here.  Or am I missing something?

No you are right, I don't use Windows since forever, I discovered from
this thread is not portable.  I guess we'll need to rename also here.

  Andrea

--
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> Loading a new compilation unit B that redefines the same functions
> defined by A does not guarantees much, some of the old definitions of A
> could be still in use somewhere, in that case A will not be collected.

> So yeah I think we need a specific mechanism (kill-emacs hook as you
> suggest) to do the cleanup when Emacs goes down.

I was thinking about something like this:

1) Call `native-comp-unload`.

2) This should inspect the eln file and put all the subrs defined in it on a
list. This should also copy the original bytecode from the eln file and store it
somewhere.

3) Then `garbage-collect` is called. This should find all references to the
subrs in the list and swap them atomically for references to functions
from the bytecode.

4) After the previous step the GC should be able to collect the DLL handle
knowing that no references to it remain.

What do you think?

> No you are right, I don't use Windows since forever, I discovered from
> this thread is not portable.  I guess we'll need to rename also here.

But someone needs to delete the old eln file. Let's say that we use
GetModuleFileNameA() to know if another Emacs instance has decided to rename the
eln file and then we delete it on close if its suffix is ".eln.old".

This algorithm has a big race condition though. If Emacs1 renames the file after
Emacs2 has checked that it has not been renamed, then the file won't be deleted.
If we put the "old eln" in the $TEMP folder this may not be a big issue though.

Nicolas.

El jue., 14 may. 2020 a las 12:03, Andrea Corallo (<akrl@sdf.org>) escribió:
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> From: Andrea Corallo <akrl@sdf.org>
> >> Cc: bug-gnu-emacs@gnu.org,
> >>         Nicolas Bértolo
> >>  <nicolasbertolo@gmail.com>,
> >>         41242@debbugs.gnu.org
> >> Date: Thu, 14 May 2020 11:17:11 +0000
> >>
> >> > Windows doesn't let you delete a shared library that's loaded by a
> >> > process, but it does let you rename it.  So I think the solution would
> >> > be to rename the .eln file to something like .eln.old, and then let
> >> > the GC driven unload machinery to delete that old file.
> >>
> >> Do we have guarantees that each object is collected before Emacs is
> >> eventually closed?  I thought is not the case.
> >
> > I don't know enough about the "GC driven unload" you mentioned, but if
> > that is not bulletproof enough, we could add a kill-emacs hook to take
> > care of this.  And if push comes to shove, we could use a Windows API
> > that causes a file to be deleted when the last handle on it is closed,
> > but that would add complexity, so I think we should try easier ways
> > first.
>
> Now simply when the compilation unit object is collected, the handle is
> closed.  The compilation unit is collected when is not reachable by any
> of the functions defined into.
>
> Loading a new compilation unit B that redefines the same functions
> defined by A does not guarantees much, some of the old definitions of A
> could be still in use somewhere, in that case A will not be collected.
>
> So yeah I think we need a specific mechanism (kill-emacs hook as you
> suggest) to do the cleanup when Emacs goes down.
>
> >> > Btw, what happens if more than one Emacs session have the same .eln file loaded, and one of them wants to recompile it?
> >>
> >> Now to avoid this problem we always delete the file before recompiling.
> >
> > But that's unportable, and won't work on Windows, for the same reasons
> > as the issue we are discussing here.  Or am I missing something?
>
> No you are right, I don't use Windows since forever, I discovered from
> this thread is not portable.  I guess we'll need to rename also here.
>
>   Andrea
>
> --
> akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 13:50:48 -0300
> Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> 
> 1) Call `native-comp-unload`.
> 
> 2) This should inspect the eln file and put all the subrs defined in it on a
> list. This should also copy the original bytecode from the eln file and store it
> somewhere.
> 
> 3) Then `garbage-collect` is called. This should find all references to the
> subrs in the list and swap them atomically for references to functions
> from the bytecode.
> 
> 4) After the previous step the GC should be able to collect the DLL handle
> knowing that no references to it remain.
> 
> What do you think?

Do we really need this complexity?

> > No you are right, I don't use Windows since forever, I discovered from
> > this thread is not portable.  I guess we'll need to rename also here.
> 
> But someone needs to delete the old eln file. Let's say that we use
> GetModuleFileNameA() to know if another Emacs instance has decided to rename the
> eln file and then we delete it on close if its suffix is ".eln.old".
> 
> This algorithm has a big race condition though. If Emacs1 renames the file after
> Emacs2 has checked that it has not been renamed, then the file won't be deleted.

Why do you need to check if it's renamed?  Just rename always.

> If we put the "old eln" in the $TEMP folder this may not be a big issue though.

It is not good to move to $TEMP because that one could be on a
different volume, and Windows won't let you do that with a DLL that is
loaded into a process.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> Why do you need to check if it's renamed?  Just rename always.

You need to see if the file has been renamed because someone needs
to delete the file

> It is not good to move to $TEMP because that one could be on a
> different volume, and Windows won't let you do that with a DLL that is
> loaded into a process.

Touché.

El jue., 14 may. 2020 a las 14:28, Eli Zaretskii (<eliz@gnu.org>) escribió:
>
> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Thu, 14 May 2020 13:50:48 -0300
> > Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> >
> > 1) Call `native-comp-unload`.
> >
> > 2) This should inspect the eln file and put all the subrs defined in it on a
> > list. This should also copy the original bytecode from the eln file and store it
> > somewhere.
> >
> > 3) Then `garbage-collect` is called. This should find all references to the
> > subrs in the list and swap them atomically for references to functions
> > from the bytecode.
> >
> > 4) After the previous step the GC should be able to collect the DLL handle
> > knowing that no references to it remain.
> >
> > What do you think?
>
> Do we really need this complexity?
>
> > > No you are right, I don't use Windows since forever, I discovered from
> > > this thread is not portable.  I guess we'll need to rename also here.
> >
> > But someone needs to delete the old eln file. Let's say that we use
> > GetModuleFileNameA() to know if another Emacs instance has decided to rename the
> > eln file and then we delete it on close if its suffix is ".eln.old".
> >
> > This algorithm has a big race condition though. If Emacs1 renames the file after
> > Emacs2 has checked that it has not been renamed, then the file won't be deleted.
>
> Why do you need to check if it's renamed?  Just rename always.
>
> > If we put the "old eln" in the $TEMP folder this may not be a big issue though.
>
> It is not good to move to $TEMP because that one could be on a
> different volume, and Windows won't let you do that with a DLL that is
> loaded into a process.

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 14:35:36 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > Why do you need to check if it's renamed?  Just rename always.
> 
> You need to see if the file has been renamed because someone needs
> to delete the file

If you always rename, then you need to always delete, no?

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> If you always rename, then you need to always delete, no?

The Emacs instance that renames the file may not be able to delete the file
because another Emacs instance may keep an open handle.

It is this last Emacs instance that should delete the file, not the
one that renamed it.

El jue., 14 may. 2020 a las 14:57, Eli Zaretskii (<eliz@gnu.org>) escribió:
>
> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Thu, 14 May 2020 14:35:36 -0300
> > Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> >
> > > Why do you need to check if it's renamed?  Just rename always.
> >
> > You need to see if the file has been renamed because someone needs
> > to delete the file
>
> If you always rename, then you need to always delete, no?

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 15:00:14 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > If you always rename, then you need to always delete, no?
> 
> The Emacs instance that renames the file may not be able to delete the file
> because another Emacs instance may keep an open handle.
> 
> It is this last Emacs instance that should delete the file, not the
> one that renamed it.

Then I guess on Windows we will have to live with the limitation that
a .eln file used by another session cannot be recompiled.

(I must confess that the idea to use libgccjit and shared libraries
for native compilation looks less and less attractive to me due to
these complications.)

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> (I must confess that the idea to use libgccjit and shared libraries
> for native compilation looks less and less attractive to me due to
> these complications.)

Come on, do we get demotivated that early!? :) :)

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> If you always rename, then you need to always delete, no?
>
> The Emacs instance that renames the file may not be able to delete the file
> because another Emacs instance may keep an open handle.
>
> It is this last Emacs instance that should delete the file, not the
> one that renamed it.

Mmmhh, probably also a lock file should be deposed by each Emacs using
the file?  That way the last session closing could check and remove?

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Achim Gratz]

Andrea Corallo writes:
> Mmmhh, probably also a lock file should be deposed by each Emacs using
> the file?  That way the last session closing could check and remove?

You might be interested in this:
https://archive.fosdem.org/2018/schedule/speaker/michael_haubenwallner/

(Gentoo PRefix running on Cygwin, which means that in-use libraries need
to be replaceable.)


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Loading a new compilation unit B that redefines the same functions
>> defined by A does not guarantees much, some of the old definitions of A
>> could be still in use somewhere, in that case A will not be collected.
>
>> So yeah I think we need a specific mechanism (kill-emacs hook as you
>> suggest) to do the cleanup when Emacs goes down.
>
> I was thinking about something like this:
>
> 1) Call `native-comp-unload`.
>
> 2) This should inspect the eln file and put all the subrs defined in it on a
> list. This should also copy the original bytecode from the eln file and store it
> somewhere.
>
> 3) Then `garbage-collect` is called. This should find all references to the
> subrs in the list and swap them atomically for references to functions
> from the bytecode.
>
> 4) After the previous step the GC should be able to collect the DLL handle
> knowing that no references to it remain.
>
> What do you think?

It can't work for the following reasons:

1- eln files do not retain the orginal function bytecode.

2- in any point of the code you can get the symbol-function of a defined
   function and leak it everywhere.  We can't technically "swap"
   functions definitions, the best we do is just redefine them that is
   set a certain symbol-function (what late load does).  The old
   definitions can always still be used if the referennce is still
   present somewhere.  Even worst the function you want to remove could
   be active in the stack!

One option would be to add a field into the compilation unit object like
'pending_for_removal', each time any of this is unloaded by GC the file
is removed if the field suggests that.  At the very last when Emacs is
closing we go through all the compilation units and remove the files
that are still pending.

>> No you are right, I don't use Windows since forever, I discovered from
>> this thread is not portable.  I guess we'll need to rename also here.
>
> But someone needs to delete the old eln file. Let's say that we use
> GetModuleFileNameA() to know if another Emacs instance has decided to rename the
> eln file and then we delete it on close if its suffix is ".eln.old".
>
> This algorithm has a big race condition though. If Emacs1 renames the file after
> Emacs2 has checked that it has not been renamed, then the file won't be deleted.
> If we put the "old eln" in the $TEMP folder this may not be a big issue though.

Yes renaming for me here was moving into a temporary folder.

This could be a very simple option also for the first problem if we
accept we can leave some file there from time to time.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> 1- eln files do not retain the orginal function bytecode.

That should be easy to change.

> in any point of the code you can get the symbol-function of a defined
   function and leak it everywhere.

This is why I said the GC should do it. It should be able to find all
references.

> We can't technically "swap" functions definitions, the best we do is just
> redefine them that is set a certain symbol-function (what late load does).

When the GC finds a Lisp_Object that points to a subr we want to unload,
it should replace it with one that points to its bytecode equivalent version.

> Even worst the function you want to remove could be active in the stack!

You are right. I hadn't considered this. It can still be done, but would need to
wait until the function finishes.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> 1- eln files do not retain the orginal function bytecode.
>
> That should be easy to change.

Sure but we don't want to do that because there's no reason to bloat the
eln.

>> in any point of the code you can get the symbol-function of a defined
>    function and leak it everywhere.
>
> This is why I said the GC should do it. It should be able to find all
> references.

Yes but I can't *remove* objects that are referenced niether swap them.
See below.

>> We can't technically "swap" functions definitions, the best we do is just
>> redefine them that is set a certain symbol-function (what late load does).
>
> When the GC finds a Lisp_Object that points to a subr we want to unload,
> it should replace it with one that points to its bytecode equivalent version.

No it cannot, if an object has been tested to say satisfy a predicate
you cannot change it for another one.  It would break tons of code and
make the system totally un-debuggable.

>> Even worst the function you want to remove could be active in the stack!
>
> You are right. I hadn't considered this. It can still be done, but would need to
> wait until the function finishes.

Yes, but when are all functions you want to get rid deactivated?

Here we are really complexifying a problem that is not.  IMO renaming
and having a list to do the clean-up are sufficient tools to solve it.

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> Here we are really complexifying a problem that is not.  IMO renaming
> and having a list to do the clean-up are sufficient tools to solve it.

You're right. I just think that renaming and a list do not guarantee that we
always delete the files when we should.

Consider this case:

* Emacs1 and Emacs2 are two Emacs instances that load the
same foo.eln file.

* Emacs1 decides to recompile foo.el. To do that it renames foo.eln to
  foo.eln.old. It creates a new foo.eln and tries to delete foo.eln.old. It
  fails because Emacs2 has an open handle to it.

* When Emacs2 closes it realizes that foo.eln has been renamed to foo.eln.old
  and deletes it.

That is the good case.

If we are unlucky this is what may happen:

* Emacs2 begins to close. It checks that foo.eln has not been renamed. Therefore
  it does not delete it. But it does not call FreeLibrary yet.

* Emacs1 renames foo.eln to foo.eln.old. It tries to delete it but fails since
  Emacs2 has an open handle.

* Emacs2 finally calls FreeLibrary() and closes.

In this case we are left over with a stale foo.eln.old. I don't think we can
have a race free algorithm.

We could add a "GC" step to `load`, where it tries to find stale .eln.old
files and removes them.

Nicolas.

El jue., 14 may. 2020 a las 15:13, Andrea Corallo (<akrl@sdf.org>) escribió:
>
> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>
> >> 1- eln files do not retain the orginal function bytecode.
> >
> > That should be easy to change.
>
> Sure but we don't want to do that because there's no reason to bloat the
> eln.
>
> >> in any point of the code you can get the symbol-function of a defined
> >    function and leak it everywhere.
> >
> > This is why I said the GC should do it. It should be able to find all
> > references.
>
> Yes but I can't *remove* objects that are referenced niether swap them.
> See below.
>
> >> We can't technically "swap" functions definitions, the best we do is just
> >> redefine them that is set a certain symbol-function (what late load does).
> >
> > When the GC finds a Lisp_Object that points to a subr we want to unload,
> > it should replace it with one that points to its bytecode equivalent version.
>
> No it cannot, if an object has been tested to say satisfy a predicate
> you cannot change it for another one.  It would break tons of code and
> make the system totally un-debuggable.
>
> >> Even worst the function you want to remove could be active in the stack!
> >
> > You are right. I hadn't considered this. It can still be done, but would need to
> > wait until the function finishes.
>
> Yes, but when are all functions you want to get rid deactivated?
>
> Here we are really complexifying a problem that is not.  IMO renaming
> and having a list to do the clean-up are sufficient tools to solve it.
>
> --
> akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Here we are really complexifying a problem that is not.  IMO renaming
>> and having a list to do the clean-up are sufficient tools to solve it.
>
> You're right. I just think that renaming and a list do not guarantee that we
> always delete the files when we should.
>
> Consider this case:
>
> * Emacs1 and Emacs2 are two Emacs instances that load the
> same foo.eln file.
>
> * Emacs1 decides to recompile foo.el. To do that it renames foo.eln to
>   foo.eln.old. It creates a new foo.eln and tries to delete foo.eln.old. It
>   fails because Emacs2 has an open handle to it.
>
> * When Emacs2 closes it realizes that foo.eln has been renamed to foo.eln.old
>   and deletes it.
>
> That is the good case.
>
> If we are unlucky this is what may happen:
>
> * Emacs2 begins to close. It checks that foo.eln has not been renamed. Therefore
>   it does not delete it. But it does not call FreeLibrary yet.
>
> * Emacs1 renames foo.eln to foo.eln.old. It tries to delete it but fails since
>   Emacs2 has an open handle.
>
> * Emacs2 finally calls FreeLibrary() and closes.
>
> In this case we are left over with a stale foo.eln.old. I don't think we can
> have a race free algorithm.
>
> We could add a "GC" step to `load`, where it tries to find stale .eln.old
> files and removes them.
>
> Nicolas.

I see.  But I suspect it could work just if each Emacs sessions depose a
file to signal is activelly using a certain .eln.

The last session can retrive the current filename of the handle and
delete it.

Do you think it works?

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> Do you think it works?

I don't know. What do you have in mind?

Another option:

We could use a global mutex shared between all Emacs processes.
https://docs.microsoft.com/en-us/windows/win32/sync/mutex-objects

They would have to acquire that mutex before any operation using eln files.
This is a complicated solution, though.

Nicolás.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Do you think it works?
>
> I don't know. What do you have in mind?

Not much more than that.

As you said the problem is to decide who has the duty to remove the file
at last.  If each Emacs deposes a file says .foo.eln-pidxxx for the
whole time is using foo.eln should be easy for the last Emacs to
understand it is really the last and has to do the clean-up in the case
foo.eln was renamed in foo.eln*whatever

I think it could work. (?)

> Another option:
>
> We could use a global mutex shared between all Emacs processes.
> https://docs.microsoft.com/en-us/windows/win32/sync/mutex-objects
>
> They would have to acquire that mutex before any operation using eln files.
> This is a complicated solution, though.
>
> Nicolás.
>

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> As you said the problem is to decide who has the duty to remove the file
> at last.  If each Emacs deposes a file says .foo.eln-pidxxx for the
> whole time is using foo.eln should be easy for the last Emacs to
> understand it is really the last and has to do the clean-up in the case
> foo.eln was renamed in foo.eln*whatever

That file would be create when opening foo.eln, but when
Emacs is closing we don't know what file it refers to:
- foo.eln
- foo.eln.old
- foo.eln.old2
- foo.eln.oldN

These last files could be created if foo.eln.old exists at the time of renaming.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> As you said the problem is to decide who has the duty to remove the file
>> at last.  If each Emacs deposes a file says .foo.eln-pidxxx for the
>> whole time is using foo.eln should be easy for the last Emacs to
>> understand it is really the last and has to do the clean-up in the case
>> foo.eln was renamed in foo.eln*whatever
>
> That file would be create when opening foo.eln, but when
> Emacs is closing we don't know what file it refers to:
> - foo.eln
> - foo.eln.old
> - foo.eln.old2
> - foo.eln.oldN
>
> These last files could be created if foo.eln.old exists at the time of renaming.

Yes, but I think we could say: the last Emacs closing that used any file
that was (at a certain point in life) foo.eln removes all the old
foo.eln.*

If is the last using any of the foo.eln* it can do that safely no?

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> Yes, but I think we could say: the last Emacs closing that used any file
> that was (at a certain point in life) foo.eln removes all the old
> foo.eln.*

I think this would work :).
We could even remove the pid file. Just do the equivalent of `rm $ELN.old*`
after FreeLibrary(). If the deletion fails then that means that another Emacs
has loaded that file. It would take of files left over from crashes too.

We would need to change `package-delete` though. It would no longer fully
delete the directory. Maybe other functions in `package.el` would need
to be updated to deal with these changes.

Nicolas.

El jue., 14 may. 2020 a las 16:58, Andrea Corallo (<akrl@sdf.org>) escribió:
>
> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>
> >> As you said the problem is to decide who has the duty to remove the file
> >> at last.  If each Emacs deposes a file says .foo.eln-pidxxx for the
> >> whole time is using foo.eln should be easy for the last Emacs to
> >> understand it is really the last and has to do the clean-up in the case
> >> foo.eln was renamed in foo.eln*whatever
> >
> > That file would be create when opening foo.eln, but when
> > Emacs is closing we don't know what file it refers to:
> > - foo.eln
> > - foo.eln.old
> > - foo.eln.old2
> > - foo.eln.oldN
> >
> > These last files could be created if foo.eln.old exists at the time of renaming.
>
> Yes, but I think we could say: the last Emacs closing that used any file
> that was (at a certain point in life) foo.eln removes all the old
> foo.eln.*
>
> If is the last using any of the foo.eln* it can do that safely no?
>
> --
> akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Yes, but I think we could say: the last Emacs closing that used any file
>> that was (at a certain point in life) foo.eln removes all the old
>> foo.eln.*
>
> I think this would work :).

Very good

> We could even remove the pid file. Just do the equivalent of `rm $ELN.old*`
> after FreeLibrary(). If the deletion fails then that means that another Emacs
> has loaded that file. It would take of files left over from crashes too.

Ah okay I thought (probably had to read better) something goes wrong if
you remove when you should not.  Then is even easier yes!

> We would need to change `package-delete` though. It would no longer fully
> delete the directory. Maybe other functions in `package.el` would need
> to be updated to deal with these changes.

If you diff the full branch I had to adjust few thing in Emacs too to
have it working, I believe is expected.  You'll check for the presence
of the native compiler in Lisp with the function you've introduced in
one of your patches.

I believe also that the renaming mechanism should be transparent on all
posix where is not necessary.

Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> We would need to change `package-delete` though. It would no longer fully
> delete the directory. Maybe other functions in `package.el` would need
> to be updated to deal with these changes.

Maybe not. We could move the .eln to the parent of the argument to
`delete-directory`.

For example move

elpa\28.0\develop\company-20200510.1614\eln-x86_64-w64-mingw32-683c5a1b96c51930\company.eln

to

elpa\28.0\develop\company.eln.old

The code to find the .eln.old files to delete would have to check if
it is a package it is dealing with and search for .eln.old files in an
upper folder.

Nicolas.

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 16:48:06 -0300
> Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> 
> > As you said the problem is to decide who has the duty to remove the file
> > at last.  If each Emacs deposes a file says .foo.eln-pidxxx for the
> > whole time is using foo.eln should be easy for the last Emacs to
> > understand it is really the last and has to do the clean-up in the case
> > foo.eln was renamed in foo.eln*whatever
> 
> That file would be create when opening foo.eln, but when
> Emacs is closing we don't know what file it refers to:
> - foo.eln
> - foo.eln.old
> - foo.eln.old2
> - foo.eln.oldN
> 
> These last files could be created if foo.eln.old exists at the time of renaming.

We could try deleting all of them.  Those that are still in use will
not be deleted.

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 16:00:02 -0300
> Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> 
> We could use a global mutex shared between all Emacs processes.
> https://docs.microsoft.com/en-us/windows/win32/sync/mutex-objects
> 
> They would have to acquire that mutex before any operation using eln files.
> This is a complicated solution, though.

If we want a mutex, we can use the offending .eln file as that mutex:
if deletion fails, the mutex is taken.

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 15:40:28 -0300
> Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> 
> * Emacs1 and Emacs2 are two Emacs instances that load the
> same foo.eln file.
> 
> * Emacs1 decides to recompile foo.el. To do that it renames foo.eln to
>   foo.eln.old. It creates a new foo.eln and tries to delete foo.eln.old. It
>   fails because Emacs2 has an open handle to it.
> 
> * When Emacs2 closes it realizes that foo.eln has been renamed to foo.eln.old
>   and deletes it.

How can Emacs2 realize that foo.eln was renamed?

> If we are unlucky this is what may happen:
> 
> * Emacs2 begins to close. It checks that foo.eln has not been renamed. Therefore
>   it does not delete it. But it does not call FreeLibrary yet.
> 
> * Emacs1 renames foo.eln to foo.eln.old. It tries to delete it but fails since
>   Emacs2 has an open handle.
> 
> * Emacs2 finally calls FreeLibrary() and closes.
> 
> In this case we are left over with a stale foo.eln.old. I don't think we can
> have a race free algorithm.

We will have to tell users not to do that on Windows.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> How can Emacs2 realize that foo.eln was renamed?

Using GetModuleFileNameA
https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-getmodulefilenamea

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 16:36:49 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > How can Emacs2 realize that foo.eln was renamed?
> 
> Using GetModuleFileNameA
> https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-getmodulefilenamea

Does that reflect renaming _after_ the library was loaded?

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> Does that reflect renaming _after_ the library was loaded?

I get that idea from the documentation. I haven't actually tested it myself.

The ProcessExplorer utility can show the name of the loaded DLLs
of a process. If I rename one the changed name will show up in ProcessExplorer.
I wonder how they do it. Too bad we don't have its source code.

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Fri, 15 May 2020 09:33:32 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > Does that reflect renaming _after_ the library was loaded?
> 
> I get that idea from the documentation. I haven't actually tested it myself.

It should be easy to write a program to test this.

> The ProcessExplorer utility can show the name of the loaded DLLs
> of a process. If I rename one the changed name will show up in ProcessExplorer.
> I wonder how they do it.

They use methods that are generally unacceptable in general-purpose
applications, including injecting code into other processes, using
undocumented features and APIs, etc.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

To summarize:

The best idea seems to be to rename the .eln file when removing the package or
recompiling. We need to reach a consensus on where to put the old .eln file,
though.

There are two options:

- Put it in the same folder as the original .eln file. This means that
  `package-delete` will not be able to delete the directory. Additionally, it
  will be tricky to handle left over files from an instance of Emacs that
  crashes.

- Another option is to move them to `package-user-dir`. This option means that
  `package-delete` will be able to delete the directory.

What option do you prefer?

The implementation I have in mind is roughly like this:

- `package-delete` iterates over the .eln files in the package directory. It
  tries to delete it, if it fails it is moved to somewhere (see point above).

- When Emacs GCs a native compilation unit it should check if it has been
  renamed (need to check if GetModuleFileNameA is fit for this). If it has, it
  tries to delete it. If it fails, then some other Emacs instance must be using
  it.

- The last step before calling exit() should FreeLibrary() all remaining .eln
  files and run the equivalent of `rm $package_user_dir/*.eln.old`.

I think this would work and should be simple to implement. If I get your OK I'll
try to do it this weekend.

Nicolas

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Fri, 15 May 2020 16:44:04 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> To summarize:
> 
> The best idea seems to be to rename the .eln file when removing the package or
> recompiling. We need to reach a consensus on where to put the old .eln file,
> though.
> 
> There are two options:
> 
> - Put it in the same folder as the original .eln file. This means that
>   `package-delete` will not be able to delete the directory. Additionally, it
>   will be tricky to handle left over files from an instance of Emacs that
>   crashes.
> 
> - Another option is to move them to `package-user-dir`. This option means that
>   `package-delete` will be able to delete the directory.
> 
> What option do you prefer?

I'm not sure I understand why we are talking only about package.el.
Wouldn't the same problem happen if the user recompiles a .el file for
which a .eln file already exists and is loaded into the session?  And
similarly when Emacs is being built and all the *.el files are being
compiled or recompiled, sometimes by several Emacs processes running
in parallel via "make -jN"?

I think the solution should handle all of these use cases, not just
that of package.el upgrading a package.  Do you agree?

> - `package-delete` iterates over the .eln files in the package directory. It
>   tries to delete it, if it fails it is moved to somewhere (see point above).
> 
> - When Emacs GCs a native compilation unit it should check if it has been
>   renamed (need to check if GetModuleFileNameA is fit for this). If it has, it
>   tries to delete it. If it fails, then some other Emacs instance must be using
>   it.
> 
> - The last step before calling exit() should FreeLibrary() all remaining .eln
>   files and run the equivalent of `rm $package_user_dir/*.eln.old`.

Sounds OK to me, I don't think we came up with anything better during
the discussion till now.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
>> Date: Fri, 15 May 2020 16:44:04 -0300
>> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
>> 
>> To summarize:
>> 
>> The best idea seems to be to rename the .eln file when removing the package or
>> recompiling. We need to reach a consensus on where to put the old .eln file,
>> though.
>> 
>> There are two options:
>> 
>> - Put it in the same folder as the original .eln file. This means that
>>   `package-delete` will not be able to delete the directory. Additionally, it
>>   will be tricky to handle left over files from an instance of Emacs that
>>   crashes.
>> 
>> - Another option is to move them to `package-user-dir`. This option means that
>>   `package-delete` will be able to delete the directory.
>> 
>> What option do you prefer?
>
> I'm not sure I understand why we are talking only about package.el.
> Wouldn't the same problem happen if the user recompiles a .el file for
> which a .eln file already exists and is loaded into the session?

IIUC the complication of package.el that makes it different from the
general cases you have described is that the eln-... sub-directory
has to be removed to be able to clean-up entirely the package.

> And
> similarly when Emacs is being built and all the *.el files are being
> compiled or recompiled, sometimes by several Emacs processes running
> in parallel via "make -jN"?
>
> I think the solution should handle all of these use cases, not just
> that of package.el upgrading a package.  Do you agree?
>
>> - `package-delete` iterates over the .eln files in the package directory. It
>>   tries to delete it, if it fails it is moved to somewhere (see point above).
>> 
>> - When Emacs GCs a native compilation unit it should check if it has been
>>   renamed (need to check if GetModuleFileNameA is fit for this). If it has, it
>>   tries to delete it. If it fails, then some other Emacs instance must be using
>>   it.
>> 
>> - The last step before calling exit() should FreeLibrary() all remaining .eln
>>   files and run the equivalent of `rm $package_user_dir/*.eln.old`.
>
> Sounds OK to me, I don't think we came up with anything better during
> the discussion till now.
>

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 2926 bytes --]

> I'm not sure I understand why we are talking only about package.el.
> Wouldn't the same problem happen if the user recompiles a .el file for
> which a .eln file already exists and is loaded into the session?

True, right now the .eln would not be removed. Not even in Posix.
Therefore it will continue to be loaded unless `load-prefer-newer` is true.

> And
> similarly when Emacs is being built and all the *.el files are being
> compiled or recompiled, sometimes by several Emacs processes running
> in parallel via "make -jN"

Help me understand. Are you referring to the case where a developer changes
an .el file for which an .eln file (now outdated) already exists? I think
fixing the
case above will fix this one.


El sáb., 16 may. 2020 a las 3:22, Eli Zaretskii (<eliz@gnu.org>) escribió:

> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Fri, 15 May 2020 16:44:04 -0300
> > Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> >
> > To summarize:
> >
> > The best idea seems to be to rename the .eln file when removing the
> package or
> > recompiling. We need to reach a consensus on where to put the old .eln
> file,
> > though.
> >
> > There are two options:
> >
> > - Put it in the same folder as the original .eln file. This means that
> >   `package-delete` will not be able to delete the directory.
> Additionally, it
> >   will be tricky to handle left over files from an instance of Emacs that
> >   crashes.
> >
> > - Another option is to move them to `package-user-dir`. This option
> means that
> >   `package-delete` will be able to delete the directory.
> >
> > What option do you prefer?
>
> I'm not sure I understand why we are talking only about package.el.
> Wouldn't the same problem happen if the user recompiles a .el file for
> which a .eln file already exists and is loaded into the session?  And
> similarly when Emacs is being built and all the *.el files are being
> compiled or recompiled, sometimes by several Emacs processes running
> in parallel via "make -jN"?
>
> I think the solution should handle all of these use cases, not just
> that of package.el upgrading a package.  Do you agree?
>
> > - `package-delete` iterates over the .eln files in the package
> directory. It
> >   tries to delete it, if it fails it is moved to somewhere (see point
> above).
> >
> > - When Emacs GCs a native compilation unit it should check if it has been
> >   renamed (need to check if GetModuleFileNameA is fit for this). If it
> has, it
> >   tries to delete it. If it fails, then some other Emacs instance must
> be using
> >   it.
> >
> > - The last step before calling exit() should FreeLibrary() all remaining
> .eln
> >   files and run the equivalent of `rm $package_user_dir/*.eln.old`.
>
> Sounds OK to me, I don't think we came up with anything better during
> the discussion till now.
>

[-- Attachment #2: Type: text/html, Size: 3772 bytes --]

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 16 May 2020 13:12:20 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > I'm not sure I understand why we are talking only about package.el.
> > Wouldn't the same problem happen if the user recompiles a .el file for
> > which a .eln file already exists and is loaded into the session?  
> 
> True, right now the .eln would not be removed. Not even in Posix.

No, on Posix systems we can delete the file, and it will be actually
deleted when its last handle is closed.  I believe this works with
shared libraries as well.

> > And
> > similarly when Emacs is being built and all the *.el files are being
> > compiled or recompiled, sometimes by several Emacs processes running
> > in parallel via "make -jN"
> 
> Help me understand. Are you referring to the case where a developer changes
> an .el file for which an .eln file (now outdated) already exists?

No, I mean building Emacs with "make -j10 bootstrap".

> I think fixing the case above will fix this one.

It's the same problem, yes.  Just a slightly different use case, which
could therefore have different probabilities for some aspects.  For
example, the probability of the same .el file being recompiled from
two separate sessions is relatively small, except when you consider
the "make -jN" use case.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 2719 bytes --]

> No, on Posix systems we can delete the file, and it will be actually
> deleted when its last handle is closed.  I believe this works with
> shared libraries as well.

Do we actually do it? I don't think so. I don't even know where exactly
that check should be. Maybe `eval-buffer`?

> It's the same problem, yes.  Just a slightly different use case, which
> could therefore have different probabilities for some aspects.  For
> example, the probability of the same .el file being recompiled from
> two separate sessions is relatively small, except when you consider
> the "make -jN" use case.

The probability of two Emacs recompiling the same file in the "make -jN"
case is
0. That is because the build system tells each Emacs to compile one and
only one
file that is passed as a parameter. See #41329 for a bug related to this.

The case of two Emacs sessions recompiling the same file at the same time is
actually a problem.

We could implement the following algorithm:

- Have libgccjit write the .eln to a temporary name in the destination
folder.

- While "foo.eln" exists:
  - Rename it to "foo.eln.oldN".
  - Move the new .eln file to "foo.eln"

Nicolas


El sáb., 16 may. 2020 a las 13:19, Eli Zaretskii (<eliz@gnu.org>) escribió:

> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Sat, 16 May 2020 13:12:20 -0300
> > Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> >
> > > I'm not sure I understand why we are talking only about package.el.
> > > Wouldn't the same problem happen if the user recompiles a .el file for
> > > which a .eln file already exists and is loaded into the session?
> >
> > True, right now the .eln would not be removed. Not even in Posix.
>
> No, on Posix systems we can delete the file, and it will be actually
> deleted when its last handle is closed.  I believe this works with
> shared libraries as well.
>
> > > And
> > > similarly when Emacs is being built and all the *.el files are being
> > > compiled or recompiled, sometimes by several Emacs processes running
> > > in parallel via "make -jN"
> >
> > Help me understand. Are you referring to the case where a developer
> changes
> > an .el file for which an .eln file (now outdated) already exists?
>
> No, I mean building Emacs with "make -j10 bootstrap".
>
> > I think fixing the case above will fix this one.
>
> It's the same problem, yes.  Just a slightly different use case, which
> could therefore have different probabilities for some aspects.  For
> example, the probability of the same .el file being recompiled from
> two separate sessions is relatively small, except when you consider
> the "make -jN" use case.
>

[-- Attachment #2: Type: text/html, Size: 3697 bytes --]

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 16 May 2020 13:31:41 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > No, on Posix systems we can delete the file, and it will be actually
> > deleted when its last handle is closed.  I believe this works with
> > shared libraries as well.
> 
> Do we actually do it? I don't think so. I don't even know where exactly
> that check should be. Maybe `eval-buffer`?

I'm not sure I understand what check did you have in mind.

> > It's the same problem, yes.  Just a slightly different use case, which
> > could therefore have different probabilities for some aspects.  For
> > example, the probability of the same .el file being recompiled from
> > two separate sessions is relatively small, except when you consider
> > the "make -jN" use case.
> 
> The probability of two Emacs recompiling the same file in the "make -jN" case is
> 0.

Sorry, I meant the probability of one session compiling a file while
another uses it.

> The case of two Emacs sessions recompiling the same file at the same time is
> actually a problem.
> 
> We could implement the following algorithm:
> 
> - Have libgccjit write the .eln to a temporary name in the destination folder.
> 
> - While "foo.eln" exists:
>   - Rename it to "foo.eln.oldN".
>   - Move the new .eln file to "foo.eln"

Something like that, yes.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1857 bytes --]

> I'm not sure I understand what check did you have in mind.

I misunderstood your first message. I thought you were talking about
deleting
the .eln file when it no longer is up to date after the .el file changes.

If we want that we will have to add it to `auto-compile.el`.

El sáb., 16 may. 2020 a las 13:42, Eli Zaretskii (<eliz@gnu.org>) escribió:

> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Sat, 16 May 2020 13:31:41 -0300
> > Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> >
> > > No, on Posix systems we can delete the file, and it will be actually
> > > deleted when its last handle is closed.  I believe this works with
> > > shared libraries as well.
> >
> > Do we actually do it? I don't think so. I don't even know where exactly
> > that check should be. Maybe `eval-buffer`?
>
> I'm not sure I understand what check did you have in mind.
>
> > > It's the same problem, yes.  Just a slightly different use case, which
> > > could therefore have different probabilities for some aspects.  For
> > > example, the probability of the same .el file being recompiled from
> > > two separate sessions is relatively small, except when you consider
> > > the "make -jN" use case.
> >
> > The probability of two Emacs recompiling the same file in the "make -jN"
> case is
> > 0.
>
> Sorry, I meant the probability of one session compiling a file while
> another uses it.
>
> > The case of two Emacs sessions recompiling the same file at the same
> time is
> > actually a problem.
> >
> > We could implement the following algorithm:
> >
> > - Have libgccjit write the .eln to a temporary name in the destination
> folder.
> >
> > - While "foo.eln" exists:
> >   - Rename it to "foo.eln.oldN".
> >   - Move the new .eln file to "foo.eln"
>
> Something like that, yes.
>

[-- Attachment #2: Type: text/html, Size: 2582 bytes --]

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1.1: Type: text/plain, Size: 133 bytes --]

I have written the attached patch to handle the deletion
of native compilation units still in use. I have tested that it works well.

[-- Attachment #1.2: Type: text/html, Size: 180 bytes --]

[-- Attachment #2: 0001-Improve-handling-of-native-compilation-units-still-i.patch --]
[-- Type: application/octet-stream, Size: 19734 bytes --]

From adfcca38f76af3796fe8cc59684ed5e24987f24c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Tue, 19 May 2020 15:57:31 -0300
Subject: [PATCH] Improve handling of native compilation units still in use in
 Windows

When closing emacs will inspect all directories from which it loaded
native compilation units. If it finds a ".eln.old" file it will try to
delete it, if it fails that means that another Emacs instance is using it.

When compiling a file we rename the file that was in the output path
in case it has been loaded into another Emacs instance.

When deleting a package we move any ".eln" or ".eln.old" files in the
package folder that we can't delete to `package-user-dir`. Emacs will
check that directory when closing and delete them.

* lisp/emacs-lisp/comp.el (comp--replace-output-file): Function called
from C code to finish the compilation process. It performs renaming of
the old file if necessary.
* lisp/emacs-lisp/package.el (package--delete-directory): Function to
delete a package directory. It moves native compilation units that it
can't delete to `package-user-dir'.
* src/alloc.c (cleanup_vector): Call dispose_comp_unit().
  (garbage_collect): Call finish_delayed_disposal_of_comp_units().
* src/comp.c: Restore the signal mask using unwind-protect. Store
loaded native compilation units in a hash table for disposal on
close. Store filenames of native compilation units GC'd in a linked
list to finish their disposal when the GC is over.
* src/comp.h: Introduce cfile member in Lisp_Native_Comp_Unit.
Add declarations of functions that: clean directories of unused native
compilation units, handle disposal of native compilation units.
* src/emacs.c (kill-emacs): Dispose all remaining compilation units
right right before calling exit().
* src/eval.c (internal_condition_case_3, internal_condition_case_4):
Add functions.
* src/lisp.h (internal_condition_case_3, internal_condition_case_4):
Add functions.
* src/pdumper.c (dump_do_dump_relocation): Set cfile to a copy of the
Lisp string specifying the file path.
---
 lisp/emacs-lisp/comp.el    |  25 ++++++
 lisp/emacs-lisp/package.el |  27 +++++-
 src/alloc.c                |   5 +-
 src/comp.c                 | 174 ++++++++++++++++++++++++++++++++++---
 src/comp.h                 |  27 ++++++
 src/emacs.c                |   3 +
 src/eval.c                 |  55 ++++++++++++
 src/lisp.h                 |   2 +
 src/pdumper.c              |   3 +
 9 files changed, 308 insertions(+), 13 deletions(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 012baf2560..1fb4cd98c0 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -2183,6 +2183,31 @@ comp-hint-cons
 \f
 ;; Some entry point support code.
 
+(defun comp--replace-output-file (outfile tmpfile)
+  "Replace OUTFILE with TMPFILE taking the necessary steps when
+dealing with shared libraries that may be loaded into Emacs"
+  (cond ((eq 'windows-nt system-type)
+         (ignore-errors (delete-file outfile))
+         (let ((retry t))
+           (while retry
+             (setf retry nil)
+             (condition-case _
+                 (progn
+                   ;; outfile maybe recreated by another Emacs in
+                   ;; between the following two rename-file calls
+                   (if (file-exists-p outfile)
+                       (rename-file outfile (make-temp-file-internal
+                                             (file-name-sans-extension outfile)
+                                             nil ".eln.old" nil)
+                                    t))
+                   (rename-file tmpfile outfile nil))
+               (file-already-exists (setf retry t))))))
+        ;; Remove the old eln instead of copying the new one into it
+        ;; to get a new inode and prevent crashes in case the old one
+        ;; is currently loaded.
+        (t (delete-file outfile)
+           (rename-file tmpfile outfile))))
+
 (defvar comp-files-queue ()
   "List of Elisp files to be compiled.")
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 95659840ad..368057e955 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2184,6 +2184,31 @@ package--newest-p
   (equal (cadr (assq (package-desc-name pkg) package-alist))
          pkg))
 
+(defun package--delete-directory (dir)
+  "Delete DIR recursively.
+In Windows move .eln and .eln.old files that can not be deleted to `package-user-dir'."
+  (cond ((eq 'windows-nt system-type)
+         (let ((retry t))
+           (while retry
+             (setf retry nil)
+             (condition-case err
+                 (delete-directory dir t)
+               (file-error
+                (if (and (string= "Removing old name" (cadr err))
+                         (string= "Permission denied" (caddr err))
+                         (or (string-suffix-p ".eln" (cadddr err))
+                             (string-suffix-p ".eln.old" (cadddr err))))
+                    (progn
+                      (rename-file (cadddr err)
+                                   (make-temp-file-internal
+                                    (concat package-user-dir
+                                            (file-name-base (cadddr err)))
+                                    nil ".eln.old" nil)
+                                   t)
+                      (setf retry t))
+                  (signal (car err) (cdr err))))))))
+        (t (delete-directory dir t)))
+
 (defun package-delete (pkg-desc &optional force nosave)
   "Delete package PKG-DESC.
 
@@ -2236,7 +2261,7 @@ package-delete
                   (package-desc-name pkg-used-elsewhere-by)))
           (t
            (add-hook 'post-command-hook #'package-menu--post-refresh)
-           (delete-directory dir t)
+           (package--delete-directory dir)
            ;; Remove NAME-VERSION.signed and NAME-readme.txt files.
            ;;
            ;; NAME-readme.txt files are no longer created, but they
diff --git a/src/alloc.c b/src/alloc.c
index f2b80fac88..17f5e15b35 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -3119,8 +3119,7 @@ cleanup_vector (struct Lisp_Vector *vector)
     {
       struct Lisp_Native_Comp_Unit *cu =
 	PSEUDOVEC_STRUCT (vector, Lisp_Native_Comp_Unit);
-      eassert (cu->handle);
-      dynlib_close (cu->handle);
+      dispose_comp_unit (cu, true);
     }
 }
 
@@ -6119,6 +6118,8 @@ garbage_collect (void)
       if (tot_after < tot_before)
 	malloc_probe (min (tot_before - tot_after, SIZE_MAX));
     }
+
+  finish_delayed_disposal_of_comp_units ();
 }
 
 DEFUN ("garbage-collect", Fgarbage_collect, Sgarbage_collect, 0, 0, "",
diff --git a/src/comp.c b/src/comp.c
index b43d3eddb3..35e1ec0da3 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -413,6 +413,10 @@ #define THIRD(x)				\
 #define CALL1I(fun, arg)				\
   CALLN (Ffuncall, intern_c_string (STR (fun)), arg)
 
+/* Like call2 but stringify and intern.  */
+#define CALL2I(fun, arg1, arg2)				\
+  CALLN (Ffuncall, intern_c_string (STR (fun)), arg1, arg2)
+
 #define DECL_BLOCK(name, func)				\
   gcc_jit_block *(name) =				\
     gcc_jit_function_new_block ((func), STR (name))
@@ -3830,6 +3834,14 @@ DEFUN ("comp--release-ctxt", Fcomp__release_ctxt, Scomp__release_ctxt,
   return Qt;
 }
 
+sigset_t oldset;
+
+static void restore_sigmask(void)
+{
+  pthread_sigmask (SIG_SETMASK, &oldset, 0);
+  unblock_input ();
+}
+
 DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
        Scomp__compile_ctxt_to_file,
        1, 1, 0,
@@ -3851,6 +3863,8 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
     CALL1I (comp-data-container-idx, CALL1I (comp-ctxt-d-ephemeral, Vcomp_ctxt));
 
   sigset_t oldset;
+  ptrdiff_t count;
+
   if (!noninteractive)
     {
       sigset_t blocked;
@@ -3863,6 +3877,8 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
       sigaddset (&blocked, SIGIO);
 #endif
       pthread_sigmask (SIG_BLOCK, &blocked, &oldset);
+      count = SPECPDL_INDEX ();
+      record_unwind_protect_void(restore_sigmask);
     }
   emit_ctxt_code ();
 
@@ -3902,18 +3918,10 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
 				   GCC_JIT_OUTPUT_KIND_DYNAMIC_LIBRARY,
 				   SSDATA (tmp_file));
 
-  /* Remove the old eln instead of copying the new one into it to get
-     a new inode and prevent crashes in case the old one is currently
-     loaded.  */
-  if (!NILP (Ffile_exists_p (out_file)))
-    Fdelete_file (out_file, Qnil);
-  Frename_file (tmp_file, out_file, Qnil);
+  CALL2I(comp--replace-output-file, out_file, tmp_file);
 
   if (!noninteractive)
-    {
-      pthread_sigmask (SIG_SETMASK, &oldset, 0);
-      unblock_input ();
-    }
+    unbind_to(count, Qnil);
 
   return out_file;
 }
@@ -3974,6 +3982,138 @@ helper_PSEUDOVECTOR_TYPEP_XUNTAG (Lisp_Object a, enum pvec_type code)
 			     code);
 }
 
+\f
+/*********************************/
+/* Disposal of compilation units */
+/*********************************/
+
+#ifdef WINDOWSNT
+#define OLD_ELN_SUFFIX_REGEXP build_string("\\.eln\\.old$")
+
+static Lisp_Object all_loaded_comp_units;
+
+struct delayed_comp_unit_disposal
+{
+  struct delayed_comp_unit_disposal * next;
+  char * filename;
+};
+
+struct delayed_comp_unit_disposal * delayed_comp_unit_disposal_list;
+
+static Lisp_Object
+returnQnil (Lisp_Object arg)
+{
+  return Qnil;
+}
+
+static void
+clean_comp_unit_directory (Lisp_Object filepath)
+{
+  if (NILP (filepath))
+    return;
+  Lisp_Object files_in_dir;
+  files_in_dir = internal_condition_case_4(Fdirectory_files, filepath, Qt,
+                                           OLD_ELN_SUFFIX_REGEXP, Qnil, Qt, returnQnil);
+  FOR_EACH_TAIL(files_in_dir)
+    {
+      DeleteFile (SSDATA (XCAR (files_in_dir)));
+    }
+}
+
+void clean_package_user_dir_of_old_comp_units (void)
+{
+  Lisp_Object package_user_dir = find_symbol_value (intern ("package-user-dir"));
+  if (EQ(package_user_dir, Qunbound) || !STRINGP(package_user_dir))
+    return;
+
+  clean_comp_unit_directory(package_user_dir);
+}
+
+#endif
+
+void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_handle, bool delay)
+{
+  eassert (comp_handle->handle);
+  dynlib_close (comp_handle->handle);
+#ifdef WINDOWSNT
+  if (!delay)
+    {
+      Lisp_Object dirname = internal_condition_case_1(Ffile_name_directory,
+                                                      build_string (comp_handle->cfile),
+                                                      Qt,
+                                                      returnQnil);
+      if (!NILP(dirname))
+        clean_comp_unit_directory (dirname);
+      xfree (comp_handle->cfile);
+      comp_handle->cfile = NULL;
+    }
+  else
+    {
+      struct delayed_comp_unit_disposal * head;
+      head = xmalloc (sizeof (struct delayed_comp_unit_disposal));
+      head->next = delayed_comp_unit_disposal_list;
+      head->filename = comp_handle->cfile;
+      comp_handle->cfile = NULL;
+      delayed_comp_unit_disposal_list = head;
+    }
+#else
+  xfree (comp_handle->file);
+#endif
+}
+
+static void
+register_native_comp_unit (Lisp_Object comp_u)
+{
+#ifdef WINDOWSNT
+  static EMACS_UINT count;
+
+  if (XFIXNUM(Fhash_table_count(all_loaded_comp_units)) >= MOST_POSITIVE_FIXNUM)
+    return;
+
+  while (!NILP(Fgethash(make_fixnum(count), all_loaded_comp_units, Qnil)))
+    count = (count + 1) % MOST_POSITIVE_FIXNUM;
+
+  Fputhash(make_fixnum(count), comp_u, all_loaded_comp_units);
+#endif
+}
+
+void dispose_all_remaining_comp_units (void)
+{
+#ifdef WINDOWSNT
+  struct Lisp_Hash_Table *h = XHASH_TABLE (all_loaded_comp_units);
+
+  for (ptrdiff_t i = 0; i < HASH_TABLE_SIZE (h); ++i)
+    {
+      Lisp_Object k = HASH_KEY (h, i);
+      if (!EQ (k, Qunbound))
+        {
+          Lisp_Object val = HASH_VALUE (h, i);
+          struct Lisp_Native_Comp_Unit * cu = XNATIVE_COMP_UNIT(val);
+          dispose_comp_unit(cu, false);
+        }
+    }
+#endif
+}
+
+void finish_delayed_disposal_of_comp_units (void)
+{
+#ifdef WINDOWSNT
+  for (struct delayed_comp_unit_disposal * item = delayed_comp_unit_disposal_list;
+       delayed_comp_unit_disposal_list;
+       item = delayed_comp_unit_disposal_list)
+    {
+      delayed_comp_unit_disposal_list = item->next;
+      Lisp_Object dirname
+        = internal_condition_case_1 (Ffile_name_directory,
+                                     build_string (item->filename), Qt,
+                                     returnQnil);
+      clean_comp_unit_directory (dirname);
+      xfree(item->filename);
+      xfree(item);
+    }
+#endif
+}
+
 \f
 /***********************************/
 /* Deferred compilation mechanism. */
@@ -4160,6 +4300,12 @@ load_comp_unit (struct Lisp_Native_Comp_Unit *comp_u, bool loading_dump,
       d_vec_len = XFIXNUM (Flength (comp_u->data_impure_vec));
       for (EMACS_INT i = 0; i < d_vec_len; i++)
 	data_imp_relocs[i] = AREF (comp_u->data_impure_vec, i);
+
+      /* If we register them while dumping we will get some entries in
+         the hash table that will be duplicated when pdumper calls
+         load_comp_unit. */
+      if (!will_dump_p())
+        register_native_comp_unit (comp_u_lisp_obj);
     }
 
   if (!loading_dump)
@@ -4273,6 +4419,9 @@ DEFUN ("native-elisp-load", Fnative_elisp_load, Snative_elisp_load, 1, 2, 0,
   if (!comp_u->handle)
     xsignal2 (Qnative_lisp_load_failed, file, build_string (dynlib_error ()));
   comp_u->file = file;
+#ifdef WINDOWSNT
+  comp_u->cfile = xlispstrdup(file);
+#endif
   comp_u->data_vec = Qnil;
   load_comp_unit (comp_u, false, !NILP (late_load));
 
@@ -4417,6 +4566,11 @@ syms_of_comp (void)
   staticpro (&delayed_sources);
   delayed_sources = Qnil;
 
+#ifdef WINDOWSNT
+  staticpro (&all_loaded_comp_units);
+  all_loaded_comp_units = CALLN(Fmake_hash_table, QCweakness, Qvalue);
+#endif
+
   DEFVAR_LISP ("comp-ctxt", Vcomp_ctxt,
 	       doc: /* The compiler context.  */);
   Vcomp_ctxt = Qnil;
diff --git a/src/comp.h b/src/comp.h
index e6ab32ff8e..89ef740fe6 100644
--- a/src/comp.h
+++ b/src/comp.h
@@ -44,7 +44,15 @@ #define COMP_H
   /* Same but for data that cannot be moved to pure space.
      Must be the last lisp object here.   */
   Lisp_Object data_impure_vec;
+
   dynlib_handle_ptr handle;
+#ifdef WINDOWSNT
+  /* We need to store a copy of the original file name in memory that
+     is not subject to GC because the function to dispose native
+     compilation units is called by the GC. By that time the `file'
+     string may have been sweeped. */
+  char * cfile;
+#endif
 };
 
 #ifdef HAVE_NATIVE_COMP
@@ -75,6 +83,14 @@ XNATIVE_COMP_UNIT (Lisp_Object a)
 
 extern void maybe_defer_native_compilation (Lisp_Object function_name,
 					    Lisp_Object definition);
+
+extern void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_unit, bool delay);
+
+extern void finish_delayed_disposal_of_comp_units (void);
+
+extern void dispose_all_remaining_comp_units (void);
+
+extern void clean_package_user_dir_of_old_comp_units (void);
 #else
 
 static inline void
@@ -84,6 +100,17 @@ maybe_defer_native_compilation (Lisp_Object function_name,
 
 extern void syms_of_comp (void);
 
+static inline void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_handle)
+{
+  emacs_abort();
+}
+
+static inline void dispose_all_remaining_comp_units (void)
+{}
+
+static inline void clean_package_user_dir_of_old_comp_units (void)
+{}
+
 #endif
 
 #endif
diff --git a/src/emacs.c b/src/emacs.c
index e75cb58834..b7c89b44ec 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -2393,6 +2393,9 @@ DEFUN ("kill-emacs", Fkill_emacs, Skill_emacs, 0, 1, "P",
       unlink (SSDATA (listfile));
     }
 
+  dispose_all_remaining_comp_units();
+  clean_package_user_dir_of_old_comp_units();
+
   if (FIXNUMP (arg))
     exit_code = (XFIXNUM (arg) < 0
 		 ? XFIXNUM (arg) | INT_MIN
diff --git a/src/eval.c b/src/eval.c
index 1091b08255..a68fc90285 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -1419,6 +1419,61 @@ internal_condition_case_2 (Lisp_Object (*bfun) (Lisp_Object, Lisp_Object),
     }
 }
 
+/* Like internal_condition_case_1 but call BFUN with ARG1, ARG2, ARG3 as
+   its arguments.  */
+
+Lisp_Object
+internal_condition_case_3 (Lisp_Object (*bfun) (Lisp_Object, Lisp_Object,
+                                                Lisp_Object),
+                           Lisp_Object arg1, Lisp_Object arg2, Lisp_Object arg3,
+                           Lisp_Object handlers,
+                           Lisp_Object (*hfun) (Lisp_Object))
+{
+  struct handler *c = push_handler (handlers, CONDITION_CASE);
+  if (sys_setjmp (c->jmp))
+    {
+      Lisp_Object val = handlerlist->val;
+      clobbered_eassert (handlerlist == c);
+      handlerlist = handlerlist->next;
+      return hfun (val);
+    }
+  else
+    {
+      Lisp_Object val = bfun (arg1, arg2, arg3);
+      eassert (handlerlist == c);
+      handlerlist = c->next;
+      return val;
+    }
+}
+
+/* Like internal_condition_case_1 but call BFUN with ARG1, ARG2, ARG3, ARG4 as
+   its arguments.  */
+
+Lisp_Object
+internal_condition_case_4 (Lisp_Object (*bfun) (Lisp_Object, Lisp_Object,
+                                                Lisp_Object, Lisp_Object),
+                           Lisp_Object arg1, Lisp_Object arg2,
+                           Lisp_Object arg3, Lisp_Object arg4,
+                           Lisp_Object handlers,
+                           Lisp_Object (*hfun) (Lisp_Object))
+{
+  struct handler *c = push_handler (handlers, CONDITION_CASE);
+  if (sys_setjmp (c->jmp))
+    {
+      Lisp_Object val = handlerlist->val;
+      clobbered_eassert (handlerlist == c);
+      handlerlist = handlerlist->next;
+      return hfun (val);
+    }
+  else
+    {
+      Lisp_Object val = bfun (arg1, arg2, arg3, arg4);
+      eassert (handlerlist == c);
+      handlerlist = c->next;
+      return val;
+    }
+}
+
 /* Like internal_condition_case but call BFUN with NARGS as first,
    and ARGS as second argument.  */
 
diff --git a/src/lisp.h b/src/lisp.h
index e242546d10..eeac20598c 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4138,6 +4138,8 @@ xsignal (Lisp_Object error_symbol, Lisp_Object data)
 extern Lisp_Object internal_condition_case (Lisp_Object (*) (void), Lisp_Object, Lisp_Object (*) (Lisp_Object));
 extern Lisp_Object internal_condition_case_1 (Lisp_Object (*) (Lisp_Object), Lisp_Object, Lisp_Object, Lisp_Object (*) (Lisp_Object));
 extern Lisp_Object internal_condition_case_2 (Lisp_Object (*) (Lisp_Object, Lisp_Object), Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object (*) (Lisp_Object));
+extern Lisp_Object internal_condition_case_3 (Lisp_Object (*) (Lisp_Object, Lisp_Object, Lisp_Object), Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object (*) (Lisp_Object));
+extern Lisp_Object internal_condition_case_4 (Lisp_Object (*) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object), Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object (*) (Lisp_Object));
 extern Lisp_Object internal_condition_case_n
     (Lisp_Object (*) (ptrdiff_t, Lisp_Object *), ptrdiff_t, Lisp_Object *,
      Lisp_Object, Lisp_Object (*) (Lisp_Object, ptrdiff_t, Lisp_Object *));
diff --git a/src/pdumper.c b/src/pdumper.c
index f837dfc38d..9b0bd472d6 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -5312,6 +5312,9 @@ dump_do_dump_relocation (const uintptr_t dump_base,
 	  concat2 (Vinvocation_directory,
 		   installation_state == LOCAL_BUILD
 		   ? XCDR (comp_u->file) : XCAR (comp_u->file));
+#ifdef WINDOWSNT
+        comp_u->cfile = xlispstrdup(comp_u->file);
+#endif
 	comp_u->handle = dynlib_open (SSDATA (comp_u->file));
 	if (!comp_u->handle)
 	  error ("%s", dynlib_error ());
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1.1: Type: text/plain, Size: 149 bytes --]

The following two patches handle minor issues:

* Removal of the emacs_root_dir() hack.
* Determine the number of processors for native compilation.

[-- Attachment #1.2: Type: text/html, Size: 219 bytes --]

[-- Attachment #2: 0001-Windows-Use-NUMBER_OF_PROCESSORS-environment-variabl.patch --]
[-- Type: application/octet-stream, Size: 1429 bytes --]

From 1fb5c38748c12735ee1d6eb6e794d61720e4fc11 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Wed, 13 May 2020 16:22:17 -0300
Subject: [PATCH] Windows: Use NUMBER_OF_PROCESSORS environment variable.

* lisp/emacs-lisp/comp.el (comp-effective-async-max-jobs): Use
NUMBER_OF_PROCESSORS environment variable if system is Windows NT,
"nproc" if it is in PATH or a default of 1.
---
 lisp/emacs-lisp/comp.el | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index d32f93a1e1..26bb79fcd1 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -2208,9 +2208,11 @@ comp-async-runnings
     (if (zerop comp-async-jobs-number)
         (or num-cpus
             (setf num-cpus
-                  ;; Half of the CPUs or at least one.
-                  ;; FIXME portable?
-                  (max 1 (/ (string-to-number (shell-command-to-string "nproc"))
+                  (max 1 (/ (cond ((eq 'windows-nt system-type)
+                                   (string-to-number (getenv "NUMBER_OF_PROCESSORS")))
+                                  ((executable-find "nproc")
+                                   (string-to-number (shell-command-to-string "nproc")))
+                                  (t 1))
                             2))))
       comp-async-jobs-number)))
 
-- 
2.25.1.windows.1


[-- Attachment #3: 0001-Determine-the-emacs-root-dir-only-when-necessary.patch --]
[-- Type: application/octet-stream, Size: 4337 bytes --]

From a7176a64d4d882874d4a81cc3924e03fa2a09ce8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Wed, 13 May 2020 18:31:42 -0300
Subject: [PATCH] Determine the emacs root dir only when necessary.

* src/fileio.c: Introduce function emacs_root_dir(). Refactor
`expand-file-name` to use it.
* src/lisp.h: Separate emacs_root_dir() into dos_emacs_root_dir() and
w32_emacs_root_dir().
* src/msdos.c: Rename emacs_root_dir() to dos_emacs_root_dir().
* src/w32.c: Rename emacs_root_dir() to w32_emacs_root_dir().
---
 src/fileio.c | 37 ++++++++++++++++++++++---------------
 src/lisp.h   | 11 +++++++----
 src/msdos.c  |  2 +-
 src/w32.c    |  2 +-
 4 files changed, 31 insertions(+), 21 deletions(-)

diff --git a/src/fileio.c b/src/fileio.c
index 2f1d2f8243..e20fa93c65 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -781,6 +781,18 @@ user_homedir (char const *name)
   return pw->pw_dir;
 }
 
+static Lisp_Object
+emacs_root_dir()
+{
+#ifdef DOS
+      return build_string (dos_emacs_root_dir ());
+#elif defined(WINDOWSNT)
+      return build_string (w32_emacs_root_dir ());
+#else
+      return build_string ("/");
+#endif
+}
+
 DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
        doc: /* Convert filename NAME to absolute, and canonicalize it.
 Second arg DEFAULT-DIRECTORY is directory to start with if NAME is relative
@@ -851,21 +863,16 @@ DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
     }
 
   /* As a last resort, we may have to use the root as
-     default_directory below.  */
-  Lisp_Object root;
-#ifdef DOS_NT
-      /* "/" is not considered a root directory on DOS_NT, so using it
-	 as default_directory causes an infinite recursion in, e.g.,
-	 the following:
+     default_directory below.
 
-            (let (default-directory)
-	      (expand-file-name "a"))
+     "/" is not considered a root directory on DOS_NT, so using it
+     as default_directory causes an infinite recursion in, e.g.,
+     the following:
 
-	 To avoid this, we use the root of the current drive.  */
-      root = build_string (emacs_root_dir ());
-#else
-      root = build_string ("/");
-#endif
+        (let (default-directory)
+          (expand-file-name "a"))
+
+     To avoid this, we use the root of the current drive.  */
 
   /* Use the buffer's default-directory if DEFAULT_DIRECTORY is omitted.  */
   if (NILP (default_directory))
@@ -891,13 +898,13 @@ DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
 	      Lisp_Object absdir
 		= STRINGP (Vinvocation_directory)
 		&& file_name_absolute_no_tilde_p (Vinvocation_directory)
-		? Vinvocation_directory : root;
+		? Vinvocation_directory : emacs_root_dir();
 	      default_directory = Fexpand_file_name (dir, absdir);
 	    }
 	}
     }
   if (! STRINGP (default_directory))
-    default_directory = root;
+    default_directory = emacs_root_dir();
 
   handler = Ffind_file_name_handler (default_directory, Qexpand_file_name);
   if (!NILP (handler))
diff --git a/src/lisp.h b/src/lisp.h
index 3d082911f5..834b3e586c 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4719,10 +4719,13 @@ maybe_disable_address_randomization (int argc, char **argv)
 extern void malloc_probe (size_t);
 extern void syms_of_profiler (void);
 
-#ifdef DOS_NT
-/* Defined in msdos.c, w32.c.  */
-extern char *emacs_root_dir (void);
-#endif /* DOS_NT */
+#ifdef MSDOS
+/* Defined in msdos.c.  */
+extern char *dos_emacs_root_dir (void);
+#elif defined(WINDOWSNT)
+/* Defined in w32.c.  */
+extern char *w32_emacs_root_dir (void);
+#endif /* MSDOS */
 
 #ifdef HAVE_NATIVE_COMP
 INLINE bool
diff --git a/src/msdos.c b/src/msdos.c
index b5f06c99c3..0827cc96cd 100644
--- a/src/msdos.c
+++ b/src/msdos.c
@@ -3350,7 +3350,7 @@ getdefdir (int drive, char *dst)
 }
 
 char *
-emacs_root_dir (void)
+dos_emacs_root_dir (void)
 {
   static char root_dir[4];
 
diff --git a/src/w32.c b/src/w32.c
index 0f69e652a5..1ec0094c8e 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -3147,7 +3147,7 @@ #define SET_ENV_BUF_SIZE (4 * MAX_PATH)	/* to cover EMACSLOADPATH */
 /* Called from expand-file-name when default-directory is not a string.  */
 
 char *
-emacs_root_dir (void)
+w32_emacs_root_dir (void)
 {
   static char root_dir[MAX_UTF8_PATH];
   const char *p;
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Tue, 19 May 2020 16:25:53 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> * lisp/emacs-lisp/comp.el (comp-effective-async-max-jobs): Use
> NUMBER_OF_PROCESSORS environment variable if system is Windows NT,
> "nproc" if it is in PATH or a default of 1.

This shouldn't be necessary: we already have a function to determine
the number of processors, see get_native_system_info in w32.c.  If you
need the result exported to Lisp, we can define a new variable which
will be populated with the value.

> Subject: [PATCH] Determine the emacs root dir only when necessary.
> 
> * src/fileio.c: Introduce function emacs_root_dir(). Refactor
> `expand-file-name` to use it.
> * src/lisp.h: Separate emacs_root_dir() into dos_emacs_root_dir() and
> w32_emacs_root_dir().
> * src/msdos.c: Rename emacs_root_dir() to dos_emacs_root_dir().
> * src/w32.c: Rename emacs_root_dir() to w32_emacs_root_dir().

Can you explain what problem this solves, and how?  It is especially
important to understand when will be emacs_root_dir first called
during a session.  That's because it calls filename_from_ansi, which
AFAIR needs some setup that happens at the beginning of a session.

Thanks.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]

> Can you explain what problem this solves, and how?  It is especially
> important to understand when will be emacs_root_dir first called
> during a session.  That's because it calls filename_from_ansi, which
> AFAIR needs some setup that happens at the beginning of a session.

When loading a dump file that contains native compiled elisp we try to find
the
.eln file. This uses Ffile_exists_p(). This function, in turn, calls
Fexpand_file_name(). This function will use the root directory as
`default_directory' as a fallback.

Getting the root directory requires reading the $emacs_dir environment
variable.
This is setup later in the initialization process. This caused a crash.

Fexpand_file_name() was trying to obtain the root directory even when it
was not
necessary because `default-directory' was not nil. My patch makes sure that
this
only happens when necessary.

It turns out that the dump loading process does not set `default-directory'
to
nil, therefore Fexpand_file_name() does not need to find out the root
directory
and we avoid reading an environment variable that is not set yet.

With this patch we avoid calling filename_from_ansi() too early (It is not
the
reason why Emacs crashed, but it is still a good idea to call it after it
has
been setup properly.

Nico

[-- Attachment #2: Type: text/html, Size: 1446 bytes --]

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Nicolas Bértolo <nicolasbertolo@gmail.com>,
>>         41242@debbugs.gnu.org
>> Date: Sat, 23 May 2020 08:35:43 +0000
>> 
>> >> When loading a dump file that contains native compiled elisp we try to find the
>> >> .eln file. This uses Ffile_exists_p(). This function, in turn, calls
>> >> Fexpand_file_name(). This function will use the root directory as
>> >> `default_directory' as a fallback.
>> >> 
>> >> Getting the root directory requires reading the $emacs_dir environment variable.
>> >> This is setup later in the initialization process. This caused a crash.
>> >
>> > Can you tell why this doesn't happen when *.elc files are used
>> > instead?  Also, what do you mean by "loading a dump file that contains
>> > native compiled elisp"? how did native compiled ELisp get into the
>> > dump file in the first place?
>> 
>> Hi Eli,
>> 
>> you may be interested in this paragraph
>> 
>> https://akrl.sdf.org/gccemacs.html#orgb063106
>
> Thanks.  I guess this changes a lot in how Emacs starts up?  E.g.,
> what happens if the .el file was in the meantime modified and
> recompiled into the .eln file? we will load the new versions instead
> of the one that was preloaded, right?

This is something we have to define.  Currently we only complain if one
of the defined functions that was dumped cannot be found in the new
.eln.  My preference would be to sign each .eln used for dump to make
sure what we are loading is what we dumped and refuse to load otherwise.

BTW reloading from dump the "dumped" eln are not located by searching in
the load-path for all suffix.  The eln position is stored into the
compilation unit object for performance reasons.  The performance
problem Nicolas is discussing is related to the non dumped .elns loaded
at startup I believe.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: nicolasbertolo@gmail.com, 41242@debbugs.gnu.org
> Date: Sat, 23 May 2020 10:37:18 +0000
> 
> >> https://akrl.sdf.org/gccemacs.html#orgb063106
> >
> > Thanks.  I guess this changes a lot in how Emacs starts up?  E.g.,
> > what happens if the .el file was in the meantime modified and
> > recompiled into the .eln file? we will load the new versions instead
> > of the one that was preloaded, right?
> 
> This is something we have to define.  Currently we only complain if one
> of the defined functions that was dumped cannot be found in the new
> .eln.  My preference would be to sign each .eln used for dump to make
> sure what we are loading is what we dumped and refuse to load otherwise.
> 
> BTW reloading from dump the "dumped" eln are not located by searching in
> the load-path for all suffix.  The eln position is stored into the
> compilation unit object for performance reasons.

Are you saying we store the absolute file name of the .eln files in
the pdumper file?  If so, how can Emacs start up if the .eln files
were moved to another location, e.g. as part of "make install", or
more generally if Emacs was relocated since it was dumped?

> The performance problem Nicolas is discussing is related to the non
> dumped .elns loaded at startup I believe.

This is not about the performance issue, this is about the crash
because emacs_dir was not yet defined where Emacs needed it: a
different issue uncovered by Nicolas.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: nicolasbertolo@gmail.com, 41242@debbugs.gnu.org
>> Date: Sat, 23 May 2020 10:37:18 +0000
>>
>> >> https://akrl.sdf.org/gccemacs.html#orgb063106
>> >
>> > Thanks.  I guess this changes a lot in how Emacs starts up?  E.g.,
>> > what happens if the .el file was in the meantime modified and
>> > recompiled into the .eln file? we will load the new versions instead
>> > of the one that was preloaded, right?
>>
>> This is something we have to define.  Currently we only complain if one
>> of the defined functions that was dumped cannot be found in the new
>> .eln.  My preference would be to sign each .eln used for dump to make
>> sure what we are loading is what we dumped and refuse to load otherwise.
>>
>> BTW reloading from dump the "dumped" eln are not located by searching in
>> the load-path for all suffix.  The eln position is stored into the
>> compilation unit object for performance reasons.
>
> Are you saying we store the absolute file name of the .eln files in
> the pdumper file?  If so, how can Emacs start up if the .eln files
> were moved to another location, e.g. as part of "make install", or
> more generally if Emacs was relocated since it was dumped?

To be precise we store the relative path of the .eln from the emacs
executable, both for the local build both for the file position we will
have after a "make install".

Reloading the first compilation unit the code detect in which of this
two cases we are (this is where file-exists is called) and this
information is used for all the following compilaiton unit to be
revived.

>> The performance problem Nicolas is discussing is related to the non
>> dumped .elns loaded at startup I believe.
>
> This is not about the performance issue, this is about the crash
> because emacs_dir was not yet defined where Emacs needed it: a
> different issue uncovered by Nicolas.

Ops got confused between the two branches of this same thread.

--
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: nicolasbertolo@gmail.com, 41242@debbugs.gnu.org
> Date: Sat, 23 May 2020 11:21:03 +0000
> 
> >> BTW reloading from dump the "dumped" eln are not located by searching in
> >> the load-path for all suffix.  The eln position is stored into the
> >> compilation unit object for performance reasons.
> >
> > Are you saying we store the absolute file name of the .eln files in
> > the pdumper file?  If so, how can Emacs start up if the .eln files
> > were moved to another location, e.g. as part of "make install", or
> > more generally if Emacs was relocated since it was dumped?
> 
> To be precise we store the relative path of the .eln from the emacs
> executable, both for the local build both for the file position we will
> have after a "make install".
> 
> Reloading the first compilation unit the code detect in which of this
> two cases we are (this is where file-exists is called) and this
> information is used for all the following compilaiton unit to be
> revived.

Do we indeed have only 2 use cases regarding the relative file name of
.eln files, and not more?

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> To be precise we store the relative path of the .eln from the emacs
>> executable, both for the local build both for the file position we will
>> have after a "make install".
>> 
>> Reloading the first compilation unit the code detect in which of this
>> two cases we are (this is where file-exists is called) and this
>> information is used for all the following compilaiton unit to be
>> revived.
>
> Do we indeed have only 2 use cases regarding the relative file name of
> .eln files, and not more?

If is a question regarding the current implementation of the branch the
answer is yes.

If is a more generic question then the answer is AFAIK yes, but we could
add others if they are known at compile time and needed.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

> This is something we have to define.  Currently we only complain if one
> of the defined functions that was dumped cannot be found in the new
> .eln.  My preference would be to sign each .eln used for dump to make
> sure what we are loading is what we dumped and refuse to load otherwise.

What if we find out where the linker has put the shared library, copy that
region
of memory into the dump file and when loading Emacs we mmap that data into
same address it was?
It is essentially saving the result of the linker for later use. This would
require
no ASLR and doing it ASAP to prevent the something from using that address
space.

Another option: statically link the .eln files (we'd need libgccjit to
create static libraries)
into the final Emacs executable. This would take care of function
definitions and
loading the dump would take care of the rest.

Nicolas

[-- Attachment #2: Type: text/html, Size: 1079 bytes --]

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> This is something we have to define.  Currently we only complain if
> one
>> of the defined functions that was dumped cannot be found in the new
>> .eln.  My preference would be to sign each .eln used for dump to
> make
>> sure what we are loading is what we dumped and refuse to load
> otherwise.
>
> What if we find out where the linker has put the shared library, copy
> that region
> of memory into the dump file and when loading Emacs we mmap that data
> into
> same address it was?
> It is essentially saving the result of the linker for later use. This
> would require
> no ASLR and doing it ASAP to prevent the something from using that
> address space.

I don't think would fly: You are not garanteed to be able to obtain the
same mmaped address anyway and we cannot go for a solution that does not
support ASLR.  In general to be portable it cannot rely on assumptions
or low level tricks.  I think these are (at least part of) the reasons
why we moved away from unexec.

> Another option: statically link the .eln files (we'd need libgccjit
> to create static libraries)
> into the final Emacs executable. This would take care of function
> definitions and
> loading the dump would take care of the rest.

Is not that simple, loading eln is mutating the environment with side
effects, function definition is just a part of that.

Even more important we must support subsequent dumps.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

 > I don't think would fly: You are not garanteed to be able to obtain the
> same mmaped address anyway and we cannot go for a solution that does not
> support ASLR.  In general to be portable it cannot rely on assumptions
> or low level tricks.  I think these are (at least part of) the reasons
> why we moved away from unexec.

AFAIU ASLR is disabled already, at least in Windows.
> Is not that simple, loading eln is mutating the environment with side
> effects, function definition is just a part of that.

I know, but linking against a static .eln would just make the symbols
available,
not anything else. The mutation of the environment would happen when loading
the dump.

> Even more important we must support subsequent dumps.

You are right. I hadn't considered this.

Nicolas

[-- Attachment #2: Type: text/html, Size: 1264 bytes --]

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> I don't think would fly: You are not garanteed to be able to obtain
> the
>> same mmaped address anyway and we cannot go for a solution that
> does not
>> support ASLR.  In general to be portable it cannot rely on
> assumptions
>> or low level tricks.  I think these are (at least part of) the
> reasons
>> why we moved away from unexec.
>
> AFAIU ASLR is disabled already, at least in Windows.

The fact that is now disabled does not imply that we want to prevent us
from using it for the future, that said AFAIU given the personality of
my running Emacs is zeroed (i'm on GNU/Linux) it is running with ASLR
enabled.  See also 'maybe_disable_address_randomization'.

>> Is not that simple, loading eln is mutating the environment with
> side
>> effects, function definition is just a part of that.
>
> I know, but linking against a static .eln would just make the symbols
> available,
> not anything else. The mutation of the environment would happen when
> loading
> the dump.

Is more complex because we can have in the eln code that at top level
relies on functions as:

(defun foo ()
  ..)

(some code that depends on the definition of foo)

(defun foo ()
  ..)

So function definition and other top level executon is mixed.  I think
this could be worked around but is tricky.

>> Even more important we must support subsequent dumps.
>
> You are right. I hadn't considered this.

Yeah I think this the real no go.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 23 May 2020 12:26:56 -0300
> Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> 
> AFAIU ASLR is disabled already, at least in Windows.

I don't think so, I see it on modern versions of Windows all the time.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

> I don't think so, I see it on modern versions of Windows all the time.

Using ProcessExplorer this is what I see in my machine:

- emacs.exe is always loaded @ 0x40000000.
- .eln files do not have the ASLR flag enabled and "Image Base" == "Base".

Nicolas

[-- Attachment #2: Type: text/html, Size: 409 bytes --]

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 23 May 2020 13:20:06 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > I don't think so, I see it on modern versions of Windows all the time. 
> 
> Using ProcessExplorer this is what I see in my machine:
> 
> - emacs.exe is always loaded @ 0x40000000.
> - .eln files do not have the ASLR flag enabled and "Image Base" == "Base".

My observation was based on the fact that addresses of the same
objects that I see in the debugger are different from session to
session on Windows 7 and Windows 10, but not on XP.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

> My observation was based on the fact that addresses of the same
> objects that I see in the debugger are different from session to
> session on Windows 7 and Windows 10, but not on XP.

That would mean that the heap is randomized, right? But the code could
still be always loaded at the same address.

Coming back to the performance problem when loading: apart from reducing
the number of files probed, we could try parallelizing openp() using a
thread
pool. What do you think?

Nicolas

El sáb., 23 may. 2020 a las 14:04, Eli Zaretskii (<eliz@gnu.org>) escribió:

> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Sat, 23 May 2020 13:20:06 -0300
> > Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> >
> > > I don't think so, I see it on modern versions of Windows all the time.
> >
> > Using ProcessExplorer this is what I see in my machine:
> >
> > - emacs.exe is always loaded @ 0x40000000.
> > - .eln files do not have the ASLR flag enabled and "Image Base" ==
> "Base".
>
> My observation was based on the fact that addresses of the same
> objects that I see in the debugger are different from session to
> session on Windows 7 and Windows 10, but not on XP.
>

[-- Attachment #2: Type: text/html, Size: 1928 bytes --]

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 23 May 2020 14:20:09 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > My observation was based on the fact that addresses of the same
> > objects that I see in the debugger are different from session to
> > session on Windows 7 and Windows 10, but not on XP. 
> 
> That would mean that the heap is randomized, right? But the code could
> still be always loaded at the same address.

I don't remember if it was only the heap, or also addresses of other
areas.  For example, I think the stack was also in a different area.

> Coming back to the performance problem when loading: apart from reducing
> the number of files probed, we could try parallelizing openp() using a thread
> pool. What do you think?

I'd start by reducing the number of probed files, and then I'd
benchmark the results and see if it's "good enough".  Threads add
another dimension of complexity, so I'd only go there if we have a
very good reason.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

 > I'd start by reducing the number of probed files, and then I'd
> benchmark the results and see if it's "good enough".  Threads add
> another dimension of complexity, so I'd only go there if we have a
> very good reason.

Just to show some numbers:
I run a fairly heavy Spacemacs configuration.
Without any patch it takes 80 seconds to start.
Reducing the number of probed files takes this down to 40 seconds.

The VTune profiler tells me it is spending 80% of the time waiting for
openp().

I'll rewrite the patch to reduce the number of probed files.

[-- Attachment #2: Type: text/html, Size: 754 bytes --]

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 23 May 2020 14:47:03 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> Just to show some numbers:
> I run a fairly heavy Spacemacs configuration.
> Without any patch it takes 80 seconds to start.
> Reducing the number of probed files takes this down to 40 seconds.
> 
> The VTune profiler tells me it is spending 80% of the time waiting for openp().

If it takes us 32 seconds to run openp, then how many files do we try
to probe?  32 sec is eternity!

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 389 bytes --]

 > If it takes us 32 seconds to run openp, then how many files do we try
> to probe?  32 sec is eternity!

`load-path' has 380 entries, and load-prefer-newer is true. That is why it
is so slow.
Loading foo.el from package foo requires probing 6 different extensions
(.eln, .dll, .elc.gz, .elc, .el, .el.gz) in 380 directories.
6*380 = 2280 calls to wopen() per call to `load` or `require`

[-- Attachment #2: Type: text/html, Size: 500 bytes --]

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 23 May 2020 15:29:25 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > If it takes us 32 seconds to run openp, then how many files do we try
> > to probe?  32 sec is eternity! 
> 
> `load-path' has 380 entries

Is this due to Spacemacs setup, or something else?  Why would you have
so many directories in your load-path?

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

> Why would you have so many directories in your load-path?

Because I have many Spacemacs layers installed, each installs many packages
and each package has its own directory in `package-user-dir`.

El sáb., 23 may. 2020 a las 15:37, Eli Zaretskii (<eliz@gnu.org>) escribió:

> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Sat, 23 May 2020 15:29:25 -0300
> > Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> >
> > > If it takes us 32 seconds to run openp, then how many files do we try
> > > to probe?  32 sec is eternity!
> >
> > `load-path' has 380 entries
>
> Is this due to Spacemacs setup, or something else?  Why would you have
> so many directories in your load-path?
>

[-- Attachment #2: Type: text/html, Size: 1294 bytes --]

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1.1: Type: text/plain, Size: 949 bytes --]

I have come up with this patch to reduce the number of files
probed when `load' is called..

El sáb., 23 may. 2020 a las 15:43, Nicolas Bértolo (<
nicolasbertolo@gmail.com>) escribió:

> > Why would you have so many directories in your load-path?
>
> Because I have many Spacemacs layers installed, each installs many packages
> and each package has its own directory in `package-user-dir`.
>
> El sáb., 23 may. 2020 a las 15:37, Eli Zaretskii (<eliz@gnu.org>)
> escribió:
>
>> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
>> > Date: Sat, 23 May 2020 15:29:25 -0300
>> > Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
>> >
>> > > If it takes us 32 seconds to run openp, then how many files do we try
>> > > to probe?  32 sec is eternity!
>> >
>> > `load-path' has 380 entries
>>
>> Is this due to Spacemacs setup, or something else?  Why would you have
>> so many directories in your load-path?
>>
>

[-- Attachment #1.2: Type: text/html, Size: 1807 bytes --]

[-- Attachment #2: 0001-Reduce-the-number-of-files-probed-when-finding-a-lis.patch --]
[-- Type: application/octet-stream, Size: 12671 bytes --]

From e19d43350e30bd5816dd8fd33c6770351ea902b7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Sat, 23 May 2020 18:49:52 -0300
Subject: [PATCH] Reduce the number of files probed when finding a lisp file.

* src/lread.c (get-load-suffixes): Do not add any suffix to files that
need to be loaded by the dynamic linker.
(effective_load_path): Remove function.
(load): Don't add any suffix if file ends in a suffix already.
(openp): Add a parameter `add_eln_dir` that should be used when we
desire the dir elt/eln-hash/ to be searched for each path in `path`.
* src/lisp.h: Add new parameter to openp.
* src/callproc.c: Add the new parameter to calls to openp().
* src/charset.c: Add the new parameter to calls to openp().
* src/emacs.c: Add the new parameter to calls to openp().
* src/image.c: Add the new parameter to calls to openp().
* src/process.c: Add the new parameter to calls to openp().
* src/w32.c: Add the new parameter to calls to openp().
* src/w32proc.c: Add the new parameter to calls to openp().
---
 src/callproc.c |   2 +-
 src/charset.c  |   2 +-
 src/emacs.c    |   3 +-
 src/image.c    |   4 +-
 src/lisp.h     |   2 +-
 src/lread.c    | 103 +++++++++++++++++++++++++++++--------------------
 src/process.c  |   2 +-
 src/w32.c      |   2 +-
 src/w32proc.c  |   2 +-
 9 files changed, 72 insertions(+), 50 deletions(-)

diff --git a/src/callproc.c b/src/callproc.c
index 65c858393a..421e42de11 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -436,7 +436,7 @@ call_process (ptrdiff_t nargs, Lisp_Object *args, int filefd,
   {
     int ok;
 
-    ok = openp (Vexec_path, args[0], Vexec_suffixes, &path,
+    ok = openp (Vexec_path, args[0], false, Vexec_suffixes, &path,
 		make_fixnum (X_OK), false);
     if (ok < 0)
       report_file_error ("Searching for program", args[0]);
diff --git a/src/charset.c b/src/charset.c
index 8635aad3ed..da3ace743f 100644
--- a/src/charset.c
+++ b/src/charset.c
@@ -486,7 +486,7 @@ load_charset_map_from_file (struct charset *charset, Lisp_Object mapfile,
   ptrdiff_t count = SPECPDL_INDEX ();
   record_unwind_protect_nothing ();
   specbind (Qfile_name_handler_alist, Qnil);
-  fd = openp (Vcharset_map_path, mapfile, suffixes, NULL, Qnil, false);
+  fd = openp (Vcharset_map_path, mapfile, false, suffixes, NULL, Qnil, false);
   fp = fd < 0 ? 0 : fdopen (fd, "r");
   if (!fp)
     {
diff --git a/src/emacs.c b/src/emacs.c
index b7c89b44ec..ce7b210c6f 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -449,7 +449,8 @@ set_invocation_vars (char *argv0, char const *original_pwd)
     {
       Lisp_Object found;
       int yes = openp (Vexec_path, Vinvocation_name,
-		       Vexec_suffixes, &found, make_fixnum (X_OK), false);
+		       false, Vexec_suffixes, &found, make_fixnum (X_OK),
+                       false);
       if (yes == 1)
 	{
 	  /* Add /: to the front of the name
diff --git a/src/image.c b/src/image.c
index c8a192aaaf..80ef801913 100644
--- a/src/image.c
+++ b/src/image.c
@@ -508,7 +508,7 @@ image_create_bitmap_from_file (struct frame *f, Lisp_Object file)
     }
 
   /* Search bitmap-file-path for the file, if appropriate.  */
-  if (openp (Vx_bitmap_file_path, file, Qnil, &found,
+  if (openp (Vx_bitmap_file_path, file, false, Qnil, &found,
 	     make_fixnum (R_OK), false)
       < 0)
     return -1;
@@ -2983,7 +2983,7 @@ image_find_image_fd (Lisp_Object file, int *pfd)
 		       Vx_bitmap_file_path);
 
   /* Try to find FILE in data-directory/images, then x-bitmap-file-path.  */
-  fd = openp (search_path, file, Qnil, &file_found,
+  fd = openp (search_path, file, false, Qnil, &file_found,
 	      pfd ? Qt : make_fixnum (R_OK), false);
   if (fd >= 0 || fd == -2)
     {
diff --git a/src/lisp.h b/src/lisp.h
index 3e7e51f839..0bf2cfd741 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4096,7 +4096,7 @@ LOADHIST_ATTACH (Lisp_Object x)
 extern bool suffix_p (Lisp_Object, const char *);
 extern Lisp_Object save_match_data_load (Lisp_Object, Lisp_Object, Lisp_Object,
 					 Lisp_Object, Lisp_Object);
-extern int openp (Lisp_Object, Lisp_Object, Lisp_Object,
+extern int openp (Lisp_Object, Lisp_Object, bool, Lisp_Object,
                   Lisp_Object *, Lisp_Object, bool);
 enum { S2N_IGNORE_TRAILING = 1 };
 extern Lisp_Object string_to_number (char const *, int, ptrdiff_t *);
diff --git a/src/lread.c b/src/lread.c
index 01f359ca58..9c3a682bb7 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1056,33 +1056,25 @@ DEFUN ("get-load-suffixes", Fget_load_suffixes, Sget_load_suffixes, 0, 0, 0,
     {
       Lisp_Object exts = Vload_file_rep_suffixes;
       Lisp_Object suffix = XCAR (suffixes);
-      FOR_EACH_TAIL (exts)
-	lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
+      if (false
+#ifdef HAVE_MODULES
+          || strcmp (MODULES_SUFFIX, SSDATA (suffix)) == 0
+#ifdef MODULES_SECONDARY_SUFFIX
+          || strcmp (MODULES_SECONDARY_SUFFIX, SSDATA (suffix)) == 0
+#endif
+#endif
+#ifdef HAVE_NATIVE_COMP
+          || strcmp (NATIVE_ELISP_SUFFIX, SSDATA (suffix)) == 0
+#endif
+          )
+	lst = Fcons (suffix, lst);
+      else
+        FOR_EACH_TAIL (exts)
+          lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
     }
   return Fnreverse (lst);
 }
 
-static Lisp_Object
-effective_load_path (void)
-{
-#ifndef HAVE_NATIVE_COMP
-  return Vload_path;
-#else
-  Lisp_Object lp = Vload_path;
-  Lisp_Object new_lp = Qnil;
-  FOR_EACH_TAIL (lp)
-    {
-      Lisp_Object el = XCAR (lp);
-      new_lp =
-	Fcons (concat2 (Ffile_name_as_directory (el),
-			Vcomp_native_path_postfix),
-	       new_lp);
-      new_lp = Fcons (el, new_lp);
-    }
-  return Fnreverse (new_lp);
-#endif
-}
-
 /* Return true if STRING ends with SUFFIX.  */
 bool
 suffix_p (Lisp_Object string, const char *suffix)
@@ -1217,6 +1209,9 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 #ifdef MODULES_SECONDARY_SUFFIX
               || suffix_p (file, MODULES_SECONDARY_SUFFIX)
 #endif
+#endif
+#ifdef HAVE_NATIVE_COMP
+              || suffix_p (file, NATIVE_ELISP_SUFFIX)
 #endif
 	      )
 	    must_suffix = Qnil;
@@ -1236,8 +1231,8 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 	}
 
       fd =
-	openp (effective_load_path (), file, suffixes, &found, Qnil,
-	       load_prefer_newer);
+	openp (Vload_path, file, true, suffixes, &found, Qnil,
+               load_prefer_newer);
     }
 
   if (fd == -1)
@@ -1600,7 +1595,7 @@ DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2,
   (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate)
 {
   Lisp_Object file;
-  int fd = openp (path, filename, suffixes, &file, predicate, false);
+  int fd = openp (path, filename, Qnil, suffixes, &file, predicate, false);
   if (NILP (predicate) && fd >= 0)
     emacs_close (fd);
   return file;
@@ -1633,8 +1628,9 @@ DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2,
    or if a non-nil and non-t PREDICATE is specified.  */
 
 int
-openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
-       Lisp_Object *storeptr, Lisp_Object predicate, bool newer)
+openp (Lisp_Object path, Lisp_Object str, bool fix_load_paths,
+       Lisp_Object suffixes, Lisp_Object *storeptr, Lisp_Object predicate,
+       bool newer)
 {
   ptrdiff_t fn_size = 100;
   char buf[100];
@@ -1643,7 +1639,7 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
   ptrdiff_t want_length;
   Lisp_Object filename;
   Lisp_Object string, tail, encoded_fn, save_string;
-  ptrdiff_t max_suffix_len = 0;
+  ptrdiff_t max_extra_len = 0;
   int last_errno = ENOENT;
   int save_fd = -1;
   USE_SAFE_ALLOCA;
@@ -1658,8 +1654,15 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
   FOR_EACH_TAIL_SAFE (tail)
     {
       CHECK_STRING_CAR (tail);
-      max_suffix_len = max (max_suffix_len,
-			    SBYTES (XCAR (tail)));
+      char * suf = SSDATA (XCAR (tail));
+      ptrdiff_t len = SBYTES (XCAR (tail));
+      if (fix_load_paths && strcmp(NATIVE_ELISP_SUFFIX, suf) == 0)
+        {
+          CHECK_STRING (Vcomp_native_path_postfix);
+          len += 2;
+          len += SBYTES (Vcomp_native_path_postfix);
+        }
+      max_extra_len = max (max_extra_len, len);
     }
 
   string = filename = encoded_fn = save_string = Qnil;
@@ -1677,7 +1680,7 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
      executable. */
   FOR_EACH_TAIL_SAFE (path)
    {
-    ptrdiff_t baselen, prefixlen;
+    ptrdiff_t dirnamelen, prefixlen, basenamewext_len;
 
     if (EQ (path, just_use_str))
       filename = str;
@@ -1694,22 +1697,27 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
 	  continue;
       }
 
+
     /* Calculate maximum length of any filename made from
        this path element/specified file name and any possible suffix.  */
-    want_length = max_suffix_len + SBYTES (filename);
+    want_length = max_extra_len + SBYTES (filename);
     if (fn_size <= want_length)
       {
 	fn_size = 100 + want_length;
 	fn = SAFE_ALLOCA (fn_size);
       }
 
+    Lisp_Object dirnamewslash = Ffile_name_directory(filename);
+    Lisp_Object basenamewext = Ffile_name_nondirectory(filename);
+
     /* Copy FILENAME's data to FN but remove starting /: if any.  */
-    prefixlen = ((SCHARS (filename) > 2
-		  && SREF (filename, 0) == '/'
-		  && SREF (filename, 1) == ':')
+    prefixlen = ((SCHARS (dirnamewslash) > 2
+		  && SREF (dirnamewslash, 0) == '/'
+		  && SREF (dirnamewslash, 1) == ':')
 		 ? 2 : 0);
-    baselen = SBYTES (filename) - prefixlen;
-    memcpy (fn, SDATA (filename) + prefixlen, baselen);
+    dirnamelen = SBYTES (dirnamewslash) - prefixlen;
+    basenamewext_len = SBYTES (basenamewext);
+    memcpy (fn, SDATA (dirnamewslash) + prefixlen, dirnamelen);
 
     /* Loop over suffixes.  */
     AUTO_LIST1 (empty_string_only, empty_unibyte_string);
@@ -1719,10 +1727,23 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
 	Lisp_Object suffix = XCAR (tail);
 	ptrdiff_t fnlen, lsuffix = SBYTES (suffix);
 	Lisp_Object handler;
+        bool add_native_comp_dir = fix_load_paths &&
+          (strcmp(SSDATA(suffix), NATIVE_ELISP_SUFFIX) == 0);
+        ptrdiff_t lmiddledir = add_native_comp_dir ?
+            SBYTES(Vcomp_native_path_postfix) + 1 : 0;
+
+        if (add_native_comp_dir)
+          {
+            memcpy(fn + dirnamelen, SDATA(Vcomp_native_path_postfix),
+                   lmiddledir - 1);
+            fn[dirnamelen+(lmiddledir-1)] = '/';
+          }
 
-	/* Make complete filename by appending SUFFIX.  */
-	memcpy (fn + baselen, SDATA (suffix), lsuffix + 1);
-	fnlen = baselen + lsuffix;
+        memcpy (fn + dirnamelen + lmiddledir, SDATA (basenamewext), basenamewext_len);
+        /* Make complete filename by appending SUFFIX.  */
+        memcpy (fn + dirnamelen + lmiddledir + basenamewext_len,
+                SDATA (suffix), lsuffix + 1);
+        fnlen = dirnamelen + lmiddledir + basenamewext_len + lsuffix;
 
 	/* Check that the file exists and is not a directory.  */
 	/* We used to only check for handlers on non-absolute file names:
diff --git a/src/process.c b/src/process.c
index 6e5bcf307a..e092fa0a0c 100644
--- a/src/process.c
+++ b/src/process.c
@@ -1903,7 +1903,7 @@ DEFUN ("make-process", Fmake_process, Smake_process, 0, MANY, 0,
 	       && IS_DEVICE_SEP (SREF (program, 1))))
 	{
 	  tem = Qnil;
-	  openp (Vexec_path, program, Vexec_suffixes, &tem,
+	  openp (Vexec_path, program, false, Vexec_suffixes, &tem,
 		 make_fixnum (X_OK), false);
 	  if (NILP (tem))
 	    report_file_error ("Searching for program", program);
diff --git a/src/w32.c b/src/w32.c
index fd1f0e059e..3622a555d7 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -10177,7 +10177,7 @@ check_windows_init_file (void)
 	 need to ENCODE_FILE here, but we do need to convert the file
 	 names from UTF-8 to ANSI.  */
       init_file = build_string ("term/w32-win");
-      fd = openp (Vload_path, init_file, Fget_load_suffixes (), NULL, Qnil, 0);
+      fd = openp (Vload_path, init_file, true, Fget_load_suffixes (), NULL, Qnil, 0);
       if (fd < 0)
 	{
 	  Lisp_Object load_path_print = Fprin1_to_string (Vload_path, Qnil);
diff --git a/src/w32proc.c b/src/w32proc.c
index 16e32e4c58..85f2ccf016 100644
--- a/src/w32proc.c
+++ b/src/w32proc.c
@@ -1918,7 +1918,7 @@ sys_spawnve (int mode, char *cmdname, char **argv, char **envp)
     {
       program = build_string (cmdname);
       full = Qnil;
-      openp (Vexec_path, program, Vexec_suffixes, &full, make_fixnum (X_OK), 0);
+      openp (Vexec_path, program, false, Vexec_suffixes, &full, make_fixnum (X_OK), 0);
       if (NILP (full))
 	{
 	  errno = EINVAL;
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> I have come up with this patch to reduce the number of files
> probed when `load' is called..

Yes, also if you are working in the area feel free to remove the
effective_load_path approach because it breaks `load-prefer-newer'.

If you are working on this I won't touch it not to conflict.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Maybe the developers of Spacemacs should help write code to speed up
that case.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Coming back to the performance problem when loading: apart from reducing
>> the number of files probed, we could try parallelizing openp() using a thread
>> pool. What do you think?
>
> I'd start by reducing the number of probed files, and then I'd
> benchmark the results and see if it's "good enough".  Threads add
> another dimension of complexity, so I'd only go there if we have a
> very good reason.

Agree, also I think before that would be necessary to prove that
parallelizing in user space let the kernel scale up in performance in
serving the syscalls.  I'm not really sure about that and the observed
bottle neck is apparently there.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 23 May 2020 11:41:19 -0300
> Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> 
> Another option: statically link the .eln files (we'd need libgccjit to create static libraries)
> into the final Emacs executable. This would take care of function definitions and
> loading the dump would take care of the rest.

That would preclude re-dumping Emacs, no?

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

 > That would preclude re-dumping Emacs, no?

I guess so. I didn't think about that option.
As Andrea said, this is a no go.


El sáb., 23 may. 2020 a las 12:52, Eli Zaretskii (<eliz@gnu.org>) escribió:

> > From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> > Date: Sat, 23 May 2020 11:41:19 -0300
> > Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> >
> > Another option: statically link the .eln files (we'd need libgccjit to
> create static libraries)
> > into the final Emacs executable. This would take care of function
> definitions and
> > loading the dump would take care of the rest.
>
> That would preclude re-dumping Emacs, no?
>

[-- Attachment #2: Type: text/html, Size: 1249 bytes --]

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
>> Date: Tue, 19 May 2020 16:25:53 -0300
>> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
>> 
>> * lisp/emacs-lisp/comp.el (comp-effective-async-max-jobs): Use
>> NUMBER_OF_PROCESSORS environment variable if system is Windows NT,
>> "nproc" if it is in PATH or a default of 1.
>
> This shouldn't be necessary: we already have a function to determine
> the number of processors, see get_native_system_info in w32.c.  If you
> need the result exported to Lisp, we can define a new variable which
> will be populated with the value.
>
>> Subject: [PATCH] Determine the emacs root dir only when necessary.
>> 
>> * src/fileio.c: Introduce function emacs_root_dir(). Refactor
>> `expand-file-name` to use it.
>> * src/lisp.h: Separate emacs_root_dir() into dos_emacs_root_dir() and
>> w32_emacs_root_dir().
>> * src/msdos.c: Rename emacs_root_dir() to dos_emacs_root_dir().
>> * src/w32.c: Rename emacs_root_dir() to w32_emacs_root_dir().
>
> Can you explain what problem this solves, and how?  It is especially
> important to understand when will be emacs_root_dir first called
> during a session.  That's because it calls filename_from_ansi, which
> AFAIR needs some setup that happens at the beginning of a session.

I'm not into what is happening on Windows and why is working differently
than Posix, but I can add this as a context:

While we are restarting from dump to reload the native compilation unit
in dump_do_dump_relocation we are calling Ffile_exists_p
(pdumper.c:5307) and this is calling Fexpand_file_name.

We need to do that because we need the eln file to revive the
compilation unit object.  Perhaps another way would be to use something
alternative to Ffile_exists_p?

But I've a question: is this a problem of Fexpand_file_name not working
correctly because something not initialized or a behavior we are trying
to change?

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Tue, 19 May 2020 16:23:06 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> I have written the attached patch to handle the deletion
> of native compilation units still in use. I have tested that it works well.

Thanks.  I question the wisdom of handling the deletion from Lisp: if
this is specific to MS-Windows, then doing this from C would allow us
finer control on what happens.  But this can be tackled later if
needed, so I'm not going to object to your code.

Your legal paperwork is done, so these changes can be installed on the
branch.  Andrea, please review the changes, and feel free to push at
your discretion.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> Your legal paperwork is done, so these changes can be installed on the
> branch.  Andrea, please review the changes, and feel free to push at
> your discretion.

Thanks, I'll start going through them this evening I guess in patch
chronological order.

  Andrea

PS Nicolas if some patch is not up to date feel free to post the latest
version (or link to a public git repo if you prefer and have one).

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1.1: Type: text/plain, Size: 133 bytes --]

> Thanks, I'll start going through them this evening I guess in patch
> chronological order.

Thank you.

Here's the latest version.

[-- Attachment #1.2: Type: text/html, Size: 240 bytes --]

[-- Attachment #2: 0003-Handle-setjmp-taking-two-arguments-in-Windows.patch --]
[-- Type: application/octet-stream, Size: 4674 bytes --]

From becf94b352a6c646d703eaa057b0870e6fda6532 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 15:56:09 -0300
Subject: [PATCH 3/9] Handle setjmp() taking two arguments in Windows.

* src/comp.c: Add `define_setjmp_deps()` and `emit_setjmp()` which
abstract over this difference in behavior between operating systems.

WARNING: Not all cases are handled by this patch. The Mingw-64
setjmp.h header deals with many other combinations. I don't think it
is a good idea to replicate the logic of that header inside
emacs. (Maybe a few lines in the configure script could be added to
handle this problem?)
---
 src/comp.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 52 insertions(+), 6 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 00f9d4b74a..0d46628e6a 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -22,6 +22,7 @@
 
 #ifdef HAVE_NATIVE_COMP
 
+#include <setjmp.h>
 #include <stdlib.h>
 #include <stdio.h>
 #include <signal.h>
@@ -74,10 +75,15 @@ #define DECL_BLOCK(name, func)				\
   gcc_jit_block *(name) =				\
     gcc_jit_function_new_block ((func), STR (name))
 
-#ifdef HAVE__SETJMP
-#define SETJMP _setjmp
+#ifndef _WIN32
+# ifdef HAVE__SETJMP
+#  define SETJMP _setjmp
+# else
+#  define SETJMP setjmp
+# endif
 #else
-#define SETJMP setjmp
+/* snippet from MINGW-64 setjmp.h */
+# define SETJMP _setjmp
 #endif
 #define SETJMP_NAME SETJMP
 
@@ -174,6 +180,9 @@ #define F_RELOC_MAX_SIZE 1500
   gcc_jit_function *setcdr;
   gcc_jit_function *check_type;
   gcc_jit_function *check_impure;
+#ifdef _WIN32
+  gcc_jit_function *setjmp_ctx_func;
+#endif
   Lisp_Object func_blocks_h; /* blk_name -> gcc_block.  */
   Lisp_Object exported_funcs_h; /* c-func-name -> gcc_jit_function *.  */
   Lisp_Object imported_funcs_h; /* subr_name -> gcc_jit_field *reloc_field.  */
@@ -1474,6 +1483,29 @@ emit_limple_call_ref (Lisp_Object insn, bool direct)
 			direct);
 }
 
+static gcc_jit_rvalue *
+emit_setjmp (gcc_jit_rvalue *buf)
+{
+#ifndef _WIN32
+  gcc_jit_rvalue *args[] = {buf};
+  return emit_call (intern_c_string (STR (SETJMP_NAME)), comp.int_type, 1, args,
+                   false);
+#else
+  /* _setjmp (buf, __builtin_frame_address (0)) */
+  gcc_jit_rvalue *args[2];
+
+  args[0] = gcc_jit_context_new_rvalue_from_int (comp.ctxt, comp.unsigned_type, 0);
+
+  args[1] = gcc_jit_context_new_call(comp.ctxt,
+                                     NULL,
+                                     comp.setjmp_ctx_func,
+                                     1, args);
+  args[0] = buf;
+  return emit_call (intern_c_string (STR (SETJMP_NAME)), comp.int_type, 2, args,
+                    false);
+#endif
+}
+
 /* Register an handler for a non local exit.  */
 
 static void
@@ -1500,8 +1532,7 @@ emit_limple_push_handler (gcc_jit_rvalue *handler, gcc_jit_rvalue *handler_type,
 	NULL);
 
   gcc_jit_rvalue *res;
-  res =
-    emit_call (intern_c_string (STR (SETJMP_NAME)), comp.int_type, 1, args, false);
+  res = emit_setjmp(args[0]);
   emit_cond_jump (res, handler_bb, guarded_bb);
 }
 
@@ -2060,8 +2091,14 @@ #define ADD_IMPORTED(f_name, ret_type, nargs, args)			       \
   args[1] = comp.int_type;
   ADD_IMPORTED (push_handler, comp.handler_ptr_type, 2, args);
 
+#ifndef _WIN32
   args[0] = gcc_jit_type_get_pointer (gcc_jit_struct_as_type (comp.jmp_buf_s));
   ADD_IMPORTED (SETJMP_NAME, comp.int_type, 1, args);
+#else
+  args[0] = gcc_jit_type_get_pointer (gcc_jit_struct_as_type (comp.jmp_buf_s));
+  args[1] = comp.void_ptr_type;
+  ADD_IMPORTED (SETJMP_NAME, comp.int_type, 2, args);
+#endif
 
   ADD_IMPORTED (record_unwind_protect_excursion, comp.void_type, 0, NULL);
 
@@ -2301,7 +2338,7 @@ define_jmp_buf (void)
       gcc_jit_context_new_array_type (comp.ctxt,
 				      NULL,
 				      comp.char_type,
-				      sizeof (jmp_buf)),
+				      sizeof (sys_jmp_buf)),
       "stuff");
   comp.jmp_buf_s =
     gcc_jit_context_new_struct_type (comp.ctxt,
@@ -2969,6 +3006,14 @@ define_CHECK_IMPURE (void)
     gcc_jit_block_end_with_void_return (err_block, NULL);
 }
 
+static void
+define_setjmp_deps (void)
+{
+  comp.setjmp_ctx_func
+    = gcc_jit_context_get_builtin_function (comp.ctxt,
+                                            "__builtin_frame_address");
+}
+
 /* Define a function to convert boolean into t or nil */
 
 static void
@@ -3357,6 +3402,7 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
   define_PSEUDOVECTORP ();
   define_CHECK_TYPE ();
   define_CHECK_IMPURE ();
+  define_setjmp_deps ();
   define_bool_to_lisp_obj ();
   define_setcar_setcdr ();
   define_add1_sub1 ();
-- 
2.25.1.windows.1


[-- Attachment #3: 0004-Handle-LISP_WORDS_ARE_POINTERS-and-CHECK_LISP_OBJECT.patch --]
[-- Type: application/octet-stream, Size: 18158 bytes --]

From 44ce2cb89856709fa4581ed4e3b8091b2d2c351c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 14:30:14 -0300
Subject: [PATCH 4/9] Handle LISP_WORDS_ARE_POINTERS and
 CHECK_LISP_OBJECT_TYPE.

* src/comp.c: Introduce the Lisp_X, Lisp_Word, and Lisp_Word_tag
types. These types are used instead of long or long long. Use
emacs_int_type and emacs_uint_types where appropriate.
(emit_coerce): Add special logic that handles the case when
Lisp_Object is a struct. This is necessary for handling the
--enable-check-lisp-object-type configure option.

* src/lisp.h: Since libgccjit does not support opaque unions, change
Lisp_X to be struct. This is done to ensure that the same types are
used in the same binary. It is probably unnecessary since only a
pointer to it is used.
---
 src/comp.c | 328 ++++++++++++++++++++++++++++++++++++-----------------
 src/lisp.h |   5 +-
 2 files changed, 228 insertions(+), 105 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 0d46628e6a..0d5880ad3c 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -116,6 +116,16 @@ #define F_RELOC_MAX_SIZE 1500
   gcc_jit_type *char_ptr_type;
   gcc_jit_type *ptrdiff_type;
   gcc_jit_type *uintptr_type;
+#if LISP_WORDS_ARE_POINTERS
+  gcc_jit_struct *lisp_X_s;
+  gcc_jit_type *lisp_X;
+#endif
+  gcc_jit_type *lisp_word_type;
+  gcc_jit_type *lisp_word_tag_type;
+#ifdef LISP_OBJECT_IS_STRUCT
+  gcc_jit_field *lisp_obj_i;
+  gcc_jit_struct *lisp_obj_s;
+#endif
   gcc_jit_type *lisp_obj_type;
   gcc_jit_type *lisp_obj_ptr_type;
   /* struct Lisp_Cons */
@@ -158,7 +168,8 @@ #define F_RELOC_MAX_SIZE 1500
   gcc_jit_field *cast_union_as_c_p;
   gcc_jit_field *cast_union_as_v_p;
   gcc_jit_field *cast_union_as_lisp_cons_ptr;
-  gcc_jit_field *cast_union_as_lisp_obj;
+  gcc_jit_field *cast_union_as_lisp_word;
+  gcc_jit_field *cast_union_as_lisp_word_tag;
   gcc_jit_field *cast_union_as_lisp_obj_ptr;
   gcc_jit_function *func; /* Current function being compiled.  */
   bool func_has_non_local; /* From comp-func has-non-local slot.  */
@@ -347,8 +358,10 @@ type_to_cast_field (gcc_jit_type *type)
     field = comp.cast_union_as_c_p;
   else if (type == comp.lisp_cons_ptr_type)
     field = comp.cast_union_as_lisp_cons_ptr;
-  else if (type == comp.lisp_obj_type)
-    field = comp.cast_union_as_lisp_obj;
+  else if (type == comp.lisp_word_type)
+    field = comp.cast_union_as_lisp_word;
+  else if (type == comp.lisp_word_tag_type)
+    field = comp.cast_union_as_lisp_word_tag;
   else if (type == comp.lisp_obj_ptr_type)
     field = comp.cast_union_as_lisp_obj_ptr;
   else
@@ -627,6 +640,31 @@ emit_coerce (gcc_jit_type *new_type, gcc_jit_rvalue *obj)
   if (new_type == old_type)
     return obj;
 
+#ifdef LISP_OBJECT_IS_STRUCT
+  if (old_type == comp.lisp_obj_type)
+    {
+      gcc_jit_rvalue *lwordobj =
+        gcc_jit_rvalue_access_field (obj, NULL, comp.lisp_obj_i);
+      return emit_coerce (new_type, lwordobj);
+    }
+
+  if (new_type == comp.lisp_obj_type)
+    {
+      gcc_jit_rvalue *lwordobj =
+        emit_coerce (comp.lisp_word_type, obj);
+
+      gcc_jit_lvalue *tmp_s
+        = gcc_jit_function_new_local (comp.func, NULL, comp.lisp_obj_type,
+                                      format_string ("lisp_obj_%td", i++));
+
+      gcc_jit_block_add_assignment (comp.block, NULL,
+                                    gcc_jit_lvalue_access_field (tmp_s, NULL,
+                                                                 comp.lisp_obj_i),
+                                    lwordobj);
+      return gcc_jit_lvalue_as_rvalue (tmp_s);
+    }
+#endif
+
   gcc_jit_field *orig_field =
     type_to_cast_field (old_type);
   gcc_jit_field *dest_field = type_to_cast_field (new_type);
@@ -664,14 +702,8 @@ emit_binary_op (enum gcc_jit_binary_op op,
 /* Should come with libgccjit.  */
 
 static gcc_jit_rvalue *
-emit_rvalue_from_long_long (long long n)
+emit_rvalue_from_long_long (gcc_jit_type *type, long long n)
 {
-#ifndef WIDE_EMACS_INT
-  xsignal1 (Qnative_ice,
-	    build_string ("emit_rvalue_from_long_long called in non wide int"
-			  " configuration"));
-#endif
-
   emit_comment (format_string ("emit long long: %lld", n));
 
   gcc_jit_rvalue *high =
@@ -697,7 +729,7 @@ emit_rvalue_from_long_long (long long n)
 		      32));
 
   return
-    emit_coerce (comp.long_long_type,
+    emit_coerce (type,
       emit_binary_op (
 	GCC_JIT_BINARY_OP_BITWISE_OR,
 	comp.unsigned_long_long_type,
@@ -712,29 +744,135 @@ emit_rvalue_from_long_long (long long n)
 }
 
 static gcc_jit_rvalue *
-emit_most_positive_fixnum (void)
+emit_rvalue_from_unsigned_long_long (gcc_jit_type *type, unsigned long long n)
+{
+  emit_comment (format_string ("emit unsigned long long: %llu", n));
+
+  gcc_jit_rvalue *high =
+    gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+					  comp.unsigned_long_long_type,
+					  n >> 32);
+  gcc_jit_rvalue *low =
+    emit_binary_op (GCC_JIT_BINARY_OP_RSHIFT,
+		    comp.unsigned_long_long_type,
+		    emit_binary_op (GCC_JIT_BINARY_OP_LSHIFT,
+				    comp.unsigned_long_long_type,
+				    gcc_jit_context_new_rvalue_from_long (
+				      comp.ctxt,
+				      comp.unsigned_long_long_type,
+				      n),
+				    gcc_jit_context_new_rvalue_from_int (
+				      comp.ctxt,
+				      comp.unsigned_long_long_type,
+				      32)),
+		    gcc_jit_context_new_rvalue_from_int (
+		      comp.ctxt,
+		      comp.unsigned_long_long_type,
+		      32));
+
+  return emit_coerce (
+           type,
+           emit_binary_op (
+             GCC_JIT_BINARY_OP_BITWISE_OR,
+             comp.unsigned_long_long_type,
+             emit_binary_op (
+               GCC_JIT_BINARY_OP_LSHIFT,
+               comp.unsigned_long_long_type,
+               high,
+               gcc_jit_context_new_rvalue_from_int (comp.ctxt,
+                                                    comp.unsigned_long_long_type,
+                                                    32)),
+             low));
+}
+
+static gcc_jit_rvalue *
+emit_rvalue_from_emacs_uint (EMACS_UINT val)
+{
+  if (val != (long) val)
+    {
+      return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+                                                   comp.emacs_uint_type,
+                                                   val);
+    }
+  else
+    {
+      return emit_rvalue_from_unsigned_long_long (comp.emacs_uint_type, val);
+    }
+}
+
+static gcc_jit_rvalue *
+emit_rvalue_from_emacs_int (EMACS_INT val)
+{
+  if (val != (long) val)
+    {
+      return emit_rvalue_from_long_long (comp.emacs_int_type, val);
+    }
+  else
+    {
+      return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+                                                   comp.emacs_int_type, val);
+    }
+}
+
+static gcc_jit_rvalue *
+emit_rvalue_from_lisp_word_tag (Lisp_Word_tag val)
 {
-#if EMACS_INT_MAX > LONG_MAX
-  return emit_rvalue_from_long_long (MOST_POSITIVE_FIXNUM);
+  if (val != (long) val)
+    {
+      return emit_rvalue_from_unsigned_long_long (comp.lisp_word_tag_type, val);
+    }
+  else
+    {
+      return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+                                                   comp.lisp_word_tag_type,
+                                                   val);
+    }
+}
+
+static gcc_jit_rvalue *
+emit_rvalue_from_lisp_word (Lisp_Word val)
+{
+#if LISP_WORDS_ARE_POINTERS
+  return gcc_jit_context_new_rvalue_from_ptr (comp.ctxt,
+                                              comp.lisp_word_type,
+                                              val);
 #else
-  return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
-					       comp.emacs_int_type,
-					       MOST_POSITIVE_FIXNUM);
+  if (val != (long) val)
+    {
+      return emit_rvalue_from_unsigned_long_long (comp.lisp_word_type, val);
+    }
+  else
+    {
+      return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
+                                                   comp.lisp_word_type,
+                                                   val);
+    }
 #endif
 }
 
 static gcc_jit_rvalue *
-emit_most_negative_fixnum (void)
+emit_rvalue_from_lisp_obj (Lisp_Object obj)
 {
-#if EMACS_INT_MAX > LONG_MAX
-  return emit_rvalue_from_long_long (MOST_NEGATIVE_FIXNUM);
+#ifdef LISP_OBJECT_IS_STRUCT
+  return emit_coerce(comp.lisp_obj_type,
+                     emit_rvalue_from_lisp_word (obj.i));
 #else
-  return gcc_jit_context_new_rvalue_from_long (comp.ctxt,
-					       comp.emacs_int_type,
-					       MOST_NEGATIVE_FIXNUM);
+  return emit_rvalue_from_lisp_word (obj);
 #endif
 }
 
+static gcc_jit_rvalue *
+emit_most_positive_fixnum (void)
+{
+  return emit_rvalue_from_emacs_int(MOST_POSITIVE_FIXNUM);
+}
+
+static gcc_jit_rvalue *
+emit_most_negative_fixnum (void)
+{
+  return emit_rvalue_from_emacs_int (MOST_NEGATIVE_FIXNUM);
+}
+
 /*
    Emit the equivalent of:
    (typeof_ptr) ((uintptr) ptr + size_of_ptr_ref * i)
@@ -769,7 +907,7 @@ emit_ptr_arithmetic (gcc_jit_rvalue *ptr, gcc_jit_type *ptr_type,
 emit_XLI (gcc_jit_rvalue *obj)
 {
   emit_comment ("XLI");
-  return obj;
+  return emit_coerce (comp.emacs_int_type, obj);
 }
 
 static gcc_jit_lvalue *
@@ -779,54 +917,40 @@ emit_lval_XLI (gcc_jit_lvalue *obj)
   return obj;
 }
 
-/*
+
 static gcc_jit_rvalue *
 emit_XLP (gcc_jit_rvalue *obj)
 {
   emit_comment ("XLP");
 
-  return gcc_jit_rvalue_access_field (obj,
-				      NULL,
-				      comp.lisp_obj_as_ptr);
+  return emit_coerce(comp.void_ptr_type, obj);
 }
 
-static gcc_jit_lvalue *
-emit_lval_XLP (gcc_jit_lvalue *obj)
-{
-  emit_comment ("lval_XLP");
+/* TODO */
+/* static gcc_jit_lvalue * */
+/* emit_lval_XLP (gcc_jit_lvalue *obj) */
+/* { */
+/*   emit_comment ("lval_XLP"); */
+
+/*   return gcc_jit_lvalue_access_field (obj, */
+/* 				      NULL, */
+/* 				      comp.lisp_obj_as_ptr); */
+/* } */
 
-  return gcc_jit_lvalue_access_field (obj,
-				      NULL,
-				      comp.lisp_obj_as_ptr);
-} */
 static gcc_jit_rvalue *
-emit_XUNTAG (gcc_jit_rvalue *a, gcc_jit_type *type, long long lisp_word_tag)
+emit_XUNTAG (gcc_jit_rvalue *a, gcc_jit_type *type, Lisp_Word_tag lisp_word_tag)
 {
   /* #define XUNTAG(a, type, ctype) ((ctype *)
      ((char *) XLP (a) - LISP_WORD_TAG (type))) */
   emit_comment ("XUNTAG");
 
-#ifndef WIDE_EMACS_INT
   return emit_coerce (
 	   gcc_jit_type_get_pointer (type),
 	   emit_binary_op (
 	     GCC_JIT_BINARY_OP_MINUS,
-	     comp.emacs_int_type,
-	     emit_XLI (a),
-	     gcc_jit_context_new_rvalue_from_int (
-	       comp.ctxt,
-	       comp.emacs_int_type,
-	       lisp_word_tag)));
-#else
-  return emit_coerce (
-	   gcc_jit_type_get_pointer (type),
-	   emit_binary_op (
-	     GCC_JIT_BINARY_OP_MINUS,
-	     comp.unsigned_long_long_type,
-	     /* FIXME Should be XLP.  */
-	     emit_XLI (a),
-	     emit_rvalue_from_long_long (lisp_word_tag)));
-#endif
+	     comp.uintptr_type,
+	     emit_XLP (a),
+	     emit_rvalue_from_lisp_word_tag(lisp_word_tag)));
 }
 
 static gcc_jit_rvalue *
@@ -853,7 +977,7 @@ emit_EQ (gcc_jit_rvalue *x, gcc_jit_rvalue *y)
 }
 
 static gcc_jit_rvalue *
-emit_TAGGEDP (gcc_jit_rvalue *obj, ptrdiff_t tag)
+emit_TAGGEDP (gcc_jit_rvalue *obj, Lisp_Word_tag tag)
 {
    /* (! (((unsigned) (XLI (a) >> (USE_LSB_TAG ? 0 : VALBITS)) \
 	- (unsigned) (tag)) \
@@ -1054,17 +1178,7 @@ emit_make_fixnum_LSB_TAG (gcc_jit_rvalue *n)
 			comp.emacs_int_type,
 			tmp, comp.lisp_int0);
 
-  gcc_jit_lvalue *res = gcc_jit_function_new_local (comp.func,
-						    NULL,
-						    comp.lisp_obj_type,
-						    "lisp_obj_fixnum");
-
-  gcc_jit_block_add_assignment (comp.block,
-				NULL,
-				emit_lval_XLI (res),
-				tmp);
-
-  return gcc_jit_lvalue_as_rvalue (res);
+  return emit_coerce (comp.lisp_obj_type, tmp);
 }
 
 static gcc_jit_rvalue *
@@ -1076,10 +1190,8 @@ emit_make_fixnum_MSB_TAG (gcc_jit_rvalue *n)
     return XIL (n);
   */
 
-  gcc_jit_rvalue *intmask =
-    emit_coerce (comp.emacs_uint_type,
-		 emit_rvalue_from_long_long ((EMACS_INT_MAX
-					      >> (INTTYPEBITS - 1))));
+  gcc_jit_rvalue *intmask = emit_rvalue_from_emacs_uint (INTMASK);
+
   n = emit_binary_op (GCC_JIT_BINARY_OP_BITWISE_AND,
 		      comp.emacs_uint_type,
 		      intmask, n);
@@ -1090,12 +1202,10 @@ emit_make_fixnum_MSB_TAG (gcc_jit_rvalue *n)
 		    emit_binary_op (GCC_JIT_BINARY_OP_LSHIFT,
 				    comp.emacs_uint_type,
 				    comp.lisp_int0,
-				    gcc_jit_context_new_rvalue_from_int (
-				      comp.ctxt,
-				      comp.emacs_uint_type,
-				      VALBITS)),
+                                    emit_rvalue_from_emacs_uint(VALBITS)),
 		    n);
-  return emit_XLI (emit_coerce (comp.emacs_int_type, n));
+
+  return emit_coerce (comp.lisp_obj_type, n);
 }
 
 
@@ -1114,17 +1224,10 @@ emit_const_lisp_obj (Lisp_Object obj)
   emit_comment (format_string ("const lisp obj: %s",
 			       SSDATA (Fprin1_to_string (obj, Qnil))));
 
-  if (NIL_IS_ZERO && EQ (obj, Qnil))
+  if (EQ (obj, Qnil))
     {
       gcc_jit_rvalue *n;
-#ifdef WIDE_EMACS_INT
-      eassert (NIL_IS_ZERO);
-      n = emit_rvalue_from_long_long (0);
-#else
-      n = gcc_jit_context_new_rvalue_from_ptr (comp.ctxt,
-					       comp.void_ptr_type,
-					       NULL);
-#endif
+      n = emit_rvalue_from_lisp_word ((Lisp_Word) iQnil);
       return emit_coerce (comp.lisp_obj_type, n);
     }
 
@@ -1345,16 +1448,7 @@ emit_mvar_val (Lisp_Object mvar)
 	  /* We can still emit directly objects that are self-contained in a
 	     word (read fixnums).  */
 	  emit_comment (SSDATA (Fprin1_to_string (constant, Qnil)));
-	  gcc_jit_rvalue *word;
-#ifdef WIDE_EMACS_INT
-	  word = emit_rvalue_from_long_long (constant);
-#else
-	  word =
-	    gcc_jit_context_new_rvalue_from_ptr (comp.ctxt,
-						 comp.void_ptr_type,
-						 XLP (constant));
-#endif
-	  return emit_coerce (comp.lisp_obj_type, word);
+          return emit_rvalue_from_lisp_obj(constant);
 	}
       /* Other const objects are fetched from the reloc array.  */
       return emit_const_lisp_obj (constant);
@@ -2518,11 +2612,16 @@ define_cast_union (void)
 			       NULL,
 			       comp.lisp_cons_ptr_type,
 			       "cons_ptr");
-  comp.cast_union_as_lisp_obj =
+  comp.cast_union_as_lisp_word =
     gcc_jit_context_new_field (comp.ctxt,
 			       NULL,
-			       comp.lisp_obj_type,
-			       "lisp_obj");
+			       comp.lisp_word_type,
+			       "lisp_word");
+  comp.cast_union_as_lisp_word_tag =
+    gcc_jit_context_new_field (comp.ctxt,
+                               NULL,
+                               comp.lisp_word_tag_type,
+                               "lisp_word_tag");
   comp.cast_union_as_lisp_obj_ptr =
     gcc_jit_context_new_field (comp.ctxt,
 			       NULL,
@@ -2543,7 +2642,8 @@ define_cast_union (void)
       comp.cast_union_as_c_p,
       comp.cast_union_as_v_p,
       comp.cast_union_as_lisp_cons_ptr,
-      comp.cast_union_as_lisp_obj,
+      comp.cast_union_as_lisp_word,
+      comp.cast_union_as_lisp_word_tag,
       comp.cast_union_as_lisp_obj_ptr };
   comp.cast_union_type =
     gcc_jit_context_new_union_type (comp.ctxt,
@@ -2810,8 +2910,8 @@ define_add1_sub1 (void)
 					  GCC_JIT_COMPARISON_NE,
 					  n_fixnum,
 					  i == 0
-					  ? emit_most_positive_fixnum ()
-					  : emit_most_negative_fixnum ())),
+					  ? emit_rvalue_from_emacs_int (MOST_POSITIVE_FIXNUM)
+					  : emit_rvalue_from_emacs_int (MOST_NEGATIVE_FIXNUM))),
 	inline_block,
 	fcall_block);
 
@@ -3307,9 +3407,31 @@ DEFUN ("comp--init-ctxt", Fcomp__init_ctxt, Scomp__init_ctxt,
   comp.emacs_uint_type = gcc_jit_context_get_int_type (comp.ctxt,
 						       sizeof (EMACS_UINT),
 						       false);
-  /* No XLP is emitted for now so lets define this always as integer
-     disregarding LISP_WORDS_ARE_POINTERS value.  */
-  comp.lisp_obj_type = comp.emacs_int_type;
+#if LISP_WORDS_ARE_POINTERS
+  comp.lisp_X_s = gcc_jit_context_new_opaque_struct (comp.ctxt,
+                                                     NULL,
+                                                     "Lisp_X");
+  comp.lisp_X = gcc_jit_struct_as_type(comp.lisp_X_s);
+  comp.lisp_word_type = gcc_jit_type_get_pointer(comp.lisp_X);
+#else
+  comp.lisp_word_type = comp.emacs_int_type;
+#endif
+  comp.lisp_word_tag_type
+    = gcc_jit_context_get_int_type (comp.ctxt, sizeof (Lisp_Word_tag), false);
+#ifdef LISP_OBJECT_IS_STRUCT
+  comp.lisp_obj_i = gcc_jit_context_new_field (comp.ctxt,
+                                               NULL,
+                                               comp.lisp_word_type,
+                                               "i");
+  comp.lisp_obj_s = gcc_jit_context_new_struct_type (comp.ctxt,
+                                                     NULL,
+                                                     "Lisp_Object",
+                                                     1,
+                                                     &comp.lisp_obj_i);
+  comp.lisp_obj_type = gcc_jit_struct_as_type (comp.lisp_obj_s);
+#else
+  comp.lisp_obj_type = comp.lisp_word_type;
+#endif
   comp.lisp_obj_ptr_type = gcc_jit_type_get_pointer (comp.lisp_obj_type);
   comp.one =
     gcc_jit_context_new_rvalue_from_int (comp.ctxt,
diff --git a/src/lisp.h b/src/lisp.h
index 834b3e586c..e242546d10 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -299,12 +299,12 @@ #define GCALIGNED(type) (alignof (type) % GCALIGNMENT == 0)
 
 /* Lisp_Word is a scalar word suitable for holding a tagged pointer or
    integer.  Usually it is a pointer to a deliberately-incomplete type
-   'union Lisp_X'.  However, it is EMACS_INT when Lisp_Objects and
+   'struct Lisp_X'.  However, it is EMACS_INT when Lisp_Objects and
    pointers differ in width.  */
 
 #define LISP_WORDS_ARE_POINTERS (EMACS_INT_MAX == INTPTR_MAX)
 #if LISP_WORDS_ARE_POINTERS
-typedef union Lisp_X *Lisp_Word;
+typedef struct Lisp_X *Lisp_Word;
 #else
 typedef EMACS_INT Lisp_Word;
 #endif
@@ -573,6 +573,7 @@ #define ENUM_BF(TYPE) enum TYPE
 
 #ifdef CHECK_LISP_OBJECT_TYPE
 typedef struct Lisp_Object { Lisp_Word i; } Lisp_Object;
+# define LISP_OBJECT_IS_STRUCT
 # define LISP_INITIALLY(w) {w}
 # undef CHECK_LISP_OBJECT_TYPE
 enum CHECK_LISP_OBJECT_TYPE { CHECK_LISP_OBJECT_TYPE = true };
-- 
2.25.1.windows.1


[-- Attachment #4: 0001-Determine-the-emacs-root-dir-only-when-necessary.patch --]
[-- Type: application/octet-stream, Size: 4341 bytes --]

From a7176a64d4d882874d4a81cc3924e03fa2a09ce8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Wed, 13 May 2020 18:31:42 -0300
Subject: [PATCH 1/9] Determine the emacs root dir only when necessary.

* src/fileio.c: Introduce function emacs_root_dir(). Refactor
`expand-file-name` to use it.
* src/lisp.h: Separate emacs_root_dir() into dos_emacs_root_dir() and
w32_emacs_root_dir().
* src/msdos.c: Rename emacs_root_dir() to dos_emacs_root_dir().
* src/w32.c: Rename emacs_root_dir() to w32_emacs_root_dir().
---
 src/fileio.c | 37 ++++++++++++++++++++++---------------
 src/lisp.h   | 11 +++++++----
 src/msdos.c  |  2 +-
 src/w32.c    |  2 +-
 4 files changed, 31 insertions(+), 21 deletions(-)

diff --git a/src/fileio.c b/src/fileio.c
index 2f1d2f8243..e20fa93c65 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -781,6 +781,18 @@ user_homedir (char const *name)
   return pw->pw_dir;
 }
 
+static Lisp_Object
+emacs_root_dir()
+{
+#ifdef DOS
+      return build_string (dos_emacs_root_dir ());
+#elif defined(WINDOWSNT)
+      return build_string (w32_emacs_root_dir ());
+#else
+      return build_string ("/");
+#endif
+}
+
 DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
        doc: /* Convert filename NAME to absolute, and canonicalize it.
 Second arg DEFAULT-DIRECTORY is directory to start with if NAME is relative
@@ -851,21 +863,16 @@ DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
     }
 
   /* As a last resort, we may have to use the root as
-     default_directory below.  */
-  Lisp_Object root;
-#ifdef DOS_NT
-      /* "/" is not considered a root directory on DOS_NT, so using it
-	 as default_directory causes an infinite recursion in, e.g.,
-	 the following:
+     default_directory below.
 
-            (let (default-directory)
-	      (expand-file-name "a"))
+     "/" is not considered a root directory on DOS_NT, so using it
+     as default_directory causes an infinite recursion in, e.g.,
+     the following:
 
-	 To avoid this, we use the root of the current drive.  */
-      root = build_string (emacs_root_dir ());
-#else
-      root = build_string ("/");
-#endif
+        (let (default-directory)
+          (expand-file-name "a"))
+
+     To avoid this, we use the root of the current drive.  */
 
   /* Use the buffer's default-directory if DEFAULT_DIRECTORY is omitted.  */
   if (NILP (default_directory))
@@ -891,13 +898,13 @@ DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
 	      Lisp_Object absdir
 		= STRINGP (Vinvocation_directory)
 		&& file_name_absolute_no_tilde_p (Vinvocation_directory)
-		? Vinvocation_directory : root;
+		? Vinvocation_directory : emacs_root_dir();
 	      default_directory = Fexpand_file_name (dir, absdir);
 	    }
 	}
     }
   if (! STRINGP (default_directory))
-    default_directory = root;
+    default_directory = emacs_root_dir();
 
   handler = Ffind_file_name_handler (default_directory, Qexpand_file_name);
   if (!NILP (handler))
diff --git a/src/lisp.h b/src/lisp.h
index 3d082911f5..834b3e586c 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4719,10 +4719,13 @@ maybe_disable_address_randomization (int argc, char **argv)
 extern void malloc_probe (size_t);
 extern void syms_of_profiler (void);
 
-#ifdef DOS_NT
-/* Defined in msdos.c, w32.c.  */
-extern char *emacs_root_dir (void);
-#endif /* DOS_NT */
+#ifdef MSDOS
+/* Defined in msdos.c.  */
+extern char *dos_emacs_root_dir (void);
+#elif defined(WINDOWSNT)
+/* Defined in w32.c.  */
+extern char *w32_emacs_root_dir (void);
+#endif /* MSDOS */
 
 #ifdef HAVE_NATIVE_COMP
 INLINE bool
diff --git a/src/msdos.c b/src/msdos.c
index b5f06c99c3..0827cc96cd 100644
--- a/src/msdos.c
+++ b/src/msdos.c
@@ -3350,7 +3350,7 @@ getdefdir (int drive, char *dst)
 }
 
 char *
-emacs_root_dir (void)
+dos_emacs_root_dir (void)
 {
   static char root_dir[4];
 
diff --git a/src/w32.c b/src/w32.c
index 0f69e652a5..1ec0094c8e 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -3147,7 +3147,7 @@ #define SET_ENV_BUF_SIZE (4 * MAX_PATH)	/* to cover EMACSLOADPATH */
 /* Called from expand-file-name when default-directory is not a string.  */
 
 char *
-emacs_root_dir (void)
+w32_emacs_root_dir (void)
 {
   static char root_dir[MAX_UTF8_PATH];
   const char *p;
-- 
2.25.1.windows.1


[-- Attachment #5: 0002-Do-not-block-SIGIO-in-platforms-that-don-t-have-it.patch --]
[-- Type: application/octet-stream, Size: 909 bytes --]

From e8e224fbf97c6346a355820e62aa4239a9c63a86 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 16:02:58 -0300
Subject: [PATCH 2/9] Do not block SIGIO in platforms that don't have it.

* src/comp.c (comp--compile-ctxt-to-file): Add a preprocessor check to
avoid blocking SIGIO in platforms that don't have it.
---
 src/comp.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/comp.c b/src/comp.c
index e3a80adfa9..00f9d4b74a 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -3345,7 +3345,9 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
       sigemptyset (&blocked);
       sigaddset (&blocked, SIGALRM);
       sigaddset (&blocked, SIGINT);
+#ifdef USABLE_SIGIO
       sigaddset (&blocked, SIGIO);
+#endif
       pthread_sigmask (SIG_BLOCK, &blocked, &oldset);
     }
   emit_ctxt_code ();
-- 
2.25.1.windows.1


[-- Attachment #6: 0005-Remove-a-layer-of-indirection-for-access-to-pure-sto.patch --]
[-- Type: application/octet-stream, Size: 3345 bytes --]

From 8dbe0c99ecbaa1b153ca6b3af02f9b5e96d63d0b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 16:23:10 -0300
Subject: [PATCH 5/9] Remove a layer of indirection for access to pure storage.

* src/comp.c: Taking the address of an array is the same as casting it
to a pointer. Therefore, the C expression `(EMACS_INT **) &pure` is in
fact adding a layer of indirection that is not necessary. The fix is
to cast the `pure` array to a pointer and store that in a void pointer
that is part of the compiled shared library.
---
 src/comp.c | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 0d5880ad3c..69525acfc0 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -38,7 +38,7 @@
 
 /* C symbols emitted for the load relocation mechanism.  */
 #define CURRENT_THREAD_RELOC_SYM "current_thread_reloc"
-#define PURE_RELOC_SYM "pure_reloc"
+#define PURE_PTR_SYM "pure_ptr"
 #define DATA_RELOC_SYM "d_reloc"
 #define DATA_RELOC_IMPURE_SYM "d_reloc_imp"
 #define DATA_RELOC_EPHEMERAL_SYM "d_reloc_eph"
@@ -152,7 +152,7 @@ #define F_RELOC_MAX_SIZE 1500
   gcc_jit_type *thread_state_ptr_type;
   gcc_jit_rvalue *current_thread_ref;
   /* Other globals.  */
-  gcc_jit_rvalue *pure_ref;
+  gcc_jit_rvalue *pure_ptr;
   /* libgccjit has really limited support for casting therefore this union will
      be used for the scope.  */
   gcc_jit_type *cast_union_type;
@@ -1419,8 +1419,7 @@ emit_PURE_P (gcc_jit_rvalue *ptr)
 	GCC_JIT_BINARY_OP_MINUS,
 	comp.uintptr_type,
 	ptr,
-	gcc_jit_lvalue_as_rvalue (
-	  gcc_jit_rvalue_dereference (comp.pure_ref, NULL))),
+        comp.pure_ptr),
       gcc_jit_context_new_rvalue_from_int (comp.ctxt,
 					   comp.uintptr_type,
 					   PURESIZE));
@@ -2244,14 +2243,14 @@ emit_ctxt_code (void)
 	gcc_jit_type_get_pointer (comp.thread_state_ptr_type),
 	CURRENT_THREAD_RELOC_SYM));
 
-  comp.pure_ref =
+  comp.pure_ptr =
     gcc_jit_lvalue_as_rvalue (
       gcc_jit_context_new_global (
 	comp.ctxt,
 	NULL,
 	GCC_JIT_GLOBAL_EXPORTED,
-	gcc_jit_type_get_pointer (comp.void_ptr_type),
-	PURE_RELOC_SYM));
+        comp.void_ptr_type,
+	PURE_PTR_SYM));
 
   gcc_jit_context_new_global (
 	comp.ctxt,
@@ -3767,13 +3766,13 @@ load_comp_unit (struct Lisp_Native_Comp_Unit *comp_u, bool loading_dump,
     {
       struct thread_state ***current_thread_reloc =
 	dynlib_sym (handle, CURRENT_THREAD_RELOC_SYM);
-      EMACS_INT ***pure_reloc = dynlib_sym (handle, PURE_RELOC_SYM);
+      void **pure_ptr = dynlib_sym (handle, PURE_PTR_SYM);
       Lisp_Object *data_relocs = dynlib_sym (handle, DATA_RELOC_SYM);
       Lisp_Object *data_imp_relocs = dynlib_sym (handle, DATA_RELOC_IMPURE_SYM);
       void **freloc_link_table = dynlib_sym (handle, FUNC_LINK_TABLE_SYM);
 
       if (!(current_thread_reloc
-	    && pure_reloc
+	    && pure_ptr
 	    && data_relocs
 	    && data_imp_relocs
 	    && data_eph_relocs
@@ -3784,7 +3783,7 @@ load_comp_unit (struct Lisp_Native_Comp_Unit *comp_u, bool loading_dump,
 	xsignal1 (Qnative_lisp_file_inconsistent, comp_u->file);
 
       *current_thread_reloc = &current_thread;
-      *pure_reloc = (EMACS_INT **)&pure;
+      *pure_ptr = pure;
 
       /* Imported functions.  */
       *freloc_link_table = freloc.link_table;
-- 
2.25.1.windows.1


[-- Attachment #7: 0008-Windows-Use-NUMBER_OF_PROCESSORS-environment-variabl.patch --]
[-- Type: application/octet-stream, Size: 1433 bytes --]

From 1fb5c38748c12735ee1d6eb6e794d61720e4fc11 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Wed, 13 May 2020 16:22:17 -0300
Subject: [PATCH 8/9] Windows: Use NUMBER_OF_PROCESSORS environment variable.

* lisp/emacs-lisp/comp.el (comp-effective-async-max-jobs): Use
NUMBER_OF_PROCESSORS environment variable if system is Windows NT,
"nproc" if it is in PATH or a default of 1.
---
 lisp/emacs-lisp/comp.el | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index d32f93a1e1..26bb79fcd1 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -2208,9 +2208,11 @@ comp-async-runnings
     (if (zerop comp-async-jobs-number)
         (or num-cpus
             (setf num-cpus
-                  ;; Half of the CPUs or at least one.
-                  ;; FIXME portable?
-                  (max 1 (/ (string-to-number (shell-command-to-string "nproc"))
+                  (max 1 (/ (cond ((eq 'windows-nt system-type)
+                                   (string-to-number (getenv "NUMBER_OF_PROCESSORS")))
+                                  ((executable-find "nproc")
+                                   (string-to-number (shell-command-to-string "nproc")))
+                                  (t 1))
                             2))))
       comp-async-jobs-number)))
 
-- 
2.25.1.windows.1


[-- Attachment #8: 0009-Improve-handling-of-native-compilation-units-still-i.patch --]
[-- Type: application/octet-stream, Size: 19739 bytes --]

From d5ee4185a9f23462cdb3fa34cafef508ddb9b81b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Tue, 19 May 2020 15:57:31 -0300
Subject: [PATCH 9/9] Improve handling of native compilation units still in use
 in Windows

When closing emacs will inspect all directories from which it loaded
native compilation units. If it finds a ".eln.old" file it will try to
delete it, if it fails that means that another Emacs instance is using it.

When compiling a file we rename the file that was in the output path
in case it has been loaded into another Emacs instance.

When deleting a package we move any ".eln" or ".eln.old" files in the
package folder that we can't delete to `package-user-dir`. Emacs will
check that directory when closing and delete them.

* lisp/emacs-lisp/comp.el (comp--replace-output-file): Function called
from C code to finish the compilation process. It performs renaming of
the old file if necessary.
* lisp/emacs-lisp/package.el (package--delete-directory): Function to
delete a package directory. It moves native compilation units that it
can't delete to `package-user-dir'.
* src/alloc.c (cleanup_vector): Call dispose_comp_unit().
  (garbage_collect): Call finish_delayed_disposal_of_comp_units().
* src/comp.c: Restore the signal mask using unwind-protect. Store
loaded native compilation units in a hash table for disposal on
close. Store filenames of native compilation units GC'd in a linked
list to finish their disposal when the GC is over.
* src/comp.h: Introduce cfile member in Lisp_Native_Comp_Unit.
Add declarations of functions that: clean directories of unused native
compilation units, handle disposal of native compilation units.
* src/emacs.c (kill-emacs): Dispose all remaining compilation units
right right before calling exit().
* src/eval.c (internal_condition_case_3, internal_condition_case_4):
Add functions.
* src/lisp.h (internal_condition_case_3, internal_condition_case_4):
Add functions.
* src/pdumper.c (dump_do_dump_relocation): Set cfile to a copy of the
Lisp string specifying the file path.
---
 lisp/emacs-lisp/comp.el    |  25 ++++++
 lisp/emacs-lisp/package.el |  27 +++++-
 src/alloc.c                |   5 +-
 src/comp.c                 | 174 ++++++++++++++++++++++++++++++++++---
 src/comp.h                 |  27 ++++++
 src/emacs.c                |   3 +
 src/eval.c                 |  55 ++++++++++++
 src/lisp.h                 |   2 +
 src/pdumper.c              |   3 +
 9 files changed, 308 insertions(+), 13 deletions(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 26bb79fcd1..ee72c92991 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -2183,6 +2183,31 @@ comp-hint-cons
 \f
 ;; Some entry point support code.
 
+(defun comp--replace-output-file (outfile tmpfile)
+  "Replace OUTFILE with TMPFILE taking the necessary steps when
+dealing with shared libraries that may be loaded into Emacs"
+  (cond ((eq 'windows-nt system-type)
+         (ignore-errors (delete-file outfile))
+         (let ((retry t))
+           (while retry
+             (setf retry nil)
+             (condition-case _
+                 (progn
+                   ;; outfile maybe recreated by another Emacs in
+                   ;; between the following two rename-file calls
+                   (if (file-exists-p outfile)
+                       (rename-file outfile (make-temp-file-internal
+                                             (file-name-sans-extension outfile)
+                                             nil ".eln.old" nil)
+                                    t))
+                   (rename-file tmpfile outfile nil))
+               (file-already-exists (setf retry t))))))
+        ;; Remove the old eln instead of copying the new one into it
+        ;; to get a new inode and prevent crashes in case the old one
+        ;; is currently loaded.
+        (t (delete-file outfile)
+           (rename-file tmpfile outfile))))
+
 (defvar comp-files-queue ()
   "List of Elisp files to be compiled.")
 
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 95659840ad..c1c54b3c9a 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2184,6 +2184,31 @@ package--newest-p
   (equal (cadr (assq (package-desc-name pkg) package-alist))
          pkg))
 
+(defun package--delete-directory (dir)
+  "Delete DIR recursively.
+In Windows move .eln and .eln.old files that can not be deleted to `package-user-dir'."
+  (cond ((eq 'windows-nt system-type)
+         (let ((retry t))
+           (while retry
+             (setf retry nil)
+             (condition-case err
+                 (delete-directory dir t)
+               (file-error
+                (if (and (string= "Removing old name" (cadr err))
+                         (string= "Permission denied" (caddr err))
+                         (or (string-suffix-p ".eln" (cadddr err))
+                             (string-suffix-p ".eln.old" (cadddr err))))
+                    (progn
+                      (rename-file (cadddr err)
+                                   (make-temp-file-internal
+                                    (concat package-user-dir
+                                            (file-name-base (cadddr err)))
+                                    nil ".eln.old" nil)
+                                   t)
+                      (setf retry t))
+                  (signal (car err) (cdr err))))))))
+        (t (delete-directory dir t))))
+
 (defun package-delete (pkg-desc &optional force nosave)
   "Delete package PKG-DESC.
 
@@ -2236,7 +2261,7 @@ package-delete
                   (package-desc-name pkg-used-elsewhere-by)))
           (t
            (add-hook 'post-command-hook #'package-menu--post-refresh)
-           (delete-directory dir t)
+           (package--delete-directory dir)
            ;; Remove NAME-VERSION.signed and NAME-readme.txt files.
            ;;
            ;; NAME-readme.txt files are no longer created, but they
diff --git a/src/alloc.c b/src/alloc.c
index f2b80fac88..17f5e15b35 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -3119,8 +3119,7 @@ cleanup_vector (struct Lisp_Vector *vector)
     {
       struct Lisp_Native_Comp_Unit *cu =
 	PSEUDOVEC_STRUCT (vector, Lisp_Native_Comp_Unit);
-      eassert (cu->handle);
-      dynlib_close (cu->handle);
+      dispose_comp_unit (cu, true);
     }
 }
 
@@ -6119,6 +6118,8 @@ garbage_collect (void)
       if (tot_after < tot_before)
 	malloc_probe (min (tot_before - tot_after, SIZE_MAX));
     }
+
+  finish_delayed_disposal_of_comp_units ();
 }
 
 DEFUN ("garbage-collect", Fgarbage_collect, Sgarbage_collect, 0, 0, "",
diff --git a/src/comp.c b/src/comp.c
index b43d3eddb3..35e1ec0da3 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -413,6 +413,10 @@ #define THIRD(x)				\
 #define CALL1I(fun, arg)				\
   CALLN (Ffuncall, intern_c_string (STR (fun)), arg)
 
+/* Like call2 but stringify and intern.  */
+#define CALL2I(fun, arg1, arg2)				\
+  CALLN (Ffuncall, intern_c_string (STR (fun)), arg1, arg2)
+
 #define DECL_BLOCK(name, func)				\
   gcc_jit_block *(name) =				\
     gcc_jit_function_new_block ((func), STR (name))
@@ -3830,6 +3834,14 @@ DEFUN ("comp--release-ctxt", Fcomp__release_ctxt, Scomp__release_ctxt,
   return Qt;
 }
 
+sigset_t oldset;
+
+static void restore_sigmask(void)
+{
+  pthread_sigmask (SIG_SETMASK, &oldset, 0);
+  unblock_input ();
+}
+
 DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
        Scomp__compile_ctxt_to_file,
        1, 1, 0,
@@ -3851,6 +3863,8 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
     CALL1I (comp-data-container-idx, CALL1I (comp-ctxt-d-ephemeral, Vcomp_ctxt));
 
   sigset_t oldset;
+  ptrdiff_t count;
+
   if (!noninteractive)
     {
       sigset_t blocked;
@@ -3863,6 +3877,8 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
       sigaddset (&blocked, SIGIO);
 #endif
       pthread_sigmask (SIG_BLOCK, &blocked, &oldset);
+      count = SPECPDL_INDEX ();
+      record_unwind_protect_void(restore_sigmask);
     }
   emit_ctxt_code ();
 
@@ -3902,18 +3918,10 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
 				   GCC_JIT_OUTPUT_KIND_DYNAMIC_LIBRARY,
 				   SSDATA (tmp_file));
 
-  /* Remove the old eln instead of copying the new one into it to get
-     a new inode and prevent crashes in case the old one is currently
-     loaded.  */
-  if (!NILP (Ffile_exists_p (out_file)))
-    Fdelete_file (out_file, Qnil);
-  Frename_file (tmp_file, out_file, Qnil);
+  CALL2I(comp--replace-output-file, out_file, tmp_file);
 
   if (!noninteractive)
-    {
-      pthread_sigmask (SIG_SETMASK, &oldset, 0);
-      unblock_input ();
-    }
+    unbind_to(count, Qnil);
 
   return out_file;
 }
@@ -3974,6 +3982,138 @@ helper_PSEUDOVECTOR_TYPEP_XUNTAG (Lisp_Object a, enum pvec_type code)
 			     code);
 }
 
+\f
+/*********************************/
+/* Disposal of compilation units */
+/*********************************/
+
+#ifdef WINDOWSNT
+#define OLD_ELN_SUFFIX_REGEXP build_string("\\.eln\\.old$")
+
+static Lisp_Object all_loaded_comp_units;
+
+struct delayed_comp_unit_disposal
+{
+  struct delayed_comp_unit_disposal * next;
+  char * filename;
+};
+
+struct delayed_comp_unit_disposal * delayed_comp_unit_disposal_list;
+
+static Lisp_Object
+returnQnil (Lisp_Object arg)
+{
+  return Qnil;
+}
+
+static void
+clean_comp_unit_directory (Lisp_Object filepath)
+{
+  if (NILP (filepath))
+    return;
+  Lisp_Object files_in_dir;
+  files_in_dir = internal_condition_case_4(Fdirectory_files, filepath, Qt,
+                                           OLD_ELN_SUFFIX_REGEXP, Qnil, Qt, returnQnil);
+  FOR_EACH_TAIL(files_in_dir)
+    {
+      DeleteFile (SSDATA (XCAR (files_in_dir)));
+    }
+}
+
+void clean_package_user_dir_of_old_comp_units (void)
+{
+  Lisp_Object package_user_dir = find_symbol_value (intern ("package-user-dir"));
+  if (EQ(package_user_dir, Qunbound) || !STRINGP(package_user_dir))
+    return;
+
+  clean_comp_unit_directory(package_user_dir);
+}
+
+#endif
+
+void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_handle, bool delay)
+{
+  eassert (comp_handle->handle);
+  dynlib_close (comp_handle->handle);
+#ifdef WINDOWSNT
+  if (!delay)
+    {
+      Lisp_Object dirname = internal_condition_case_1(Ffile_name_directory,
+                                                      build_string (comp_handle->cfile),
+                                                      Qt,
+                                                      returnQnil);
+      if (!NILP(dirname))
+        clean_comp_unit_directory (dirname);
+      xfree (comp_handle->cfile);
+      comp_handle->cfile = NULL;
+    }
+  else
+    {
+      struct delayed_comp_unit_disposal * head;
+      head = xmalloc (sizeof (struct delayed_comp_unit_disposal));
+      head->next = delayed_comp_unit_disposal_list;
+      head->filename = comp_handle->cfile;
+      comp_handle->cfile = NULL;
+      delayed_comp_unit_disposal_list = head;
+    }
+#else
+  xfree (comp_handle->file);
+#endif
+}
+
+static void
+register_native_comp_unit (Lisp_Object comp_u)
+{
+#ifdef WINDOWSNT
+  static EMACS_UINT count;
+
+  if (XFIXNUM(Fhash_table_count(all_loaded_comp_units)) >= MOST_POSITIVE_FIXNUM)
+    return;
+
+  while (!NILP(Fgethash(make_fixnum(count), all_loaded_comp_units, Qnil)))
+    count = (count + 1) % MOST_POSITIVE_FIXNUM;
+
+  Fputhash(make_fixnum(count), comp_u, all_loaded_comp_units);
+#endif
+}
+
+void dispose_all_remaining_comp_units (void)
+{
+#ifdef WINDOWSNT
+  struct Lisp_Hash_Table *h = XHASH_TABLE (all_loaded_comp_units);
+
+  for (ptrdiff_t i = 0; i < HASH_TABLE_SIZE (h); ++i)
+    {
+      Lisp_Object k = HASH_KEY (h, i);
+      if (!EQ (k, Qunbound))
+        {
+          Lisp_Object val = HASH_VALUE (h, i);
+          struct Lisp_Native_Comp_Unit * cu = XNATIVE_COMP_UNIT(val);
+          dispose_comp_unit(cu, false);
+        }
+    }
+#endif
+}
+
+void finish_delayed_disposal_of_comp_units (void)
+{
+#ifdef WINDOWSNT
+  for (struct delayed_comp_unit_disposal * item = delayed_comp_unit_disposal_list;
+       delayed_comp_unit_disposal_list;
+       item = delayed_comp_unit_disposal_list)
+    {
+      delayed_comp_unit_disposal_list = item->next;
+      Lisp_Object dirname
+        = internal_condition_case_1 (Ffile_name_directory,
+                                     build_string (item->filename), Qt,
+                                     returnQnil);
+      clean_comp_unit_directory (dirname);
+      xfree(item->filename);
+      xfree(item);
+    }
+#endif
+}
+
 \f
 /***********************************/
 /* Deferred compilation mechanism. */
@@ -4160,6 +4300,12 @@ load_comp_unit (struct Lisp_Native_Comp_Unit *comp_u, bool loading_dump,
       d_vec_len = XFIXNUM (Flength (comp_u->data_impure_vec));
       for (EMACS_INT i = 0; i < d_vec_len; i++)
 	data_imp_relocs[i] = AREF (comp_u->data_impure_vec, i);
+
+      /* If we register them while dumping we will get some entries in
+         the hash table that will be duplicated when pdumper calls
+         load_comp_unit. */
+      if (!will_dump_p())
+        register_native_comp_unit (comp_u_lisp_obj);
     }
 
   if (!loading_dump)
@@ -4273,6 +4419,9 @@ DEFUN ("native-elisp-load", Fnative_elisp_load, Snative_elisp_load, 1, 2, 0,
   if (!comp_u->handle)
     xsignal2 (Qnative_lisp_load_failed, file, build_string (dynlib_error ()));
   comp_u->file = file;
+#ifdef WINDOWSNT
+  comp_u->cfile = xlispstrdup(file);
+#endif
   comp_u->data_vec = Qnil;
   load_comp_unit (comp_u, false, !NILP (late_load));
 
@@ -4417,6 +4566,11 @@ syms_of_comp (void)
   staticpro (&delayed_sources);
   delayed_sources = Qnil;
 
+#ifdef WINDOWSNT
+  staticpro (&all_loaded_comp_units);
+  all_loaded_comp_units = CALLN(Fmake_hash_table, QCweakness, Qvalue);
+#endif
+
   DEFVAR_LISP ("comp-ctxt", Vcomp_ctxt,
 	       doc: /* The compiler context.  */);
   Vcomp_ctxt = Qnil;
diff --git a/src/comp.h b/src/comp.h
index e6ab32ff8e..89ef740fe6 100644
--- a/src/comp.h
+++ b/src/comp.h
@@ -44,7 +44,15 @@ #define COMP_H
   /* Same but for data that cannot be moved to pure space.
      Must be the last lisp object here.   */
   Lisp_Object data_impure_vec;
+
   dynlib_handle_ptr handle;
+#ifdef WINDOWSNT
+  /* We need to store a copy of the original file name in memory that
+     is not subject to GC because the function to dispose native
+     compilation units is called by the GC. By that time the `file'
+     string may have been sweeped. */
+  char * cfile;
+#endif
 };
 
 #ifdef HAVE_NATIVE_COMP
@@ -75,6 +83,14 @@ XNATIVE_COMP_UNIT (Lisp_Object a)
 
 extern void maybe_defer_native_compilation (Lisp_Object function_name,
 					    Lisp_Object definition);
+
+extern void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_unit, bool delay);
+
+extern void finish_delayed_disposal_of_comp_units (void);
+
+extern void dispose_all_remaining_comp_units (void);
+
+extern void clean_package_user_dir_of_old_comp_units (void);
 #else
 
 static inline void
@@ -84,6 +100,17 @@ maybe_defer_native_compilation (Lisp_Object function_name,
 
 extern void syms_of_comp (void);
 
+static inline void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_handle)
+{
+  emacs_abort();
+}
+
+static inline void dispose_all_remaining_comp_units (void)
+{}
+
+static inline void clean_package_user_dir_of_old_comp_units (void)
+{}
+
 #endif
 
 #endif
diff --git a/src/emacs.c b/src/emacs.c
index e75cb58834..b7c89b44ec 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -2393,6 +2393,9 @@ DEFUN ("kill-emacs", Fkill_emacs, Skill_emacs, 0, 1, "P",
       unlink (SSDATA (listfile));
     }
 
+  dispose_all_remaining_comp_units();
+  clean_package_user_dir_of_old_comp_units();
+
   if (FIXNUMP (arg))
     exit_code = (XFIXNUM (arg) < 0
 		 ? XFIXNUM (arg) | INT_MIN
diff --git a/src/eval.c b/src/eval.c
index 1091b08255..a68fc90285 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -1419,6 +1419,61 @@ internal_condition_case_2 (Lisp_Object (*bfun) (Lisp_Object, Lisp_Object),
     }
 }
 
+/* Like internal_condition_case_1 but call BFUN with ARG1, ARG2, ARG3 as
+   its arguments.  */
+
+Lisp_Object
+internal_condition_case_3 (Lisp_Object (*bfun) (Lisp_Object, Lisp_Object,
+                                                Lisp_Object),
+                           Lisp_Object arg1, Lisp_Object arg2, Lisp_Object arg3,
+                           Lisp_Object handlers,
+                           Lisp_Object (*hfun) (Lisp_Object))
+{
+  struct handler *c = push_handler (handlers, CONDITION_CASE);
+  if (sys_setjmp (c->jmp))
+    {
+      Lisp_Object val = handlerlist->val;
+      clobbered_eassert (handlerlist == c);
+      handlerlist = handlerlist->next;
+      return hfun (val);
+    }
+  else
+    {
+      Lisp_Object val = bfun (arg1, arg2, arg3);
+      eassert (handlerlist == c);
+      handlerlist = c->next;
+      return val;
+    }
+}
+
+/* Like internal_condition_case_1 but call BFUN with ARG1, ARG2, ARG3, ARG4 as
+   its arguments.  */
+
+Lisp_Object
+internal_condition_case_4 (Lisp_Object (*bfun) (Lisp_Object, Lisp_Object,
+                                                Lisp_Object, Lisp_Object),
+                           Lisp_Object arg1, Lisp_Object arg2,
+                           Lisp_Object arg3, Lisp_Object arg4,
+                           Lisp_Object handlers,
+                           Lisp_Object (*hfun) (Lisp_Object))
+{
+  struct handler *c = push_handler (handlers, CONDITION_CASE);
+  if (sys_setjmp (c->jmp))
+    {
+      Lisp_Object val = handlerlist->val;
+      clobbered_eassert (handlerlist == c);
+      handlerlist = handlerlist->next;
+      return hfun (val);
+    }
+  else
+    {
+      Lisp_Object val = bfun (arg1, arg2, arg3, arg4);
+      eassert (handlerlist == c);
+      handlerlist = c->next;
+      return val;
+    }
+}
+
 /* Like internal_condition_case but call BFUN with NARGS as first,
    and ARGS as second argument.  */
 
diff --git a/src/lisp.h b/src/lisp.h
index e242546d10..eeac20598c 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4138,6 +4138,8 @@ xsignal (Lisp_Object error_symbol, Lisp_Object data)
 extern Lisp_Object internal_condition_case (Lisp_Object (*) (void), Lisp_Object, Lisp_Object (*) (Lisp_Object));
 extern Lisp_Object internal_condition_case_1 (Lisp_Object (*) (Lisp_Object), Lisp_Object, Lisp_Object, Lisp_Object (*) (Lisp_Object));
 extern Lisp_Object internal_condition_case_2 (Lisp_Object (*) (Lisp_Object, Lisp_Object), Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object (*) (Lisp_Object));
+extern Lisp_Object internal_condition_case_3 (Lisp_Object (*) (Lisp_Object, Lisp_Object, Lisp_Object), Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object (*) (Lisp_Object));
+extern Lisp_Object internal_condition_case_4 (Lisp_Object (*) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object), Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object (*) (Lisp_Object));
 extern Lisp_Object internal_condition_case_n
     (Lisp_Object (*) (ptrdiff_t, Lisp_Object *), ptrdiff_t, Lisp_Object *,
      Lisp_Object, Lisp_Object (*) (Lisp_Object, ptrdiff_t, Lisp_Object *));
diff --git a/src/pdumper.c b/src/pdumper.c
index f837dfc38d..9b0bd472d6 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -5312,6 +5312,9 @@ dump_do_dump_relocation (const uintptr_t dump_base,
 	  concat2 (Vinvocation_directory,
 		   installation_state == LOCAL_BUILD
 		   ? XCDR (comp_u->file) : XCAR (comp_u->file));
+#ifdef WINDOWSNT
+        comp_u->cfile = xlispstrdup(comp_u->file);
+#endif
 	comp_u->handle = dynlib_open (SSDATA (comp_u->file));
 	if (!comp_u->handle)
 	  error ("%s", dynlib_error ());
-- 
2.25.1.windows.1


[-- Attachment #9: 0007-Load-libgccjit-dynamically-in-Windows.patch --]
[-- Type: application/octet-stream, Size: 26333 bytes --]

From 6f5bc7a5e2a0a4ef5258ab246b387396cc1723de Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 11 May 2020 20:43:06 -0300
Subject: [PATCH 7/9] Load libgccjit dynamically in Windows.

* configure.ac: don't add linker flags if compiling on
Windows. Compile dynlib.c if modules or native compilation are
enabled. Always compile comp.c
* lisp/term/w32-win.el: Map 'gccjit to "libgccjit.dll" in
`dynamic-library-alist`.
* src/Makefile.in: Update comments. Update to handle changes in
configure.ac.
* src/comp.c: Add declarations of used libgccjit functions using
DEF_DLL_FN. Add calls to load_gccjit_if_necessary() where necessary.
Add `native-comp-available-p`
* src/comp.h: Remove Fnative_elisp_load. Add syms_of_comp().
* src/emacs.c (main): Always call syms_of_comp()
* src/w32.c (globals_of_w32): Clear Vlibrary_cache when starting
because the libraries loaded when dumping will not be loaded when
starting.
* src/w32fns.c: Add Qgccjit symbol.
---
 configure.ac         |  19 ++-
 lisp/term/w32-win.el |   3 +-
 src/Makefile.in      |   9 +-
 src/comp.c           | 374 ++++++++++++++++++++++++++++++++++++++++++-
 src/comp.h           |   6 +-
 src/emacs.c          |   2 -
 src/w32.c            |   4 +
 src/w32fns.c         |   1 +
 8 files changed, 398 insertions(+), 20 deletions(-)

diff --git a/configure.ac b/configure.ac
index 23b94cf6ca..ea0144f404 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3666,6 +3666,7 @@ AC_DEFUN
 LIBMODULES=
 HAVE_MODULES=no
 MODULES_OBJ=
+NEED_DYNLIB=no
 case $opsys in
   cygwin|mingw32) MODULES_SUFFIX=".dll" ;;
   darwin) MODULES_SUFFIX=".dylib" ;;
@@ -3701,7 +3702,8 @@ AC_DEFUN
 fi
 
 if test "${HAVE_MODULES}" = yes; then
-   MODULES_OBJ="dynlib.o emacs-module.o"
+   MODULES_OBJ="emacs-module.o"
+   NEED_DYNLIB=yes
    AC_DEFINE(HAVE_MODULES, 1, [Define to 1 if dynamic modules are enabled])
    AC_DEFINE_UNQUOTED(MODULES_SUFFIX, "$MODULES_SUFFIX",
      [System extension for dynamic libraries])
@@ -3785,7 +3787,6 @@ AC_DEFUN
 
 HAVE_NATIVE_COMP=no
 LIBGCCJIT_LIB=
-COMP_OBJ=
 if test "${with_nativecomp}" != "no"; then
     emacs_save_LIBS=$LIBS
     LIBS="-lgccjit"
@@ -3793,8 +3794,11 @@ AC_DEFUN
       [AC_LINK_IFELSE([libgccjit_smoke_test], [], [libgccjit_not_found])])
     LIBS=$emacs_save_LIBS
     HAVE_NATIVE_COMP=yes
-    LIBGCCJIT_LIB="-lgccjit -ldl"
-    COMP_OBJ="comp.o"
+    # mingw32 loads the library dynamically.
+    if test "${opsys}" != "mingw32"; then
+      LIBGCCJIT_LIB="-lgccjit -ldl"
+    fi
+    NEED_DYNLIB=yes
     AC_DEFINE(HAVE_NATIVE_COMP, 1, [Define to 1 if you have the libgccjit library (-lgccjit).])
 fi
 if test "${HAVE_NATIVE_COMP}" = yes && test "${HAVE_PDUMPER}" = no; then
@@ -3804,7 +3808,12 @@ AC_DEFUN
   [System extension for native compiled elisp])
 AC_SUBST(HAVE_NATIVE_COMP)
 AC_SUBST(LIBGCCJIT_LIB)
-AC_SUBST(COMP_OBJ)
+
+DYNLIB_OBJ=
+if test "${NEED_DYNLIB}" = yes; then
+  DYNLIB_OBJ="dynlib.o"
+fi
+AC_SUBST(DYNLIB_OBJ)
 
 ### Use -lpng if available, unless '--with-png=no'.
 HAVE_PNG=no
diff --git a/lisp/term/w32-win.el b/lisp/term/w32-win.el
index 5901e0295e..6b9716ca30 100644
--- a/lisp/term/w32-win.el
+++ b/lisp/term/w32-win.el
@@ -289,7 +289,8 @@ libgnutls-version
        '(libxml2 "libxml2-2.dll" "libxml2.dll")
        '(zlib "zlib1.dll" "libz-1.dll")
        '(lcms2 "liblcms2-2.dll")
-       '(json "libjansson-4.dll")))
+       '(json "libjansson-4.dll")
+       '(gccjit "libgccjit.dll")))
 
 ;;; multi-tty support
 (defvar w32-initialized nil
diff --git a/src/Makefile.in b/src/Makefile.in
index 63f909ae14..85709184da 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -241,7 +241,7 @@ LIBZ =
 
 ## system-specific libs for dynamic modules, else empty
 LIBMODULES = @LIBMODULES@
-## dynlib.o emacs-module.o if modules enabled, else empty
+## emacs-module.o if modules enabled, else empty
 MODULES_OBJ = @MODULES_OBJ@
 
 XRANDR_LIBS = @XRANDR_LIBS@
@@ -327,8 +327,9 @@ GMP_LIB =
 GMP_OBJ = @GMP_OBJ@
 
 LIBGCCJIT = @LIBGCCJIT_LIB@
-## dynlib.o comp.o if native compiler is enabled, otherwise empty.
-COMP_OBJ = @COMP_OBJ@
+
+## dynlib.o if necessary, else empty
+DYNLIB_OBJ = @DYNLIB_OBJ@
 
 RUN_TEMACS = ./temacs
 
@@ -418,7 +419,7 @@ base_obj =
 	cmds.o casetab.o casefiddle.o indent.o search.o regex-emacs.o undo.o \
 	alloc.o pdumper.o data.o doc.o editfns.o callint.o \
 	eval.o floatfns.o fns.o font.o print.o lread.o $(MODULES_OBJ) \
-	syntax.o $(UNEXEC_OBJ) bytecode.o $(COMP_OBJ) \
+	syntax.o $(UNEXEC_OBJ) bytecode.o comp.o $(DYNLIB_OBJ) \
 	process.o gnutls.o callproc.o \
 	region-cache.o sound.o timefns.o atimer.o \
 	doprnt.o intervals.o textprop.o composite.o xml.o lcms.o $(NOTIFY_OBJ) \
diff --git a/src/comp.c b/src/comp.c
index 69525acfc0..b43d3eddb3 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -20,6 +20,8 @@
 
 #include <config.h>
 
+#include "lisp.h"
+
 #ifdef HAVE_NATIVE_COMP
 
 #include <setjmp.h>
@@ -28,7 +30,6 @@
 #include <signal.h>
 #include <libgccjit.h>
 
-#include "lisp.h"
 #include "puresize.h"
 #include "window.h"
 #include "dynlib.h"
@@ -36,6 +37,347 @@
 #include "blockinput.h"
 #include "sha512.h"
 
+\f
+/********************************/
+/* Dynamic loading of libgccjit */
+/********************************/
+
+#ifdef WINDOWSNT
+# include "w32common.h"
+
+#undef gcc_jit_block_add_assignment
+#undef gcc_jit_block_add_comment
+#undef gcc_jit_block_add_eval
+#undef gcc_jit_block_end_with_conditional
+#undef gcc_jit_block_end_with_jump
+#undef gcc_jit_block_end_with_return
+#undef gcc_jit_block_end_with_void_return
+#undef gcc_jit_context_acquire
+#undef gcc_jit_context_compile_to_file
+#undef gcc_jit_context_dump_reproducer_to_file
+#undef gcc_jit_context_dump_to_file
+#undef gcc_jit_context_get_builtin_function
+#undef gcc_jit_context_get_first_error
+#undef gcc_jit_context_get_int_type
+#undef gcc_jit_context_get_type
+#undef gcc_jit_context_new_array_access
+#undef gcc_jit_context_new_array_type
+#undef gcc_jit_context_new_binary_op
+#undef gcc_jit_context_new_call
+#undef gcc_jit_context_new_call_through_ptr
+#undef gcc_jit_context_new_comparison
+#undef gcc_jit_context_new_field
+#undef gcc_jit_context_new_function
+#undef gcc_jit_context_new_function_ptr_type
+#undef gcc_jit_context_new_global
+#undef gcc_jit_context_new_opaque_struct
+#undef gcc_jit_context_new_param
+#undef gcc_jit_context_new_rvalue_from_int
+#undef gcc_jit_context_new_rvalue_from_long
+#undef gcc_jit_context_new_rvalue_from_ptr
+#undef gcc_jit_context_new_struct_type
+#undef gcc_jit_context_new_unary_op
+#undef gcc_jit_context_new_union_type
+#undef gcc_jit_context_release
+#undef gcc_jit_context_set_bool_option
+#undef gcc_jit_context_set_int_option
+#undef gcc_jit_context_set_logfile
+#undef gcc_jit_function_get_param
+#undef gcc_jit_function_new_block
+#undef gcc_jit_function_new_local
+#undef gcc_jit_lvalue_access_field
+#undef gcc_jit_lvalue_as_rvalue
+#undef gcc_jit_lvalue_get_address
+#undef gcc_jit_param_as_lvalue
+#undef gcc_jit_param_as_rvalue
+#undef gcc_jit_rvalue_access_field
+#undef gcc_jit_rvalue_dereference
+#undef gcc_jit_rvalue_dereference_field
+#undef gcc_jit_rvalue_get_type
+#undef gcc_jit_struct_as_type
+#undef gcc_jit_struct_set_fields
+#undef gcc_jit_type_get_pointer
+
+/* In alphabetical order */
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_rvalue_from_int,
+            (gcc_jit_context *ctxt, gcc_jit_type *numeric_type, int value));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_lvalue_as_rvalue,
+            (gcc_jit_lvalue *lvalue));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_rvalue_access_field,
+            (gcc_jit_rvalue *struct_or_union, gcc_jit_location *loc,
+             gcc_jit_field *field));
+DEF_DLL_FN (void, gcc_jit_block_add_comment,
+            (gcc_jit_block *block, gcc_jit_location *loc, const char *text));
+DEF_DLL_FN (void, gcc_jit_context_release, (gcc_jit_context *ctxt));
+DEF_DLL_FN (const char *, gcc_jit_context_get_first_error,
+            (gcc_jit_context *ctxt));
+DEF_DLL_FN (gcc_jit_block *, gcc_jit_function_new_block,
+            (gcc_jit_function *func, const char *name));
+DEF_DLL_FN (gcc_jit_context *, gcc_jit_context_acquire, (void));
+DEF_DLL_FN (gcc_jit_field *, gcc_jit_context_new_field,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, gcc_jit_type *type,
+             const char *name));
+DEF_DLL_FN (gcc_jit_function *, gcc_jit_context_get_builtin_function,
+            (gcc_jit_context *ctxt, const char *name));
+DEF_DLL_FN (gcc_jit_function *, gcc_jit_context_new_function,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_function_kind kind, gcc_jit_type *return_type,
+             const char *name, int num_params, gcc_jit_param **params,
+             int is_variadic));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_context_new_array_access,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, gcc_jit_rvalue *ptr,
+             gcc_jit_rvalue *index));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_context_new_global,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_global_kind kind, gcc_jit_type *type,
+             const char *name));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_function_new_local,
+            (gcc_jit_function *func, gcc_jit_location *loc, gcc_jit_type *type,
+             const char *name));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_lvalue_access_field,
+            (gcc_jit_lvalue *struct_or_union, gcc_jit_location *loc,
+             gcc_jit_field *field));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_param_as_lvalue, (gcc_jit_param *param));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_rvalue_dereference,
+            (gcc_jit_rvalue *rvalue, gcc_jit_location *loc));
+DEF_DLL_FN (gcc_jit_lvalue *, gcc_jit_rvalue_dereference_field,
+            (gcc_jit_rvalue *ptr, gcc_jit_location *loc, gcc_jit_field *field));
+DEF_DLL_FN (gcc_jit_param *, gcc_jit_context_new_param,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, gcc_jit_type *type,
+             const char *name));
+DEF_DLL_FN (gcc_jit_param *, gcc_jit_function_get_param,
+            (gcc_jit_function *func, int index));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_binary_op,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_binary_op op, gcc_jit_type *result_type,
+             gcc_jit_rvalue *a, gcc_jit_rvalue *b));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_call,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             gcc_jit_function *func, int numargs , gcc_jit_rvalue **args));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_call_through_ptr,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             gcc_jit_rvalue *fn_ptr, int numargs, gcc_jit_rvalue **args));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_comparison,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_comparison op, gcc_jit_rvalue *a, gcc_jit_rvalue *b));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_rvalue_from_long,
+            (gcc_jit_context *ctxt, gcc_jit_type *numeric_type, long value));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_rvalue_from_ptr,
+            (gcc_jit_context *ctxt, gcc_jit_type *pointer_type, void *value));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_unary_op,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             enum gcc_jit_unary_op op, gcc_jit_type *result_type,
+             gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_lvalue_get_address,
+            (gcc_jit_lvalue *lvalue, gcc_jit_location *loc));
+DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_param_as_rvalue, (gcc_jit_param *param));
+DEF_DLL_FN (gcc_jit_struct *, gcc_jit_context_new_opaque_struct,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, const char *name));
+DEF_DLL_FN (gcc_jit_struct *, gcc_jit_context_new_struct_type,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, const char *name,
+             int num_fields, gcc_jit_field **fields));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_get_int_type,
+            (gcc_jit_context *ctxt, int num_bytes, int is_signed));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_get_type,
+            (gcc_jit_context *ctxt, enum gcc_jit_types type_));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_new_array_type,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             gcc_jit_type *element_type, int num_elements));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_new_function_ptr_type,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc,
+             gcc_jit_type *return_type, int num_params,
+             gcc_jit_type **param_types, int is_variadic));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_context_new_union_type,
+            (gcc_jit_context *ctxt, gcc_jit_location *loc, const char *name,
+             int num_fields, gcc_jit_field **fields));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_rvalue_get_type, (gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_struct_as_type,
+            (gcc_jit_struct *struct_type));
+DEF_DLL_FN (gcc_jit_type *, gcc_jit_type_get_pointer, (gcc_jit_type *type));
+DEF_DLL_FN (void, gcc_jit_block_add_assignment,
+            (gcc_jit_block *block, gcc_jit_location *loc, gcc_jit_lvalue *lvalue,
+             gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (void, gcc_jit_block_add_eval,
+            (gcc_jit_block *block, gcc_jit_location *loc,
+             gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (void, gcc_jit_block_end_with_conditional,
+            (gcc_jit_block *block, gcc_jit_location *loc,
+             gcc_jit_rvalue *boolval, gcc_jit_block *on_true,
+             gcc_jit_block *on_false));
+DEF_DLL_FN (void, gcc_jit_block_end_with_jump,
+            (gcc_jit_block *block, gcc_jit_location *loc,
+             gcc_jit_block *target));
+DEF_DLL_FN (void, gcc_jit_block_end_with_return,
+            (gcc_jit_block *block, gcc_jit_location *loc,
+             gcc_jit_rvalue *rvalue));
+DEF_DLL_FN (void, gcc_jit_block_end_with_void_return,
+            (gcc_jit_block *block, gcc_jit_location *loc));
+DEF_DLL_FN (void, gcc_jit_context_compile_to_file,
+            (gcc_jit_context *ctxt, enum gcc_jit_output_kind output_kind,
+             const char *output_path));
+DEF_DLL_FN (void, gcc_jit_context_dump_reproducer_to_file,
+            (gcc_jit_context *ctxt, const char *path));
+DEF_DLL_FN (void, gcc_jit_context_dump_to_file,
+            (gcc_jit_context *ctxt, const char *path, int update_locations));
+DEF_DLL_FN (void, gcc_jit_context_set_bool_option,
+            (gcc_jit_context *ctxt, enum gcc_jit_bool_option opt, int value));
+DEF_DLL_FN (void, gcc_jit_context_set_int_option,
+            (gcc_jit_context *ctxt, enum gcc_jit_int_option opt, int value));
+DEF_DLL_FN (void, gcc_jit_context_set_logfile,
+            (gcc_jit_context *ctxt, FILE *logfile, int flags, int verbosity));
+DEF_DLL_FN (void, gcc_jit_struct_set_fields,
+            (gcc_jit_struct *struct_type, gcc_jit_location *loc, int num_fields,
+             gcc_jit_field **fields));
+
+static bool
+init_gccjit_functions (void)
+{
+  HMODULE library;
+
+  if (!(library = w32_delayed_load (Qgccjit)))
+    {
+      return false;
+    }
+
+  /* In alphabetical order */
+  LOAD_DLL_FN(library, gcc_jit_block_add_assignment);
+  LOAD_DLL_FN(library, gcc_jit_block_add_comment);
+  LOAD_DLL_FN(library, gcc_jit_block_add_eval);
+  LOAD_DLL_FN(library, gcc_jit_block_end_with_conditional);
+  LOAD_DLL_FN(library, gcc_jit_block_end_with_jump);
+  LOAD_DLL_FN(library, gcc_jit_block_end_with_return);
+  LOAD_DLL_FN(library, gcc_jit_block_end_with_void_return);
+  LOAD_DLL_FN(library, gcc_jit_context_acquire);
+  LOAD_DLL_FN(library, gcc_jit_context_compile_to_file);
+  LOAD_DLL_FN(library, gcc_jit_context_dump_reproducer_to_file);
+  LOAD_DLL_FN(library, gcc_jit_context_dump_to_file);
+  LOAD_DLL_FN(library, gcc_jit_context_get_builtin_function);
+  LOAD_DLL_FN(library, gcc_jit_context_get_first_error);
+  LOAD_DLL_FN(library, gcc_jit_context_get_int_type);
+  LOAD_DLL_FN(library, gcc_jit_context_get_type);
+  LOAD_DLL_FN(library, gcc_jit_context_new_array_access);
+  LOAD_DLL_FN(library, gcc_jit_context_new_array_type);
+  LOAD_DLL_FN(library, gcc_jit_context_new_binary_op);
+  LOAD_DLL_FN(library, gcc_jit_context_new_call);
+  LOAD_DLL_FN(library, gcc_jit_context_new_call_through_ptr);
+  LOAD_DLL_FN(library, gcc_jit_context_new_comparison);
+  LOAD_DLL_FN(library, gcc_jit_context_new_field);
+  LOAD_DLL_FN(library, gcc_jit_context_new_function);
+  LOAD_DLL_FN(library, gcc_jit_context_new_function_ptr_type);
+  LOAD_DLL_FN(library, gcc_jit_context_new_global);
+  LOAD_DLL_FN(library, gcc_jit_context_new_opaque_struct);
+  LOAD_DLL_FN(library, gcc_jit_context_new_param);
+  LOAD_DLL_FN(library, gcc_jit_context_new_rvalue_from_int);
+  LOAD_DLL_FN(library, gcc_jit_context_new_rvalue_from_long);
+  LOAD_DLL_FN(library, gcc_jit_context_new_rvalue_from_ptr);
+  LOAD_DLL_FN(library, gcc_jit_context_new_struct_type);
+  LOAD_DLL_FN(library, gcc_jit_context_new_unary_op);
+  LOAD_DLL_FN(library, gcc_jit_context_new_union_type);
+  LOAD_DLL_FN(library, gcc_jit_context_release);
+  LOAD_DLL_FN(library, gcc_jit_context_set_bool_option);
+  LOAD_DLL_FN(library, gcc_jit_context_set_int_option);
+  LOAD_DLL_FN(library, gcc_jit_context_set_logfile);
+  LOAD_DLL_FN(library, gcc_jit_function_get_param);
+  LOAD_DLL_FN(library, gcc_jit_function_new_block);
+  LOAD_DLL_FN(library, gcc_jit_function_new_local);
+  LOAD_DLL_FN(library, gcc_jit_lvalue_access_field);
+  LOAD_DLL_FN(library, gcc_jit_lvalue_as_rvalue);
+  LOAD_DLL_FN(library, gcc_jit_lvalue_get_address);
+  LOAD_DLL_FN(library, gcc_jit_param_as_lvalue);
+  LOAD_DLL_FN(library, gcc_jit_param_as_rvalue);
+  LOAD_DLL_FN(library, gcc_jit_rvalue_access_field);
+  LOAD_DLL_FN(library, gcc_jit_rvalue_dereference);
+  LOAD_DLL_FN(library, gcc_jit_rvalue_dereference_field);
+  LOAD_DLL_FN(library, gcc_jit_rvalue_get_type);
+  LOAD_DLL_FN(library, gcc_jit_struct_as_type);
+  LOAD_DLL_FN(library, gcc_jit_struct_set_fields);
+  LOAD_DLL_FN(library, gcc_jit_type_get_pointer);
+
+  return true;
+}
+
+/* In alphabetical order */
+#define gcc_jit_block_add_assignment fn_gcc_jit_block_add_assignment
+#define gcc_jit_block_add_comment fn_gcc_jit_block_add_comment
+#define gcc_jit_block_add_eval fn_gcc_jit_block_add_eval
+#define gcc_jit_block_end_with_conditional fn_gcc_jit_block_end_with_conditional
+#define gcc_jit_block_end_with_jump fn_gcc_jit_block_end_with_jump
+#define gcc_jit_block_end_with_return fn_gcc_jit_block_end_with_return
+#define gcc_jit_block_end_with_void_return fn_gcc_jit_block_end_with_void_return
+#define gcc_jit_context_acquire fn_gcc_jit_context_acquire
+#define gcc_jit_context_compile_to_file fn_gcc_jit_context_compile_to_file
+#define gcc_jit_context_dump_reproducer_to_file fn_gcc_jit_context_dump_reproducer_to_file
+#define gcc_jit_context_dump_to_file fn_gcc_jit_context_dump_to_file
+#define gcc_jit_context_get_builtin_function fn_gcc_jit_context_get_builtin_function
+#define gcc_jit_context_get_first_error fn_gcc_jit_context_get_first_error
+#define gcc_jit_context_get_int_type fn_gcc_jit_context_get_int_type
+#define gcc_jit_context_get_type fn_gcc_jit_context_get_type
+#define gcc_jit_context_new_array_access fn_gcc_jit_context_new_array_access
+#define gcc_jit_context_new_array_type fn_gcc_jit_context_new_array_type
+#define gcc_jit_context_new_binary_op fn_gcc_jit_context_new_binary_op
+#define gcc_jit_context_new_call fn_gcc_jit_context_new_call
+#define gcc_jit_context_new_call_through_ptr fn_gcc_jit_context_new_call_through_ptr
+#define gcc_jit_context_new_comparison fn_gcc_jit_context_new_comparison
+#define gcc_jit_context_new_field fn_gcc_jit_context_new_field
+#define gcc_jit_context_new_function fn_gcc_jit_context_new_function
+#define gcc_jit_context_new_function_ptr_type fn_gcc_jit_context_new_function_ptr_type
+#define gcc_jit_context_new_global fn_gcc_jit_context_new_global
+#define gcc_jit_context_new_opaque_struct fn_gcc_jit_context_new_opaque_struct
+#define gcc_jit_context_new_param fn_gcc_jit_context_new_param
+#define gcc_jit_context_new_rvalue_from_int fn_gcc_jit_context_new_rvalue_from_int
+#define gcc_jit_context_new_rvalue_from_long fn_gcc_jit_context_new_rvalue_from_long
+#define gcc_jit_context_new_rvalue_from_ptr fn_gcc_jit_context_new_rvalue_from_ptr
+#define gcc_jit_context_new_struct_type fn_gcc_jit_context_new_struct_type
+#define gcc_jit_context_new_unary_op fn_gcc_jit_context_new_unary_op
+#define gcc_jit_context_new_union_type fn_gcc_jit_context_new_union_type
+#define gcc_jit_context_release fn_gcc_jit_context_release
+#define gcc_jit_context_set_bool_option fn_gcc_jit_context_set_bool_option
+#define gcc_jit_context_set_int_option fn_gcc_jit_context_set_int_option
+#define gcc_jit_context_set_logfile fn_gcc_jit_context_set_logfile
+#define gcc_jit_function_get_param fn_gcc_jit_function_get_param
+#define gcc_jit_function_new_block fn_gcc_jit_function_new_block
+#define gcc_jit_function_new_local fn_gcc_jit_function_new_local
+#define gcc_jit_lvalue_access_field fn_gcc_jit_lvalue_access_field
+#define gcc_jit_lvalue_as_rvalue fn_gcc_jit_lvalue_as_rvalue
+#define gcc_jit_lvalue_get_address fn_gcc_jit_lvalue_get_address
+#define gcc_jit_param_as_lvalue fn_gcc_jit_param_as_lvalue
+#define gcc_jit_param_as_rvalue fn_gcc_jit_param_as_rvalue
+#define gcc_jit_rvalue_access_field fn_gcc_jit_rvalue_access_field
+#define gcc_jit_rvalue_dereference fn_gcc_jit_rvalue_dereference
+#define gcc_jit_rvalue_dereference_field fn_gcc_jit_rvalue_dereference_field
+#define gcc_jit_rvalue_get_type fn_gcc_jit_rvalue_get_type
+#define gcc_jit_struct_as_type fn_gcc_jit_struct_as_type
+#define gcc_jit_struct_set_fields fn_gcc_jit_struct_set_fields
+#define gcc_jit_type_get_pointer fn_gcc_jit_type_get_pointer
+
+#endif
+
+static bool
+load_gccjit_if_necessary (bool mandatory)
+{
+#ifdef WINDOWSNT
+  static bool tried_to_initialize_once;
+  static bool gccjit_initialized;
+
+  if (!tried_to_initialize_once)
+    {
+      tried_to_initialize_once = true;
+      Lisp_Object status;
+      gccjit_initialized = init_gccjit_functions ();
+      status = gccjit_initialized ? Qt : Qnil;
+      Vlibrary_cache = Fcons (Fcons (Qgccjit, status), Vlibrary_cache);
+    }
+
+  if (mandatory && !gccjit_initialized)
+    xsignal1(Qnative_compiler_error, build_string("libgccjit not found"));
+
+  return gccjit_initialized;
+#else
+  return true;
+#endif
+}
+
+\f
 /* C symbols emitted for the load relocation mechanism.  */
 #define CURRENT_THREAD_RELOC_SYM "current_thread_reloc"
 #define PURE_PTR_SYM "pure_ptr"
@@ -3328,6 +3670,8 @@ DEFUN ("comp--init-ctxt", Fcomp__init_ctxt, Scomp__init_ctxt,
        doc: /* Initialize the native compiler context. Return t on success.  */)
      (void)
 {
+  load_gccjit_if_necessary(true);
+
   if (comp.ctxt)
     {
       xsignal1 (Qnative_ice,
@@ -3474,6 +3818,8 @@ DEFUN ("comp--release-ctxt", Fcomp__release_ctxt, Scomp__release_ctxt,
        doc: /* Release the native compiler context.  */)
      (void)
 {
+  load_gccjit_if_necessary(true);
+
   if (comp.ctxt)
     gcc_jit_context_release (comp.ctxt);
 
@@ -3490,6 +3836,8 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
        doc: /* Compile as native code the current context to file.  */)
      (Lisp_Object base_name)
 {
+  load_gccjit_if_necessary(true);
+
   CHECK_STRING (base_name);
 
   gcc_jit_context_set_int_option (comp.ctxt,
@@ -3660,6 +4008,9 @@ maybe_defer_native_compilation (Lisp_Object function_name,
       fflush (f);
     }
 #endif
+  if (!load_gccjit_if_necessary(false))
+    return;
+
   if (!comp_deferred_compilation
       || noninteractive
       || !NILP (Vpurify_flag)
@@ -3928,10 +4279,26 @@ DEFUN ("native-elisp-load", Fnative_elisp_load, Snative_elisp_load, 1, 2, 0,
   return Qt;
 }
 
+#endif /* HAVE_NATIVE_COMP */
+
+DEFUN ("native-comp-available-p", Fnative_comp_available_p,
+       Snative_comp_available_p, 0, 0, 0,
+       doc: /* Returns t if native compilation of Lisp files is available in
+this instance of Emacs. */)
+  (void)
+{
+#ifdef HAVE_NATIVE_COMP
+  return load_gccjit_if_necessary(false) ? Qt : Qnil;
+#else
+  return Qnil;
+#endif
+}
+
 \f
 void
 syms_of_comp (void)
 {
+#ifdef HAVE_NATIVE_COMP
   /* Compiler control customizes.  */
   DEFVAR_BOOL ("comp-deferred-compilation", comp_deferred_compilation,
 	       doc: /* If t compile asyncronously every .elc file loaded.  */);
@@ -4073,6 +4440,7 @@ syms_of_comp (void)
 	       doc: /* Hash table symbol-name -> function-value.  For
 		       internal use during  */);
   Vcomp_deferred_pending_h = CALLN (Fmake_hash_table, QCtest, Qeq);
-}
+#endif
 
-#endif /* HAVE_NATIVE_COMP */
+  defsubr (&Snative_comp_available_p);
+}
diff --git a/src/comp.h b/src/comp.h
index cbdcaccd5f..e6ab32ff8e 100644
--- a/src/comp.h
+++ b/src/comp.h
@@ -82,11 +82,7 @@ maybe_defer_native_compilation (Lisp_Object function_name,
 				Lisp_Object definition)
 {}
 
-static inline Lisp_Object
-Fnative_elisp_load (Lisp_Object file, Lisp_Object late_load)
-{
-  eassume (false);
-}
+extern void syms_of_comp (void);
 
 #endif
 
diff --git a/src/emacs.c b/src/emacs.c
index 2c90825742..e75cb58834 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1606,10 +1606,8 @@ main (int argc, char **argv)
   init_json ();
 #endif
 
-#ifdef HAVE_NATIVE_COMP
   if (!initialized)
     syms_of_comp ();
-#endif
 
   no_loadup
     = argmatch (argv, argc, "-nl", "--no-loadup", 6, NULL, &skip_args);
diff --git a/src/w32.c b/src/w32.c
index 1ec0094c8e..fd1f0e059e 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -10586,6 +10586,10 @@ globals_of_w32 (void)
 #endif
 
   w32_crypto_hprov = (HCRYPTPROV)0;
+
+  /* We need to forget about libraries that were loaded during the
+     dumping process (e.g. libgccjit) */
+  Vlibrary_cache = Qnil;
 }
 
 /* For make-serial-process  */
diff --git a/src/w32fns.c b/src/w32fns.c
index e595b0285a..eeb73489dd 100644
--- a/src/w32fns.c
+++ b/src/w32fns.c
@@ -10462,6 +10462,7 @@ syms_of_w32fns (void)
   DEFSYM (Qzlib, "zlib");
   DEFSYM (Qlcms2, "lcms2");
   DEFSYM (Qjson, "json");
+  DEFSYM (Qgccjit, "gccjit");
 
   Fput (Qundefined_color, Qerror_conditions,
 	pure_list (Qundefined_color, Qerror));
-- 
2.25.1.windows.1


[-- Attachment #10: 0006-Workaround-the-32768-chars-command-line-limit-in-Win.patch --]
[-- Type: application/octet-stream, Size: 2422 bytes --]

From 5d0afae4f1ecc0577baf52545b8b656766a1d304 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 8 May 2020 14:04:06 -0300
Subject: [PATCH 6/9] Workaround the 32768 chars command line limit in Windows.

* lisp/emacs-lisp/comp.el (comp-run-async-workers): Pass the
compilation commands through a temporary file that is loaded by the
child process. This is also done all other operating systems, even
those that support long command lines. It should not be a problem
since libgccjit uses temporary files too.
---
 lisp/emacs-lisp/comp.el | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index c2a95feec1..d32f93a1e1 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -2239,6 +2239,9 @@ comp-run-async-workers
                             (message "Compiling %s..." ,source-file)
                             (native-compile ,source-file ,(and load t))))
                    (source-file1 source-file) ;; Make the closure works :/
+                   (temp-file (make-temp-file
+                               (concat "emacs-async-comp-" (file-name-base source-file) "-")
+                               nil ".el" (prin1-to-string expr)))
                    (load1 load)
                    (process (make-process
                              :name (concat "Compiling: " source-file)
@@ -2246,13 +2249,14 @@ comp-run-async-workers
                              :command (list
                                        (expand-file-name invocation-name
                                                          invocation-directory)
-                                       "--batch" "--eval" (prin1-to-string expr))
+                                       "--batch" "-l" temp-file)
                              :sentinel
                              (lambda (process _event)
                                (run-hook-with-args
                                 'comp-async-cu-done-hook
                                 source-file)
                                (accept-process-output process)
+                               (ignore-errors (delete-file temp-file))
                                (when (and load1
                                           (zerop (process-exit-status process)))
                                  (native-elisp-load
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Thanks, I'll start going through them this evening I guess in patch
>> chronological order.

Ok this is in:

68fad7a8fc @ Do not block SIGIO in platforms that don't have it.

Looking at 0003-Handle-setjmp-taking-two-arguments-in-Windows.patch

> +static void
> +define_setjmp_deps (void)
> +{
> +  comp.setjmp_ctx_func
> +    = gcc_jit_context_get_builtin_function (comp.ctxt,
> +                                            "__builtin_frame_address");
> +}

This won't compile on Posix because setjmp_ctx_func is under #ifdef
_WIN32.

Given setjmp_ctx_func is used only in emit_setjmp I suggest to invoke
directly gcc_jit_context_get_builtin_function there while emitting the
corresponding call.

The following patches do not currently apply, please update them from
your rebase branch:

0004-Handle-LISP_WORDS_ARE_POINTERS-and-CHECK_LISP_OBJECT.patch
0005-Remove-a-layer-of-indirection-for-access-to-pure-sto.patch

While you are refreshing patches please have a look into the GNU style
of the diff (spaces in function calls and in pointer declaration) ;)

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

Sorry, still looking at
0003-Handle-setjmp-taking-two-arguments-in-Windows.patch forgot to ask
if you could comment on this change:

>  				      comp.char_type,
> -				      sizeof (jmp_buf)),
> +				      sizeof (sys_jmp_buf)),

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> Date: Wed, 20 May 2020 17:29:37 +0000
> 
> Sorry, still looking at
> 0003-Handle-setjmp-taking-two-arguments-in-Windows.patch forgot to ask
> if you could comment on this change:
> 
> >  				      comp.char_type,
> > -				      sizeof (jmp_buf)),
> > +				      sizeof (sys_jmp_buf)),

I think the latter is more portable, see lisp.h around line 2150.

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> I think the latter is more portable, see lisp.h around line 2150.

I see thanks.

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

> This won't compile on Posix because setjmp_ctx_func is under #ifdef
> _WIN32.

Fixed.

> Given setjmp_ctx_func is used only in emit_setjmp I suggest to invoke
> directly gcc_jit_context_get_builtin_function there while emitting the
> corresponding call.

Done.

> The following patches do not currently apply, please update them from
> your rebase branch:

> 0004-Handle-LISP_WORDS_ARE_POINTERS-and-CHECK_LISP_OBJECT.patch
> 0005-Remove-a-layer-of-indirection-for-access-to-pure-sto.patch

Done.

> While you are refreshing patches please have a look into the GNU style
> of the diff (spaces in function calls and in pointer declaration) ;)

Fixed :).

> Sorry, still looking at
> 0003-Handle-setjmp-taking-two-arguments-in-Windows.patch forgot to ask
> if you could comment on this change:
> >                                     comp.char_type,
> > -                                   sizeof (jmp_buf)),
> > +                                   sizeof (sys_jmp_buf)),

It is just a little bit more portable.

I uploaded new patches to https://github.com/nicber/emacs branch
feature/native-comp. Please tell me I if you'd rather I posted patches here.

[-- Attachment #2: Type: text/html, Size: 1448 bytes --]

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> I uploaded new patches to https://github.com/nicber/emacs branch
> feature/native-comp. Please tell me I if you'd rather I posted
> patches here.

Git works for me thanks.

I've pushed

05b08f2644 * Handle setjmp() taking two arguments in Windows.
5ff2cbdb04 * Remove a layer of indirection for access to pure storage.

I took the freedom to do myself some minors to avoid a re-spin, you can
have a look.

I started testing Handle-LISP_WORDS_ARE_POINTERS-and-CHECK_LISP_OBJECT
because is sensitive.  The space of the configurations is relatively big
and I want to cover the one I can here so I think I'll be done tomorrow.

Thanks

  Andrea

PS are you cooking-up also libgccjit patches?

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 372 bytes --]

 > PS are you cooking-up also libgccjit patches?
Yes, the C++ side is done.
I need help with the build system side of things.
* libiberty.a needs to be copied by hand between folders inside the build
directory.
* libgccjit.dll does not get created in the "bin" folder.
  It is created as libgccjit.so.0.0.1 in the "lib" folder, so it needs to
copied and renamed.

Nicolas

[-- Attachment #2: Type: text/html, Size: 584 bytes --]

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

I keep on working on top of "Handle LISP_WORDS_ARE_POINTERS and
CHECK_LISP_OBJECT_TYPE".

The x86_64 works like a charm and compile time is not affected.

I did some fixing for i386 --enable-check-lisp-object-type, now I'm on
wide-int where we get ICE in GCC.  Hope to sort out quickly but can't
guarantee.

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> PS are you cooking-up also libgccjit patches?
> Yes, the C++ side is done.
> I need help with the build system side of things.
> * libiberty.a needs to be copied by hand between folders inside the
> build directory.
> * libgccjit.dll does not get created in the "bin" folder.
>   It is created as libgccjit.so.0.0.1 in the "lib" folder, so it
> needs to copied and renamed.

Yeah build system is always tricky, but I guess defining a new target to
rename and copy or modifying the existing one for the new name should
not be a problem no?  If you need help the better place to ask and
discuss is jit@gcc.gnu.org tho.

  Andrea

--
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Andrea Corallo]

I've installed "Handle LISP_WORDS_ARE_POINTERS..." plus some fixes
mostly unrelated.

We still ICE GCC (at least my 10 and trunk) on i386 at speed 2 with
--enable-check-lisp-object-type.  This really looks like a libgccjit
bug.  Tomorrow I'll try to look into it better and to progress into
applying patches.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 15:40:28 -0300
> Cc: Eli Zaretskii <eliz@gnu.org>, 41242@debbugs.gnu.org
> 
> * Emacs1 renames foo.eln to foo.eln.old. It tries to delete it but fails since
>   Emacs2 has an open handle.
> 
> * Emacs2 finally calls FreeLibrary() and closes.
> 
> In this case we are left over with a stale foo.eln.old. I don't think we can
> have a race free algorithm.

When Emacs1 fails to delete foo.eln.old, it could record the file's
deletion to be attempted again when it exits.  That won't solve all
the cases, but it will solve some of them.

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: nicolasbertolo@gmail.com, 41242@debbugs.gnu.org
> Date: Thu, 14 May 2020 15:03:08 +0000
> 
> Now simply when the compilation unit object is collected, the handle is
> closed.  The compilation unit is collected when is not reachable by any
> of the functions defined into.
> 
> Loading a new compilation unit B that redefines the same functions
> defined by A does not guarantees much, some of the old definitions of A
> could be still in use somewhere, in that case A will not be collected.
> 
> So yeah I think we need a specific mechanism (kill-emacs hook as you
> suggest) to do the cleanup when Emacs goes down.

Some list of files to remove and a function to remove them at exit
should be sufficient.

bug#41242: Port feature/native-comp to Windows

[Nicolas Bértolo]

> And if push comes to shove, we could use a Windows API
> that causes a file to be deleted when the last handle on it is closed,
> but that would add complexity, so I think we should try easier ways
> first.

If you are talking about FILE_FLAG_DELETE_ON_CLOSE:
I have tried that in the patch I have for the third suggestion.
When Emacs closes normally the temporary files are not deleted.
I don't know why that is. It is as if the LoadLibrary function disabled
the delete on close flag.

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Thu, 14 May 2020 13:24:02 -0300
> Cc: Andrea Corallo <akrl@sdf.org>, 41242@debbugs.gnu.org
> 
> > And if push comes to shove, we could use a Windows API
> > that causes a file to be deleted when the last handle on it is closed,
> > but that would add complexity, so I think we should try easier ways
> > first.
> 
> If you are talking about FILE_FLAG_DELETE_ON_CLOSE:
> I have tried that in the patch I have for the third suggestion.
> When Emacs closes normally the temporary files are not deleted.
> I don't know why that is. It is as if the LoadLibrary function disabled
> the delete on close flag.

Like I said: let's try more traditional ways first ;-)

bug#41242: Port feature/native-comp to Windows

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Wed, 13 May 2020 16:26:57 -0300
> 
> There are a few remaining issues:
> 
> * The loading process is very slow. This is especially annoying when coupled
>   with autoloading. For example, autoloading Helm may stall Emacs for 5 seconds
>   in my machine.
> 
>   I have thought a possible solution to this problem: load the byte-compiled
>   file and put the native-compiled version in a list. Then load that list one by
>   one on an idle-timer, that way the UI freezes should be minimized. This could
>   reuse the "late loading" machinery implemented for deferred compilation.

I believe we didn't investigate this issue well enough to understand
what happens and why.  Would it be possible to investigate this soon?
It could be after the patches are installed.

Thanks.

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Andrea Corallo]

Hi Nicolas,

following some comments on - Improve handling of native compilation
etc.

Please review all the GNU code-style of this diff.  I've annotated some
to be fixed but there are quite a number more of fixes of the same kind
to be done.

> When closing emacs will inspect all directories from which it loaded
> native compilation units. If it finds a ".eln.old" file it will try to
> delete it, if it fails that means that another Emacs instance is using it.
>
> When compiling a file we rename the file that was in the output path
> in case it has been loaded into another Emacs instance.
>
> When deleting a package we move any ".eln" or ".eln.old" files in the
> package folder that we can't delete to `package-user-dir`. Emacs will
> check that directory when closing and delete them.
>
> * lisp/emacs-lisp/comp.el (comp--replace-output-file): Function called
> from C code to finish the compilation process. It performs renaming of
> the old file if necessary.
> * lisp/emacs-lisp/package.el (package--delete-directory): Function to
> delete a package directory. It moves native compilation units that it
> can't delete to `package-user-dir'.
> * src/alloc.c (cleanup_vector): Call dispose_comp_unit().
>   (garbage_collect): Call finish_delayed_disposal_of_comp_units().
> * src/comp.c: Restore the signal mask using unwind-protect. Store
> loaded native compilation units in a hash table for disposal on
> close. Store filenames of native compilation units GC'd in a linked
> list to finish their disposal when the GC is over.

Please annotate in the changelog the new functions ex:
finish_delayed_disposal_of_comp_units, dispose_all_remaining_comp_units,
register_native_comp_unit are missing.

> * src/comp.h: Introduce cfile member in Lisp_Native_Comp_Unit.
> Add declarations of functions that: clean directories of unused native
> compilation units, handle disposal of native compilation units.
> * src/emacs.c (kill-emacs): Dispose all remaining compilation units
> right right before calling exit().
> * src/eval.c (internal_condition_case_3, internal_condition_case_4):
> Add functions.
> * src/lisp.h (internal_condition_case_3, internal_condition_case_4):
> Add functions.
> * src/pdumper.c (dump_do_dump_relocation): Set cfile to a copy of the
> Lisp string specifying the file path.

> diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
> index 012baf2560..1fb4cd98c0 100644
> --- a/lisp/emacs-lisp/comp.el
> +++ b/lisp/emacs-lisp/comp.el
> @@ -2183,6 +2183,31 @@ comp-hint-cons
>  \f
>  ;; Some entry point support code.
>
> +(defun comp--replace-output-file (outfile tmpfile)
> +  "Replace OUTFILE with TMPFILE taking the necessary steps when
> +dealing with shared libraries that may be loaded into Emacs"
> +  (cond ((eq 'windows-nt system-type)
> +         (ignore-errors (delete-file outfile))
> +         (let ((retry t))
> +           (while retry
> +             (setf retry nil)
> +             (condition-case _
> +                 (progn
> +                   ;; outfile maybe recreated by another Emacs in
> +                   ;; between the following two rename-file calls
> +                   (if (file-exists-p outfile)
> +                       (rename-file outfile (make-temp-file-internal
> +                                             (file-name-sans-extension outfile)
> +                                             nil ".eln.old" nil)

Isn't better to just add .old? So we will have cases of foo.eln.old.old
instead of foo.eln.old.eln.old ?

> +                                    t))
> +                   (rename-file tmpfile outfile nil))
> +               (file-already-exists (setf retry t))))))
> +        ;; Remove the old eln instead of copying the new one into it
> +        ;; to get a new inode and prevent crashes in case the old one
> +        ;; is currently loaded.
> +        (t (delete-file outfile)
> +           (rename-file tmpfile outfile))))
> +
>  (defvar comp-files-queue ()
>    "List of Elisp files to be compiled.")



> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index 95659840ad..c1c54b3c9a 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -2184,6 +2184,31 @@ If some packages are not installed propose to install them."
>    (equal (cadr (assq (package-desc-name pkg) package-alist))
>           pkg))
>
> +(defun package--delete-directory (dir)
> +  "Delete DIR recursively.
> +In Windows move .eln and .eln.old files that can not be deleted to `package-user-dir'."

80 column lines limit.  I think also this should be transparent when
native-comp-available-p say native comp is not available (for now
compiler and load machinery are bundled).

> +  (cond ((eq 'windows-nt system-type)
> +         (let ((retry t))
> +           (while retry
> +             (setf retry nil)
> +             (condition-case err
> +                 (delete-directory dir t)
> +               (file-error
> +                (if (and (string= "Removing old name" (cadr err))
> +                         (string= "Permission denied" (caddr err))
> +                         (or (string-suffix-p ".eln" (cadddr err))
> +                             (string-suffix-p ".eln.old" (cadddr err))))

I think would be good to destructure err using something like
cl-destructuring-bind or pcase or even just using a let + some naming to
make this more readable.

> +                    (progn
> +                      (rename-file (cadddr err)
> +                                   (make-temp-file-internal
> +                                    (concat package-user-dir
> +                                            (file-name-base (cadddr err)))
> +                                    nil ".eln.old" nil)
> +                                   t)
> +                      (setf retry t))
> +                  (signal (car err) (cdr err))))))))
> +        (t (delete-directory dir t))))
> +
>  (defun package-delete (pkg-desc &optional force nosave)
>    "Delete package PKG-DESC.
>
> @@ -2236,7 +2261,7 @@ If NOSAVE is non-nil, the package is not removed from
>                    (package-desc-name pkg-used-elsewhere-by)))
>            (t
>             (add-hook 'post-command-hook #'package-menu--post-refresh)
> -           (delete-directory dir t)
> +           (package--delete-directory dir)
>             ;; Remove NAME-VERSION.signed and NAME-readme.txt files.
>             ;;
>             ;; NAME-readme.txt files are no longer created, but they
> diff --git a/src/alloc.c b/src/alloc.c
> index d6ba4d9790..420168ec4d 100644
> --- a/src/alloc.c
> +++ b/src/alloc.c
> @@ -3119,8 +3119,7 @@ cleanup_vector (struct Lisp_Vector *vector)
>      {
>        struct Lisp_Native_Comp_Unit *cu =
>  	PSEUDOVEC_STRUCT (vector, Lisp_Native_Comp_Unit);
> -      eassert (cu->handle);
> -      dynlib_close (cu->handle);
> +      dispose_comp_unit (cu, true);
>      }
>  }
>
> @@ -6117,6 +6116,8 @@ garbage_collect (void)
>        if (tot_after < tot_before)
>  	malloc_probe (min (tot_before - tot_after, SIZE_MAX));
>      }
> +
> +  finish_delayed_disposal_of_comp_units ();

Could you describe why we need to call this each garbage collection?
Isn't sufficient to do it when emacs is exiting?

>  }
>
>  DEFUN ("garbage-collect", Fgarbage_collect, Sgarbage_collect, 0, 0, "",
> diff --git a/src/comp.c b/src/comp.c
> index dd45599cc4..77c3006c56 100644
> --- a/src/comp.c
> +++ b/src/comp.c
> @@ -413,6 +413,10 @@ load_gccjit_if_necessary (bool mandatory)
>  #define CALL1I(fun, arg)				\
>    CALLN (Ffuncall, intern_c_string (STR (fun)), arg)
>
> +/* Like call2 but stringify and intern.  */
> +#define CALL2I(fun, arg1, arg2)				\
> +  CALLN (Ffuncall, intern_c_string (STR (fun)), arg1, arg2)
> +
>  #define DECL_BLOCK(name, func)				\
>    gcc_jit_block *(name) =				\
>      gcc_jit_function_new_block ((func), STR (name))
> @@ -3828,6 +3832,14 @@ DEFUN ("comp--release-ctxt", Fcomp__release_ctxt, Scomp__release_ctxt,
>    return Qt;
>  }
>
> +sigset_t oldset;

I think we have all static data at the top.

That said this is unclear to me because in comp--compile-ctxt-to-file
oldset is automatic and shadows this static, so I think we'll save in
the the automatic and later we just restore the (always zeroed) static
one.

> +static void restore_sigmask(void)
                             ^^^
                             space
> +{
> +  pthread_sigmask (SIG_SETMASK, &oldset, 0);
> +  unblock_input ();
> +}
> +
>  DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
>         Scomp__compile_ctxt_to_file,
>         1, 1, 0,
> @@ -3849,6 +3861,8 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
>      CALL1I (comp-data-container-idx, CALL1I (comp-ctxt-d-ephemeral, Vcomp_ctxt));
>
>    sigset_t oldset;
> +  ptrdiff_t count;
> +
>    if (!noninteractive)
>      {
>        sigset_t blocked;
> @@ -3861,6 +3875,8 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
>        sigaddset (&blocked, SIGIO);
>  #endif
>        pthread_sigmask (SIG_BLOCK, &blocked, &oldset);
> +      count = SPECPDL_INDEX ();
> +      record_unwind_protect_void(restore_sigmask);
                                  ^^^
                                  space

>      }
>    emit_ctxt_code ();
>
> @@ -3899,18 +3915,10 @@ DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
>  				   GCC_JIT_OUTPUT_KIND_DYNAMIC_LIBRARY,
>  				   SSDATA (tmp_file));
>
> -  /* Remove the old eln instead of copying the new one into it to get
> -     a new inode and prevent crashes in case the old one is currently
> -     loaded.  */
> -  if (!NILP (Ffile_exists_p (out_file)))
> -    Fdelete_file (out_file, Qnil);
> -  Frename_file (tmp_file, out_file, Qnil);
> +  CALL2I(comp--replace-output-file, out_file, tmp_file);
          ^^^
          space
>
>    if (!noninteractive)
> -    {
> -      pthread_sigmask (SIG_SETMASK, &oldset, 0);
> -      unblock_input ();
> -    }
> +    unbind_to(count, Qnil);
>
>    return out_file;
>  }
> @@ -3972,6 +3980,138 @@ helper_PSEUDOVECTOR_TYPEP_XUNTAG (Lisp_Object a, enum pvec_type code)
>  }
>
>  \f
> +/*********************************/
> +/* Disposal of compilation units */
> +/*********************************/
> +
> +#ifdef WINDOWSNT
> +#define OLD_ELN_SUFFIX_REGEXP build_string("\\.eln\\.old$")

I think instead of $  \\' is more correct.

> +static Lisp_Object all_loaded_comp_units;

All hash table in this files are postfixed as _h

> +struct delayed_comp_unit_disposal
> +{
> +  struct delayed_comp_unit_disposal * next;
                                       ^^^
                                     no space here
> +  char * filename;
          ^^
          likewise
> +};

Why an ad-hoc C structure and not a simple cons?  I think it would be
simpler and safer to use just a lisp list here.  Is it because we need
to add during GC?  If yes, comment :)

> +struct delayed_comp_unit_disposal * delayed_comp_unit_disposal_list;
                                     ^^
                                     likewise and the followings
> +
> +static Lisp_Object
> +returnQnil (Lisp_Object arg)

No camel case in function names.

> +{
> +  return Qnil;
> +}

I think each of the following functions really needs a comment line to
explain the scope of each of them + one preamble comment to explain all
the rename mechanism how is expected to work and the two datastructures
involved.

> +static void
> +clean_comp_unit_directory (Lisp_Object filepath)
> +{
> +  if (NILP (filepath))
> +    return;
> +  Lisp_Object files_in_dir;
> +  files_in_dir = internal_condition_case_4(Fdirectory_files, filepath, Qt,
> +                                           OLD_ELN_SUFFIX_REGEXP, Qnil, Qt, returnQnil);

80 columns

> +  FOR_EACH_TAIL(files_in_dir)
> +    {
> +      DeleteFile (SSDATA (XCAR (files_in_dir)));
> +    }
> +}
> +
> +void clean_package_user_dir_of_old_comp_units (void)
      ^^^
      new lines
> +{
> +  Lisp_Object package_user_dir = find_symbol_value (intern ("package-user-dir"));
> +  if (EQ(package_user_dir, Qunbound) || !STRINGP(package_user_dir))
> +    return;
> +
> +  clean_comp_unit_directory(package_user_dir);
> +}
> +
> +#endif
> +
> +void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_handle, bool delay)
      ^^^
      likewise
> +{
> +  eassert (comp_handle->handle);
> +  dynlib_close (comp_handle->handle);
> +#ifdef WINDOWSNT
> +  if (!delay)
> +    {
> +      Lisp_Object dirname = internal_condition_case_1(Ffile_name_directory,
> +                                                      build_string (comp_handle->cfile),
> +                                                      Qt,
> +                                                      returnQnil);
> +      if (!NILP(dirname))
> +        clean_comp_unit_directory (dirname);

I think we need to comment here why when we dispose the compilation unit
we try to clean the full directory.

> +      xfree (comp_handle->cfile);
> +      comp_handle->cfile = NULL;
> +    }
> +  else
> +    {
> +      struct delayed_comp_unit_disposal * head;
> +      head = xmalloc (sizeof (struct delayed_comp_unit_disposal));
> +      head->next = delayed_comp_unit_disposal_list;
> +      head->filename = comp_handle->cfile;
> +      comp_handle->cfile = NULL;
> +      delayed_comp_unit_disposal_list = head;
> +    }
> +#else
> +  xfree (comp_handle->file);
> +#endif
> +}

Also, wasn't the plan to try to delete the file and in case of failure
to put it in a list?  Here when delay is true this goes directly in the
list.  Could you explain why and add comment?

> +static void
> +register_native_comp_unit (Lisp_Object comp_u)
> +{
> +#ifdef WINDOWSNT
> +  static EMACS_UINT count;
> +
> +  if (XFIXNUM(Fhash_table_count(all_loaded_comp_units)) >= MOST_POSITIVE_FIXNUM)
> +    return;
> +
> +  while (!NILP(Fgethash(make_fixnum(count), all_loaded_comp_units, Qnil)))
> +    count = (count + 1) % MOST_POSITIVE_FIXNUM;

Given you are doing all of this just to get a key (we'll not use) I
think would be wise to just create the key using gensym.

> +  Fputhash(make_fixnum(count), comp_u, all_loaded_comp_units);
> +#endif
> +}
>
> +void dispose_all_remaining_comp_units (void)
> +{
> +#ifdef WINDOWSNT
> +  struct Lisp_Hash_Table *h = XHASH_TABLE (all_loaded_comp_units);
> +
> +  for (ptrdiff_t i = 0; i < HASH_TABLE_SIZE (h); ++i)
> +    {
> +      Lisp_Object k = HASH_KEY (h, i);
> +      if (!EQ (k, Qunbound))
> +        {
> +          Lisp_Object val = HASH_VALUE (h, i);
> +          struct Lisp_Native_Comp_Unit * cu = XNATIVE_COMP_UNIT(val);
> +          dispose_comp_unit(cu, false);
> +        }
> +    }
> +#endif
> +}
> +



> +void finish_delayed_disposal_of_comp_units (void)
> +{
> +#ifdef WINDOWSNT
> +  for (struct delayed_comp_unit_disposal * item = delayed_comp_unit_disposal_list;
> +       delayed_comp_unit_disposal_list;
> +       item = delayed_comp_unit_disposal_list)
> +    {
> +      delayed_comp_unit_disposal_list = item->next;
> +      Lisp_Object dirname
> +        = internal_condition_case_1 (Ffile_name_directory,
> +                                     build_string (item->filename), Qt,
> +                                     returnQnil);
> +      clean_comp_unit_directory (dirname);
> +      xfree(item->filename);
> +      xfree(item);
> +    }
> +#endif
> +}
> +
> +\f
>  /***********************************/
>  /* Deferred compilation mechanism. */
>  /***********************************/
> @@ -4192,6 +4332,12 @@ load_comp_unit (struct Lisp_Native_Comp_Unit *comp_u, bool loading_dump,
>        d_vec_len = XFIXNUM (Flength (comp_u->data_impure_vec));
>        for (EMACS_INT i = 0; i < d_vec_len; i++)
>  	data_imp_relocs[i] = AREF (comp_u->data_impure_vec, i);
> +
> +      /* If we register them while dumping we will get some entries in
> +         the hash table that will be duplicated when pdumper calls
> +         load_comp_unit. */
> +      if (!will_dump_p())
> +        register_native_comp_unit (comp_u_lisp_obj);
>      }
>
>    if (!loading_dump)
> @@ -4349,6 +4495,9 @@ DEFUN ("native-elisp-load", Fnative_elisp_load, Snative_elisp_load, 1, 2, 0,
>    if (!comp_u->handle)
>      xsignal2 (Qnative_lisp_load_failed, file, build_string (dynlib_error ()));
>    comp_u->file = file;
> +#ifdef WINDOWSNT
> +  comp_u->cfile = xlispstrdup(file);
> +#endif
>    comp_u->data_vec = Qnil;
>    comp_u->lambda_gc_guard = CALLN (Fmake_hash_table, QCtest, Qeq);
>    comp_u->lambda_c_name_idx_h = CALLN (Fmake_hash_table, QCtest, Qequal);
> @@ -4497,6 +4646,11 @@ syms_of_comp (void)
>    staticpro (&delayed_sources);
>    delayed_sources = Qnil;
>
> +#ifdef WINDOWSNT
> +  staticpro (&all_loaded_comp_units);
> +  all_loaded_comp_units = CALLN(Fmake_hash_table, QCweakness, Qvalue);
> +#endif
> +
>    DEFVAR_LISP ("comp-ctxt", Vcomp_ctxt,
>  	       doc: /* The compiler context.  */);
>    Vcomp_ctxt = Qnil;
> diff --git a/src/comp.h b/src/comp.h
> index 36e7cdf441..0b790fc7cb 100644
> --- a/src/comp.h
> +++ b/src/comp.h
> @@ -52,7 +52,15 @@ struct Lisp_Native_Comp_Unit
>    /* STUFFS WE DO NOT DUMP!!  */
>    Lisp_Object *data_imp_relocs;
>    bool loaded_once;
> +
>    dynlib_handle_ptr handle;
> +#ifdef WINDOWSNT
> +  /* We need to store a copy of the original file name in memory that
> +     is not subject to GC because the function to dispose native
> +     compilation units is called by the GC. By that time the `file'
> +     string may have been sweeped. */
> +  char * cfile;
> +#endif
>  };
>
>  #ifdef HAVE_NATIVE_COMP
> @@ -83,6 +91,14 @@ extern void syms_of_comp (void);
>
>  extern void maybe_defer_native_compilation (Lisp_Object function_name,
>  					    Lisp_Object definition);
> +
> +extern void dispose_comp_unit (struct Lisp_Native_Comp_Unit *
>  comp_unit, bool delay);
> +
> +extern void finish_delayed_disposal_of_comp_units (void);
> +
> +extern void dispose_all_remaining_comp_units (void);
> +
> +extern void clean_package_user_dir_of_old_comp_units (void);
>  #else
>
>  static inline void
> @@ -92,6 +108,17 @@ maybe_defer_native_compilation (Lisp_Object function_name,
>
>  extern void syms_of_comp (void);
>
> +static inline void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_handle)

Newline after ret type for this and the following definitions.

> +{
> +  emacs_abort();
> +}

emacs_abort is still not declared here so it does not compile.  Maybe we
can just put an eassert (false).

> +static inline void dispose_all_remaining_comp_units (void)
> +{}
> +
> +static inline void clean_package_user_dir_of_old_comp_units (void)
> +{}
> +


Thanks

  Andrea

--
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 26604 bytes --]

Hi Andrea,

Thanks for the feedback.

> Please review all the GNU code-style of this diff.  I've annotated some
> to be fixed but there are quite a number more of fixes of the same kind
> to be done.

This is such a pain, I sometimes forget to change Emacs to GNU style and
sometimes I'll write GNU style at work.

> > +(defun comp--replace-output-file (outfile tmpfile)
> > +  "Replace OUTFILE with TMPFILE taking the necessary steps when
> > +dealing with shared libraries that may be loaded into Emacs"
> > +  (cond ((eq 'windows-nt system-type)
> > +         (ignore-errors (delete-file outfile))
> > +         (let ((retry t))
> > +           (while retry
> > +             (setf retry nil)
> > +             (condition-case _
> > +                 (progn
> > +                   ;; outfile maybe recreated by another Emacs in
> > +                   ;; between the following two rename-file calls
> > +                   (if (file-exists-p outfile)
> > +                       (rename-file outfile (make-temp-file-internal
> > +                                             (file-name-sans-extension
outfile)
> > +                                             nil ".eln.old" nil)

> Isn't better to just add .old? So we will have cases of foo.eln.old.old
> instead of foo.eln.old.eln.old ?

We can't do that because foo.eln.old might exist already. This code tries to
rename an existing foo.eln to fooXXXXXX.eln.old where XXXXXX are random. We
should never try to rename fooXXXXXX.eln.old if it already exists.

> > +(defun package--delete-directory (dir)
> > +  "Delete DIR recursively.
> > +In Windows move .eln and .eln.old files that can not be deleted to
`package-user-dir'."

> 80 column lines limit.  I think also this should be transparent when
> native-comp-available-p say native comp is not available (for now
> compiler and load machinery are bundled).

We might be trying to delete an .eln file that another Emacs instance has
loaded. I know it sounds kind of strange, but if that is not the case then
`package--delete-directory` will just call `delete-directory`.

> I think would be good to destructure err using something like
> cl-destructuring-bind or pcase or even just using a let + some naming to
> make this more readable.

Good idea.

> > @@ -6117,6 +6116,8 @@ garbage_collect (void)
> >        if (tot_after < tot_before)
> >       malloc_probe (min (tot_before - tot_after, SIZE_MAX));
> >      }
> > +
> > +  finish_delayed_disposal_of_comp_units ();

> Could you describe why we need to call this each garbage collection?
> Isn't sufficient to do it when emacs is exiting?

I thought it was cleaner to delete *.eln.old ASAP. It would make it
necessary to
store the original files in a list, anyway.

> That said this is unclear to me because in comp--compile-ctxt-to-file
> oldset is automatic and shadows this static, so I think we'll save in
> the the automatic and later we just restore the (always zeroed) static
> one.

You are right. I'll fix it.

> +struct delayed_comp_unit_disposal
> +{
> +  struct delayed_comp_unit_disposal * next;
                                       ^^^
                                     no space here
> +  char * filename;
          ^^
          likewise
> +};

> Why an ad-hoc C structure and not a simple cons?  I think it would be
> simpler and safer to use just a lisp list here.  Is it because we need
> to add during GC?  If yes, comment :)

Exactly. Since filename needs to be a C string, I figured that using a C
struct
would be simpler that trying to wrap the C string in a Lisp object.

> I think each of the following functions really needs a comment line to
> explain the scope of each of them + one preamble comment to explain all
> the rename mechanism how is expected to work and the two datastructures
> involved.

Sure.

> +{
> +  eassert (comp_handle->handle);
> +  dynlib_close (comp_handle->handle);
> +#ifdef WINDOWSNT
> +  if (!delay)
> +    {
> +      Lisp_Object dirname =
internal_condition_case_1(Ffile_name_directory,
> +                                                      build_string
(comp_handle->cfile),
> +                                                      Qt,
> +                                                      returnQnil);
> +      if (!NILP(dirname))
> +        clean_comp_unit_directory (dirname);

> I think we need to comment here why when we dispose the compilation unit
> we try to clean the full directory.

This is because ideally we'd try to delete fooXXXXXX.eln.old if we are
disposing
the "foo" compilation unit and the file has been renamed. It turns out that
there is no simple way to figure out the current name of a loaded module in
Windows. GetModuleFileName returns the filename it had at the time we
loaded it.
Deleting all the *.eln.old in the directory is the (simple) workaround.

> +      xfree (comp_handle->cfile);
> +      comp_handle->cfile = NULL;
> +    }
> +  else
> +    {
> +      struct delayed_comp_unit_disposal * head;
> +      head = xmalloc (sizeof (struct delayed_comp_unit_disposal));
> +      head->next = delayed_comp_unit_disposal_list;
> +      head->filename = comp_handle->cfile;
> +      comp_handle->cfile = NULL;
> +      delayed_comp_unit_disposal_list = head;
> +    }
> +#else
> +  xfree (comp_handle->file);
> +#endif
> +}

> Also, wasn't the plan to try to delete the file and in case of failure
> to put it in a list?  Here when delay is true this goes directly in the
> list.  Could you explain why and add comment?

There are two reasons:

We don't have a simple way of getting the current name of the module (see
above).

Deleting all the files in the directory is implemented using
Fdirectory_files,
that allocates Lisp objects, and doing that in the middle of a GC sweep is
cause
of crashes (already tried).

> Given you are doing all of this just to get a key (we'll not use) I
> think would be wise to just create the key using gensym.

Great!

Nicolas.

El sáb., 23 may. 2020 a las 19:58, Andrea Corallo (<akrl@sdf.org>) escribió:

> Hi Nicolas,
>
> following some comments on - Improve handling of native compilation
> etc.
>
> Please review all the GNU code-style of this diff.  I've annotated some
> to be fixed but there are quite a number more of fixes of the same kind
> to be done.
>
> > When closing emacs will inspect all directories from which it loaded
> > native compilation units. If it finds a ".eln.old" file it will try to
> > delete it, if it fails that means that another Emacs instance is using
> it.
> >
> > When compiling a file we rename the file that was in the output path
> > in case it has been loaded into another Emacs instance.
> >
> > When deleting a package we move any ".eln" or ".eln.old" files in the
> > package folder that we can't delete to `package-user-dir`. Emacs will
> > check that directory when closing and delete them.
> >
> > * lisp/emacs-lisp/comp.el (comp--replace-output-file): Function called
> > from C code to finish the compilation process. It performs renaming of
> > the old file if necessary.
> > * lisp/emacs-lisp/package.el (package--delete-directory): Function to
> > delete a package directory. It moves native compilation units that it
> > can't delete to `package-user-dir'.
> > * src/alloc.c (cleanup_vector): Call dispose_comp_unit().
> >   (garbage_collect): Call finish_delayed_disposal_of_comp_units().
> > * src/comp.c: Restore the signal mask using unwind-protect. Store
> > loaded native compilation units in a hash table for disposal on
> > close. Store filenames of native compilation units GC'd in a linked
> > list to finish their disposal when the GC is over.
>
> Please annotate in the changelog the new functions ex:
> finish_delayed_disposal_of_comp_units, dispose_all_remaining_comp_units,
> register_native_comp_unit are missing.
>
> > * src/comp.h: Introduce cfile member in Lisp_Native_Comp_Unit.
> > Add declarations of functions that: clean directories of unused native
> > compilation units, handle disposal of native compilation units.
> > * src/emacs.c (kill-emacs): Dispose all remaining compilation units
> > right right before calling exit().
> > * src/eval.c (internal_condition_case_3, internal_condition_case_4):
> > Add functions.
> > * src/lisp.h (internal_condition_case_3, internal_condition_case_4):
> > Add functions.
> > * src/pdumper.c (dump_do_dump_relocation): Set cfile to a copy of the
> > Lisp string specifying the file path.
>
> > diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
> > index 012baf2560..1fb4cd98c0 100644
> > --- a/lisp/emacs-lisp/comp.el
> > +++ b/lisp/emacs-lisp/comp.el
> > @@ -2183,6 +2183,31 @@ comp-hint-cons
> >
> >  ;; Some entry point support code.
> >
> > +(defun comp--replace-output-file (outfile tmpfile)
> > +  "Replace OUTFILE with TMPFILE taking the necessary steps when
> > +dealing with shared libraries that may be loaded into Emacs"
> > +  (cond ((eq 'windows-nt system-type)
> > +         (ignore-errors (delete-file outfile))
> > +         (let ((retry t))
> > +           (while retry
> > +             (setf retry nil)
> > +             (condition-case _
> > +                 (progn
> > +                   ;; outfile maybe recreated by another Emacs in
> > +                   ;; between the following two rename-file calls
> > +                   (if (file-exists-p outfile)
> > +                       (rename-file outfile (make-temp-file-internal
> > +                                             (file-name-sans-extension
> outfile)
> > +                                             nil ".eln.old" nil)
>
> Isn't better to just add .old? So we will have cases of foo.eln.old.old
> instead of foo.eln.old.eln.old ?
>
> > +                                    t))
> > +                   (rename-file tmpfile outfile nil))
> > +               (file-already-exists (setf retry t))))))
> > +        ;; Remove the old eln instead of copying the new one into it
> > +        ;; to get a new inode and prevent crashes in case the old one
> > +        ;; is currently loaded.
> > +        (t (delete-file outfile)
> > +           (rename-file tmpfile outfile))))
> > +
> >  (defvar comp-files-queue ()
> >    "List of Elisp files to be compiled.")
>
>
>
> > diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> > index 95659840ad..c1c54b3c9a 100644
> > --- a/lisp/emacs-lisp/package.el
> > +++ b/lisp/emacs-lisp/package.el
> > @@ -2184,6 +2184,31 @@ If some packages are not installed propose to
> install them."
> >    (equal (cadr (assq (package-desc-name pkg) package-alist))
> >           pkg))
> >
> > +(defun package--delete-directory (dir)
> > +  "Delete DIR recursively.
> > +In Windows move .eln and .eln.old files that can not be deleted to
> `package-user-dir'."
>
> 80 column lines limit.  I think also this should be transparent when
> native-comp-available-p say native comp is not available (for now
> compiler and load machinery are bundled).
>
> > +  (cond ((eq 'windows-nt system-type)
> > +         (let ((retry t))
> > +           (while retry
> > +             (setf retry nil)
> > +             (condition-case err
> > +                 (delete-directory dir t)
> > +               (file-error
> > +                (if (and (string= "Removing old name" (cadr err))
> > +                         (string= "Permission denied" (caddr err))
> > +                         (or (string-suffix-p ".eln" (cadddr err))
> > +                             (string-suffix-p ".eln.old" (cadddr err))))
>
> I think would be good to destructure err using something like
> cl-destructuring-bind or pcase or even just using a let + some naming to
> make this more readable.
>
> > +                    (progn
> > +                      (rename-file (cadddr err)
> > +                                   (make-temp-file-internal
> > +                                    (concat package-user-dir
> > +                                            (file-name-base (cadddr
> err)))
> > +                                    nil ".eln.old" nil)
> > +                                   t)
> > +                      (setf retry t))
> > +                  (signal (car err) (cdr err))))))))
> > +        (t (delete-directory dir t))))
> > +
> >  (defun package-delete (pkg-desc &optional force nosave)
> >    "Delete package PKG-DESC.
> >
> > @@ -2236,7 +2261,7 @@ If NOSAVE is non-nil, the package is not removed
> from
> >                    (package-desc-name pkg-used-elsewhere-by)))
> >            (t
> >             (add-hook 'post-command-hook #'package-menu--post-refresh)
> > -           (delete-directory dir t)
> > +           (package--delete-directory dir)
> >             ;; Remove NAME-VERSION.signed and NAME-readme.txt files.
> >             ;;
> >             ;; NAME-readme.txt files are no longer created, but they
> > diff --git a/src/alloc.c b/src/alloc.c
> > index d6ba4d9790..420168ec4d 100644
> > --- a/src/alloc.c
> > +++ b/src/alloc.c
> > @@ -3119,8 +3119,7 @@ cleanup_vector (struct Lisp_Vector *vector)
> >      {
> >        struct Lisp_Native_Comp_Unit *cu =
> >       PSEUDOVEC_STRUCT (vector, Lisp_Native_Comp_Unit);
> > -      eassert (cu->handle);
> > -      dynlib_close (cu->handle);
> > +      dispose_comp_unit (cu, true);
> >      }
> >  }
> >
> > @@ -6117,6 +6116,8 @@ garbage_collect (void)
> >        if (tot_after < tot_before)
> >       malloc_probe (min (tot_before - tot_after, SIZE_MAX));
> >      }
> > +
> > +  finish_delayed_disposal_of_comp_units ();
>
> Could you describe why we need to call this each garbage collection?
> Isn't sufficient to do it when emacs is exiting?
>
> >  }
> >
> >  DEFUN ("garbage-collect", Fgarbage_collect, Sgarbage_collect, 0, 0, "",
> > diff --git a/src/comp.c b/src/comp.c
> > index dd45599cc4..77c3006c56 100644
> > --- a/src/comp.c
> > +++ b/src/comp.c
> > @@ -413,6 +413,10 @@ load_gccjit_if_necessary (bool mandatory)
> >  #define CALL1I(fun, arg)                             \
> >    CALLN (Ffuncall, intern_c_string (STR (fun)), arg)
> >
> > +/* Like call2 but stringify and intern.  */
> > +#define CALL2I(fun, arg1, arg2)                              \
> > +  CALLN (Ffuncall, intern_c_string (STR (fun)), arg1, arg2)
> > +
> >  #define DECL_BLOCK(name, func)                               \
> >    gcc_jit_block *(name) =                            \
> >      gcc_jit_function_new_block ((func), STR (name))
> > @@ -3828,6 +3832,14 @@ DEFUN ("comp--release-ctxt", Fcomp__release_ctxt,
> Scomp__release_ctxt,
> >    return Qt;
> >  }
> >
> > +sigset_t oldset;
>
> I think we have all static data at the top.
>
> That said this is unclear to me because in comp--compile-ctxt-to-file
> oldset is automatic and shadows this static, so I think we'll save in
> the the automatic and later we just restore the (always zeroed) static
> one.
>
> > +static void restore_sigmask(void)
>                              ^^^
>                              space
> > +{
> > +  pthread_sigmask (SIG_SETMASK, &oldset, 0);
> > +  unblock_input ();
> > +}
> > +
> >  DEFUN ("comp--compile-ctxt-to-file", Fcomp__compile_ctxt_to_file,
> >         Scomp__compile_ctxt_to_file,
> >         1, 1, 0,
> > @@ -3849,6 +3861,8 @@ DEFUN ("comp--compile-ctxt-to-file",
> Fcomp__compile_ctxt_to_file,
> >      CALL1I (comp-data-container-idx, CALL1I (comp-ctxt-d-ephemeral,
> Vcomp_ctxt));
> >
> >    sigset_t oldset;
> > +  ptrdiff_t count;
> > +
> >    if (!noninteractive)
> >      {
> >        sigset_t blocked;
> > @@ -3861,6 +3875,8 @@ DEFUN ("comp--compile-ctxt-to-file",
> Fcomp__compile_ctxt_to_file,
> >        sigaddset (&blocked, SIGIO);
> >  #endif
> >        pthread_sigmask (SIG_BLOCK, &blocked, &oldset);
> > +      count = SPECPDL_INDEX ();
> > +      record_unwind_protect_void(restore_sigmask);
>                                   ^^^
>                                   space
>
> >      }
> >    emit_ctxt_code ();
> >
> > @@ -3899,18 +3915,10 @@ DEFUN ("comp--compile-ctxt-to-file",
> Fcomp__compile_ctxt_to_file,
> >                                  GCC_JIT_OUTPUT_KIND_DYNAMIC_LIBRARY,
> >                                  SSDATA (tmp_file));
> >
> > -  /* Remove the old eln instead of copying the new one into it to get
> > -     a new inode and prevent crashes in case the old one is currently
> > -     loaded.  */
> > -  if (!NILP (Ffile_exists_p (out_file)))
> > -    Fdelete_file (out_file, Qnil);
> > -  Frename_file (tmp_file, out_file, Qnil);
> > +  CALL2I(comp--replace-output-file, out_file, tmp_file);
>           ^^^
>           space
> >
> >    if (!noninteractive)
> > -    {
> > -      pthread_sigmask (SIG_SETMASK, &oldset, 0);
> > -      unblock_input ();
> > -    }
> > +    unbind_to(count, Qnil);
> >
> >    return out_file;
> >  }
> > @@ -3972,6 +3980,138 @@ helper_PSEUDOVECTOR_TYPEP_XUNTAG (Lisp_Object a,
> enum pvec_type code)
> >  }
> >
> >
> > +/*********************************/
> > +/* Disposal of compilation units */
> > +/*********************************/
> > +
> > +#ifdef WINDOWSNT
> > +#define OLD_ELN_SUFFIX_REGEXP build_string("\\.eln\\.old$")
>
> I think instead of $  \\' is more correct.
>
> > +static Lisp_Object all_loaded_comp_units;
>
> All hash table in this files are postfixed as _h
>
> > +struct delayed_comp_unit_disposal
> > +{
> > +  struct delayed_comp_unit_disposal * next;
>                                        ^^^
>                                      no space here
> > +  char * filename;
>           ^^
>           likewise
> > +};
>
> Why an ad-hoc C structure and not a simple cons?  I think it would be
> simpler and safer to use just a lisp list here.  Is it because we need
> to add during GC?  If yes, comment :)
>
> > +struct delayed_comp_unit_disposal * delayed_comp_unit_disposal_list;
>                                      ^^
>                                      likewise and the followings
> > +
> > +static Lisp_Object
> > +returnQnil (Lisp_Object arg)
>
> No camel case in function names.
>
> > +{
> > +  return Qnil;
> > +}
>
> I think each of the following functions really needs a comment line to
> explain the scope of each of them + one preamble comment to explain all
> the rename mechanism how is expected to work and the two datastructures
> involved.
>
> > +static void
> > +clean_comp_unit_directory (Lisp_Object filepath)
> > +{
> > +  if (NILP (filepath))
> > +    return;
> > +  Lisp_Object files_in_dir;
> > +  files_in_dir = internal_condition_case_4(Fdirectory_files, filepath,
> Qt,
> > +                                           OLD_ELN_SUFFIX_REGEXP, Qnil,
> Qt, returnQnil);
>
> 80 columns
>
> > +  FOR_EACH_TAIL(files_in_dir)
> > +    {
> > +      DeleteFile (SSDATA (XCAR (files_in_dir)));
> > +    }
> > +}
> > +
> > +void clean_package_user_dir_of_old_comp_units (void)
>       ^^^
>       new lines
> > +{
> > +  Lisp_Object package_user_dir = find_symbol_value (intern
> ("package-user-dir"));
> > +  if (EQ(package_user_dir, Qunbound) || !STRINGP(package_user_dir))
> > +    return;
> > +
> > +  clean_comp_unit_directory(package_user_dir);
> > +}
> > +
> > +#endif
> > +
> > +void dispose_comp_unit (struct Lisp_Native_Comp_Unit * comp_handle,
> bool delay)
>       ^^^
>       likewise
> > +{
> > +  eassert (comp_handle->handle);
> > +  dynlib_close (comp_handle->handle);
> > +#ifdef WINDOWSNT
> > +  if (!delay)
> > +    {
> > +      Lisp_Object dirname =
> internal_condition_case_1(Ffile_name_directory,
> > +                                                      build_string
> (comp_handle->cfile),
> > +                                                      Qt,
> > +                                                      returnQnil);
> > +      if (!NILP(dirname))
> > +        clean_comp_unit_directory (dirname);
>
> I think we need to comment here why when we dispose the compilation unit
> we try to clean the full directory.
>
> > +      xfree (comp_handle->cfile);
> > +      comp_handle->cfile = NULL;
> > +    }
> > +  else
> > +    {
> > +      struct delayed_comp_unit_disposal * head;
> > +      head = xmalloc (sizeof (struct delayed_comp_unit_disposal));
> > +      head->next = delayed_comp_unit_disposal_list;
> > +      head->filename = comp_handle->cfile;
> > +      comp_handle->cfile = NULL;
> > +      delayed_comp_unit_disposal_list = head;
> > +    }
> > +#else
> > +  xfree (comp_handle->file);
> > +#endif
> > +}
>
> Also, wasn't the plan to try to delete the file and in case of failure
> to put it in a list?  Here when delay is true this goes directly in the
> list.  Could you explain why and add comment?
>
> > +static void
> > +register_native_comp_unit (Lisp_Object comp_u)
> > +{
> > +#ifdef WINDOWSNT
> > +  static EMACS_UINT count;
> > +
> > +  if (XFIXNUM(Fhash_table_count(all_loaded_comp_units)) >=
> MOST_POSITIVE_FIXNUM)
> > +    return;
> > +
> > +  while (!NILP(Fgethash(make_fixnum(count), all_loaded_comp_units,
> Qnil)))
> > +    count = (count + 1) % MOST_POSITIVE_FIXNUM;
>
> Given you are doing all of this just to get a key (we'll not use) I
> think would be wise to just create the key using gensym.
>
> > +  Fputhash(make_fixnum(count), comp_u, all_loaded_comp_units);
> > +#endif
> > +}
> >
> > +void dispose_all_remaining_comp_units (void)
> > +{
> > +#ifdef WINDOWSNT
> > +  struct Lisp_Hash_Table *h = XHASH_TABLE (all_loaded_comp_units);
> > +
> > +  for (ptrdiff_t i = 0; i < HASH_TABLE_SIZE (h); ++i)
> > +    {
> > +      Lisp_Object k = HASH_KEY (h, i);
> > +      if (!EQ (k, Qunbound))
> > +        {
> > +          Lisp_Object val = HASH_VALUE (h, i);
> > +          struct Lisp_Native_Comp_Unit * cu = XNATIVE_COMP_UNIT(val);
> > +          dispose_comp_unit(cu, false);
> > +        }
> > +    }
> > +#endif
> > +}
> > +
>
>
>
> > +void finish_delayed_disposal_of_comp_units (void)
> > +{
> > +#ifdef WINDOWSNT
> > +  for (struct delayed_comp_unit_disposal * item =
> delayed_comp_unit_disposal_list;
> > +       delayed_comp_unit_disposal_list;
> > +       item = delayed_comp_unit_disposal_list)
> > +    {
> > +      delayed_comp_unit_disposal_list = item->next;
> > +      Lisp_Object dirname
> > +        = internal_condition_case_1 (Ffile_name_directory,
> > +                                     build_string (item->filename), Qt,
> > +                                     returnQnil);
> > +      clean_comp_unit_directory (dirname);
> > +      xfree(item->filename);
> > +      xfree(item);
> > +    }
> > +#endif
> > +}
> > +
> > +
> >  /***********************************/
> >  /* Deferred compilation mechanism. */
> >  /***********************************/
> > @@ -4192,6 +4332,12 @@ load_comp_unit (struct Lisp_Native_Comp_Unit
> *comp_u, bool loading_dump,
> >        d_vec_len = XFIXNUM (Flength (comp_u->data_impure_vec));
> >        for (EMACS_INT i = 0; i < d_vec_len; i++)
> >       data_imp_relocs[i] = AREF (comp_u->data_impure_vec, i);
> > +
> > +      /* If we register them while dumping we will get some entries in
> > +         the hash table that will be duplicated when pdumper calls
> > +         load_comp_unit. */
> > +      if (!will_dump_p())
> > +        register_native_comp_unit (comp_u_lisp_obj);
> >      }
> >
> >    if (!loading_dump)
> > @@ -4349,6 +4495,9 @@ DEFUN ("native-elisp-load", Fnative_elisp_load,
> Snative_elisp_load, 1, 2, 0,
> >    if (!comp_u->handle)
> >      xsignal2 (Qnative_lisp_load_failed, file, build_string
> (dynlib_error ()));
> >    comp_u->file = file;
> > +#ifdef WINDOWSNT
> > +  comp_u->cfile = xlispstrdup(file);
> > +#endif
> >    comp_u->data_vec = Qnil;
> >    comp_u->lambda_gc_guard = CALLN (Fmake_hash_table, QCtest, Qeq);
> >    comp_u->lambda_c_name_idx_h = CALLN (Fmake_hash_table, QCtest,
> Qequal);
> > @@ -4497,6 +4646,11 @@ syms_of_comp (void)
> >    staticpro (&delayed_sources);
> >    delayed_sources = Qnil;
> >
> > +#ifdef WINDOWSNT
> > +  staticpro (&all_loaded_comp_units);
> > +  all_loaded_comp_units = CALLN(Fmake_hash_table, QCweakness, Qvalue);
> > +#endif
> > +
> >    DEFVAR_LISP ("comp-ctxt", Vcomp_ctxt,
> >              doc: /* The compiler context.  */);
> >    Vcomp_ctxt = Qnil;
> > diff --git a/src/comp.h b/src/comp.h
> > index 36e7cdf441..0b790fc7cb 100644
> > --- a/src/comp.h
> > +++ b/src/comp.h
> > @@ -52,7 +52,15 @@ struct Lisp_Native_Comp_Unit
> >    /* STUFFS WE DO NOT DUMP!!  */
> >    Lisp_Object *data_imp_relocs;
> >    bool loaded_once;
> > +
> >    dynlib_handle_ptr handle;
> > +#ifdef WINDOWSNT
> > +  /* We need to store a copy of the original file name in memory that
> > +     is not subject to GC because the function to dispose native
> > +     compilation units is called by the GC. By that time the `file'
> > +     string may have been sweeped. */
> > +  char * cfile;
> > +#endif
> >  };
> >
> >  #ifdef HAVE_NATIVE_COMP
> > @@ -83,6 +91,14 @@ extern void syms_of_comp (void);
> >
> >  extern void maybe_defer_native_compilation (Lisp_Object function_name,
> >                                           Lisp_Object definition);
> > +
> > +extern void dispose_comp_unit (struct Lisp_Native_Comp_Unit *
> >  comp_unit, bool delay);
> > +
> > +extern void finish_delayed_disposal_of_comp_units (void);
> > +
> > +extern void dispose_all_remaining_comp_units (void);
> > +
> > +extern void clean_package_user_dir_of_old_comp_units (void);
> >  #else
> >
> >  static inline void
> > @@ -92,6 +108,17 @@ maybe_defer_native_compilation (Lisp_Object
> function_name,
> >
> >  extern void syms_of_comp (void);
> >
> > +static inline void dispose_comp_unit (struct Lisp_Native_Comp_Unit *
> comp_handle)
>
> Newline after ret type for this and the following definitions.
>
> > +{
> > +  emacs_abort();
> > +}
>
> emacs_abort is still not declared here so it does not compile.  Maybe we
> can just put an eassert (false).
>
> > +static inline void dispose_all_remaining_comp_units (void)
> > +{}
> > +
> > +static inline void clean_package_user_dir_of_old_comp_units (void)
> > +{}
> > +
>
>
> Thanks
>
>   Andrea
>
> --
> akrl@sdf.org
>

[-- Attachment #2: Type: text/html, Size: 32081 bytes --]

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> Hi Andrea,
>
> Thanks for the feedback.
>
>> Please review all the GNU code-style of this diff.  I've annotated
> some
>> to be fixed but there are quite a number more of fixes of the same
> kind
>> to be done.
>
> This is such a pain, I sometimes forget to change Emacs to GNU style
> and
> sometimes I'll write GNU style at work.

Isn't this good? ;)

Joking aside, you can also set it on a directory base.  Another trick if
your are not mechanically used to C GNU style is to run
check_GNU_style.sh on your patch (comes in the contrib folder of GCC).

>> > +(defun comp--replace-output-file (outfile tmpfile)
>> > +  "Replace OUTFILE with TMPFILE taking the necessary steps when
>> > +dealing with shared libraries that may be loaded into Emacs"
>> > +  (cond ((eq 'windows-nt system-type)
>> > +         (ignore-errors (delete-file outfile))
>> > +         (let ((retry t))
>> > +           (while retry
>> > +             (setf retry nil)
>> > +             (condition-case _
>> > +                 (progn
>> > +                   ;; outfile maybe recreated by another Emacs
> in
>> > +                   ;; between the following two rename-file
> calls
>> > +                   (if (file-exists-p outfile)
>> > +                       (rename-file outfile
> (make-temp-file-internal
>> > +                                            
> (file-name-sans-extension outfile)
>> > +                                             nil ".eln.old" nil)
>
>> Isn't better to just add .old? So we will have cases of
> foo.eln.old.old
>> instead of foo.eln.old.eln.old ?
>
> We can't do that because foo.eln.old might exist already. This code
> tries to
> rename an existing foo.eln to fooXXXXXX.eln.old where XXXXXX are
> random. We
> should never try to rename fooXXXXXX.eln.old if it already exists.

Understand, I had in my mind was cleaner to add .old till the first non
used filename available was found but I guess is the same, probably even
simpler.

>> > +(defun package--delete-directory (dir)
>> > +  "Delete DIR recursively.
>> > +In Windows move .eln and .eln.old files that can not be deleted
> to `package-user-dir'."
>
>> 80 column lines limit.  I think also this should be transparent
> when
>> native-comp-available-p say native comp is not available (for now
>> compiler and load machinery are bundled).
>
> We might be trying to delete an .eln file that another Emacs instance
> has
> loaded. I know it sounds kind of strange, but if that is not the case
> then
> `package--delete-directory` will just call `delete-directory`.

It doesn't sound strange.  I see probably is better like this for
systems running Emacs native-comp and stock at the same time.

>> I think would be good to destructure err using something like
>> cl-destructuring-bind or pcase or even just using a let + some
> naming to
>> make this more readable.
>
> Good idea.
>
>> > @@ -6117,6 +6116,8 @@ garbage_collect (void)
>> >        if (tot_after < tot_before)
>> >       malloc_probe (min (tot_before - tot_after, SIZE_MAX));
>> >      }
>> > +
>> > +  finish_delayed_disposal_of_comp_units ();
>
>> Could you describe why we need to call this each garbage
> collection?
>> Isn't sufficient to do it when emacs is exiting?
>
> I thought it was cleaner to delete *.eln.old ASAP. It would make it
> necessary to
> store the original files in a list, anyway.

The problem is that GC is called (especially by default) *very*
frequently, bounding GC performance to filesystem accesses is really not
a good idea IMO because we have no control over this last.

You could not see a difference here because:

- spaceemacs GC settings runs it way less often coming with a bigger
  gc-cons-threshold by default

- GC euristincs being GC slow decides to give-up a little and accept
  running less often leading to more fragmentation

- filesystem is blazingly fast

- you haven't measured ;)

>> That said this is unclear to me because in
> comp--compile-ctxt-to-file
>> oldset is automatic and shadows this static, so I think we'll save
> in
>> the the automatic and later we just restore the (always zeroed)
> static
>> one.
>
> You are right. I'll fix it.
>
>> +struct delayed_comp_unit_disposal
>> +{
>> +  struct delayed_comp_unit_disposal * next;
>                                        ^^^
>                                      no space here
>> +  char * filename;
>           ^^
>           likewise
>> +};
>
>> Why an ad-hoc C structure and not a simple cons?  I think it would
> be
>> simpler and safer to use just a lisp list here.  Is it because we
> need
>> to add during GC?  If yes, comment :)
>
> Exactly. Since filename needs to be a C string, I figured that using
> a C struct
> would be simpler that trying to wrap the C string in a Lisp object.

So the reason is not to allocate lisp objs during GC correct?

>> I think each of the following functions really needs a comment line
> to
>> explain the scope of each of them + one preamble comment to explain
> all
>> the rename mechanism how is expected to work and the two
> datastructures
>> involved.
>
> Sure.
>
>> +{
>> +  eassert (comp_handle->handle);
>> +  dynlib_close (comp_handle->handle);
>> +#ifdef WINDOWSNT
>> +  if (!delay)
>> +    {
>> +      Lisp_Object dirname = internal_condition_case_1
> (Ffile_name_directory,
>> +                                                      build_string
> (comp_handle->cfile),
>> +                                                      Qt,
>> +                                                      returnQnil);
>> +      if (!NILP(dirname))
>> +        clean_comp_unit_directory (dirname);
>
>> I think we need to comment here why when we dispose the compilation
> unit
>> we try to clean the full directory.
>
> This is because ideally we'd try to delete fooXXXXXX.eln.old if we
> are disposing
> the "foo" compilation unit and the file has been renamed. It turns
> out that
> there is no simple way to figure out the current name of a loaded
> module in
> Windows. GetModuleFileName returns the filename it had at the time we
> loaded it.
> Deleting all the *.eln.old in the directory is the (simple)
> workaround.

Uh I see.  I think this is worth noting with a comment in the code too.

>> +      xfree (comp_handle->cfile);
>> +      comp_handle->cfile = NULL;
>> +    }
>> +  else
>> +    {
>> +      struct delayed_comp_unit_disposal * head;
>> +      head = xmalloc (sizeof (struct delayed_comp_unit_disposal));
>> +      head->next = delayed_comp_unit_disposal_list;
>> +      head->filename = comp_handle->cfile;
>> +      comp_handle->cfile = NULL;
>> +      delayed_comp_unit_disposal_list = head;
>> +    }
>> +#else
>> +  xfree (comp_handle->file);
>> +#endif
>> +}
>
>> Also, wasn't the plan to try to delete the file and in case of
> failure
>> to put it in a list?  Here when delay is true this goes directly in
> the
>> list.  Could you explain why and add comment?
>
> There are two reasons:
>
> We don't have a simple way of getting the current name of the module
> (see
> above).
>
> Deleting all the files in the directory is implemented using
> Fdirectory_files,
> that allocates Lisp objects, and doing that in the middle of a GC
> sweep is cause
> of crashes (already tried).
>
>> Given you are doing all of this just to get a key (we'll not use) I
>> think would be wise to just create the key using gensym.
>

As said please annotate the ratio of this decisions in form of comments
in the code, otherwise will be trick to follow for somebody who has not
followed this thread.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

> The problem is that GC is called (especially by default) *very*
> frequently, bounding GC performance to filesystem accesses is really not
> a good idea IMO because we have no control over this last.
>
> You could not see a difference here because:
>
> - spaceemacs GC settings runs it way less often coming with a bigger
>   gc-cons-threshold by default
>
> - GC euristincs being GC slow decides to give-up a little and accept
>   running less often leading to more fragmentation
>
> - filesystem is blazingly fast
>
> - you haven't measured ;)

Actually unloading a native compilation unit is such an unfrequent operation
that all that finish_delayed_disposal_of_comp_units() does is compare a
pointer
to NULL. It will not slowdown the GC at all.

Anyway, I could change this to run on an idle timer or just handle it when
Emacs
closes. Which do you prefer?

> So the reason is not to allocate lisp objs during GC correct?

Correct.

Nicolas

[-- Attachment #2: Type: text/html, Size: 1124 bytes --]

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> The problem is that GC is called (especially by default) *very*
>> frequently, bounding GC performance to filesystem accesses is
> really not
>> a good idea IMO because we have no control over this last.
>>
>> You could not see a difference here because:
>>
>> - spaceemacs GC settings runs it way less often coming with a
> bigger
>>   gc-cons-threshold by default
>>
>> - GC euristincs being GC slow decides to give-up a little and
> accept
>>   running less often leading to more fragmentation
>>
>> - filesystem is blazingly fast
>>
>> - you haven't measured ;)
>
> Actually unloading a native compilation unit is such an unfrequent
> operation
> that all that finish_delayed_disposal_of_comp_units() does is compare
> a pointer
> to NULL. It will not slowdown the GC at all.
>
> Anyway, I could change this to run on an idle timer or just handle it
> when Emacs
> closes. Which do you prefer?

What you say is correct, collecting a compilation unit is very
infrequent now.  But code could decide to native compile functions each
time a performance critical operation has to be done, real world code
does that already relying on the byte-compiler.

I think to start with doing the clean-up when Emacs is closing is
sufficient, we can always add the timer in case we feel the need.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1660 bytes --]

I pushed a new version of the patch to my git repo.

Thanks, Nico

El dom., 24 may. 2020 a las 16:14, Andrea Corallo (<akrl@sdf.org>) escribió:

> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>
> >> The problem is that GC is called (especially by default) *very*
> >> frequently, bounding GC performance to filesystem accesses is
> > really not
> >> a good idea IMO because we have no control over this last.
> >>
> >> You could not see a difference here because:
> >>
> >> - spaceemacs GC settings runs it way less often coming with a
> > bigger
> >>   gc-cons-threshold by default
> >>
> >> - GC euristincs being GC slow decides to give-up a little and
> > accept
> >>   running less often leading to more fragmentation
> >>
> >> - filesystem is blazingly fast
> >>
> >> - you haven't measured ;)
> >
> > Actually unloading a native compilation unit is such an unfrequent
> > operation
> > that all that finish_delayed_disposal_of_comp_units() does is compare
> > a pointer
> > to NULL. It will not slowdown the GC at all.
> >
> > Anyway, I could change this to run on an idle timer or just handle it
> > when Emacs
> > closes. Which do you prefer?
>
> What you say is correct, collecting a compilation unit is very
> infrequent now.  But code could decide to native compile functions each
> time a performance critical operation has to be done, real world code
> does that already relying on the byte-compiler.
>
> I think to start with doing the clean-up when Emacs is closing is
> sufficient, we can always add the timer in case we feel the need.
>
> Thanks
>
>   Andrea
>
> --
> akrl@sdf.org
>

[-- Attachment #2: Type: text/html, Size: 2314 bytes --]

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> I pushed a new version of the patch to my git repo.
>
> Thanks, Nico

Hi Nico,

I see the patch is adding internal_condition_case_3 and
internal_condition_case_4 but I don't see them used.  Is this a refuse
of some previous version?

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

Hi Andrea,

> I see the patch is adding internal_condition_case_3 and
> internal_condition_case_4 but I don't see them used.  Is this a refuse
> of some previous version?

internal_condition_case_4 is used in clean_comp_unit_directory().
internal_condition_case_3 is not used, but I added it for completeness.

I have pushed a new patch that reduces the number of files probed when
loading.
It should work for Posix too.

Nico


El lun., 25 may. 2020 a las 11:05, Andrea Corallo (<akrl@sdf.org>) escribió:

> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>
> > I pushed a new version of the patch to my git repo.
> >
> > Thanks, Nico
>
> Hi Nico,
>
> I see the patch is adding internal_condition_case_3 and
> internal_condition_case_4 but I don't see them used.  Is this a refuse
> of some previous version?
>
> Thanks
>
>   Andrea
>
> --
> akrl@sdf.org
>

[-- Attachment #2: Type: text/html, Size: 1500 bytes --]

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> Hi Andrea,
>
>> I see the patch is adding internal_condition_case_3 and
>> internal_condition_case_4 but I don't see them used.  Is this a
> refuse
>> of some previous version? 
>
> internal_condition_case_4 is used in clean_comp_unit_directory().
> internal_condition_case_3 is not used, but I added it for
> completeness.

Ops sorry, doing last checks in parallel to other stuffs I failed to
search correctly :/

Okay, have a look just pushed.

> I have pushed a new patch that reduces the number of files probed
> when loading.
> It should work for Posix too.

Super, I'll have a look this evening.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>
>> Hi Andrea,
>>
>>> I see the patch is adding internal_condition_case_3 and
>>> internal_condition_case_4 but I don't see them used.  Is this a
>> refuse
>>> of some previous version? 
>>
>> internal_condition_case_4 is used in clean_comp_unit_directory().
>> internal_condition_case_3 is not used, but I added it for
>> completeness.
>
> Ops sorry, doing last checks in parallel to other stuffs I failed to
> search correctly :/
>
> Okay, have a look just pushed.
>
>> I have pushed a new patch that reduces the number of files probed
>> when loading.
>> It should work for Posix too.
>
> Super, I'll have a look this evening.
>
> Thanks
>
>   Andrea

Hi Nico,

please put the two patches in GNU code style, as mentioned you can run
check_GNU_style.sh and check against the output.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Improve handling of native compilation...

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 156 bytes --]

> please put the two patches in GNU code style, as mentioned you can run
> check_GNU_style.sh and check against the output.
It should be correct now.

Nico

[-- Attachment #2: Type: text/html, Size: 244 bytes --]

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Andrea Corallo]

[-- Attachment #1: Type: text/plain, Size: 2935 bytes --]

Hi all,

I've attached the current Nico's patch in case someone wants to engage,
here some comments from me.

> * src/fileio.c: Introduce function emacs_root_dir(). Refactor
> `expand-file-name` to use it.
> * src/lisp.h: Separate emacs_root_dir() into dos_emacs_root_dir() and
> w32_emacs_root_dir().
> * src/msdos.c: Rename emacs_root_dir() to dos_emacs_root_dir().
> * src/w32.c: Rename emacs_root_dir() to w32_emacs_root_dir().

I think would be good to mention in the commit message/changelog the
reason of this change, why it wasn't working and why it is now.

> diff --git a/src/fileio.c b/src/fileio.c
> index 2f1d2f8243..e9be811841 100644
> --- a/src/fileio.c
> +++ b/src/fileio.c
> @@ -781,6 +781,18 @@ user_homedir (char const *name)
>    return pw->pw_dir;
>  }
>  
> +static Lisp_Object
> +emacs_root_dir (void)
> +{
> +#ifdef DOS
> +      return build_string (dos_emacs_root_dir ());
> +#elif defined (WINDOWSNT)
> +      return build_string (w32_emacs_root_dir ());
> +#else
> +      return build_string ("/");
> +#endif
> +}

I believe the indentation of these returns should be two regular spaces.

>  DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
>         doc: /* Convert filename NAME to absolute, and canonicalize it.
>  Second arg DEFAULT-DIRECTORY is directory to start with if NAME is relative
> @@ -851,21 +863,16 @@ the root directory.  */)
>      }
>  
>    /* As a last resort, we may have to use the root as
> -     default_directory below.  */
> -  Lisp_Object root;
> -#ifdef DOS_NT
> -      /* "/" is not considered a root directory on DOS_NT, so using it
> -	 as default_directory causes an infinite recursion in, e.g.,
> -	 the following:
> +     default_directory below.
>  
> -            (let (default-directory)
> -	      (expand-file-name "a"))
> +     "/" is not considered a root directory on DOS_NT, so using it
> +     as default_directory causes an infinite recursion in, e.g.,
> +     the following:
>  
> -	 To avoid this, we use the root of the current drive.  */
> -      root = build_string (emacs_root_dir ());
> -#else
> -      root = build_string ("/");
> -#endif
> +        (let (default-directory)
> +          (expand-file-name "a"))
> +
> +     To avoid this, we use the root of the current drive.  */

I suspect this commentary is not very ideal here given now the code has
been moved into emacs_root_dir, maybe the commentary should go there.

>    /* Use the buffer's default-directory if DEFAULT_DIRECTORY is omitted.  */
>    if (NILP (default_directory))
> @@ -891,13 +898,13 @@ the root directory.  */)
>  	      Lisp_Object absdir
>  		= STRINGP (Vinvocation_directory)
>  		&& file_name_absolute_no_tilde_p (Vinvocation_directory)
> -		? Vinvocation_directory : root;
> +		? Vinvocation_directory : emacs_root_dir ();
>  	      default_directory = Fexpand_file_name (dir, absdir);
>  	    }
>  	}
>      }

Thanks

  Andrea

-- 
akrl@sdf.org

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Determine-the-emacs-root-dir-only-when-necessary.patch --]
[-- Type: text/x-diff, Size: 4178 bytes --]

From 1ead6bad59d0c2739dce13d3720c0264f2c5c531 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 25 May 2020 17:55:23 -0300
Subject: [PATCH] Determine the emacs root dir only when necessary.

* src/fileio.c: Introduce function emacs_root_dir(). Refactor
`expand-file-name` to use it.
* src/lisp.h: Separate emacs_root_dir() into dos_emacs_root_dir() and
w32_emacs_root_dir().
* src/msdos.c: Rename emacs_root_dir() to dos_emacs_root_dir().
* src/w32.c: Rename emacs_root_dir() to w32_emacs_root_dir().
---
 src/fileio.c | 37 ++++++++++++++++++++++---------------
 src/lisp.h   | 11 +++++++----
 src/msdos.c  |  2 +-
 src/w32.c    |  2 +-
 4 files changed, 31 insertions(+), 21 deletions(-)

diff --git a/src/fileio.c b/src/fileio.c
index 2f1d2f8243..e9be811841 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -781,6 +781,18 @@ user_homedir (char const *name)
   return pw->pw_dir;
 }
 
+static Lisp_Object
+emacs_root_dir (void)
+{
+#ifdef DOS
+      return build_string (dos_emacs_root_dir ());
+#elif defined (WINDOWSNT)
+      return build_string (w32_emacs_root_dir ());
+#else
+      return build_string ("/");
+#endif
+}
+
 DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
        doc: /* Convert filename NAME to absolute, and canonicalize it.
 Second arg DEFAULT-DIRECTORY is directory to start with if NAME is relative
@@ -851,21 +863,16 @@ the root directory.  */)
     }
 
   /* As a last resort, we may have to use the root as
-     default_directory below.  */
-  Lisp_Object root;
-#ifdef DOS_NT
-      /* "/" is not considered a root directory on DOS_NT, so using it
-	 as default_directory causes an infinite recursion in, e.g.,
-	 the following:
+     default_directory below.
 
-            (let (default-directory)
-	      (expand-file-name "a"))
+     "/" is not considered a root directory on DOS_NT, so using it
+     as default_directory causes an infinite recursion in, e.g.,
+     the following:
 
-	 To avoid this, we use the root of the current drive.  */
-      root = build_string (emacs_root_dir ());
-#else
-      root = build_string ("/");
-#endif
+        (let (default-directory)
+          (expand-file-name "a"))
+
+     To avoid this, we use the root of the current drive.  */
 
   /* Use the buffer's default-directory if DEFAULT_DIRECTORY is omitted.  */
   if (NILP (default_directory))
@@ -891,13 +898,13 @@ the root directory.  */)
 	      Lisp_Object absdir
 		= STRINGP (Vinvocation_directory)
 		&& file_name_absolute_no_tilde_p (Vinvocation_directory)
-		? Vinvocation_directory : root;
+		? Vinvocation_directory : emacs_root_dir ();
 	      default_directory = Fexpand_file_name (dir, absdir);
 	    }
 	}
     }
   if (! STRINGP (default_directory))
-    default_directory = root;
+    default_directory = emacs_root_dir ();
 
   handler = Ffind_file_name_handler (default_directory, Qexpand_file_name);
   if (!NILP (handler))
diff --git a/src/lisp.h b/src/lisp.h
index 52242791aa..3dd8b9ad14 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4749,10 +4749,13 @@ extern bool profiler_memory_running;
 extern void malloc_probe (size_t);
 extern void syms_of_profiler (void);
 
-#ifdef DOS_NT
-/* Defined in msdos.c, w32.c.  */
-extern char *emacs_root_dir (void);
-#endif /* DOS_NT */
+#ifdef MSDOS
+/* Defined in msdos.c.  */
+extern char *dos_emacs_root_dir (void);
+#elif defined (WINDOWSNT)
+/* Defined in w32.c.  */
+extern char *w32_emacs_root_dir (void);
+#endif /* MSDOS */
 
 #ifdef HAVE_NATIVE_COMP
 INLINE bool
diff --git a/src/msdos.c b/src/msdos.c
index b5f06c99c3..0827cc96cd 100644
--- a/src/msdos.c
+++ b/src/msdos.c
@@ -3350,7 +3350,7 @@ getdefdir (int drive, char *dst)
 }
 
 char *
-emacs_root_dir (void)
+dos_emacs_root_dir (void)
 {
   static char root_dir[4];
 
diff --git a/src/w32.c b/src/w32.c
index 38bbc49656..a3f5844574 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -3147,7 +3147,7 @@ init_environment (char ** argv)
 /* Called from expand-file-name when default-directory is not a string.  */
 
 char *
-emacs_root_dir (void)
+w32_emacs_root_dir (void)
 {
   static char root_dir[MAX_UTF8_PATH];
   const char *p;
-- 
2.20.1

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Date: Wed, 27 May 2020 21:02:45 +0000
> Cc: 41242@debbugs.gnu.org
> 
> I've attached the current Nico's patch in case someone wants to engage,
> here some comments from me.

I agree with all your comments.

Thanks.

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Nicolas Bértolo]

[-- Attachment #1.1: Type: text/plain, Size: 83 bytes --]

Hi,

I have taken your comments into consideration and updated the patch.

Nicolas

[-- Attachment #1.2: Type: text/html, Size: 167 bytes --]

[-- Attachment #2: 0001-Determine-the-emacs-root-dir-only-when-necessary.patch --]
[-- Type: application/octet-stream, Size: 5412 bytes --]

From 6283738729b8986afe6f6d7e0117b0a04d3ffeb7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 25 May 2020 17:55:23 -0300
Subject: [PATCH] Determine the emacs root dir only when necessary.

When loading a dump file that contains native compiled elisp we try to
find the .eln file. This uses Ffile_exists_p(). This function, in
turn, calls Fexpand_file_name(). This function will use the root
directory as `default_directory' as a fallback.

Getting the root directory requires reading the $emacs_dir environment
variable.  This is setup later in the initialization process. This
caused a crash.

Fexpand_file_name() was trying to obtain the root directory even when
it was not necessary because `default-directory' was not nil.

It turns out that the dump loading process does not set
`default-directory' to nil, therefore Fexpand_file_name() does not
need to find out the root directory and we avoid reading an
environment variable that is not set yet.

With this patch we avoid calling filename_from_ansi() too early (It is
not the reason why Emacs crashed, but it is still important to call it
after it has been setup properly.

* src/fileio.c: Introduce function emacs_root_dir(). Refactor
`expand-file-name` to use it.
* src/lisp.h: Separate emacs_root_dir() into dos_emacs_root_dir() and
w32_emacs_root_dir().
* src/msdos.c: Rename emacs_root_dir() to dos_emacs_root_dir().
* src/w32.c: Rename emacs_root_dir() to w32_emacs_root_dir().
---
 src/fileio.c | 45 ++++++++++++++++++++++++++-------------------
 src/lisp.h   | 11 +++++++----
 src/msdos.c  |  2 +-
 src/w32.c    |  2 +-
 4 files changed, 35 insertions(+), 25 deletions(-)

diff --git a/src/fileio.c b/src/fileio.c
index 2f1d2f8243..11374ac56d 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -781,6 +781,30 @@ user_homedir (char const *name)
   return pw->pw_dir;
 }
 
+/* As a last resort, we may have to use the root as
+   default_directory in `expand-file-name'.
+
+   "/" is not considered a root directory on DOS_NT, so using it
+   as default_directory causes an infinite recursion in, e.g.,
+   the following:
+
+   (let (default-directory)
+          (expand-file-name "a"))
+
+   To avoid this, we use the root of the current drive. */
+
+static Lisp_Object
+emacs_root_dir (void)
+{
+#ifdef DOS
+  return build_string (dos_emacs_root_dir ());
+#elif defined (WINDOWSNT)
+  return build_string (w32_emacs_root_dir ());
+#else
+  return build_string ("/");
+#endif
+}
+
 DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
        doc: /* Convert filename NAME to absolute, and canonicalize it.
 Second arg DEFAULT-DIRECTORY is directory to start with if NAME is relative
@@ -850,23 +874,6 @@ DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
       error ("Invalid handler in `file-name-handler-alist'");
     }
 
-  /* As a last resort, we may have to use the root as
-     default_directory below.  */
-  Lisp_Object root;
-#ifdef DOS_NT
-      /* "/" is not considered a root directory on DOS_NT, so using it
-	 as default_directory causes an infinite recursion in, e.g.,
-	 the following:
-
-            (let (default-directory)
-	      (expand-file-name "a"))
-
-	 To avoid this, we use the root of the current drive.  */
-      root = build_string (emacs_root_dir ());
-#else
-      root = build_string ("/");
-#endif
-
   /* Use the buffer's default-directory if DEFAULT_DIRECTORY is omitted.  */
   if (NILP (default_directory))
     {
@@ -891,13 +898,13 @@ DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
 	      Lisp_Object absdir
 		= STRINGP (Vinvocation_directory)
 		&& file_name_absolute_no_tilde_p (Vinvocation_directory)
-		? Vinvocation_directory : root;
+		? Vinvocation_directory : emacs_root_dir ();
 	      default_directory = Fexpand_file_name (dir, absdir);
 	    }
 	}
     }
   if (! STRINGP (default_directory))
-    default_directory = root;
+    default_directory = emacs_root_dir ();
 
   handler = Ffind_file_name_handler (default_directory, Qexpand_file_name);
   if (!NILP (handler))
diff --git a/src/lisp.h b/src/lisp.h
index 5f921d58dc..6936bbbb39 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4749,10 +4749,13 @@ maybe_disable_address_randomization (int argc, char **argv)
 extern void malloc_probe (size_t);
 extern void syms_of_profiler (void);
 
-#ifdef DOS_NT
-/* Defined in msdos.c, w32.c.  */
-extern char *emacs_root_dir (void);
-#endif /* DOS_NT */
+#ifdef MSDOS
+/* Defined in msdos.c.  */
+extern char *dos_emacs_root_dir (void);
+#elif defined (WINDOWSNT)
+/* Defined in w32.c.  */
+extern char *w32_emacs_root_dir (void);
+#endif /* MSDOS */
 
 #ifdef HAVE_NATIVE_COMP
 INLINE bool
diff --git a/src/msdos.c b/src/msdos.c
index b5f06c99c3..0827cc96cd 100644
--- a/src/msdos.c
+++ b/src/msdos.c
@@ -3350,7 +3350,7 @@ getdefdir (int drive, char *dst)
 }
 
 char *
-emacs_root_dir (void)
+dos_emacs_root_dir (void)
 {
   static char root_dir[4];
 
diff --git a/src/w32.c b/src/w32.c
index 3e71d0d383..bce8371164 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -3147,7 +3147,7 @@ #define SET_ENV_BUF_SIZE (4 * MAX_PATH)	/* to cover EMACSLOADPATH */
 /* Called from expand-file-name when default-directory is not a string.  */
 
 char *
-emacs_root_dir (void)
+w32_emacs_root_dir (void)
 {
   static char root_dir[MAX_UTF8_PATH];
   const char *p;
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> Hi,
>
> I have taken your comments into consideration and updated the patch.

Hi thanks, looks more clear to me.

question: what if instead of using Ffile_exists we just use fopen to
check if the file exists in dump_do_dump_relocation?

I think the origin of "the trouble" is just there while checking if a
file exists, the path in discussion should be already absolute by
construction so I suspect we do not need Fexpand_file to come into play.

Haven't tried, but if it works looks to me cleaner then entering in
logic where not everything is initialized.  It's true that now you have
verified that with your patch the execution path does not involve
variables to be initialized, but the logic could change in the future.

What do you think?

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: 41242@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
> Date: Fri, 29 May 2020 12:12:49 +0000
> 
> question: what if instead of using Ffile_exists we just use fopen to
> check if the file exists in dump_do_dump_relocation?
> 
> I think the origin of "the trouble" is just there while checking if a
> file exists, the path in discussion should be already absolute by
> construction so I suspect we do not need Fexpand_file to come into play.

Will that work if the files were moved?

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: 41242@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
>> Date: Fri, 29 May 2020 12:12:49 +0000
>> 
>> question: what if instead of using Ffile_exists we just use fopen to
>> check if the file exists in dump_do_dump_relocation?
>> 
>> I think the origin of "the trouble" is just there while checking if a
>> file exists, the path in discussion should be already absolute by
>> construction so I suspect we do not need Fexpand_file to come into play.
>
> Will that work if the files were moved?

I think so yes, the absolute path under discussion is generated on
purpose using Vinvocation_directory as follow:

pdumper.c:5304
=====
	if (installation_state == UNKNOWN)
	  /* Check just once if is a local build or Emacs got installed. */
	  installation_state =
	    NILP (Ffile_exists_p (concat2 (Vinvocation_directory,
					   XCAR (comp_u->file))))
	    ? LOCAL_BUILD : INSTALLED;
====

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>
>> Hi,
>>
>> I have taken your comments into consideration and updated the patch.
>
> Hi thanks, looks more clear to me.
>
> question: what if instead of using Ffile_exists we just use fopen to
> check if the file exists in dump_do_dump_relocation?
>
> I think the origin of "the trouble" is just there while checking if a
> file exists, the path in discussion should be already absolute by
> construction so I suspect we do not need Fexpand_file to come into play.
>
> Haven't tried, but if it works looks to me cleaner then entering in
> logic where not everything is initialized.  It's true that now you have
> verified that with your patch the execution path does not involve
> variables to be initialized, but the logic could change in the future.
>
> What do you think?
>
> Thanks
>
>   Andrea

I've pushed 15c121ee0b "* Avoid calling Ffile_exists_p too early"
implementing the discussed idea.

Should do the job in Windows too, please give it a try.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Nicolas Bértolo]

> I've pushed 15c121ee0b "* Avoid calling Ffile_exists_p too early"
> implementing the discussed idea.

> Should do the job in Windows too, please give it a try.

It works well, thank you.

Last night I implemented a very similar patch, but I didn't send it
because it was too late. You beat me to it.

Nico

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> I've pushed 15c121ee0b "* Avoid calling Ffile_exists_p too early"
>> implementing the discussed idea.
>
>> Should do the job in Windows too, please give it a try.
>
> It works well, thank you.

Great

> Last night I implemented a very similar patch, but I didn't send it
> because it was too late. You beat me to it.
>
> Nico

Apologies, feel free to just drop a message to signal you are willing to
or already working on something so I don't overlap.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

I have found a few bugs in Windows.
Patches attached (they are in my repo too).

- It is still a good idea to avoid the call to getenv("emacs_dir"), if only for
  performance reasons.

- We can't use "gensym" for register_native_comp_unit() because the dynamic
  library may not have been loaded yet when loading a dump file.

- Commit `f5dceed09a8234548d5b3acb76d443569533cab9` "* lisp/loadup.el: Use new
  'native-comp-available-p'." causes load_gccjit_if_necessary() to be called in
  temacs. This didn't work because because term/w32-win.el had not been loaded
  yet. In particular, we need `dynamic-library-alist` to be defined to know the
  name of the libgccjit DLL. I have defined this in syms_of_emacs(). This
  definition should be active only while dumping.

- This last bug is kinda confusing. I'm not sure about my diagnosis. The list
  `delayed_comp_unit_disposal_list` has nodes allocated with xmalloc(). It seems
  that these blocks allocated with xmalloc() get GC'd or they get corrupted
  somehow and thus they don't survive until Emacs is about to close, which is
  when we need the list. I solved it by allocating the data and nodes with
  HeapAlloc().

Thanks, Nico.

[-- Attachment #2: 0001-Determine-the-emacs-root-dir-only-when-necessary.patch --]
[-- Type: application/octet-stream, Size: 4650 bytes --]

From 42e5d34f9f17ec85da376fb6c18569ba1cc353d4 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 25 May 2020 17:55:23 -0300
Subject: [PATCH 1/4] Determine the emacs root dir only when necessary.

Fexpand_file_name() tries to obtain the root directory even when it
is not necessary because `default-directory' is not nil.

Avoid a call to getenv () unless it's necessary.

* src/fileio.c: Introduce function emacs_root_dir(). Refactor
`expand-file-name` to use it.
* src/lisp.h: Separate emacs_root_dir() into dos_emacs_root_dir() and
w32_emacs_root_dir().
* src/msdos.c: Rename emacs_root_dir() to dos_emacs_root_dir().
* src/w32.c: Rename emacs_root_dir() to w32_emacs_root_dir().
---
 src/fileio.c | 45 ++++++++++++++++++++++++++-------------------
 src/lisp.h   | 11 +++++++----
 src/msdos.c  |  2 +-
 src/w32.c    |  2 +-
 4 files changed, 35 insertions(+), 25 deletions(-)

diff --git a/src/fileio.c b/src/fileio.c
index 2f1d2f8243..11374ac56d 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -781,6 +781,30 @@ user_homedir (char const *name)
   return pw->pw_dir;
 }
 
+/* As a last resort, we may have to use the root as
+   default_directory in `expand-file-name'.
+
+   "/" is not considered a root directory on DOS_NT, so using it
+   as default_directory causes an infinite recursion in, e.g.,
+   the following:
+
+   (let (default-directory)
+          (expand-file-name "a"))
+
+   To avoid this, we use the root of the current drive. */
+
+static Lisp_Object
+emacs_root_dir (void)
+{
+#ifdef DOS
+  return build_string (dos_emacs_root_dir ());
+#elif defined (WINDOWSNT)
+  return build_string (w32_emacs_root_dir ());
+#else
+  return build_string ("/");
+#endif
+}
+
 DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
        doc: /* Convert filename NAME to absolute, and canonicalize it.
 Second arg DEFAULT-DIRECTORY is directory to start with if NAME is relative
@@ -850,23 +874,6 @@ DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
       error ("Invalid handler in `file-name-handler-alist'");
     }
 
-  /* As a last resort, we may have to use the root as
-     default_directory below.  */
-  Lisp_Object root;
-#ifdef DOS_NT
-      /* "/" is not considered a root directory on DOS_NT, so using it
-	 as default_directory causes an infinite recursion in, e.g.,
-	 the following:
-
-            (let (default-directory)
-	      (expand-file-name "a"))
-
-	 To avoid this, we use the root of the current drive.  */
-      root = build_string (emacs_root_dir ());
-#else
-      root = build_string ("/");
-#endif
-
   /* Use the buffer's default-directory if DEFAULT_DIRECTORY is omitted.  */
   if (NILP (default_directory))
     {
@@ -891,13 +898,13 @@ DEFUN ("expand-file-name", Fexpand_file_name, Sexpand_file_name, 1, 2, 0,
 	      Lisp_Object absdir
 		= STRINGP (Vinvocation_directory)
 		&& file_name_absolute_no_tilde_p (Vinvocation_directory)
-		? Vinvocation_directory : root;
+		? Vinvocation_directory : emacs_root_dir ();
 	      default_directory = Fexpand_file_name (dir, absdir);
 	    }
 	}
     }
   if (! STRINGP (default_directory))
-    default_directory = root;
+    default_directory = emacs_root_dir ();
 
   handler = Ffind_file_name_handler (default_directory, Qexpand_file_name);
   if (!NILP (handler))
diff --git a/src/lisp.h b/src/lisp.h
index 23b105e550..9704e78cf4 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4746,10 +4746,13 @@ maybe_disable_address_randomization (int argc, char **argv)
 extern void malloc_probe (size_t);
 extern void syms_of_profiler (void);
 
-#ifdef DOS_NT
-/* Defined in msdos.c, w32.c.  */
-extern char *emacs_root_dir (void);
-#endif /* DOS_NT */
+#ifdef MSDOS
+/* Defined in msdos.c.  */
+extern char *dos_emacs_root_dir (void);
+#elif defined (WINDOWSNT)
+/* Defined in w32.c.  */
+extern char *w32_emacs_root_dir (void);
+#endif /* MSDOS */
 
 #ifdef HAVE_NATIVE_COMP
 INLINE bool
diff --git a/src/msdos.c b/src/msdos.c
index b5f06c99c3..0827cc96cd 100644
--- a/src/msdos.c
+++ b/src/msdos.c
@@ -3350,7 +3350,7 @@ getdefdir (int drive, char *dst)
 }
 
 char *
-emacs_root_dir (void)
+dos_emacs_root_dir (void)
 {
   static char root_dir[4];
 
diff --git a/src/w32.c b/src/w32.c
index 3e71d0d383..bce8371164 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -3147,7 +3147,7 @@ #define SET_ENV_BUF_SIZE (4 * MAX_PATH)	/* to cover EMACSLOADPATH */
 /* Called from expand-file-name when default-directory is not a string.  */
 
 char *
-emacs_root_dir (void)
+w32_emacs_root_dir (void)
 {
   static char root_dir[MAX_UTF8_PATH];
   const char *p;
-- 
2.25.1.windows.1


[-- Attachment #3: 0002-Do-not-call-gensym-too-early-when-loading-a-dump-fil.patch --]
[-- Type: application/octet-stream, Size: 1354 bytes --]

From fc2b770805d064d3369ddbee0468ffb85b3c3a72 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 29 May 2020 21:03:00 -0300
Subject: [PATCH 2/4] Do not call `gensym' too early when loading a dump file.

This happened when subr.eln was not the first native compilation unit
to be loaded. register_native_comp_unit() is called when loading a
native compilation unit and that in turn used to call `gensym', which
was not loaded yet. This led to a SIGSEGV.

* src/comp.c (register_native_comp_unit): Replace the call to `gensym'
with an ad-hoc counter.
---
 src/comp.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/src/comp.c b/src/comp.c
index 32a98173d5..9698393f70 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -4120,7 +4120,15 @@ finish_delayed_disposal_of_comp_units (void)
 register_native_comp_unit (Lisp_Object comp_u)
 {
 #ifdef WINDOWSNT
-  Fputhash (CALL1I (gensym, Qnil), comp_u, all_loaded_comp_units_h);
+  /* We have to do this since we can't use `gensym'. This function is
+     called early when loading a dump file and subr.el may not have
+     been loaded yet. */
+  static EMACS_UINT count;
+
+  if (count >= MOST_POSITIVE_FIXNUM)
+    return;
+
+  Fputhash(make_fixnum(count++), comp_u, all_loaded_comp_units_h);
 #endif
 }
 
-- 
2.25.1.windows.1


[-- Attachment #4: 0003-Fix-loading-of-libgccjit.dll-while-dumping-in-Window.patch --]
[-- Type: application/octet-stream, Size: 1544 bytes --]

From 314fc82ae3a5050ed1b6fd06bb057f26b740ee2a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 29 May 2020 21:08:37 -0300
Subject: [PATCH 3/4] Fix loading of libgccjit.dll while dumping in Windows.

loadup.el calls `native-comp-available-p', that calls
load_gccjit_if_necessary() in Windows. That function tries to load
libgccjit using the mappings defined in `dynamic-library-alist'. That
mapping is filled by term/w32-win.el, but that file may be loaded too
late.

* src/emacs.c (syms_of_emacs): Add libgccjit to the
`dynamic-library-alist' used when starting to dump so
`native-comp-available-p' always works in Windows.
---
 src/emacs.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/emacs.c b/src/emacs.c
index 35ae9d0e66..90035bc3a3 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -3053,7 +3053,15 @@ syms_of_emacs (void)
 
 Also note that this is not a generic facility for accessing external
 libraries; only those already known by Emacs will be loaded.  */);
+#ifdef WINDOWSNT
+  /* We may need to load libgccjit when dumping before term/w32-win.el
+     defines `dynamic-library-alist`. This will fail if that variable
+     is empty, so add libgccjit.dll to it. */
+  Vdynamic_library_alist = list1(list2(Qgccjit,
+                                       build_string("libgccjit.dll")));
+#else
   Vdynamic_library_alist = Qnil;
+#endif
   Fput (intern_c_string ("dynamic-library-alist"), Qrisky_local_variable, Qt);
 
 #ifdef WINDOWSNT
-- 
2.25.1.windows.1


[-- Attachment #5: 0004-Use-Windows-allocation-APIs-for-delayed_comp_unit_di.patch --]
[-- Type: application/octet-stream, Size: 3187 bytes --]

From ee82ba57fb5397719296d882fc7b0bf5131e19b8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 29 May 2020 21:25:55 -0300
Subject: [PATCH 4/4] Use Windows allocation APIs for
 delayed_comp_unit_disposal_list.

That list and associated information need to be available right until
Emacs closes. This is not realiable when using xmalloc().

* src/comp.c: Use HeapFree and HeapAlloc instead of xfree() and
xmalloc() when allocating nodes of delayed_comp_unit_disposal_list and
the "cfile" member of Lisp_Native_Comp_Unit.
* src/pdumper.c (dump_do_dump_relocation): Use HeapAlloc for
allocating the "cfile" member of Lisp_Native_Comp_Unit.
---
 src/comp.c    | 13 ++++++++-----
 src/pdumper.c |  4 +++-
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 9698393f70..996168bbb5 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -4108,8 +4108,8 @@ finish_delayed_disposal_of_comp_units (void)
       Lisp_Object dirname = internal_condition_case_1 (
           Ffile_name_directory, build_string (item->filename), Qt, return_nil);
       clean_comp_unit_directory (dirname);
-      xfree (item->filename);
-      xfree (item);
+      HeapFree (GetProcessHeap (), 0, item->filename);
+      HeapFree (GetProcessHeap (), 0, item);
     }
 }
 #endif
@@ -4152,13 +4152,15 @@ dispose_comp_unit (struct Lisp_Native_Comp_Unit *comp_handle, bool delay)
           return_nil);
       if (!NILP (dirname))
         clean_comp_unit_directory (dirname);
-      xfree (comp_handle->cfile);
+      HeapFree (GetProcessHeap (), 0, comp_handle->cfile);
       comp_handle->cfile = NULL;
     }
   else
     {
       struct delayed_comp_unit_disposal *head;
-      head = xmalloc (sizeof (struct delayed_comp_unit_disposal));
+      head = HeapAlloc (GetProcessHeap (),
+                        0,
+                        sizeof (struct delayed_comp_unit_disposal));
       head->next = delayed_comp_unit_disposal_list;
       head->filename = comp_handle->cfile;
       comp_handle->cfile = NULL;
@@ -4552,7 +4554,8 @@ DEFUN ("native-elisp-load", Fnative_elisp_load, Snative_elisp_load, 1, 2, 0,
     xsignal2 (Qnative_lisp_load_failed, file, build_string (dynlib_error ()));
   comp_u->file = file;
 #ifdef WINDOWSNT
-  comp_u->cfile = xlispstrdup (file);
+  comp_u->cfile = HeapAlloc (GetProcessHeap (), 0, 1 + SBYTES (file));
+  strcpy (comp_u->cfile, SSDATA (file));
 #endif
   comp_u->data_vec = Qnil;
   comp_u->lambda_gc_guard = CALLN (Fmake_hash_table, QCtest, Qeq);
diff --git a/src/pdumper.c b/src/pdumper.c
index 29e3560ee5..68a7a9a536 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -5321,7 +5321,9 @@ dump_do_dump_relocation (const uintptr_t dump_base,
 		   installation_state == INSTALLED
 		   ? XCAR (comp_u->file) : XCDR (comp_u->file));
 #ifdef WINDOWSNT
-	comp_u->cfile = xlispstrdup (comp_u->file);
+        comp_u->cfile = HeapAlloc (GetProcessHeap (), 0,
+                                   1 + SBYTES (comp_u->file));
+        strcpy (comp_u->cfile, SSDATA (comp_u->file));
 #endif
 	comp_u->handle = dynlib_open (SSDATA (comp_u->file));
 	if (!comp_u->handle)
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> I have found a few bugs in Windows.
> Patches attached (they are in my repo too).
>
> - It is still a good idea to avoid the call to getenv("emacs_dir"), if only for
>   performance reasons.

I guess this should then go into master right?  Probably also others
likes to review it, in case should not be discussed in this thread.

> - We can't use "gensym" for register_native_comp_unit() because the dynamic
>   library may not have been loaded yet when loading a dump file.

The code LGTM, I just think instead of make_fixum you could use make_int
to have some more room (not sure how much is realistic this scenario but
comes for free).

> - Commit `f5dceed09a8234548d5b3acb76d443569533cab9` "* lisp/loadup.el: Use new
>   'native-comp-available-p'." causes load_gccjit_if_necessary() to be called in
>   temacs. This didn't work because because term/w32-win.el had not been loaded
>   yet. In particular, we need `dynamic-library-alist` to be defined to know the
>   name of the libgccjit DLL. I have defined this in syms_of_emacs(). This
>   definition should be active only while dumping.

I'll look into.

> - This last bug is kinda confusing. I'm not sure about my diagnosis. The list
>   `delayed_comp_unit_disposal_list` has nodes allocated with xmalloc(). It seems
>   that these blocks allocated with xmalloc() get GC'd or they get corrupted
>   somehow and thus they don't survive until Emacs is about to close, which is
>   when we need the list. I solved it by allocating the data and nodes with
>   HeapAlloc().

I think this issue deserves to be precisely understood.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

> I guess this should then go into master right?  Probably also others
> likes to review it, in case should not be discussed in this thread.

Ok, later I will open a bug report about it.

> - We can't use "gensym" for register_native_comp_unit() because the dynamic
>   library may not have been loaded yet when loading a dump file.

> The code LGTM, I just think instead of make_fixum you could use make_int
> to have some more room (not sure how much is realistic this scenario but
> comes for free).

See new attached patch.

> I'll look into.

Thanks.

> I think this issue deserves to be precisely understood.

I was not even close. The code crashes when iterating over the
`all_loaded_comp_units_h` weak hash table. We get a SIGSEGV when iterating over
a native compilation unit that has already been swept by the GC. For some
reason the hash table does not get updated. Maybe the code I used to iterate
over the hash table is not ok for weak hash tables?

=============
  struct Lisp_Hash_Table *h = XHASH_TABLE (all_loaded_comp_units_h);

  for (ptrdiff_t i = 0; i < HASH_TABLE_SIZE (h); ++i)
    {
      Lisp_Object k = HASH_KEY (h, i);
      if (!EQ (k, Qunbound))
        {
          Lisp_Object val = HASH_VALUE (h, i);
          struct Lisp_Native_Comp_Unit *cu = XNATIVE_COMP_UNIT (val);
          dispose_comp_unit (cu, false);
        }
    }
=============

Thanks, Nico.

[-- Attachment #2: 0001-Do-not-call-gensym-too-early-when-loading-a-dump-fil.patch --]
[-- Type: application/octet-stream, Size: 1288 bytes --]

From 61f943bee786f79a7164d22a2e9b4eb224fcb596 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 29 May 2020 21:03:00 -0300
Subject: [PATCH] Do not call `gensym' too early when loading a dump file.

This happened when subr.eln was not the first native compilation unit
to be loaded. register_native_comp_unit() is called when loading a
native compilation unit and that in turn used to call `gensym', which
was not loaded yet. This led to a SIGSEGV.

* src/comp.c (register_native_comp_unit): Replace the call to `gensym'
with an ad-hoc counter.
---
 src/comp.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/src/comp.c b/src/comp.c
index 32a98173d5..310ad76fbe 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -4120,7 +4120,12 @@ finish_delayed_disposal_of_comp_units (void)
 register_native_comp_unit (Lisp_Object comp_u)
 {
 #ifdef WINDOWSNT
-  Fputhash (CALL1I (gensym, Qnil), comp_u, all_loaded_comp_units_h);
+  /* We have to do this since we can't use `gensym'. This function is
+     called early when loading a dump file and subr.el may not have
+     been loaded yet. */
+  static intmax_t count;
+
+  Fputhash(make_int(count++), comp_u, all_loaded_comp_units_h);
 #endif
 }
 
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> ---
>  src/comp.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/src/comp.c b/src/comp.c
> index 32a98173d5..310ad76fbe 100644
> --- a/src/comp.c
> +++ b/src/comp.c
> @@ -4120,7 +4120,12 @@ finish_delayed_disposal_of_comp_units (void)
>  register_native_comp_unit (Lisp_Object comp_u)
>  {
>  #ifdef WINDOWSNT
> -  Fputhash (CALL1I (gensym, Qnil), comp_u, all_loaded_comp_units_h);
> +  /* We have to do this since we can't use `gensym'. This function is
> +     called early when loading a dump file and subr.el may not have
> +     been loaded yet. */
> +  static intmax_t count;
> +
> +  Fputhash(make_int(count++), comp_u, all_loaded_comp_units_h);
>  #endif
>  }

Again as suggested, *please* run 'check_GNU_style.sh' on your patches if
you are not used to GNU code style to fix it.

Presenting a patch correctly formatted, well tested and fully understood
is a sign of respect for reviewers and the time they are going to invest
in the review process.

We are all volunteers and we all have to cope with time constraints.
Investing time in reviews means subtracting it to other activities
including working on other patches and features.

We aim for code quality rather then quantity or other metrics.

The followings are to be considered as basic features we want for all
patches (not just this) to be applied to this branch:

- Compiles and bootstrap --with-nativecomp --without-nativecomp

- Formatting is correct

Obviously we can always make mistakes that is totally okay, but does not
have to be a routine that is expected to be fixed by reviewers.

Please apply these suggestions to all patches that are submitted or
pending for review to speed-up the process so we can leave the
discussion for interesting topics.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Nicolas Bértolo]

> Again as suggested, *please* run 'check_GNU_style.sh' on your patches if
> you are not used to GNU code style to fix it.

I will set it as a git hook, so I won't be able to commit unless the code is
well formatted.

> Presenting a patch correctly formatted, well tested and fully understood
> is a sign of respect for reviewers and the time they are going to invest
> in the review process.

> We are all volunteers and we all have to cope with time constraints.
> Investing time in reviews means subtracting it to other activities
> including working on other patches and features.

I have great respect for you and Eli, and for all the time you have spent
reviewing my patches. I am sorry that my lack of attention has been taken as a
lack respect for you. It will not happen again.

> We aim for code quality rather then quantity or other metrics.

> The followings are to be considered as basic features we want for all
> patches (not just this) to be applied to this branch:

> - Compiles and bootstrap --with-nativecomp --without-nativecomp

I had setup an AppVeyor instance that compiles my repo without native-comp on
Windows. I could not detect the build problems in my latest patch for some
unknown reason. I didn't expect that to happen. I will add two instances that
build the code on GNU/Linux with and without native-comp, that should help me
catch more build errors.

> - Formatting is correct

> Obviously we can always make mistakes that is totally okay, but does not
> have to be a routine that is expected to be fixed by reviewers.

> Please apply these suggestions to all patches that are submitted or
> pending for review to speed-up the process so we can leave the
> discussion for interesting topics.

I am really sorry for wasting your time like this. It will not happen again.

Nico

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 63 bytes --]

I have reformatted the two bug fixes I attached earlier.

Nico

[-- Attachment #2: 0001-Do-not-call-gensym-too-early-when-loading-a-dump-fil.patch --]
[-- Type: application/octet-stream, Size: 1295 bytes --]

From 0db0f857f3b15b57c144583132425a19cd3b562f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 29 May 2020 21:03:00 -0300
Subject: [PATCH 1/2] Do not call `gensym' too early when loading a dump file.

This happened when subr.eln was not the first native compilation unit
to be loaded. register_native_comp_unit() is called when loading a
native compilation unit and that in turn used to call `gensym', which
was not loaded yet. This led to a SIGSEGV.

* src/comp.c (register_native_comp_unit): Replace the call to `gensym'
with an ad-hoc counter.
---
 src/comp.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/src/comp.c b/src/comp.c
index 32a98173d5..d3bff1e4cf 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -4120,7 +4120,12 @@ finish_delayed_disposal_of_comp_units (void)
 register_native_comp_unit (Lisp_Object comp_u)
 {
 #ifdef WINDOWSNT
-  Fputhash (CALL1I (gensym, Qnil), comp_u, all_loaded_comp_units_h);
+  /* We have to do this since we can't use `gensym'. This function is
+     called early when loading a dump file and subr.el may not have
+     been loaded yet.  */
+  static intmax_t count;
+
+  Fputhash (make_int (count++), comp_u, all_loaded_comp_units_h);
 #endif
 }
 
-- 
2.25.1.windows.1


[-- Attachment #3: 0002-Fix-loading-of-libgccjit.dll-while-dumping-in-Window.patch --]
[-- Type: application/octet-stream, Size: 1626 bytes --]

From 87ac79e958bd5658cdc50eca747be8699f46acc7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Fri, 29 May 2020 21:08:37 -0300
Subject: [PATCH 2/2] Fix loading of libgccjit.dll while dumping in Windows.

loadup.el calls `native-comp-available-p', that calls
load_gccjit_if_necessary() in Windows. That function tries to load
libgccjit using the mappings defined in `dynamic-library-alist'. That
mapping is filled by term/w32-win.el, but that file may be loaded too
late.

* src/emacs.c (syms_of_emacs): Add libgccjit to the
`dynamic-library-alist' used when starting to dump so
`native-comp-available-p' always works in Windows.
---
 src/emacs.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/src/emacs.c b/src/emacs.c
index 35ae9d0e66..6c86931d2b 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -3053,7 +3053,18 @@ syms_of_emacs (void)
 
 Also note that this is not a generic facility for accessing external
 libraries; only those already known by Emacs will be loaded.  */);
+#ifdef WINDOWSNT
+  /* We may need to load libgccjit when dumping before term/w32-win.el
+     defines `dynamic-library-alist`. This will fail if that variable
+     is empty, so add libgccjit.dll to it.  */
+  if (will_dump_p ())
+    Vdynamic_library_alist = list1 (list2 (Qgccjit,
+                                           build_string ("libgccjit.dll")));
+  else
+    Vdynamic_library_alist = Qnil;
+#else
   Vdynamic_library_alist = Qnil;
+#endif
   Fput (intern_c_string ("dynamic-library-alist"), Qrisky_local_variable, Qt);
 
 #ifdef WINDOWSNT
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> I have reformatted the two bug fixes I attached earlier.

Appreciated, I've applied the two.

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Determine the emacs root dir...

[Eli Zaretskii]

> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Sat, 30 May 2020 10:23:55 -0300
> Cc: 41242@debbugs.gnu.org
> 
> - Commit `f5dceed09a8234548d5b3acb76d443569533cab9` "* lisp/loadup.el: Use new
>   'native-comp-available-p'." causes load_gccjit_if_necessary() to be called in
>   temacs. This didn't work because because term/w32-win.el had not been loaded
>   yet. In particular, we need `dynamic-library-alist` to be defined to know the
>   name of the libgccjit DLL. I have defined this in syms_of_emacs(). This
>   definition should be active only while dumping.

If this is only for dumping, then please make sure it has the proper
condition to be executed only at dump time.

> - This last bug is kinda confusing. I'm not sure about my diagnosis. The list
>   `delayed_comp_unit_disposal_list` has nodes allocated with xmalloc(). It seems
>   that these blocks allocated with xmalloc() get GC'd or they get corrupted
>   somehow and thus they don't survive until Emacs is about to close, which is
>   when we need the list. I solved it by allocating the data and nodes with
>   HeapAlloc().

I don't understand: the malloc implementation in the Windows build
calls HeapAlloc, so I see no reason why the latter should work while
the former doesn't.  There's some other factor at work here.

In any case, it's a definite no-no to call Windows specific APIs in a
general-purpose source file, so this patch as is cannot be acceptable.

Thanks.

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

[-- Attachment #1: Type: text/plain, Size: 8472 bytes --]

Hi Nico,

my comments on "Reduce the number of files probed when finding a lisp
file.", the full patch attached in case somebody likes to engage.

> diff --git a/src/lread.c b/src/lread.c
> index 46725d9b0f..aaf4ad1a37 100644
> --- a/src/lread.c
> +++ b/src/lread.c
> @@ -1056,33 +1056,25 @@ This uses the variables `load-suffixes' and `load-file-rep-suffixes'.  */)
>      {
>        Lisp_Object exts = Vload_file_rep_suffixes;
>        Lisp_Object suffix = XCAR (suffixes);
> -      FOR_EACH_TAIL (exts)
> -	lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
> +      if (false
> +#ifdef HAVE_MODULES
> +          || strcmp (MODULES_SUFFIX, SSDATA (suffix)) == 0
> +#ifdef MODULES_SECONDARY_SUFFIX
> +          || strcmp (MODULES_SECONDARY_SUFFIX, SSDATA (suffix)) == 0
> +#endif
> +#endif
> +#ifdef HAVE_NATIVE_COMP
> +          || strcmp (NATIVE_ELISP_SUFFIX, SSDATA (suffix)) == 0
> +#endif
> +          )

We can also use NATIVE_COMP_FLAG instead of HAVE_NATIVE_COMP to limit
the macrology when we are not forced to ifdef out code that involves
data structures only defined with native-comp.

> +	lst = Fcons (suffix, lst);
> +      else
> +        FOR_EACH_TAIL (exts)
> +          lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
>      }
>    return Fnreverse (lst);
>  }
>
> -static Lisp_Object
> -effective_load_path (void)
> -{
> -#ifndef HAVE_NATIVE_COMP
> -  return Vload_path;
> -#else
> -  Lisp_Object lp = Vload_path;
> -  Lisp_Object new_lp = Qnil;
> -  FOR_EACH_TAIL (lp)
> -    {
> -      Lisp_Object el = XCAR (lp);
> -      new_lp =
> -	Fcons (concat2 (Ffile_name_as_directory (el),
> -			Vcomp_native_path_postfix),
> -	       new_lp);
> -      new_lp = Fcons (el, new_lp);
> -    }
> -  return Fnreverse (new_lp);
> -#endif
> -}
> -
>  /* Return true if STRING ends with SUFFIX.  */
>  bool
>  suffix_p (Lisp_Object string, const char *suffix)
> @@ -1217,6 +1209,9 @@ Return t if the file exists and loads successfully.  */)
>  #ifdef MODULES_SECONDARY_SUFFIX
>                || suffix_p (file, MODULES_SECONDARY_SUFFIX)
>  #endif
> +#endif
> +#ifdef HAVE_NATIVE_COMP
> +              || suffix_p (file, NATIVE_ELISP_SUFFIX)
>  #endif
>  	      )

Same as previous comment.

>  	    must_suffix = Qnil;
> @@ -1236,8 +1231,8 @@ Return t if the file exists and loads successfully.  */)
>  	}
>
>        fd =
> -	openp (effective_load_path (), file, suffixes, &found, Qnil,
> -	       load_prefer_newer);
> +	openp (Vload_path, file, true, suffixes, &found, Qnil,
> +               load_prefer_newer);
>      }
>
>    if (fd == -1)
> @@ -1600,7 +1595,7 @@ directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
>    (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate)
>  {
>    Lisp_Object file;
> -  int fd = openp (path, filename, suffixes, &file, predicate, false);
> +  int fd = openp (path, filename, Qnil, suffixes, &file, predicate, false);
                                     ^^^
                                     false?

>    if (NILP (predicate) && fd >= 0)
>      emacs_close (fd);
>    return file;
> @@ -1611,6 +1606,9 @@ directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
>     On success, return a file descriptor (or 1 or -2 as described below).
>     On failure, return -1 and set errno.
>
> +   If ADD_ELN_DIR is nonzero, use the `comp-native-path-postfix'
> +   variable to change the searched path if the suffix is .eln.
> +

Is there a specific reason to add a parameter to exclude/includes elns?

I'd say we want to have an homogeneous behavior across all call sites
no?

>     SUFFIXES is a list of strings containing possible suffixes.
>     The empty suffix is automatically added if the list is empty.
>
> @@ -1633,8 +1631,9 @@ directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
>     or if a non-nil and non-t PREDICATE is specified.  */
>
>  int
> -openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
> -       Lisp_Object *storeptr, Lisp_Object predicate, bool newer)
> +openp (Lisp_Object path, Lisp_Object str, bool add_eln_dir,
> +       Lisp_Object suffixes, Lisp_Object *storeptr, Lisp_Object predicate,
> +       bool newer)
>  {
>    ptrdiff_t fn_size = 100;
>    char buf[100];
> @@ -1643,7 +1642,7 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
>    ptrdiff_t want_length;
>    Lisp_Object filename;
>    Lisp_Object string, tail, encoded_fn, save_string;
> -  ptrdiff_t max_suffix_len = 0;
> +  ptrdiff_t max_extra_len = 0;
>    int last_errno = ENOENT;
>    int save_fd = -1;
>    USE_SAFE_ALLOCA;
> @@ -1658,8 +1657,15 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
>    FOR_EACH_TAIL_SAFE (tail)
>      {
>        CHECK_STRING_CAR (tail);
> -      max_suffix_len = max (max_suffix_len,
> -			    SBYTES (XCAR (tail)));
> +      char * suf = SSDATA (XCAR (tail));
              ^^
              no space

> +      ptrdiff_t len = SBYTES (XCAR (tail));
> +      if (add_eln_dir && strcmp (NATIVE_ELISP_SUFFIX, suf) == 0)
> +        {
> +          CHECK_STRING (Vcomp_native_path_postfix);
> +          len += 2;
> +          len += SBYTES (Vcomp_native_path_postfix);
> +        }
> +      max_extra_len = max (max_extra_len, len);
>      }
>
>    string = filename = encoded_fn = save_string = Qnil;
> @@ -1677,7 +1683,7 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
>       executable. */
>    FOR_EACH_TAIL_SAFE (path)
>     {
> -    ptrdiff_t baselen, prefixlen;
> +    ptrdiff_t dirnamelen, prefixlen, basenamewext_len;
>
>      if (EQ (path, just_use_str))
>        filename = str;
> @@ -1694,22 +1700,27 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
>  	  continue;
>        }
>
> +
>      /* Calculate maximum length of any filename made from
>         this path element/specified file name and any possible suffix.  */
> -    want_length = max_suffix_len + SBYTES (filename);
> +    want_length = max_extra_len + SBYTES (filename);
>      if (fn_size <= want_length)
>        {
>  	fn_size = 100 + want_length;
>  	fn = SAFE_ALLOCA (fn_size);
>        }
>
> +    Lisp_Object dirnamewslash = Ffile_name_directory (filename);
> +    Lisp_Object basenamewext = Ffile_name_nondirectory (filename);
> +
>      /* Copy FILENAME's data to FN but remove starting /: if any.  */
> -    prefixlen = ((SCHARS (filename) > 2
> -		  && SREF (filename, 0) == '/'
> -		  && SREF (filename, 1) == ':')
> +    prefixlen = ((SCHARS (dirnamewslash) > 2
> +		  && SREF (dirnamewslash, 0) == '/'
> +		  && SREF (dirnamewslash, 1) == ':')
>  		 ? 2 : 0);
> -    baselen = SBYTES (filename) - prefixlen;
> -    memcpy (fn, SDATA (filename) + prefixlen, baselen);
> +    dirnamelen = SBYTES (dirnamewslash) - prefixlen;
> +    basenamewext_len = SBYTES (basenamewext);
> +    memcpy (fn, SDATA (dirnamewslash) + prefixlen, dirnamelen);
>
>      /* Loop over suffixes.  */
>      AUTO_LIST1 (empty_string_only, empty_unibyte_string);
> @@ -1719,10 +1730,24 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
>  	Lisp_Object suffix = XCAR (tail);
>  	ptrdiff_t fnlen, lsuffix = SBYTES (suffix);
>  	Lisp_Object handler;
> +        bool add_native_comp_dir = add_eln_dir
> +          && (strcmp (SSDATA (suffix), NATIVE_ELISP_SUFFIX) == 0);

I think this should be optimized for non native comp builds with a
NATIVE_COMP_FLAG && in front.

> +        ptrdiff_t lmiddledir = add_native_comp_dir
> +            ? SBYTES (Vcomp_native_path_postfix) + 1 : 0;
> +
> +        if (add_native_comp_dir)
> +          {
> +            memcpy (fn + dirnamelen, SDATA (Vcomp_native_path_postfix),
> +                   lmiddledir - 1);

Vcomp_native_path_postfix is only defined with HAVE_NATIVE_COMP so I
guess this breaks the stock build and my previous suggestion is not
applicable.

> +            fn[dirnamelen+(lmiddledir-1)] = '/';

Spaces around operators + -.

I could not compile this patch because non all the calls to openp has
been updated for the new parameter (my question on adding this stands).

In general please recall to check the stock build when working on
infrastructure integration, it's quite easy to break.

Generally speaking I think the behavior we want to have is that when a
.eln file is specified this is loaded without re-adding
Vcomp_native_path_postfix.  I could not test it but I suspect this is
not handled.

Thanks this is good to have in when tuned.

  Andrea

--
akrl@sdf.org

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Reduce-the-number-of-files-probed-when-finding-a-lis.patch --]
[-- Type: text/x-diff, Size: 13196 bytes --]

From d6b6ac651c62e6a5cd9701d94419cfd53e95976e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 25 May 2020 18:05:23 -0300
Subject: [PATCH] Reduce the number of files probed when finding a lisp file.

* src/lread.c (get-load-suffixes): Do not add any suffix to files that
need to be loaded by the dynamic linker.
(effective_load_path): Remove function.
(load): Don't add any suffix if file ends in a suffix already.
(openp): Add a parameter `add_eln_dir` that should be used when we
desire the dir elt/eln-hash/ to be searched for each path in `path`.
* src/lisp.h: Add new parameter to openp.
* src/callproc.c: Add the new parameter to calls to openp().
* src/charset.c: Add the new parameter to calls to openp().
* src/emacs.c: Add the new parameter to calls to openp().
* src/image.c: Add the new parameter to calls to openp().
* src/process.c: Add the new parameter to calls to openp().
* src/w32.c: Add the new parameter to calls to openp().
* src/w32proc.c: Add the new parameter to calls to openp().
---
 src/callproc.c |   2 +-
 src/charset.c  |   2 +-
 src/emacs.c    |   3 +-
 src/image.c    |   4 +-
 src/lisp.h     |   2 +-
 src/lread.c    | 107 ++++++++++++++++++++++++++++++-------------------
 src/process.c  |   2 +-
 src/w32.c      |   3 +-
 src/w32proc.c  |   3 +-
 9 files changed, 78 insertions(+), 50 deletions(-)

diff --git a/src/callproc.c b/src/callproc.c
index 65c858393a..421e42de11 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -436,7 +436,7 @@ call_process (ptrdiff_t nargs, Lisp_Object *args, int filefd,
   {
     int ok;
 
-    ok = openp (Vexec_path, args[0], Vexec_suffixes, &path,
+    ok = openp (Vexec_path, args[0], false, Vexec_suffixes, &path,
 		make_fixnum (X_OK), false);
     if (ok < 0)
       report_file_error ("Searching for program", args[0]);
diff --git a/src/charset.c b/src/charset.c
index 8635aad3ed..da3ace743f 100644
--- a/src/charset.c
+++ b/src/charset.c
@@ -486,7 +486,7 @@ load_charset_map_from_file (struct charset *charset, Lisp_Object mapfile,
   ptrdiff_t count = SPECPDL_INDEX ();
   record_unwind_protect_nothing ();
   specbind (Qfile_name_handler_alist, Qnil);
-  fd = openp (Vcharset_map_path, mapfile, suffixes, NULL, Qnil, false);
+  fd = openp (Vcharset_map_path, mapfile, false, suffixes, NULL, Qnil, false);
   fp = fd < 0 ? 0 : fdopen (fd, "r");
   if (!fp)
     {
diff --git a/src/emacs.c b/src/emacs.c
index cd4f7a0b28..35ae9d0e66 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -454,7 +454,8 @@ set_invocation_vars (char *argv0, char const *original_pwd)
     {
       Lisp_Object found;
       int yes = openp (Vexec_path, Vinvocation_name,
-		       Vexec_suffixes, &found, make_fixnum (X_OK), false);
+		       false, Vexec_suffixes, &found, make_fixnum (X_OK),
+                       false);
       if (yes == 1)
 	{
 	  /* Add /: to the front of the name
diff --git a/src/image.c b/src/image.c
index c8a192aaaf..80ef801913 100644
--- a/src/image.c
+++ b/src/image.c
@@ -508,7 +508,7 @@ image_create_bitmap_from_file (struct frame *f, Lisp_Object file)
     }
 
   /* Search bitmap-file-path for the file, if appropriate.  */
-  if (openp (Vx_bitmap_file_path, file, Qnil, &found,
+  if (openp (Vx_bitmap_file_path, file, false, Qnil, &found,
 	     make_fixnum (R_OK), false)
       < 0)
     return -1;
@@ -2983,7 +2983,7 @@ image_find_image_fd (Lisp_Object file, int *pfd)
 		       Vx_bitmap_file_path);
 
   /* Try to find FILE in data-directory/images, then x-bitmap-file-path.  */
-  fd = openp (search_path, file, Qnil, &file_found,
+  fd = openp (search_path, file, false, Qnil, &file_found,
 	      pfd ? Qt : make_fixnum (R_OK), false);
   if (fd >= 0 || fd == -2)
     {
diff --git a/src/lisp.h b/src/lisp.h
index 52242791aa..5f921d58dc 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4094,7 +4094,7 @@ LOADHIST_ATTACH (Lisp_Object x)
 extern bool suffix_p (Lisp_Object, const char *);
 extern Lisp_Object save_match_data_load (Lisp_Object, Lisp_Object, Lisp_Object,
 					 Lisp_Object, Lisp_Object);
-extern int openp (Lisp_Object, Lisp_Object, Lisp_Object,
+extern int openp (Lisp_Object, Lisp_Object, bool, Lisp_Object,
                   Lisp_Object *, Lisp_Object, bool);
 enum { S2N_IGNORE_TRAILING = 1 };
 extern Lisp_Object string_to_number (char const *, int, ptrdiff_t *);
diff --git a/src/lread.c b/src/lread.c
index 46725d9b0f..aaf4ad1a37 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1056,33 +1056,25 @@ This uses the variables `load-suffixes' and `load-file-rep-suffixes'.  */)
     {
       Lisp_Object exts = Vload_file_rep_suffixes;
       Lisp_Object suffix = XCAR (suffixes);
-      FOR_EACH_TAIL (exts)
-	lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
+      if (false
+#ifdef HAVE_MODULES
+          || strcmp (MODULES_SUFFIX, SSDATA (suffix)) == 0
+#ifdef MODULES_SECONDARY_SUFFIX
+          || strcmp (MODULES_SECONDARY_SUFFIX, SSDATA (suffix)) == 0
+#endif
+#endif
+#ifdef HAVE_NATIVE_COMP
+          || strcmp (NATIVE_ELISP_SUFFIX, SSDATA (suffix)) == 0
+#endif
+          )
+	lst = Fcons (suffix, lst);
+      else
+        FOR_EACH_TAIL (exts)
+          lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
     }
   return Fnreverse (lst);
 }
 
-static Lisp_Object
-effective_load_path (void)
-{
-#ifndef HAVE_NATIVE_COMP
-  return Vload_path;
-#else
-  Lisp_Object lp = Vload_path;
-  Lisp_Object new_lp = Qnil;
-  FOR_EACH_TAIL (lp)
-    {
-      Lisp_Object el = XCAR (lp);
-      new_lp =
-	Fcons (concat2 (Ffile_name_as_directory (el),
-			Vcomp_native_path_postfix),
-	       new_lp);
-      new_lp = Fcons (el, new_lp);
-    }
-  return Fnreverse (new_lp);
-#endif
-}
-
 /* Return true if STRING ends with SUFFIX.  */
 bool
 suffix_p (Lisp_Object string, const char *suffix)
@@ -1217,6 +1209,9 @@ Return t if the file exists and loads successfully.  */)
 #ifdef MODULES_SECONDARY_SUFFIX
               || suffix_p (file, MODULES_SECONDARY_SUFFIX)
 #endif
+#endif
+#ifdef HAVE_NATIVE_COMP
+              || suffix_p (file, NATIVE_ELISP_SUFFIX)
 #endif
 	      )
 	    must_suffix = Qnil;
@@ -1236,8 +1231,8 @@ Return t if the file exists and loads successfully.  */)
 	}
 
       fd =
-	openp (effective_load_path (), file, suffixes, &found, Qnil,
-	       load_prefer_newer);
+	openp (Vload_path, file, true, suffixes, &found, Qnil,
+               load_prefer_newer);
     }
 
   if (fd == -1)
@@ -1600,7 +1595,7 @@ directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
   (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate)
 {
   Lisp_Object file;
-  int fd = openp (path, filename, suffixes, &file, predicate, false);
+  int fd = openp (path, filename, Qnil, suffixes, &file, predicate, false);
   if (NILP (predicate) && fd >= 0)
     emacs_close (fd);
   return file;
@@ -1611,6 +1606,9 @@ directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
    On success, return a file descriptor (or 1 or -2 as described below).
    On failure, return -1 and set errno.
 
+   If ADD_ELN_DIR is nonzero, use the `comp-native-path-postfix'
+   variable to change the searched path if the suffix is .eln.
+
    SUFFIXES is a list of strings containing possible suffixes.
    The empty suffix is automatically added if the list is empty.
 
@@ -1633,8 +1631,9 @@ directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
    or if a non-nil and non-t PREDICATE is specified.  */
 
 int
-openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
-       Lisp_Object *storeptr, Lisp_Object predicate, bool newer)
+openp (Lisp_Object path, Lisp_Object str, bool add_eln_dir,
+       Lisp_Object suffixes, Lisp_Object *storeptr, Lisp_Object predicate,
+       bool newer)
 {
   ptrdiff_t fn_size = 100;
   char buf[100];
@@ -1643,7 +1642,7 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
   ptrdiff_t want_length;
   Lisp_Object filename;
   Lisp_Object string, tail, encoded_fn, save_string;
-  ptrdiff_t max_suffix_len = 0;
+  ptrdiff_t max_extra_len = 0;
   int last_errno = ENOENT;
   int save_fd = -1;
   USE_SAFE_ALLOCA;
@@ -1658,8 +1657,15 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
   FOR_EACH_TAIL_SAFE (tail)
     {
       CHECK_STRING_CAR (tail);
-      max_suffix_len = max (max_suffix_len,
-			    SBYTES (XCAR (tail)));
+      char * suf = SSDATA (XCAR (tail));
+      ptrdiff_t len = SBYTES (XCAR (tail));
+      if (add_eln_dir && strcmp (NATIVE_ELISP_SUFFIX, suf) == 0)
+        {
+          CHECK_STRING (Vcomp_native_path_postfix);
+          len += 2;
+          len += SBYTES (Vcomp_native_path_postfix);
+        }
+      max_extra_len = max (max_extra_len, len);
     }
 
   string = filename = encoded_fn = save_string = Qnil;
@@ -1677,7 +1683,7 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
      executable. */
   FOR_EACH_TAIL_SAFE (path)
    {
-    ptrdiff_t baselen, prefixlen;
+    ptrdiff_t dirnamelen, prefixlen, basenamewext_len;
 
     if (EQ (path, just_use_str))
       filename = str;
@@ -1694,22 +1700,27 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
 	  continue;
       }
 
+
     /* Calculate maximum length of any filename made from
        this path element/specified file name and any possible suffix.  */
-    want_length = max_suffix_len + SBYTES (filename);
+    want_length = max_extra_len + SBYTES (filename);
     if (fn_size <= want_length)
       {
 	fn_size = 100 + want_length;
 	fn = SAFE_ALLOCA (fn_size);
       }
 
+    Lisp_Object dirnamewslash = Ffile_name_directory (filename);
+    Lisp_Object basenamewext = Ffile_name_nondirectory (filename);
+
     /* Copy FILENAME's data to FN but remove starting /: if any.  */
-    prefixlen = ((SCHARS (filename) > 2
-		  && SREF (filename, 0) == '/'
-		  && SREF (filename, 1) == ':')
+    prefixlen = ((SCHARS (dirnamewslash) > 2
+		  && SREF (dirnamewslash, 0) == '/'
+		  && SREF (dirnamewslash, 1) == ':')
 		 ? 2 : 0);
-    baselen = SBYTES (filename) - prefixlen;
-    memcpy (fn, SDATA (filename) + prefixlen, baselen);
+    dirnamelen = SBYTES (dirnamewslash) - prefixlen;
+    basenamewext_len = SBYTES (basenamewext);
+    memcpy (fn, SDATA (dirnamewslash) + prefixlen, dirnamelen);
 
     /* Loop over suffixes.  */
     AUTO_LIST1 (empty_string_only, empty_unibyte_string);
@@ -1719,10 +1730,24 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
 	Lisp_Object suffix = XCAR (tail);
 	ptrdiff_t fnlen, lsuffix = SBYTES (suffix);
 	Lisp_Object handler;
+        bool add_native_comp_dir = add_eln_dir
+          && (strcmp (SSDATA (suffix), NATIVE_ELISP_SUFFIX) == 0);
+        ptrdiff_t lmiddledir = add_native_comp_dir
+            ? SBYTES (Vcomp_native_path_postfix) + 1 : 0;
+
+        if (add_native_comp_dir)
+          {
+            memcpy (fn + dirnamelen, SDATA (Vcomp_native_path_postfix),
+                   lmiddledir - 1);
+            fn[dirnamelen+(lmiddledir-1)] = '/';
+          }
 
-	/* Make complete filename by appending SUFFIX.  */
-	memcpy (fn + baselen, SDATA (suffix), lsuffix + 1);
-	fnlen = baselen + lsuffix;
+        memcpy (fn + dirnamelen + lmiddledir, SDATA (basenamewext),
+                basenamewext_len);
+        /* Make complete filename by appending SUFFIX.  */
+        memcpy (fn + dirnamelen + lmiddledir + basenamewext_len,
+                SDATA (suffix), lsuffix + 1);
+        fnlen = dirnamelen + lmiddledir + basenamewext_len + lsuffix;
 
 	/* Check that the file exists and is not a directory.  */
 	/* We used to only check for handlers on non-absolute file names:
diff --git a/src/process.c b/src/process.c
index 6e5bcf307a..e092fa0a0c 100644
--- a/src/process.c
+++ b/src/process.c
@@ -1903,7 +1903,7 @@ usage: (make-process &rest ARGS)  */)
 	       && IS_DEVICE_SEP (SREF (program, 1))))
 	{
 	  tem = Qnil;
-	  openp (Vexec_path, program, Vexec_suffixes, &tem,
+	  openp (Vexec_path, program, false, Vexec_suffixes, &tem,
 		 make_fixnum (X_OK), false);
 	  if (NILP (tem))
 	    report_file_error ("Searching for program", program);
diff --git a/src/w32.c b/src/w32.c
index 38bbc49656..3e71d0d383 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -10187,7 +10187,8 @@ check_windows_init_file (void)
 	 need to ENCODE_FILE here, but we do need to convert the file
 	 names from UTF-8 to ANSI.  */
       init_file = build_string ("term/w32-win");
-      fd = openp (Vload_path, init_file, Fget_load_suffixes (), NULL, Qnil, 0);
+      fd = openp (Vload_path, init_file, true, Fget_load_suffixes (), NULL,
+                  Qnil, 0);
       if (fd < 0)
 	{
 	  Lisp_Object load_path_print = Fprin1_to_string (Vload_path, Qnil);
diff --git a/src/w32proc.c b/src/w32proc.c
index 16e32e4c58..4aa4c9fa0a 100644
--- a/src/w32proc.c
+++ b/src/w32proc.c
@@ -1918,7 +1918,8 @@ sys_spawnve (int mode, char *cmdname, char **argv, char **envp)
     {
       program = build_string (cmdname);
       full = Qnil;
-      openp (Vexec_path, program, Vexec_suffixes, &full, make_fixnum (X_OK), 0);
+      openp (Vexec_path, program, false, Vexec_suffixes, &full,
+             make_fixnum (X_OK), 0);
       if (NILP (full))
 	{
 	  errno = EINVAL;
-- 
2.20.1

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

Hi Andrea,

> I could not compile this patch because non all the calls to openp has
> been updated for the new parameter (my question on adding this stands).

Sorry. I didn't check the GNU/Linux build.

> In general please recall to check the stock build when working on
> infrastructure integration, it's quite easy to break.
I have tested that this new patch builds and bootstraps in windows x64,
Ubuntu 18.04 amd64. Both with and without nativecomp.

> Generally speaking I think the behavior we want to have is that when a
> .eln file is specified this is loaded without re-adding
> Vcomp_native_path_postfix.  I could not test it but I suspect this is
> not handled.

I tested moving company.eln to a directory in load-path without
`comp-native-path-postfix` and then ran (load "company.eln") and it was
loaded from that directory.

Nico.

[-- Attachment #2: 0001-Reduce-the-number-of-files-probed-when-finding-a-lis.patch --]
[-- Type: application/octet-stream, Size: 10669 bytes --]

From d659a8a7ed9918d7a77655109abc6db5338ac354 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 25 May 2020 18:05:23 -0300
Subject: [PATCH] Reduce the number of files probed when finding a lisp file.

* src/lread.c (get-load-suffixes): Do not add any suffix to files that
need to be loaded by the dynamic linker.
(effective_load_path): Remove function.
(load): Don't add any suffix if file ends in a suffix already.
(effective_load_path): Remove function.
(openp_add_middle_dir_to_suffixes): Add helper function to create
pairs of middle directories and suffixes.
(openp_max_middledir_and_suffix_len): Add helper function to count the
number of bytes needed to store the middle directory and suffix.
(openp_fill_filename_buffer): Add helper function to copy middle
directory, basename and suffix to the filename buffer.
---
 src/lread.c | 203 +++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 154 insertions(+), 49 deletions(-)

diff --git a/src/lread.c b/src/lread.c
index 9f849eda42..c6666e4ea3 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1056,31 +1056,25 @@ DEFUN ("get-load-suffixes", Fget_load_suffixes, Sget_load_suffixes, 0, 0, 0,
     {
       Lisp_Object exts = Vload_file_rep_suffixes;
       Lisp_Object suffix = XCAR (suffixes);
-      FOR_EACH_TAIL (exts)
-	lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
-    }
-  return Fnreverse (lst);
-}
+      bool native_code_suffix = (NATIVE_COMP_FLAG
+        && strcmp (NATIVE_ELISP_SUFFIX, SSDATA (suffix)) == 0);
 
-static Lisp_Object
-effective_load_path (void)
-{
-#ifndef HAVE_NATIVE_COMP
-  return Vload_path;
-#else
-  Lisp_Object lp = Vload_path;
-  Lisp_Object new_lp = Qnil;
-  FOR_EACH_TAIL (lp)
-    {
-      Lisp_Object el = XCAR (lp);
-      new_lp =
-	Fcons (concat2 (Ffile_name_as_directory (el),
-			Vcomp_native_path_postfix),
-	       new_lp);
-      new_lp = Fcons (el, new_lp);
-    }
-  return Fnreverse (new_lp);
+#ifdef HAVE_MODULES
+      native_code_suffix = native_code_suffix
+        || strcmp (MODULES_SUFFIX, SSDATA (suffix)) == 0;
+#ifdef MODULES_SECONDARY_SUFFIX
+      native_code_suffix = native_code_suffix
+        || strcmp (MODULES_SECONDARY_SUFFIX, SSDATA (suffix)) == 0;
+#endif
 #endif
+
+      if (native_code_suffix)
+	lst = Fcons (suffix, lst);
+      else
+        FOR_EACH_TAIL (exts)
+          lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
+    }
+  return Fnreverse (lst);
 }
 
 /* Return true if STRING ends with SUFFIX.  */
@@ -1218,7 +1212,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
               || suffix_p (file, MODULES_SECONDARY_SUFFIX)
 #endif
 #endif
-	      )
+              || (NATIVE_COMP_FLAG && suffix_p (file, NATIVE_ELISP_SUFFIX)))
 	    must_suffix = Qnil;
 	  /* Don't insist on adding a suffix
 	     if the argument includes a directory name.  */
@@ -1236,8 +1230,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 	}
 
       fd =
-	openp (effective_load_path (), file, suffixes, &found, Qnil,
-	       load_prefer_newer);
+	openp (Vload_path, file, suffixes, &found, Qnil, load_prefer_newer);
     }
 
   if (fd == -1)
@@ -1606,6 +1599,116 @@ DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2,
   return file;
 }
 
+/* This function turns a list of suffixes into a list of middle dirs
+   and suffixes.  If the suffix is not NATIVE_ELISP_SUFFIX then its
+   suffix is nil and it is added to the list as is.  Instead, if it
+   suffix is NATIVE_ELISP_SUFFIX then two elements are added to the
+   list.  The first one has middledir equal to nil and the second uses
+   comp-native-path-postfix as middledir.  This is because we'd like
+   to search for dir/foo.eln before dir/middledir/foo.eln.
+
+For example, it turns this:
+
+(".eln" ".elc" ".elc.gz" ".el" ".el.gz")
+
+ into this:
+
+((nil . ".eln")
+ (comp-native-path-postfix . ".eln")
+ (nil . ".elc")
+ (nil . ".elc.gz")
+ (nil . ".el")
+ (nil . ".el.gz"))
+*/
+static Lisp_Object
+openp_add_middle_dir_to_suffixes (Lisp_Object suffixes)
+{
+  Lisp_Object tail = suffixes;
+  Lisp_Object extended_suf = Qnil;
+  FOR_EACH_TAIL_SAFE (tail)
+    {
+#ifdef HAVE_NATIVE_COMP
+      CHECK_STRING_CAR (tail);
+      char * suf = SSDATA (XCAR (tail));
+      if (strcmp (NATIVE_ELISP_SUFFIX, suf) == 0)
+        {
+          CHECK_STRING (Vcomp_native_path_postfix);
+          /* Here we add them in the opposite order so that nreverse
+             corrects it.  */
+          extended_suf = Fcons (Fcons (Qnil, XCAR (tail)), extended_suf);
+          extended_suf = Fcons (Fcons (Vcomp_native_path_postfix, XCAR (tail)),
+                                extended_suf);
+        }
+      else
+#endif
+        {
+          extended_suf = Fcons (Fcons (Qnil, XCAR (tail)), extended_suf);
+        }
+    }
+
+  suffixes = Fnreverse (extended_suf);
+  return suffixes;
+}
+
+/*  This function takes a list of middledirs and suffixes and returns
+    the maximum buffer space that this part of the filename will
+    need.  */
+static ptrdiff_t
+openp_max_middledir_and_suffix_len (Lisp_Object middledir_and_suffixes)
+{
+  ptrdiff_t max_extra_len = 0;
+  Lisp_Object tail = middledir_and_suffixes;
+  FOR_EACH_TAIL_SAFE (tail)
+    {
+      Lisp_Object middledir_and_suffix = XCAR (tail);
+      Lisp_Object middledir = XCAR (middledir_and_suffix);
+      Lisp_Object suffix = XCDR (middledir_and_suffix);
+      ptrdiff_t len = SBYTES (suffix);
+      if (!NILP (middledir))
+          len += 2 + SBYTES (middledir); /* Add two slashes.  */
+      max_extra_len = max (max_extra_len, len);
+    }
+  return max_extra_len;
+}
+
+/*  This function completes the FN buffer with the middledir,
+    basenameme, and suffix.  It takes the directory length in DIRNAME,
+    but it requires that it has been copied already to the start of
+    the buffer.
+
+    After this function the FN buffer will be (depending on middledir)
+    dirname/middledir/basename.suffix
+    or
+    dirname/basename.suffix
+*/
+static ptrdiff_t
+openp_fill_filename_buffer (char *fn, ptrdiff_t dirnamelen,
+                            Lisp_Object basenamewext,
+                            Lisp_Object middledir_and_suffix)
+{
+  Lisp_Object middledir = XCAR (middledir_and_suffix);
+  Lisp_Object suffix = XCDR (middledir_and_suffix);
+  ptrdiff_t basenamewext_len = SBYTES (basenamewext);
+  ptrdiff_t fnlen, lsuffix = SBYTES (suffix);
+  ptrdiff_t lmiddledir = 0;
+  if (!NILP (middledir))
+    {
+      /* Add 1 for the slash.  */
+      lmiddledir = SBYTES (middledir) + 1;
+      memcpy (fn + dirnamelen, SDATA (middledir),
+              lmiddledir - 1);
+      fn[dirnamelen + (lmiddledir - 1)] = '/';
+    }
+
+  memcpy (fn + dirnamelen + lmiddledir, SDATA (basenamewext),
+          basenamewext_len);
+  /* Make complete filename by appending SUFFIX.  */
+  memcpy (fn + dirnamelen + lmiddledir + basenamewext_len,
+          SDATA (suffix), lsuffix + 1);
+  fnlen = dirnamelen + lmiddledir + basenamewext_len + lsuffix;
+  return fnlen;
+}
+
 /* Search for a file whose name is STR, looking in directories
    in the Lisp list PATH, and trying suffixes from SUFFIX.
    On success, return a file descriptor (or 1 or -2 as described below).
@@ -1643,7 +1746,8 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
   ptrdiff_t want_length;
   Lisp_Object filename;
   Lisp_Object string, tail, encoded_fn, save_string;
-  ptrdiff_t max_suffix_len = 0;
+  Lisp_Object middledir_and_suffixes;
+  ptrdiff_t max_extra_len = 0;
   int last_errno = ENOENT;
   int save_fd = -1;
   USE_SAFE_ALLOCA;
@@ -1654,13 +1758,9 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
 
   CHECK_STRING (str);
 
-  tail = suffixes;
-  FOR_EACH_TAIL_SAFE (tail)
-    {
-      CHECK_STRING_CAR (tail);
-      max_suffix_len = max (max_suffix_len,
-			    SBYTES (XCAR (tail)));
-    }
+  middledir_and_suffixes = openp_add_middle_dir_to_suffixes (suffixes);
+
+  max_extra_len = openp_max_middledir_and_suffix_len (middledir_and_suffixes);
 
   string = filename = encoded_fn = save_string = Qnil;
 
@@ -1677,7 +1777,7 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
      executable. */
   FOR_EACH_TAIL_SAFE (path)
    {
-    ptrdiff_t baselen, prefixlen;
+    ptrdiff_t dirnamelen, prefixlen;
 
     if (EQ (path, just_use_str))
       filename = str;
@@ -1694,35 +1794,40 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
 	  continue;
       }
 
+
     /* Calculate maximum length of any filename made from
        this path element/specified file name and any possible suffix.  */
-    want_length = max_suffix_len + SBYTES (filename);
+    want_length = max_extra_len + SBYTES (filename);
     if (fn_size <= want_length)
       {
 	fn_size = 100 + want_length;
 	fn = SAFE_ALLOCA (fn_size);
       }
 
+    Lisp_Object dirnamewslash = Ffile_name_directory (filename);
+    Lisp_Object basenamewext = Ffile_name_nondirectory (filename);
+
     /* Copy FILENAME's data to FN but remove starting /: if any.  */
-    prefixlen = ((SCHARS (filename) > 2
-		  && SREF (filename, 0) == '/'
-		  && SREF (filename, 1) == ':')
+    prefixlen = ((SCHARS (dirnamewslash) > 2
+		  && SREF (dirnamewslash, 0) == '/'
+		  && SREF (dirnamewslash, 1) == ':')
 		 ? 2 : 0);
-    baselen = SBYTES (filename) - prefixlen;
-    memcpy (fn, SDATA (filename) + prefixlen, baselen);
+    dirnamelen = SBYTES (dirnamewslash) - prefixlen;
+    memcpy (fn, SDATA (dirnamewslash) + prefixlen, dirnamelen);
 
-    /* Loop over suffixes.  */
-    AUTO_LIST1 (empty_string_only, empty_unibyte_string);
-    tail = NILP (suffixes) ? empty_string_only : suffixes;
+    /* Loop over middledir_and_suffixes.  */
+    AUTO_LIST1 (empty_string_only, Fcons (Qnil, empty_unibyte_string));
+    tail = NILP (middledir_and_suffixes) ? empty_string_only
+                                         : middledir_and_suffixes;
     FOR_EACH_TAIL_SAFE (tail)
       {
-	Lisp_Object suffix = XCAR (tail);
-	ptrdiff_t fnlen, lsuffix = SBYTES (suffix);
+	Lisp_Object middledir_and_suffix = XCAR (tail);
+        Lisp_Object suffix = XCDR (middledir_and_suffix);
 	Lisp_Object handler;
 
-	/* Make complete filename by appending SUFFIX.  */
-	memcpy (fn + baselen, SDATA (suffix), lsuffix + 1);
-	fnlen = baselen + lsuffix;
+        ptrdiff_t fnlen = openp_fill_filename_buffer (fn, dirnamelen,
+                                                      basenamewext,
+                                                      middledir_and_suffix);
 
 	/* Check that the file exists and is not a directory.  */
 	/* We used to only check for handlers on non-absolute file names:
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

Last night I came up with a fresh idea to properly solve the problem with slow
loads. Considering that it happens when `load-path` contains hundreds of
directories I thought that a caching mechanism would help.

This "cache" would consist of a mapping of files that would get probed when
(load "foo") runs. It would be implemented as a hash table. This could get
stored in `package-user-dir` and package.el would be in charge of updating it
when installing or removing packages, just like autoloads.

The contents of this hash table could be something like this:

The directory company-20200525.101 could have a file load-cache.el with:

company -> ("company-20200525.101/eln-hash/company.eln"
            "company-20200525.101/company.el"
            "company-20200525.101/company.elc")
[...]

The directory helm-20200517.509 could have a file load-cache.el with:

helm -> ("helm-20200517.509/eln-hash/helm.eln"
         "helm-20200517.509/helm.el"
         "helm-20200517.509/helm.elc")
[...]

When `load-path` changes we could update the in-memory hash table by loading all
the load-cache.el files.

Then, when (require 'foo) runs, the loading code could look at the hash
table and only fopen() the files associated with the feature we are loading.
This would reduce the number of calls to fopen() from thousands to ~3 in the
worst case.

Of course, this feature would be disabled by default.

What do you think?

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> Last night I came up with a fresh idea to properly solve the problem with slow
> loads. Considering that it happens when `load-path` contains hundreds of
> directories I thought that a caching mechanism would help.
>
> This "cache" would consist of a mapping of files that would get probed when
> (load "foo") runs. It would be implemented as a hash table. This could get
> stored in `package-user-dir` and package.el would be in charge of updating it
> when installing or removing packages, just like autoloads.
>
> The contents of this hash table could be something like this:
>
> The directory company-20200525.101 could have a file load-cache.el with:
>
> company -> ("company-20200525.101/eln-hash/company.eln"
>             "company-20200525.101/company.el"
>             "company-20200525.101/company.elc")
> [...]
>
> The directory helm-20200517.509 could have a file load-cache.el with:
>
> helm -> ("helm-20200517.509/eln-hash/helm.eln"
>          "helm-20200517.509/helm.el"
>          "helm-20200517.509/helm.elc")
> [...]
>
> When `load-path` changes we could update the in-memory hash table by loading all
> the load-cache.el files.
>
> Then, when (require 'foo) runs, the loading code could look at the hash
> table and only fopen() the files associated with the feature we are loading.
> This would reduce the number of calls to fopen() from thousands to ~3 in the
> worst case.
>
> Of course, this feature would be disabled by default.
>
> What do you think?

Hi Nico,

could you recall me why this is specific to the native-comp branch?
Isn't this a problem that should affect any Emacs on Windows?

Thanks

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

> Isn't this a problem that should affect any Emacs on Windows?

You are right. I will open a separate bug.

Nico

El lun., 1 jun. 2020 a las 4:21, Andrea Corallo (<akrl@sdf.org>) escribió:
>
> Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>
> > Last night I came up with a fresh idea to properly solve the problem with slow
> > loads. Considering that it happens when `load-path` contains hundreds of
> > directories I thought that a caching mechanism would help.
> >
> > This "cache" would consist of a mapping of files that would get probed when
> > (load "foo") runs. It would be implemented as a hash table. This could get
> > stored in `package-user-dir` and package.el would be in charge of updating it
> > when installing or removing packages, just like autoloads.
> >
> > The contents of this hash table could be something like this:
> >
> > The directory company-20200525.101 could have a file load-cache.el with:
> >
> > company -> ("company-20200525.101/eln-hash/company.eln"
> >             "company-20200525.101/company.el"
> >             "company-20200525.101/company.elc")
> > [...]
> >
> > The directory helm-20200517.509 could have a file load-cache.el with:
> >
> > helm -> ("helm-20200517.509/eln-hash/helm.eln"
> >          "helm-20200517.509/helm.el"
> >          "helm-20200517.509/helm.elc")
> > [...]
> >
> > When `load-path` changes we could update the in-memory hash table by loading all
> > the load-cache.el files.
> >
> > Then, when (require 'foo) runs, the loading code could look at the hash
> > table and only fopen() the files associated with the feature we are loading.
> > This would reduce the number of calls to fopen() from thousands to ~3 in the
> > worst case.
> >
> > Of course, this feature would be disabled by default.
> >
> > What do you think?
>
> Hi Nico,
>
> could you recall me why this is specific to the native-comp branch?
> Isn't this a problem that should affect any Emacs on Windows?
>
> Thanks
>
>   Andrea
>
> --
> akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> Hi Andrea,
>
>> I could not compile this patch because non all the calls to openp has
>> been updated for the new parameter (my question on adding this stands).
>
> Sorry. I didn't check the GNU/Linux build.
>
>> In general please recall to check the stock build when working on
>> infrastructure integration, it's quite easy to break.
> I have tested that this new patch builds and bootstraps in windows x64,
> Ubuntu 18.04 amd64. Both with and without nativecomp.
>
>> Generally speaking I think the behavior we want to have is that when a
>> .eln file is specified this is loaded without re-adding
>> Vcomp_native_path_postfix.  I could not test it but I suspect this is
>> not handled.
>
> I tested moving company.eln to a directory in load-path without
> `comp-native-path-postfix` and then ran (load "company.eln") and it was
> loaded from that directory.
>
> Nico.

Hi Nico,

thanks this looks better.  I've two nits and two hopefully smarter
observations:

> +++ b/src/lread.c
> @@ -1056,31 +1056,25 @@ DEFUN ("get-load-suffixes", Fget_load_suffixes, Sget_load_suffixes, 0, 0, 0,
>      {
>        Lisp_Object exts = Vload_file_rep_suffixes;
>        Lisp_Object suffix = XCAR (suffixes);
> -      FOR_EACH_TAIL (exts)
> -	lst = Fcons (concat2 (suffix, XCAR (exts)), lst);
> -    }
> -  return Fnreverse (lst);
> -}
> +      bool native_code_suffix = (NATIVE_COMP_FLAG
> +        && strcmp (NATIVE_ELISP_SUFFIX, SSDATA (suffix)) == 0);

I think with the outer parentesys the second line should go be indented
at the level of NATIVE_COMP_FLAG.  I'd probably remove the parenthesys
and put the newline after the equal.

> -static Lisp_Object
> -effective_load_path (void)
> -{
> -#ifndef HAVE_NATIVE_COMP
> -  return Vload_path;

[...]

> +#ifdef HAVE_NATIVE_COMP
> +      CHECK_STRING_CAR (tail);
> +      char * suf = SSDATA (XCAR (tail));
> +      if (strcmp (NATIVE_ELISP_SUFFIX, suf) == 0)
> +        {
> +          CHECK_STRING (Vcomp_native_path_postfix);
> +          /* Here we add them in the opposite order so that nreverse
> +             corrects it.  */
> +          extended_suf = Fcons (Fcons (Qnil, XCAR (tail)), extended_suf);
> +          extended_suf = Fcons (Fcons (Vcomp_native_path_postfix, XCAR (tail)),
> +                                extended_suf);
> +        }
> +      else
> +#endif
> +        {
> +          extended_suf = Fcons (Fcons (Qnil, XCAR (tail)), extended_suf);
> +        }

I think GNU style does not want these brackets if the statement is just
one.

Okay interesting stuffs:

In which folders are we going to search if we do (load "...a/path/foo.eln")?

I believe in this case we should search the file only in "...a/path/"
because the user really want to load this specific file.  Am I correct?

That said IMO this logic is sufficiently complex to deserve a minimum of
testing to make sure we have it under control.  Not sure if the best
place is files-tests.el or comp-tests.el.

Maybe Eli likes to gives his opinion on this last point and on the patch
in general.

Thanks

  Andrea

--
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

> In which folders are we going to search if we do (load "...a/path/foo.eln")?

> I believe in this case we should search the file only in "...a/path/"
> because the user really want to load this specific file.  Am I correct?

I'm not sure we want the load function to be that smart. This is from
the manual:

-----
The load function is not clever about looking at filename. In the perverse case
of a file named foo.el.el, evaluation of (load "foo.el") will indeed find it.
-----

I think we should respect that principle when dealing with .eln files, even if
it leads to trying to open absurd filenames.

I did some tests, and these are the files probed in each case:

(load "dir/foo.eln")

"{elt}/dir/foo.eln.eln"
"{elt}/dir/eln-hash/foo.eln.eln"
"{elt}/dir/foo.eln.dll"
"{elt}/dir/foo.eln.elc"
"{elt}/dir/foo.eln.elc.gz"
"{elt}/dir/foo.eln.el"
"{elt}/dir/foo.eln.el.gz"
"{elt}/dir/foo.eln"
"{elt}/dir/foo.eln.gz"

where {elt} is an element from `load-path`.

(load "C:/dir/foo.eln")

"c:/dir/foo.eln.eln"
"c:/dir/eln-hash/foo.eln.eln"
"c:/dir/foo.eln.dll"
"c:/dir/foo.eln.elc"
"c:/dir/foo.eln.elc.gz"
"c:/dir/foo.eln.el"
"c:/dir/foo.eln.el.gz"
"c:/dir/foo.eln"
"c:/dir/foo.eln.gz"

(load "dir/foo.eln" nil nil t) <- nosuffix: t

"{elt}/dir/foo.eln"

(load "C:/dir/foo.eln" nil nil t) <- nosuffix: t

"C:/dir/foo.eln"

> That said IMO this logic is sufficiently complex to deserve a minimum of
> testing to make sure we have it under control.  Not sure if the best
> place is files-tests.el or comp-tests.el.

I agree about this. I am not sure what is the best way to do it. The list of
files probed is inaccessible from Lisp.

Thanks,

Nico.

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> In which folders are we going to search if we do (load "...a/path/foo.eln")?
>
>> I believe in this case we should search the file only in "...a/path/"
>> because the user really want to load this specific file.  Am I correct?
>
> I'm not sure we want the load function to be that smart. This is from
> the manual:
>
> -----
> The load function is not clever about looking at filename. In the perverse case
> of a file named foo.el.el, evaluation of (load "foo.el") will indeed find it.
> -----
>
> I think we should respect that principle when dealing with .eln files, even if
> it leads to trying to open absurd filenames.

Mmmh you are maybe right, but the directory and the filename are two
different things, I've to think about it.  I'd be curious of other
opinions.

> I did some tests, and these are the files probed in each case:
>
> (load "dir/foo.eln")
>
> "{elt}/dir/foo.eln.eln"
> "{elt}/dir/eln-hash/foo.eln.eln"
> "{elt}/dir/foo.eln.dll"
> "{elt}/dir/foo.eln.elc"
> "{elt}/dir/foo.eln.elc.gz"
> "{elt}/dir/foo.eln.el"
> "{elt}/dir/foo.eln.el.gz"
> "{elt}/dir/foo.eln"
> "{elt}/dir/foo.eln.gz"
>
> where {elt} is an element from `load-path`.
>
> (load "C:/dir/foo.eln")
>
> "c:/dir/foo.eln.eln"
> "c:/dir/eln-hash/foo.eln.eln"
> "c:/dir/foo.eln.dll"
> "c:/dir/foo.eln.elc"
> "c:/dir/foo.eln.elc.gz"
> "c:/dir/foo.eln.el"
> "c:/dir/foo.eln.el.gz"
> "c:/dir/foo.eln"
> "c:/dir/foo.eln.gz"
>
> (load "dir/foo.eln" nil nil t) <- nosuffix: t
>
> "{elt}/dir/foo.eln"
>
> (load "C:/dir/foo.eln" nil nil t) <- nosuffix: t
>
> "C:/dir/foo.eln"
>
>> That said IMO this logic is sufficiently complex to deserve a minimum of
>> testing to make sure we have it under control.  Not sure if the best
>> place is files-tests.el or comp-tests.el.
>
> I agree about this. I am not sure what is the best way to do it. The list of
> files probed is inaccessible from Lisp.

Yes but we can always compile a file and load it to see what's inside,
this is what we do in comp-tests.el, technically is not a problem if we
think is necessary.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: 41242@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
> Date: Mon, 01 Jun 2020 19:24:43 +0000
> 
> In which folders are we going to search if we do (load "...a/path/foo.eln")?
> 
> I believe in this case we should search the file only in "...a/path/"
> because the user really want to load this specific file.  Am I correct?

Isn't that already so when we look for *.elc files?

> That said IMO this logic is sufficiently complex to deserve a minimum of
> testing to make sure we have it under control.  Not sure if the best
> place is files-tests.el or comp-tests.el.
> 
> Maybe Eli likes to gives his opinion on this last point and on the patch
> in general.

I think the logic should be consistent with how we search for Lisp
files in general.

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: 41242@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
>> Date: Mon, 01 Jun 2020 19:24:43 +0000
>> 
>> In which folders are we going to search if we do (load "...a/path/foo.eln")?
>> 
>> I believe in this case we should search the file only in "...a/path/"
>> because the user really want to load this specific file.  Am I correct?
>
> Isn't that already so when we look for *.elc files?

Yes but here the hash directory that we use to disambiguate the triplet
comes into play so we search there too.  This is what Nico posted about
what we would probe for a load.

(load "C:/dir/foo.eln")

"c:/dir/foo.eln.eln"
"c:/dir/eln-hash/foo.eln.eln"
"c:/dir/foo.eln.dll"
"c:/dir/foo.eln.elc"
"c:/dir/foo.eln.elc.gz"
"c:/dir/foo.eln.el"
"c:/dir/foo.eln.el.gz"
"c:/dir/foo.eln"
"c:/dir/foo.eln.gz"

My argument was that in the case of (load "C:/dir/foo.eln") we should
try to load only "c:/dir/foo.eln" without having to look into
"c:/dir/eln-hash/".

But Nico pointed out (probably correctly) that the function is already
quite dumb regarding ignoring extentions and is probably not worth doing
an exception for this.

>> That said IMO this logic is sufficiently complex to deserve a minimum of
>> testing to make sure we have it under control.  Not sure if the best
>> place is files-tests.el or comp-tests.el.
>> 
>> Maybe Eli likes to gives his opinion on this last point and on the patch
>> in general.
>
> I think the logic should be consistent with how we search for Lisp
> files in general.
>

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 1988 bytes --]

Hi,

The recent patches broke the Windows build. There is one patch that
fixes it and another one that improves the way to check for the gccjit version
functions.

Nico.

El mar., 2 jun. 2020 a las 13:24, Andrea Corallo (<akrl@sdf.org>) escribió:
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> From: Andrea Corallo <akrl@sdf.org>
> >> Cc: 41242@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
> >> Date: Mon, 01 Jun 2020 19:24:43 +0000
> >>
> >> In which folders are we going to search if we do (load "...a/path/foo.eln")?
> >>
> >> I believe in this case we should search the file only in "...a/path/"
> >> because the user really want to load this specific file.  Am I correct?
> >
> > Isn't that already so when we look for *.elc files?
>
> Yes but here the hash directory that we use to disambiguate the triplet
> comes into play so we search there too.  This is what Nico posted about
> what we would probe for a load.
>
> (load "C:/dir/foo.eln")
>
> "c:/dir/foo.eln.eln"
> "c:/dir/eln-hash/foo.eln.eln"
> "c:/dir/foo.eln.dll"
> "c:/dir/foo.eln.elc"
> "c:/dir/foo.eln.elc.gz"
> "c:/dir/foo.eln.el"
> "c:/dir/foo.eln.el.gz"
> "c:/dir/foo.eln"
> "c:/dir/foo.eln.gz"
>
> My argument was that in the case of (load "C:/dir/foo.eln") we should
> try to load only "c:/dir/foo.eln" without having to look into
> "c:/dir/eln-hash/".
>
> But Nico pointed out (probably correctly) that the function is already
> quite dumb regarding ignoring extentions and is probably not worth doing
> an exception for this.
>
> >> That said IMO this logic is sufficiently complex to deserve a minimum of
> >> testing to make sure we have it under control.  Not sure if the best
> >> place is files-tests.el or comp-tests.el.
> >>
> >> Maybe Eli likes to gives his opinion on this last point and on the patch
> >> in general.
> >
> > I think the logic should be consistent with how we search for Lisp
> > files in general.
> >
>
> --
> akrl@sdf.org

[-- Attachment #2: 0001-Fix-DLL-imports-of-gccjit-version-functions.patch --]
[-- Type: application/octet-stream, Size: 3196 bytes --]

From 80f6a7fc7a531e15ea909af9e254515e8888005a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 1 Jun 2020 19:53:00 -0300
Subject: [PATCH 1/2] Fix DLL imports of gccjit version functions.

* src/comp.c (init_gccjit_functions): Use LOAD_DLL_FN_OPT macro to
load gcc_jit_version_major, gcc_jit_version_major and
gcc_jit_version_patchlevel.
* src/w32common.h (LOAD_DLL_FN_OPT): Add macro optionally load a
function from a DLL.
---
 src/comp.c      | 18 ++++++++++++------
 src/w32common.h |  8 ++++++++
 2 files changed, 20 insertions(+), 6 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index d0574ac5ef3..8e7582b3e65 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -98,6 +98,9 @@
 #undef gcc_jit_struct_as_type
 #undef gcc_jit_struct_set_fields
 #undef gcc_jit_type_get_pointer
+#undef gcc_jit_version_major
+#undef gcc_jit_version_minor
+#undef gcc_jit_version_patchlevel
 
 /* In alphabetical order */
 DEF_DLL_FN (gcc_jit_rvalue *, gcc_jit_context_new_rvalue_from_int,
@@ -231,9 +234,9 @@ DEF_DLL_FN (void, gcc_jit_context_set_logfile,
 DEF_DLL_FN (void, gcc_jit_struct_set_fields,
             (gcc_jit_struct *struct_type, gcc_jit_location *loc, int num_fields,
              gcc_jit_field **fields));
-DEF_DLL_FN (int, gcc_jit_version_major);
-DEF_DLL_FN (int, gcc_jit_version_minor);
-DEF_DLL_FN (int, gcc_jit_version_patchlevel);
+DEF_DLL_FN (int, gcc_jit_version_major, (void));
+DEF_DLL_FN (int, gcc_jit_version_minor, (void));
+DEF_DLL_FN (int, gcc_jit_version_patchlevel, (void));
 
 static bool
 init_gccjit_functions (void)
@@ -297,9 +300,9 @@ init_gccjit_functions (void)
   LOAD_DLL_FN (library, gcc_jit_struct_as_type);
   LOAD_DLL_FN (library, gcc_jit_struct_set_fields);
   LOAD_DLL_FN (library, gcc_jit_type_get_pointer);
-  LOAD_DLL_FN (library, gcc_jit_version_major);
-  LOAD_DLL_FN (library, gcc_jit_version_minor);
-  LOAD_DLL_FN (library, gcc_jit_version_patchlevel);
+  LOAD_DLL_FN_OPT (library, gcc_jit_version_major);
+  LOAD_DLL_FN_OPT (library, gcc_jit_version_minor);
+  LOAD_DLL_FN_OPT (library, gcc_jit_version_patchlevel);
 
   return true;
 }
@@ -358,6 +361,9 @@ #define gcc_jit_rvalue_get_type fn_gcc_jit_rvalue_get_type
 #define gcc_jit_struct_as_type fn_gcc_jit_struct_as_type
 #define gcc_jit_struct_set_fields fn_gcc_jit_struct_set_fields
 #define gcc_jit_type_get_pointer fn_gcc_jit_type_get_pointer
+#define gcc_jit_version_major fn_gcc_jit_version_major
+#define gcc_jit_version_minor fn_gcc_jit_version_minor
+#define gcc_jit_version_patchlevel fn_gcc_jit_version_patchlevel
 
 #endif
 
diff --git a/src/w32common.h b/src/w32common.h
index eb7faa1939a..bd01fd40401 100644
--- a/src/w32common.h
+++ b/src/w32common.h
@@ -81,6 +81,14 @@ #define LOAD_DLL_FN(lib, func)						\
     }									\
   while (false)
 
+/* Load a function from the DLL, and don't fail if it does not exist.  */
+#define LOAD_DLL_FN_OPT(lib, func)                                      \
+  do									\
+    {									\
+      fn_##func = (W32_PFN_##func) get_proc_addr (lib, #func);		\
+    }									\
+  while (false)
+
 #ifdef HAVE_HARFBUZZ
 extern bool hbfont_init_w32_funcs (HMODULE);
 #endif
-- 
2.25.1.windows.1


[-- Attachment #3: 0002-Remove-kludge-for-detecting-if-gcc_jit_version_major.patch --]
[-- Type: application/octet-stream, Size: 1868 bytes --]

From 8aa7673f626bfa2c12e6f9c8bbec102ce2fb72d1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Mon, 1 Jun 2020 19:56:37 -0300
Subject: [PATCH 2/2] Remove kludge for detecting if gcc_jit_version_major can
 be called.

* src/comp.c (comp-libgccjit-version): Use 'dlsym' in Posix and the
normal function loading mechanism in Windows.
---
 src/comp.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 8e7582b3e65..084d0b6745b 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -365,6 +365,8 @@ #define gcc_jit_version_major fn_gcc_jit_version_major
 #define gcc_jit_version_minor fn_gcc_jit_version_minor
 #define gcc_jit_version_patchlevel fn_gcc_jit_version_patchlevel
 
+#else
+#include <dlfcn.h>
 #endif
 
 static bool
@@ -3999,15 +4001,18 @@ DEFUN ("comp-libgccjit-version", Fcomp_libgccjit_version,
 #if defined (LIBGCCJIT_HAVE_gcc_jit_version) || defined (WINDOWSNT)
   load_gccjit_if_necessary (true);
 
-  /* FIXME this kludge is quite bad.  Can we dynamically load on all
-     operating systems?  */
-#pragma GCC diagnostic ignored "-Waddress"
-  return gcc_jit_version_major
-    ? list3 (make_fixnum (gcc_jit_version_major ()),
-	     make_fixnum (gcc_jit_version_minor ()),
-	     make_fixnum (gcc_jit_version_patchlevel ()))
-    : Qnil;
-#pragma GCC diagnostic pop
+  bool ok_to_call;
+
+#if defined (WINDOWSNT)
+  ok_to_call = !!fn_gcc_jit_version_major;
+#else
+  ok_to_call = !!dlsym (RTLD_DEFAULT, "gcc_jit_version_major");
+#endif
+
+  return ok_to_call ? list3 (make_fixnum (gcc_jit_version_major ()),
+                             make_fixnum (gcc_jit_version_minor ()),
+                             make_fixnum (gcc_jit_version_patchlevel ()))
+                    : Qnil;
 #else
   return Qnil;
 #endif
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> Hi,
>
> The recent patches broke the Windows build. There is one patch that
> fixes it and another one that improves the way to check for the gccjit version
> functions.

I've pushed "Fix DLL imports of gccjit version functions." on the second
I've some perplexity.

> Nico.
> From 8aa7673f626bfa2c12e6f9c8bbec102ce2fb72d1 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
> Date: Mon, 1 Jun 2020 19:56:37 -0300
> Subject: [PATCH 2/2] Remove kludge for detecting if gcc_jit_version_major can
>  be called.
>
> * src/comp.c (comp-libgccjit-version): Use 'dlsym' in Posix and the
> normal function loading mechanism in Windows.
> ---
>  src/comp.c | 23 ++++++++++++++---------
>  1 file changed, 14 insertions(+), 9 deletions(-)
>
> diff --git a/src/comp.c b/src/comp.c
> index 8e7582b3e65..084d0b6745b 100644
> --- a/src/comp.c
> +++ b/src/comp.c
> @@ -365,6 +365,8 @@ #define gcc_jit_version_major fn_gcc_jit_version_major
>  #define gcc_jit_version_minor fn_gcc_jit_version_minor
>  #define gcc_jit_version_patchlevel fn_gcc_jit_version_patchlevel
>  
> +#else
> +#include <dlfcn.h>
>  #endif
>  
>  static bool
> @@ -3999,15 +4001,18 @@ DEFUN ("comp-libgccjit-version", Fcomp_libgccjit_version,
>  #if defined (LIBGCCJIT_HAVE_gcc_jit_version) || defined (WINDOWSNT)
>    load_gccjit_if_necessary (true);
>  
> -  /* FIXME this kludge is quite bad.  Can we dynamically load on all
> -     operating systems?  */
> -#pragma GCC diagnostic ignored "-Waddress"
> -  return gcc_jit_version_major
> -    ? list3 (make_fixnum (gcc_jit_version_major ()),
> -	     make_fixnum (gcc_jit_version_minor ()),
> -	     make_fixnum (gcc_jit_version_patchlevel ()))
> -    : Qnil;
> -#pragma GCC diagnostic pop
> +  bool ok_to_call;
> +
> +#if defined (WINDOWSNT)
> +  ok_to_call = !!fn_gcc_jit_version_major;
> +#else
> +  ok_to_call = !!dlsym (RTLD_DEFAULT, "gcc_jit_version_major");
> +#endif
> +
> +  return ok_to_call ? list3 (make_fixnum (gcc_jit_version_major ()),
> +                             make_fixnum (gcc_jit_version_minor ()),
> +                             make_fixnum (gcc_jit_version_patchlevel ()))
> +                    : Qnil;
>  #else
>    return Qnil;
>  #endif

First given is giving away two pragmas for adding three preprocessor
directives I'd say it is adding more kludge then what is removing.

Second this code would fail both link time and load time if the linker
cannot resolve one of these three functions making the test on the
result of dlsym in the run time pointless.  To work one needs to go
through function pointers as you are doing in Window, but some macrology
is requested otherwise would be a kludge^2.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

> Second this code would fail both link time and load time if the linker
> cannot resolve one of these three functions making the test on the
> result of dlsym in the run time pointless.

I thought that on GNU/Linux the dynamic linker would just resolve a function
when necessary (i.e. function call) and not when starting the executable. I
assumed that lazy binding was the default, and then the call to dlsym would
prevent a crash.

This is what the man page of 'ld' says about the '-z lazy' option:

When generating an executable or shared library, mark it to tell the dynamic
linker to defer function call resolution to the point when the function is
called (lazy binding), rather than at load time. Lazy binding is the default.

I checked and the call to gcc to link temacs does not include '-z now', so lazy
binding should be in effect and the dlsym trick should work.

Anyway, if you think that it's ugly I respect that and won't object to it not
getting merged.

Thanks, Nico.

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Second this code would fail both link time and load time if the linker
>> cannot resolve one of these three functions making the test on the
>> result of dlsym in the run time pointless.
>
> I thought that on GNU/Linux the dynamic linker would just resolve a function
> when necessary (i.e. function call) and not when starting the executable. I
> assumed that lazy binding was the default, and then the call to dlsym would
> prevent a crash.
>
> This is what the man page of 'ld' says about the '-z lazy' option:
>
> When generating an executable or shared library, mark it to tell the dynamic
> linker to defer function call resolution to the point when the function is
> called (lazy binding), rather than at load time. Lazy binding is the default.

Yes, but even if you use lazy linking this code would not link if the
linker cannot verify the presence of these functions.  In this case all
users with a non 10 libgccjit.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

> Yes, but even if you use lazy linking this code would not link if the
> linker cannot verify the presence of these functions.  In this case all
> users with a non 10 libgccjit.

Shouldn't the first #ifdef remove the code for those users? That one checks for
LIBGCCJIT_HAVE_gcc_jit_version.

Nico.

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Yes, but even if you use lazy linking this code would not link if the
>> linker cannot verify the presence of these functions.  In this case all
>> users with a non 10 libgccjit.
>
> Shouldn't the first #ifdef remove the code for those users? That one checks for
> LIBGCCJIT_HAVE_gcc_jit_version.

Oh my, possibly.  This code is exactly the definition of what a terrible
kludge is.

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

I did the suggested style modification myself and applied the patch as
e38678b268.

Thanks I'm closing this.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

> I did the suggested style modification myself and applied the patch as
> e38678b268.

Thanks.

I didn't have much time during the week to look at how to implement the testing
part. I have just came up with an idea that is abusing file-name-handler-alist.
If for tests I add a handler that logs all operations we can indirectly access
the list of files probed.

What do you think of this as a way of testing "load"?

Nico.

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> I did the suggested style modification myself and applied the patch as
>> e38678b268.
>
> Thanks.
>
> I didn't have much time during the week to look at how to implement the testing
> part. I have just came up with an idea that is abusing file-name-handler-alist.
> If for tests I add a handler that logs all operations we can indirectly access
> the list of files probed.
>
> What do you think of this as a way of testing "load"?
>
> Nico.

Hi Nico,

mmmh sounds a bit heavy but I'm not able to tell if there is a better
and comprehensive at the same time way of testing it.  Perhaps if you
are chasing the hash bug I guess is more important now.

If there's no reasonable way of testing it either we write a couple of
ad hoc tests or we say we are fine with that.

Not sure if you have seen this
https://lists.gnu.org/archive/html/emacs-devel/2020-06/msg00050.html
there is also the possibility that we are going to change radically all
the eln load mechanism.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

> Perhaps if you are chasing the hash bug I guess is more important now.

I agree, but that bug keeps eluding me. I thought of not using a Lisp hashtable
and moving to an adhoc C list may be better.

> there is also the possibility that we are going to change radically all
> the eln load mechanism.

No I hadn't. That sounds good. But there are a few issues that I'm not sure
about. How would you build the hash? Only the filename? The full path? Both of
these options have problems. The current approach makes it easy to avoid
filename collisions and to associate .eln files with their source code.

Right now I'm trying to fix a bug that prevents the Windows build from dumping.
It only happens in 64 bits and nativecomp does not make a difference, but I
haven't been able to reproduce it in master. It occurs during the marking stage
of the GC.

I'll have to bisect the newest changes from master, as I see there has been work
on the GC fixing bugs.

Nico.

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

>> Perhaps if you are chasing the hash bug I guess is more important now.
>
> I agree, but that bug keeps eluding me. I thought of not using a Lisp hashtable
> and moving to an adhoc C list may be better.
>
>> there is also the possibility that we are going to change radically all
>> the eln load mechanism.
>
> No I hadn't. That sounds good. But there are a few issues that I'm not sure
> about. How would you build the hash? Only the filename? The full path? Both of
> these options have problems. The current approach makes it easy to avoid
> filename collisions and to associate .eln files with their source code.

Yeah there are a lot of open questions.  Certainly if we go for this
solution it has to be fully transparent and very robust.  Please join
the thread there with your observations.

> Right now I'm trying to fix a bug that prevents the Windows build from dumping.
> It only happens in 64 bits and nativecomp does not make a difference, but I
> haven't been able to reproduce it in master. It occurs during the marking stage
> of the GC.

Mmmh strange, recent GC fixes *should* not impact you.  Will be
interesting to see what's the issue.

> I'll have to bisect the newest changes from master, as I see there has been work
> on the GC fixing bugs.
>
> Nico.

  Andrea

-- 
akrl@sdf.org

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Nicolas Bértolo]

[-- Attachment #1: Type: text/plain, Size: 328 bytes --]

Hi,

I have attached a bugfix for the issue that causes crashes when closing Emacs.

Ideally I would figure out why iterating over  a weak hash-table does not skip
elements that were already GC'd, but I could not do it. I feel fixing
this bug is
very important, so I used a C array to keep the list of native
compilation units.

[-- Attachment #2: 0001-Use-a-C-array-to-keep-the-list-of-live-native-compil.patch --]
[-- Type: application/octet-stream, Size: 7537 bytes --]

From 717437852b7e91a5fb9cf87dab21b4661f3bb3b4 Mon Sep 17 00:00:00 2001
From: Nicolas Bertolo <nicolasbertolo@gmail.com>
Date: Sun, 7 Jun 2020 15:54:50 -0300
Subject: [PATCH] Use a C array to keep the list of live native compilation
 units.

I was getting crashes related to the access to the hashtable when
Emacs was about to close. I couldn't debug them, so I replaced it with
a simple C array.

* src/lisp.h (allocate_native_comp_unit): register every native
compilation unit allocated.
* src/comp.h: Expose register_native_comp_unit () to lisp.h.
* src/comp.c: Remove all_loaded_comp_units_h. Replace it with a
dynamically sized array.
---
 src/comp.c | 94 +++++++++++++++++++++++++++++++-----------------------
 src/comp.h |  5 +++
 src/lisp.h |  9 ++++--
 3 files changed, 66 insertions(+), 42 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index 521cadcb10c..1a7eafaaadf 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -4111,17 +4111,17 @@ helper_PSEUDOVECTOR_TYPEP_XUNTAG (Lisp_Object a, enum pvec_type code)
 
   There are two data structures used:
 
-  - The `all_loaded_comp_units_h` hashtable.
+  - The `all_loaded_comp_units` list.
 
-  This hashtable is used like an array of weak references to native
-  compilation units.  This hash table is filled by load_comp_unit ()
-  and dispose_all_remaining_comp_units () iterates over all values
-  that were not disposed by the GC and performs all disposal steps
-  when Emacs is closing.
+  This list is filled by allocate_native_comp_unit () and
+  dispose_all_remaining_comp_units () iterates over all values that
+  remain and performs all disposal steps when Emacs is closing.  The
+  dispose_comp_unit () function removes entries that were disposed by
+  the GC.
 
   - The `delayed_comp_unit_disposal_list` list.
 
-  This is were the dispose_comp_unit () function, when called by the
+  This is where the dispose_comp_unit () function, when called by the
   GC sweep stage, stores the original filenames of the disposed native
   compilation units.  This is an ad-hoc C structure instead of a Lisp
   cons because we need to allocate instances of this structure during
@@ -4136,7 +4136,35 @@ helper_PSEUDOVECTOR_TYPEP_XUNTAG (Lisp_Object a, enum pvec_type code)
 #ifdef WINDOWSNT
 #define OLD_ELN_SUFFIX_REGEXP build_string ("\\.eln\\.old\\'")
 
-static Lisp_Object all_loaded_comp_units_h;
+struct all_loaded_comp_units_s {
+  struct Lisp_Native_Comp_Unit **mem;
+  size_t size;
+  size_t capacity;
+};
+
+static struct all_loaded_comp_units_s all_loaded_comp_units;
+
+static void
+loaded_comp_units_remove (struct Lisp_Native_Comp_Unit * comp_u)
+{
+  size_t i;
+  bool found = false;
+  for (i = 0 ; i < all_loaded_comp_units.size; ++i)
+    if (all_loaded_comp_units.mem[i] == comp_u)
+      {
+        found = true;
+        break;
+      }
+  if (!found)
+    emacs_abort ();
+
+  size_t elements_on_right = all_loaded_comp_units.size - i - 1;
+  memmove (&all_loaded_comp_units.mem[i],
+           &all_loaded_comp_units.mem[i + 1],
+           elements_on_right * sizeof (struct Lisp_Native_Comp_Unit *));
+
+  all_loaded_comp_units.mem[--all_loaded_comp_units.size] = NULL;
+}
 
 /* We need to allocate instances of this struct during a GC sweep.
    This is why it can't be transformed into a simple cons.  */
@@ -4193,17 +4221,9 @@ clean_package_user_dir_of_old_comp_units (void)
 void
 dispose_all_remaining_comp_units (void)
 {
-  struct Lisp_Hash_Table *h = XHASH_TABLE (all_loaded_comp_units_h);
-
-  for (ptrdiff_t i = 0; i < HASH_TABLE_SIZE (h); ++i)
+  for (ptrdiff_t i = all_loaded_comp_units.size - 1; i >= 0; --i)
     {
-      Lisp_Object k = HASH_KEY (h, i);
-      if (!EQ (k, Qunbound))
-        {
-          Lisp_Object val = HASH_VALUE (h, i);
-          struct Lisp_Native_Comp_Unit *cu = XNATIVE_COMP_UNIT (val);
-          dispose_comp_unit (cu, false);
-        }
+      dispose_comp_unit (all_loaded_comp_units.mem[i], false);
     }
 }
 
@@ -4227,22 +4247,26 @@ finish_delayed_disposal_of_comp_units (void)
       xfree (item);
     }
 }
-#endif
 
 /* This function puts the compilation unit in the
-  `all_loaded_comp_units_h` hashmap.  */
-static void
-register_native_comp_unit (Lisp_Object comp_u)
+  `all_loaded_comp_units` list.  */
+void
+register_native_comp_unit (struct Lisp_Native_Comp_Unit * comp_u)
 {
-#ifdef WINDOWSNT
-  /* We have to do this since we can't use `gensym'. This function is
-     called early when loading a dump file and subr.el may not have
-     been loaded yet.  */
-  static intmax_t count;
+  /*  Grow the array if necessary.  */
+  if (all_loaded_comp_units.size + 1 > all_loaded_comp_units.capacity)
+    {
+      all_loaded_comp_units.capacity = max (1,
+                                            2 * all_loaded_comp_units.capacity);
+      all_loaded_comp_units.mem
+        = xrealloc (all_loaded_comp_units.mem,
+                    all_loaded_comp_units.capacity
+                    * sizeof (struct Lisp_Native_Comp_Unit *));
+    }
 
-  Fputhash (make_int (count++), comp_u, all_loaded_comp_units_h);
-#endif
+  all_loaded_comp_units.mem[all_loaded_comp_units.size++] = comp_u;
 }
+#endif
 
 /* This function disposes compilation units.  It is called during the GC sweep
    stage and when Emacs is closing.
@@ -4257,6 +4281,7 @@ dispose_comp_unit (struct Lisp_Native_Comp_Unit *comp_handle, bool delay)
   eassert (comp_handle->handle);
   dynlib_close (comp_handle->handle);
 #ifdef WINDOWSNT
+  loaded_comp_units_remove (comp_handle);
   if (!delay)
     {
       Lisp_Object dirname = internal_condition_case_1 (
@@ -4501,12 +4526,6 @@ load_comp_unit (struct Lisp_Native_Comp_Unit *comp_u, bool loading_dump,
       d_vec_len = XFIXNUM (Flength (comp_u->data_impure_vec));
       for (EMACS_INT i = 0; i < d_vec_len; i++)
 	data_imp_relocs[i] = AREF (comp_u->data_impure_vec, i);
-
-      /* If we register them while dumping we will get some entries in
-	 the hash table that will be duplicated when pdumper calls
-	 load_comp_unit.  */
-      if (!will_dump_p ())
-	register_native_comp_unit (comp_u_lisp_obj);
     }
 
   if (!loading_dump)
@@ -4818,11 +4837,6 @@ syms_of_comp (void)
   staticpro (&delayed_sources);
   delayed_sources = Qnil;
 
-#ifdef WINDOWSNT
-  staticpro (&all_loaded_comp_units_h);
-  all_loaded_comp_units_h = CALLN (Fmake_hash_table, QCweakness, Qvalue);
-#endif
-
   DEFVAR_LISP ("comp-ctxt", Vcomp_ctxt,
 	       doc: /* The compiler context.  */);
   Vcomp_ctxt = Qnil;
diff --git a/src/comp.h b/src/comp.h
index 507379bf5e6..e4196d4a5f9 100644
--- a/src/comp.h
+++ b/src/comp.h
@@ -101,6 +101,11 @@ XNATIVE_COMP_UNIT (Lisp_Object a)
 
 extern void clean_package_user_dir_of_old_comp_units (void);
 
+#ifdef WINDOWSNT
+extern void
+register_native_comp_unit (struct Lisp_Native_Comp_Unit * comp_unit);
+#endif
+
 #else /* #ifdef HAVE_NATIVE_COMP */
 
 static inline void
diff --git a/src/lisp.h b/src/lisp.h
index 22cd166c954..f64463afeec 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4764,8 +4764,13 @@ SUBR_NATIVE_COMPILEDP (Lisp_Object a)
 INLINE struct Lisp_Native_Comp_Unit *
 allocate_native_comp_unit (void)
 {
-  return ALLOCATE_ZEROED_PSEUDOVECTOR (struct Lisp_Native_Comp_Unit,
-				       data_impure_vec, PVEC_NATIVE_COMP_UNIT);
+  struct Lisp_Native_Comp_Unit * comp_u
+    = ALLOCATE_ZEROED_PSEUDOVECTOR (struct Lisp_Native_Comp_Unit,
+                                    data_impure_vec, PVEC_NATIVE_COMP_UNIT);
+#ifdef WINDOWSNT
+  register_native_comp_unit (comp_u);
+#endif
+  return comp_u;
 }
 #else
 INLINE bool
-- 
2.25.1.windows.1

bug#41242: Port feature/native-comp to Windows - Reduce the number of files probed when finding a lisp file.

[Andrea Corallo]

Nicolas Bértolo <nicolasbertolo@gmail.com> writes:

> Hi,
>
> I have attached a bugfix for the issue that causes crashes when closing Emacs.
>
> Ideally I would figure out why iterating over  a weak hash-table does not skip
> elements that were already GC'd, but I could not do it. I feel fixing
> this bug is
> very important, so I used a C array to keep the list of native
> compilation units.

Hi Nico,

at a quick look your code to itereate looks correct and very similar to
other we have in the codebase so this is important to understand what is
going on.  Have you tried to run a reproducer like the following?

======
;;; -*- lexical-binding: t; -*-

(setq gc-cons-threshold most-positive-fixnum)
(defun foo ())
(load (native-compile #'foo))

(setq xxx (make-hash-table :weakness 'value))
(puthash 1 (subr-native-comp-unit (symbol-function #'foo)) xxx)

(defun foo ())
(maphash (lambda (k v)
	   (message "%s %s" k v))
	 xxx)

(garbage-collect)
(maphash (lambda (k v)
	   (message "%s %s" k v))
	 xxx)
======

The thing is that 'mark_and_sweep_weak_table_contents' is called after
convetional mark and before conventional sweep.  This will eventually
call 'sweep_weak_table' that sets the Qunbound as key.

Should be relativelly easy to see that the CU is sweeped there, and in
case is not see the reason.

A possibility where I bet is that something goes wrong dumping and
reviving a weak hashtable (like this is not in 'weak_hash_tables').

You can quickly check against that moving the creation of your hash such
that is created at each startup.

Hope it helps

  Andrea

-- 
akrl@sdf.org