Dynamic loading progress

Dynamic loading progress

[Aurélien Aptel]

[-- Attachment #1: Type: text/plain, Size: 1731 bytes --]

Hi,

I've tried to test Dave Love's patch [1] but it's now bitrotten, I was
not able to make it work.

So I've started my own thing (drawn from Dave's initial work):

* Use of libtool (ltdl)
  - add configure option for ltdl
  - update dependency in configure.ac / Makefile.in
  - add HAVE_LTDL test macro
* New Lisp function `load-module' which takes a compiled module file
path and loads it
  - check for plugin_is_GPL_compatible symbol
  - check presence of `init' symbol
  - run it as a void (*) () call (no return value/arg).

At first I was calling the libtool init function (lt_dlinit) in
syms_of_lread() but it made every subsequent call to lt_dlopen
segfault. I think the problem was coming from calling it before
dumping because as soon as I moved it to the body of `load-module' it
started working.

It works for very simple stuff so far. If I put a printf in the module
init function it is called successfully. But as soon as I try to use
DEFUN, gcc complains:

In file included from fmod.c:17:0:
fmod.c:30:16: error: ‘Ffmod’ undeclared here (not in a function)
 DEFUN ("fmod", Ffmod, Sfmod, 2, 2, 0,
                ^
emacs-master/src/lisp.h:2688:26: note: in definition of macro ‘DEFUN’
        { .a ## maxargs = fnname },     \
                          ^

So I'm stuck here for now... Any help/feedback welcome :)

I've attached a patch and an example module (compilation/loading
instructions in comments). ltdl is enabled by default in the configure
script, you just need libtool in order to try it. I know it is not as
polished as Dave's patch, I'm just trying to get it to work before
making it clean.

1: http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

[-- Attachment #2: dynamic-loading.diff --]
[-- Type: text/plain, Size: 4552 bytes --]

diff -ru old/configure.ac new/configure.ac
--- old/configure.ac	2014-07-01 16:06:11.000000000 +0200
+++ new/configure.ac	2014-07-01 19:28:08.030520500 +0200
@@ -323,6 +323,8 @@
 OPTION_DEFAULT_ON([selinux],[don't compile with SELinux support])
 OPTION_DEFAULT_ON([gnutls],[don't use -lgnutls for SSL/TLS support])
 OPTION_DEFAULT_ON([zlib],[don't compile with zlib decompression support])
+OPTION_DEFAULT_ON([ltdl], [don't compile with dynamic module loading support])
+

 AC_ARG_WITH([file-notification],[AS_HELP_STRING([--with-file-notification=LIB],
  [use a file notification library (LIB one of: yes, gfile, inotify, w32, no)])],
@@ -3126,6 +3128,18 @@
 fi
 AC_SUBST(LIBZ)

+HAVE_LTDL=no
+LIBLTDL=
+if test "${with_ltdl}" != "no"; then
+  AC_CHECK_HEADER(ltdl.h, HAVE_LTDL=yes, HAVE_LTDL=no)
+  AC_CHECK_LIB(ltdl, lt_dlopen, HAVE_LTDL=yes, HAVE_LTDL=no)
+fi
+if test "${HAVE_LTDL}" = "yes"; then
+  AC_DEFINE(HAVE_LTDL, 1, [Define to 1 if you have the ltdl library (-lltdl).])
+  LIBLTDL=-lltdl
+fi
+AC_SUBST(LIBLTDL)
+
 ### Use -lpng if available, unless `--with-png=no'.
 HAVE_PNG=no
 LIBPNG=
@@ -4989,7 +5003,7 @@
 emacs_config_features=
 for opt in XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS \
   GCONF GSETTINGS NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT \
-  LIBOTF XFT ZLIB; do
+  LIBOTF XFT ZLIB LTDL; do

     case $opt in
       NOTIFY|ACL) eval val=\${${opt}_SUMMARY} ;;
@@ -5028,6 +5042,7 @@
 echo "  Does Emacs use -lotf?                                   ${HAVE_LIBOTF}"
 echo "  Does Emacs use -lxft?                                   ${HAVE_XFT}"
 echo "  Does Emacs directly use zlib?                           ${HAVE_ZLIB}"
+echo "  Does Emacs use -lltdl?                                  ${HAVE_LTDL}"

 echo "  Does Emacs use toolkit scroll bars?                     ${USE_TOOLKIT_SCROLL_BARS}"
 echo
diff -ru old/src/lread.c new/src/lread.c
--- old/src/lread.c	2014-07-01 16:06:11.000000000 +0200
+++ new/src/lread.c	2014-07-02 22:06:58.865218515 +0200
@@ -1,3 +1,4 @@
+
 /* Lisp parsing and input streams.

 Copyright (C) 1985-1989, 1993-1995, 1997-2014 Free Software Foundation,
@@ -64,6 +65,10 @@
 #define file_tell ftell
 #endif

+#ifdef HAVE_LTDL
+#include <ltdl.h>
+#endif
+
 /* Hash table read constants.  */
 static Lisp_Object Qhash_table, Qdata;
 static Lisp_Object Qtest, Qsize;
@@ -999,6 +1004,49 @@
   return Fnreverse (lst);
 }

+DEFUN ("load-module", Fload_module, Sload_module, 1, 1, 0,
+       doc: /* Dymamically load a compiled module. */)
+  (Lisp_Object file)
+{
+#ifdef HAVE_LTDL
+    static int lt_init_done = 0;
+    lt_dlhandle handle;
+    void (*module_init) ();
+    void *gpl_sym;
+
+    if (!lt_init_done)
+      {
+        int ret = lt_dlinit ();
+        if (ret)
+          {
+            const char* s = lt_dlerror ();
+            error ("ltdl init fail: %s", s);
+          }
+        lt_init_done = 1;
+      }
+
+    CHECK_STRING (file);
+
+    handle = lt_dlopen (SDATA (file));
+    if (!handle)
+      error ("Cannot load file %s", SDATA (file));
+
+    gpl_sym = lt_dlsym (handle, "plugin_is_GPL_compatible");
+    if (!gpl_sym)
+      error ("Module %s is not GPL compatible", SDATA (file));
+
+    module_init = (void (*) ()) lt_dlsym (handle, "init");
+    if (!module_init)
+      error ("Module %s does not have an init function.", SDATA (file));
+
+    module_init ();
+
+    return Qt;
+#else
+    return Qnil;
+#endif
+}
+
 DEFUN ("load", Fload, Sload, 1, 5, 0,
        doc: /* Execute a file of Lisp code named FILE.
 First try FILE with `.elc' appended, then try with `.el',
@@ -4486,6 +4534,7 @@
   defsubr (&Sget_file_char);
   defsubr (&Smapatoms);
   defsubr (&Slocate_file_internal);
+  defsubr (&Sload_module);

   DEFVAR_LISP ("obarray", Vobarray,
 	       doc: /* Symbol table for use by `intern' and `read'.
Only in new/src: lread.o
Only in new/src: macros.o
Only in new/src: Makefile
diff -ru old/src/Makefile.in new/src/Makefile.in
--- old/src/Makefile.in	2014-07-01 16:06:11.000000000 +0200
+++ new/src/Makefile.in	2014-07-01 19:35:02.941344222 +0200
@@ -225,6 +225,8 @@

 LIBZ = @LIBZ@

+LIBLTDL = @LIBLTDL@
+
 XRANDR_LIBS = @XRANDR_LIBS@
 XRANDR_CFLAGS = @XRANDR_CFLAGS@

@@ -408,7 +410,7 @@
    $(LIBS_TERMCAP) $(GETLOADAVG_LIBS) $(SETTINGS_LIBS) $(LIBSELINUX_LIBS) \
    $(FREETYPE_LIBS) $(FONTCONFIG_LIBS) $(LIBOTF_LIBS) $(M17N_FLT_LIBS) \
    $(LIBGNUTLS_LIBS) $(LIB_PTHREAD) \
-   $(GFILENOTIFY_LIBS) $(LIB_MATH) $(LIBZ)
+   $(GFILENOTIFY_LIBS) $(LIB_MATH) $(LIBZ) $(LIBLTDL)

 all: emacs$(EXEEXT) $(OTHER_FILES)
 .PHONY: all

[-- Attachment #3: fmod.c --]
[-- Type: text/x-csrc, Size: 818 bytes --]

/*
  example module

  compilation:

    $ gcc -Wall -Iemacs/src -Iemacs/lib -fPIC -c fmod.c
    $ gcc -shared -o fmod.so fmod.o

  loading:

    $ ./emacs -Q --batch --eval '(load-module "../../fmod.so")'
    hello world

*/

#include <config.h>
#include <lisp.h>

#include <math.h>
#include <stdio.h>

/* emacs checks for this symbol before running the module */
int plugin_is_GPL_compatible;

/* XXX: DEFUN doesn't work yet... :( */

#if 0

/* define a new lisp function */
DEFUN ("fmod", Ffmod, Sfmod, 2, 2, 0,
       doc: /* Returns the floating-point remainder of NUMER/DENOM */)
     (Lisp_Object numer, Lisp_Object denom)
{
  return make_float (fmod (extract_float (numer), extract_float (denom)));
}

#endif

/* entry point of the module */
void init ()
{
  // defsubr (&Sfmod);
  printf ("hello world\n");
}

Re: Dynamic loading progress

[Andreas Schwab]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> It works for very simple stuff so far. If I put a printf in the module
> init function it is called successfully. But as soon as I try to use
> DEFUN, gcc complains:
>
> In file included from fmod.c:17:0:
> fmod.c:30:16: error: ‘Ffmod’ undeclared here (not in a function)
>  DEFUN ("fmod", Ffmod, Sfmod, 2, 2, 0,
>                 ^
> emacs-master/src/lisp.h:2688:26: note: in definition of macro ‘DEFUN’
>         { .a ## maxargs = fnname },     \
>                           ^
>
> So I'm stuck here for now... Any help/feedback welcome :)

You need to declare Ffmod with EXFUN.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Dynamic loading progress

[Aurélien Aptel]

[-- Attachment #1: Type: text/plain, Size: 1965 bytes --]

On Wed, Jul 2, 2014 at 11:33 PM, Andreas Schwab <schwab@linux-m68k.org> wrote:
> You need to declare Ffmod with EXFUN.

Thanks, it almost worked. In order to use emacs symbols from a
dynamically loaded module I had to add a linker option to export the
symbols from emacs. The option is -Wl,--export-dynamic. (updated in
configure.ac)

I then ran into another problem: the module is loaded correctly but
emacs_abort() is called inside defsubr(). The actual call was in
intern_c_string_1 where I made the following change:

 Lisp_Object
 intern_c_string_1 (const char *str, ptrdiff_t len)
 {
   Lisp_Object obarray = check_obarray (Vobarray);
   Lisp_Object tem = oblookup (obarray, str, len, len);

   if (SYMBOLP (tem))
     return tem;

   if (NILP (Vpurify_flag))
-    /* Creating a non-pure string from a string literal not
-       implemented yet.  We could just use make_string here and live
-       with the extra copy.  */
-    emacs_abort ();
+    return Fintern (make_string (str, len), obarray);

   return Fintern (make_pure_c_string (str, len), obarray);
 }

I'm not familiar with all the dumping intricacies but judging from the
comment this change seems harmless (Can someone confirm this?).
I've made a modules directory with the sample module and a Makefile
which has to be run separately for now.

Here's the whole procedure the test it for anyone interested:

# get a recent copy of emacs sources
$ wget -O e.zip https://github.com/mirrors/emacs/archive/master.zip
$ unzip e.zip

# apply attached diff
$ cd emacs-master
$ patch -p1 < path/to/dynamic-loading-2.diff

# compile
$ ./autogen.sh && ./configure && make -j8

# compile & test module
$ make -C modules
$ ./src/emacs -Q --batch --eval '(progn (load-module
"modules/fmod.so") (message "%f" (fmod 2.4 2)))'
0.400000

Pretty cool if you ask me :)

The next steps are:
- handle doc strings
- handle features related stuff (provide, require, run-after-load hooks, ...)
- making it portable
- ?

[-- Attachment #2: dynamic-loading-2.diff --]
[-- Type: text/plain, Size: 6184 bytes --]

diff --git a/configure.ac b/configure.ac
index 04c75e3..2614f46 100644
--- a/configure.ac
+++ b/configure.ac
@@ -323,6 +323,8 @@ OPTION_DEFAULT_ON([gsettings],[don't compile with GSettings support])
 OPTION_DEFAULT_ON([selinux],[don't compile with SELinux support])
 OPTION_DEFAULT_ON([gnutls],[don't use -lgnutls for SSL/TLS support])
 OPTION_DEFAULT_ON([zlib],[don't compile with zlib decompression support])
+OPTION_DEFAULT_ON([ltdl], [don't compile with dynamic module loading support])
+
 
 AC_ARG_WITH([file-notification],[AS_HELP_STRING([--with-file-notification=LIB],
  [use a file notification library (LIB one of: yes, gfile, inotify, w32, no)])],
@@ -3126,6 +3128,18 @@ if test "${HAVE_ZLIB}" = "yes"; then
 fi
 AC_SUBST(LIBZ)
 
+HAVE_LTDL=no
+LIBLTDL=
+if test "${with_ltdl}" != "no"; then
+  AC_CHECK_HEADER(ltdl.h, HAVE_LTDL=yes, HAVE_LTDL=no)
+  AC_CHECK_LIB(ltdl, lt_dlopen, HAVE_LTDL=yes, HAVE_LTDL=no)
+fi
+if test "${HAVE_LTDL}" = "yes"; then
+  AC_DEFINE(HAVE_LTDL, 1, [Define to 1 if you have the ltdl library (-lltdl).])
+  LIBLTDL="-lltdl -Wl,--export-dynamic"
+fi
+AC_SUBST(LIBLTDL)
+
 ### Use -lpng if available, unless `--with-png=no'.
 HAVE_PNG=no
 LIBPNG=
@@ -4989,7 +5003,7 @@ optsep=
 emacs_config_features=
 for opt in XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS \
   GCONF GSETTINGS NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT \
-  LIBOTF XFT ZLIB; do
+  LIBOTF XFT ZLIB LTDL; do
 
     case $opt in
       NOTIFY|ACL) eval val=\${${opt}_SUMMARY} ;;
@@ -5028,6 +5042,7 @@ echo "  Does Emacs use -lm17n-flt?                              ${HAVE_M17N_FLT}
 echo "  Does Emacs use -lotf?                                   ${HAVE_LIBOTF}"
 echo "  Does Emacs use -lxft?                                   ${HAVE_XFT}"
 echo "  Does Emacs directly use zlib?                           ${HAVE_ZLIB}"
+echo "  Does Emacs use -lltdl?                                  ${HAVE_LTDL}"
 
 echo "  Does Emacs use toolkit scroll bars?                     ${USE_TOOLKIT_SCROLL_BARS}"
 echo
diff --git a/modules/Makefile b/modules/Makefile
new file mode 100644
index 0000000..52fb876
--- /dev/null
+++ b/modules/Makefile
@@ -0,0 +1,8 @@
+
+all: fmod.so
+
+%.so: %.o
+	gcc -shared -o $@ $<
+
+%.o: %.c
+	gcc -ggdb3 -Wall -I../src -I../lib -fPIC -c $<
diff --git a/modules/fmod.c b/modules/fmod.c
new file mode 100644
index 0000000..2fdd01e
--- /dev/null
+++ b/modules/fmod.c
@@ -0,0 +1,25 @@
+#include <config.h>
+#include <lisp.h>
+
+#include <math.h>
+
+/* emacs checks for this symbol before running the module */
+
+int plugin_is_GPL_compatible;
+
+/* define a new lisp function */
+
+EXFUN (Ffmod, 2);
+DEFUN ("fmod", Ffmod, Sfmod, 2, 2, 0,
+       doc: "Returns the floating-point remainder of NUMER/DENOM")
+  (Lisp_Object numer, Lisp_Object denom)
+{
+  return make_float (fmod (extract_float (numer), extract_float (denom)));
+}
+
+/* entry point of the module */
+
+void init ()
+{
+  defsubr (&Sfmod);
+}
diff --git a/src/Makefile.in b/src/Makefile.in
index a13f7b8..780245a 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -225,6 +225,8 @@ LIBXML2_CFLAGS = @LIBXML2_CFLAGS@
 
 LIBZ = @LIBZ@
 
+LIBLTDL = @LIBLTDL@
+
 XRANDR_LIBS = @XRANDR_LIBS@
 XRANDR_CFLAGS = @XRANDR_CFLAGS@
 
@@ -408,7 +410,7 @@ LIBES = $(LIBS) $(W32_LIBS) $(LIBS_GNUSTEP) $(LIBX_BASE) $(LIBIMAGE) \
    $(LIBS_TERMCAP) $(GETLOADAVG_LIBS) $(SETTINGS_LIBS) $(LIBSELINUX_LIBS) \
    $(FREETYPE_LIBS) $(FONTCONFIG_LIBS) $(LIBOTF_LIBS) $(M17N_FLT_LIBS) \
    $(LIBGNUTLS_LIBS) $(LIB_PTHREAD) \
-   $(GFILENOTIFY_LIBS) $(LIB_MATH) $(LIBZ)
+   $(GFILENOTIFY_LIBS) $(LIB_MATH) $(LIBZ) $(LIBLTDL)
 
 all: emacs$(EXEEXT) $(OTHER_FILES)
 .PHONY: all
diff --git a/src/lread.c b/src/lread.c
index f252993..e431be6 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1,3 +1,4 @@
+
 /* Lisp parsing and input streams.
 
 Copyright (C) 1985-1989, 1993-1995, 1997-2014 Free Software Foundation,
@@ -64,6 +65,10 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #define file_tell ftell
 #endif
 
+#ifdef HAVE_LTDL
+#include <ltdl.h>
+#endif
+
 /* Hash table read constants.  */
 static Lisp_Object Qhash_table, Qdata;
 static Lisp_Object Qtest, Qsize;
@@ -999,6 +1004,49 @@ This uses the variables `load-suffixes' and `load-file-rep-suffixes'.  */)
   return Fnreverse (lst);
 }
 
+DEFUN ("load-module", Fload_module, Sload_module, 1, 1, 0,
+       doc: /* Dymamically load a compiled module. */)
+  (Lisp_Object file)
+{
+#ifdef HAVE_LTDL
+    static int lt_init_done = 0;
+    lt_dlhandle handle;
+    void (*module_init) ();
+    void *gpl_sym;
+
+    if (!lt_init_done)
+      {
+        int ret = lt_dlinit ();
+        if (ret)
+          {
+            const char* s = lt_dlerror ();
+            error ("ltdl init fail: %s", s);
+          }
+        lt_init_done = 1;
+      }
+
+    CHECK_STRING (file);
+
+    handle = lt_dlopen (SDATA (file));
+    if (!handle)
+      error ("Cannot load file %s", SDATA (file));
+
+    gpl_sym = lt_dlsym (handle, "plugin_is_GPL_compatible");
+    if (!gpl_sym)
+      error ("Module %s is not GPL compatible", SDATA (file));
+
+    module_init = (void (*) ()) lt_dlsym (handle, "init");
+    if (!module_init)
+      error ("Module %s does not have an init function.", SDATA (file));
+
+    module_init ();
+
+    return Qt;
+#else
+    return Qnil;
+#endif
+}
+
 DEFUN ("load", Fload, Sload, 1, 5, 0,
        doc: /* Execute a file of Lisp code named FILE.
 First try FILE with `.elc' appended, then try with `.el',
@@ -3805,10 +3853,7 @@ intern_c_string_1 (const char *str, ptrdiff_t len)
     return tem;
 
   if (NILP (Vpurify_flag))
-    /* Creating a non-pure string from a string literal not
-       implemented yet.  We could just use make_string here and live
-       with the extra copy.  */
-    emacs_abort ();
+    return Fintern (make_string (str, len), obarray);
 
   return Fintern (make_pure_c_string (str, len), obarray);
 }
@@ -4486,6 +4531,7 @@ syms_of_lread (void)
   defsubr (&Sget_file_char);
   defsubr (&Smapatoms);
   defsubr (&Slocate_file_internal);
+  defsubr (&Sload_module);
 
   DEFVAR_LISP ("obarray", Vobarray,
 	       doc: /* Symbol table for use by `intern' and `read'.

Re: Dynamic loading progress

[Dmitry Antipov]

On 07/03/2014 09:09 PM, Aurélien Aptel wrote:

> The next steps are:
> - handle doc strings
> - handle features related stuff (provide, require, run-after-load hooks, ...)
> - making it portable
> - ?

IMHO unloading should be the first item in this list.
(I'm curious whether we have a method to undo defsubr).

And lread.c is about reading input streams, so it's
better to move module-related stuff in fns.c.

Dmitry

Re: Dynamic loading progress

[Stefan Monnier]

> IMHO unloading should be the first item in this list.

Unloading of Elisp packages is barely supported, so unloading of FFI
packages is not very high priority for me.


        Stefan

Re: Dynamic loading progress

[Stefan Monnier]

> At first I was calling the libtool init function (lt_dlinit) in
> syms_of_lread() but it made every subsequent call to lt_dlopen
> segfault. I think the problem was coming from calling it before
> dumping because as soon as I moved it to the body of `load-module' it
> started working.

It should maybe be called in an init_* function rather than
a syms_of_* function: the difference is that syms_of_* functions are
called only once, before dumping, whereas the init_* functions are
re-executed every time Emacs is started (i.e. before and after the
dump).

But calling it from Fload_module is probably fine as well (and maybe
even better).

> -    /* Creating a non-pure string from a string literal not
> -       implemented yet.  We could just use make_string here and live
> -       with the extra copy.  */
> -    emacs_abort ();
> +    return Fintern (make_string (str, len), obarray);

That's OK, tho you might keep a comment about the need to make
a duplicate copy of the string because a heap-allocated Lisp_String
can't point to a string literal.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

I've focused on the docstring part these last few days.

Dave's patch was using Emacs' make-docfile utility to generate a .doc
file (from a .c file) per module which was loaded along the module by
calling its own version of snarf_documentation that skipped some
checks.

I've tried to do the same thing but there are several problems which I
suspect where already there in Dave's patch.

The doc member of a subr struct is not a pointer to the doc string but
an offset in the build global doc file (Vfile_doc_name aka etc/DOC).
This saves memory but since we now have multiple doc files it doesn't
work. If you update Vfile_doc_name to the module's doc file,
Fdocumentation will only work on this module's functions.

We could:
- switch back to storing the doc string in memory for modules
- make an alist or something to map a function name to a doc file
- <your alternative solution>

Re: Dynamic loading progress

[Stefan Monnier]

> The doc member of a subr struct is not a pointer to the doc string but
> an offset in the build global doc file (Vfile_doc_name aka etc/DOC).
> This saves memory but since we now have multiple doc files it doesn't
> work. If you update Vfile_doc_name to the module's doc file,
> Fdocumentation will only work on this module's functions.

The `doc' field could simply be changed to be of type Lisp_Object and
then follow the usual representation used for lambda expressions
and variables where it can be:
- a single integer: an offset into the one true DOC file.
- a cons (FILE . OFFSET): says that the doc can be found in FILE starting
  at OFFSET.
- a string.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

I've made a github repo of my work so far:
https://github.com/aaptel/emacs-dynamic-module

On Mon, Jul 7, 2014 at 3:19 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> The `doc' field could simply be changed to be of type Lisp_Object and
> then follow the usual representation used for lambda expressions

I've followed your advice.

a) I've updated the type of the doc field. I'm initializing it to 0 in
the DEFUN macro and I set it to Qnil in defsubr(). I'm not
initializing the field directly to Qnil because the docs reads:

  You must not use C initializers for static or global variables unless
  the variables are never written once Emacs is dumped. These variables
  with initializers are allocated in an area of memory that becomes
  read-only (on certain operating systems) as a result of dumping
  Emacs. See Pure Storage.

Instead of duplicating the Fsnarf_documentation code for handling
modules I've added a boolean 'module' parameter. If the function
snarfs a module documentation, don't overwrite the global DOC file
path and store a (FILE . OFFSET) cons instead of a simple offset. This
part works correctly.

b) Now, when fetching a function docstring with Fdocumentation, I
check the types and let the existing get_doc_string() handle the field
itself.

It should work but I've ran into a strange problem: the doc field is
set correctly in step a) but the field becomes either an invalid lisp
object or garbage at step b).

I've added debug print statement (using safe_debug_print):

# this is after calling load-module
---
fmod
("/home/knarf/prog/c/emacs/dyn/modules/fmod.doc" . -16)
---

---
fmod-test1
("/home/knarf/prog/c/emacs/dyn/modules/fmod.doc" . -98)
---

---
fmod-test2
("/home/knarf/prog/c/emacs/dyn/modules/fmod.doc" . -125)
---

---
fmod-test3
("/home/knarf/prog/c/emacs/dyn/modules/fmod.doc" . -152)
---

# now calling (documentation 'fmod...)

--Fdocumentation
fmod
#<INVALID_LISP_OBJECT 0x0322a986>
---

--Fdocumentation
fmod-test1
#<INVALID_LISP_OBJECT 0x0322a996>
---

--Fdocumentation
fmod-test2
((keymap #^[nil nil keymap
#^^[3 0 set-mark-comma...
# [garbage]
---

It suspect it has to do with garbage collecting since I have not done
anything special to protect the doc field. I'm not sure how to
proceed.

Re: Dynamic loading progress

[Stefan Monnier]

> It should work but I've ran into a strange problem: the doc field is
> set correctly in step a) but the field becomes either an invalid lisp
> object or garbage at step b).
[...]
> It suspect it has to do with garbage collecting since I have not done
> anything special to protect the doc field. I'm not sure how to
> proceed.

Right.  You need to tell the GC to trace this new field.

Look at the mark_object function in alloc.c.

You could change its handling of PVEC_SUBR such that it calls itself
recursively on the `doc' field and then make sure the VECTOR_MARKED_P bit
gets reset to false in gc_sweep.

Or maybe force those `doc' references to be kept alive by adding
globally reachable references to them (e.g. have a global list of all
the cons cells you ever added to such a `doc' field, and then
`staticpro' that list).


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Jul 10, 2014 at 12:18 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> You could change its handling of PVEC_SUBR such that it calls itself
> recursively on the `doc' field and then make sure the VECTOR_MARKED_P bit
> gets reset to false in gc_sweep.

I understand that marking a symbol which is bound to a subr means we
have to mark the subr i.e. the doc field since we can't free a subr.
But why do we need to do something else? The doc string being either a
cons, an int or a string it will be automatically handled in the
respective sweep_xxx functions. There are no global list of subr that
can be swept AFAIK.

I've only made the mark_object() recursive call. Getting a module
documentation still doesn't work but at least the conses are valid
now.

> Or maybe force those `doc' references to be kept alive by adding
> globally reachable references to them (e.g. have a global list of all
> the cons cells you ever added to such a `doc' field, and then
> `staticpro' that list).

The first solution is better IMHO, it's more intrusive but it limits
the scope of the problem to the GC.

Re: Dynamic loading progress

[Stefan Monnier]

>> You could change its handling of PVEC_SUBR such that it calls itself
>> recursively on the `doc' field and then make sure the VECTOR_MARKED_P bit
>> gets reset to false in gc_sweep.
> I understand that marking a symbol which is bound to a subr means we
> have to mark the subr i.e. the doc field since we can't free a subr.
> But why do we need to do something else? The doc string being either a
> cons, an int or a string it will be automatically handled in the
> respective sweep_xxx functions. There are no global list of subr that
> can be swept AFAIK.

The issue is that when we get a vectorlike object, the first thing we
want to do is to check the VECTOR_MARKED_P and stop right there if it's
already marked.

So in order to look inside the Subrs, we'd either have to check SUBRP
before checking VECTOR_MARKED_P (thus slowing doing GC for every one),
or we'd have to make sure that the SUBRP that are dyn-loaded have
a VECTOR_MARKED_P that says false (either always or at least a the
beginning of the GC).

> The first solution is better IMHO, it's more intrusive but it limits
> the scope of the problem to the GC.

I tend to agree.  Maybe resetting VECTOR_MARKED_P on Subrs to false
directly from mark_object (and only if the `doc' field is non-trivial)
is the easiest solution.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

Sorry, I still don't get it.

On Thu, Jul 10, 2014 at 8:04 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> The issue is that when we get a vectorlike object, the first thing we
> want to do is to check the VECTOR_MARKED_P and stop right there if it's
> already marked.

You're talking about the marking pass here, right?

in mark_object ():

    case Lisp_Vectorlike:
      {
        register struct Lisp_Vector *ptr = XVECTOR (obj);
        register ptrdiff_t pvectype;

        if (VECTOR_MARKED_P (ptr))
          break;

If it's already marked, then the doc field is already marked too so
breaking here is correct.

Re: Dynamic loading progress

[Stefan Monnier]

> If it's already marked, then the doc field is already marked too so
> breaking here is correct.

But if "it's already marked" there are two possible cases:
- it was just marked earlier in the current GC and indeed then we're
  good to go.
- it was marked in some earlier GC and the mark was not reset since then
  (normally, all marks are reset during gc_sweep, but this is currently
  not done for Subrs, AFAIK).  In that case we're in trouble, because
  the doc needs to be re-marked during this GC, otherwise it might
  get reclaimed.


        Stefan

Re: Dynamic loading progress

[Ted Zlatanov]

On Thu, 3 Jul 2014 19:09:46 +0200 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> Pretty cool if you ask me :)

It's absolutely wonderful.

AA> The next steps are:
AA> - handle doc strings
AA> - handle features related stuff (provide, require, run-after-load hooks, ...)
AA> - making it portable
AA> - ?

I am assuming that structurally, your code is OK with all the
developers, because we've had no protests :)

I'd love to get a version of this in a branch and try it for JSON and
YAML parsing library integration. Is it ready for that kind of abuse?

Portability is not a big deal IMO, it's OK to start with GNU/Linux
support.

We previously discussed signed modules in addition to unsigned. Can your
patch provide that?

Ted

Re: Dynamic loading progress

[Stefan Monnier]

> I'd love to get a version of this in a branch and try it for JSON and
> YAML parsing library integration. Is it ready for that kind of abuse?

AFAIC, it's OK to start installing such experimental code in trunk, as
long as it's properly #ifdef'd.

> We previously discussed signed modules in addition to unsigned.  Can your
> patch provide that?

The FFI itself shouldn't care about signed modules.  This should be done
by the module download&installation code (which should most likely be
in package.el).


        Stefan

Re: Dynamic loading progress

[Ted Zlatanov]

On Wed, 24 Sep 2014 13:27:39 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> We previously discussed signed modules in addition to unsigned.  Can your
>> patch provide that?

SM> The FFI itself shouldn't care about signed modules.  This should be done
SM> by the module download&installation code (which should most likely be
SM> in package.el).

I personally wouldn't make it trivially easy to circumvent in ELisp, but
it was not a blocking question anyhow.  The key is to get the basics
working as you mentioned.

Ted

Re: Dynamic loading progress

[Aurélien Aptel]

Good news everyone!

As of my last commit (7a38270cd) dynamic loading is working pretty
well.

I've written a few toy modules in the "modules" folder. They can be
used as examples and cover various use-case.

- The `fmod' module simply expose the fmod(3) function from the math
  lib.

- The `elisp' module evaluates code written in Lisp from C.

- The `opaque' module show how any C structs/type can be embedded in a
  Lisp_Object.

- The `curl' module can actually be useful as it links against libcurl
  and lets you fetch any http url supported by curl. You can use the
  same curl handle for multiple request in order to reuse the
  connection when possible (faster!).

* Usage

Dynamic loading can be enabled/disabled via the configure
script. libltdl/ltdl.h is all you need to make it work. I've turned
the option on by default (--with-ltdl).

  # build emacs
  $ git clone https://github.com/aaptel/emacs-dynamic-module.git emacs-dyn
  $ cd emacs-dyn
  $ ./autogen.sh
  $ ./configure
  $ make

  # build all the modules
  $ for i in modules/*; do make -C "$i"; done

  # run emacs! (no init file)
  $ ./src/emacs -Q

Switch to *scratch* to test interactively the curl module:

  (add-to-list 'load-path "path/to/emacs/modules/curl")
  (require 'curl)
  => t

  ;; alternatively you can use
  ;;     (load "path/to/curl/curl.so")
  ;; or
  ;;     (load-module "path/to/curl/curl.so")

  (defvar c)
  (setq c (curl-make)) ;; allocate/init a curl handle
  => #<save-value <pointer 0x1cf56d0> <unused> <unused> <unused>>

  (curl-fetch-url c "http://example.com") ;; make and store request
  => t

  (curl-content c)
  "<!doctype html>
  <html>
  <head>
      <title>Example Domain</title>
  ..."

  (curl-free c) ;; free the curl handle
  => t


I've just tested the build procedure above before posting and *of course*
for some reason a clean emacs build fail on something completely
unrelated. Even at my first commit where I didn't make any change yet.

  make[1]: *** No rule to make target 'ctags.c', needed by 'ctags'.  Stop.

I suppose it comes from my setup? Gahh.. Anyway...

* Implementation details and potential issues

** Loading

The actual loading is done in Fload_module in lread.c. It is called
from Fload when appropriate. See my previous emails.

When dynamic modules are enabled, the `load-suffixes' variable has the
additional .so suffix. This variable is used by `load' which in turn
calls openp() to do the actual search in `load-path'.

Adapting `load' to support modules made `require' and `provide' work
with no additional work.

Nothing is preventing a module from being loaded several times apart
from the `features' mechanism. This might be problematic.

** Documentation

I have updated several documentation related functions in order to
support multiple DOCFILE (each module has its own). See my previous
emails.

The xref on function documentation (go to definition) is currently
broken. I have not looked into it (issue #6).

I have not fixed the issue (#1) Stefan mentioned earlier about a
memory leak in the GC on the subr doc field because I *still* don't
get it :( Any help is welcomed.

** Opaque user types

Adding a new proper type is too much work for a module. You need to
provide a reader, a printer, add a bunch of enums, etc.

I've reused the Save_Value type to store arbitrary pointers. The
object hierarchy is as follows:

Lisp_Object > Lisp_Misc > Lisp_Save_Value > void* pointer

A major problem with this is that you can't make type predicates
e.g. `curl-handle-p'. A solution would be to make a new generic user
type composed of a user pointer and a cookie or some sort of ID we can
test. The space has to be large enough to limit collisions.

** Resource management

An allocated curl handle has to be explicitly freed with
`curl-free'. Could we extend the GC to provide some sort of destructor
mechanism? Just a thought.

Also, Lisp_Objects contained in a user type are not protected from the
GC. This can be a problem.

** Native type size

EMACS_INT_MAX is not always equal to INT_MAX depending on the
architecture. Libraries can often use special value to signal
something (e.g. (unsigned)-1 which is 0xffff..) which might not be fit
in a Lisp type. What can we do about it? Is it really a problem? Emacs
Lisp provides other way to signal errors and a module doesn't have to
expose low-level stuff like this.

** Packaging

Right now each module has its own Makefile and is built separately from
emacs. We need to think of a way to package modules for distribution
that is practical and takes into account what the module needs from
the Emacs build (which configure option should be enabled to build
module xxx?).

** Portability

The .so suffix is hardcoded but via libltdl loading should work on
most systems.

Re: Dynamic loading progress

[Aurélien Aptel]

Just a few things I forgot:

You can find more in commit messages and in the issue tracker.

https://github.com/aaptel/emacs-dynamic-module/commits/master
https://github.com/aaptel/emacs-dynamic-module/issues

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Thu, 9 Oct 2014 19:08:47 +0200
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> 
> As of my last commit (7a38270cd) dynamic loading is working pretty
> well.

Thanks.

>   ;; alternatively you can use
>   ;;     (load "path/to/curl/curl.so")
>   ;; or
>   ;;     (load-module "path/to/curl/curl.so")

The .so extension is platform specific.  Since Emacs tries to keep
platform dependencies out of the Lisp level, how about chopping the
extension, and adding to load-module the necessary code to look for
the extension specific to the underlying platform?

> ** Portability
> 
> The .so suffix is hardcoded but via libltdl loading should work on
> most systems.

Well, how about removing this issue by using lt_dlopenext?

Re: Dynamic loading progress

[Stefan Monnier]

> The .so extension is platform specific.  Since Emacs tries to keep
> platform dependencies out of the Lisp level, how about chopping the
> extension, and adding to load-module the necessary code to look for
> the extension specific to the underlying platform?

I think we can circumvent this problem by changing the suffix we add to
load-suffixes depending on the platform (e.g. based on dlopenext).


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>,
>   emacs-devel@gnu.org
> Date: Thu, 09 Oct 2014 15:42:35 -0400
> 
> > The .so extension is platform specific.  Since Emacs tries to keep
> > platform dependencies out of the Lisp level, how about chopping the
> > extension, and adding to load-module the necessary code to look for
> > the extension specific to the underlying platform?
> 
> I think we can circumvent this problem by changing the suffix we add to
> load-suffixes depending on the platform (e.g. based on dlopenext).

You mean, use foo.so in Lisp and have .so changed to something else?
That's ugly, and we never do that anywhere else.  Why start now?  Just
use "foo" and set up the extensions as appropriate for the platform,
as we do with programs.

Re: Dynamic loading progress

[Stefan Monnier]

>> I think we can circumvent this problem by changing the suffix we add to
>> load-suffixes depending on the platform (e.g. based on dlopenext).
> You mean, use foo.so in Lisp and have .so changed to something else?

No, I means to add ".so" or ".dll" to `load-suffixes' depending on the
host system.


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: aurelien.aptel+emacs@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 09 Oct 2014 20:53:13 -0400
> 
> >> I think we can circumvent this problem by changing the suffix we add to
> >> load-suffixes depending on the platform (e.g. based on dlopenext).
> > You mean, use foo.so in Lisp and have .so changed to something else?
> 
> No, I means to add ".so" or ".dll" to `load-suffixes' depending on the
> host system.

In that case, that's what I was proposing.  I think.

Re: Dynamic loading progress

[Stefan Monnier]

>> >> I think we can circumvent this problem by changing the suffix we add to
>> >> load-suffixes depending on the platform (e.g. based on dlopenext).
>> > You mean, use foo.so in Lisp and have .so changed to something else?
>> No, I means to add ".so" or ".dll" to `load-suffixes' depending on the
>> host system.
> In that case, that's what I was proposing.  I think.

Yes, I think we're in violent agreement,


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Oct 9, 2014 at 7:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> Well, how about removing this issue by using lt_dlopenext?

I use the .so extension to know whether a file is a binary module or
elisp (compiled or source). In order to let libltdl guess & try
various extensions with dlopenext I have to leave `load-suffixes'
alone and try to load the file as a module everytime. If it's actually
a module, continue loading it, else load as elisp. I find this messy.

What Stefan is proposing is to add the plateform specific suffixes in
`load-suffixes' at compilation time. Somehing like this:

/*
   Somwhere in a platform abstraction header (which one?)
   (I don't know the actual system defines)
*/

#ifdef HAVE_LTDL
  #if   defined(WINDOWS)
    #define MODULE_SUFFIX ".dll"
  #elif defined(UNIX)
    #define MODULE_SUFFIX ".so"
  #elif defined(MAC)
    #define MODULE_SUFFIX ".dylib"
  #else
    #error "Modules not supported on this system"
  #endif
#endif

/*
  Now in `load-suffixes' definition
*/

#ifdef HAVE_LTDL
  Vload_suffixes = list3 (build_pure_c_string (MODULE_SUFFIX),
                          build_pure_c_string (".elc"),
                          build_pure_c_string (".el"));
#else
  Vload_suffixes = list2 (build_pure_c_string (".elc"),
                          build_pure_c_string (".el"));
#endif

I'm not familiar enough with every system but having only one
extension per system is maybe limiting. After a quick search I found
that Mac OSX use both .so and .dylib.

Another solution would be to define a new Lisp variable
`load-module-suffixes' with every possible module extensions (system
specific or not) and use that to test if a file is a module or not.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Fri, 10 Oct 2014 17:05:06 +0200
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> On Thu, Oct 9, 2014 at 7:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > Well, how about removing this issue by using lt_dlopenext?
> 
> I use the .so extension to know whether a file is a binary module or
> elisp (compiled or source).

You cannot know that by the extension alone, anyway.  It's unreliable.
So that problem cannot be solved by using the extension explicitly.

> I'm not familiar enough with every system but having only one
> extension per system is maybe limiting.

Emacs can use a list of extensions, not just a single extension.

Re: Dynamic loading progress

[Stefan Monnier]

>> I use the .so extension to know whether a file is a binary module or
>> elisp (compiled or source).
> You cannot know that by the extension alone, anyway.

Of course, we can: if the extension says ".el" or ".elc" we know it's
Elisp (compiled or not), and if it says ".dll", ".so" or something like
that, we know it's a binary module.

> It's unreliable.

If the user wants to use ".so" on an Elisp module, she'll get what she
paid for.  I'm really not worried.

>> I'm not familiar enough with every system but having only one
>> extension per system is maybe limiting.
> Emacs can use a list of extensions, not just a single extension.

Yes, it's really easy to fix.


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>,
>   emacs-devel@gnu.org
> Date: Fri, 10 Oct 2014 14:42:56 -0400
> 
> >> I use the .so extension to know whether a file is a binary module or
> >> elisp (compiled or source).
> > You cannot know that by the extension alone, anyway.
> 
> Of course, we can: if the extension says ".el" or ".elc" we know it's
> Elisp (compiled or not), and if it says ".dll", ".so" or something like
> that, we know it's a binary module.

That's until you bump into a maliciously named file.

> > It's unreliable.
> 
> If the user wants to use ".so" on an Elisp module, she'll get what she
> paid for.  I'm really not worried.

You don't have to, because loading a shared library will validate it
anyway.

Re: Dynamic loading progress

[Stefan Monnier]

>> >> I use the .so extension to know whether a file is a binary module or
>> >> elisp (compiled or source).
>> > You cannot know that by the extension alone, anyway.
>> Of course, we can: if the extension says ".el" or ".elc" we know it's
>> Elisp (compiled or not), and if it says ".dll", ".so" or something like
>> that, we know it's a binary module.
> That's until you bump into a maliciously named file.

I don't think it matters: the same problem can happen by playing with
file's contents rather than file names.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

I'm with Stefan, I will make a new `load-module-suffixes' variable.
Let's not turn this into a bikeshed discussion.

I would like to hear your comments/feedback/ideas on the points I
mentioned in my previous announcement:

- should we provide a destructor mechanism in the GC?
- should we add a new generic lisp type for opaque pointers?
- how should modules be packaged?

Re: Dynamic loading progress

[Stefan Monnier]

> - should we provide a destructor mechanism in the GC?

You mean "finalizers"?  Sounds fine, yes.  The GC code already has code
that does similar things to handle various kinds of "weak" pointers.

> - should we add a new generic lisp type for opaque pointers?

Fine by me, yes.

> - how should modules be packaged?

As GNU ELPA packages, of course.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

On Tue, Oct 14, 2014 at 8:11 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> - should we provide a destructor mechanism in the GC?
>
> You mean "finalizers"?  Sounds fine, yes.  The GC code already has code
> that does similar things to handle various kinds of "weak" pointers.

I will look into it, thanks.

>> - should we add a new generic lisp type for opaque pointers?
>
> Fine by me, yes.

Ok. I think this new type needs an unique module identifier field in
order to differentiate one module type from another (per-module type
predicates).
This field can be a random int, it can be computed from the module
struct directly, a short fixed size string (eg the module name +
version), the opaque type size, or a mix of all of these... (I'm
thinking out loud)

>> - how should modules be packaged?
>
> As GNU ELPA packages, of course.

Several headers are needed in order to build a module, and they have
to be the one your current Emacs was built with (eg config.h). I think
this means Emacs should copy those in the system include dir when
installing. Can we generate a config.h from a built emacs binary? Also
the whole thing has to be portable (build commands, paths, ...). Well
it's going to be a nightmare to get right.

In the mean time I wrote a basic YAML parsing module [1] using
libyaml. You can read yaml from a file, a buffer or a string. I've
tried to use direct access to lisp types (string, buffer) when reading
yaml streams to prevent too much copying/type conversion overhead. I
haven't done much error checking yet so it's not very robust but it
works:

(require 'yaml)
(yaml-parse-string "---
invoice: 34843
date   : 2001-01-23
bill-to: &id001
    given  : Chris
    family : Dumars
    address:
        lines: |
            458 Walkman Dr.
            Suite #292
        city    : Royal Oak
        state   : MI
        postal  : 48046
ship-to: *id001
")
=> (#s(hash-table
       data
       ("invoice" "34843"
    "date" "2001-01-23"
    "bill-to" #s(hash-table
             data
             ("given" "Chris"
              "family" "Dumars"
              "address" #s(hash-table
                   data
                   ("lines" "458 Walkman Dr.\nSuite #292\n"
                    "city" "Royal Oak"
                    "state" "MI"
                    "postal" "48046"))))
    "ship-to" #s(hash-table
             data
             ("given" "Chris"
              "family" "Dumars"
              "address" #s(hash-table
                   data
                   ("lines" "458 Walkman Dr.\nSuite #292\n"
                    "city" "Royal Oak"
                    "state" "MI"
                    "postal" "48046")))))))

(I have edited the output to make it more readable)

1: https://github.com/aaptel/emacs-dynamic-module/blob/master/modules/yaml/yaml.c

Re: Dynamic loading progress

[Stefan Monnier]

> Ok. I think this new type needs an unique module identifier field in
> order to differentiate one module type from another (per-module type
> predicates).

It might make more sense to have a "type" field which holds
a Lisp_Object so we can trivially make it unique by using an object that
we just allocated.

>>> - how should modules be packaged?
>> As GNU ELPA packages, of course.
> Several headers are needed in order to build a module, and they have
> to be the one your current Emacs was built with (eg config.h).

Right, those headers need to be provided by Emacs.

> I think this means Emacs should copy those in the system include dir
> when installing.

If we let Emacs run the C compiler, it can trivially add the -I args
to specify where those headers are found.  This way they don't need to
be installed in the system's include dir (which is always a problem for
non-root users, and makes it more tricky to handle multiple installs of
different Emacs versions).

> Can we generate a config.h from a built emacs binary?

I don't think so.  But we can copy the config.h (and whichever other
include file we might need) when installing Emacs.

> Also the whole thing has to be portable (build commands, paths,
> ...).  Well it's going to be a nightmare to get right.

Yup!

> In the mean time I wrote a basic YAML parsing module [1] using
> libyaml.  You can read yaml from a file, a buffer or a string.
> I've tried to use direct access to lisp types (string, buffer) when
> reading yaml streams to prevent too much copying/type conversion
> overhead.  I haven't done much error checking yet so it's not very
> robust but it works:

Yay!


        Stefan

Re: Dynamic loading progress

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> - how should modules be packaged?
>>> As GNU ELPA packages, of course.
>> Several headers are needed in order to build a module, and they have
>> to be the one your current Emacs was built with (eg config.h).
>
> Right, those headers need to be provided by Emacs.
>
>> I think this means Emacs should copy those in the system include dir
>> when installing.
>
> If we let Emacs run the C compiler, it can trivially add the -I args
> to specify where those headers are found.  This way they don't need to
> be installed in the system's include dir (which is always a problem for
> non-root users, and makes it more tricky to handle multiple installs of
> different Emacs versions).

Let's not assume C; I'm planning on writing a module in Ada, with a
Makefile and Ada compiler config files.

One step in the Makefile will be to generate Ada import headers from the
C headers, so I do still need those headers.

Another module I'm planning will have texinfo sources; that needs to run
'makeinfo'.

It would be excellent if 'package install' could run an arbitrary
script, assuming the user has the required tools on PATH. (It would also
be nice if the script gave good error messages about missing tools, but
that's not Emacs's problem).

Emacs could export an environment variable (or more than one) providing
the location of the Emacs C headers.

Perhaps the package meta information could specify whether to run gcc
with -I, or to run an arbitrary build script with environment vars.


Just requiring that a compiler be installed will put off a large number
of users, so a binary distribution would be nice.

For binary distributions, or sources with lots of other dependencies, an
option is to provide the non-elisp code as an OS package thru the
standard OS distribution mechanisms. Debian and Mingw64 packages would
cover a lot of users (although I have some using Mac; I know nothing
about packaging on Mac, except it seems to be harder than anywhere else).

Then the ELPA module can list that as a dependency, and simply tell the
user to install it outside Emacs if the package load fails.

Or the package build script mentioned above could invoke the appropriate
OS install commands.

The OS distribution mechanisms have a lot of support for building
things; let's not reinvent that wheel.

-- 
-- Stephe

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

>   # build emacs
>   $ git clone https://github.com/aaptel/emacs-dynamic-module.git emacs-dyn
>   $ cd emacs-dyn
>   $ ./autogen.sh
>   $ ./configure
>   $ make
>
>   # build all the modules
>   $ for i in modules/*; do make -C "$i"; done

What happened to the 'modules' directory? 

It's no longer in this repository (I just cloned it), and it's not in
the Savannah dynamics-modules* branches either.

-- 
-- Stephe

Re: Dynamic loading progress

[Ted Zlatanov]

On Sat, 27 Dec 2014 15:57:05 -0600 Stephen Leake <stephen_leake@stephe-leake.org> wrote: 

SL> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>> # build emacs
>> $ git clone https://github.com/aaptel/emacs-dynamic-module.git emacs-dyn
>> $ cd emacs-dyn
>> $ ./autogen.sh
>> $ ./configure
>> $ make
>> 
>> # build all the modules
>> $ for i in modules/*; do make -C "$i"; done

SL> What happened to the 'modules' directory? 

SL> It's no longer in this repository (I just cloned it), and it's not in
SL> the Savannah dynamics-modules* branches either.

Aurelien asked for more time to deal with some minor issues, but the
branch dynamic-modules-rc2 definitely has that directory. It just needs
rebasing and testing.

Ted

Re: Dynamic loading progress

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sat, 27 Dec 2014 15:57:05 -0600 Stephen Leake <stephen_leake@stephe-leake.org> wrote: 
>
> SL> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>>> # build emacs
>>> $ git clone https://github.com/aaptel/emacs-dynamic-module.git emacs-dyn
>>> $ cd emacs-dyn
>>> $ ./autogen.sh
>>> $ ./configure
>>> $ make
>>> 
>>> # build all the modules
>>> $ for i in modules/*; do make -C "$i"; done
>
> SL> What happened to the 'modules' directory? 
>
> SL> It's no longer in this repository (I just cloned it), and it's not in
> SL> the Savannah dynamics-modules* branches either.
>
> Aurelien asked for more time to deal with some minor issues, but the
> branch dynamic-modules-rc2 definitely has that directory. 

I had some git misunderstandings trying to checkout a
separate workspace for this branch. I see it now, thanks.

-- 
-- Stephe

Re: Dynamic loading progress

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sat, 27 Dec 2014 15:57:05 -0600 Stephen Leake <stephen_leake@stephe-leake.org> wrote: 
>
> SL> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>>> # build emacs
>>> $ git clone https://github.com/aaptel/emacs-dynamic-module.git emacs-dyn
>>> $ cd emacs-dyn
>>> $ ./autogen.sh
>>> $ ./configure
>>> $ make
>>> 
>>> # build all the modules
>>> $ for i in modules/*; do make -C "$i"; done
>
> SL> What happened to the 'modules' directory? 
>
> SL> It's no longer in this repository (I just cloned it), and it's not in
> SL> the Savannah dynamics-modules* branches either.
>
> Aurelien asked for more time to deal with some minor issues, but the
> branch dynamic-modules-rc2 definitely has that directory. It just needs
> rebasing and testing.

I rebased to master as of a couple days ago. I'm getting a compilation
error:

./temacs --batch --load loadup bootstrap
Loading loadup.el (source)...
Using load-path (c:/Projects/emacs/dynamic-modules-rc2/lisp c:/Projects/emacs/dynamic-modules-rc2/lisp/emacs-lisp c:/Projects/emacs/dynamic-modules-rc2/lisp/language c:/Projects/emacs/dynamic-modules-rc2/lisp/international c:/Projects/emacs/dynamic-modules-rc2/lisp/textmodes c:/Projects/emacs/dynamic-modules-rc2/lisp/vc)
Loading emacs-lisp/byte-run (source)...
Loading emacs-lisp/backquote (source)...
Symbol's function definition is void: defun
Makefile:825: recipe for target 'bootstrap-emacs.exe' failed
make[1]: *** [bootstrap-emacs.exe] Error 127
make[1]: Leaving directory '/c/Projects/emacs/dynamic-modules-rc2/src'
Makefile:398: recipe for target 'src' failed
make: *** [src] Error 2


I got the same error before the rebase.

I'm building on Mingw64; I'll try on Debian.

-- 
-- Stephe

Re: Dynamic loading progress

[Aurélien Aptel]

Hi!

I'm just letting everyone know I'm working on my patch again. I've
started by merging the last 2-3 months commits in master but of course
there are merge conflicts... It's taking more time to resolve than
anticipated.

There's a macro conflict with "Qc". I had to rename Qc to Qc_ in
modules/opaque/opaque.c because globals.h defines Qc to something.

I'm also working on a small test suite since I (and maintainers)
quickly want to know when my code breaks.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sat, 31 Jan 2015 19:57:42 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> There's a macro conflict with "Qc". I had to rename Qc to Qc_ in
> modules/opaque/opaque.c because globals.h defines Qc to something.

Most probably, that's not the right solution.  You should instead
refrain from declaring Qc, since make-doc and globals.h now take care
of that.  See how other Q* symbols are defined and declared.

Thanks for your work.

Re: Dynamic loading progress

[Aurélien Aptel]

On Sat, Jan 31, 2015 at 8:30 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> Most probably, that's not the right solution.  You should instead
> refrain from declaring Qc, since make-doc and globals.h now take care
> of that.  See how other Q* symbols are defined and declared.

You can't expect modules to keep up with core global variables. At
least not for such generic, short variables which shouldn't be global
in the first place. What do you think?

Re: Dynamic loading progress

[Aurélien Aptel]

The next thing I will work on is the new module Lisp_Object type and
the custom destructor mechanism for it in the GC.

Re: Dynamic loading progress

[Stephen J. Turnbull]

Aurélien Aptel writes:

 > You can't expect modules to keep up with core global variables.

No, you can't.  You *can*, however, expect modules to maintain proper
namespace hygiene for *symbols*.  And *must*: this is the Land of
(Emacs) Lisp, where symbols are all global.

 > At least not for such generic, short variables which shouldn't be
 > global in the first place. What do you think?

I think `Fintern' is your friend.

That has worked in XEmacs for the last 15 years.  I don't see why it
won't work in Emacs.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Wed, 4 Feb 2015 22:58:08 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> On Sat, Jan 31, 2015 at 8:30 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > Most probably, that's not the right solution.  You should instead
> > refrain from declaring Qc, since make-doc and globals.h now take care
> > of that.  See how other Q* symbols are defined and declared.
> 
> You can't expect modules to keep up with core global variables. At
> least not for such generic, short variables which shouldn't be global
> in the first place. What do you think?

I thought figuring out how to solve this is part of the job.  Modules
will not really be workable unless this issue is resolved in a way
that doesn't break them with every change in the core.

Re: Dynamic loading progress

[Aurélien Aptel]

Ok, what I've pushed on my branch is now working.

For some reasons, DEFSYM was a no-op in module code so I've replaced
DEFSYM calls with:

    MQfoo = intern ("foo")

Are there any drawbacks?

On Thu, Feb 5, 2015 at 4:50 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> I thought figuring out how to solve this is part of the job.  Modules
> will not really be workable unless this issue is resolved in a way
> that doesn't break them with every change in the core.

I've settled for a M prefix on all global module variables. Well only
symbols right now since that's all I'm using.

The whole reason why I've put them global is because that's what the
core does but as it turns out it's not needed here. The fact that
globals are actually macros exanding to a global struct member is not
desirable here as it prevents normal C lexical scoping.

By the way, there are some conventions that I'd like to understand
like prefixes: V is for lisp variables, F for functions, S for subr, Q
is for symbols (Q as in Quote since S was taken?). Am I missing
something?

One last thing, I need a deep `equal' function that also works
hash-tables (every key and values must be equal) in yaml/test.el to
compare nested lisp objects. Do I need to hand roll this myself?

Re: Dynamic loading progress

[Paul Eggert]

Aurélien Aptel wrote:
> I've replaced
> DEFSYM calls with:
>
>      MQfoo = intern ("foo")
>
> Are there any drawbacks?

It's a bit slower.  Symbols like Qabove are compile-time constants; on my 
platform, for example, Qabove == 8688, and this integer is known at 
compile-time.  This gives C compilers more opportunities for optimization than 
variables like MQfoo do.

> The fact that
> globals are actually macros exanding to a global struct member is not
> desirable here as it prevents normal C lexical scoping.

That sounds like it might be more trouble than it's worth.  These symbols are 
globally unique anyway, in Elisp, so it's a bit clearer to keep them globally 
unique in C as well, rather than attempt to reuse them with C lexical scoping.

Re: Dynamic loading progress

[Stefan Monnier]

> Aurélien Aptel wrote:
>> I've replaced
>> DEFSYM calls with:
>> MQfoo = intern ("foo")
>> Are there any drawbacks?

That looks good.

> It's a bit slower.  Symbols like Qabove are compile-time constants; on my
> platform, for example, Qabove == 8688, and this integer is known at
> compile-time.  This gives C compilers more opportunities for optimization
> than variables like MQfoo do.

But he's talking about code in plugins, i.e. code which needs to be
compiled seaprately from Emacs itself and where the .(s)o file needs to
be re-usable with several different Emacs executables, so the special
optimization as used for Qabove is not an option.


        Stefan

Re: Dynamic loading progress

[Paul Eggert]

Stefan Monnier wrote:
> he's talking about code in plugins, i.e. code which needs to be
> compiled seaprately from Emacs itself and where the .(s)o file needs to
> be re-usable with several different Emacs executables

First, in that case the value of Qabove is known at link-time, and compilers 
should be able to do a better job with link-time constants than with variables.

Second, even plugins can be subject to link-time optimization, if one's linker 
is smart enough, as gold is.

I hadn't considered either of these possibilities when adding the lispsym 
support, so Aurélien's approach of calling 'intern' is the best we can do now. 
However, I suggest packaging those calls into a function ('module_defsym', say?) 
so that its implementation can be improved later, if someone wants to work on that.

One other thing: I don't see the point of prefixing global module symbols with 
'MQ' as opposed to 'Q'.  Users of these symbols shouldn't care whether a module 
defined them, or the core interpreter defined them.

Re: Dynamic loading progress

[Aurélien Aptel]

On Mon, Feb 9, 2015 at 7:26 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> One other thing: I don't see the point of prefixing global module symbols
> with 'MQ' as opposed to 'Q'.  Users of these symbols shouldn't care whether
> a module defined them, or the core interpreter defined them.

Global vars in core end up as macros.
I was using Qc in the opaque module but it broke recently when core
defined it too.
Since it's hard to keep up with what core defines, adding a prefix
prevents the clash.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Mon, 9 Feb 2015 10:58:55 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>, 
> 	Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> I was using Qc in the opaque module but it broke recently when core
> defined it too.

Can you explain why your Qc was globally visible?  Can't you arrange
for it to remain un-exported?

Re: Dynamic loading progress

[Aurélien Aptel]

On Mon, Feb 9, 2015 at 4:45 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> Can you explain why your Qc was globally visible?  Can't you arrange
> for it to remain un-exported?

I thought it was a convention? See one of my previous message. Anyway
it doesn't matter if it's global: Qc expands to a member of a struct.
No matter where I declare Qc in my module, the preprocessor will
expand it into something invalid. I'm not actually sure what it
expands to right now, I'm at work now and I can't test it.

Any existing solutions for the hash-table equality test I was talking
about earlier?

Re: Dynamic loading progress

[Paul Eggert]

Aurélien Aptel wrote:
> Since it's hard to keep up with what core defines, adding a prefix
> prevents the clash.

Ah, OK, that explains things.

Though this raises a related issue: as I understand it currently modules must 
never refer to any Qxxx symbols (other than Qnil), since these symbols may have 
different values in different Emacs implementations.  And this includes any 
macros or inline functions the modules may invoke.  For example, a module must 
never call 'functionp', since 'functionp' is an inline function that refers to Qt.

This is something that needs to be fixed, surely.  And once we have a fix for 
it, perhaps the fix will mean that we don't need that "M" prefix.

Re: Dynamic loading progress

[Stefan Monnier]

I think what this boils down to, is that we need a "emacs.h" or
something like that which modules would include instead of including
lisp.h.


        Stefan


>>>>> "Paul" == Paul Eggert <eggert@cs.ucla.edu> writes:

> Aurélien Aptel wrote:
>> Since it's hard to keep up with what core defines, adding a prefix
>> prevents the clash.

> Ah, OK, that explains things.

> Though this raises a related issue: as I understand it currently modules
> must never refer to any Qxxx symbols (other than Qnil), since these symbols
> may have different values in different Emacs implementations.  And this
> includes any macros or inline functions the modules may invoke.
> For example, a module must never call 'functionp', since 'functionp' is an
> inline function that refers to Qt.

> This is something that needs to be fixed, surely.  And once we have a fix
> for it, perhaps the fix will mean that we don't need that "M" prefix.

Re: Dynamic loading progress

[Paul Eggert]

On 02/10/2015 12:40 PM, Stefan Monnier wrote:
> I think what this boils down to, is that we need a "emacs.h" or
> something like that which modules would include instead of including
> lisp.h.

Yes, I was thinking of something along those lines as well. "emacs.h" 
could, for example, define a symbol EMACS_MODULE that is 1 (whereas it's 
0 for core Emacs code), and then globals.h could conditionally define 
Qxxx to be something that's dynamically linkable but slower if 
EMACS_MODULE is 1, and to be something that's not dynamically linkable 
but faster if EMACS_MODULE is 0.

Re: Dynamic loading progress

[Stefan Monnier]

>> I think what this boils down to, is that we need a "emacs.h" or
>> something like that which modules would include instead of including
>> lisp.h.
> Yes, I was thinking of something along those lines as well. "emacs.h" could,
> for example, define a symbol EMACS_MODULE that is 1 (whereas it's 0 for core
> Emacs code), and then globals.h could conditionally define Qxxx to be
> something that's dynamically linkable but slower if EMACS_MODULE is 1, and
> to be something that's not dynamically linkable but faster if EMACS_MODULE
> is 0.

I was thinking of making emacs.h a "real file" which would contain
a subset of the current lisp.h and then lisp.h would #include it.


        Stefan

Re: Dynamic loading progress

[Ted Zlatanov]

On Tue, 10 Feb 2015 20:45:03 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

SM> I was thinking of making emacs.h a "real file" which would contain
SM> a subset of the current lisp.h and then lisp.h would #include it.

On Wed, 11 Feb 2015 05:26:12 -0800 Paul Eggert <eggert@cs.ucla.edu> wrote: 
PE> you start your module this way:

PE> #define EMACS_MODULE
PE> #include <lisp.h>

PE> or better, we package up an <emacs.h> file the way Stefan suggested.

I'd love to see emacs.h become a real API for modules.  But `#define EMACS_MODULE'
is a good idea regardless, for cleanliness.  I would even define it to
be "1.0" to indicate module version 1 (the initial).

Ted

Re: Dynamic loading progress

[Paul Eggert]

To help make progress on the plugin compatibility issue I installed commit 
65563fd7714271582d5146c09202c0f7a0631fe5 in the master.  A plugin should do this:

#define DEFINE_NONNIL_Q_SYMBOL_MACROS false

before including globals.h either directly or indirectly.  This #define could be 
part of any new emacs.h.

We're not out of the woods yet, I'm afraid.  For example, suppose a new version 
of Emacs adds a new kind of function, and that the implementation of functionp 
changes.  And suppose a plugin is compiled with the old version of Emacs and 
uses the FUNCTIONP macro.  Since FUNCTIONP's expansion calls functionp and 
functionp is an extern inline function, the plugin may use the old version of 
functionp or it may use the new one -- the C standard allows either behavior. 
Either way, the plugin may misbehave.

This sort of thing suggests that it may be difficult to guarantee upward 
compatibility of plugins from one version of Emacs to the next.  And perhaps we 
shouldn't even try.

This raises another question: what exactly should go into emacs.h?  The more we 
put into emacs.h, the more likely plugin compatibility won't survive the 
transition from one version of Emacs to the next.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Thu, 12 Feb 2015 18:40:19 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> This raises another question: what exactly should go into emacs.h?  The more we 
> put into emacs.h, the more likely plugin compatibility won't survive the 
> transition from one version of Emacs to the next.

Right.  emacs.h should include the bare minimum, and each function,
macro, or variable we put there should have a very good reason for
being there.

IOW, we should design the interface between the modules and Emacs, and
we should do this ASAP, before the status quo forces us to make
compromises we will be unhappy with down the line.

I would start by coming up with the minimum set of requirements a
module needs to be able to communicate with Emacs.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

On 02/13/2015 12:37 AM, Eli Zaretskii wrote:
>> Date: Thu, 12 Feb 2015 18:40:19 -0800
>> From: Paul Eggert <eggert@cs.ucla.edu>
>>
>> This raises another question: what exactly should go into emacs.h?  The more we 
>> put into emacs.h, the more likely plugin compatibility won't survive the 
>> transition from one version of Emacs to the next.
> 
> Right.  emacs.h should include the bare minimum, and each function,
> macro, or variable we put there should have a very good reason for
> being there.
> 
> IOW, we should design the interface between the modules and Emacs, and
> we should do this ASAP, before the status quo forces us to make
> compromises we will be unhappy with down the line.

That's why this whole module effort is misguided and dangerous. With a
CFFI-style approach to loading code dynamically, shared modules don't
need to know anything about Emacs internal structure, so there are no
ABI or source compatibility issues.

I still think a CFFI port would be less work than trying to extract a
stable Emacs internals API that modules can include.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Fri, 13 Feb 2015 04:17:17 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: emacs-devel@gnu.org
> 
> That's why this whole module effort is misguided and dangerous. With a
> CFFI-style approach to loading code dynamically, shared modules don't
> need to know anything about Emacs internal structure, so there are no
> ABI or source compatibility issues.
> 
> I still think a CFFI port would be less work than trying to extract a
> stable Emacs internals API that modules can include.

I don't understand how CFFI would solve the problem, at least not all
of it.  We'd still need to write code to allow modules create Lisp
objects, right?  That code would still be Emacs version dependent,
right?  What am I missing?

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]

On 02/13/2015 07:01 AM, Eli Zaretskii wrote:
>> Date: Fri, 13 Feb 2015 04:17:17 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: emacs-devel@gnu.org
>>
>> That's why this whole module effort is misguided and dangerous. With a
>> CFFI-style approach to loading code dynamically, shared modules don't
>> need to know anything about Emacs internal structure, so there are no
>> ABI or source compatibility issues.
>>
>> I still think a CFFI port would be less work than trying to extract a
>> stable Emacs internals API that modules can include.
> 
> I don't understand how CFFI would solve the problem, at least not all
> of it.  We'd still need to write code to allow modules create Lisp
> objects, right?  That code would still be Emacs version dependent,
> right?  What am I missing?

In a CFFI world, modules don't know about lisp at all. Lisp code
describes the signatures of functions in loaded modules and calls into
the automatically generated wrapper functions. The called code has no
idea it's being called from lisp. If native code needs to create an
object, it creates it in the normal way and returns a pointer; lisp then
manages that pointer as an opaque value, possibly supplying it to
further calls to that module's functions.

This way, instead of loaded modules needing to track Emacs' historically
unstable internal data structures, elisp needs to track module ABIs,
which _have_ historically been stable; elisp interfaces can also be
updated without requiring users have a compiler.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Fri, 13 Feb 2015 07:06:17 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> This way, instead of loaded modules needing to track Emacs' historically
> unstable internal data structures, elisp needs to track module ABIs,

How's that better?  That's the same problem in the opposite direction,
only much worse, since the modules out there are not under our control
at all.

Maybe it works in Python because Python is much more standardized and
stable that Emacs Lisp.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

On 02/13/2015 07:49 AM, Eli Zaretskii wrote:
>> Date: Fri, 13 Feb 2015 07:06:17 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
>>
>> This way, instead of loaded modules needing to track Emacs' historically
>> unstable internal data structures, elisp needs to track module ABIs,
> 
> How's that better?  That's the same problem in the opposite direction,
> only much worse, since the modules out there are not under our control
> at all.

The risk isn't symmetric:

1) Shared libraries traditionally expose stable ABIs; when ABIs change
incompatibly, libraries change sonames. Shared libraries need stable
ABIs whether or not Emacs loads them: they're linked into other programs
and aren't necessarily rebuilt at the same time as their dependencies.
The Emacs core has never had a stable ABI, and creating one would
constrain our optimization opportunities.

(That is, libpng.so's ABI ought to change a lot less often than
/usr/bin/emacs's: the former promises to provide a stable ABI and the
latter never has.)

2) If a shared library's ABI does change, a Lisp update can fix the
problem. On the other hand, if a strongly-coupled module becomes
incompatible with the Emacs core, the remedy involves recompiling either
Emacs or the module. It's much easier to update Lisp than to rebuild
native code.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Fri, 13 Feb 2015 07:58:12 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> 1) Shared libraries traditionally expose stable ABIs; when ABIs change
> incompatibly, libraries change sonames. Shared libraries need stable
> ABIs whether or not Emacs loads them: they're linked into other programs
> and aren't necessarily rebuilt at the same time as their dependencies.
> The Emacs core has never had a stable ABI, and creating one would
> constrain our optimization opportunities.
> 
> (That is, libpng.so's ABI ought to change a lot less often than
> /usr/bin/emacs's: the former promises to provide a stable ABI and the
> latter never has.)

Sorry, I don't share your optimism about the stability of their ABI.
In particular, image libraries are a bad example on which to build
your point: look at the versonitis we need to employ in w32-win.el to
support all their different ABIs.  Each new revision there breaks
Emacs build with them.

And we didn't yet start talking about problems with passing FILE
pointers and malloc'ed buffers between Emacs and the modules.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2445 bytes --]

On 02/13/2015 08:10 AM, Eli Zaretskii wrote:
>> Date: Fri, 13 Feb 2015 07:58:12 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
>>
>> 1) Shared libraries traditionally expose stable ABIs; when ABIs change
>> incompatibly, libraries change sonames. Shared libraries need stable
>> ABIs whether or not Emacs loads them: they're linked into other programs
>> and aren't necessarily rebuilt at the same time as their dependencies.
>> The Emacs core has never had a stable ABI, and creating one would
>> constrain our optimization opportunities.
>>
>> (That is, libpng.so's ABI ought to change a lot less often than
>> /usr/bin/emacs's: the former promises to provide a stable ABI and the
>> latter never has.)
> 
> Sorry, I don't share your optimism about the stability of their ABI.

My optimism is rooted in reality. Most shared library interfaces are
stable the vast majority of the time. If ABIs were generally unstable,
we'd have to rebuild programs constantly, yet I can run very old
binaries (especially on Windows) without trouble.

> In particular, image libraries are a bad example on which to build
> your point: look at the versonitis we need to employ in w32-win.el to
> support all their different ABIs.  Each new revision there breaks
> Emacs build with them.

w32-win.el isn't that bad, especially for a module that loads quite a
few different image libraries.

Besides: using modules tightly coupled to the Emacs core doesn't
actually help. Instead of elisp coupled to one module's stable ABI, you
have a piece of native code tightly coupled not only to that module's
stable ABI, but _also_ to the volatile Emacs internal ABI. As jwz would
say, now you have two problems.

Using a CFFI-like approach both reduces the volatility of the interface
(since now there's a strong boundary in only one direction) and makes it
easier to fix problems when they do come up.

> And we didn't yet start talking about problems with passing FILE
> pointers and malloc'ed buffers between Emacs and the modules.

You have that problem anyway: a tightly-bound Emacs module for bridging
the Emacs core and some third-party library would have to link against
_some_ libc, and unless both the Emacs core and the third-party library
used the same libc, you'd lose. A better idea is not to pass C runtime
objects between dynamically loaded components.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Fri, 13 Feb 2015 08:20:09 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> > Sorry, I don't share your optimism about the stability of their ABI.
> 
> My optimism is rooted in reality.

So is mine.  I guess our experiences differ.

> w32-win.el isn't that bad, especially for a module that loads quite a
> few different image libraries.

It's not bad when you get it ready.  But the effort it took to find
out which versions changed ABI, to design the way of testing that, and
to have all the #ifdef's in Emacs core to support all of those
versions, and continue maintaining that as new versions arrive, is
anything but trivial, take it from someone who did most of that.

> Besides: using modules tightly coupled to the Emacs core doesn't
> actually help. Instead of elisp coupled to one module's stable ABI, you
> have a piece of native code tightly coupled not only to that module's
> stable ABI, but _also_ to the volatile Emacs internal ABI. As jwz would
> say, now you have two problems.

No, we have only one problem: design and implement a minimal set of
APIs necessary for modules to work, and keep it minimal.  All the rest
is the problem of module developers.

It makes little sense to me to turn the table and make module
compatibility something that core developers need to take care of.

> > And we didn't yet start talking about problems with passing FILE
> > pointers and malloc'ed buffers between Emacs and the modules.
> 
> You have that problem anyway: a tightly-bound Emacs module for bridging
> the Emacs core and some third-party library would have to link against
> _some_ libc, and unless both the Emacs core and the third-party library
> used the same libc, you'd lose.

A well-designed set of APIs between Emacs and modules will avoid this
and other similar problems.  This has already been done at least
twice, in 2 different ways: in Make and in Gawk.  I see no reason why
Emacs couldn't do something similar to one of those methods.

> A better idea is not to pass C runtime objects between dynamically
> loaded components.

Exactly.  But this requires the set of APIs I'm talking about to
support this without making non-trivial modules impossible.

Re: Dynamic loading progress

[Paul Eggert]

Eli Zaretskii wrote:
> A well-designed set of APIs between Emacs and modules will avoid this
> and other similar problems.  This has already been done at least
> twice, in 2 different ways: in Make and in Gawk.

Sure, but the cffi/libffi approach has seen a lot more uptake than that: GCC, 
CPython, OpenJDK, PLT Scheme, etc., etc.  It's quite a well-trodden path.

> we have only one problem: design and implement a minimal set of
> APIs necessary for modules to work, and keep it minimal.

cffi/libffi provides a good framework for that, no?  Why reinvent the wheel? 
Because that's what we'd need to do, when designing this new emacs.h file.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]

On 02/13/2015 09:08 AM, Paul Eggert wrote:
> Eli Zaretskii wrote:
>> A well-designed set of APIs between Emacs and modules will avoid this
>> and other similar problems.  This has already been done at least
>> twice, in 2 different ways: in Make and in Gawk.
> 
> Sure, but the cffi/libffi approach has seen a lot more uptake than that:
> GCC, CPython, OpenJDK, PLT Scheme, etc., etc.  It's quite a well-trodden
> path.
> 
>> we have only one problem: design and implement a minimal set of
>> APIs necessary for modules to work, and keep it minimal.
> 
> cffi/libffi provides a good framework for that, no?  Why reinvent the
> wheel? Because that's what we'd need to do, when designing this new
> emacs.h file.

I was imagining emacs.h looking a lot like jni.h: modules would export
some kind of emacs_init symbol that the Emacs core would call --- say,
something like this:

struct emacs_runtime {
  /* Return NULL on failure */
  struct emacs_interface* (*init)(
    int interface_version,
    const char* my_license);
};

struct emacs_interface_1 {
  /* Err, something like this */
  struct emacs_value* (*defun)(const char* name, int nargs);
  struct emacs_value* (*funcall)(struct emacs_value* func, ...);
};

/* Exported by loadable modules and called on load.
   Return 0 on success or -1 on error. */
extern int emacs_init(struct emacs_runtime*);

This way, Emacs itself doesn't need to export any symbols and it can
support multiple interface versions sanely.

I'd still prefer a CFFI-style interface though.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Fri, 13 Feb 2015 09:44:47 -0800
> From: Daniel Colascione <dancol@dancol.org>
> Cc: emacs-devel@gnu.org
> 
> struct emacs_runtime {
>   /* Return NULL on failure */
>   struct emacs_interface* (*init)(
>     int interface_version,
>     const char* my_license);
> };
> 
> struct emacs_interface_1 {
>   /* Err, something like this */
>   struct emacs_value* (*defun)(const char* name, int nargs);
>   struct emacs_value* (*funcall)(struct emacs_value* func, ...);
> };
> 
> /* Exported by loadable modules and called on load.
>    Return 0 on success or -1 on error. */
> extern int emacs_init(struct emacs_runtime*);
> 
> This way, Emacs itself doesn't need to export any symbols and it can
> support multiple interface versions sanely.

IOW, you suggest that the interface will be through function
pointers.  That's how Gawk does it.

This method (and any other) will need function pointers to create Lisp
data structures, though.  Like strings, lists, vectors, etc.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 3345 bytes --]

On 02/13/2015 08:55 AM, Eli Zaretskii wrote:
>> Date: Fri, 13 Feb 2015 08:20:09 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
>>
>>> Sorry, I don't share your optimism about the stability of their ABI.
>>
>> My optimism is rooted in reality.
> 
> So is mine.  I guess our experiences differ.
> 
>> w32-win.el isn't that bad, especially for a module that loads quite a
>> few different image libraries.
> 
> It's not bad when you get it ready.  But the effort it took to find
> out which versions changed ABI, to design the way of testing that, and
> to have all the #ifdef's in Emacs core to support all of those
> versions, and continue maintaining that as new versions arrive, is
> anything but trivial, take it from someone who did most of that.
> 
>> Besides: using modules tightly coupled to the Emacs core doesn't
>> actually help. Instead of elisp coupled to one module's stable ABI, you
>> have a piece of native code tightly coupled not only to that module's
>> stable ABI, but _also_ to the volatile Emacs internal ABI. As jwz would
>> say, now you have two problems.
> 
> No, we have only one problem: design and implement a minimal set of
> APIs necessary for modules to work, and keep it minimal.  All the rest
> is the problem of module developers.

Well, at least the proposed module design should allow the creation of a
CFFI interface _on top_ of it --- a bit like how we can use Java's JNI
to implement JNA.

By the way: Java does it the proposed Emacs way (via JNI), and the CLR
does it the way I propose (via P/Invoke). In my experience,
P/Invoke-using modules have been far less painful to use and maintain
than JNI-using ones.

Speaking of JNI: if you insist on tightly coupling modules to the Emacs
core, it's better to use a versioned system like JNI than to just
dynamically export symbols from the Emacs core. This way, module A can
request interface version 1 and module B can request interface version
2, and as long as a particular Emacs supports both interfaces 1 and 2,
you can load both modules into the same Emacs without trouble and
without rebuilding anything.

> It makes little sense to me to turn the table and make module
> compatibility something that core developers need to take care of.

With the proposed design, core developers have to take pains to make
Emacs itself export a stable ABI. With a CFFI-style interface, Emacs
doesn't have to maintain a stable ABI at all. To me, that sounds like
making life for core developers easier.

>>> And we didn't yet start talking about problems with passing FILE
>>> pointers and malloc'ed buffers between Emacs and the modules.
>>
>> You have that problem anyway: a tightly-bound Emacs module for bridging
>> the Emacs core and some third-party library would have to link against
>> _some_ libc, and unless both the Emacs core and the third-party library
>> used the same libc, you'd lose.
> 
> A well-designed set of APIs between Emacs and modules will avoid this
> and other similar problems.  This has already been done at least
> twice, in 2 different ways: in Make and in Gawk.  I see no reason why
> Emacs couldn't do something similar to one of those methods.

Woah. I had no idea that make supported loading dynamic objects.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stephen Leake]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/13/2015 07:01 AM, Eli Zaretskii wrote:
>>> Date: Fri, 13 Feb 2015 04:17:17 -0800
>>> From: Daniel Colascione <dancol@dancol.org>
>>> CC: emacs-devel@gnu.org
>>>
>>> That's why this whole module effort is misguided and dangerous. With a
>>> CFFI-style approach to loading code dynamically, shared modules don't
>>> need to know anything about Emacs internal structure, so there are no
>>> ABI or source compatibility issues.
>>>
>>> I still think a CFFI port would be less work than trying to extract a
>>> stable Emacs internals API that modules can include.
>> 
>> I don't understand how CFFI would solve the problem, at least not all
>> of it.  We'd still need to write code to allow modules create Lisp
>> objects, right?  That code would still be Emacs version dependent,
>> right?  What am I missing?
>
> In a CFFI world, modules don't know about lisp at all. Lisp code
> describes the signatures of functions in loaded modules and calls into
> the automatically generated wrapper functions. The called code has no
> idea it's being called from lisp. 

Which means (I think), that module code cannot call lisp functions.

My use case requires calling lisp functions; in particular,
put-text-property, but also other higher-level lisp functions in
ada-mode.

I need a compiled module to implement an LALR parser for the Ada
language that is fast enough for very large files.

-- 
-- Stephe

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Fri, 13 Feb 2015 15:48:07 -0600
> 
> My use case requires calling lisp functions; in particular,
> put-text-property, but also other higher-level lisp functions in
> ada-mode.

Why do you need to call Lisp in the module?  I think it's a no-no for
modules to call Lisp; that should be done in the Lisp code that uses
the module.

> I need a compiled module to implement an LALR parser for the Ada
> language that is fast enough for very large files.

Why does an implementation of a parser need to call Lisp?

Re: Dynamic loading progress

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: Stephen Leake <stephen_leake@stephe-leake.org>
 > > Date: Fri, 13 Feb 2015 15:48:07 -0600
 > > 
 > > My use case requires calling lisp functions; in particular,
 > > put-text-property, but also other higher-level lisp functions in
 > > ada-mode.
 > 
 > Why do you need to call Lisp in the module?  I think it's a no-no for
 > modules to call Lisp; that should be done in the Lisp code that uses
 > the module.

I don't understand.  XEmacs's ELLs (dynamically loadable modules) have
no problem with calling Lisp (use Ffuncall) or using symbols (use
Fintern).  The only requirement that I've seen expressed in this
thread that XEmacs ELLs don't satisfy is that a module should be
compilable for use with multiple versions of XEmacs (of course modules
actually work with multiple versions, but that's not officially
supported).

XEmacs also doesn't really support unloading, although there's an
unload function (there is no real guarantee that everything of
interest will be cleaned), but I can't really see why that's an
important requirement.  It's mostly a convenience to enable a fast
compile-load-test-unload cycle, and if it works half the time, it's a
significant saving over compile-restart-load-test.

 > > I need a compiled module to implement an LALR parser for the Ada
 > > language that is fast enough for very large files.
 > 
 > Why does an implementation of a parser need to call Lisp?

I can't give you a reason in the case of a parser, but in general
compiled Lisp functions may want to provide Lisp hooks, or implement
inner loops for simple functionality in C but do flexible handling of
more complex functionality in Lisp.  Perhaps a similar motivation is
relevant to the Ada parser.

Re: Dynamic loading progress

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
>     emacs-devel@gnu.org
> Date: Sat, 14 Feb 2015 21:18:51 +0900
> 
> Eli Zaretskii writes:
>  > > From: Stephen Leake <stephen_leake@stephe-leake.org>
>  > > Date: Fri, 13 Feb 2015 15:48:07 -0600
>  > > 
>  > > My use case requires calling lisp functions; in particular,
>  > > put-text-property, but also other higher-level lisp functions in
>  > > ada-mode.
>  > 
>  > Why do you need to call Lisp in the module?  I think it's a no-no for
>  > modules to call Lisp; that should be done in the Lisp code that uses
>  > the module.
> 
> I don't understand.  XEmacs's ELLs (dynamically loadable modules) have
> no problem with calling Lisp (use Ffuncall) or using symbols (use
> Fintern).  The only requirement that I've seen expressed in this
> thread that XEmacs ELLs don't satisfy is that a module should be
> compilable for use with multiple versions of XEmacs

This last requirement is what is being discussed here.  The capability
of creating Lisp objects depends on the internals, and AFAIU we would
like to avoid having that dependency.

>  > Why does an implementation of a parser need to call Lisp?
> 
> I can't give you a reason in the case of a parser, but in general
> compiled Lisp functions may want to provide Lisp hooks, or implement
> inner loops for simple functionality in C but do flexible handling of
> more complex functionality in Lisp.

I wasn't talking about using Lisp by modules, I was talking
specifically about calling Lisp from C.  Implementing new primitives
for performance purposes doesn't need that, at least it isn't clear to
me why it would.

Re: Dynamic loading progress

[Stefan Monnier]

> The capability of creating Lisp objects depends on the internals,

It doesn't have to be.


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>, stephen_leake@stephe-leake.org,
>         emacs-devel@gnu.org
> Date: Sat, 14 Feb 2015 09:38:38 -0500
> 
> > The capability of creating Lisp objects depends on the internals,
> 
> It doesn't have to be.

Not sure what you mean here; can you elaborate?

What I had in mind was all the recent changes to internals of Lisp
objects.

Re: Dynamic loading progress

[Stefan Monnier]

>> > The capability of creating Lisp objects depends on the internals,
>> It doesn't have to be.
> Not sure what you mean here; can you elaborate?

You don't need to know anything about the Lisp_Object representation
(other than it's an EMACS_INT) to call Fcons, Fintern, Fcar, ...


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: stephen@xemacs.org, stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> Date: Sun, 15 Feb 2015 13:36:20 -0500
> 
> >> > The capability of creating Lisp objects depends on the internals,
> >> It doesn't have to be.
> > Not sure what you mean here; can you elaborate?
> 
> You don't need to know anything about the Lisp_Object representation
> (other than it's an EMACS_INT) to call Fcons, Fintern, Fcar, ...

Not every Lisp object has such a function.

And what about the other direction: access a Lisp object's data from
C?  That is done almost entirely in C macros.

Re: Dynamic loading progress

[Stefan Monnier]

> Not every Lisp object has such a function.

But we can provide such functions very easily.


        Stefan

Re: Dynamic loading progress

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > I don't understand.  XEmacs's ELLs (dynamically loadable modules)
 > > have no problem with calling Lisp (use Ffuncall) or using symbols
 > > (use Fintern).  The only requirement that I've seen expressed in
 > > this thread that XEmacs ELLs don't satisfy is that a module
 > > should be compilable for use with multiple versions of XEmacs
 > 
 > This last requirement is what is being discussed here.  The capability
 > of creating Lisp objects depends on the internals, and AFAIU we would
 > like to avoid having that dependency.

It's not very hard to get backward compatibility (old compiled modules
work in new Emacs).  Define a stable ABI, and then stick to it.

It shouldn't be that hard to do it.  Most Lisp functionality of most
types is accessed through the object header, which points to a class
descriptor, which is basically a table of methods.  In XEmacs there's
a constructor, a finalizer (optional), a printer, a gc-marker
(optional), and a couple of others I forget offhand.  Add a length
indicator at the beginning, never change the order of methods from
past versions, and I think you're done.  If you add new methods in new
Emacs, no problem, just add them at the end of the table.  If you
deprecate old ones, just leave the slots and insert an unimplemented
marker (probably (slot_fun *) NULL will do but you could get fancy and
have a slot_deprecated_fun to put in that slot).  Sure, unneeded slots
may accumulate over time, but you're going to have only a few dozen
types at most and a couple unused slots per type.

Then require that module functions access other modules (including
core) only through Lisp (ie, Ffuncall and Fintern), which has a very
stylized calling convention.  If you find more efficiency is needed by
some modules, you can expand the allowed functions in new Emacs.

 > I wasn't talking about using Lisp by modules, I was talking
 > specifically about calling Lisp from C.  Implementing new
 > primitives for performance purposes doesn't need that, at least it
 > isn't clear to me why it would.

I just don't see what the problem is.  We call Lisp from C all the
time, that's what Ffuncall *is*.  It appears in most backtraces. :-)
I don't know about Emacs conventions back when you had GCPROs, but in
XEmacs you see functions marked "can GC" all over the place, precisely
because they call Lisp.

In the above, I'm being a little facetious, but it seems to me that
calling Lisp from C is much safer and more straightforward than
calling C from C.  Up to GCPRO, but Emacs doesn't have that any more,
correct?

Re: Dynamic loading progress

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: stephen_leake@stephe-leake.org,
>     emacs-devel@gnu.org
> Date: Sun, 15 Feb 2015 01:48:50 +0900
> 
> Eli Zaretskii writes:
> 
>  > > I don't understand.  XEmacs's ELLs (dynamically loadable modules)
>  > > have no problem with calling Lisp (use Ffuncall) or using symbols
>  > > (use Fintern).  The only requirement that I've seen expressed in
>  > > this thread that XEmacs ELLs don't satisfy is that a module
>  > > should be compilable for use with multiple versions of XEmacs
>  > 
>  > This last requirement is what is being discussed here.  The capability
>  > of creating Lisp objects depends on the internals, and AFAIU we would
>  > like to avoid having that dependency.
> 
> It's not very hard to get backward compatibility (old compiled modules
> work in new Emacs).  Define a stable ABI, and then stick to it.

I simply don't believe this goal to be practically achievable, if
there are no limitations on the interface.

I also am not sure that people would accept forward incompatibility
easily: consider the number of people who use bleeding-edge Org, Gnus,
etc. with the last official release of Emacs.  Why would they agree to
lose that when part of a module is in C?

> It shouldn't be that hard to do it.  Most Lisp functionality of most
> types is accessed through the object header, which points to a class
> descriptor, which is basically a table of methods.  In XEmacs there's
> a constructor, a finalizer (optional), a printer, a gc-marker
> (optional), and a couple of others I forget offhand.  Add a length
> indicator at the beginning, never change the order of methods from
> past versions, and I think you're done.  If you add new methods in new
> Emacs, no problem, just add them at the end of the table.  If you
> deprecate old ones, just leave the slots and insert an unimplemented
> marker (probably (slot_fun *) NULL will do but you could get fancy and
> have a slot_deprecated_fun to put in that slot).  Sure, unneeded slots
> may accumulate over time, but you're going to have only a few dozen
> types at most and a couple unused slots per type.

Emacs objects aren't built that way, so part of what you say is
inapplicable.  But the real problems don't happen at that level, they
are elsewhere, see below.

> Then require that module functions access other modules (including
> core) only through Lisp (ie, Ffuncall and Fintern), which has a very
> stylized calling convention.

And what about accessing Lisp objects?  Most of them are accessed
through C macros, whose definitions change as the internals change.
What can you do with those to preserve compatibility across Emacs
versions?

>  > I wasn't talking about using Lisp by modules, I was talking
>  > specifically about calling Lisp from C.  Implementing new
>  > primitives for performance purposes doesn't need that, at least it
>  > isn't clear to me why it would.
> 
> I just don't see what the problem is.  We call Lisp from C all the
> time, that's what Ffuncall *is*.  It appears in most backtraces. :-)

It appears in most backtraces because the Lisp interpreter calls it
when it sees funcall in Lisp.

More to the point: I simply don't see any reason for calling Lisp from
C.  If a module needs to do something in Lisp, let it do it in Lisp
directly, as every external package for Emacs does.  C code should be
reserved for providing additional primitives that core Emacs lacks,
and that's all.  At least that should be the starting point, IMO.

Re: Dynamic loading progress

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: "Stephen J. Turnbull" <stephen@xemacs.org>

 > I simply don't believe this goal to be practically achievable, if
 > there are no limitations on the interface.

The limitation is obvious: in principle, all Lisp objects are accessed
through Lisp, with the exception of numbers.  Even for strings and
buffers you can call Lisp "coding system" functions.  You may need to
add an "internal char[] buffer" type to serve as source/sink (I'm sure
you already have one for stream I/O, but it probably doesn't have a
formal or stable definition).

 > I also am not sure that people would accept forward incompatibility
 > easily: consider the number of people who use bleeding-edge Org, Gnus,
 > etc. with the last official release of Emacs.  Why would they agree to
 > lose that when part of a module is in C?

For the same reasons they do when a package decides to use some new
feature of Lisp that is forward incompatible.  Mostly, they have no
choice if the package decides to do that.  Upgrade their Emacs or lose
out.  But I don't think you need to worry that the package maintainers
will gratuitously break compatibility for users of C modules.  If
Emacs is unwilling to be disciplined, package maintainers won't use C
modules, or will call through Lisp and otherwise treat external Lisp
objects as opaque.

 > And what about accessing Lisp objects?  Most of them are accessed
 > through C macros, whose definitions change as the internals change.
 > What can you do with those to preserve compatibility across Emacs
 > versions?

Prohibit such change, or add accessor functions with a stable ABI
encapsulating the accessor macros to a table for use by externally
compiled modules loaded at runtime.  A C function call is very cheap
compared to a Lisp funcall.  If this is too expensive the module
probably shouldn't be based on Emacs at all, or it can be moved into
core as has been done traditionally.  It's that simple, both as a
constraint on core development and as a software engineering
technique.  This is not rocket science, and hasn't been for a decade
and a half now.  It's just work, which the core developers can and
should impose on the folks who want the feature.

If you expect you must change some ABI in the future and can't/won't
use the method table approach, tell module writers to access them
*through* Lisp and live with funcall overhead.  I assure you there are
plenty of features that can benefit from C code that can live very
happily even with that restriction.

 > More to the point: I simply don't see any reason for calling Lisp from
 > C.  If a module needs to do something in Lisp, let it do it in Lisp
 > directly, as every external package for Emacs does.  C code should
 > be reserved for providing additional primitives that core Emacs
 > lacks, and that's all.  At least that should be the starting point,
 > IMO.

But you're not the one who wants this feature.  The people who do want
this feature, in both XEmacs (which has more than a decade of
experience with ELLs) and now in Emacs, have accessed, or think they
need to access, Lisp objects from their C code.

Re: Dynamic loading progress

[Steinar Bang]

>>>>> "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>:

> But you're not the one who wants this feature.  The people who do want
> this feature, in both XEmacs (which has more than a decade of
> experience with ELLs) and now in Emacs, have accessed, or think they
> need to access, Lisp objects from their C code.

A good use case seems to be the Ada parser mentioned in this thread,
where it is desirable to call lisp code from the parser semantic
actions.

Re: Dynamic loading progress

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
> Cc: stephen_leake@stephe-leake.org,
>     emacs-devel@gnu.org
> Date: Sun, 15 Feb 2015 17:03:21 +0900
> 
>  > I simply don't believe this goal to be practically achievable, if
>  > there are no limitations on the interface.
> 
> The limitation is obvious: in principle, all Lisp objects are accessed
> through Lisp

Not sure what that means in practice.  What would be the definition of
a Lisp_Object visible to a module?  Will it be what we have on lisp.h,
or some opaque object?

> with the exception of numbers.

Why the exception?  What if the internal representation of a number
changes?  (It already did, a few weeks ago.)

> Even for strings and buffers you can call Lisp "coding system"
> functions.  You may need to add an "internal char[] buffer" type to
> serve as source/sink (I'm sure you already have one for stream I/O,
> but it probably doesn't have a formal or stable definition).

I don't follow: why coding-systems are relevant here?  Perhaps you
have XEmacs implementation in mind.

>  > I also am not sure that people would accept forward incompatibility
>  > easily: consider the number of people who use bleeding-edge Org, Gnus,
>  > etc. with the last official release of Emacs.  Why would they agree to
>  > lose that when part of a module is in C?
> 
> For the same reasons they do when a package decides to use some new
> feature of Lisp that is forward incompatible.

Using Lisp objects is not a new feature, so I don't think this analogy
will hold water.

> If Emacs is unwilling to be disciplined, package maintainers won't
> use C modules, or will call through Lisp and otherwise treat
> external Lisp objects as opaque.

I think they will rather apply pressure to avoid changes that break
binary compatibility, which in the long run will limit development.
Not sure if we want that; something to discuss, I guess.

>  > And what about accessing Lisp objects?  Most of them are accessed
>  > through C macros, whose definitions change as the internals change.
>  > What can you do with those to preserve compatibility across Emacs
>  > versions?
> 
> Prohibit such change, or add accessor functions with a stable ABI
> encapsulating the accessor macros to a table for use by externally
> compiled modules loaded at runtime.

That's what I tried to do in my suggestion for the API, but I don't
hold my breath to see it accepted.  Stephen Leake already said he
didn't like it; I have no reason to believe I will hear anything
different from others.  The way the current modules on the branch are
written are a good sign of what will follow, I think.

> If you expect you must change some ABI in the future and can't/won't
> use the method table approach, tell module writers to access them
> *through* Lisp and live with funcall overhead.

How can funcall help you access Lisp objects and convert their values
to something that C can grok?  Sooner or later, you will have to
access the internals of the objects.

So the funcall issue is irrelevant to accessing Lisp objects.

>  > More to the point: I simply don't see any reason for calling Lisp from
>  > C.  If a module needs to do something in Lisp, let it do it in Lisp
>  > directly, as every external package for Emacs does.  C code should
>  > be reserved for providing additional primitives that core Emacs
>  > lacks, and that's all.  At least that should be the starting point,
>  > IMO.
> 
> But you're not the one who wants this feature.  The people who do want
> this feature, in both XEmacs (which has more than a decade of
> experience with ELLs) and now in Emacs, have accessed, or think they
> need to access, Lisp objects from their C code.

Yes, I understand that.  And because of that I don't believe the
"stable API" paradigm will be accepted.  I think people will want to
write modules with the same freedom and flexibility we write core
code.  Which means the "tightly-coupled" paradigm.

Re: Dynamic loading progress

[Stefan Monnier]

> a Lisp_Object visible to a module?  Will it be what we have on lisp.h,
> or some opaque object?

It'll be opaque (i.e. EMACS_INT and that's it, everything else happens
via function calls).


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>,
>         stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> Date: Sun, 15 Feb 2015 13:47:56 -0500
> 
> > a Lisp_Object visible to a module?  Will it be what we have on lisp.h,
> > or some opaque object?
> 
> It'll be opaque (i.e. EMACS_INT and that's it, everything else happens
> via function calls).

But even a definition of EMACS_INT comes with a lot of baggage, just
look at lisp.h with fresh eyes.

IOW, you should present a more complete proposal, if you want to
convince (including convince yourself ;-) that this concept can hold
enough water.

Re: Dynamic loading progress

[Stefan Monnier]

> But even a definition of EMACS_INT comes with a lot of baggage, just
> look at lisp.h with fresh eyes.

M-x module-make RET would very easily be in a position to provide the
right definition of EMACS_INT.


        Stefan

Re: Dynamic loading progress

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Fri, 13 Feb 2015 15:48:07 -0600
>> 
>> My use case requires calling lisp functions; in particular,
>> put-text-property, but also other higher-level lisp functions in
>> ada-mode.
>
> Why do you need to call Lisp in the module?  I think it's a no-no for
> modules to call Lisp; that should be done in the Lisp code that uses
> the module.
>
>> I need a compiled module to implement an LALR parser for the Ada
>> language that is fast enough for very large files.
>
> Why does an implementation of a parser need to call Lisp?

Emacs ada-mode does indentation in two steps; first it parses the source
code, and the parser actions are lisp functions that eventually call
put-text-property to store information about the syntax and/or semantics
on many identifiers. Then the indentation code uses those text
properties to compute indentation.

I have a generalized LALR parser implemented in elisp that is fast
enough for many user's Ada files, but some users have much bigger files,
and it takes them 10 seconds to parse. So I need a faster
implementation. So far my benchmarking says I can get close with a
machine compiled parser.

So the module would contain the generalized LALR parser; the actions of
the parser would still be calls to the lisp functions.

Of course, lisp put-text-property is implemented in C, so I could
reimplement the parser actions in the module, and call that C function
directly.

But it would be convenient to be able to call the higher-level lisp
parser action functions from the module; that would simplify
maintaining the module and elisp parser implementations in parallel.

-- 
-- Stephe

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 14 Feb 2015 09:32:54 -0600
> 
> Emacs ada-mode does indentation in two steps; first it parses the source
> code, and the parser actions are lisp functions that eventually call
> put-text-property to store information about the syntax and/or semantics
> on many identifiers. Then the indentation code uses those text
> properties to compute indentation.
> 
> I have a generalized LALR parser implemented in elisp that is fast
> enough for many user's Ada files, but some users have much bigger files,
> and it takes them 10 seconds to parse. So I need a faster
> implementation. So far my benchmarking says I can get close with a
> machine compiled parser.
> 
> So the module would contain the generalized LALR parser; the actions of
> the parser would still be calls to the lisp functions.

But this means your mode is in Lisp to begin with, and it just calls
functions implemented in C to work faster.  So you could leave the
calls to put-text-property in Lisp, and have the functions implemented
in the module return the information required to compute the arguments
to put-text-property.  Or am I missing something?

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sat, 14 Feb 2015 18:02:22 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Stephen Leake <stephen_leake@stephe-leake.org>
> > Date: Sat, 14 Feb 2015 09:32:54 -0600
> > 
> > Emacs ada-mode does indentation in two steps; first it parses the source
> > code, and the parser actions are lisp functions that eventually call
> > put-text-property to store information about the syntax and/or semantics
> > on many identifiers. Then the indentation code uses those text
> > properties to compute indentation.
> > 
> > I have a generalized LALR parser implemented in elisp that is fast
> > enough for many user's Ada files, but some users have much bigger files,
> > and it takes them 10 seconds to parse. So I need a faster
> > implementation. So far my benchmarking says I can get close with a
> > machine compiled parser.
> > 
> > So the module would contain the generalized LALR parser; the actions of
> > the parser would still be calls to the lisp functions.
> 
> But this means your mode is in Lisp to begin with, and it just calls
> functions implemented in C to work faster.  So you could leave the
> calls to put-text-property in Lisp, and have the functions implemented
> in the module return the information required to compute the arguments
> to put-text-property.  Or am I missing something?

And btw, what exactly do you mean by "call Lisp" in this case?  Call
put-text-property, which is implemented in C in its entirety, doesn't
constitute a call to Lisp in my book.

Of course, that means we will want to export Fput_text_property...

Re: Dynamic loading progress

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 14 Feb 2015 18:02:22 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> > From: Stephen Leake <stephen_leake@stephe-leake.org>
>> > Date: Sat, 14 Feb 2015 09:32:54 -0600
>> > 
>> > Emacs ada-mode does indentation in two steps; first it parses the source
>> > code, and the parser actions are lisp functions that eventually call
>> > put-text-property to store information about the syntax and/or semantics
>> > on many identifiers. Then the indentation code uses those text
>> > properties to compute indentation.
>> > 
>> > I have a generalized LALR parser implemented in elisp that is fast
>> > enough for many user's Ada files, but some users have much bigger files,
>> > and it takes them 10 seconds to parse. So I need a faster
>> > implementation. So far my benchmarking says I can get close with a
>> > machine compiled parser.
>> > 
>> > So the module would contain the generalized LALR parser; the actions of
>> > the parser would still be calls to the lisp functions.
>> 
>> But this means your mode is in Lisp to begin with, and it just calls
>> functions implemented in C to work faster.  So you could leave the
>> calls to put-text-property in Lisp, and have the functions implemented
>> in the module return the information required to compute the arguments
>> to put-text-property.  Or am I missing something?
>
> And btw, what exactly do you mean by "call Lisp" in this case?  Call
> put-text-property, which is implemented in C in its entirety, doesn't
> constitute a call to Lisp in my book.

The parser actions are higher level lisp functions; they do call
put-text-property, but there is significant logic in the lisp, and I'd
rather not re-implement that in the module, to simplify maintenance.
I'll have to maintain both the elisp and module implementations, for two
reasons:

- emacs 24 doesn't support modules

- some (many?) ada-mode users won't bother to compile the module (and I
  don't plan to distribute a binary).

> Of course, that means we will want to export Fput_text_property...

Right, that might be needed at some point. Right now I'll settle for
'funcall'.

-- 
-- Stephe

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1719 bytes --]

On 02/14/2015 04:39 PM, Stephen Leake wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> Date: Sat, 14 Feb 2015 18:02:22 +0200
>>> From: Eli Zaretskii <eliz@gnu.org>
>>> Cc: emacs-devel@gnu.org
>>>
>>>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>>>> Date: Sat, 14 Feb 2015 09:32:54 -0600
>>>>
>>>> Emacs ada-mode does indentation in two steps; first it parses the source
>>>> code, and the parser actions are lisp functions that eventually call
>>>> put-text-property to store information about the syntax and/or semantics
>>>> on many identifiers. Then the indentation code uses those text
>>>> properties to compute indentation.
>>>>
>>>> I have a generalized LALR parser implemented in elisp that is fast
>>>> enough for many user's Ada files, but some users have much bigger files,
>>>> and it takes them 10 seconds to parse. So I need a faster
>>>> implementation. So far my benchmarking says I can get close with a
>>>> machine compiled parser.

You don't a parser in C. You need a smarter parser. Look up
"incremental", please.

>>>>
>>>> So the module would contain the generalized LALR parser; the actions of
>>>> the parser would still be calls to the lisp functions.

The right way to let C libraries call Lisp functions is to let C
libraries accept C callback functions, then thunk from C callbacks to
Lisp functions in the CFFI layer. That's what other CFFI implementations
do. This way, your library _still_ doesn't need to know about specific
Lisp internals.

The module proposal in this thread will harm Emacs development because
it'll put unnecessary constraints on the elisp implementations. Look at
how much trouble PyPy has with CPython modules.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stephen Leake]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/14/2015 04:39 PM, Stephen Leake wrote:
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>>> Date: Sat, 14 Feb 2015 18:02:22 +0200
>>>> From: Eli Zaretskii <eliz@gnu.org>
>>>> Cc: emacs-devel@gnu.org
>>>>
>>>>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>>>>> Date: Sat, 14 Feb 2015 09:32:54 -0600
>>>>>
>>>>> Emacs ada-mode does indentation in two steps; first it parses the source
>>>>> code, and the parser actions are lisp functions that eventually call
>>>>> put-text-property to store information about the syntax and/or semantics
>>>>> on many identifiers. Then the indentation code uses those text
>>>>> properties to compute indentation.
>>>>>
>>>>> I have a generalized LALR parser implemented in elisp that is fast
>>>>> enough for many user's Ada files, but some users have much bigger files,
>>>>> and it takes them 10 seconds to parse. So I need a faster
>>>>> implementation. So far my benchmarking says I can get close with a
>>>>> machine compiled parser.
>
> You don't a parser in C. You need a smarter parser. Look up
> "incremental", please.

Yes, an incremental parser would help. I have looked at that; I'd much
rather write that in Ada than in lisp.

I tried using an Ada parser in a separate process communication via
pipes over stdin/stdout; the communications overhead was too much.

So a module implementation would still be a good thing. For the moment,
just moving the current parser into a module is the first step; if that
is not fast enough, implementing an incremental parser is the next step.

>>>>> So the module would contain the generalized LALR parser; the actions of
>>>>> the parser would still be calls to the lisp functions.
>
> The right way to let C libraries call Lisp functions is to let C
> libraries accept C callback functions, then thunk from C callbacks to
> Lisp functions in the CFFI layer. That's what other CFFI implementations
> do. This way, your library _still_ doesn't need to know about specific
> Lisp internals.
>
> The module proposal in this thread will harm Emacs development because
> it'll put unnecessary constraints on the elisp implementations. Look at
> how much trouble PyPy has with CPython modules.

It appears that the people actually working on this issue don't have
experience with the various forms of module interface.

Perhaps you could take a stab at writing a CFFI layer that is sufficient
for the curl module?

-- 
-- Stephe

Re: Dynamic loading progress

[Stefan Monnier]

> The parser actions are higher level lisp functions; they do call
> put-text-property, but there is significant logic in the lisp, and I'd
> rather not re-implement that in the module, to simplify maintenance.

I don't see why calling Elisp from a C (or Ada) module would cause any
kind of problem, so this approach looks fine to me.


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Sun, 15 Feb 2015 13:43:45 -0500
> Cc: emacs-devel@gnu.org
> 
> > The parser actions are higher level lisp functions; they do call
> > put-text-property, but there is significant logic in the lisp, and I'd
> > rather not re-implement that in the module, to simplify maintenance.
> 
> I don't see why calling Elisp from a C (or Ada) module would cause any
> kind of problem, so this approach looks fine to me.

Look at the modules on the branch to see where this is going.  It
certainly contradicts your "no need to know about internals of Lisp
objects" concept.

Re: Dynamic loading progress

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Look at the modules on the branch to see where this is going.  It
 > certainly contradicts your "no need to know about internals of Lisp
 > objects" concept.

No, it simply means that the current modules are programmed by people
who know about dlopen and Emacs internals and have not thought about
designing a runtime environment for the modules.

Daniel's proposal makes a lot of sense, and is better than the current
S?XEmacs design.

Re: Dynamic loading progress

[Aurélien Aptel]

I've let this sit for a few days, didn't expect so much feedback, good!

I was in favor of a tight integration (like Linux kernel modules which
are in the same repository as the kernel) about after reading
everything I'm not sure what to think anymore. Keeping modules
uptodate with Emacs while also backporting features to older Emacs is
insane, that's for sure.

In any case rewriting current modules is not a problem, they are very
simple and meant as examples/POCs.

On Mon, Feb 16, 2015 at 5:52 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
> No, it simply means that the current modules are programmed by people
> who know about dlopen and Emacs internals and have not thought about
> designing a runtime environment for the modules.

I'm far from an expert, yes. I'm just the guy who wanted to extend
Emacs in C without recompiling it and acted on it.

A real FFI would be better than a module API but I'm not sure how to
handle certain things: making/passing struct pointers and callbacks,
mainly. I don't know if it can be efficient either.

> Daniel's proposal makes a lot of sense, and is better than the current
> S?XEmacs design.

Daniel's proposal is very good yes, it's the most convincing solution
I think. And as he said the FFI can be implemented on top of it. I
just hope all the type conversion and intern() calls won't ruin
performances.

Re: Dynamic loading progress

[joakim]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> The parser actions are higher level lisp functions; they do call
>> put-text-property, but there is significant logic in the lisp, and I'd
>> rather not re-implement that in the module, to simplify maintenance.
>
> I don't see why calling Elisp from a C (or Ada) module would cause any
> kind of problem, so this approach looks fine to me.

This is just my experience from implementing Xwidgets FWIW.

Originally I tried having external code calling Emacs internals, but
that was very unreliable. Then I switched to generating events that was
handled by the Emacs eventloop, and that now works fine.

The other way around, Emacs calling code in a module via the Gnome
Introspection API works fine, except marshalling all the types isnt
trivial.


It would be nice if we could find some synergies between gnome object
introspection and other FFI types.

>
>
>         Stefan
>

-- 
Joakim Verona

Re: Dynamic loading progress

[Stefan Monnier]

> This is just my experience from implementing Xwidgets FWIW.
> Originally I tried having external code calling Emacs internals, but
> that was very unreliable.

That's not because it was called from external code.  It's because it
was called asynchronously.

What I meant by the above is that it's perfectly OK for module code to
call back Elisp in cases such as when Fmapcar calls Elisp:
synchronous calls.

There are indeed various restrictions about *when* you can run Elisp.
These apply to module code, but they also apply to core code.


        Stefan

Re: Dynamic loading progress

[joakim]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> This is just my experience from implementing Xwidgets FWIW.
>> Originally I tried having external code calling Emacs internals, but
>> that was very unreliable.
>
> That's not because it was called from external code.  It's because it
> was called asynchronously.

Yes, I should have clarified that.

>
> What I meant by the above is that it's perfectly OK for module code to
> call back Elisp in cases such as when Fmapcar calls Elisp:
> synchronous calls.
>
> There are indeed various restrictions about *when* you can run Elisp.
> These apply to module code, but they also apply to core code.
>
>
>         Stefan
>

-- 
Joakim Verona

Re: Dynamic loading progress

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sat, 14 Feb 2015 09:32:54 -0600
>> 
>> Emacs ada-mode does indentation in two steps; first it parses the source
>> code, and the parser actions are lisp functions that eventually call
>> put-text-property to store information about the syntax and/or semantics
>> on many identifiers. Then the indentation code uses those text
>> properties to compute indentation.
>> 
>> I have a generalized LALR parser implemented in elisp that is fast
>> enough for many user's Ada files, but some users have much bigger files,
>> and it takes them 10 seconds to parse. So I need a faster
>> implementation. So far my benchmarking says I can get close with a
>> machine compiled parser.
>> 
>> So the module would contain the generalized LALR parser; the actions of
>> the parser would still be calls to the lisp functions.
>
> But this means your mode is in Lisp to begin with, and it just calls
> functions implemented in C to work faster.  So you could leave the
> calls to put-text-property in Lisp, and have the functions implemented
> in the module return the information required to compute the arguments
> to put-text-property.  Or am I missing something?

The parser actions are callbacks; when the parser recognizes a complete
grammar sentence (ie a statement, function declaration, etc), it calls
an action (which in this case is a lisp function), then resumes parsing.
Each file contains many grammar sentences, so there are many calls to
lisp functions from within the C parser. (Actually, the parser is in
Ada, but that's a separate issue).

I could redefine the parser actions to store the required info in a
list, and at the end of parsing return that list, and then process the
list in elisp. But I've already tried that with the parser in an
external process; just processing the returned list is too slow. I need
direct calls.

-- 
-- Stephe

Re: Dynamic loading progress

[Stefan Monnier]

> I would start by coming up with the minimum set of requirements a
> module needs to be able to communicate with Emacs.

Easy: take the existing sample modules and see what they need.


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 13 Feb 2015 14:11:33 -0500
> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel@gnu.org
> 
> > I would start by coming up with the minimum set of requirements a
> > module needs to be able to communicate with Emacs.
> 
> Easy: take the existing sample modules and see what they need.

No one seems to care, so I will bite the bullet.  The below is based
on looking at the modules branch.

It looks like modules need:

 . A function to replace the DEFUN macro and the related defsubr call.

 . Some definition of Lisp_Object

   This already presents at least 2 problems.  The first is 32- vs
   64-bit issues.  We need some way of checking that a 32-bit module
   isn't loaded by a 64-bit Emacs, and vice versa.

   The other problem is more serious: Emacs can be built with or
   without --check-lisp-object-type, and with or without --wide-int;
   how will the module know which implementation to use?

   Perhaps the solution is to make Lisp_Object an opaque data type for
   the modules, and not use the definition we have on lisp.h.

 . Functions to access values of Lisp objects.  We shouldn't rely on C
   macros like XINT and XWINDOW for that, because macros track too
   closely the internals of each object.  So I propose to use
   functions that will be exposed via the API.

 . Functions to create Lisp objects and/or set their values.  Again,
   we shouldn't rely on the likes of XSETCONS.  There should be a
   function make_FOO for each object type FOO.  This includes creating
   symbols, so 'intern' should not be needed by modules.

 . A function to create a Lisp object that is a container for a C
   struct or pointer.

The above means that most of the C source files under modules/ should
be thoroughly rewritten.  They are currently written as if the code
were an integral part of Emacs on one of the Emacs C files.  Going
that way means that modules will have to be recompiled from sources
for each Emacs version, and practically will have to live in the Emacs
tree.  Maybe this is what we want, I don't know.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sat, 14 Feb 2015 21:12:02 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> The above means that most of the C source files under modules/ should
> be thoroughly rewritten.  They are currently written as if the code
> were an integral part of Emacs on one of the Emacs C files.

Btw, one thing that the current code in modules/ entirely ignores is
encoding of file names, URLs, and buffer/string text: the current code
simply hands C pointers to string/buffer internals to C APIs.  Modules
should encode/decode them when they communicate with C library APIs,
otherwise the result will be subtle bugs with non-ASCII text.

Re: Dynamic loading progress

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 3439 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Fri, 13 Feb 2015 14:11:33 -0500
>> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel@gnu.org
>>
>> > I would start by coming up with the minimum set of requirements a
>> > module needs to be able to communicate with Emacs.
>>
>> Easy: take the existing sample modules and see what they need.
>
> No one seems to care, so I will bite the bullet.

I took a stab at creating emacs_module_api.h; attached.

To do this, I replaced "#include <lisp.h>"
in modules/curl/curl.c with "#include <emacs_modules_api.h>", and copied
stuff from lisp.h and other headers to emacs_module_api.h until it
compiled. Note the FIXMEs; I could not figure out what was going on in
some places, so I just kludged it for now.

I don't know if this is useful, but it is my interpretation of what
Stefan proposed; lisp.h would "#include emacs_modules_api.h". (He called
it "emacs.h", but I think including 'modules' in the name is better).

> The below is based
> on looking at the modules branch.
>
> It looks like modules need:
>
>  . A function to replace the DEFUN macro and the related defsubr call.
>
>  . Some definition of Lisp_Object
>
>    This already presents at least 2 problems.  The first is 32- vs
>    64-bit issues.  We need some way of checking that a 32-bit module
>    isn't loaded by a 64-bit Emacs, and vice versa.

Good point. That's a special case of "don't load a module compiled for
processor X in an Emacs compiled for processor Y". gnu binutils has
facilities for that.

>    The other problem is more serious: Emacs can be built with or
>    without --check-lisp-object-type, and with or without --wide-int;
>    how will the module know which implementation to use?

Similar check when the module is loaded. Which means we need some
metainfo for each module, and a standard function to retrieve it.

>  . Functions to access values of Lisp objects.  We shouldn't rely on C
>    macros like XINT and XWINDOW for that, because macros track too
>    closely the internals of each object.  So I propose to use
>    functions that will be exposed via the API.

If we follow Stefan's suggestion, then either this function approach is
not viable, or we need another #ifdef for module vs emacs.

> The above means that most of the C source files under modules/ should
> be thoroughly rewritten.

Yes.

> They are currently written as if the code were an integral part of
> Emacs on one of the Emacs C files. Going that way means that modules
> will have to be recompiled from sources for each Emacs version, and
> practically will have to live in the Emacs tree. Maybe this is what we
> want, I don't know.

This is the model I had in mind. Since I need max speed with mixed
Ada/lisp code, I need tight integration with the core. The people who
need that speed for Ada mode will simply have to recompile the module
for each Emacs release; they are coders, so that's not a problem.

On the other hand, modules for other use cases (I have in mind a spam
filter for Gnus) would not need to be nearly so tightly bound, so a more
limited/safe API would be appropriate.


Rather than splitting out emacs_module_api.h from lisp.h, we could add
__declspec(dllexport) to a subset of the functions in lisp.h; I suspect
that will be a small number. I'll give that a try next.

Is there an equivalent on Linux (I don't recall ever seeing one)?

--
-- Stephe

[-- Attachment #2: emacs_modules_api.h --]
[-- Type: application/octet-stream, Size: 37490 bytes --]

/* Emacs modules API */

/* FIXME: for Windows, add __declspec(dllexport) to all (some?) of these */
  /* #ifdef WINDOWS */
  /* # ifdef emacs */
  /* #  define EMACS_EXPORT  __declspec(dllexport) */
  /* # else */
  /* #  define EMACS_EXPORT  __declspec(dllimport) */
  /* # endif */
  /* # else */
  /* #  define EMACS_EXPORT */
  /* #endif */

/* FIXME: <stdint.h> vs ../nt/inc/stdint.h:37 */
#include <stdint.h>

/* conf_post.h:35 */
#include <stdbool.h>

/* The type of bool bitfields.  Needed to compile Objective-C with
   standard GCC.  It was also needed to port to pre-C99 compilers,
   although we don't care about that any more.  */
/* conf_post.h:40 */
#if NS_IMPL_GNUSTEP
typedef unsigned int bool_bf;
#else
typedef bool bool_bf;
#endif

/* When not using Clang, assume its attributes and features are absent.  */
/* conf_post.h:62 */
#ifndef __has_attribute
# define __has_attribute(a) false
#endif
#ifndef __has_feature
# define __has_feature(a) false
#endif

/* conf_post.h:226 */
#if (__clang__                                                          \
     ? __has_attribute (externally_visible)                             \
     : (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 1)))
#define EXTERNALLY_VISIBLE __attribute__((externally_visible))
#else
#define EXTERNALLY_VISIBLE
#endif



/* FIXME: need to define INLINE */
/* conf_post.h:318 */
#define INLINE /* empty */

/* Define a TYPE constant ID as an externally visible name.  Use like this:

      DEFINE_GDB_SYMBOL_BEGIN (TYPE, ID)
      # define ID (some integer preprocessor expression of type TYPE)
      DEFINE_GDB_SYMBOL_END (ID)

   This hack is for the benefit of compilers that do not make macro
   definitions or enums visible to the debugger.  It's used for symbols
   that .gdbinit needs.  */
/* lisp.h:11 */
#define DECLARE_GDB_SYM(type, id) type const id EXTERNALLY_VISIBLE
#ifdef MAIN_PROGRAM
# define DEFINE_GDB_SYMBOL_BEGIN(type, id) DECLARE_GDB_SYM (type, id)
# define DEFINE_GDB_SYMBOL_END(id) = id;
#else
# define DEFINE_GDB_SYMBOL_BEGIN(type, id) extern DECLARE_GDB_SYM (type, id)
# define DEFINE_GDB_SYMBOL_END(val) ;
#endif

/* Number of bits in a Lisp_Object tag.  */
/* lisp.h:70 */
DEFINE_GDB_SYMBOL_BEGIN (int, GCTYPEBITS)
#define GCTYPEBITS 3
DEFINE_GDB_SYMBOL_END (GCTYPEBITS)

/* EMACS_INT - signed integer wide enough to hold an Emacs value
   EMACS_INT_MAX - maximum value of EMACS_INT; can be used in #if
   pI - printf length modifier for EMACS_INT
   EMACS_UINT - unsigned variant of EMACS_INT */
/* lisp.h:87 */
#ifndef EMACS_INT_MAX
# if INTPTR_MAX <= 0
#  error "INTPTR_MAX misconfigured"
# elif INTPTR_MAX <= INT_MAX >> NONPOINTER_BITS && !defined WIDE_EMACS_INT
typedef int EMACS_INT;
typedef unsigned int EMACS_UINT;
#  define EMACS_INT_MAX INT_MAX
#  define pI ""
# elif INTPTR_MAX <= LONG_MAX >> NONPOINTER_BITS && !defined WIDE_EMACS_INT
typedef long int EMACS_INT;
typedef unsigned long EMACS_UINT;
#  define EMACS_INT_MAX LONG_MAX
#  define pI "l"
/* Check versus LLONG_MAX, not LLONG_MAX >> NONPOINTER_BITS.
   In theory this is not safe, but in practice it seems to be OK.  */
# elif INTPTR_MAX <= LLONG_MAX
typedef long long int EMACS_INT;
typedef unsigned long long int EMACS_UINT;
#  define EMACS_INT_MAX LLONG_MAX
#  define pI "ll"
# else
#  error "INTPTR_MAX too large"
# endif
#endif

/* Number of bits in some machine integer types.  */
/* lisp.h:134 */
enum
  {
    BITS_PER_CHAR      = CHAR_BIT,
    BITS_PER_SHORT     = CHAR_BIT * sizeof (short),
    BITS_PER_LONG      = CHAR_BIT * sizeof (long int),
    BITS_PER_EMACS_INT = CHAR_BIT * sizeof (EMACS_INT)
  };

/* Define Emacs versions of <assert.h>'s 'assert (COND)' and <verify.h>'s
   'assume (COND)'.  COND should be free of side effects, as it may or
   may not be evaluated.

   'eassert (COND)' checks COND at runtime if ENABLE_CHECKING is
   defined and suppress_checking is false, and does nothing otherwise.
   Emacs dies if COND is checked and is false.  The suppress_checking
   variable is initialized to 0 in alloc.c.  Set it to 1 using a
   debugger to temporarily disable aborting on detected internal
   inconsistencies or error conditions.

   In some cases, a good compiler may be able to optimize away the
   eassert macro even if ENABLE_CHECKING is true, e.g., if XSTRING (x)
   uses eassert to test STRINGP (x), but a particular use of XSTRING
   is invoked only after testing that STRINGP (x) is true, making the
   test redundant.

   eassume is like eassert except that it also causes the compiler to
   assume that COND is true afterwards, regardless of whether runtime
   checking is enabled.  This can improve performance in some cases,
   though it can degrade performance in others.  It's often suboptimal
   for COND to call external functions or access volatile storage.  */
/* lisp.h:199 */
#ifndef ENABLE_CHECKING
# define eassert(cond) ((void) (false && (cond))) /* Check COND compiles.  */
# define eassume(cond) assume (cond)
#else /* ENABLE_CHECKING */

extern _Noreturn void die (const char *, const char *, int);

extern bool suppress_checking EXTERNALLY_VISIBLE;

# define eassert(cond)                                          \
   (suppress_checking || (cond)                                 \
    ? (void) 0                                                  \
    : die (# cond, __FILE__, __LINE__))
# define eassume(cond)                                          \
   (suppress_checking                                           \
    ? assume (cond)                                             \
    : (cond)                                                    \
    ? (void) 0                                                  \
    : die (# cond, __FILE__, __LINE__))
#endif /* ENABLE_CHECKING */

/* lisp.h:239 */
enum Lisp_Bits
  {
    /* 2**GCTYPEBITS.  This must be a macro that expands to a literal
       integer constant, for MSVC.  */
#define GCALIGNMENT 8

    /* Number of bits in a Lisp_Object value, not counting the tag.  */
    VALBITS = BITS_PER_EMACS_INT - GCTYPEBITS,

    /* Number of bits in a Lisp fixnum tag.  */
    INTTYPEBITS = GCTYPEBITS - 1,

    /* Number of bits in a Lisp fixnum value, not counting the tag.  */
    FIXNUM_BITS = VALBITS + 1
  };

/* The maximum value that can be stored in a EMACS_INT, assuming all
   bits other than the type bits contribute to a nonnegative signed value.
   This can be used in #if, e.g., '#if USB_TAG' below expands to an
   expression involving VAL_MAX.  */
/* lisp.h:263 */
#define VAL_MAX (EMACS_INT_MAX >> (GCTYPEBITS - 1))

/* Whether the least-significant bits of an EMACS_INT contain the tag.
   On hosts where pointers-as-ints do not exceed VAL_MAX / 2, USE_LSB_TAG is:
    a. unnecessary, because the top bits of an EMACS_INT are unused, and
    b. slower, because it typically requires extra masking.
   So, USE_LSB_TAG is true only on hosts where it might be useful.  */
/* lisp.h:270 */
/* FIXME: not working; just set it */
DEFINE_GDB_SYMBOL_BEGIN (bool, USE_LSB_TAG)
#define USE_LSB_TAG 0 /* (VAL_MAX / 2 < INTPTR_MAX) */
DEFINE_GDB_SYMBOL_END (USE_LSB_TAG)

/* FIXME: */
/* #if !USE_LSB_TAG && !defined WIDE_EMACS_INT */
/* # error "USE_LSB_TAG not supported on this platform; please report this." \ */
/*         "Try 'configure --with-wide-int' to work around the problem." */
/* error !; */
/* #endif */

/* lisp.h:280 */
#ifndef alignas
# define alignas(alignment) /* empty */
# if USE_LSB_TAG
#  error "USE_LSB_TAG requires alignas"
# endif
#endif

/* lisp.h:287 */
#ifdef HAVE_STRUCT_ATTRIBUTE_ALIGNED
# define GCALIGNED __attribute__ ((aligned (GCALIGNMENT)))
#else
# define GCALIGNED /* empty */
#endif

/* Some operations are so commonly executed that they are implemented
   as macros, not functions, because otherwise runtime performance would
   suffer too much when compiling with GCC without optimization.
   There's no need to inline everything, just the operations that
   would otherwise cause a serious performance problem.

   For each such operation OP, define a macro lisp_h_OP that contains
   the operation's implementation.  That way, OP can be implemented
   via a macro definition like this:

     #define OP(x) lisp_h_OP (x)

   and/or via a function definition like this:

     LISP_MACRO_DEFUN (OP, Lisp_Object, (Lisp_Object x), (x))

   which macro-expands to this:

     Lisp_Object (OP) (Lisp_Object x) { return lisp_h_OP (x); }

   without worrying about the implementations diverging, since
   lisp_h_OP defines the actual implementation.  The lisp_h_OP macros
   are intended to be private to this include file, and should not be
   used elsewhere.

   FIXME: Remove the lisp_h_OP macros, and define just the inline OP
   functions, once most developers have access to GCC 4.8 or later and
   can use "gcc -Og" to debug.  Maybe in the year 2016.  See
   Bug#11935.

   Commentary for these macros can be found near their corresponding
   functions, below.  */

#if CHECK_LISP_OBJECT_TYPE
# define lisp_h_XLI(o) ((o).i)
# define lisp_h_XIL(i) ((Lisp_Object) { i })
#else
# define lisp_h_XLI(o) (o)
# define lisp_h_XIL(i) (i)
#endif
#define lisp_h_CHECK_LIST_CONS(x, y) CHECK_TYPE (CONSP (x), Qlistp, y)
#define lisp_h_CHECK_NUMBER(x) CHECK_TYPE (INTEGERP (x), Qintegerp, x)
#define lisp_h_CHECK_SYMBOL(x) CHECK_TYPE (SYMBOLP (x), Qsymbolp, x)
#define lisp_h_CHECK_TYPE(ok, predicate, x) \
   ((ok) ? (void) 0 : (void) wrong_type_argument (predicate, x))
#define lisp_h_CONSP(x) (XTYPE (x) == Lisp_Cons)
#define lisp_h_EQ(x, y) (XLI (x) == XLI (y))
#define lisp_h_FLOATP(x) (XTYPE (x) == Lisp_Float)
#define lisp_h_INTEGERP(x) ((XTYPE (x) & (Lisp_Int0 | ~Lisp_Int1)) == Lisp_Int0)
#define lisp_h_MARKERP(x) (MISCP (x) && XMISCTYPE (x) == Lisp_Misc_Marker)
#define lisp_h_MISCP(x) (XTYPE (x) == Lisp_Misc)
#define lisp_h_NILP(x) EQ (x, Qnil)
#define lisp_h_SET_SYMBOL_VAL(sym, v) \
   (eassert ((sym)->redirect == SYMBOL_PLAINVAL), (sym)->val.value = (v))
#define lisp_h_SYMBOL_CONSTANT_P(sym) (XSYMBOL (sym)->constant)
#define lisp_h_SYMBOL_VAL(sym) \
   (eassert ((sym)->redirect == SYMBOL_PLAINVAL), (sym)->val.value)
#define lisp_h_SYMBOLP(x) (XTYPE (x) == Lisp_Symbol)
#define lisp_h_VECTORLIKEP(x) (XTYPE (x) == Lisp_Vectorlike)
#define lisp_h_XCAR(c) XCONS (c)->car
#define lisp_h_XCDR(c) XCONS (c)->u.cdr
#define lisp_h_XCONS(a) \
   (eassert (CONSP (a)), (struct Lisp_Cons *) XUNTAG (a, Lisp_Cons))
#define lisp_h_XHASH(a) XUINT (a)
#define lisp_h_XPNTR(a) \
   (SYMBOLP (a) ? XSYMBOL (a) : (void *) ((intptr_t) (XLI (a) & VALMASK)))
#ifndef GC_CHECK_CONS_LIST
# define lisp_h_check_cons_list() ((void) 0)
#endif
#if USE_LSB_TAG
# define lisp_h_make_number(n) \
    XIL ((EMACS_INT) (((EMACS_UINT) (n) << INTTYPEBITS) + Lisp_Int0))
# define lisp_h_XFASTINT(a) XINT (a)
# define lisp_h_XINT(a) (XLI (a) >> INTTYPEBITS)
# define lisp_h_XSYMBOL(a) \
    (eassert (SYMBOLP (a)), \
     (struct Lisp_Symbol *) ((uintptr_t) XLI (a) - Lisp_Symbol \
                             + (char *) lispsym))
# define lisp_h_XTYPE(a) ((enum Lisp_Type) (XLI (a) & ~VALMASK))
# define lisp_h_XUNTAG(a, type) ((void *) (intptr_t) (XLI (a) - (type)))
#endif

/* When compiling via gcc -O0, define the key operations as macros, as
   Emacs is too slow otherwise.  To disable this optimization, compile
   with -DINLINING=false.  */
#if (defined __NO_INLINE__ \
     && ! defined __OPTIMIZE__ && ! defined __OPTIMIZE_SIZE__ \
     && ! (defined INLINING && ! INLINING))
# define XLI(o) lisp_h_XLI (o)
# define XIL(i) lisp_h_XIL (i)
# define CHECK_LIST_CONS(x, y) lisp_h_CHECK_LIST_CONS (x, y)
# define CHECK_NUMBER(x) lisp_h_CHECK_NUMBER (x)
# define CHECK_SYMBOL(x) lisp_h_CHECK_SYMBOL (x)
# define CHECK_TYPE(ok, predicate, x) lisp_h_CHECK_TYPE (ok, predicate, x)
# define CONSP(x) lisp_h_CONSP (x)
# define EQ(x, y) lisp_h_EQ (x, y)
# define FLOATP(x) lisp_h_FLOATP (x)
# define INTEGERP(x) lisp_h_INTEGERP (x)
# define MARKERP(x) lisp_h_MARKERP (x)
# define MISCP(x) lisp_h_MISCP (x)
# define NILP(x) lisp_h_NILP (x)
# define SET_SYMBOL_VAL(sym, v) lisp_h_SET_SYMBOL_VAL (sym, v)
# define SYMBOL_CONSTANT_P(sym) lisp_h_SYMBOL_CONSTANT_P (sym)
# define SYMBOL_VAL(sym) lisp_h_SYMBOL_VAL (sym)
# define SYMBOLP(x) lisp_h_SYMBOLP (x)
# define VECTORLIKEP(x) lisp_h_VECTORLIKEP (x)
# define XCAR(c) lisp_h_XCAR (c)
# define XCDR(c) lisp_h_XCDR (c)
# define XCONS(a) lisp_h_XCONS (a)
# define XHASH(a) lisp_h_XHASH (a)
# define XPNTR(a) lisp_h_XPNTR (a)
# ifndef GC_CHECK_CONS_LIST
#  define check_cons_list() lisp_h_check_cons_list ()
# endif
# if USE_LSB_TAG
#  define make_number(n) lisp_h_make_number (n)
#  define XFASTINT(a) lisp_h_XFASTINT (a)
#  define XINT(a) lisp_h_XINT (a)
#  define XSYMBOL(a) lisp_h_XSYMBOL (a)
#  define XTYPE(a) lisp_h_XTYPE (a)
#  define XUNTAG(a, type) lisp_h_XUNTAG (a, type)
# endif
#endif

/* Define NAME as a lisp.h inline function that returns TYPE and has
   arguments declared as ARGDECLS and passed as ARGS.  ARGDECLS and
   ARGS should be parenthesized.  Implement the function by calling
   lisp_h_NAME ARGS.  */
/* lisp.h:423 */
#define LISP_MACRO_DEFUN(name, type, argdecls, args) \
  INLINE type (name) argdecls { return lisp_h_##name args; }

/* Lisp integers use 2 tags, to give them one extra bit, thus
   extending their range from, e.g., -2^28..2^28-1 to -2^29..2^29-1.  */
/* lisp.h:437 */
#define INTMASK (EMACS_INT_MAX >> (INTTYPEBITS - 1))

/* Idea stolen from GDB.  Pedantic GCC complains about enum bitfields,
   MSVC doesn't support them, and xlc and Oracle Studio c99 complain
   vociferously about them.  */
/* lisp.h:443 */
#if (defined __STRICT_ANSI__ || defined _MSC_VER || defined __IBMC__ \
     || (defined __SUNPRO_C && __STDC__))
#define ENUM_BF(TYPE) unsigned int
#else
#define ENUM_BF(TYPE) enum TYPE
#endif

/* lisp.h:451 */
enum Lisp_Type
  {
    /* Symbol.  XSYMBOL (object) points to a struct Lisp_Symbol.  */
    Lisp_Symbol = 0,

    /* Miscellaneous.  XMISC (object) points to a union Lisp_Misc,
       whose first member indicates the subtype.  */
    Lisp_Misc = 1,

    /* Integer.  XINT (obj) is the integer value.  */
    Lisp_Int0 = 2,
    Lisp_Int1 = USE_LSB_TAG ? 6 : 3,

    /* String.  XSTRING (object) points to a struct Lisp_String.
       The length of the string, and its contents, are stored therein.  */
    Lisp_String = 4,

    /* Vector of Lisp objects, or something resembling it.
       XVECTOR (object) points to a struct Lisp_Vector, which contains
       the size and contents.  The size field also contains the type
       information, if it's not a real vector object.  */
    Lisp_Vectorlike = 5,

    /* Cons.  XCONS (object) points to a struct Lisp_Cons.  */
    Lisp_Cons = USE_LSB_TAG ? 3 : 6,

    Lisp_Float = 7
  };

/* This is the set of data types that share a common structure.
   The first member of the structure is a type code from this set.
   The enum values are arbitrary, but we'll use large numbers to make it
   more likely that we'll spot the error if a random word in memory is
   mistakenly interpreted as a Lisp_Misc.  */
/* lisp.h:485 */
enum Lisp_Misc_Type
  {
    Lisp_Misc_Free = 0x5eab,
    Lisp_Misc_Marker,
    Lisp_Misc_Overlay,
    Lisp_Misc_Save_Value,
    /* Currently floats are not a misc type,
       but let's define this in case we want to change that.  */
    Lisp_Misc_Float,
#ifdef HAVE_LTDL
    Lisp_Misc_Module,
#endif
    /* This is not a type code.  It is for range checking.  */
    Lisp_Misc_Limit
  };

/* If you want to define a new Lisp data type, here are some
   instructions.  See the thread at
   http://lists.gnu.org/archive/html/emacs-devel/2012-10/msg00561.html
   for more info.

   First, there are already a couple of Lisp types that can be used if
   your new type does not need to be exposed to Lisp programs nor
   displayed to users.  These are Lisp_Save_Value, a Lisp_Misc
   subtype; and PVEC_OTHER, a kind of vectorlike object.  The former
   is suitable for temporarily stashing away pointers and integers in
   a Lisp object.  The latter is useful for vector-like Lisp objects
   that need to be used as part of other objects, but which are never
   shown to users or Lisp code (search for PVEC_OTHER in xterm.c for
   an example).

   These two types don't look pretty when printed, so they are
   unsuitable for Lisp objects that can be exposed to users.

   To define a new data type, add one more Lisp_Misc subtype or one
   more pseudovector subtype.  Pseudovectors are more suitable for
   objects with several slots that need to support fast random access,
   while Lisp_Misc types are for everything else.  A pseudovector object
   provides one or more slots for Lisp objects, followed by struct
   members that are accessible only from C.  A Lisp_Misc object is a
   wrapper for a C struct that can contain anything you like.

   Explicit freeing is discouraged for Lisp objects in general.  But if
   you really need to exploit this, use Lisp_Misc (check free_misc in
   alloc.c to see why).  There is no way to free a vectorlike object.

   To add a new pseudovector type, extend the pvec_type enumeration;
   to add a new Lisp_Misc, extend the Lisp_Misc_Type enumeration.

   For a Lisp_Misc, you will also need to add your entry to union
   Lisp_Misc (but make sure the first word has the same structure as
   the others, starting with a 16-bit member of the Lisp_Misc_Type
   enumeration and a 1-bit GC markbit) and make sure the overall size
   of the union is not increased by your addition.

   For a new pseudovector, it's highly desirable to limit the size
   of your data type by VBLOCK_BYTES_MAX bytes (defined in alloc.c).
   Otherwise you will need to change sweep_vectors (also in alloc.c).

   Then you will need to add switch branches in print.c (in
   print_object, to print your object, and possibly also in
   print_preprocess) and to alloc.c, to mark your object (in
   mark_object) and to free it (in gc_sweep).  The latter is also the
   right place to call any code specific to your data type that needs
   to run when the object is recycled -- e.g., free any additional
   resources allocated for it that are not Lisp objects.  You can even
   make a pointer to the function that frees the resources a slot in
   your object -- this way, the same object could be used to represent
   several disparate C structures.  */

/* lisp.h:567 */
#ifdef CHECK_LISP_OBJECT_TYPE

typedef struct { EMACS_INT i; } Lisp_Object;

#define LISP_INITIALLY(i) {i}

#undef CHECK_LISP_OBJECT_TYPE
enum CHECK_LISP_OBJECT_TYPE { CHECK_LISP_OBJECT_TYPE = true };
#else /* CHECK_LISP_OBJECT_TYPE */

/* If a struct type is not wanted, define Lisp_Object as just a number.  */

typedef EMACS_INT Lisp_Object;
#define LISP_INITIALLY(i) (i)
enum CHECK_LISP_OBJECT_TYPE { CHECK_LISP_OBJECT_TYPE = false };
#endif /* CHECK_LISP_OBJECT_TYPE */
/* lisp.h:582 */

/* lisp.h:612 */
INLINE bool STRINGP (Lisp_Object);

/* lisp.h:621 */
INLINE void *(XUNTAG) (Lisp_Object, int);

/* lisp.h:649 */
enum symbol_redirect
{
  SYMBOL_PLAINVAL  = 4,
  SYMBOL_VARALIAS  = 1,
  SYMBOL_LOCALIZED = 2,
  SYMBOL_FORWARDED = 3
};

/* lisp.h:657 */
struct Lisp_Symbol
{
  bool_bf gcmarkbit : 1;

  /* Indicates where the value can be found:
     0 : it's a plain var, the value is in the `value' field.
     1 : it's a varalias, the value is really in the `alias' symbol.
     2 : it's a localized var, the value is in the `blv' object.
     3 : it's a forwarding variable, the value is in `forward'.  */
  ENUM_BF (symbol_redirect) redirect : 3;

  /* Non-zero means symbol is constant, i.e. changing its value
     should signal an error.  If the value is 3, then the var
     can be changed, but only by `defconst'.  */
  unsigned constant : 2;

  /* Interned state of the symbol.  This is an enumerator from
     enum symbol_interned.  */
  unsigned interned : 2;

  /* True means that this variable has been explicitly declared
     special (with `defvar' etc), and shouldn't be lexically bound.  */
  bool_bf declared_special : 1;

  /* True if pointed to from purespace and hence can't be GC'd.  */
  bool_bf pinned : 1;

  /* The symbol's name, as a Lisp string.  */
  Lisp_Object name;

  /* Value of the symbol or Qunbound if unbound.  Which alternative of the
     union is used depends on the `redirect' field above.  */
  union {
    Lisp_Object value;
    struct Lisp_Symbol *alias;
    struct Lisp_Buffer_Local_Value *blv;
    union Lisp_Fwd *fwd;
  } val;

  /* Function value of the symbol or Qnil if not fboundp.  */
  Lisp_Object function;

  /* The symbol's property list.  */
  Lisp_Object plist;

  /* Next symbol in obarray bucket, if the symbol is interned.  */
  struct Lisp_Symbol *next;
};

/* lisp.h:710 */
#define EXFUN(fnname, maxargs) \
  extern Lisp_Object fnname DEFUN_ARGS_ ## maxargs

/* Note that the weird token-substitution semantics of ANSI C makes
   this work for MANY and UNEVALLED.  */
#define DEFUN_ARGS_MANY         (ptrdiff_t, Lisp_Object *)
#define DEFUN_ARGS_UNEVALLED    (Lisp_Object)
#define DEFUN_ARGS_0    (void)
#define DEFUN_ARGS_1    (Lisp_Object)
#define DEFUN_ARGS_2    (Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_3    (Lisp_Object, Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_4    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_5    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, \
                         Lisp_Object)
#define DEFUN_ARGS_6    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, \
                         Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_7    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, \
                         Lisp_Object, Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_8    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, \
                         Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object)

/* Yield an integer that contains TAG along with PTR.  */
#define TAG_PTR(tag, ptr) \
  ((USE_LSB_TAG ? (tag) : (EMACS_UINT) (tag) << VALBITS) + (uintptr_t) (ptr))

/* Yield an integer that contains a symbol tag along with OFFSET.
   OFFSET should be the offset in bytes from 'lispsym' to the symbol.  */
/* lisp.h:737 */
#define TAG_SYMOFFSET(offset)                               \
  TAG_PTR (Lisp_Symbol,                                     \
           ((uintptr_t) (offset) >> (USE_LSB_TAG ? 0 : GCTYPEBITS)))


/* lisp.h:754 */
/* FIXME: need to define Qt etc, EXFUNs from globals.h */
struct Lisp_Symbol alignas (GCALIGNMENT) lispsym[1131];
#define iQnil 0
#define Qnil builtin_lisp_symbol (iQnil)
#define iQt 955
#define Qt builtin_lisp_symbol (iQt)
EXFUN (Fprovide, 2);


/* Convert a Lisp_Object to the corresponding EMACS_INT and vice versa.
   At the machine level, these operations are no-ops.  */
/* lisp.h:758 */
LISP_MACRO_DEFUN (XLI, EMACS_INT, (Lisp_Object o), (o))
LISP_MACRO_DEFUN (XIL, Lisp_Object, (EMACS_INT i), (i))

/* In a pseudovector, the size field actually contains a word with one
   PSEUDOVECTOR_FLAG bit set, and one of the following values extracted
   with PVEC_TYPE_MASK to indicate the actual type.  */
/* lisp.h:776 */
enum pvec_type
{
  PVEC_NORMAL_VECTOR,
  PVEC_FREE,
  PVEC_PROCESS,
  PVEC_FRAME,
  PVEC_WINDOW,
  PVEC_BOOL_VECTOR,
  PVEC_BUFFER,
  PVEC_HASH_TABLE,
  PVEC_TERMINAL,
  PVEC_WINDOW_CONFIGURATION,
  PVEC_SUBR,
  PVEC_OTHER,
  /* These should be last, check internal_equal to see why.  */
  PVEC_COMPILED,
  PVEC_CHAR_TABLE,
  PVEC_SUB_CHAR_TABLE,
  PVEC_FONT /* Should be last because it's used for range checking.  */
};

/* lisp.h:787 */
enum More_Lisp_Bits
  {
    /* For convenience, we also store the number of elements in these bits.
       Note that this size is not necessarily the memory-footprint size, but
       only the number of Lisp_Object fields (that need to be traced by GC).
       The distinction is used, e.g., by Lisp_Process, which places extra
       non-Lisp_Object fields at the end of the structure.  */
    PSEUDOVECTOR_SIZE_BITS = 12,
    PSEUDOVECTOR_SIZE_MASK = (1 << PSEUDOVECTOR_SIZE_BITS) - 1,

    /* To calculate the memory footprint of the pseudovector, it's useful
       to store the size of non-Lisp area in word_size units here.  */
    PSEUDOVECTOR_REST_BITS = 12,
    PSEUDOVECTOR_REST_MASK = (((1 << PSEUDOVECTOR_REST_BITS) - 1)
                              << PSEUDOVECTOR_SIZE_BITS),

    /* Used to extract pseudovector subtype information.  */
    PSEUDOVECTOR_AREA_BITS = PSEUDOVECTOR_SIZE_BITS + PSEUDOVECTOR_REST_BITS,
    PVEC_TYPE_MASK = 0x3f << PSEUDOVECTOR_AREA_BITS
  };

/* Mask for the value (as opposed to the type bits) of a Lisp object.  */
DEFINE_GDB_SYMBOL_BEGIN (EMACS_INT, VALMASK)
/* lisp.h:827 */
# define VALMASK (USE_LSB_TAG ? - (1 << GCTYPEBITS) : VAL_MAX)
DEFINE_GDB_SYMBOL_END (VALMASK)

/* lisp.h:833 */
#if USE_LSB_TAG

LISP_MACRO_DEFUN (make_number, Lisp_Object, (EMACS_INT n), (n))
LISP_MACRO_DEFUN (XINT, EMACS_INT, (Lisp_Object a), (a))
LISP_MACRO_DEFUN (XFASTINT, EMACS_INT, (Lisp_Object a), (a))
LISP_MACRO_DEFUN (XSYMBOL, struct Lisp_Symbol *, (Lisp_Object a), (a))
LISP_MACRO_DEFUN (XTYPE, enum Lisp_Type, (Lisp_Object a), (a))
LISP_MACRO_DEFUN (XUNTAG, void *, (Lisp_Object a, int type), (a, type))

#else /* ! USE_LSB_TAG */

/* Although compiled only if ! USE_LSB_TAG, the following functions
   also work when USE_LSB_TAG; this is to aid future maintenance when
   the lisp_h_* macros are eventually removed.  */

/* Make a Lisp integer representing the value of the low order
   bits of N.  */
INLINE Lisp_Object
make_number (EMACS_INT n)
{
  EMACS_INT int0 = Lisp_Int0;
  if (USE_LSB_TAG)
    {
      EMACS_UINT u = n;
      n = u << INTTYPEBITS;
      n += int0;
    }
  else
    {
      n &= INTMASK;
      n += (int0 << VALBITS);
    }
  return XIL (n);
}

/* Extract A's value as a signed integer.  */
INLINE EMACS_INT
XINT (Lisp_Object a)
{
  EMACS_INT i = XLI (a);
  if (! USE_LSB_TAG)
    {
      EMACS_UINT u = i;
      i = u << INTTYPEBITS;
    }
  return i >> INTTYPEBITS;
}

/* Like XINT (A), but may be faster.  A must be nonnegative.
   If ! USE_LSB_TAG, this takes advantage of the fact that Lisp
   integers have zero-bits in their tags.  */
INLINE EMACS_INT
XFASTINT (Lisp_Object a)
{
  EMACS_INT int0 = Lisp_Int0;
  EMACS_INT n = USE_LSB_TAG ? XINT (a) : XLI (a) - (int0 << VALBITS);
  eassert (0 <= n);
  return n;
}

/* Extract A's value as a symbol.  */
INLINE struct Lisp_Symbol *
XSYMBOL (Lisp_Object a)
{
  uintptr_t i = (uintptr_t) XUNTAG (a, Lisp_Symbol);
  if (! USE_LSB_TAG)
    i <<= GCTYPEBITS;
  void *p = (char *) lispsym + i;
  return p;
}

/* Extract A's type.  */
INLINE enum Lisp_Type
XTYPE (Lisp_Object a)
{
  EMACS_UINT i = XLI (a);
  return USE_LSB_TAG ? i & ~VALMASK : i >> VALBITS;
}

/* Extract A's pointer value, assuming A's type is TYPE.  */
INLINE void *
XUNTAG (Lisp_Object a, int type)
{
  intptr_t i = USE_LSB_TAG ? XLI (a) - type : XLI (a) & VALMASK;
  return (void *) i;
}

#endif /* ! USE_LSB_TAG */
/* lisp.h:920 */

/* lisp.h:975 */
INLINE struct Lisp_String *
XSTRING (Lisp_Object a)
{
  eassert (STRINGP (a));
  return XUNTAG (a, Lisp_String);
}

/* lisp.h:1061 */
INLINE Lisp_Object
make_lisp_symbol (struct Lisp_Symbol *sym)
{
  Lisp_Object a = XIL (TAG_SYMOFFSET ((char *) sym - (char *) lispsym));
  eassert (XSYMBOL (a) == sym);
  return a;
}

/* lisp.h:1070 */
INLINE Lisp_Object
builtin_lisp_symbol (int index)
{
  return make_lisp_symbol (lispsym + index);
}

/* See the macros in intervals.h.  */
/* lisp.h:1149 */
typedef struct interval *INTERVAL;

/* In a string or vector, the sign bit of the `size' is the gc mark bit.  */
/* lisp.h:1233 */
struct GCALIGNED Lisp_String
  {
    ptrdiff_t size;
    ptrdiff_t size_byte;
    INTERVAL intervals;         /* Text properties in this string.  */
    unsigned char *data;
  };

/* lisp.h:1285 */
INLINE unsigned char *
SDATA (Lisp_Object string)
{
  return XSTRING (string)->data;
}
INLINE char *
SSDATA (Lisp_Object string)
{
  /* Avoid "differ in sign" warnings.  */
  return (char *) SDATA (string);
}

/* Header of vector-like objects.  This documents the layout constraints on
   vectors and pseudovectors (objects of PVEC_xxx subtype).  It also prevents
   compilers from being fooled by Emacs's type punning: XSETPSEUDOVECTOR
   and PSEUDOVECTORP cast their pointers to struct vectorlike_header *,
   because when two such pointers potentially alias, a compiler won't
   incorrectly reorder loads and stores to their size fields.  See
   Bug#8546.  */
/* lisp.h:1344 */
struct vectorlike_header
  {
    /* The only field contains various pieces of information:
       - The MSB (ARRAY_MARK_FLAG) holds the gcmarkbit.
       - The next bit (PSEUDOVECTOR_FLAG) indicates whether this is a plain
         vector (0) or a pseudovector (1).
       - If PSEUDOVECTOR_FLAG is 0, the rest holds the size (number
         of slots) of the vector.
       - If PSEUDOVECTOR_FLAG is 1, the rest is subdivided into three fields:
         - a) pseudovector subtype held in PVEC_TYPE_MASK field;
         - b) number of Lisp_Objects slots at the beginning of the object
           held in PSEUDOVECTOR_SIZE_MASK field.  These objects are always
           traced by the GC;
         - c) size of the rest fields held in PSEUDOVECTOR_REST_MASK and
           measured in word_size units.  Rest fields may also include
           Lisp_Objects, but these objects usually needs some special treatment
           during GC.
         There are some exceptions.  For PVEC_FREE, b) is always zero.  For
         PVEC_BOOL_VECTOR and PVEC_SUBR, both b) and c) are always zero.
         Current layout limits the pseudovectors to 63 PVEC_xxx subtypes,
         4095 Lisp_Objects in GC-ed area and 4095 word-sized other slots.  */
    ptrdiff_t size;
  };

/* This structure describes a built-in function.
   It is generated by the DEFUN macro only.
   defsubr makes it into a Lisp object.  */
/* lisp.h:1670 */
struct Lisp_Subr
  {
    struct vectorlike_header header;
    union {
      Lisp_Object (*a0) (void);
      Lisp_Object (*a1) (Lisp_Object);
      Lisp_Object (*a2) (Lisp_Object, Lisp_Object);
      Lisp_Object (*a3) (Lisp_Object, Lisp_Object, Lisp_Object);
      Lisp_Object (*a4) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
      Lisp_Object (*a5) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
      Lisp_Object (*a6) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
      Lisp_Object (*a7) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
      Lisp_Object (*a8) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
      Lisp_Object (*aUNEVALLED) (Lisp_Object args);
      Lisp_Object (*aMANY) (ptrdiff_t, Lisp_Object *);
    } function;
    short min_args, max_args;
    const char *symbol_name;
    const char *intspec;
    Lisp_Object doc;
  };

INLINE struct Lisp_Save_Value *XSAVE_VALUE (Lisp_Object);

/* Types of data which may be saved in a Lisp_Save_Value.  */

/* lisp.h:2045 */
enum
  {
    SAVE_UNUSED,
    SAVE_INTEGER,
    SAVE_FUNCPOINTER,
    SAVE_POINTER,
    SAVE_OBJECT
  };

/* Number of bits needed to store one of the above values.  */
enum { SAVE_SLOT_BITS = 3 };

/* Number of slots in a save value where save_type is nonzero.  */
enum { SAVE_VALUE_SLOTS = 4 };

/* Bit-width and values for struct Lisp_Save_Value's save_type member.  */

enum { SAVE_TYPE_BITS = SAVE_VALUE_SLOTS * SAVE_SLOT_BITS + 1 };

enum Lisp_Save_Type
  {
    SAVE_TYPE_INT_INT = SAVE_INTEGER + (SAVE_INTEGER << SAVE_SLOT_BITS),
    SAVE_TYPE_INT_INT_INT
      = (SAVE_INTEGER + (SAVE_TYPE_INT_INT << SAVE_SLOT_BITS)),
    SAVE_TYPE_OBJ_OBJ = SAVE_OBJECT + (SAVE_OBJECT << SAVE_SLOT_BITS),
    SAVE_TYPE_OBJ_OBJ_OBJ = SAVE_OBJECT + (SAVE_TYPE_OBJ_OBJ << SAVE_SLOT_BITS),
    SAVE_TYPE_OBJ_OBJ_OBJ_OBJ
      = SAVE_OBJECT + (SAVE_TYPE_OBJ_OBJ_OBJ << SAVE_SLOT_BITS),
    SAVE_TYPE_PTR_INT = SAVE_POINTER + (SAVE_INTEGER << SAVE_SLOT_BITS),
    SAVE_TYPE_PTR_OBJ = SAVE_POINTER + (SAVE_OBJECT << SAVE_SLOT_BITS),
    SAVE_TYPE_PTR_PTR = SAVE_POINTER + (SAVE_POINTER << SAVE_SLOT_BITS),
    SAVE_TYPE_FUNCPTR_PTR_OBJ
      = SAVE_FUNCPOINTER + (SAVE_TYPE_PTR_OBJ << SAVE_SLOT_BITS),

    /* This has an extra bit indicating it's raw memory.  */
    SAVE_TYPE_MEMORY = SAVE_TYPE_PTR_INT + (1 << (SAVE_TYPE_BITS - 1))
  };

/* Special object used to hold a different values for later use.

   This is mostly used to package C integers and pointers to call
   record_unwind_protect when two or more values need to be saved.
   For example:

   ...
     struct my_data *md = get_my_data ();
     ptrdiff_t mi = get_my_integer ();
     record_unwind_protect (my_unwind, make_save_ptr_int (md, mi));
   ...

   Lisp_Object my_unwind (Lisp_Object arg)
   {
     struct my_data *md = XSAVE_POINTER (arg, 0);
     ptrdiff_t mi = XSAVE_INTEGER (arg, 1);
     ...
   }

   If ENABLE_CHECKING is in effect, XSAVE_xxx macros do type checking of the
   saved objects and raise eassert if type of the saved object doesn't match
   the type which is extracted.  In the example above, XSAVE_INTEGER (arg, 2)
   and XSAVE_OBJECT (arg, 0) are wrong because nothing was saved in slot 2 and
   slot 0 is a pointer.  */

typedef void (*voidfuncptr) (void);

/* lisp.h:2110 */
struct Lisp_Save_Value
  {
    ENUM_BF (Lisp_Misc_Type) type : 16; /* = Lisp_Misc_Save_Value */
    bool_bf gcmarkbit : 1;
    unsigned spacer : 32 - (16 + 1 + SAVE_TYPE_BITS);

    /* V->data may hold up to SAVE_VALUE_SLOTS entries.  The type of
       V's data entries are determined by V->save_type.  E.g., if
       V->save_type == SAVE_TYPE_PTR_OBJ, V->data[0] is a pointer,
       V->data[1] is an integer, and V's other data entries are unused.

       If V->save_type == SAVE_TYPE_MEMORY, V->data[0].pointer is the address of
       a memory area containing V->data[1].integer potential Lisp_Objects.  */
    ENUM_BF (Lisp_Save_Type) save_type : SAVE_TYPE_BITS;
    union {
      void *pointer;
      voidfuncptr funcpointer;
      ptrdiff_t integer;
      Lisp_Object object;
    } data[SAVE_VALUE_SLOTS];
  };

/* Return the type of V's Nth saved value.  */
INLINE int
save_type (struct Lisp_Save_Value *v, int n)
{
  eassert (0 <= n && n < SAVE_VALUE_SLOTS);
  return (v->save_type >> (SAVE_SLOT_BITS * n) & ((1 << SAVE_SLOT_BITS) - 1));
}

/* Get and set the Nth saved pointer.  */

/* lisp.h:2143 */
INLINE void *
XSAVE_POINTER (Lisp_Object obj, int n)
{
  eassert (save_type (XSAVE_VALUE (obj), n) == SAVE_POINTER);
  return XSAVE_VALUE (obj)->data[n].pointer;
}

/* Define a built-in function for calling from Lisp.
 `lname' should be the name to give the function in Lisp,
    as a null-terminated C string.
 `fnname' should be the name of the function in C.
    By convention, it starts with F.
 `sname' should be the name for the C constant structure
    that records information on this function for internal use.
    By convention, it should be the same as `fnname' but with S instead of F.
    It's too bad that C macros can't compute this from `fnname'.
 `minargs' should be a number, the minimum number of arguments allowed.
 `maxargs' should be a number, the maximum number of arguments allowed,
    or else MANY or UNEVALLED.
    MANY means pass a vector of evaluated arguments,
         in the form of an integer number-of-arguments
         followed by the address of a vector of Lisp_Objects
         which contains the argument values.
    UNEVALLED means pass the list of unevaluated arguments
 `intspec' says how interactive arguments are to be fetched.
    If the string starts with a `(', `intspec' is evaluated and the resulting
    list is the list of arguments.
    If it's a string that doesn't start with `(', the value should follow
    the one of the doc string for `interactive'.
    A null string means call interactively with no arguments.
 `doc' is documentation for the user.  */

/* This version of DEFUN declares a function prototype with the right
   arguments, so we can catch errors with maxargs at compile-time.  */
/* lisp.h:2812 */
#ifdef _MSC_VER
#define DEFUN(lname, fnname, sname, minargs, maxargs, intspec, doc)     \
   Lisp_Object fnname DEFUN_ARGS_ ## maxargs ;                          \
   static struct Lisp_Subr alignas (GCALIGNMENT) sname =                \
   { { (PVEC_SUBR << PSEUDOVECTOR_AREA_BITS)                            \
       | (sizeof (struct Lisp_Subr) / sizeof (EMACS_INT)) },            \
      { (Lisp_Object (__cdecl *)(void))fnname },                        \
       minargs, maxargs, lname, intspec, 0};                            \
   Lisp_Object fnname
#else  /* not _MSC_VER */
#define DEFUN(lname, fnname, sname, minargs, maxargs, intspec, doc)     \
   static struct Lisp_Subr alignas (GCALIGNMENT) sname =                \
     { { PVEC_SUBR << PSEUDOVECTOR_AREA_BITS },                         \
       { .a ## maxargs = fnname },                                      \
       minargs, maxargs, lname, intspec, 0};                            \
   Lisp_Object fnname
#endif

/* defsubr (Sname);
   is how we define the symbol for function `name' at start-up time.  */
/* lisp.h:2839 */
extern void defsubr (struct Lisp_Subr *);

/* lisp.h:3777 */
extern Lisp_Object make_string (const char *, ptrdiff_t);

/* lisp.h:3881 */
extern Lisp_Object make_save_ptr (void *);

/* Defined in lread.c.  */
/* lisp.h:3960 */
extern Lisp_Object intern_1 (const char *, ptrdiff_t);

/* lisp.h:3987 */
INLINE Lisp_Object
intern (const char *str)
{
  return intern_1 (str, strlen (str));
}

/* end of file */

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 14 Feb 2015 19:02:48 -0600
> 
> I took a stab at creating emacs_module_api.h; attached.
> 
> To do this, I replaced "#include <lisp.h>"
> in modules/curl/curl.c with "#include <emacs_modules_api.h>", and copied
> stuff from lisp.h and other headers to emacs_module_api.h until it
> compiled. Note the FIXMEs; I could not figure out what was going on in
> some places, so I just kludged it for now.

Thanks.

I started to write comments to that file, but gave up in despair.  My
overall conclusion from reviewing it is that, if we want the
tightly-coupled version of modules, we might as well compile them as
any other C source file in the Emacs tree, i.e. we could simply
include lisp.h in its entirety, and all the other headers that it
pulls in.  I see no other workable alternative that would let modules
use Lisp objects transparently (as opposed to using them as opaque
objects), call Lisp primitives directly like we do in Emacs sources,
etc.

> >    This already presents at least 2 problems.  The first is 32- vs
> >    64-bit issues.  We need some way of checking that a 32-bit module
> >    isn't loaded by a 64-bit Emacs, and vice versa.

> Good point. That's a special case of "don't load a module compiled for
> processor X in an Emacs compiled for processor Y". gnu binutils has
> facilities for that.

Not sure how Binutils could help here.  AFAIK, dlopen and dlsym (and
their equivalents) don't invoke Binutils on any platform.

> >    The other problem is more serious: Emacs can be built with or
> >    without --check-lisp-object-type, and with or without --wide-int;
> >    how will the module know which implementation to use?
> 
> Similar check when the module is loaded. Which means we need some
> metainfo for each module, and a standard function to retrieve it.

That "metainfo", whatever it will be, will also have to support
compilation of modules: we need to know at compile time what values of
USE_LSB_TAG, ENABLE_CHECKING, WIDE_EMACS_INT, optimization flags,
etc. were used when Emacs was built, because those affect the layout
of objects inside Emacs and also the availability of certain functions
in the Emacs binary.

> >  . Functions to access values of Lisp objects.  We shouldn't rely on C
> >    macros like XINT and XWINDOW for that, because macros track too
> >    closely the internals of each object.  So I propose to use
> >    functions that will be exposed via the API.
> 
> If we follow Stefan's suggestion, then either this function approach is
> not viable, or we need another #ifdef for module vs emacs.

If we don't use the function approach, we can only have modules that
are tightly coupled with the version of Emacs for which they were
written and compiled.  We have no hope of binary compatibility with
other versions of Emacs, except by sheer luck, and even source-level
compatibility will be a challenge.

> > They are currently written as if the code were an integral part of
> > Emacs on one of the Emacs C files. Going that way means that modules
> > will have to be recompiled from sources for each Emacs version, and
> > practically will have to live in the Emacs tree. Maybe this is what we
> > want, I don't know.
> 
> This is the model I had in mind. Since I need max speed with mixed
> Ada/lisp code, I need tight integration with the core. The people who
> need that speed for Ada mode will simply have to recompile the module
> for each Emacs release; they are coders, so that's not a problem.

Being an Ada programmer (or even a C programmer) doesn't necessarily
mean you know how to fix compile- and link-time problems with Emacs.
You've just bumped into that yourself.

In practice, I think this means maintainers of modules will have to
adapt their modules to each Emacs version, and perhaps also keep
several branches, one each for every version they want to support.

> Rather than splitting out emacs_module_api.h from lisp.h, we could add
> __declspec(dllexport) to a subset of the functions in lisp.h; I suspect
> that will be a small number.

What for?  If we are going to expose most or all of lisp.h to modules,
we are actually saying that all of Emacs internals should be exposed
as well, so all of the lisp.h stuff should be externally visible, as
it is used all over the place in Emacs sources, and modules will want
to be able to do the same, under the "tightly-coupled" concept.

> Is there an equivalent on Linux (I don't recall ever seeing one)?

You want '__attribute__((visibility ("default")))', and you want to
use -fvisibility=hidden on the command line.  See

  https://gcc.gnu.org/wiki/Visibility

But this looks futile wrt lisp.h, see above.  Perhaps we could use
that for other public functions in other headers, but I expect that
paradigm to collapse very quickly, since modules will want to call
Lisp primitives directly, not through funcall.  Look at the modules
already on the branch: they include buffer.h and character.h, and call
stuff like Fgethash, Fputhash, and Fmake_hash_table.  How much time
will pass before we see modules call Finsert_file_contents,
Fmake_frame, and Fredisplay?  How can we explain to module developers
that they need to go through funcall?  And then "more internal"
internals will quickly follow: scan_words, for example, or
recenter_overlay_lists.  It's a slippery slope, but I don't see how we
could prevent the slip if we are willing to go with "tightly-coupled"
modules.

Re: Dynamic loading progress

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sat, 14 Feb 2015 19:02:48 -0600
>> 
>> I took a stab at creating emacs_module_api.h; attached.
>> 
>> To do this, I replaced "#include <lisp.h>"
>> in modules/curl/curl.c with "#include <emacs_modules_api.h>", and copied
>> stuff from lisp.h and other headers to emacs_module_api.h until it
>> compiled. Note the FIXMEs; I could not figure out what was going on in
>> some places, so I just kludged it for now.
>
> Thanks.
>
> I started to write comments to that file, but gave up in despair.  My
> overall conclusion from reviewing it is that, if we want the
> tightly-coupled version of modules, we might as well compile them as
> any other C source file in the Emacs tree, i.e. we could simply
> include lisp.h in its entirety, and all the other headers that it
> pulls in.  I see no other workable alternative that would let modules
> use Lisp objects transparently (as opposed to using them as opaque
> objects), call Lisp primitives directly like we do in Emacs sources,
> etc.

I agree.

>> >    This already presents at least 2 problems.  The first is 32- vs
>> >    64-bit issues.  We need some way of checking that a 32-bit module
>> >    isn't loaded by a 64-bit Emacs, and vice versa.
>
>> Good point. That's a special case of "don't load a module compiled for
>> processor X in an Emacs compiled for processor Y". gnu binutils has
>> facilities for that.
>
> Not sure how Binutils could help here.  AFAIK, dlopen and dlsym (and
> their equivalents) don't invoke Binutils on any platform.

I was thinking Emacs could invoke some binutils function to query the
.dll for this info, before calling dlopen.

>> >    The other problem is more serious: Emacs can be built with or
>> >    without --check-lisp-object-type, and with or without --wide-int;
>> >    how will the module know which implementation to use?
>> 
>> Similar check when the module is loaded. Which means we need some
>> metainfo for each module, and a standard function to retrieve it.
>
> That "metainfo", whatever it will be, will also have to support
> compilation of modules: we need to know at compile time what values of
> USE_LSB_TAG, ENABLE_CHECKING, WIDE_EMACS_INT, optimization flags,
> etc. were used when Emacs was built, because those affect the layout
> of objects inside Emacs and also the availability of certain functions
> in the Emacs binary.

Right.

One solution is to only distribute source, and provide an elisp function
'module-make' (suggested by Stefan) that provides command-line options
to a module makefile for all of those settings.

>> >  . Functions to access values of Lisp objects.  We shouldn't rely on C
>> >    macros like XINT and XWINDOW for that, because macros track too
>> >    closely the internals of each object.  So I propose to use
>> >    functions that will be exposed via the API.
>> 
>> If we follow Stefan's suggestion, then either this function approach is
>> not viable, or we need another #ifdef for module vs emacs.
>
> If we don't use the function approach, we can only have modules that
> are tightly coupled with the version of Emacs for which they were
> written and compiled.  We have no hope of binary compatibility with
> other versions of Emacs, except by sheer luck, and even source-level
> compatibility will be a challenge.

Right.

>> > They are currently written as if the code were an integral part of
>> > Emacs on one of the Emacs C files. Going that way means that modules
>> > will have to be recompiled from sources for each Emacs version, and
>> > practically will have to live in the Emacs tree. Maybe this is what we
>> > want, I don't know.
>> 
>> This is the model I had in mind. Since I need max speed with mixed
>> Ada/lisp code, I need tight integration with the core. The people who
>> need that speed for Ada mode will simply have to recompile the module
>> for each Emacs release; they are coders, so that's not a problem.
>
> Being an Ada programmer (or even a C programmer) doesn't necessarily
> mean you know how to fix compile- and link-time problems with Emacs.
> You've just bumped into that yourself.

Yes, and I'm becomming more sympathetic to your point of view :).

> In practice, I think this means maintainers of modules will have to
> adapt their modules to each Emacs version, and perhaps also keep
> several branches, one each for every version they want to support.

Yes, that was my plan. I do that now at the elisp level.

>> Rather than splitting out emacs_module_api.h from lisp.h, we could add
>> __declspec(dllexport) to a subset of the functions in lisp.h; I suspect
>> that will be a small number.
>
> What for?  

Mostly just to get a confirmed list of the functions that curl.c
currently calls. But I guess the link-time errors I started with is good
enough for that.

So let's take a stab at something closer to your approach.

I'm not familiar with the idioms for making an opaque type in C; would
that just be a pointer to void? something like:

typedef void*  Lisp_Object_Ptr;

-- 
-- Stephe

implementing curl module with opaque emacs types

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> So let's take a stab at something closer to your approach.
>
> I'm not familiar with the idioms for making an opaque type in C; would
> that just be a pointer to void? something like:
>
> typedef void*  Lisp_Object_Ptr;

Here is the list of Emacs functions that curl.c currently uses

    Fprovide
    SSDATA
    XSAVE_POINTER
    builtin_lisp_symbol (for Qnil)
    defsubr
    intern
    make_string

We can either implement these in a new version of emacs_module_api.h/.c,
or rewrite curl.c to not use them.

-- 
-- Stephe

Re: implementing curl module with opaque emacs types

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sun, 15 Feb 2015 13:27:40 -0600
> 
> Here is the list of Emacs functions that curl.c currently uses
> 
>     Fprovide
>     SSDATA
>     XSAVE_POINTER
>     builtin_lisp_symbol (for Qnil)
>     defsubr
>     intern
>     make_string

It also uses XSAVE_VALUE (called by XSAVE_POINTER) and save_type and
die, indirectly via eassert, when ENABLE_CHECKING is defined.

Btw, its use of make_string is incorrect: it should use
make_unibyte_string string and then decode it.  Which will require
more functions.  Likewise with SSDATA: the string that is used as a
URL should be encoded before it is passed to libcurl functions.

Re: implementing curl module with opaque emacs types

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> So let's take a stab at something closer to your approach.
>>
>> I'm not familiar with the idioms for making an opaque type in C; would
>> that just be a pointer to void? something like:
>>
>> typedef void*  Lisp_Object_Ptr;
>
> Here is the list of Emacs functions that curl.c currently uses
>
>     Fprovide
>     SSDATA
>     XSAVE_POINTER
>     builtin_lisp_symbol (for Qnil)
>     defsubr
>     intern
>     make_string
>
> We can either implement these in a new version of emacs_module_api.h/.c,
> or rewrite curl.c to not use them.

Attached is a first attempt at emacs_module_api.h for this. The only
change in curl.c is:

-#include <config.h>
-#include <lisp.h>
+/* #include <config.h> */
+/* #include <lisp.h> */
+#include "emacs_modules_api.h"


This gives compilation errors:

curl.c: In function 'init':
curl.c:113:13: error: 'Scurl_make' undeclared (first use in this function)
   defsubr (&Scurl_make);
             ^
curl.c:113:13: note: each undeclared identifier is reported only once for each function it appears in
curl.c:114:13: error: 'Scurl_fetch_url' undeclared (first use in this function)
   defsubr (&Scurl_fetch_url);
             ^
curl.c:115:13: error: 'Scurl_content' undeclared (first use in this function)
   defsubr (&Scurl_content);
             ^
curl.c:116:13: error: 'Scurl_free' undeclared (first use in this function)
   defsubr (&Scurl_free);
             ^
Makefile:14: recipe for target 'curl.o' failed

This is because I deleted the declaration of 'sname' from the DEFUN
macro, because I couldn't figure out how to implement it simply.

It seems EXFUN is redundant with the first part of DEFUN, so I'm not
sure why the EXFUN statments are in curl.c.

So we either need a lisp wrapper to declare the module functions, with
some way to lookup the EXFUN symbols to set the function values, or we
need to implement a C function (not macro) that creates the struct for
defsubr.

-- 
-- Stephe

[-- Attachment #2: emacs_modules_api.h --]
[-- Type: application/octet-stream, Size: 3889 bytes --]

/* Emacs modules API */

#ifdef WINDOWSNT
# ifdef emacs
   /* compiling emacs */
#  define EMACS_EXPORT  __declspec(dllexport)
# else
   /* compiling a module */
#  define EMACS_EXPORT  __declspec(dllimport)
# endif
#else
  /* non-Windows OS */
# define EMACS_EXPORT __attribute__((visibility ("default")))
#endif

#include <stdint.h>
#include <stdbool.h>

/* lisp.h:87 */
#ifndef EMACS_INT_MAX
# if INTPTR_MAX <= 0
#  error "INTPTR_MAX misconfigured"
# elif INTPTR_MAX <= INT_MAX >> NONPOINTER_BITS && !defined WIDE_EMACS_INT
typedef int EMACS_INT;
typedef unsigned int EMACS_UINT;
#  define EMACS_INT_MAX INT_MAX
#  define pI ""
# elif INTPTR_MAX <= LONG_MAX >> NONPOINTER_BITS && !defined WIDE_EMACS_INT
typedef long int EMACS_INT;
typedef unsigned long EMACS_UINT;
#  define EMACS_INT_MAX LONG_MAX
#  define pI "l"
/* Check versus LLONG_MAX, not LLONG_MAX >> NONPOINTER_BITS.
   In theory this is not safe, but in practice it seems to be OK.  */
# elif INTPTR_MAX <= LLONG_MAX
typedef long long int EMACS_INT;
typedef unsigned long long int EMACS_UINT;
#  define EMACS_INT_MAX LLONG_MAX
#  define pI "ll"
# else
#  error "INTPTR_MAX too large"
# endif
#endif

/* lisp.h:567 */
#ifdef CHECK_LISP_OBJECT_TYPE

typedef struct { EMACS_INT i; } Lisp_Object;

#define LISP_INITIALLY(i) {i}

#undef CHECK_LISP_OBJECT_TYPE
enum CHECK_LISP_OBJECT_TYPE { CHECK_LISP_OBJECT_TYPE = true };
#else /* CHECK_LISP_OBJECT_TYPE */

/* If a struct type is not wanted, define Lisp_Object as just a number.  */

typedef EMACS_INT Lisp_Object;
#define LISP_INITIALLY(i) (i)
enum CHECK_LISP_OBJECT_TYPE { CHECK_LISP_OBJECT_TYPE = false };
#endif /* CHECK_LISP_OBJECT_TYPE */
/* lisp.h:582 */

/* lisp.h:612 */
EMACS_EXPORT bool STRINGP (Lisp_Object);

/* lisp.h:710 */
#define EXFUN(fnname, maxargs) \
  extern Lisp_Object fnname DEFUN_ARGS_ ## maxargs

/* Note that the weird token-substitution semantics of ANSI C makes
   this work for MANY and UNEVALLED.  */
#define DEFUN_ARGS_MANY         (ptrdiff_t, Lisp_Object *)
#define DEFUN_ARGS_UNEVALLED    (Lisp_Object)
#define DEFUN_ARGS_0    (void)
#define DEFUN_ARGS_1    (Lisp_Object)
#define DEFUN_ARGS_2    (Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_3    (Lisp_Object, Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_4    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_5    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, \
                         Lisp_Object)
#define DEFUN_ARGS_6    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, \
                         Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_7    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, \
                         Lisp_Object, Lisp_Object, Lisp_Object)
#define DEFUN_ARGS_8    (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, \
                         Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object)

/* lisp.h:754 */
/* FIXME: need to define Qt etc, EXFUNs from globals.h */
#define iQnil 0
#define Qnil builtin_lisp_symbol (iQnil)
#define iQt 955
#define Qt builtin_lisp_symbol (iQt)
EXFUN (Fprovide, 2);

/* lisp.h:1070 */
EMACS_EXPORT Lisp_Object builtin_lisp_symbol (int index);

/* lisp.h:1285 */
EMACS_EXPORT unsigned char *SDATA (Lisp_Object string);

EMACS_EXPORT char *SSDATA (Lisp_Object string);

/* lisp.h:2143 */
EMACS_EXPORT void *
XSAVE_POINTER (Lisp_Object obj, int n);

/* lisp.h:2812 */
/* FIXME: not clear how to declare sname here. */
#define DEFUN(lname, fnname, sname, minargs, maxargs, intspec, doc)     \
   Lisp_Object fnname DEFUN_ARGS_ ## maxargs ;\
   Lisp_Object fnname

/* lisp.h:2839 */
struct Lisp_Subr;
EMACS_EXPORT extern void defsubr (struct Lisp_Subr *);

/* lisp.h:3777 */
EMACS_EXPORT extern Lisp_Object make_string (const char *, ptrdiff_t);

/* lisp.h:3882 */
EMACS_EXPORT extern Lisp_Object make_save_ptr (void *);

/* lisp.h:3987 */
Lisp_Object
intern (const char *str);

/* end of file */

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sun, 15 Feb 2015 13:09:25 -0600
> 
> > That "metainfo", whatever it will be, will also have to support
> > compilation of modules: we need to know at compile time what values of
> > USE_LSB_TAG, ENABLE_CHECKING, WIDE_EMACS_INT, optimization flags,
> > etc. were used when Emacs was built, because those affect the layout
> > of objects inside Emacs and also the availability of certain functions
> > in the Emacs binary.
> 
> Right.
> 
> One solution is to only distribute source, and provide an elisp function
> 'module-make' (suggested by Stefan) that provides command-line options
> to a module makefile for all of those settings.

That's a beginning, but it cannot be all there is to, because modules
should be distributable in binary form as well, at least some Linux
distributions will probably want that.

> So let's take a stab at something closer to your approach.
> 
> I'm not familiar with the idioms for making an opaque type in C; would
> that just be a pointer to void? something like:
> 
> typedef void*  Lisp_Object_Ptr;

I'd go with

  typedef intmax_t Lisp_Object;

This should be wide enough to hold both a pointer and the widest
integer type, although I think its being wide enough for a pointer is
not guaranteed by the C standard.

Another suggestion, from Stefan, was to use EMACS_INT, but I'm not
sure we will be able to define it correctly without dragging in too
much stuff from lisp.h that is used there to define EMACS_INT.  Maybe
we should have a separate definition of EMACS_INT for modules.

Re: Dynamic loading progress

[Stefan Monnier]

> This is the model I had in mind.  Since I need max speed with mixed
> Ada/lisp code, I need tight integration with the core.  The people who

The non-tight integration will mostly impact uses of things like CONSP,
XCAR, XCDR, ... but I'm far from convinced that making those go through
function calls would slow you down enough to make much of a difference.

Of course, we should provide functions to get efficient access to the
buffer text, along the lines of INC_BOTH, and FETCH_CHAR.


        Stefan

Re: Dynamic loading progress

[Stephen Leake]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> This is the model I had in mind.  Since I need max speed with mixed
>> Ada/lisp code, I need tight integration with the core.  The people who
>
> The non-tight integration will mostly impact uses of things like CONSP,
> XCAR, XCDR, ... but I'm far from convinced that making those go through
> function calls would slow you down enough to make much of a
> difference.

Ok. I'm willing to go with a non-tight implementation and see how fast
it is.

> Of course, we should provide functions to get efficient access to the
> buffer text, along the lines of INC_BOTH, and FETCH_CHAR.

Yes, the parser does need efficient access to the buffer text. Although
one option is to keep the current elisp lexer, and only pass tokens to
the module parser. Lots of things to benchmark :).

-- 
-- Stephe

Re: Dynamic loading progress

[Stefan Monnier]

> Yes, the parser does need efficient access to the buffer text.

This is very important, IMHO, because that's one of the most compelling
reasons to use modules instead of subprocesses: it allows much faster
uptodate access to buffer's contents.


        Stefan

Re: Dynamic loading progress

[Ted Zlatanov]

On Mon, 16 Feb 2015 14:02:45 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Yes, the parser does need efficient access to the buffer text.
SM> This is very important, IMHO, because that's one of the most compelling
SM> reasons to use modules instead of subprocesses: it allows much faster
SM> uptodate access to buffer's contents.

Perhaps we should move on with "pure data" modules and implement "buffer
interaction" modules later?  I see the utility, but would love to get
something basic in place for the common cases like parsing data.

Ted

Re: Dynamic loading progress

[Stefan Monnier]

> Perhaps we should move on with "pure data" modules and implement "buffer
> interaction" modules later?

It's definitely not a requirement before it can be installed into
master.  My criterion for installation into master is basically that the
probability we'll soon reimplement it differently is sufficiently low
(and for the code to be clean enough).


        Stefan

Re: Dynamic loading progress

[Ted Zlatanov]

On Tue, 21 Apr 2015 10:06:35 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Perhaps we should move on with "pure data" modules and implement "buffer
>> interaction" modules later?

SM> It's definitely not a requirement before it can be installed into
SM> master.  My criterion for installation into master is basically that the
SM> probability we'll soon reimplement it differently is sufficiently low
SM> (and for the code to be clean enough).

Thanks. I'll concentrate on testing the simple "pure data" cases and see
if a usable general data access model can emerge from that. If not,
we'll figure out what works...

Ted

Re: Dynamic loading progress

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > It looks like modules need:
 > 
 >  . A function to replace the DEFUN macro and the related defsubr call.

Aha, so you already figured this out for yourself.

 >  . Some definition of Lisp_Object
 > 
 >    This already presents at least 2 problems.  The first is 32- vs
 >    64-bit issues.  We need some way of checking that a 32-bit module
 >    isn't loaded by a 64-bit Emacs, and vice versa.
 > 
 >    The other problem is more serious: Emacs can be built with or
 >    without --check-lisp-object-type, and with or without --wide-int;
 >    how will the module know which implementation to use?

XEmacs builds a separate compiler driver ellcc that knows how to build
modules of the same configuration.

 >    Perhaps the solution is to make Lisp_Object an opaque data type for
 >    the modules, and not use the definition we have on lisp.h.

I don't know how you identify type of objects, but since you only have
a couple of tag bits left, I have to assume you use a header word of
some kind.  In XEmacs the header word is a pointer to a table of
functions that define the common functionality of Lisp objects used by
the interpreter and the virtual machines.  Whether a method table
pointer or an enum, that header word is pretty much the only thing
that's going to be common to all types.  Seems to me that's pretty
close to opaque already.

 >  . Functions to access values of Lisp objects.

Not values, attributes.  You don't want to just unbox the Lisp object
and give modules access to internals.  Rather, each object type should
export a specific API in the form of accessors and mutators for its
Lisp-visible attributes.

This is what XEmacs already does, it's what Python does, and it's
those two examples that give me the confidence that it will work for
Emacs too.  Sounds like it will be a pile of work, but you don't have
to convert all types immediately.

 >  . Functions to create Lisp objects and/or set their values.  Again,
 >    we shouldn't rely on the likes of XSETCONS.  There should be a
 >    function make_FOO for each object type FOO.  This includes creating
 >    symbols, so 'intern' should not be needed by modules.

`intern' *is* the creation function for symbols.  Uninterned symbols
are hardly useful for modules; they already have a private namespace.

 >  . A function to create a Lisp object that is a container for a C
 >    struct or pointer.
 > 
 > The above means that most of the C source files under modules/ should
 > be thoroughly rewritten.

Sounds like a good idea to me.  Wouldn't hurt if you rewrote most of
the C source under src/ that way too. ;-)

Re: Dynamic loading progress

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>     eggert@cs.ucla.edu,
>     emacs-devel@gnu.org
> Date: Sun, 15 Feb 2015 18:21:54 +0900
> 
>  >  . Some definition of Lisp_Object
>  > 
>  >    This already presents at least 2 problems.  The first is 32- vs
>  >    64-bit issues.  We need some way of checking that a 32-bit module
>  >    isn't loaded by a 64-bit Emacs, and vice versa.
>  > 
>  >    The other problem is more serious: Emacs can be built with or
>  >    without --check-lisp-object-type, and with or without --wide-int;
>  >    how will the module know which implementation to use?
> 
> XEmacs builds a separate compiler driver ellcc that knows how to build
> modules of the same configuration.

That won't support binary distributions of modules, would it?

>  >    Perhaps the solution is to make Lisp_Object an opaque data type for
>  >    the modules, and not use the definition we have on lisp.h.
> 
> I don't know how you identify type of objects, but since you only have
> a couple of tag bits left, I have to assume you use a header word of
> some kind.  In XEmacs the header word is a pointer to a table of
> functions that define the common functionality of Lisp objects used by
> the interpreter and the virtual machines.  Whether a method table
> pointer or an enum, that header word is pretty much the only thing
> that's going to be common to all types.  Seems to me that's pretty
> close to opaque already.

Making a Lisp_Object opaque, one way or the other, is easy.  Getting
people to accept that it will be opaque is harder.  I posted my
suggestion to try to convince that this is the way to go, but I'm not
holding my breath, frankly.

>  >  . Functions to access values of Lisp objects.
> 
> Not values, attributes.

I don't think the difference matters for the purposes of this
discussion.  The intent is to allow extracting a C int from Emacs
integer, a C double from a Lisp float, a C char array from a Lisp
string, a pointer to buffer text for buffers, etc.

> You don't want to just unbox the Lisp object and give modules access
> to internals.  Rather, each object type should export a specific API
> in the form of accessors and mutators for its Lisp-visible
> attributes.

I don't think modules will settle for using only the Lisp attributes,
they would like access to C attributes as well.

>  > The above means that most of the C source files under modules/ should
>  > be thoroughly rewritten.
> 
> Sounds like a good idea to me.  Wouldn't hurt if you rewrote most of
> the C source under src/ that way too. ;-)

I'm not holding my breath.

Re: Dynamic loading progress

[Aurélien Aptel]

On Tue, Feb 10, 2015 at 7:58 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> Though this raises a related issue: as I understand it currently modules
> must never refer to any Qxxx symbols (other than Qnil), since these symbols
> may have different values in different Emacs implementations.  And this
> includes any macros or inline functions the modules may invoke.  For
> example, a module must never call 'functionp', since 'functionp' is an
> inline function that refers to Qt.

I get it now, thanks :)

I understand you made this change for performance reasons. Was it
really a bottleneck though? I don't want to undermine your work and I
probably don't know enough to begin with so bear with me. I am under
the impression you implemented a clever change that wasn't critical
but made the module implementation a lot harder. Reverting it or
disabling it for modules might be a solution? What do you think?

Re: Dynamic loading progress

[Aurélien Aptel]

In other news, I stayed up waaay too late yesterday but I got the
destructor thing working! It's pushed on my branch.

I added a new Lisp_Misc subtype called Lisp_Module which stores:
- a module id
- a void* user pointer
- a function pointer to a destructor to release the resources of the
user pointer

There's a module_make_id() function that module can used to get a new
unique id and a module_make_object(id, dtor, userp) that allocates a
new module Lisp_Object.

The destructor is called in by the GC in sweep_misc(), which sweeps
the marker list (all misc subtypes use the marker list, maybe this
should be documented somewhere).

I've added a new source file for module functions in src/module.c.
I've tried to ifdef'd everything properly: --without-ltdl has no
runtime cost, as it should be.

I have made an example module that uses this new lisp type in
modules/memtest along with a ERT test.

You can browse the commit here [1], it's relatively easy to follow.

One thing I haven't checked is whether adding the Lisp_Module to the
Lisp_Misc union changes its size (the existing comments say this is
important).

1: https://github.com/aaptel/emacs-dynamic-module/commit/c59f2deaae99ca85b0a4fcdd53a3d8ed41d995cd

Re: Dynamic loading progress

[Aurélien Aptel]

On Wed, Feb 11, 2015 at 10:43 AM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:
> There's a module_make_id() function that module can used to get a new
> unique id and a module_make_object(id, dtor, userp) that allocates a
> new module Lisp_Object.

I should clarify this. A module is expected to use the same id for all
its objects. It will be used to make per-module predicates and
probably other things.

Re: Dynamic loading progress

[Stefan Monnier]

> I added a new Lisp_Misc subtype called Lisp_Module which stores:
> - a module id
> - a void* user pointer
> - a function pointer to a destructor to release the resources of the
> user pointer

I think I got confused by the name: these objects don't represent
modules, right (despite the name of their type)?  They are objects
created by a particular module (stored in the module-id).

> One thing I haven't checked is whether adding the Lisp_Module to the
> Lisp_Misc union changes its size (the existing comments say this is
> important).

The Lisp_Misc union is for objects with up to 5 "words" (or 6 "words" if
you count the word that holds the header).  Your objects have only
2 "words" (or 3 if you count the header where you added the "id"), so
you're perfectly fine, with room to grow.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

On Wed, Feb 11, 2015 at 4:35 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> I think I got confused by the name: these objects don't represent
> modules, right (despite the name of their type)?  They are objects
> created by a particular module (stored in the module-id).

Correct. I started with Lisp_Module_Object but thought it was redundant.

We could make /module/ module objects too. Let's call them Module
Descriptors to avoid any confusions. It could be a struct containing
functions pointers for printing, moving, copying, freeing a module
object, ... Module objects could then use a pointer to this struct as
a module id.

I don't have a lot of experience implementing programming languages,
especially features like this. I don't want to rush things and end up
with a complex C++-like system so feel free to comment if you think
this is overengineering it.

> The Lisp_Misc union is for objects with up to 5 "words" (or 6 "words" if
> you count the word that holds the header).  Your objects have only
> 2 "words" (or 3 if you count the header where you added the "id"), so
> you're perfectly fine, with room to grow.

I'm not sure what "words" means here. Pointers are 64 bits wide on 64
bits systems, does this make a difference?

Re: Dynamic loading progress

[Stefan Monnier]

>> I think I got confused by the name: these objects don't represent
>> modules, right (despite the name of their type)?  They are objects
>> created by a particular module (stored in the module-id).
> Correct. I started with Lisp_Module_Object but thought it was redundant.

I think the undesirable part of this name is "module" ;-)

At least, AFAICT, while this feature is introduced so that modules can
use it, it is not directly dependent on modules.

So, we could call it something else like "Lisp_Opaque_finalized_Object"
(I'm sure you can all easily come up with an even longer name).

>> The Lisp_Misc union is for objects with up to 5 "words" (or 6 "words" if
>> you count the word that holds the header).  Your objects have only
>> 2 "words" (or 3 if you count the header where you added the "id"), so
>> you're perfectly fine, with room to grow.
> I'm not sure what "words" means here.

It means "the natural size of things on the particular system".

> Pointers are 64 bits wide on 64 bits systems,

Right, so "words" are 64bit on such a system.


        Stefan

Re: Dynamic loading progress

[Paul Eggert]

Aurélien Aptel wrote:

> disabling it for modules might be a solution?

Yes, that's the idea. A kludgey way to do that right now is to have your module 
do this:

#include <lisp.h>
#undef Qfoo
#undef Qbar
...

with an #undef for every Qfoo symbol your code uses either directly or 
indirectly via macros or inline functions.  Obviously we need something better 
than that, namely, we modify make-docfile.c so that instead of generating this:

#define Qfoo builtin_lisp_symbol (iQfoo)

it generates this:

#ifndef EMACS_MODULE
# define Qfoo builtin_lisp_symbol (iQfoo)
#endif

(and similarly for Qbar, etc.), and then you start your module this way:

#define EMACS_MODULE
#include <lisp.h>

or better, we package up an <emacs.h> file the way Stefan suggested.

Re: Dynamic loading progress

[Ted Zlatanov]

On Mon, 9 Feb 2015 01:04:44 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> Ok, what I've pushed on my branch is now working.

Thanks!  I pushed today's version to the Emacs repo in branch feature/aptel/dynamic-modules-rc3

AA> One last thing, I need a deep `equal' function that also works
AA> hash-tables (every key and values must be equal) in yaml/test.el to
AA> compare nested lisp objects. Do I need to hand roll this myself?

Your `yaml-equal' seems OK to me :)

Ted

Re: Dynamic loading progress

[Aurélien Aptel]

On Tue, Feb 10, 2015 at 3:56 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> Thanks!  I pushed today's version to the Emacs repo in branch feature/aptel/dynamic-modules-rc3

Thanks Ted. I remember some people were already having build issues
with rc2. Can you confirm you can build rc3 (and my last commit, while
we're at it)?

> Your `yaml-equal' seems OK to me :)

I had some help from bpalmer on #emacs, I personally don't speak CL loop :)

Re: Dynamic loading progress

[Ted Zlatanov]

On Wed, 11 Feb 2015 10:55:19 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Tue, Feb 10, 2015 at 3:56 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Thanks!  I pushed today's version to the Emacs repo in branch feature/aptel/dynamic-modules-rc3

AA> Thanks Ted. I remember some people were already having build issues
AA> with rc2. Can you confirm you can build rc3 (and my last commit, while
AA> we're at it)?

I pushed your latest commit defining the module structure as well.

I did "make bootstrap" but now get:

Symbol's function definition is void: defun

Can you explain how to test? I'm able to build the master branch, then
switched to yours and did "make maintainer-clean; make bootstrap" but
get that error. I didn't get this error when I tested your code last
time, but it seems like something I'm not cleaning up.  Sorry if this is
a FAQ.

I'd like to merge this as soon as possible, even if there are rough
edges, so we can start finding them. I think if I test this and it's OK,
we should coordinate the ChangeLogs and documentation as we clean things
up. It may make sense to squash some of the trivial commits as well.

Thanks for your great work.

Ted

Re: Dynamic loading progress

[Aurélien Aptel]

On Wed, Feb 11, 2015 at 5:05 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> I did "make bootstrap" but now get:
>
> Symbol's function definition is void: defun

I have no idea what is causing that :|
Sometimes I also get "Symbol is void: ^?" but it doesn't make the tests fail...

> Can you explain how to test? I'm able to build the master branch, then

Run ./modules/tests.py. It should work no matter the current working directory.

> switched to yours and did "make maintainer-clean; make bootstrap" but
> get that error. I didn't get this error when I tested your code last
> time, but it seems like something I'm not cleaning up.  Sorry if this is
> a FAQ.

I don't trust the nested polylingual spaghetti monster that is
autoconf and usually run

    ./autogen.sh && ./configure --with-ltdl && make -kj8

 after big changes just to make sure. I have a i7 CPU with 8 core at
home so it's ok :) (now if I had a SSD...)

> I'd like to merge this as soon as possible, even if there are rough
> edges, so we can start finding them. I think if I test this and it's OK,
> we should coordinate the ChangeLogs and documentation as we clean things
> up. It may make sense to squash some of the trivial commits as well.

Glad to hear it.

Re: Dynamic loading progress

[Ted Zlatanov]

On Wed, 11 Feb 2015 17:24:51 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Wed, Feb 11, 2015 at 5:05 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> I did "make bootstrap" but now get:
>> 
>> Symbol's function definition is void: defun

AA> I have no idea what is causing that :|

I think it's because --with-ltdl is not the default.  It should be,
right?  Modules are enabled by default?

AA>     ./autogen.sh && ./configure --with-ltdl && make -kj8

OK, so the problem is that my system (Ubuntu 14.10) has that library but
it's not autodetected.  What can I do to help you find the problem?

AA> Run ./modules/tests.py. It should work no matter the current working directory.

I got:

% ./modules/tests.py

[*] testing module yaml...
[*] building module...
gcc -ggdb3 -Wall -I../../src -I../../lib `pkg-config yaml-0.1 --cflags` -fPIC -c yaml.c
gcc -shared `pkg-config yaml-0.1 --libs` -o yaml.so yaml.o
../../lib-src/make-docfile yaml.c > yaml.doc
rm yaml.o
[*] running test.el...
Eager macro-expansion failure: (void-variable )
Running 5 tests (2015-02-11 16:13:26-0500)
/home/tzz/source/emacs/emacs/src/emacs: symbol lookup error: /home/tzz/source/emacs/emacs/modules/yaml/yaml.so: undefined symbol: yaml_parser_initialize
[!] yaml tests failed

I have libyaml-dev installed, but `pkg-config yaml-0.1 --cflags` returns
nothing.  That may be expected--I don't see a yaml.pc anywhere.

Ted

Re: Dynamic loading progress

[Aurélien Aptel]

On Wed, Feb 11, 2015 at 10:17 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> I think it's because --with-ltdl is not the default.  It should be,
> right?  Modules are enabled by default?

They used to but Eli or Stefan said it should be off by default as
it's still experimental.

> AA>     ./autogen.sh && ./configure --with-ltdl && make -kj8
>
> OK, so the problem is that my system (Ubuntu 14.10) has that library but
> it's not autodetected.  What can I do to help you find the problem?

http://diobla.info/code/load-example/
Can you make this work? If you had to change the flags, which one did you use?

As an alternative solution, we could bundle ltdl with emacs:

http://www.gnu.org/software/libtool/manual/html_node/Distributing-libltdl.html

> /home/tzz/source/emacs/emacs/src/emacs: symbol lookup error: /home/tzz/source/emacs/emacs/modules/yaml/yaml.so: undefined symbol: yaml_parser_initialize
> [!] yaml tests failed
>
> I have libyaml-dev installed, but `pkg-config yaml-0.1 --cflags` returns
> nothing.  That may be expected--I don't see a yaml.pc anywhere.

This has more to do with --libs but it doesn't matter: if a package
doesn't exist pkg-config prints an error message.
Again, maybe try to find the right flags to build a sample libyaml
program? We can go from there.

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Wed, Feb 11, 2015 at 10:17 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> I think it's because --with-ltdl is not the default.  It should be,
>> right?  Modules are enabled by default?
>
> They used to but Eli or Stefan said it should be off by default as
> it's still experimental.

Which means the build is broken for the no modules config; that has to
be fixed before merging to master.

-- 
-- Stephe

Re: Dynamic loading progress

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 11 Feb 2015 17:24:51 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 
>
> AA> On Wed, Feb 11, 2015 at 5:05 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>> I did "make bootstrap" but now get:
>>> 
>>> Symbol's function definition is void: defun
>
> AA> I have no idea what is causing that :|
>
> I think it's because --with-ltdl is not the default.  It should be,
> right?  Modules are enabled by default?

-with-ltdl fixes the problem for me on Windows mingw64. More testing later.


-- 
-- Stephe

Re: Dynamic loading progress

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> On Wed, 11 Feb 2015 17:24:51 +0100 Aurélien Aptel
>> <aurelien.aptel+emacs@gmail.com> wrote:
>>
>> AA> On Wed, Feb 11, 2015 at 5:05 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>>> I did "make bootstrap" but now get:
>>>> 
>>>> Symbol's function definition is void: defun
>>
>> AA> I have no idea what is causing that :|
>>
>> I think it's because --with-ltdl is not the default.  It should be,
>> right?  Modules are enabled by default?
>
> -with-ltdl fixes the problem for me on Windows mingw64. More testing later.

When I run tests.py, I get:

$ ./tests.py
  File "./tests.py", line 47
    print ""
           ^
SyntaxError: invalid syntax

Mingw64 has python 3.3


I can also run this under Cygwin, which has python 2.7.8; it does not
give that error.

So I'm assuming this is a python 2/3 incompatibility; is there a simple
way to accomodate both?

-- 
-- Stephe

Re: Dynamic loading progress

[Davis Herring]

> $ ./tests.py
>   File "./tests.py", line 47
>     print ""
>            ^
> SyntaxError: invalid syntax
[...]
> So I'm assuming this is a python 2/3 incompatibility; is there a simple
> way to accomodate both?

print("") means the same thing in each.  For more complicated cases, one
can write "from __future__ import print_function" to make Python >=2.6
act like Python 3.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Dynamic loading progress

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> On Wed, 11 Feb 2015 17:24:51 +0100 Aurélien Aptel
>> <aurelien.aptel+emacs@gmail.com> wrote:
>>
>> AA> On Wed, Feb 11, 2015 at 5:05 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>>> I did "make bootstrap" but now get:
>>>> 
>>>> Symbol's function definition is void: defun
>>
>> AA> I have no idea what is causing that :|
>>
>> I think it's because --with-ltdl is not the default.  It should be,
>> right?  Modules are enabled by default?
>
> -with-ltdl fixes the problem for me on Windows mingw64. More testing later.

I had to add to options to the 'gcc' line in modules/curl/Makefile:

    -std=c99 -I$(ROOT)/nt/inc

Normally those would be added by configure. Perhaps they could be set
via CFLAGS by the perl script when on mingw64?

But then I get link errors:

stephe@takver$ make
gcc -shared `pkg-config libcurl --libs` -o curl.so curl.o
curl.o: In function `Fcurl_make':
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:55: undefined reference to `__imp_curl_easy_init'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:56: undefined reference to `make_save_ptr'
curl.o: In function `Fcurl_fetch_url':
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:67: undefined reference to `XSAVE_POINTER'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:69: undefined reference to `SSDATA'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:69: undefined reference to `__imp_curl_easy_setopt'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:70: undefined reference to `__imp_curl_easy_setopt'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:71: undefined reference to `__imp_curl_easy_setopt'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:72: undefined reference to `__imp_curl_easy_setopt'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:73: undefined reference to `__imp_curl_easy_perform'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:77: undefined reference to `__imp_curl_easy_strerror'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:78: undefined reference to `make_string'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:81: undefined reference to `builtin_lisp_symbol'
curl.o: In function `Fcurl_content':
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:89: undefined reference to `XSAVE_POINTER'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:90: undefined reference to `make_string'
curl.o: In function `Fcurl_free':
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:98: undefined reference to `XSAVE_POINTER'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:100: undefined reference to `__imp_curl_easy_cleanup'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:102: undefined reference to `builtin_lisp_symbol'
curl.o: In function `init':
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:107: undefined reference to `__imp_curl_global_init'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:110: undefined reference to `intern'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:112: undefined reference to `defsubr'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:113: undefined reference to `defsubr'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:114: undefined reference to `defsubr'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:115: undefined reference to `defsubr'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:117: undefined reference to `builtin_lisp_symbol'
C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:117: undefined reference to `Fprovide'
collect2.exe: error: ld returned 1 exit status
Makefile:9: recipe for target 'curl.so' failed
make: *** [curl.so] Error 1

pkg-config libcurl --libs returns -L/mingw64/lib -lcurl -lwldap32 -lz -lws2_32 

The "__imp_curl*" functions are provided by
/msys64/mingw64/lib/libcurl.dll.a; I assume that's what matches
'-lcurl'. So I don't understand those errors.

The other symbols appear to be Emacs symbols? There was some discussion
about a special flag for exporting those on Windows; is that '-Xlinker
-E'? I guess that goes in src/Makefile somewhere?

We also discussed creating an 'emacs.h' file to export those symbols
individually; I can work on that after I get the brute-force approach
working.

-- 
-- Stephe

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Thu, 12 Feb 2015 14:09:53 -0600
> 
> I had to add to options to the 'gcc' line in modules/curl/Makefile:
> 
>     -std=c99 -I$(ROOT)/nt/inc

Why do you need -std=c99?  What happens if you don't use it?  GCC
defaults to -std=gnu99, which is a superset of c99, so I don't
understand why you needed to insist on strict C99.

I also don't like the -I$(ROOT)/nt/inc thing, it should only be needed
for Emacs-specific system-level stuff.  Why was it needed here?

> Normally those would be added by configure. Perhaps they could be set
> via CFLAGS by the perl script when on mingw64?

Sorry, I don't understand: the Emacs configure script already adds
that directory to the include path.  So what's wrong?

> But then I get link errors:
> 
> stephe@takver$ make
> gcc -shared `pkg-config libcurl --libs` -o curl.so curl.o
> curl.o: In function `Fcurl_make':
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:55: undefined reference to `__imp_curl_easy_init'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:56: undefined reference to `make_save_ptr'
> curl.o: In function `Fcurl_fetch_url':
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:67: undefined reference to `XSAVE_POINTER'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:69: undefined reference to `SSDATA'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:69: undefined reference to `__imp_curl_easy_setopt'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:70: undefined reference to `__imp_curl_easy_setopt'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:71: undefined reference to `__imp_curl_easy_setopt'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:72: undefined reference to `__imp_curl_easy_setopt'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:73: undefined reference to `__imp_curl_easy_perform'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:77: undefined reference to `__imp_curl_easy_strerror'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:78: undefined reference to `make_string'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:81: undefined reference to `builtin_lisp_symbol'
> curl.o: In function `Fcurl_content':
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:89: undefined reference to `XSAVE_POINTER'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:90: undefined reference to `make_string'
> curl.o: In function `Fcurl_free':
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:98: undefined reference to `XSAVE_POINTER'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:100: undefined reference to `__imp_curl_easy_cleanup'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:102: undefined reference to `builtin_lisp_symbol'
> curl.o: In function `init':
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:107: undefined reference to `__imp_curl_global_init'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:110: undefined reference to `intern'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:112: undefined reference to `defsubr'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:113: undefined reference to `defsubr'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:114: undefined reference to `defsubr'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:115: undefined reference to `defsubr'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:117: undefined reference to `builtin_lisp_symbol'
> C:\Projects\emacs\feature\aptel\dynamic-modules-rc3\modules\curl/curl.c:117: undefined reference to `Fprovide'
> collect2.exe: error: ld returned 1 exit status
> Makefile:9: recipe for target 'curl.so' failed
> make: *** [curl.so] Error 1
> 
> pkg-config libcurl --libs returns -L/mingw64/lib -lcurl -lwldap32 -lz -lws2_32 
> 
> The "__imp_curl*" functions are provided by
> /msys64/mingw64/lib/libcurl.dll.a; I assume that's what matches
> '-lcurl'. So I don't understand those errors.

GNU ld is a one-pass linker, so the libraries should be after the
object files.

> The other symbols appear to be Emacs symbols? There was some discussion
> about a special flag for exporting those on Windows; is that '-Xlinker
> -E'? I guess that goes in src/Makefile somewhere?

The unresolved externals that are defined by Emacs are expected: you
need to link against an import library produced as part of building
emacs.exe.  Otherwise all this stuff will not work on Windows, because
the Windows port of the linker must see the import library at link
time.

The flag to produce the import library should be something like this:

   -Wl,--out-implib=libemacs.dll.a

Then you need to link the shared library against -lemacs.

And the link command line should produce curl.dll, not curl.so.

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Feb 12, 2015 at 9:34 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> Sorry, I don't understand: the Emacs configure script already adds
> that directory to the include path.  So what's wrong?

He's talking about the module Makefile, which is very basic and
standalone (no includes or preprocessing).


> The unresolved externals that are defined by Emacs are expected: you
> need to link against an import library produced as part of building
> emacs.exe.  Otherwise all this stuff will not work on Windows, because
> the Windows port of the linker must see the import library at link
> time.
>
> The flag to produce the import library should be something like this:
>
>    -Wl,--out-implib=libemacs.dll.a
>
> Then you need to link the shared library against -lemacs.

The configure script adds this to Emacs LD flags when modules are enabled:

    -lltdl -Wl,--export-dynamic

What does that do on Windows? I have very little experience in C
programming on Windows, especially using the GNU toolchain.

I want to make this work but this is unknown territory for me. Is
there an up-to-date guide to setup a simple build environment (libs
and all) for Emacs on Windows?

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Feb 12, 2015 at 10:51 PM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:
> I want to make this work but this is unknown territory for me. Is
> there an up-to-date guide to setup a simple build environment (libs
> and all) for Emacs on Windows?

Or even better, using Wine? Guess I'm asking for too much but
switching OS really is a PITA.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Thu, 12 Feb 2015 22:55:25 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> On Thu, Feb 12, 2015 at 10:51 PM, Aurélien Aptel
> <aurelien.aptel+emacs@gmail.com> wrote:
> > I want to make this work but this is unknown territory for me. Is
> > there an up-to-date guide to setup a simple build environment (libs
> > and all) for Emacs on Windows?
> 
> Or even better, using Wine? Guess I'm asking for too much but
> switching OS really is a PITA.

I never used Wine, but up front I don't see why nt/INSTALL won't DTRT
for that case as well.  AFAIK, Wine is just a Windows emulator, and
setting up for building Emacs certainly doesn't call for messing with
the OS ;-)

Re: Dynamic loading progress

[Stefan Monnier]

> Or even better, using Wine? Guess I'm asking for too much but
> switching OS really is a PITA.

FWIW, I have used Wine to *run* the w32 build of Emacs and it worked
well enough (not for real use, but to reproduce a particular bug and
come up with a fix).

I would expect that Wine can indeed similarly be used to build the
w32 version of Emacs.


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Thu, 12 Feb 2015 22:51:22 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> On Thu, Feb 12, 2015 at 9:34 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > Sorry, I don't understand: the Emacs configure script already adds
> > that directory to the include path.  So what's wrong?
> 
> He's talking about the module Makefile, which is very basic and
> standalone (no includes or preprocessing).

I don't really understand what that means, but I hope you will be able
to help him.

> > The unresolved externals that are defined by Emacs are expected: you
> > need to link against an import library produced as part of building
> > emacs.exe.  Otherwise all this stuff will not work on Windows, because
> > the Windows port of the linker must see the import library at link
> > time.
> >
> > The flag to produce the import library should be something like this:
> >
> >    -Wl,--out-implib=libemacs.dll.a
> >
> > Then you need to link the shared library against -lemacs.
> 
> The configure script adds this to Emacs LD flags when modules are enabled:
> 
>     -lltdl -Wl,--export-dynamic
> 
> What does that do on Windows?

Nothing useful in this case, AFAIK.  Reverse dynamic linking doesn't
work on Windows.

On Windows, when you link a shared library, the linker must be
presented with either all the dependency object files, or with
their import libraries.  That's because the linker records inside the
shared library data about the dependent object files and their names.

In this case, we need to provide this information about Emacs, since
(AFAIU) the module has unresolved externals that need to be
dynamically linked to the corresponding Emacs symbols.  So we need
either to link the module against emacs.exe, or against an import
library generated when Emacs is built.  The latter is accomplished by
using the --out-implib= option to the linker.

Btw, when using the -Wl,--out-implib=libemacs.dll.a switch on the link
command line, we will be by default exporting _all_ the externally
visible symbols.  That doesn't sound right (think DSO), and is even a
source of possible problems, since Emacs on Windows redirects quite a
few C library functions to its own implementations, and also adds some
C library functions unavailable in the MS runtime.  Making them
visible to modules might mean trouble for those modules, if they rely
on the standard or different implementations of those functions.  So
we should really have the emacs.h header file with only the functions
and variables we want to export, and on Windows we should mark those
exported functions and variables with dllexport attribute, so that
only they are exported.  (It is possible that we should also use some
visibility attributes on ELF platforms, for similar reasons.)

One other potential issue is with passing file names to modules.
Emacs on Windows pretends that file-name-coding-system is UTF-8, so
using ENCODE_FILE will produce a UTF-8 encoded file name that Windows
runtime cannot grok.  Therefore, the glue code, either in Lisp or in
C, between Emacs and the module will have to convert the file name to
the current system codepage (because most libraries only support
that on Windows).

> Is there an up-to-date guide to setup a simple build environment
> (libs and all) for Emacs on Windows?

Yes, see nt/INSTALL.

Re: Dynamic loading progress

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> I want to make this work but this is unknown territory for me. Is
> there an up-to-date guide to setup a simple build environment (libs
> and all) for Emacs on Windows?

Attached are my notes on installing msys2/mingw64 and building emacs.

-- 
-- Stephe

[-- Attachment #2: install_emacs_msys2.text --]
[-- Type: application/octet-stream, Size: 4601 bytes --]

notes on installing Emacs from source on Windows 7 using MinGW64 64 or 32 bit via Msys2.

Here we give sources for the various packages needed. We give
directions to the download, rather than an exact URL, to make it easy
to check for new versions.

These instructions work on a new computer, with only 64 bit Windows 7
installed.

Msys2 provides all of the tools required, and several of the packages.

These instructions are derived from http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/

There are more detailed instructions at http://sourceforge.net/p/msys2/tickets/32/?limit=10&page=1#216b

For more help, consult the mailing list at https://lists.sourceforge.net/lists/listinfo/msys2-users

01. MSYS2 install

    - if you have a 64 bit machine:

        From http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/ download the 64 bit installer

        run msys2-x86_64-20141113.exe

        install to the default install directory c:/msys64

    - if you have a 32 bit machine, the 32 bit installer (also works on a 64 bit machine, but is unnecessarily slower)

        run msys2-i686-20141113.exe

        install to the default install directory c:/msys32


    The rest of these instructions assume a 64 bit machine; modify the paths accordingly.

    The installer offers to "Run Msys 64/32bit now."; check that.

    That starts an msys shell.

    Later, to update safely:

    - exit all msys2 processes, start a new shell from Windows Explorer: c:\msys64\mingw64_shell.bat

    - In the msys shell:

        $ pacman -Sy
            # updates local package database from central repository

        $ pacman -S --needed \
        automake diffutils make autoconf pkg-config tar texinfo \
        mingw-w64-i686-gcc mingw-w64-i686-libpng mingw-w64-i686-libjpeg-turbo mingw-w64-i686-libtiff \
        mingw-w64-i686-giflib mingw-w64-i686-xpm-nox mingw-w64-i686-gnutls mingw-w64-i686-libxml2 \
        mingw-w64-i686-librsvg

02. gcc 64 bit toolchain install

    - in the mingw64 shell (started with c:\msys64\mingw64_shell.bat):

       $ pacman -S --needed \
        automake diffutils make autoconf pkg-config tar texinfo \
        mingw-w64-x86_64-gcc mingw-w64-x86_64-libpng mingw-w64-x86_64-libjpeg-turbo mingw-w64-x86_64-libtiff \
        mingw-w64-x86_64-giflib mingw-w64-x86_64-xpm-nox mingw-w64-x86_64-gnutls mingw-w64-x86_64-libxml2 \
        mingw-w64-x86_64-librsvg

02a. for debugging emacs:

    $ pacman -S gdb

03  install libtool

    msys only has an msys libtool package, not a mingw64 package

    download libtool-2.4.tar.xz from http://ftp.gnu.org

    in msys64:
    $ cd /usr/src
    $ tar xz libtool-2.4.tar.xz
    $ cd libtool-2.5
    $ ./configure --prefix=/mingw64
    $ make install

03a. build Emacs from source tarball

    download http://alpha.gnu.org/gnu/emacs/pretest/emacs-24.3.93.tar.xz

    $ mkdir /c/Projects
    $ cd /c/Projects
    $ tar xf emacs-24.3.93.tar.xz
    $ cd emacs-24.3.93
    $ ./configure --build=i686-pc-mingw32 --prefix=/mingw64 --with-ltdl
        # explicit --build not needed in emacs 25
    $ make

03b. build Emacs from source git repository

    we build out of tree, in case we want to also build with another compiler or options

    $ git clone ssh://<user>@git.sv.gnu.org/srv/git/emacs.git master
    $ cd master
    $ ./autogen.sh
    $ mkdir ../master-build-mingw64
    $ cd ../master-build-mingw64
    $ ../master/configure  --with-ltdl
    $ make

    to configure with debug optimization:
    $ export CFLAGS="-g3 -gdwarf-2 -O0" ; ../master/configure

    useful git links:
    http://www.emacswiki.org/emacs/GitForEmacsDevs
    https://savannah.gnu.org/git/?group=emacs

03c. run modules tests

    pacman does not have mingw64 libcurl
        download curl-7.40.0.tar.bz2 from http://curl.haxx.se/download.html

    in msys64:
    $ cd /usr/src
    $ tar xf curl-7.40.0.tar.bz2
    $ cd curl-7.40.0
    $ ./configure --prefix=/mingw64
    $ make install
    $ pacman -S python
    $ cd /c/Projects/emacs/master/modules
    $ ./tests.py

04. build Emacs docs

    this requires extra tools from cygwin (tex stuff not in msys2 yet):

    texinfo
    texinfo-tex
    texlive-collection-latexextra
    texlive-collection-latexrecommended
    texlive-collection-fontsrecommended

    However, Cygwin has texinfo 5, which is significantly slower than
    texinfo 4. Texinfo 4 is available at:

    https://sourceforge.net/projects/ezwinports/files/?source=navbar

Re: Dynamic loading progress

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Thu, 12 Feb 2015 14:09:53 -0600
>>
>> I had to add to options to the 'gcc' line in modules/curl/Makefile:
>>
>>     -std=c99 -I$(ROOT)/nt/inc
>
> Why do you need -std=c99?  What happens if you don't use it?

It complains about the "register" keyword in string.h stpcpy

>GCC
> defaults to -std=gnu99,

Apparently not in Mingw64

> I also don't like the -I$(ROOT)/nt/inc thing, it should only be needed
> for Emacs-specific system-level stuff.  Why was it needed here?

some file included ms-w32.h:

stephe@takver$ make
gcc -std=c99 -ggdb3 -Wall -I../../src -I../../lib `pkg-config libcurl --cflags` -fPIC -c curl.c
curl.c:1:0: warning: -fPIC ignored for target (all code is position independent)
 #include <stdio.h>
 ^
In file included from ../../src/config.h:1866:0,
                 from curl.c:6:
../../src/conf_post.h:32:27: fatal error: ms-w32.h: No such file or directory

>> Normally those would be added by configure. Perhaps they could be set
>> via CFLAGS by the perl script when on mingw64?
>
> Sorry, I don't understand: the Emacs configure script already adds
> that directory to the include path.  So what's wrong?

I'm not running the top level Emacs makefile; I'm running
modules/curl/Makefile, in order to build the curl module.

>> But then I get link errors:
>>
>> <snip>
>
> GNU ld is a one-pass linker, so the libraries should be after the
> object files.

Ah, right; I forgot about link order. That fixes the libcurl problem.
But the original order worked on Debian.

>> The other symbols appear to be Emacs symbols? There was some discussion
>> about a special flag for exporting those on Windows; is that '-Xlinker
>> -E'? I guess that goes in src/Makefile somewhere?
>
> The unresolved externals that are defined by Emacs are expected: you
> need to link against an import library produced as part of building
> emacs.exe.

Right, but I was expecting this makefile to do that. Especially since it
"just works" on Debian.

> Otherwise all this stuff will not work on Windows, because
> the Windows port of the linker must see the import library at link
> time.

Apparently that is not true on Debian.

> The flag to produce the import library should be something like this:
>
>    -Wl,--out-implib=libemacs.dll.a

And that goes in src/Makefile somewhere? Not on the "emacs$(EXEEXT)"
line, since that doesn't run gcc. So it must go on the temacs$(EXEEXT)
line (line 693)? And until we write emacs.h, it also needs -Xlinker -E
(or can that be -Wl,-E ?).

That should be added by configure when Windows and --with-ltdl are
specified. 

Hmm. We already have:

LIBLTDL = -lltdl -Wl,--export-dynamic

which is included in LIBES, and thus in the temacs.exe link. However,
that produces a warning:

C:/msys64/mingw64/lib/gcc/x86_64-w64-mingw32/4.9.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
warning: --export-dynamic is not supported for PE+ targets, did you mean
--export-all-symbols?

So apparently that should be changed to --export-all-symbols? Doing that
produces libemacs.dll.a

"--export-dynamic" is currently set in ./configure.ac line 3266; that
needs to be customized for the OS. Apparently the right way to do that
is with something like:

  if test "${opsys}" = "mingw64" || test "${opsys}" = "mingw32"; then
     #FIXME: should export only symbols defined in emacs.h (which doesn't exist yet)
     LIBLTDL="-lltdl -Wl,--export-all-symbols  -Wl,--out-implib=libemacs.dll.a"
  else
     LIBLTDL="-lltdl -Wl,--export-dynamic"
  fi


> Then you need to link the shared library against -lemacs.
>
> And the link command line should produce curl.dll, not curl.so.

Ok, that produced curl.dll. But doing:

src/emacs.exe -L modules/curl &
(require 'curl)

fails, with "eval: Cannot open load file: No such file or directory,
curl". That same test works on Debian.

Hmm. There is no curl/test.el, so this is _not_ designed to be part of
the test suite. Sigh. I'll work on the other module/* directories, which
do have test.el files.

So one question is how to make this Makefile do the right thing
for the current operating system. The intent is for this Makefile to be
part of the Emacs test suite, so it needs to not require user intervention.

One option is to rename modules/curl/Makefile to
modules/curl/Makefile.in and use configure in the modules directories.
Another is to set flags in modules/tests.py. I would prefer the former,
since that makes each Makefile self-documenting.

--
-- Stephe

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Fri, 13 Feb 2015 15:09:08 -0600
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Stephen Leake <stephen_leake@stephe-leake.org>
> >> Date: Thu, 12 Feb 2015 14:09:53 -0600
> >>
> >> I had to add to options to the 'gcc' line in modules/curl/Makefile:
> >>
> >>     -std=c99 -I$(ROOT)/nt/inc
> >
> > Why do you need -std=c99?  What happens if you don't use it?
> 
> It complains about the "register" keyword in string.h stpcpy
> 
> >GCC
> > defaults to -std=gnu99,
> 
> Apparently not in Mingw64

I don't think this is true, as that is a general feature of GCC
independent of the platform.  But in any case, -std=c99 is rarely what
you want, since it's a rare program that conforms to strict ANSI C99
spec.  Use -std=gnu99 instead.

> > I also don't like the -I$(ROOT)/nt/inc thing, it should only be needed
> > for Emacs-specific system-level stuff.  Why was it needed here?
> 
> some file included ms-w32.h:
> 
> stephe@takver$ make
> gcc -std=c99 -ggdb3 -Wall -I../../src -I../../lib `pkg-config libcurl --cflags` -fPIC -c curl.c
> curl.c:1:0: warning: -fPIC ignored for target (all code is position independent)
>  #include <stdio.h>
>  ^
> In file included from ../../src/config.h:1866:0,
>                  from curl.c:6:
> ../../src/conf_post.h:32:27: fatal error: ms-w32.h: No such file or directory

This is wrong on several levels.  First, the module shouldn't use
"-fPIC" unconditionally, as on some platforms, such as Windows, it
invokes a warning and is otherwise ignored.

More importantly, it is dead wrong for modules to include headers in
lib/, and in particular they should NEVER include Emacs's config.h.
That's because a config.h produced for a program or library should
never be used by another program or library -- it includes stuff
specific to that program/library.

On Windows, doing so will get you in trouble because ms-w32.h includes
all kinds of tricks and function redirections/redefinitions that will
most probably break any non-trivial code which interfaces with an
external library.  The above is an example of such trouble, and the
"solution" of add -I../nt -s a wrong solution.

So the question now becomes: what happens if you _remove_ the
"-I../../lib" option from the GCC command line?

If that doesn't work, then we really need an emacs.h file ASAP, and
the module should include only that file, without including <config.h>
inside that file.

> >> Normally those would be added by configure. Perhaps they could be set
> >> via CFLAGS by the perl script when on mingw64?
> >
> > Sorry, I don't understand: the Emacs configure script already adds
> > that directory to the include path.  So what's wrong?
> 
> I'm not running the top level Emacs makefile; I'm running
> modules/curl/Makefile, in order to build the curl module.
> 
> > GNU ld is a one-pass linker, so the libraries should be after the
> > object files.
> 
> Ah, right; I forgot about link order. That fixes the libcurl problem.
> But the original order worked on Debian.

On Debian, you don't need to link against the import library, that's
why.

> >> The other symbols appear to be Emacs symbols? There was some discussion
> >> about a special flag for exporting those on Windows; is that '-Xlinker
> >> -E'? I guess that goes in src/Makefile somewhere?
> >
> > The unresolved externals that are defined by Emacs are expected: you
> > need to link against an import library produced as part of building
> > emacs.exe.
> 
> Right, but I was expecting this makefile to do that. Especially since it
> "just works" on Debian.
> 
> > Otherwise all this stuff will not work on Windows, because
> > the Windows port of the linker must see the import library at link
> > time.
> 
> Apparently that is not true on Debian.

Yes, and I tried to explain why.  Dynamic linking on Posix systems
works differently: the reference to external symbols is left at zero,
and is resolved at run time.  On Windows, these references cannot be
left unresolved, so using the import library allows the linker to
resolve them to an "import symbol" which is just an indirect call
through a dispatch table.

> > The flag to produce the import library should be something like this:
> >
> >    -Wl,--out-implib=libemacs.dll.a
> 
> And that goes in src/Makefile somewhere? Not on the "emacs$(EXEEXT)"
> line, since that doesn't run gcc. So it must go on the temacs$(EXEEXT)
> line (line 693)?

Yes, it should be part of the temacs link command.

> And until we write emacs.h, it also needs -Xlinker -E (or can that
> be -Wl,-E ?).

I don't see what does a header file have to do with linking.  Why do
you think you need that option?

> That should be added by configure when Windows and --with-ltdl are
> specified. 

I'd imagine --with-modules, not --with-ltdl, but I didn't take a look
at the branch yet.

> Hmm. We already have:
> 
> LIBLTDL = -lltdl -Wl,--export-dynamic
> 
> which is included in LIBES, and thus in the temacs.exe link. However,
> that produces a warning:
> 
> C:/msys64/mingw64/lib/gcc/x86_64-w64-mingw32/4.9.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
> warning: --export-dynamic is not supported for PE+ targets, did you mean
> --export-all-symbols?
> 
> So apparently that should be changed to --export-all-symbols? Doing that
> produces libemacs.dll.a

No, using --export-all-symbols is wrong.  See my other message in this
thread.  We don't want to export all the Emacs symbols to the outside
world.  Please use -Wl,--out-implib= instead.

> > Then you need to link the shared library against -lemacs.
> >
> > And the link command line should produce curl.dll, not curl.so.
> 
> Ok, that produced curl.dll. But doing:
> 
> src/emacs.exe -L modules/curl &
> (require 'curl)
> 
> fails, with "eval: Cannot open load file: No such file or directory,
> curl".

That most probably means the module support stuff doesn't tell Emacs
where to find the library.  Or maybe it tries to load curl.so or
something.  I really don't know, you should debug this.

> That same test works on Debian.

GNU/Linux is what Aurélien tested this on, so it shouldn't surprise
you it works on GNU/Linux, but not on Windows.  IOW, it's not an
interesting data point.

> So one question is how to make this Makefile do the right thing
> for the current operating system. The intent is for this Makefile to be
> part of the Emacs test suite, so it needs to not require user intervention.

Indeed, this is part of the job of figuring out how to use modules in
Emacs.  Apparently, that part was not yet done.

> One option is to rename modules/curl/Makefile to
> modules/curl/Makefile.in and use configure in the modules directories.

Which configure? the Emacs configure?  That'd be a wrong solution,
IMO, as it will require the Emacs maintainers to know about modules in
order to DTRT for them in our mainline configure script.

So if Autoconf should be used to generate Makefile's in the modules,
it should be their own configure scripts.

> Another is to set flags in modules/tests.py.

That requires Python, which is an additional requirement for the build
environment.  I'd rather we used Emacs for that.

Alternatively, if the number of different settings is small, we could
assume GNU Make and use its conditional directives to cater to the few
different platforms.  That's much easier, but doesn't scale well to
large and complex modules.  But perhaps that's not a big deal, if we
assume modules will not be large and complex.

Re: Dynamic loading progress

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I also don't like the -I$(ROOT)/nt/inc thing, it should only be needed
>> > for Emacs-specific system-level stuff.  Why was it needed here?
>> 
>> some file included ms-w32.h:
>> 
>> stephe@takver$ make
>> gcc -std=c99 -ggdb3 -Wall -I../../src -I../../lib `pkg-config libcurl --cflags` -fPIC -c curl.c
>> curl.c:1:0: warning: -fPIC ignored for target (all code is position independent)
>>  #include <stdio.h>
>>  ^
>> In file included from ../../src/config.h:1866:0,
>>                  from curl.c:6:
>> ../../src/conf_post.h:32:27: fatal error: ms-w32.h: No such file or directory
>
> This is wrong on several levels.  First, the module shouldn't use
> "-fPIC" unconditionally, as on some platforms, such as Windows, it
> invokes a warning and is otherwise ignored.

Right. More reason to use configure.

> So the question now becomes: what happens if you _remove_ the
> "-I../../lib" option from the GCC command line?

gcc -std=gnu99 -ggdb3 -Wall -I../../src `pkg-config libcurl --cflags` -c curl.c
In file included from curl.c:8:0:
../../src/lisp.h:32:22: fatal error: intprops.h: No such file or directory
 #include <intprops.h>
                      ^
compilation terminated.


intprops.h is in lib; that explains why -I../../lib is needed.

> If that doesn't work, then we really need an emacs.h file ASAP, and
> the module should include only that file, without including <config.h>
> inside that file.

Nor <lisp.h>. And probably not lots of other things.

>> >    -Wl,--out-implib=libemacs.dll.a
>> 
>> And that goes in src/Makefile somewhere? Not on the "emacs$(EXEEXT)"
>> line, since that doesn't run gcc. So it must go on the temacs$(EXEEXT)
>> line (line 693)?
>
> Yes, it should be part of the temacs link command.
>
>> And until we write emacs.h, it also needs -Xlinker -E (or can that
>> be -Wl,-E ?).
>
> I don't see what does a header file have to do with linking.  

-Wl,-E is short for -Wl,--export-all-symbols; that makes all symbols
declared in Emacs visible to the module. (Similarly, --export-dynamic
makes all symbols visible on Debian).

I'm assuming emacs.h will include some annotation to make the symbols
declared in that file visible, so we won't need --export-all-symbols
(but more below).

>> That should be added by configure when Windows and --with-ltdl are
>> specified. 
>
> I'd imagine --with-modules, not --with-ltdl, but I didn't take a look
> at the branch yet.

It's currently --with-ltdl, but I agree --with-modules is a better
option name.


>> Hmm. We already have:
>> 
>> LIBLTDL = -lltdl -Wl,--export-dynamic
>> 
>> which is included in LIBES, and thus in the temacs.exe link. However,
>> that produces a warning:
>> 
>> C:/msys64/mingw64/lib/gcc/x86_64-w64-mingw32/4.9.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
>> warning: --export-dynamic is not supported for PE+ targets, did you mean
>> --export-all-symbols?
>> 
>> So apparently that should be changed to --export-all-symbols? Doing that
>> produces libemacs.dll.a
>
> No, using --export-all-symbols is wrong.  See my other message in this
> thread.  We don't want to export all the Emacs symbols to the outside
> world.  Please use -Wl,--out-implib= instead.

How does that know which symbols to export? The help for Gnu ld
--out-implib doesn't say.

Testing; apparently it exports all symbols - emacs.dll.a did not change
size. So --export-all-symbols is redundant, but I don't see how we are
going to restrict what symbols are available to the module.

Unless we are proposing to do that solely via .h files, not at the
linker level. I think that's better - compile time errors are easier to
understand than link time errors. We'll have to see how well that
works. 


>> So one question is how to make this Makefile do the right thing
>> for the current operating system. The intent is for this Makefile to be
>> part of the Emacs test suite, so it needs to not require user intervention.
>
> Indeed, this is part of the job of figuring out how to use modules in
> Emacs.  Apparently, that part was not yet done.
>
>> One option is to rename modules/curl/Makefile to
>> modules/curl/Makefile.in and use configure in the modules directories.
>
> Which configure? the Emacs configure?  That'd be a wrong solution,
> IMO, as it will require the Emacs maintainers to know about modules in
> order to DTRT for them in our mainline configure script.

Not modules in general, just the few that are in the Emacs test suite.

> So if Autoconf should be used to generate Makefile's in the modules,
> it should be their own configure scripts.

That is true for normal modules, yes.

>> Another is to set flags in modules/tests.py.
>
> That requires Python, which is an additional requirement for the build
> environment.  I'd rather we used Emacs for that.

Ok.

> Alternatively, if the number of different settings is small, we could
> assume GNU Make and use its conditional directives to cater to the few
> different platforms.  That's much easier, but doesn't scale well to
> large and complex modules.  But perhaps that's not a big deal, if we
> assume modules will not be large and complex.

Is there an acceptable (ie portable and reliable) way to determine the
OS in a Makefile? I'm not aware of one (short of Gnu config.guess).

-- 
-- Stephe

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 14 Feb 2015 09:13:03 -0600
> 
> > So the question now becomes: what happens if you _remove_ the
> > "-I../../lib" option from the GCC command line?
> 
> gcc -std=gnu99 -ggdb3 -Wall -I../../src `pkg-config libcurl --cflags` -c curl.c
> In file included from curl.c:8:0:
> ../../src/lisp.h:32:22: fatal error: intprops.h: No such file or directory
>  #include <intprops.h>
>                       ^
> compilation terminated.
> 
> 
> intprops.h is in lib; that explains why -I../../lib is needed.

No, it means including lisp.h without any changes will get you in
trouble.

> > If that doesn't work, then we really need an emacs.h file ASAP, and
> > the module should include only that file, without including <config.h>
> > inside that file.
> 
> Nor <lisp.h>. And probably not lots of other things.

None of the Emacs header files in src/ include config.h.  But they
might include other headers not appropriate for modules, like
intprops.h above.

This really is something that needs to be figured out and fixed before
the modules are announced operational, at least on MS-Windows.

> >> >    -Wl,--out-implib=libemacs.dll.a
> >> 
> >> And that goes in src/Makefile somewhere? Not on the "emacs$(EXEEXT)"
> >> line, since that doesn't run gcc. So it must go on the temacs$(EXEEXT)
> >> line (line 693)?
> >
> > Yes, it should be part of the temacs link command.
> >
> >> And until we write emacs.h, it also needs -Xlinker -E (or can that
> >> be -Wl,-E ?).
> >
> > I don't see what does a header file have to do with linking.  
> 
> -Wl,-E is short for -Wl,--export-all-symbols; that makes all symbols
> declared in Emacs visible to the module. (Similarly, --export-dynamic
> makes all symbols visible on Debian).
> 
> I'm assuming emacs.h will include some annotation to make the symbols
> declared in that file visible, so we won't need --export-all-symbols
> (but more below).

As soon as some symbols are declared for export, -Wl,--out-implib=
takes care to export only them, whereas -E will still export everything.

> > No, using --export-all-symbols is wrong.  See my other message in this
> > thread.  We don't want to export all the Emacs symbols to the outside
> > world.  Please use -Wl,--out-implib= instead.
> 
> How does that know which symbols to export? The help for Gnu ld
> --out-implib doesn't say.
> 
> Testing; apparently it exports all symbols - emacs.dll.a did not change
> size. So --export-all-symbols is redundant

Yes, that's why I said --export-all-symbols should not be used.  You
don't need it.

> but I don't see how we are going to restrict what symbols are
> available to the module.

The way to restrict the export is to declare the symbols you want to
export with __declspec(dllexport).  Then only those symbols will be
exported.

Note that, since emacs.h will be included by modules as well, we will
need something like

  #ifdef WINDOWS
  # ifdef emacs
  #  define EMACS_EXPORT  __declspec(dllexport)
  # else
  #  define EMACS_EXPORT  __declspec(dllimport)
  # endif
  # else
  #  define EMACS_EXPORT
  #endif

That's because they should be exported by Emacs, but imported by
modules.

> Is there an acceptable (ie portable and reliable) way to determine the
> OS in a Makefile? I'm not aware of one (short of Gnu config.guess).

Windows can be detected by looking at environment variables.  Since
Make converts them to Make variables, they are easily accessed.

Another possibility is to invoke "gcc -dumpmachine" via the $shell
function.  We could also invoke uname in a similar manner, but that
requires MSYS or Coreutils to be installed.  We could even invoke Make
itself with the --version option, and see what follows the "Built for"
header.

So ways to do that exist, we just need to decide whether we want to go
that way.

Re: Dynamic loading progress

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> but I don't see how we are going to restrict what symbols are
>> available to the module.
>
> The way to restrict the export is to declare the symbols you want to
> export with __declspec(dllexport).  Then only those symbols will be
> exported.
>
> Note that, since emacs.h will be included by modules as well, we will
> need something like
>
>   #ifdef WINDOWS
>   # ifdef emacs
>   #  define EMACS_EXPORT  __declspec(dllexport)
>   # else
>   #  define EMACS_EXPORT  __declspec(dllimport)
>   # endif
>   # else
>   #  define EMACS_EXPORT
>   #endif
>
> That's because they should be exported by Emacs, but imported by
> modules.

Right, that makes sense.

>> Is there an acceptable (ie portable and reliable) way to determine the
>> OS in a Makefile? I'm not aware of one (short of Gnu config.guess).
>
> Windows can be detected by looking at environment variables.  Since
> Make converts them to Make variables, they are easily accessed.
>
> Another possibility is to invoke "gcc -dumpmachine" via the $shell
> function.  We could also invoke uname in a similar manner, but that
> requires MSYS or Coreutils to be installed.  We could even invoke Make
> itself with the --version option, and see what follows the "Built for"
> header.
>
> So ways to do that exist, we just need to decide whether we want to go
> that way.

I've attempted to take that approach for other projects, and it isn't
very portable

-- 
-- Stephe

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

On 02/14/2015 05:04 PM, Stephen Leake wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> but I don't see how we are going to restrict what symbols are
>>> available to the module.
>>
>> The way to restrict the export is to declare the symbols you want to
>> export with __declspec(dllexport).  Then only those symbols will be
>> exported.
>>
>> Note that, since emacs.h will be included by modules as well, we will
>> need something like
>>
>>   #ifdef WINDOWS
>>   # ifdef emacs
>>   #  define EMACS_EXPORT  __declspec(dllexport)
>>   # else
>>   #  define EMACS_EXPORT  __declspec(dllimport)
>>   # endif
>>   # else
>>   #  define EMACS_EXPORT
>>   #endif
>>
>> That's because they should be exported by Emacs, but imported by
>> modules.
> 
> Right, that makes sense.

No it doesn't. Modules shouldn't be importing anything from Emacs: Emacs
should _give_ modules a table of all function pointers they need. It's a
lot less error-prone that way.

> 
>>> Is there an acceptable (ie portable and reliable) way to determine the
>>> OS in a Makefile? I'm not aware of one (short of Gnu config.guess).
>>
>> Windows can be detected by looking at environment variables.  Since
>> Make converts them to Make variables, they are easily accessed.
>>
>> Another possibility is to invoke "gcc -dumpmachine" via the $shell
>> function.  We could also invoke uname in a similar manner, but that
>> requires MSYS or Coreutils to be installed.  We could even invoke Make
>> itself with the --version option, and see what follows the "Built for"
>> header.
>>
>> So ways to do that exist, we just need to decide whether we want to go
>> that way.
> 
> I've attempted to take that approach for other projects, and it isn't
> very portable
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sun, 15 Feb 2015 05:50:07 -0800
> From: Daniel Colascione <dancol@dancol.org>
> 
> >>   #ifdef WINDOWS
> >>   # ifdef emacs
> >>   #  define EMACS_EXPORT  __declspec(dllexport)
> >>   # else
> >>   #  define EMACS_EXPORT  __declspec(dllimport)
> >>   # endif
> >>   # else
> >>   #  define EMACS_EXPORT
> >>   #endif
> >>
> >> That's because they should be exported by Emacs, but imported by
> >> modules.
> > 
> > Right, that makes sense.
> 
> No it doesn't. Modules shouldn't be importing anything from Emacs: Emacs
> should _give_ modules a table of all function pointers they need. It's a
> lot less error-prone that way.

I don't follow: how will the linker succeed in linking the module's
shared library, if you don't submit to it an import library with the
symbols in that table you mention?  And how do you suggest to produce
an import library, except by the above decorations?

What am I missing?

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

On 02/15/2015 09:00 AM, Eli Zaretskii wrote:
>> Date: Sun, 15 Feb 2015 05:50:07 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>>
>>>>   #ifdef WINDOWS
>>>>   # ifdef emacs
>>>>   #  define EMACS_EXPORT  __declspec(dllexport)
>>>>   # else
>>>>   #  define EMACS_EXPORT  __declspec(dllimport)
>>>>   # endif
>>>>   # else
>>>>   #  define EMACS_EXPORT
>>>>   #endif
>>>>
>>>> That's because they should be exported by Emacs, but imported by
>>>> modules.
>>>
>>> Right, that makes sense.
>>
>> No it doesn't. Modules shouldn't be importing anything from Emacs: Emacs
>> should _give_ modules a table of all function pointers they need. It's a
>> lot less error-prone that way.
> 
> I don't follow: how will the linker succeed in linking the module's
> shared library, if you don't submit to it an import library with the
> symbols in that table you mention?  And how do you suggest to produce
> an import library, except by the above decorations?
> 
> What am I missing?

Emacs doesn't provide symbols to modules. Modules provide an
initialization function to Emacs: Emacs then calls this initialization
function with a pointer to a C structure containing a table of function
pointers. Modules call these function pointers to do their work. There's
no need for Emacs itself to export any symbols.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sun, 15 Feb 2015 09:04:01 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
> Modules provide an initialization function to Emacs: Emacs then
> calls this initialization function with a pointer to a C structure
> containing a table of function pointers. Modules call these function
> pointers to do their work.

How will modules know which function in the table does what job?

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 763 bytes --]

On 02/15/2015 09:28 AM, Eli Zaretskii wrote:
>> Date: Sun, 15 Feb 2015 09:04:01 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
>>
>> Modules provide an initialization function to Emacs: Emacs then
>> calls this initialization function with a pointer to a C structure
>> containing a table of function pointers. Modules call these function
>> pointers to do their work.
> 
> How will modules know which function in the table does what job?

Positionally, as fixed in the structure that defines the table.

struct {
  void (*func1)(int);
  void (*func2)(double);
  /* etc. */
}

This way, the ABI Emacs exports is explicit, and breaking it is unlikely
to happen accidentally.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sun, 15 Feb 2015 09:31:12 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
> On 02/15/2015 09:28 AM, Eli Zaretskii wrote:
> >> Date: Sun, 15 Feb 2015 09:04:01 -0800
> >> From: Daniel Colascione <dancol@dancol.org>
> >> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> >>
> >> Modules provide an initialization function to Emacs: Emacs then
> >> calls this initialization function with a pointer to a C structure
> >> containing a table of function pointers. Modules call these function
> >> pointers to do their work.
> > 
> > How will modules know which function in the table does what job?
> 
> Positionally, as fixed in the structure that defines the table.
> 
> struct {
>   void (*func1)(int);
>   void (*func2)(double);
>   /* etc. */
> }
> 
> This way, the ABI Emacs exports is explicit, and breaking it is unlikely
> to happen accidentally.

I see.

Would you like to propose a list of functions that should be in that
table?  I don't mean function names, I mean a short description of
each one of them.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

On 02/15/2015 10:00 AM, Eli Zaretskii wrote:
>> Date: Sun, 15 Feb 2015 09:31:12 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
>>
>> On 02/15/2015 09:28 AM, Eli Zaretskii wrote:
>>>> Date: Sun, 15 Feb 2015 09:04:01 -0800
>>>> From: Daniel Colascione <dancol@dancol.org>
>>>> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
>>>>
>>>> Modules provide an initialization function to Emacs: Emacs then
>>>> calls this initialization function with a pointer to a C structure
>>>> containing a table of function pointers. Modules call these function
>>>> pointers to do their work.
>>>
>>> How will modules know which function in the table does what job?
>>
>> Positionally, as fixed in the structure that defines the table.
>>
>> struct {
>>   void (*func1)(int);
>>   void (*func2)(double);
>>   /* etc. */
>> }
>>
>> This way, the ABI Emacs exports is explicit, and breaking it is unlikely
>> to happen accidentally.
> 
> I see.
> 
> Would you like to propose a list of functions that should be in that
> table?  I don't mean function names, I mean a short description of
> each one of them.

That depends on the supported module use cases, which are being
discussed on other branches of the thread.

Keep in mind that my first preference is still to do everything with
CFFI and not export any Emacs-specific facilities at all.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sun, 15 Feb 2015 10:01:28 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
> > Would you like to propose a list of functions that should be in that
> > table?  I don't mean function names, I mean a short description of
> > each one of them.
> 
> That depends on the supported module use cases, which are being
> discussed on other branches of the thread.

You can easily look at the modules already on the branch, I think they
will give enough data points to at least start with this.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 12716 bytes --]

On 02/15/2015 10:29 AM, Eli Zaretskii wrote:
>> Date: Sun, 15 Feb 2015 10:01:28 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
>>
>>> Would you like to propose a list of functions that should be in that
>>> table?  I don't mean function names, I mean a short description of
>>> each one of them.
>>
>> That depends on the supported module use cases, which are being
>> discussed on other branches of the thread.
> 
> You can easily look at the modules already on the branch, I think they
> will give enough data points to at least start with this.

Here's a broad outline of what I have in mind.

We want an ABI powerful enough to let C modules interact with Emacs,
but decoupled enough to let the Emacs core evolve independently. The
best model we have for an interface like that is JNI. See the
specification at [1] for comparison.

The JNI designers carefully considered issues like error propagation,
GC integration, thread safety, type coercion, and so on, so let's
depart from the JNI model only where there's a good reason to do
so. We don't need all of JNI's complexity, but let's try to provide a
simple subset with an option to extend it later.

First, let's define a JNI-like entry point. Call it emacs_module_init:

  struct emacs_runtime;
  extern int emacs_module_init(struct emacs_runtime* ert);

When Emacs loads a module, it uses dlsym (or the platform equivalent)
to find this routine and call it. If it returns 0, the module loaded
successfully; otherwise, we report an error to the caller.

That's the limit of Emacs integration with the system dynamic
loader. ERT provides functions the module can use to do everything
else. We'll declare struct emacs_runtime ABI-stable; we can add fields
to the end of the struct without breaking binary compatibility.

Runtime environment
-------------------

Let's define the emacs_runtime structure like this:

  struct emacs_runtime {
    size_t size;
    struct emacs_env (*get_environment)(struct emacs_runtime* ert);
  };

The `size' member tells modules how long the emacs_runtime structure
is. (It's better to use size than an explicit version field: this way,
.size = sizeof(struct emacs_runtime) is always correct.)

Modules use `get_environment' to retrieve a pointer to an `emacs_env'
structure that lets them do interesting things; Emacs also supplied
module-registered callbacks with `emacs_env' pointers on entry.

Thread-local environments
-------------------------

The `get_environment' member lets us do anything else interesting. As
in Java, environments are thread-local. We only support one thread for
the moment, so this constraint is easy to enforce. (Just abort if we
use an emacs_env off the main thread.)

Now we can define emacs_env like this. (Ignore the forward definitions
for the moment.)

  typedef struct emacs_env_25 emacs_env;
  typedef struct emacs_value_tag* emacs_value;

  typedef emacs_value (*emacs_subr)(
    emacs_env* env,
    int nargs,
    emacs_value args[]);

  struct emacs_env {
    size_t size /* struct size */;

    emacs_value (*make_global_reference)(
     emacs_env* env,
     emacs_value any_reference);

    void (*free_global_reference)(
     emacs_env* env,
     emacs_value global_reference);

    bool (*error_check)(emacs_env* env);
    void (*clear_error)(emacs_env* env);
    bool (*get_error)(
      emacs_env* env,
      emacs_value* error_symbol_out,
      emacs_value* error_data_out);
    void (*signal_error)(
      emacs_env* env,
      emacs_value error_symbol,
      emacs_value error_data);

    emacs_value (*make_function)(
      emacs_env* env,
      int min_arity,
      int max_arity,
      emacs_subr function);

    emacs_value (*funcall)(
      emacs_env* env,
      emacs_value function,
      int nargs,
      emacs_value args[]);

    emacs_value (*intern)(
      emacs_env* env,
      const char* symbol_name);

    emacs_value (*type_of)(
      emacs_env* env,
      emacs_value value);

    int64_t (*fixnum_to_int)(
      emacs_env* env,
      emacs_value value);

    emacs_value (*make_fixnum)(
      emacs_env* env,
      int64_t value);

    double (*float_to_c_double)(
      emacs_env* env,
      emacs_value value);

    emacs_value (*make_float)(
      emacs_env* env,
      double value);

    bool (*copy_string_contents)(
      emacs_env* env,
      emacs_value value,
      char* buffer,
      size_* length_inout);

    emacs_value (*make_string)(
      emacs_env* env,
      const char* contents);
  };

That's a pretty bare-boned interface. If we want, we can add more
members for efficiency and functionality reasons. (Maybe car and cdr.)

emacs_env size
--------------

The first member in emacs_env is `size': that's just the size the
emacs_env structure provided by emacs to a module. In emacs.h, we'd
have something like this:

  struct emacs_env_25 { ... };
  struct emacs_env_26 { ... };
  ...;
  typedef struct emacs_env_26 emacs_env;

This way, modules will use the latest version by default, but can check
for a lesser version with something like this:

  if (env->size < sizeof(struct emacs_env_26)) {
    avoid_thing_that_works_only_in_emacs_26();
  }

Memory management
-----------------

Let's steal from JNI some more.

    emacs_value (*make_global_reference)(
     emacs_env* env,
     emacs_value any_reference);

    void (*free_global_reference)(
     emacs_env* env,
     emacs_value global_reference);

We'll represent all Lisp values as an opaque pointer typedef
emacs_value.  Each emacs_value is either a local or a global
reference.  Local references are valid only on the current thread and
only while the module function Emacs called is on the stack --- think
GCPRO.  Global references are valid indefinitely: each one is a GC
root.

Modules can use make_global_reference to allocate a global reference
(i.e., a GC root) for any emacs_value; modules must then free these
references explicitly.

All routines (except make_global_reference) that return emacs_value
values return local references.  It's up to modules to register
long-lived references explicitly.

This way, normal module code remains pretty efficient (since it deals
only with stack-allocated local references), but can retain arbitrary
data.

Also, we don't lock ourselves into conservative stack-scanning GC.

Error handling
--------------

We can't longjmp through arbitrary C code! We have to let modules
treat errors more conventionally.  Let's use the JNI approach:

    bool (*error_check)(emacs_env* env);
    void (*clear_error)(emacs_env* env);
    bool (*get_error)(
      emacs_env* env,
      emacs_value* error_symbol_out,
      emacs_value* error_data_out);
    void (*signal_error)(
      emacs_env* env,
      emacs_value error_symbol,
      emacs_value error_data);

We'll give each thread (initially, our only thread) a pending-error
information.

When Emacs calls a module function, the current thread's pending-error
flag will be clear.  When that module returns to Emacs, if the
thread's pending-error flag is set, Emacs signals the condition
corresponding to the current thread's error information.

When the module calls an Emacs routine that would ordinarily signal,
Emacs catches the signal at the stack frame just before control flow
would return to the module, sets the pending-error flag, and returns
to the module normally.  (Functions that would return a value on
success instead return a dummy value, like 0 or NULL.)

`error_check' just returns true iff an error is pending.
`clear_error' removes pending error information.  `get_error'
retrieves the pending error information, without clearing it, and
returns true.  If no error is set, it returns false.  signal_error
sets the current thread's pending-error flag.

To simplify the interface, we can treat `catch' and 'throw' as special
kinds of error.

Like JNI, we can just declare that it's illegal to call all but a few
specially-marked functions (like global reference deregistration) with
a pending error.

Function registration
---------------------

  typedef emacs_value (*emacs_subr)(
    emacs_env* env,
    int nargs,
    emacs_value args[]);

    emacs_value (*make_function)(
      emacs_env* env,
      int min_arity,
      int max_arity,
      emacs_subr function);

    emacs_value (*funcall)(
      emacs_env* env,
      emacs_value function,
      int nargs,
      emacs_value args[]);

    emacs_value (*intern)(
      emacs_env* env,
      const char* symbol_name);

Modules create function values using make_function; it works like
lambda; max_arity == -1 indicates a varargs function.  Modules can
register functions in the global namespace by calling a Lisp-level
function; we can also just provide a `defun' API, although strictly
speaking, one would be superfluous.

When Lisp calls a module-defined function object, Emacs calls the
emacs_subr callback with which the function was defined.

Modules can call Lisp functions using `funcall', which does the
obvious thing.  If Lisp signals or throws, `funcall' returns NULL.

`intern' also does the obvious thing.

Type coercion
-------------

emacs_value is not very useful without letting C look inside specific
types of value.

    emacs_value (*type_of)(
      emacs_env* env,
      emacs_value value);

Like Lisp type-of: returns a symbol.

    int64_t (*fixnum_to_int)(
      emacs_env* env,
      emacs_value value);

    emacs_value (*make_fixnum)(
      emacs_env* env,
      int64_t value);

These functions do the obvious thing.  They signal error on type
mismatch.  We use int64_t to handle big-integer Emacs variants on
32-bit platforms.

    double (*float_to_c_double)(
      emacs_env* env,
      emacs_value value);

    emacs_value (*make_float)(
      emacs_env* env,
      double value);

These functions do the obvious thing.  They signal on type mismatch.

    bool (*copy_string_contents)(
      emacs_env* env,
      emacs_value value,
      char* buffer,
      size_* length_inout);

    emacs_value (*make_string)(
      emacs_env* env,
      const char* contents);

These functions let C code access Lisp strings.  I imagine we'll
always produce and consume UTF-8.

`copy_string_contents' copies into a caller-allocated buffer instead
of returning a char* callers must free() --- this way, modules and the
Emacs core don't need to share the same C runtime.  We can deal with
the buffer-length issue in a number of ways: here, we just accept the
destination buffer size in *length_inout and write the total length of
the string to *length_inout on normal return.  We just truncate if
we're given too short a buffer and don't signal an error; this way,
callers can loop around and allocate a sufficiently large buffer for a
string's contents.

Other functionality
-------------------

I think the interface above is enough for complete functionality in a
module, but for efficiency, we might want to expose additional
facilities, like access to a unibyte buffer's raw representation.

Convenience library
---------------

This kind of interface is adequate, but asking callers to
env->intern() every Emacs facility they want to use is inconvenient.
It'd be nice to provide a statically-linked convenience library,
written in terms of the ABI-stable interface, that made it
straightforward to set text properties, visit files, and so on.

For example, we could provide a function like this:

bool
emacs_find_file(emacs_env* env, const char* filename)
{
  emacs_value e_filename = env->make_string(env, filename);
  if(env->error_check(env)) return false;
  emacs_value e_find_file = env->intern(env, "find-file");
  if(env->error_check(env)) return false;
  return env->funcall(env, e_find_file, &e_filename, 1) != NULL;
}

Questions
---------

1) Do we need JNI-style local variable frames and functions that
   release local references?

2) Maybe we want a separate, non-emacs_value type for global references?

3) How exactly do we represent catch/throw values?

4) Do we need to use int64_t for integers?

5) Do we need to provide direct access to string contents?

[1]
http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/jniTOC.html

[2] Let's not support JNI's deprecated implicit registration system,
where modules can register functions with the runtime by exporting
symbols with certain magical names.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

On Sun, Feb 15, 2015 at 9:20 PM, Daniel Colascione <dancol@dancol.org> wrote:
> 2) Maybe we want a separate, non-emacs_value type for global references?

What for?

> 4) Do we need to use int64_t for integers?

It's probably simpler that way.

> 5) Do we need to provide direct access to string contents?

Strings are already stored as mostly UTF-8, why would you not provide
direct access to that, at least read-only?

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Mon, 16 Feb 2015 15:05:49 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> > 5) Do we need to provide direct access to string contents?
> 
> Strings are already stored as mostly UTF-8, why would you not provide
> direct access to that, at least read-only?

It's a superset of UTF-8, yes.  However, the deviations are not
insignificant, especially for unibyte strings.  Dealing with their
data requires knowledge that most people don't have (and don't need,
IMO).

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sun, 15 Feb 2015 12:20:35 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
> Here's a broad outline of what I have in mind.

Thanks.  I think the next step is for someone to try rewriting a
couple of modules on the branch in terms of this specification, and
see if something is missing.  Volunteers are welcome.

Some comments on your write-up:

> When Emacs loads a module, it uses dlsym (or the platform equivalent)
> to find this routine and call it. If it returns 0, the module loaded
> successfully; otherwise, we report an error to the caller.

If this is all we need, then there's no need for libltdl, I think.

>   struct emacs_runtime {
>     size_t size;
>     struct emacs_env (*get_environment)(struct emacs_runtime* ert);
>   };
> 
> The `size' member tells modules how long the emacs_runtime structure
> is. (It's better to use size than an explicit version field: this way,
> .size = sizeof(struct emacs_runtime) is always correct.)

This approach requires us to change the size each time we change the
layout, even if the change itself leaves the size intact.  Using a
version field doesn't have this disadvantage.

> Thread-local environments
> -------------------------
> 
> The `get_environment' member lets us do anything else interesting. As
> in Java, environments are thread-local. We only support one thread for
> the moment, so this constraint is easy to enforce. (Just abort if we
> use an emacs_env off the main thread.)

Is it wise to design for threads at this point?  The only suggestion
on the table for "threads" is a far cry from real threads, and the
current Lisp interpreter is otherwise thread-unsafe.  Won't the thread
infrastructure add a lot of cruft that will go unneeded for the
observable future?

> We'll represent all Lisp values as an opaque pointer typedef
> emacs_value.  Each emacs_value is either a local or a global
> reference.  Local references are valid only on the current thread and
> only while the module function Emacs called is on the stack --- think
> GCPRO.  Global references are valid indefinitely: each one is a GC
> root.

With massive use of calls to Lisp functions from modules (since we
don't provide them with direct access in C to internals of many
objects), how can we ensure GC doesn't strike while the function that
created an emacs_value is still on the callstack?  You say "we don't
lock ourselves into conservative stack-scanning GC", which I interpret
as saying you don't want to rely on stack scanning to avoid a
destructive GC in this case.  But if we don't rely on that, where's
the guarantee that such emacs_value will survive GC?

> We'll represent all Lisp values as an opaque pointer typedef
> emacs_value.

This doesn't play well with --with-wide-int, where a value can be
wider than a pointer.  I think we should instead go with intmax_t or
inptr_t, whichever is wider on the host.

> Function registration
> ---------------------
> 
>   typedef emacs_value (*emacs_subr)(
>     emacs_env* env,
>     int nargs,
>     emacs_value args[]);
> 
>     emacs_value (*make_function)(
>       emacs_env* env,
>       int min_arity,
>       int max_arity,
>       emacs_subr function);

What about the doc string?

>     emacs_value (*funcall)(
>       emacs_env* env,
>       emacs_value function,
>       int nargs,
>       emacs_value args[]);

Shouldn't funcall use emacs_subr?

> Modules can register functions in the global namespace by calling a
> Lisp-level function

This is unclear, can you elaborate?  What happens if a function is not
"registered"? what's its status then?

> When Lisp calls a module-defined function object, Emacs calls the
> emacs_subr callback with which the function was defined.

This is a change in the Lisp interpreter, I think.  Why do we need
this?

> If Lisp signals or throws, `funcall' returns NULL.

I suggest some other value or indication of that.  NULL is a valid
return value, so usurping it for errors might be too harsh.

Or maybe I don't understand how will Lisp functions return values to
the module, under your suggestion.  Can you describe that?

> `intern' also does the obvious thing.

Do we need 'unintern' as well?

>     emacs_value (*type_of)(
>       emacs_env* env,
>       emacs_value value);
> 
> Like Lisp type-of: returns a symbol.

What is a "symbol", from the module's C code POV?  You show no
functions to access attributes of symbols, so it must be either one of
the other types, like an integer or a string, or a C primitive data
type, like a char * pointer.

>     int64_t (*fixnum_to_int)(
>       emacs_env* env,
>       emacs_value value);
> 
>     emacs_value (*make_fixnum)(
>       emacs_env* env,
>       int64_t value);
> 
> These functions do the obvious thing.  They signal error on type
> mismatch.  We use int64_t to handle big-integer Emacs variants on
> 32-bit platforms.

The last bit means we will need a utility function to return the valid
range of integers, so that modules can be written to support 32-bit
and 64-bit use cases without raising errors.

>     bool (*copy_string_contents)(
>       emacs_env* env,
>       emacs_value value,
>       char* buffer,
>       size_* length_inout);
> 
>     emacs_value (*make_string)(
>       emacs_env* env,
>       const char* contents);
> 
> These functions let C code access Lisp strings.  I imagine we'll
> always produce and consume UTF-8.

Strings in Emacs are of limited usability if you cannot encode and
decode them.  So this needs to be part of supported functionality, I
think.

More generally, modules that would like to process buffer or string
text will have to be able to deal with Emacs's internal encoding of
text, which means macros and functions we use in the core.  The
alternative of working only on UTF-8 encoded replica means we'd need
to encode and decode text across the module boundaries -- that's a lot
of consing.

> `copy_string_contents' copies into a caller-allocated buffer instead
> of returning a char* callers must free() --- this way, modules and the
> Emacs core don't need to share the same C runtime.  We can deal with
> the buffer-length issue in a number of ways: here, we just accept the
> destination buffer size in *length_inout and write the total length of
> the string to *length_inout on normal return.  We just truncate if
> we're given too short a buffer and don't signal an error; this way,
> callers can loop around and allocate a sufficiently large buffer for a
> string's contents.

That's an annoyance, IMO; why not provide a function to return the
required size?  Its implementation is trivial.

> I think the interface above is enough for complete functionality in a
> module, but for efficiency, we might want to expose additional
> facilities, like access to a unibyte buffer's raw representation.

I can envision a few additional missing bits:

 . direct access to buffer text (using buffer-substring means consing
   a lot of strings)
 . creation of opaque objects that should be returned to Lisp (like
   handles to objects managed by modules)
 . not sure how will a module "provide" its feature
 . some operations that must be efficient because they are typically
   done in inner loops, like regexp matching and accessing syntax of
   characters: doing that via Lisp will probably be horribly slow

> Convenience library
> ---------------

One thing that's inconvenient is the need to drag the environment
pointer through all the calls.  Why exactly is that needed?

> bool
> emacs_find_file(emacs_env* env, const char* filename)
> {
>   emacs_value e_filename = env->make_string(env, filename);
>   if(env->error_check(env)) return false;
>   emacs_value e_find_file = env->intern(env, "find-file");
>   if(env->error_check(env)) return false;
>   return env->funcall(env, e_find_file, &e_filename, 1) != NULL;
> }

This kind of code looks tedious to me, no matter if it's in Emacs or
in the module.  Just an observation.

Also, the buffer returned by find-file when it returns normally is
lost here, isn't it?

Thanks.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 11687 bytes --]

On 02/16/2015 07:43 AM, Eli Zaretskii wrote:
>> Date: Sun, 15 Feb 2015 12:20:35 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
>>
>> Here's a broad outline of what I have in mind.
> 
> Thanks.  I think the next step is for someone to try rewriting a
> couple of modules on the branch in terms of this specification, and
> see if something is missing.  Volunteers are welcome.
> 
> Some comments on your write-up:
> 
>> When Emacs loads a module, it uses dlsym (or the platform equivalent)
>> to find this routine and call it. If it returns 0, the module loaded
>> successfully; otherwise, we report an error to the caller.
> 
> If this is all we need, then there's no need for libltdl, I think.
> 
>>   struct emacs_runtime {
>>     size_t size;
>>     struct emacs_env (*get_environment)(struct emacs_runtime* ert);
>>   };
>>
>> The `size' member tells modules how long the emacs_runtime structure
>> is. (It's better to use size than an explicit version field: this way,
>> .size = sizeof(struct emacs_runtime) is always correct.)
> 
> This approach requires us to change the size each time we change the
> layout, even if the change itself leaves the size intact.  Using a
> version field doesn't have this disadvantage.

Emacs is allowed to pass a new version of the runtime and context
structures to modules expecting old versions. That can only work if the
structures are append-only, so `size' is adequate here.

>> Thread-local environments
>> -------------------------
>>
>> The `get_environment' member lets us do anything else interesting. As
>> in Java, environments are thread-local. We only support one thread for
>> the moment, so this constraint is easy to enforce. (Just abort if we
>> use an emacs_env off the main thread.)
> 
> Is it wise to design for threads at this point?  The only suggestion
> on the table for "threads" is a far cry from real threads, and the
> current Lisp interpreter is otherwise thread-unsafe.  Won't the thread
> infrastructure add a lot of cruft that will go unneeded for the
> observable future?

If it were a lot of cruft, I might agree with you, but it's not, so I'd
rather make the design future-proof.

>> We'll represent all Lisp values as an opaque pointer typedef
>> emacs_value.  Each emacs_value is either a local or a global
>> reference.  Local references are valid only on the current thread and
>> only while the module function Emacs called is on the stack --- think
>> GCPRO.  Global references are valid indefinitely: each one is a GC
>> root.
> 
> With massive use of calls to Lisp functions from modules (since we
> don't provide them with direct access in C to internals of many
> objects), how can we ensure GC doesn't strike while the function that
> created an emacs_value is still on the callstack?

We don't. Calls from modules to Lisp can certainly GC. That's why we
have local references.

> You say "we don't
> lock ourselves into conservative stack-scanning GC", which I interpret
> as saying you don't want to rely on stack scanning to avoid a
> destructive GC in this case.  But if we don't rely on that, where's
> the guarantee that such emacs_value will survive GC?

Emacs stores emacs_value values in a local reference table before
handing them to module code.

>> We'll represent all Lisp values as an opaque pointer typedef
>> emacs_value.
> 
> This doesn't play well with --with-wide-int, where a value can be
> wider than a pointer.  I think we should instead go with intmax_t or
> inptr_t, whichever is wider on the host.

emacs_value objects are not literally Lisp_Object values. They're
indirected through a reference table (either local or global) so that we
can GC them. A pointer value can address all possible memory locations,
so it's fine here. (In the wide-int case, the pointed-to table entries
are wider than pointers, but modules don't need to know that.)

>> Function registration
>> ---------------------
>>
>>   typedef emacs_value (*emacs_subr)(
>>     emacs_env* env,
>>     int nargs,
>>     emacs_value args[]);
>>
>>     emacs_value (*make_function)(
>>       emacs_env* env,
>>       int min_arity,
>>       int max_arity,
>>       emacs_subr function);
> 
> What about the doc string?

We'll set that at the lisp level. make_function is equivalent to lambda,
not defun. We can either let modules call `defun' directly or provide a
special Lisp-level function just for them.

>>     emacs_value (*funcall)(
>>       emacs_env* env,
>>       emacs_value function,
>>       int nargs,
>>       emacs_value args[]);
> 
> Shouldn't funcall use emacs_subr?

Why? It works like Lisp funcall: you can give it a function object, a
symbol, or a lambda expression. Remember that this facility lets an
Emacs module call Lisp, not the other way around.

>> Modules can register functions in the global namespace by calling a
>> Lisp-level function
> 
> This is unclear, can you elaborate?  What happens if a function is not
> "registered"? what's its status then?

It has the same status a Lisp function does after defun.

>> When Lisp calls a module-defined function object, Emacs calls the
>> emacs_subr callback with which the function was defined.
> 
> This is a change in the Lisp interpreter, I think.  Why do we need
> this?

Why would it be? Implementation-wise, all we need is an Emacs-internal
magic function for calling into modules; we can implement
module-registered functions as Lisp-level lambdas that close over all
the stuff we need to invoke the module code.

>> If Lisp signals or throws, `funcall' returns NULL.
> 
> I suggest some other value or indication of that.  NULL is a valid
> return value, so usurping it for errors might be too harsh.

No it isn't. Qnil is distinct from NULL in this model because
emacs_value is not a Lisp_Object in disguise. Qnil is not special here.

> Or maybe I don't understand how will Lisp functions return values to
> the module, under your suggestion.  Can you describe that?
> 
>> `intern' also does the obvious thing.
> 
> Do we need 'unintern' as well?

Modules can call unintern through Lisp.

>>     emacs_value (*type_of)(
>>       emacs_env* env,
>>       emacs_value value);
>>
>> Like Lisp type-of: returns a symbol.
> 
> What is a "symbol", from the module's C code POV?

It's an emacs_value. You can compare symbols (by calling `eq'), call
them as functions, or use them as function arguments. That's sufficient.

Come to think of it, though, we do need a C-level `eq'.

> You show no
> functions to access attributes of symbols, so it must be either one of
> the other types, like an integer or a string, or a C primitive data
> type, like a char * pointer.

That's deliberate. Modules should be calling Lisp functions to do that work.

>>     int64_t (*fixnum_to_int)(
>>       emacs_env* env,
>>       emacs_value value);
>>
>>     emacs_value (*make_fixnum)(
>>       emacs_env* env,
>>       int64_t value);
>>
>> These functions do the obvious thing.  They signal error on type
>> mismatch.  We use int64_t to handle big-integer Emacs variants on
>> 32-bit platforms.
> 
> The last bit means we will need a utility function to return the valid
> range of integers, so that modules can be written to support 32-bit
> and 64-bit use cases without raising errors.

Modules can call (get 'most-positive-fixnum). A utility function that
did that wouldn't hurt though.

>>     bool (*copy_string_contents)(
>>       emacs_env* env,
>>       emacs_value value,
>>       char* buffer,
>>       size_* length_inout);
>>
>>     emacs_value (*make_string)(
>>       emacs_env* env,
>>       const char* contents);
>>
>> These functions let C code access Lisp strings.  I imagine we'll
>> always produce and consume UTF-8.
> 
> Strings in Emacs are of limited usability if you cannot encode and
> decode them.  So this needs to be part of supported functionality, I
> think.
> 
> More generally, modules that would like to process buffer or string
> text will have to be able to deal with Emacs's internal encoding of
> text, which means macros and functions we use in the core.  The
> alternative of working only on UTF-8 encoded replica means we'd need
> to encode and decode text across the module boundaries -- that's a lot
> of consing.

Is the Emacs internal encoding stable? I don't have a problem with
allowing modules to access string contents: it just enlarges the
interface we'll have to support forever. Is consing strings really a
bottleneck in the use cases we have in mind?

>> `copy_string_contents' copies into a caller-allocated buffer instead
>> of returning a char* callers must free() --- this way, modules and the
>> Emacs core don't need to share the same C runtime.  We can deal with
>> the buffer-length issue in a number of ways: here, we just accept the
>> destination buffer size in *length_inout and write the total length of
>> the string to *length_inout on normal return.  We just truncate if
>> we're given too short a buffer and don't signal an error; this way,
>> callers can loop around and allocate a sufficiently large buffer for a
>> string's contents.
> 
> That's an annoyance, IMO; why not provide a function to return the
> required size?  Its implementation is trivial.

Sure --- as part of the non-ABI convenience library.

>> I think the interface above is enough for complete functionality in a
>> module, but for efficiency, we might want to expose additional
>> facilities, like access to a unibyte buffer's raw representation.
> 
> I can envision a few additional missing bits:
> 
>  . direct access to buffer text (using buffer-substring means consing
>    a lot of strings)

Is that really a problem for the use cases we want to support?

>  . creation of opaque objects that should be returned to Lisp (like
>    handles to objects managed by modules)

Sure. That'll require a new vectorlike.

>  . not sure how will a module "provide" its feature

It doesn't. You don't require a module: you load it. Most modules will
come with a small Lisp wrapper, just like Python modules.

>  . some operations that must be efficient because they are typically
>    done in inner loops, like regexp matching and accessing syntax of
>    characters: doing that via Lisp will probably be horribly slow

That seems reasonable.

>> Convenience library
>> ---------------
> 
> One thing that's inconvenient is the need to drag the environment
> pointer through all the calls.  Why exactly is that needed?

Modules have no other way of accessing Emacs internals. Dragging one
parameter through the system isn't that bad and lets us sanely track
which module is doing what.

>> bool
>> emacs_find_file(emacs_env* env, const char* filename)
>> {
>>   emacs_value e_filename = env->make_string(env, filename);
>>   if(env->error_check(env)) return false;
>>   emacs_value e_find_file = env->intern(env, "find-file");
>>   if(env->error_check(env)) return false;
>>   return env->funcall(env, e_find_file, &e_filename, 1) != NULL;
>> }
> 
> This kind of code looks tedious to me, no matter if it's in Emacs or
> in the module.  Just an observation.

We need explicit error handling. Within Emacs, we can use longjmp for
error handling. In modules, we can't.

> Also, the buffer returned by find-file when it returns normally is
> lost here, isn't it?

Sure. It's just a tiny example, and the returned buffer itself is still
GCed.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

>> You say "we don't lock ourselves into conservative stack-scanning
>> GC", which I interpret as saying you don't want to rely on stack
>> scanning to avoid a destructive GC in this case.  But if we don't
>> rely on that, where's the guarantee that such emacs_value will
>> survive GC?
> Emacs stores emacs_value values in a local reference table before
> handing them to module code.

I don't think this is worth the trouble.  I'm perfectly OK to either
require stack scanning or to require modules use the same GCPRO macros
we use in core.

If/when we get a new GC technology that is incompatible with our GCPROs
and/or stack scanning, well, then modules will break, but so will the
rest of the Emacs core code so it's only fair.


        Stefan "who didn't read the rest of this thread, sorry"

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Mon, 16 Feb 2015 10:22:18 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
> >> The `size' member tells modules how long the emacs_runtime structure
> >> is. (It's better to use size than an explicit version field: this way,
> >> .size = sizeof(struct emacs_runtime) is always correct.)
> > 
> > This approach requires us to change the size each time we change the
> > layout, even if the change itself leaves the size intact.  Using a
> > version field doesn't have this disadvantage.
> 
> Emacs is allowed to pass a new version of the runtime and context
> structures to modules expecting old versions. That can only work if the
> structures are append-only, so `size' is adequate here.

I understand that, but appending members does not necessarily increase
the size of the struct, due to alignment, bit fields, etc.

> > You say "we don't
> > lock ourselves into conservative stack-scanning GC", which I interpret
> > as saying you don't want to rely on stack scanning to avoid a
> > destructive GC in this case.  But if we don't rely on that, where's
> > the guarantee that such emacs_value will survive GC?
> 
> Emacs stores emacs_value values in a local reference table before
> handing them to module code.

That's something that wasn't in your original writeup.  Sounds like a
significant part of the design.

> >> We'll represent all Lisp values as an opaque pointer typedef
> >> emacs_value.
> > 
> > This doesn't play well with --with-wide-int, where a value can be
> > wider than a pointer.  I think we should instead go with intmax_t or
> > inptr_t, whichever is wider on the host.
> 
> emacs_value objects are not literally Lisp_Object values. They're
> indirected through a reference table (either local or global) so that we
> can GC them. A pointer value can address all possible memory locations,
> so it's fine here. (In the wide-int case, the pointed-to table entries
> are wider than pointers, but modules don't need to know that.)

How this will work when some Lisp calls a primitive implemented by a
module.  From the Lisp interpreter POV, it's just a call to a
primitive, so how will this special handling come into existence?
(And why? just to save us from picking up a wider data type?)

> > What about the doc string?
> 
> We'll set that at the lisp level.

I'm not following: what Lisp level?  What I had in mind was a module
that implements a primitive.  Are you saying that every primitive a
module wants to implement will need a Lisp part and a C part?  That
sounds cumbersome, just to save us from passing yet another argument
to make_function.

> >>     emacs_value (*funcall)(
> >>       emacs_env* env,
> >>       emacs_value function,
> >>       int nargs,
> >>       emacs_value args[]);
> > 
> > Shouldn't funcall use emacs_subr?
> 
> Why?

Because otherwise I see no point in having emacs_subr, it's almost
unused.

> >> If Lisp signals or throws, `funcall' returns NULL.
> > 
> > I suggest some other value or indication of that.  NULL is a valid
> > return value, so usurping it for errors might be too harsh.
> 
> No it isn't. Qnil is distinct from NULL in this model because
> emacs_value is not a Lisp_Object in disguise. Qnil is not special here.

I wasn't thinking about Qnil.  I simply don't see why it is a good
idea to forbid funcall from returning NULL in a normal case.

> >> `intern' also does the obvious thing.
> > 
> > Do we need 'unintern' as well?
> 
> Modules can call unintern through Lisp.

Then why provide 'intern'?  It, too, can be called from Lisp.

> >>     emacs_value (*type_of)(
> >>       emacs_env* env,
> >>       emacs_value value);
> >>
> >> Like Lisp type-of: returns a symbol.
> > 
> > What is a "symbol", from the module's C code POV?
> 
> It's an emacs_value. You can compare symbols (by calling `eq'), call
> them as functions, or use them as function arguments. That's sufficient.
> 
> Come to think of it, though, we do need a C-level `eq'.

Why not simply return a C enumeration type?

> Is the Emacs internal encoding stable?

It didn't change since Emacs 23, AFAIK.

But working with the internal representation is not my idea of fun.

> Is consing strings really a bottleneck in the use cases we have in
> mind?

I think we have no idea at this point which use cases we have in mind.

> >  . direct access to buffer text (using buffer-substring means consing
> >    a lot of strings)
> 
> Is that really a problem for the use cases we want to support?

It could be a problem for a module that processes buffer text.

> >  . not sure how will a module "provide" its feature
> 
> It doesn't. You don't require a module: you load it. Most modules will
> come with a small Lisp wrapper, just like Python modules.

So do we _require_ a Lisp wrapper?  Otherwise, how would Emacs know
whether the module is loaded or not?

> > One thing that's inconvenient is the need to drag the environment
> > pointer through all the calls.  Why exactly is that needed?
> 
> Modules have no other way of accessing Emacs internals. Dragging one
> parameter through the system isn't that bad and lets us sanely track
> which module is doing what.

The environment must be known to the module, that's for sure.  But why
does it have to be plugged into every call to every interface
function?

> > Also, the buffer returned by find-file when it returns normally is
> > lost here, isn't it?
> 
> Sure. It's just a tiny example, and the returned buffer itself is still
> GCed.

That's not what bothered me.  I don't understand how will the module
receive that return value in the first place.  An additional argument
to funcall, perhaps?

Thanks.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 9940 bytes --]

On 02/16/2015 11:29 AM, Eli Zaretskii wrote:
>> Date: Mon, 16 Feb 2015 10:22:18 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
>>
>>>> The `size' member tells modules how long the emacs_runtime structure
>>>> is. (It's better to use size than an explicit version field: this way,
>>>> .size = sizeof(struct emacs_runtime) is always correct.)
>>>
>>> This approach requires us to change the size each time we change the
>>> layout, even if the change itself leaves the size intact.  Using a
>>> version field doesn't have this disadvantage.
>>
>> Emacs is allowed to pass a new version of the runtime and context
>> structures to modules expecting old versions. That can only work if the
>> structures are append-only, so `size' is adequate here.
> 
> I understand that, but appending members does not necessarily increase
> the size of the struct, due to alignment, bit fields, etc.

If it's just a table of function pointers and everything has natural
alignment, adding members will increase the size.

>>> You say "we don't
>>> lock ourselves into conservative stack-scanning GC", which I interpret
>>> as saying you don't want to rely on stack scanning to avoid a
>>> destructive GC in this case.  But if we don't rely on that, where's
>>> the guarantee that such emacs_value will survive GC?
>>
>> Emacs stores emacs_value values in a local reference table before
>> handing them to module code.
> 
> That's something that wasn't in your original writeup.  Sounds like a
> significant part of the design.

I thought I'd made that clear. I'm glad I've been able to clarify it.
Stefan doesn't seem to think the indirection is necessary, but I'm not
sure I agree with him, especially because his scheme has a space and
speed cost on 32-bit platforms.

>>>> We'll represent all Lisp values as an opaque pointer typedef
>>>> emacs_value.
>>>
>>> This doesn't play well with --with-wide-int, where a value can be
>>> wider than a pointer.  I think we should instead go with intmax_t or
>>> inptr_t, whichever is wider on the host.
>>
>> emacs_value objects are not literally Lisp_Object values. They're
>> indirected through a reference table (either local or global) so that we
>> can GC them. A pointer value can address all possible memory locations,
>> so it's fine here. (In the wide-int case, the pointed-to table entries
>> are wider than pointers, but modules don't need to know that.)
> 
> How this will work when some Lisp calls a primitive implemented by a
> module.  From the Lisp interpreter POV, it's just a call to a
> primitive, so how will this special handling come into existence?
> (And why? just to save us from picking up a wider data type?)

I think there's some confusion about how function calls work.

From the Lisp interpreter perspective, we're calling a regular Lisp
function. That Lisp function closes over the information needed to call
a module function and then calls into an Emacs core function that
actually makes the module call.

Say a module wants to provide a function "frobricate" taking two
arguments. It defines a C-level function (call it do_frobricate) with a
signature matching emacs_subr. It then calls `make_function' like this:

  emacs_value frobricate = env->make_function(env, 2, 2, do_frobricate)
  if (frobricate == NULL) { FAILURE(); }

Now `frobricate' is a Lisp value usable in Lisp funcall; if we bind
this value to the function slot of some symbol, Lisp code can call
frobricate like any other function.

Say do_frobricate has address 0xA3B41234. Implementation-wise, the value
`frobricate` that `make_function' returned is a Lisp closure that
essentially looks like this:

  (lambda (&rest args) (%do-module-call frobricate-module 0xA3B41234 2 2
args))

%do-module-call then clears the pending-error information, packs its
arguments into a C array, and calls 0xA3B41234 like this:

  emacs_subr module_function = (module_function) 0xA3B41234;
  result = module_function(some_env, args, nargs);

(I've left out the part where we transform emacs_value to Lisp_Object
and vice versa.)

>>> What about the doc string?
>>
>> We'll set that at the lisp level.
> 
> I'm not following: what Lisp level?  What I had in mind was a module
> that implements a primitive.  Are you saying that every primitive a
> module wants to implement will need a Lisp part and a C part?  That
> sounds cumbersome, just to save us from passing yet another argument
> to make_function.

No: that's silly. I'm saying that creating a globally-named function in
Emacs has two parts: 1) create the actual _function object_. 2) bind
that function object to a name by setting some symbol's function slot to
that function object. We can do step #2 entirely in Lisp code that the
module calls, and we can set a documentation property at this time. It's
only step #1 that requires a magical incantation.

I've only described the magical, ABI-relevant bits of the interface. In
order to make a module actually usable, we'll need glue, but since we
can implement this glue in Lisp and call it from module code using the
narrow interface I've described here, I'm not worried about it.

>>>>     emacs_value (*funcall)(
>>>>       emacs_env* env,
>>>>       emacs_value function,
>>>>       int nargs,
>>>>       emacs_value args[]);
>>>
>>> Shouldn't funcall use emacs_subr?
>>
>> Why?
> 
> Because otherwise I see no point in having emacs_subr, it's almost
> unused.

emacs_subr is a typedef for a pointer to a module-provided function.
Every module-provided function is an emacs_subr.

>>>> If Lisp signals or throws, `funcall' returns NULL.
>>>
>>> I suggest some other value or indication of that.  NULL is a valid
>>> return value, so usurping it for errors might be too harsh.
>>
>> No it isn't. Qnil is distinct from NULL in this model because
>> emacs_value is not a Lisp_Object in disguise. Qnil is not special here.
> 
> I wasn't thinking about Qnil.  I simply don't see why it is a good
> idea to forbid funcall from returning NULL in a normal case.

We're not preventing functions from returning NULL in the general case.
We're deciding that NULL is not a valid value of emacs_value type. This
property makes NULL a good candidate for an error sentinel return value
for functions that would otherwise return valid emacs_value values.

>>>> `intern' also does the obvious thing.
>>>
>>> Do we need 'unintern' as well?
>>
>> Modules can call unintern through Lisp.
> 
> Then why provide 'intern'?  It, too, can be called from Lisp.

Modules have no way other than `intern' of getting an emacs_value that
corresponds to a known symbol. They can call `unintern' by first
interning the symbol "unintern", then calling it.

>>>>     emacs_value (*type_of)(
>>>>       emacs_env* env,
>>>>       emacs_value value);
>>>>
>>>> Like Lisp type-of: returns a symbol.
>>>
>>> What is a "symbol", from the module's C code POV?
>>
>> It's an emacs_value. You can compare symbols (by calling `eq'), call
>> them as functions, or use them as function arguments. That's sufficient.
>>
>> Come to think of it, though, we do need a C-level `eq'.
> 
> Why not simply return a C enumeration type?

Why make the set of elisp primitive types part of the ABI? We already
have a way to refer to symbols. There's no need to an enum, which would
just be a redundant way of naming types.

>>>  . direct access to buffer text (using buffer-substring means consing
>>>    a lot of strings)
>>
>> Is that really a problem for the use cases we want to support?
> 
> It could be a problem for a module that processes buffer text.
> 
>>>  . not sure how will a module "provide" its feature
>>
>> It doesn't. You don't require a module: you load it. Most modules will
>> come with a small Lisp wrapper, just like Python modules.
> 
> So do we _require_ a Lisp wrapper?  Otherwise, how would Emacs know
> whether the module is loaded or not?

By path, I imagine. Why don't you have this problem with (load
"myfile.el")? I'm not opposed to hooking modules into the
provide-and-require machinery, but I don't see a particular need either.

>>> One thing that's inconvenient is the need to drag the environment
>>> pointer through all the calls.  Why exactly is that needed?
>>
>> Modules have no other way of accessing Emacs internals. Dragging one
>> parameter through the system isn't that bad and lets us sanely track
>> which module is doing what.
> 
> The environment must be known to the module, that's for sure.  But why
> does it have to be plugged into every call to every interface
> function?

How else are the interface functions supposed to know the module for
which they're being called?

>>> Also, the buffer returned by find-file when it returns normally is
>>> lost here, isn't it?
>>
>> Sure. It's just a tiny example, and the returned buffer itself is still
>> GCed.
> 
> That's not what bothered me.  I don't understand how will the module
> receive that return value in the first place.  An additional argument
> to funcall, perhaps?

funcall returns an emacs_value corresponding to the value the called
function returned. Here's an alternate implementation of emacs_find_file:

emacs_value
emacs_find_file(emacs_env* env, const char* filename)
{
  emacs_value e_filename = env->make_string(env, filename);
  if(env->error_check(env)) return NULL;
  emacs_value e_find_file = env->intern(env, "find-file");
  if(env->error_check(env)) return NULL;
  return env->funcall(env, e_find_file, &e_filename, 1);
}

Now emacs_find_file returns NULL on error (with error information set in
the environment) and an emacs_value on success. The returned value
corresponds to whatever find-file returned, so in this case, it's a buffer.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Mon, 16 Feb 2015 12:01:40 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
> >>> This approach requires us to change the size each time we change the
> >>> layout, even if the change itself leaves the size intact.  Using a
> >>> version field doesn't have this disadvantage.
> >>
> >> Emacs is allowed to pass a new version of the runtime and context
> >> structures to modules expecting old versions. That can only work if the
> >> structures are append-only, so `size' is adequate here.
> > 
> > I understand that, but appending members does not necessarily increase
> > the size of the struct, due to alignment, bit fields, etc.
> 
> If it's just a table of function pointers and everything has natural
> alignment, adding members will increase the size.

Having to include only function pointers is also a kind of limitation
inherent to this method.

I guess I just don't understand why you didn't go with a version
field.  What are its disadvantages?  The value can be generated
automatically, so it isn't more error-prone than the size.

> Say do_frobricate has address 0xA3B41234. Implementation-wise, the value
> `frobricate` that `make_function' returned is a Lisp closure that
> essentially looks like this:
> 
>   (lambda (&rest args) (%do-module-call frobricate-module 0xA3B41234 2 2
> args))
> 
> %do-module-call then clears the pending-error information, packs its
> arguments into a C array, and calls 0xA3B41234 like this:
> 
>   emacs_subr module_function = (module_function) 0xA3B41234;
>   result = module_function(some_env, args, nargs);

It's the arguments that I was talking about.  AFAIU, you are saying
that %do-module-call takes arguments, which are normal Lisp_Object
values, generates the indirection table for them, which stores
_pointers_ to the actual arguments, then puts those pointers into a C
array, and calls 0xA3B41234.  Is that correct?

If so, I guess I don't see why we would need all that complexity and
extra processing.  Why not put the original Lisp_Object values into
the args[] array to begin with?

> >>> What about the doc string?
> >>
> No: that's silly. I'm saying that creating a globally-named function in
> Emacs has two parts: 1) create the actual _function object_. 2) bind
> that function object to a name by setting some symbol's function slot to
> that function object. We can do step #2 entirely in Lisp code that the
> module calls, and we can set a documentation property at this time. It's
> only step #1 that requires a magical incantation.
> 
> I've only described the magical, ABI-relevant bits of the interface. In
> order to make a module actually usable, we'll need glue, but since we
> can implement this glue in Lisp and call it from module code using the
> narrow interface I've described here, I'm not worried about it.

Sorry, I'm still not out of the woods.  Providing a doc string has
nothing to do with calling the function.  You need to have the doc
string available even if you never call the function.  The trigger for
accessing the doc string is _outside_ of the module, by one of the
help commands.  So how does all this work in your scenario above,
where we called make_function, which returned 'frobnicate', which is a
value in the function slot of some Lisp object?  Say now Emacs calls
'documentation-property' on that Lisp object or it calls
'documentation' on 'frobnicate' -- how will that produce a doc string?

> >>> Do we need 'unintern' as well?
> >>
> >> Modules can call unintern through Lisp.
> > 
> > Then why provide 'intern'?  It, too, can be called from Lisp.
> 
> Modules have no way other than `intern' of getting an emacs_value that
> corresponds to a known symbol. They can call `unintern' by first
> interning the symbol "unintern", then calling it.

You mean, modules cannot use funcall to call Lisp functions, without
some complicated dance with 'intern' before that?  Why is that needed?
why not let modules call Lisp functions by their string name, and do
the 'intern' stuff on the Emacs side of the interface?

> >>>>     emacs_value (*type_of)(
> >>>>       emacs_env* env,
> >>>>       emacs_value value);
> >>>>
> >>>> Like Lisp type-of: returns a symbol.
> >>>
> >>> What is a "symbol", from the module's C code POV?
> >>
> >> It's an emacs_value. You can compare symbols (by calling `eq'), call
> >> them as functions, or use them as function arguments. That's sufficient.
> >>
> >> Come to think of it, though, we do need a C-level `eq'.
> > 
> > Why not simply return a C enumeration type?
> 
> Why make the set of elisp primitive types part of the ABI?

An enum is just a list of named integer values.  Additions to that
list make more values valid, that's all.  How is that "part of the
ABI" that could introduce incompatibility?

OTOH, making the life of module writers harder on every turn is not a
good way of encouraging modules.

> >>>  . not sure how will a module "provide" its feature
> >>
> >> It doesn't. You don't require a module: you load it. Most modules will
> >> come with a small Lisp wrapper, just like Python modules.
> > 
> > So do we _require_ a Lisp wrapper?  Otherwise, how would Emacs know
> > whether the module is loaded or not?
> 
> By path, I imagine.

What path is that?  Where and how will Lisp code find it?

> Why don't you have this problem with (load "myfile.el")?

Because myfile.el normally does a (provide 'myfile), and then I can
use featurep.

> I'm not opposed to hooking modules into the provide-and-require
> machinery, but I don't see a particular need either.

The need is to have an easy way of checking that a module is
available, without risking loading it twice, or having to jump through
hoops.

> >>> One thing that's inconvenient is the need to drag the environment
> >>> pointer through all the calls.  Why exactly is that needed?
> >>
> >> Modules have no other way of accessing Emacs internals. Dragging one
> >> parameter through the system isn't that bad and lets us sanely track
> >> which module is doing what.
> > 
> > The environment must be known to the module, that's for sure.  But why
> > does it have to be plugged into every call to every interface
> > function?
> 
> How else are the interface functions supposed to know the module for
> which they're being called?

Why would the interface functions care?  Which parts of the
environment are needed for the interface to know about the module in
order to do its job?

Re: Dynamic loading progress

[Aurélien Aptel]

I'm with Eli on the versionning choice but it's just bike shedding at
this point.

Interning all the time seems tedious and probably (a bit?) slow but
moving the interning on Emacs side won't change the slow part. At
least on the module side you can do it once and reuse the result. Or
maybe you shouldn't if the user changes the binding in the meantime?

On Tue, Feb 17, 2015 at 4:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> By path, I imagine.
>
> What path is that?  Where and how will Lisp code find it?
>
>> Why don't you have this problem with (load "myfile.el")?
>
> Because myfile.el normally does a (provide 'myfile), and then I can
> use featurep.
>
>> I'm not opposed to hooking modules into the provide-and-require
>> machinery, but I don't see a particular need either.
>
> The need is to have an easy way of checking that a module is
> available, without risking loading it twice, or having to jump through
> hoops.

We should call `provide' in the module (like in my module branch)
using the API. Problem solved?

Concerning docstrings, we just have to teach the makedoc utility how
to extract them from this new module system. Then the module loading
code can read that along with loading the module as it's done in my
branch, where the doc string field of a Lisp_Subr can be a (cons
"doc/file/path" offset) instead of simply <offset to the global DOC
file>.

So the real remaining issues are
- module code is tedious to write because you have to check for errors
all the time
- interning all the time is slow, probably? (needs some testing)
- <insert missing issues>

Sorry, I'm sure I've overlooked some things, I'll re-read all the
discussion later.

Re: Dynamic loading progress

[Aurélien Aptel]

On Tue, Feb 17, 2015 at 6:46 PM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:
> maybe you shouldn't if the user changes the binding in the meantime?

err.. intrening returns a symbol nevermind that part.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2561 bytes --]

On 02/17/2015 09:46 AM, Aurélien Aptel wrote:
> I'm with Eli on the versionning choice but it's just bike shedding at
> this point.

I'm worried about what happens during long Emacs development cycles. Say
we're developing Emacs 26 and during its yearlong release cycle, we add
to emacs_env a `make_foo' in month 3 and an `access_bar' hook in month
6. I want to make an distribute a module that uses `make_foo' if it's
available. If we have some kind of explicit versioning enum, we'll
probably have only one per Emacs release, so modules won't be able to
detect that `foo' is available until we release Emacs 26. If we make the
version date-based, we'll have a lot of different versions, and it'll be
hard for a module author to figure out which constant they need to test
against.

A size field seems just right, since it automatically updates when and
only when we add something to the struct.

> Interning all the time seems tedious and probably (a bit?) slow but
> moving the interning on Emacs side won't change the slow part. At
> least on the module side you can do it once and reuse the result. Or
> maybe you shouldn't if the user changes the binding in the meantime?

I agree that interning all the time is tedious, but I don't intend
module authors to use this interface directly. What I've described is
just the narrow ABI-stable boundary between Emacs and modules; we can
give module authors a richer "client" library, implemented in terms of
the stable ABI, that itself doesn't need to ABI-stable (although we
should try to make it source-compatible upwardly).

I have a few ideas for making life easy for module authors.  One idea is
to make evaluating Lisp very easy. We can provide something like this:

  bool emacs_eval(emacs_env* env, const char* lisp, ...);

Then module authors can write code like this:

  success = emacs_eval(env,
    "(put-text-property %v %v 'myproperty (myfunction))",
      value_min, value_max);

Internally, we'd then call (using emacs_env::funcall) a function we'd
define in Lisp in Emacs:

  (%module-eval "(put-text-property %v %v...)" <value_min> <value_max>)

That function would then transform the %v substitutions to expressions
like (nth module-eval-arguments 0) (nth module-eval-arguments 1))

And then call

  (let ((module-eval-arguments args))
    (eval expression-from-module))

This interface isn't the most efficient thing in the world, but it's
hard to be more convenient, and it's good enough for code that isn't
executed frequently.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> I'm worried about what happens during long Emacs development
> cycles.  Say we're developing Emacs 26 and during its yearlong release
> cycle, we add to emacs_env a `make_foo' in month 3 and an `access_bar'
> hook in month 6.

We don't provide any support for non-released version of Emacs other
than "the latest on its branch", so once `access_bar' is added the
intermediate situation where `make_foo'was added but `access_bar' wasn't
simply doesn't exist anymore (that is, it may exist but any user of such
an Emacs is expected to fix the problem by upgrading (or downgrading)).

> A size field seems just right, since it automatically updates when and
> only when we add something to the struct.

IIUC there'd be a size field anyway,whether or not there's also
a version field.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

I've tried to implement modules using Daniel's plan. It turned out to
be pretty easy, mostly because I'm more familiar with the Emacs
codebase now and because it's a lot less intrusive than my previous
attempt.

It's not finished but I have a basic module working on linux. It's a
proof of concept, basically.

* fixnum <-> int64_t is the only type conversion supported
* No memory management
* No error handling
* No optional compilation/configure option
* No docstrings

No special build instructions since nothing is ifdef'd.
Once you have it running, you can build the sample module by running
make in modules/basic/.

    $ ./src/emacs -Q
    (module-load "./modules/basic/basic.so")
    => t

    (basic-sum 2 2)
    => 4

The code is available on my github repo on the dynamic-modules-2 branch.
The interesting parts are the API header, the runtime and the sample module.

git repo: https://github.com/aaptel/emacs-dynamic-module.git
browse online: https://github.com/aaptel/emacs-dynamic-module/commit/278f26a5e993f8cbf6784f35c5ea8aeb9ef04019

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> I've tried to implement modules using Daniel's plan. It turned out to
> be pretty easy, mostly because I'm more familiar with the Emacs
> codebase now and because it's a lot less intrusive than my previous
> attempt.

Thanks!

I'm a little busy right now, but I'll look at this soon.


-- 
-- Stephe

Re: Dynamic loading progress

[Stefan Monnier]

> * fixnum <-> int64_t is the only type conversion supported

Not sure I understand: can the C side see "Lisp_Object" (presumably as
an opaque type)?


        Stefan

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

On 03/04/2015 02:34 PM, Stefan Monnier wrote:
>> * fixnum <-> int64_t is the only type conversion supported
> 
> Not sure I understand: can the C side see "Lisp_Object" (presumably as
> an opaque type)?

As I'm imagining the system, no. Access is indirected. That insulates C
code from Emacs GC requirements and lets us easily support global
references.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

The dynamic-modules-2 branch is implemented from scratch (well, from
the master branch) in case that wasn't clear.

I'll summarize what the code is doing, to make sure with everyone I'm
not doing anything wrong.

Emacs core has a new module.c[0] file which provides various function
pointers to modules via a struct (actually 2 struct, there's
emacs_environment and emacs_runtime). These structs are defined in
src/emacs_module.h [1].

A module has an emacs_module_init function which is called when the
module is loaded. This function is passed an emacs_environment pointer
which is used by the module to get a emacs_runtime pointer which has
all the useful function pointers to interact with emacs.

Lisp_Object are exposed via the emacs_value type which is opaque. From
emacs_module.h:

    typedef void* emacs_value;

To convert a Lisp_Objet to an emacs_value I simply cast, currently.
From Emacs' module.c:

    static emacs_value module_make_fixnum (emacs_env *env, int64_t n)
    {
      return (emacs_value) make_number (n);
    }

(let's just ignore the overflow and error checking for now and focus
on the architecture)

Daniel's last message made me understand the GC problem. Since emacs
integers are stored directly "in" the Lisp_Object there's no problem
with the GC because there's no memory to free. But for other more
complex types (where the Lisp_Object is actually a pointer) an
emacs_value stored by the module can become invalid between 2
emacs->module calls because it was garbage collected.

So what I should do is make another Lisp_Object-like union type that
owns its memory (ie. not handled by the GC) and use that as
emacs_value (still opaque for the module). Correct?

0: https://github.com/aaptel/emacs-dynamic-module/blob/dynamic-modules-2/src/module.c
1: https://github.com/aaptel/emacs-dynamic-module/blob/dynamic-modules-2/src/emacs_module.h

Re: Dynamic loading progress

[Paul Eggert]

On 03/05/2015 05:12 AM, Aurélien Aptel wrote:
> typedef void* emacs_value;
>
> To convert a Lisp_Objet to an emacs_value I simply cast

I don't see how this can work reliably if EMACS_INT and 'void *' have 
different widths.  You should be able to reproduce the problem on x86 
with 'configure --with-wide-int'.

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Mar 5, 2015 at 6:37 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> I don't see how this can work reliably if EMACS_INT and 'void *' have
> different widths.  You should be able to reproduce the problem on x86 with
> 'configure --with-wide-int'.

I guess we could use (u)int64_t instead of void* then?

Anyway passing casted Lisp_Object won't work for pointer-type
Lisp_Object because of the GC. We'll most likely have to use pointers
to an intermediate struct (cf my previous post). Fast access to types
directly encoded in the Lisp_Object address would be nice though. I
feel like we're just going to re-implement Lisp_Object x)

Re: Dynamic loading progress

[Ted Zlatanov]

On Thu, 5 Mar 2015 14:12:29 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> The dynamic-modules-2 branch is implemented from scratch (well, from
AA> the master branch) in case that wasn't clear.
...
AA> 0: https://github.com/aaptel/emacs-dynamic-module/blob/dynamic-modules-2/src/module.c
AA> 1: https://github.com/aaptel/emacs-dynamic-module/blob/dynamic-modules-2/src/emacs_module.h

I like it. Can you provide a simple example in the branch of how it
would be used from both sides? I can restart the testing and keep it in
sync with the Emacs repo again.

Thanks
Ted

Re: Dynamic loading progress

[Aurélien Aptel]

On Tue, Apr 21, 2015 at 1:38 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> I like it. Can you provide a simple example in the branch of how it
> would be used from both sides? I can restart the testing and keep it in
> sync with the Emacs repo again.

It's already in there:

https://github.com/aaptel/emacs-dynamic-module/tree/dynamic-modules-2/modules/basic

Re: Dynamic loading progress

[Ted Zlatanov]

On Tue, 21 Apr 2015 10:58:06 +0200 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Tue, Apr 21, 2015 at 1:38 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> I like it. Can you provide a simple example in the branch of how it
>> would be used from both sides? I can restart the testing and keep it in
>> sync with the Emacs repo again.

AA> It's already in there:

AA> https://github.com/aaptel/emacs-dynamic-module/tree/dynamic-modules-2/modules/basic

Right!

As far as the coding style and barriers this enforces, I think it's just
right.  Thanks to you and Dan Colascione and the others for working on it.

As far as capabilities, the basic module should have a string exchange,
as you've started to implement in `copy_string_contents'
and `make_string'.  I think it's impossible to move forward without that.

I would go simple for now: always make a copy of the string using UTF-8
encoding in the args, assuming that the majority of C modules can deal
with it, and we can stop worrying about string sharing and encoding for
now (we can revisit them later). We should free the copy right after
calling so the module doesn't have to free it.

If you agree, maybe you should make a specific type for UTF-8 encoded
strings so we can support others in the future (byte array, wchars,
etc.).

I think `plugin_is_GPL_compatible' should be a float indicating the GPL
version.  Maybe even a char[] to express flavors, from a constrained set
of choices.

Let me know what you think.  I'll hold off pushing and testing for now
but am happy to proceed on your command.

Ted

Re: Dynamic loading progress

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 21 Apr 2015 10:58:06 +0200 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 
>
> AA> On Tue, Apr 21, 2015 at 1:38 AM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>> I like it. Can you provide a simple example in the branch of how it
>>> would be used from both sides? I can restart the testing and keep it in
>>> sync with the Emacs repo again.
>
> AA> It's already in there:
>
> AA> https://github.com/aaptel/emacs-dynamic-module/tree/dynamic-modules-2/modules/basic
>
> Right!
>
> As far as the coding style and barriers this enforces, I think it's just
> right.  Thanks to you and Dan Colascione and the others for working on
> it.

+1

> As far as capabilities, the basic module should have a string exchange,
> as you've started to implement in `copy_string_contents'
> and `make_string'.  I think it's impossible to move forward without
> that.

For my parser module, I plan on calling the elisp lexer currently in
ada-mode (which uses the sytax class information, forward-sexp, etc).
That avoids copying the entire buffer text to the module, or dealing
with direct access. I haven't got it working yet, so I'm not entirely
sure that will be fast enough, but the lexer is a small portion of the
full parser.

> I would go simple for now: always make a copy of the string using UTF-8
> encoding in the args, assuming that the majority of C modules can deal
> with it, and we can stop worrying about string sharing and encoding for
> now (we can revisit them later). We should free the copy right after
> calling so the module doesn't have to free it.

I'm not clear who is freeing what here; the module API does need to
document that clearly.

We also need to document the GCPRO issues; for example, given the
following call:

  emacs_value pairs[] =
    { env->make_fixnum (env, 1), ada_grammar_names[341], /* statement-start */
      env->make_fixnum (env, 2), ada_grammar_names[342], /* statement-other */
      env->make_fixnum (env, 4), ada_grammar_names[343]  /* statement-end */
    };

  emacs_value args_2[] = { env->funcall (env, vector, 6, pairs) };

  env->funcall (env, wisi_statement_action, 1, args_2);

where 'wisi_statement_action' is an elisp function that uses 'pairs' to
set text properties, should the module do GCPRO around the funcall, or
does the funcall handle that?

Also, we need to do something about the doc strings for lisp objects
declared in modules. I don't think they need to be available before the
module is loaded; if a particular author wants that, they can define an
elisp wrapper with autoloads.

At a minimum, default doc string should be provided that gives a pointer
to the module source code.

> I think `plugin_is_GPL_compatible' should be a float indicating the GPL
> version.  Maybe even a char[] to express flavors, from a constrained set
> of choices.

+1 for the char[]. The variable could be named 'license_terms', so it
could be used by non-GPL in special circumstances.

-- 
-- Stephe

Re: Dynamic loading progress

[Stefan Monnier]

> where 'wisi_statement_action' is an elisp function that uses 'pairs' to
> set text properties, should the module do GCPRO around the funcall, or
> does the funcall handle that?

I think it'd be OK to simply say that dynamic loading can only be used
on platforms that use the conservative stack scanning (i.e. don't need
GCPRO).
AFAIK all platforms that matter use conservative stack scanning.

>> I think `plugin_is_GPL_compatible' should be a float indicating the GPL
>> version.  Maybe even a char[] to express flavors, from a constrained set
>> of choices.
> +1 for the char[]. The variable could be named 'license_terms', so it
> could be used by non-GPL in special circumstances.

It took us many years to find a system with which the FSF is
comfortable, so I'm vetoing any bikeshedding on this name.
`plugin_is_GPL_compatible' it will be.

If you want it to be finer grained to detect incompatibilities between
Emacs's GPLv3+ and a plugin that's GPLv2-only, then please discuss it
with Richard (and maybe the GCC guys, since IIUC they use the exact
same system).
In the mean we'll just check the presence of this symbol rather than any
value associated with it.


        Stefan

Re: Dynamic loading progress

[Ted Zlatanov]

On Wed, 22 Apr 2015 13:15:52 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

SM> It took us many years to find a system with which the FSF is
SM> comfortable, so I'm vetoing any bikeshedding on this name.
SM> `plugin_is_GPL_compatible' it will be.

SM> If you want it to be finer grained to detect incompatibilities between
SM> Emacs's GPLv3+ and a plugin that's GPLv2-only, then please discuss it
SM> with Richard (and maybe the GCC guys, since IIUC they use the exact
SM> same system).
SM> In the mean we'll just check the presence of this symbol rather than any
SM> value associated with it.

I wasn't aware it was a GNU thing already.  Thank you for explaining.

Ted

Re: Dynamic loading progress

[Ted Zlatanov]

On Wed, 22 Apr 2015 11:25:51 -0500 Stephen Leake <stephen_leake@stephe-leake.org> wrote: 

SL> Ted Zlatanov <tzz@lifelogs.com> writes:

>> I would go simple for now: always make a copy of the string using UTF-8
>> encoding in the args, assuming that the majority of C modules can deal
>> with it, and we can stop worrying about string sharing and encoding for
>> now (we can revisit them later). We should free the copy right after
>> calling so the module doesn't have to free it.

SL> I'm not clear who is freeing what here; the module API does need to
SL> document that clearly.

"We" was Emacs. So the module API will state "passed strings are a
temporary copy that will be freed for you, so don't free() them, but if
you do, it won't break anything."

SL> We also need to document the GCPRO issues;
...
SL> should the module do GCPRO around the funcall, or does the funcall
SL> handle that?

IMHO Emacs should manage memory, period. Any time modules are given that
responsibility, it's an opportunity for bugs and a premature
optimization. Perhaps we'll find it necessary later to give this
responsibility to some modules, but let's not do it now.

SL> Also, we need to do something about the doc strings for lisp objects
SL> declared in modules. I don't think they need to be available before the
SL> module is loaded; if a particular author wants that, they can define an
SL> elisp wrapper with autoloads.

SL> At a minimum, default doc string should be provided that gives a pointer
SL> to the module source code.

I have no strong opinion on this, but your suggestion sounds fine.

Ted

Re: Dynamic loading progress

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 22 Apr 2015 11:25:51 -0500 Stephen Leake <stephen_leake@stephe-leake.org> wrote: 
>
> SL> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>
> SL> We also need to document the GCPRO issues;
> ...
> SL> should the module do GCPRO around the funcall, or does the funcall
> SL> handle that?
>
> IMHO Emacs should manage memory, period. Any time modules are given that
> responsibility, it's an opportunity for bugs and a premature
> optimization. Perhaps we'll find it necessary later to give this
> responsibility to some modules, but let's not do it now.

That makes sense, but doesn't actually answer my question.

Does the current funcall implementation handle GCPRO on the args
adequately, or does it need to be improved/GCPRO added in the module?

-- 
-- Stephe

Re: Dynamic loading progress

[Ted Zlatanov]

On Sun, 03 May 2015 18:44:05 -0500 Stephen Leake <stephen_leake@stephe-leake.org> wrote: 

SL> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Wed, 22 Apr 2015 11:25:51 -0500 Stephen Leake <stephen_leake@stephe-leake.org> wrote: 
>> 
SL> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
>> 
SL> We also need to document the GCPRO issues;
>> ...
SL> should the module do GCPRO around the funcall, or does the funcall
SL> handle that?
>> 
>> IMHO Emacs should manage memory, period. Any time modules are given that
>> responsibility, it's an opportunity for bugs and a premature
>> optimization. Perhaps we'll find it necessary later to give this
>> responsibility to some modules, but let's not do it now.

SL> That makes sense, but doesn't actually answer my question.

SL> Does the current funcall implementation handle GCPRO on the args
SL> adequately, or does it need to be improved/GCPRO added in the module?

(You mean the module-side funcall, right?  Not the one built in Emacs?)

Looking at it, it appears to only pass atomic data types that don't need
GC protection right now, so the implementation for passing strings is
not done.  But I'd love to get some comments from Aurélien, he must have
plans and ideas.

Ted

Re: Dynamic loading progress

[Stefan Monnier]

>>> I would go simple for now: always make a copy of the string using UTF-8
>>> encoding in the args, assuming that the majority of C modules can deal
>>> with it, and we can stop worrying about string sharing and encoding for
>>> now (we can revisit them later). We should free the copy right after
>>> calling so the module doesn't have to free it.

I don't like this idea of making copies just to avoid memory management.
Especially since we already have a GC available to solve this problem.
But I missed the context, so I don't really know what I'm talking about.

SL> We also need to document the GCPRO issues;
> ...
SL> should the module do GCPRO around the funcall, or does the funcall
SL> handle that?
> IMHO Emacs should manage memory, period. Any time modules are given that
> responsibility, it's an opportunity for bugs and a premature
> optimization.

It's an illusion to think that all memory management can be moved to the
Emacs side.  But I think the best way to move most of it to the Emacs
side is to let the modules benefit from the Emacs's GC, and hence export
the "GC protocol" to the modules side.

As mentioned elsewhere, we probably want to skip the GCPRO part of the
protocol (and hence only support dynamic loading on those platforms
where we use conservative stack scanning), so as to move even more of
the memory management onto the Emacs's side.

SL> Also, we need to do something about the doc strings for lisp objects
SL> declared in modules. I don't think they need to be available before the
SL> module is loaded; if a particular author wants that, they can define an
SL> elisp wrapper with autoloads.

Agreed.  The same holds for .el files.  The only difference is that for
.el files we have autoload.el which can auto-generate the "wrapper with
autoloads".  Maybe we could create a similar system for dynloaded
packages, but there's no hurry for it.


        Stefan

Re: Dynamic loading progress

[Ted Zlatanov]

On Sun, 03 May 2015 20:16:03 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> IMHO Emacs should manage memory, period. Any time modules are given that
>> responsibility, it's an opportunity for bugs and a premature
>> optimization.

SM> It's an illusion to think that all memory management can be moved to the
SM> Emacs side.  But I think the best way to move most of it to the Emacs
SM> side is to let the modules benefit from the Emacs's GC, and hence export
SM> the "GC protocol" to the modules side.

Exactly.  I should have said "Emacs should manage its memory."

Ted

Re: Dynamic loading progress

[Ted Zlatanov]

Stephen, Aurélien, the dynamic module progress seems stalled. Could you
summarize what's missing or blocking it?  Back in May 2015 we discussed
it but I don't see anything happening afterwards.

Thanks
Ted

Re: Dynamic loading progress

[Stephen Leake]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Stephen, Aurélien, the dynamic module progress seems stalled. Could you
> summarize what's missing or blocking it?  Back in May 2015 we discussed
> it but I don't see anything happening afterwards.

I got stuck with Ada exception handling issues, then got sidetracked by
various other things.

As far as issues blocking a merge to master, I've documented the ones I
know about as FIXMEs in module.c.

-- 
-- Stephe

Re: Dynamic loading progress

[Aurélien Aptel]

Hi all, Daniel,

Following Ted last off-list email I looked again at the module code.

Thanks again to Daniel for the API design proposal you did few month
ago and the general explanations, it was insightful and it really
helped me. I would really like to have the module API in emacs 25 so I
recently revisited my code and re-read the dynamic loading threads on
emacs-devel but I'm still stuck on the memory aspect.

I'll re-explain briefly the problem:

The module API [1] lets module writer make new Lisp_Object (hidden as
opaque pointers named 'emacs_value') via make_<type>(), funcall(),
type_of() and intern() functions. But if a module function calls core
Emacs functions, those functions it turn might call the GC, which
would free and invalidate the objects made by the module.

Here are 2 examples of the same problem:

- Local values which are created/can-be-freed inside the module function.

    a = make_string("foo")
    b = make_string("bar")
    len = funcall(eval, 1, [length_sym, b])
    // a (and b?) might be invalid now if the GC was triggered inside
the funcall call.

In the core Emacs code, we have the GCPRO macro to deal with that use-case.

- Values created and reused between 2 module calls.

    emacs_value global_a;

    module_func_A()
        global_a = make_string("a")

    module_func_B():
        // global_a might be invalid now if the GC was triggered
between func_A and func_B
        return global_a

In the core Emacs code, global_a is usually a defvar.

Daniel was talking about tables mapping emacs_value to Lisp_Object,
and having manual memory management in the module API. But I'm not
sure I understood what you meant.

Here's another proposition (which might just be what you were trying
to explain):

We add a hash table at the GC root that maps module names to a list of
values currently owned by the module.
Any markable/collectable emacs_value (all Lisp_Object types except for
numeric ones basically) created by the module API are added to that
hash table (appended at the module's list).
When a module doesn't need a value anymore, it explicitly calls a
function in the module API that removes it from the module's list.
The GC walks the object tree like usual, including the module hash table.

This is more or less what GCPRO/UNGCPRO does, I think (temporally add
objects to the GC root). And if remember correctly Stefan was against
exposing it in the module API.

I've proposed a hash table to know which values a module owns. That
way we can sort of "unload" a module by simply removing the module
entry from the hash table (we still have to deal with unbinding the
module's functions). The GC will do the rest.

Any feedback, other ideas, welcome.

1: https://github.com/aaptel/emacs-dynamic-module/blob/dynamic-modules-2/src/emacs_module.h

Re: Dynamic loading progress

[Stefan Monnier]

> - Local values which are created/can-be-freed inside the module function.
>
>     a = make_string("foo")
>     b = make_string("bar")
>     len = funcall(eval, 1, [length_sym, b])
>     // a (and b?) might be invalid now if the GC was triggered inside
> the funcall call.
>
> In the core Emacs code, we have the GCPRO macro to deal with that use-case.

GCPRO is only needed if we don't use conservative stack scanning.
Conservative stack scanning is used by 99.9% of Emacs builds (so much so
that there are always latent bugs in the GCPRO code because it sees very
little testing), so I think it's perfectly acceptable to ignore the
remaining 0.1%.

IOW we don't need to worry about local Lisp_Object values: they'll be
found by the conservative stack scanner.

> - Values created and reused between 2 module calls.
>
>     emacs_value global_a;
>
>     module_func_A()
>         global_a = make_string("a")
>
>     module_func_B():
>         // global_a might be invalid now if the GC was triggered
> between func_A and func_B
>         return global_a
>
> In the core Emacs code, global_a is usually a defvar.

We just need to export the equivalent of "staticpro" (ideally, this
should keep track of which module called it, so that we can
automatically "unpro"tect those vars when/if the module is unloaded).

> Daniel was talking about tables mapping emacs_value to Lisp_Object,

I don't want to have to convert between emacs_value and Lisp_Object,
they should be one and the same.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

On Mon, Aug 24, 2015 at 12:26 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> GCPRO is only needed if we don't use conservative stack scanning.
> Conservative stack scanning is used by 99.9% of Emacs builds (so much so
> that there are always latent bugs in the GCPRO code because it sees very
> little testing), so I think it's perfectly acceptable to ignore the
> remaining 0.1%.

Ok, we can always check for conservative scanning at compile to enable modules.

> We just need to export the equivalent of "staticpro" (ideally, this
> should keep track of which module called it, so that we can
> automatically "unpro"tect those vars when/if the module is unloaded).

So if I understand correctly, with conservative scanning there's no
need to call UNGCPRO?

> I don't want to have to convert between emacs_value and Lisp_Object,
> they should be one and the same.

Ok, I'll keep casting Lisp_Object to void* then.

Since module writer will often need to use Lisp_Object as handle for
their own struct pointers, I was thinking adding a way to register
finalizer functions to the API.

Re: Dynamic loading progress

[Paul Eggert]

Aurélien Aptel wrote:
> Ok, I'll keep casting Lisp_Object to void* then.

That's not necessarily correct, as the cast could lose information, e.g., 32-bit 
platforms configured --with-wide-int.

Re: Dynamic loading progress

[Stefan Monnier]

> Ok, we can always check for conservative scanning at compile to
> enable modules.

Exactly.

>> We just need to export the equivalent of "staticpro" (ideally, this
>> should keep track of which module called it, so that we can
>> automatically "unpro"tect those vars when/if the module is unloaded).
> So if I understand correctly, with conservative scanning there's no
> need to call UNGCPRO?

Indeed, neither GCPRO nor GCUNPRO.

> Since module writer will often need to use Lisp_Object as handle for
> their own struct pointers, I was thinking adding a way to register
> finalizer functions to the API.

We'll probably want ways to define new types and/or to register
finalizers, indeed.

But that can wait.


        Stefan

Re: Dynamic loading progress

[Tom Tromey]

Stefan> IOW we don't need to worry about local Lisp_Object values: they'll be
Stefan> found by the conservative stack scanner.

Is there a reason to keep GCPRO around?

Tom

Re: Dynamic loading progress

[Stefan Monnier]

Stefan> IOW we don't need to worry about local Lisp_Object values: they'll be
Stefan> found by the conservative stack scanner.
> Is there a reason to keep GCPRO around?

IIUC there are still some situations where we build without conservative
stack scanning.

We should probably go and enable it everywhere.  If noone complains that
should indicate that indeed conservative stack scanning is used (or ca
be used) everywhere, so we could drop the GCPROs.


        Stefan

Re: Dynamic loading progress

[Paul Eggert]

On 08/24/2015 12:13 PM, Stefan Monnier wrote:
> IIUC there are still some situations where we build without conservative
> stack scanning.
>
> We should probably go and enable it everywhere.

We did that last year, before 24.4 came out.  GC_MARK_STACK is defined 
to be GC_MAKE_GCPROS_NOOPS unless the builder compiles with 
-DGC_MARK_STACK=somethingelse.  I think nobody does that except perhaps 
when testing.

> If noone complains that
> should indicate that indeed conservative stack scanning is used (or ca
> be used) everywhere, so we could drop the GCPROs.

As far as I know, nobody has complained, so we could drop the GCPROs now 
if you like.

Perhaps we should hold a wake?  Should I bring a bottle of poteen?

Re: Dynamic loading progress

[David Kastrup]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 08/24/2015 12:13 PM, Stefan Monnier wrote:
>> IIUC there are still some situations where we build without conservative
>> stack scanning.
>>
>> We should probably go and enable it everywhere.
>
> We did that last year, before 24.4 came out.  GC_MARK_STACK is defined
> to be GC_MAKE_GCPROS_NOOPS unless the builder compiles with
> -DGC_MARK_STACK=somethingelse.  I think nobody does that except
> perhaps when testing.
>
>> If noone complains that
>> should indicate that indeed conservative stack scanning is used (or ca
>> be used) everywhere, so we could drop the GCPROs.
>
> As far as I know, nobody has complained, so we could drop the GCPROs
> now if you like.
>
> Perhaps we should hold a wake?  Should I bring a bottle of poteen?

Just have people point at each other, that should keep them awake.

-- 
David Kastrup

Re: Dynamic loading progress

[Stefan Monnier]

> We did that last year, before 24.4 came out.

Wait... what's that bell ringing in the distance?

> As far as I know, nobody has complained, so we could drop the GCPROs
> now if you like.

Indeed.

> Perhaps we should hold a wake?

Definitely.

There are still many details to figure out.
Incineration of inhumation?
Open casket?
Should we keep it private?


        Stefan

Re: Dynamic loading progress

[Paul Eggert]

Stefan Monnier wrote:
> Open casket?

I was thinking more along the lines of a limestone sarcophagus, Egyptian style.

I gave it a shot in master commit 60d1b18734fff144f1608da6228d60e4bda7b24c, so 
there should no longer be any need for GCPRO1 etc. in C source.  I hope this 
gives us one less thing to worry about with dynamic loading.

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Aug 27, 2015 at 4:29 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> I gave it a shot in master commit 60d1b18734fff144f1608da6228d60e4bda7b24c,
> so there should no longer be any need for GCPRO1 etc. in C source.  I hope
> this gives us one less thing to worry about with dynamic loading.

Cool! I was recently thinking of writing coccinelle [1] checks for
unreleased GCPRO calls.. Guess it's not needed anymore :)

1: http://coccinelle.lip6.fr/

Re: Dynamic loading progress

[Aurélien Aptel]

Quick update:

I only have the error handling left to implement from the API. I'm not
sure on how to catch error/signal and return to the module code.

I've added a basic portable dynamic loader layer (directly calling the
native loading function) so it doesn't have to depend on libtool.

Some of the remaining stuff to do:

* configure/ifdef machinery to enable/disable the feature
* expose a way to store arbitrary pointers in an emacs_value, and
maybe implement finalizers.
* add doc strings support
* start thinking about how we want to package modules

The problem with finalizer is they can't be used as destructors since
the GC is not guaranteed to be called at the end of every
dynamic/lexical scope (or is it?). But maybe that's a non-problem as
long as it's documented.

The rest of the things to do (except for the packaging thing) should
be relatively easy since I've basically already done them in my first
attempt at dynamic modules.

I've also tried to build Emacs on a windows vm at work using Stephen
Leake how-to but it fails at the configure stage saying that the build
system is not supported (pc-i686-mingw-32 ...from memory). I have to
look more into it.

Re: Dynamic loading progress

[Stefan Monnier]

> I only have the error handling left to implement from the API. I'm not
> sure on how to catch error/signal and return to the module code.

Can you give us more details about you mean by the above.
Are you talking about how to handle things like calls to Fsignal, or to
Fthrow, or how to provide access to unwind-protect and condition-case in
to the module's?

> The problem with finalizer is they can't be used as destructors since
> the GC is not guaranteed to be called at the end of every
> dynamic/lexical scope (or is it?).

Indeed it's not guaranteed at all.  It's not even guaranteed that when you
call the GC, all dead objects will be collected.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

On Sat, Sep 12, 2015 at 10:42 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> Can you give us more details about you mean by the above.
> Are you talking about how to handle things like calls to Fsignal, or to
> Fthrow, or how to provide access to unwind-protect and condition-case in
> to the module's?

The case where a module calls an emacs function that ends up calling
signal/error. I don't know enough about how signaling and
unwind-protect work. It's just black stack magic for me right now :)

I think we just need to implement funcall (from the module API) like this:

    global error = 0

    module_funcall(fun, args):
        // wrap (protect?) this with the right code
        // - to keep the control
        // - set ret to nil and error to 1 in case of error
        ret = Ffuncall(fun, args)

        return ret

The error is accessible via error_get(), error_clear() and
error_check() in the module API. error_get() is currently redundant
with error_check() unless we decide to return detailed errors.

I didn't think about the case where a module calls Fthrow but my guess
is it will just work. I have to test thoroughly what I have already
anyway, I'll see if it works.

> Indeed it's not guaranteed at all.  It's not even guaranteed that when you
> call the GC, all dead objects will be collected.

Ok. Finalizers are better than nothing I guess.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2835 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am So., 13. Sep.
2015 um 01:11 Uhr:

> On Sat, Sep 12, 2015 at 10:42 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
> > Can you give us more details about you mean by the above.
> > Are you talking about how to handle things like calls to Fsignal, or to
> > Fthrow, or how to provide access to unwind-protect and condition-case in
> > to the module's?
>
> The case where a module calls an emacs function that ends up calling
> signal/error. I don't know enough about how signaling and
> unwind-protect work. It's just black stack magic for me right now :)
>
>
They use longjmp, which doesn't work in modules. Your implementation needs
to make sure that no longjmp ever escapes to module code, i.e. that none of
the stackframes skipped by longjmp can lie inside a module. See what Daniel
wrote above:

"When Emacs calls a module function, the current thread's pending-error
flag will be clear.  When that module returns to Emacs, if the
thread's pending-error flag is set, Emacs signals the condition
corresponding to the current thread's error information.

When the module calls an Emacs routine that would ordinarily signal,
Emacs catches the signal at the stack frame just before control flow
would return to the module, sets the pending-error flag, and returns
to the module normally."


> I think we just need to implement funcall (from the module API) like this:
>
>     global error = 0
>
>     module_funcall(fun, args):
>         // wrap (protect?) this with the right code
>         // - to keep the control
>         // - set ret to nil and error to 1 in case of error
>

Here you probably need to call both internal_catch and
internal_condition_case.


>         ret = Ffuncall(fun, args)
>
>         return ret
>
> The error is accessible via error_get(), error_clear() and
> error_check() in the module API. error_get() is currently redundant
> with error_check() unless we decide to return detailed errors.
>
>
What do you mean with 'detailed errors'? At a minimum users need access to
the handler type (condition-case or catch), the catch tag, the error
symbol, and the error data. Most likely the stacktrace should also be
provided.


> I didn't think about the case where a module calls Fthrow but my guess
> is it will just work.
>

It uses a different handler type and therefore will probably not work out
of the box unless you call internal_catch.


>
> > Indeed it's not guaranteed at all.  It's not even guaranteed that when
> you
> > call the GC, all dead objects will be collected.
>
> Ok. Finalizers are better than nothing I guess.
>
>
Why would you need finalizers at all? There are usually wrong and useless
in languages with nondeterministic garbage collection.

[-- Attachment #2: Type: text/html, Size: 4026 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> The case where a module calls an emacs function that ends up calling
> signal/error.

OK, what are you worried about w.r.t that case?

> I don't know enough about how signaling and unwind-protect work. It's
> just black stack magic for me right now :)

Fsignal works using longjmp.  unwind-protect (and dynamic let-bindings)
works by adding things onto a stack which Fsignal will run just prior to
calling longjmp.

> I think we just need to implement funcall (from the module API) like this:

>     global error = 0

>     module_funcall(fun, args):
>         // wrap (protect?) this with the right code
>         // - to keep the control
>         // - set ret to nil and error to 1 in case of error
>         ret = Ffuncall(fun, args)

>         return ret

> The error is accessible via error_get(), error_clear() and
> error_check() in the module API. error_get() is currently redundant
> with error_check() unless we decide to return detailed errors.

I don't understand what kind of error handling you have in mind.
How/why/when would we use things like error_get, error_clear, etc...?

> I didn't think about the case where a module calls Fthrow but my guess
> is it will just work. I have to test thoroughly what I have already
> anyway, I'll see if it works.

Fthrow uses the same technique as Fsignal, and I think your intuition is
right: it should just work (for both).


        Stefan

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2122 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am So., 13. Sep. 2015 um
15:14 Uhr:

> > The case where a module calls an emacs function that ends up calling
> > signal/error.
>
> OK, what are you worried about w.r.t that case?
>

It's not possible to skip frames in module code using longjmp, so
module_funcall needs to set up handlers before calling Ffuncall.


>
> > I don't know enough about how signaling and unwind-protect work. It's
> > just black stack magic for me right now :)
>
> Fsignal works using longjmp.  unwind-protect (and dynamic let-bindings)
> works by adding things onto a stack which Fsignal will run just prior to
> calling longjmp.
>
> > I think we just need to implement funcall (from the module API) like
> this:
>
> >     global error = 0
>
> >     module_funcall(fun, args):
> >         // wrap (protect?) this with the right code
> >         // - to keep the control
> >         // - set ret to nil and error to 1 in case of error
> >         ret = Ffuncall(fun, args)
>
> >         return ret
>
> > The error is accessible via error_get(), error_clear() and
> > error_check() in the module API. error_get() is currently redundant
> > with error_check() unless we decide to return detailed errors.
>
> I don't understand what kind of error handling you have in mind.
> How/why/when would we use things like error_get, error_clear, etc...?
>

Because setjmp/longjmp and therefore Fthrow/Fsignal don't work, you need to
design a different error handling API that doesn't rely on non-local jumps.
There are three options: return values, output parameters, thread-local
state. All three have their pros and cons; Daniel opted for the third. The
error_... functions are needed to access that thread-local state.


>
> > I didn't think about the case where a module calls Fthrow but my guess
> > is it will just work. I have to test thoroughly what I have already
> > anyway, I'll see if it works.
>
> Fthrow uses the same technique as Fsignal, and I think your intuition is
> right: it should just work (for both).
>

No, it can't. See Daniel's initial posting where he designed the module
API.

[-- Attachment #2: Type: text/html, Size: 2941 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> It's not possible to skip frames in module code using longjmp, so

Why not?


        Stefan

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 317 bytes --]

On 09/13/2015 01:31 PM, Stefan Monnier wrote:
>> It's not possible to skip frames in module code using longjmp, so
> 
> Why not?

Because most C code isn't expecting to be unwound. Forcing non-local
flow control on module code is completely unacceptable. Emacs needs to
return with an error indication set.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

>>> It's not possible to skip frames in module code using longjmp, so
>> Why not?
> Because most C code isn't expecting to be unwound.
> Forcing non-local flow control on module code is completely
> unacceptable.  Emacs needs to return with an error indication set.

Hmm... I'm still not sure what's the issue.

AFAICT, the only odd case I can think of is the following:

   1- Emacs calls some module code via the new API.
   2- This code (which is Emacs-specific) will know about Fthrow.
   3- It may call some non-Emacs-specific code from some other library.
   4- This non-Emacs-specific code calls back to some Emacs-specific function.
   5- This Emacs-specific function calls Fthrow/Fsignal.

Where the problem only shows if/when we reach point 5.

This problem can be handled between 4 and 5 by using an appropriate
internal_condition_case.

Is there some other situation you're thinking of?


        Stefan

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2071 bytes --]

On 09/13/2015 06:58 PM, Stefan Monnier wrote:
>>>> It's not possible to skip frames in module code using longjmp, so
>>> Why not?
>> Because most C code isn't expecting to be unwound.
>> Forcing non-local flow control on module code is completely
>> unacceptable.  Emacs needs to return with an error indication set.
> 
> Hmm... I'm still not sure what's the issue.
> 
> AFAICT, the only odd case I can think of is the following:
> 
>    1- Emacs calls some module code via the new API.
>    2- This code (which is Emacs-specific) will know about Fthrow.
>    3- It may call some non-Emacs-specific code from some other library.
>    4- This non-Emacs-specific code calls back to some Emacs-specific function.
>    5- This Emacs-specific function calls Fthrow/Fsignal.
> 
> Where the problem only shows if/when we reach point 5.
> 
> This problem can be handled between 4 and 5 by using an appropriate
> internal_condition_case.
> 
> Is there some other situation you're thinking of?

Nobody is going to consider that case. It'll be hard enough to get
developers to properly unwind their own code, especially because we're
only talking about a case that's active when things go wrong. Of course
we can make non-local control flow work: it works well enough inside Emacs.

I'm objecting to this approach because it's a bug machine, especially
without the Emacs Lisp and GC machinery. It breaks C++ RAII, because
there's no way the longjmp out of Emacs can run C++ destructors. Rust
cleanups, and lots of other runtimes are similarly broken. You're asking
developers to cope with Emacs deciding simply never to return from most
module API calls. That's okay for Emacs internals, but it's very awkward
for code having to talk to Emacs.

There's no need to do it that way either. Storing the exception
information thread-locally (which means globally unless we ever actually
get threads) works with C-ABI semantics, is easier to reason about for
developers not used to Emacs internals, and provides just as much error
information.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

>> AFAICT, the only odd case I can think of is the following:
>>
>> 1- Emacs calls some module code via the new API.
>> 2- This code (which is Emacs-specific) will know about Fthrow.
>> 3- It may call some non-Emacs-specific code from some other library.
>> 4- This non-Emacs-specific code calls back to some Emacs-specific function.
>> 5- This Emacs-specific function calls Fthrow/Fsignal.
>>
>> Where the problem only shows if/when we reach point 5.
>>
>> This problem can be handled between 4 and 5 by using an appropriate
>> internal_condition_case.
>>
>> Is there some other situation you're thinking of?

> Nobody is going to consider that case.

Which other case are you thinking about then?

> It'll be hard enough to get developers to properly unwind their own
> code, especially because we're only talking about a case that's active
> when things go wrong. Of course we can make non-local control flow
> work: it works well enough inside Emacs.

You lost me.

> I'm objecting to this approach because it's a bug machine, especially

What is "this approach"?

> without the Emacs Lisp and GC machinery.

AFAIK we do have the Elisp and the GC machinery available.

> It breaks C++ RAII, because there's no way the longjmp out of Emacs
> can run C++ destructors.

This seems to refer to the "point 5" above.  As mentioned it's easy to
prevent the longjmp from escaping back into the surrounding non-Emacs
(e.g. C++) code.


        Stefan

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 3491 bytes --]

On 09/13/2015 09:18 PM, Stefan Monnier wrote:
>> It'll be hard enough to get developers to properly unwind their own
>> code, especially because we're only talking about a case that's active
>> when things go wrong. Of course we can make non-local control flow
>> work: it works well enough inside Emacs.
> 
> You lost me.
> 
>> I'm objecting to this approach because it's a bug machine, especially
> 
> What is "this approach"?

Specifically, I don't think Emacs public module API functions should
ever return non-locally. "This approach" means longjmping out to
condition-case instead of returning normally from Emacs module API
functions.

> 
>> without the Emacs Lisp and GC machinery.
> 
> AFAIK we do have the Elisp and the GC machinery available.

Yes, but the whole point of the module system is to access non-Emacs
libraries, and these libraries will have its own resource tracking
systems, which won't integrate with the Emacs GC core.

Longjmp in the Emacs core is convenient: if you want to make an Emacs
Lisp string, you can call make_string, and even if the next line of the
program longjmps, the GC will take care of releasing the resources
associated with that string.

Imagine if you had to individually arrange to call free on both the
success and error paths for every string you wanted to allocate ---
that's what life will will be like for module authors. I don't imagine
most modules will be using Emacs GCed resources for their own data, so
they'll have to arrange for cleanups manually.

>> It breaks C++ RAII, because there's no way the longjmp out of Emacs
>> can run C++ destructors.
> 
> This seems to refer to the "point 5" above.  As mentioned it's easy to
> prevent the longjmp from escaping back into the surrounding non-Emacs
> (e.g. C++) code.

I agree that it's possible. I don't agree that it's easy.

If the interface is like internal_condition_case, developers will have
to define another C-level function to act as an longjmp barrier, and
they'll have to do that for every call from their module to the Emacs
core. That's a lot of boilerplate. I'm afraid that developers will skip
this boilerplate and just make Emacs API calls directly, leading to the
presence of latent bugs that are hard to diagnose and fix. This sloppy
approach will appear to work fine, but things can go very wrong in rare
circumstances.

The problems I foresee are of the form "Emacs crashes when I hit C-g
with just the right timing", or "Emacs leaks memory if find-file fails
while this module calls it".  You can imagine Emacs deadlocking if a
module takes a long, Emacs longjmps past that lock's release, and then
later (perhaps much later) we try to take that lock and never acquire it.

Sure, it's possible to introduce similar bugs by not checking error
codes or some global error flag. The difference is that the danger is
better-understood (since many more C programs use error codes or error
flags than use longjmp), and we can diagnose the problems arising from a
failed error check more reliably than we can diagnose the problems that
can arise from a failed stack cleanup.

For example, if we use a global error flag, and we see an Emacs module
API function called with this error flag set, we can abort immediately,
pinpointing the problem. Or we can fail the call, warn the user, or do
anything else sensible. We can't even _detect_ the kind of damage
produced by missing stack cleanups.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> Yes, but the whole point of the module system is to access non-Emacs
> libraries,

But the issue only comes up once these non-Emacs libraries call back to
Emacs functions (i.e. step 5 in my example).  So only those calls need
to be protected somehow from non-local exits.

> and these libraries will have its own resource tracking
> systems, which won't integrate with the Emacs GC core.

Right, just like the libraries we currently link to from core Emacs
(libxml, libgnutls, libX, ...).   They *very rarely* call back to
Emacs functions.

> Imagine if you had to individually arrange to call free on both the
> success and error paths for every string you wanted to allocate ---
> that's what life will will be like for module authors.

No.  Only for those who choose to use use manually-managed data.

> I don't imagine most modules will be using Emacs GCed resources for

I do.

And I think we should encourage them to do so.  For that reason we want
to add finalizers, for example.

>> This seems to refer to the "point 5" above.  As mentioned it's easy to
>> prevent the longjmp from escaping back into the surrounding non-Emacs
>> (e.g. C++) code.
> I agree that it's possible. I don't agree that it's easy.

Of course it is.  We already do that in Emacs's core with things like
safe_call.


        Stefan

Re: Dynamic loading progress

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Yes, but the whole point of the module system is to access non-Emacs
>> libraries,
>
> But the issue only comes up once these non-Emacs libraries call back to
> Emacs functions (i.e. step 5 in my example).  So only those calls need
> to be protected somehow from non-local exits.

I think Daniel's point is that most module developers will not be aware
of this issue, and will be totally lost when it occurs.

In addition, how many libraries document whether they are "longjmp
safe"? I suspect that actually depends on compiler options, so it's up
to the packagers to ensure it.

Imagine someone used to writing Java code, with the attitude "the
language is totally safe, I don't need to worry about details"; now they
try to write an Emacs module, calling out to Java code all over the
place. Chaos ensues, and Emacs gets another bad wrap as "unusable".

So if the Emacs module API can be made to support those developers, it
would be a good thing.

I think his point is valid, although it would be nice for expert module
developers to be able to bypass the "safe call" if it becomes a
performance issue.

I don't think we can make writing an Emacs module as easy as writing a
Java class, but we should consider doing as much as we can to make it
easier/safer for the novice.

On the other hand, being forced to handle an exception instead of
remembering to check a status code is a proven win. So perhaps there is
a way to make all module developers aware of this issue; a well written,
short "module developer's checklist" would help here.

>> Imagine if you had to individually arrange to call free on both the
>> success and error paths for every string you wanted to allocate ---
>> that's what life will will be like for module authors.
>
> No.  Only for those who choose to use use manually-managed data.

Some module authors won't realize they've chosen that.

-- 
-- Stephe

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Mon, 14 Sep 2015 09:36:10 -0500
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> Yes, but the whole point of the module system is to access non-Emacs
> >> libraries,
> >
> > But the issue only comes up once these non-Emacs libraries call back to
> > Emacs functions (i.e. step 5 in my example).  So only those calls need
> > to be protected somehow from non-local exits.
> 
> I think Daniel's point is that most module developers will not be aware
> of this issue, and will be totally lost when it occurs.

And Stefan's point is that such an occurrence is highly improbable, to
say the least.

> In addition, how many libraries document whether they are "longjmp
> safe"?

If the longjmp never happens, this is irrelevant.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2964 bytes --]

Why are you so insistent on using non-local exits for error handling?
Representing errors more conventionally is cheap. If there's even a
small chance that I'm right about the safety and maintainability
advantages of using purely local exits, why not err on the side of
caution? AFAICS, there's no advantage to using non-local exits and
plenty of risks.

On 09/14/2015 05:14 AM, Stefan Monnier wrote:
>> Yes, but the whole point of the module system is to access non-Emacs
>> libraries,
> 
> But the issue only comes up once these non-Emacs libraries call back to
> Emacs functions (i.e. step 5 in my example).  So only those calls need
> to be protected somehow from non-local exits.

Calls back to Emacs functions will be fairly common, because modules use
these functions to manipulate Emacs data on behalf of their own callers.
Any of these functions can quit. What you're imagining will be a rare
case will in fact be common.

I also don't think it's correct that the Emacs-specific module code will
work properly in the presence of modules. You're supposing too much
patience and knowledge on the part of third-party module authors who'll
want to provide bindings not only for Emacs, but for other systems as well.

>> and these libraries will have its own resource tracking
>> systems, which won't integrate with the Emacs GC core.
> 
> Right, just like the libraries we currently link to from core Emacs
> (libxml, libgnutls, libX, ...).   They *very rarely* call back to
> Emacs functions.
> 
>> Imagine if you had to individually arrange to call free on both the
>> success and error paths for every string you wanted to allocate ---
>> that's what life will will be like for module authors.
> 
> No.  Only for those who choose to use use manually-managed data.

Who will be common, because the entire point of writing modules is to
work with the _non_-Emacs world. I can imagine one module that has both
non-trivial calls back into Emacs and quite a bit of internal state in
suitable for management with Emacs primitives. It's called Python.

Modules aren't going to be limited to small data-centric thngs like
libjpeg, and we shouldn't design the extension system in a way that
discourages more complex kinds of integration.

>> I don't imagine most modules will be using Emacs GCed resources for
> 
> I do.
> 
> And I think we should encourage them to do so.  For that reason we want
> to add finalizers, for example.
> 
>>> This seems to refer to the "point 5" above.  As mentioned it's easy to
>>> prevent the longjmp from escaping back into the surrounding non-Emacs
>>> (e.g. C++) code.
>> I agree that it's possible. I don't agree that it's easy.
> 
> Of course it is.  We already do that in Emacs's core with things like
> safe_call.

safe_call blindly suppresses errors. Errors from module-invoked
functions to propagate normally from the perspective of surrounding Lisp
code.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> Representing errors more conventionally is cheap.

AFAIK "representing errors more conventionally" means wrapping every
single function we pass to the module API.  That's definitely not cheap
in terms of performance.

> If there's even a small chance that I'm right about the safety and
> maintainability advantages of using purely local exits,

Representing errors as exceptions also has advantages.

> why not err on the side of caution? AFAICS, there's no advantage to
> using non-local exits and plenty of risks.

There are very many advantages, on the contrary.

Also, I expect most module authors will be familiar with Emacs, and
having to learn a different style will be an extra hindrance.

>> But the issue only comes up once these non-Emacs libraries call back to
>> Emacs functions (i.e. step 5 in my example).  So only those calls need
>> to be protected somehow from non-local exits.
> Calls back to Emacs functions will be fairly common, because modules use
> these functions to manipulate Emacs data on behalf of their own callers.
> Any of these functions can quit. What you're imagining will be a rare
> case will in fact be common.

You're not talking about the "step 5" case.  You're talking about the
Emacs-specific module code calling Emacs functions.  Indeed, this will be
very common.  But to write this code, you will have to know a fair bit
about Emacs internals, anyway.  It will not look like your "typical C++
code".

> I also don't think it's correct that the Emacs-specific module code will
> work properly in the presence of modules. You're supposing too much
> patience and knowledge on the part of third-party module authors who'll
> want to provide bindings not only for Emacs, but for other systems as well.

I don't forsee droves of coders who no nothing about Emacs internals and
who start writing third party modules.

In any case, I think it would be easy to layer the kind of API you
imagine on top of the kind of API I imagine, whereas the reverse seems
to be much more difficult and/or much less efficient.

So I'd rather we go with a straightforward lean API for a start.
And we can develop an alternative API on top afterwards (it might even
be possible to implement this alt-API as a module).

>> Of course it is.  We already do that in Emacs's core with things like
>> safe_call.
> safe_call blindly suppresses errors.

You're nitpicking.  Of course we'd use something slighly different which
stores the error somewhere.  Th point remains that it's easy to provide
such a wrapper.

> Errors from module-invoked functions to propagate normally from the
> perspective of surrounding Lisp code.

[ Is there a verb missing, maybe, I can't quite parse this, sorry.  ]


        Stefan

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 5170 bytes --]

On 09/14/2015 10:48 AM, Stefan Monnier wrote:
>> Representing errors more conventionally is cheap.
> 
> AFAIK "representing errors more conventionally" means wrapping every
> single function we pass to the module API.  That's definitely not cheap
> in terms of performance.

Yes, we need an internal condition-case at the boundary between the
module API and the Emacs core. I can call setjmp 54 million times per
second. (100000000 runs in 183ms.) Performance is simply not a problem.

>> If there's even a small chance that I'm right about the safety and
>> maintainability advantages of using purely local exits,
> 
> Representing errors as exceptions also has advantages.

Besides performance? Longjmp for error handling has an _expressiveness_
advantage in the Emacs core, but that's because we use GC data
structures for everything. Not so with external modules.

>> why not err on the side of caution? AFAICS, there's no advantage to
>> using non-local exits and plenty of risks.
> 
> There are very many advantages, on the contrary.

In a non-local exit scheme intended for use with third-party code and
not supported by any standardized exception system (like C++ exceptions
or SEH), I see only risks.

> 
> Also, I expect most module authors will be familiar with Emacs, and
> having to learn a different style will be an extra hindrance.

I expect them to be familiar with Emacs. There is no need to make module
developers familiarize themselves with Emacs internals. The public API
should insulate them from that.

> 
>>> But the issue only comes up once these non-Emacs libraries call back to
>>> Emacs functions (i.e. step 5 in my example).  So only those calls need
>>> to be protected somehow from non-local exits.
>> Calls back to Emacs functions will be fairly common, because modules use
>> these functions to manipulate Emacs data on behalf of their own callers.
>> Any of these functions can quit. What you're imagining will be a rare
>> case will in fact be common.
> 
> You're not talking about the "step 5" case.  You're talking about the
> Emacs-specific module code calling Emacs functions.  Indeed, this will be
> very common.  But to write this code, you will have to know a fair bit
> about Emacs internals, anyway.  It will not look like your "typical C++
> code".

Why shouldn't it? We _can_ make it look just like typical C++ code, and
to encourage module development, we should. Why make life harder for people?

>> I also don't think it's correct that the Emacs-specific module code will
>> work properly in the presence of modules. You're supposing too much
>> patience and knowledge on the part of third-party module authors who'll
>> want to provide bindings not only for Emacs, but for other systems as well.
> 
> I don't forsee droves of coders who no nothing about Emacs internals and
> who start writing third party modules.

Plenty of developers know nothing about Java or Python or Ruby or C# or
V8 internals and write native extensions for these systems. Supporting
developers who know the Lisp side of Emacs and who don't want to be
bothered working in core is one of the most important use cases of the
module system.

> 
> In any case, I think it would be easy to layer the kind of API you
> imagine on top of the kind of API I imagine, whereas the reverse seems
> to be much more difficult and/or much less efficient.

On the contrary, layering a different non-local exit scheme on top of an
error flag scheme is elegant. Imagine a C++ module library that throws a
C++ exception upon returning from Emacs and seeing the error flag set,
the translates (just before returning from C++ to Emacs) that C++
exception back into an Emacs error. Now you have a non-local error
scheme that is exactly the right thing for both languages.

> So I'd rather we go with a straightforward lean API for a start.
> And we can develop an alternative API on top afterwards (it might even
> be possible to implement this alt-API as a module).

A local-exit-only API _is_ a lean API, at least from a semantics
perspective. It's not surprising. The cost of the error flag machinery
in terms of both performance and API specification space is minuscule as
well.

>>> Of course it is.  We already do that in Emacs's core with things like
>>> safe_call.
>> safe_call blindly suppresses errors.
> 
> You're nitpicking.  Of course we'd use something slighly different which
> stores the error somewhere.  Th point remains that it's easy to provide
> such a wrapper.

So now we need some kind of explicit error propagation machinery
*anyway*, plus we're making developers worry about non-local exits.
Let's skip the latter part.

> 
>> Errors from module-invoked functions to propagate normally from the
>> perspective of surrounding Lisp code.
> 
> [ Is there a verb missing, maybe, I can't quite parse this, sorry.  ]

s/to //

I don't want C-g to break when module, core, and lisp frames are mingled
on stacks, and I expect these situations to be fairly common, especially
if we support modules with names like "python" and "guile".


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

Just, FWIW we can keep talking about it, but my opinion is quite firmly
made.  I want the first API to provide access to the "naked" primitives
with no condition-case wrapping.


        Stefan


>>>>> "Daniel" == Daniel Colascione <dancol@dancol.org> writes:

> On 09/14/2015 10:48 AM, Stefan Monnier wrote:
>>> Representing errors more conventionally is cheap.
>> 
>> AFAIK "representing errors more conventionally" means wrapping every
>> single function we pass to the module API.  That's definitely not cheap
>> in terms of performance.

> Yes, we need an internal condition-case at the boundary between the
> module API and the Emacs core. I can call setjmp 54 million times per
> second. (100000000 runs in 183ms.) Performance is simply not a problem.

>>> If there's even a small chance that I'm right about the safety and
>>> maintainability advantages of using purely local exits,
>> 
>> Representing errors as exceptions also has advantages.

> Besides performance? Longjmp for error handling has an _expressiveness_
> advantage in the Emacs core, but that's because we use GC data
> structures for everything. Not so with external modules.

>>> why not err on the side of caution? AFAICS, there's no advantage to
>>> using non-local exits and plenty of risks.
>> 
>> There are very many advantages, on the contrary.

> In a non-local exit scheme intended for use with third-party code and
> not supported by any standardized exception system (like C++ exceptions
> or SEH), I see only risks.

>> 
>> Also, I expect most module authors will be familiar with Emacs, and
>> having to learn a different style will be an extra hindrance.

> I expect them to be familiar with Emacs. There is no need to make module
> developers familiarize themselves with Emacs internals. The public API
> should insulate them from that.

>> 
>>>> But the issue only comes up once these non-Emacs libraries call back to
>>>> Emacs functions (i.e. step 5 in my example).  So only those calls need
>>>> to be protected somehow from non-local exits.
>>> Calls back to Emacs functions will be fairly common, because modules use
>>> these functions to manipulate Emacs data on behalf of their own callers.
>>> Any of these functions can quit. What you're imagining will be a rare
>>> case will in fact be common.
>> 
>> You're not talking about the "step 5" case.  You're talking about the
>> Emacs-specific module code calling Emacs functions.  Indeed, this will be
>> very common.  But to write this code, you will have to know a fair bit
>> about Emacs internals, anyway.  It will not look like your "typical C++
>> code".

> Why shouldn't it? We _can_ make it look just like typical C++ code, and
> to encourage module development, we should. Why make life harder for people?

>>> I also don't think it's correct that the Emacs-specific module code will
>>> work properly in the presence of modules. You're supposing too much
>>> patience and knowledge on the part of third-party module authors who'll
>>> want to provide bindings not only for Emacs, but for other systems as well.
>> 
>> I don't forsee droves of coders who no nothing about Emacs internals and
>> who start writing third party modules.

> Plenty of developers know nothing about Java or Python or Ruby or C# or
> V8 internals and write native extensions for these systems. Supporting
> developers who know the Lisp side of Emacs and who don't want to be
> bothered working in core is one of the most important use cases of the
> module system.

>> 
>> In any case, I think it would be easy to layer the kind of API you
>> imagine on top of the kind of API I imagine, whereas the reverse seems
>> to be much more difficult and/or much less efficient.

> On the contrary, layering a different non-local exit scheme on top of an
> error flag scheme is elegant. Imagine a C++ module library that throws a
> C++ exception upon returning from Emacs and seeing the error flag set,
> the translates (just before returning from C++ to Emacs) that C++
> exception back into an Emacs error. Now you have a non-local error
> scheme that is exactly the right thing for both languages.

>> So I'd rather we go with a straightforward lean API for a start.
>> And we can develop an alternative API on top afterwards (it might even
>> be possible to implement this alt-API as a module).

> A local-exit-only API _is_ a lean API, at least from a semantics
> perspective. It's not surprising. The cost of the error flag machinery
> in terms of both performance and API specification space is minuscule as
> well.

>>>> Of course it is.  We already do that in Emacs's core with things like
>>>> safe_call.
>>> safe_call blindly suppresses errors.
>> 
>> You're nitpicking.  Of course we'd use something slighly different which
>> stores the error somewhere.  Th point remains that it's easy to provide
>> such a wrapper.

> So now we need some kind of explicit error propagation machinery
> *anyway*, plus we're making developers worry about non-local exits.
> Let's skip the latter part.

>> 
>>> Errors from module-invoked functions to propagate normally from the
>>> perspective of surrounding Lisp code.
>> 
>> [ Is there a verb missing, maybe, I can't quite parse this, sorry.  ]

> s/to //

> I don't want C-g to break when module, core, and lisp frames are mingled
> on stacks, and I expect these situations to be fairly common, especially
> if we support modules with names like "python" and "guile".

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 460 bytes --]

On 09/14/2015 05:55 PM, Stefan Monnier wrote:
> Just, FWIW we can keep talking about it, but my opinion is quite firmly
> made.  I want the first API to provide access to the "naked" primitives
> with no condition-case wrapping.

The "first API" being the one with which we have to maintain binary
compatibility? Well, at least I haven't lost on the merits of the
design. There's no shame in losing to "because I say so". I'll stop
wasting my time.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

Some updates.

I haven't made any progress on errors. I was reading on
condition-case, setjmp/longjmp and how it's used for
signaling/throwing. I actually knew very little about it. I shouldnt
be the one writing this stuff I think... Anyway I was looking into how
I would set up a "catch-all" wrapping code so that we can implement a
funcall that always returns and I still haven't figured it out. You
said you would rather have an "unsafe" funcall that sometimes not
return to module code so I will try to do that. And realisticly if we
want this in emacs 25 it's the only way (I don't feel confident enough
to do that bit properly and I'm the only one working on this). We can
always fix it afterwards or simply have a "safe" and an "unsafe"
version of funcall in the API. I personally think having a clean
simple API from the module writer POV is better (least surprise
principle) so I'm in favor of wrapping, in the long term.

I think changing the API in the early stage is fine and I don't worry
too much about binary API changes because it's only going to be used
with GPL plugins (i.e. modules can be fixed and recompiled, most of
the time).

I've added [1] a tool that can generate modules basic code from
templates (Makefile, source file, ERT test file) and can run module
tests in batch. It's going to be easier to get started and easier for
me to write tests. I've put everything in the modules/ subdir, I know
its a bit messy having a python script there along with a lisp file
(common elisp functions for module test) but it's my setup right now
and it saves me time (i.e. deal with it! :). We can remove or change
it afterwards if we want.

Here's how you use it:

    # after successfully compiling emacs
    % cd modules

    % ./modhelp.py init modt-mymod
    writing modt-mymod/test.el...
    writing modt-mymod/Makefile...
    writing modt-mymod/modt-mymod.c...
    done! you can run ./modhelp.py test modt-mymod

    % ./modhelp.py test modt-mymod
    [*] ./modt-mymod: ------- start -------
    [*] ./modt-mymod: running make
    gcc -ggdb3 -Wall -I../../src -fPIC -c modt-mymod.c
    gcc -shared  -o modt-mymod.so modt-mymod.o
    ../../lib-src/make-docfile modt-mymod.c > modt-mymod.doc
    rm modt-mymod.o
    [*] ./modt-mymod: running test
    ../src/emacs -batch -L . -l ert -l ./modt-mymod/test.el -f
ert-run-tests-batch
    Running 1 tests (2015-09-24 14:33:44+0200)
       passed  1/1  modt-mymod-fun-test

    Ran 1 tests, 1 results as expected (2015-09-24 14:33:44+0200)


    [*] 1/1 MODULES OK

Running the `test` command without specifying the module tests all
module. (Hm.. I just noticed modules makefiles were ignored/not
commited by git. I've just fixed it)

I've started working on the configure.ac to make module a build time
option (still a WIP, I've only commited it to be able to work on it
from other computers).

1: https://github.com/aaptel/emacs-dynamic-module/commit/46c5f032ebf49275e346cae015dc39cbe3f8ebea

Re: Dynamic loading progress

[Stefan Monnier]

> I haven't made any progress on errors. I was reading on
> condition-case, setjmp/longjmp and how it's used for
> signaling/throwing. I actually knew very little about it. I shouldnt
> be the one writing this stuff I think...

That's OK.  We can add it later.

> Anyway I was looking into how
> I would set up a "catch-all" wrapping code so that we can implement a
> funcall that always returns and I still haven't figured it out.

You can grep for "safe_call" to get an idea.

> You said you would rather have an "unsafe" funcall that sometimes not
> return to module code so I will try to do that.

IIUC you've done that already, because it requires no special effort.
That's why it's the path I want to take first.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

I've implemented signal() and I've made the module support
configurable (disabled by default). Except for signal() all error
related functions are unimplemented (they are not useful because
funcall() doesn't return to module code on error right now). I've also
added a signal test.

The make_string() now takes the length of the string as a parameter.

Right now module are loaded with `module-load'. I need to modify
`load' so that it also looks for modules (which will make `require'
work).

How should module docstring be written? We either:

* pick a new syntax and teach make-docfile how to parse it. This is
the clean approach IMO, but it requires more work. We can use a
sufficiently simple syntax so that it can be embedded in comments in
other languages than C.
* define DEFVAR/DEFUN macro in module API header file to a noop so
that make-docfile can parse it as it is.

As for the finalizer, I wanted to add a function in the API:

/* Finalizer prototype */
typedef int (*emacs_finalizer_function)(void *ptr);

emacs_value make_user_ptr (emacs_env *env,
               void *ptr, emacs_finalizer_function *fin);


Or we can have a full "method table" with printer function and other
things we might need in the future. Although It's a bit overkill with
just finalizer/printer...

/* Printer prototype */
typedef emacs_value (*emacs_printer_function)(void *ptr);

/* User-ptr operations */
typedef struct {
  emacs_finalizer_function *fin;
  emacs_printer_function *print;
  /* ... */
} emacs_user_ptr_ops;

emacs_value make_user_ptr (emacs_env *env,
               void *ptr, emacs_user_ptr_ops *ops);

Re: Dynamic loading progress

[Stefan Monnier]

> I've implemented signal()

What means "implemented signal()"?  You mean, you export Fsignal to
the modules?

> Except for signal() all error related functions are unimplemented
> (they are not useful because funcall() doesn't return to module code
> on error right now).

I'm not sure what that means.  What other error-related functions?
How does this relate to whether or not "funcall() returns to module code
on error right now"?  What means "returns to module code"?

Do you provide "cdr"?  If so, what does it do when a module calls it on
a string object?  I'd expect it to signal the error and longjmp to the
nearest condition-case that catches it (presumably from outside of the
module code, unless the module code setup condition_case as well).

> * pick a new syntax and teach make-docfile how to parse it. This is
> the clean approach IMO, but it requires more work. We can use a
> sufficiently simple syntax so that it can be embedded in comments in
> other languages than C.
> * define DEFVAR/DEFUN macro in module API header file to a noop so
> that make-docfile can parse it as it is.

You're talking about how to get docstrings from the source code, but
I think we should expect Emacs to need/fetch them during "load" at which
point we'll be looking at the compiled code.  So the first question is
where should those docstrings (regardless of whether they were found and
prepared by make-docfile or by some new tool) be located?  Inside the
.so file?  Is the source code which should be placed nearby?  In some
kind of separate DOC file?

> Or we can have a full "method table" with printer function and other
> things we might need in the future. Although It's a bit overkill with
> just finalizer/printer...

Even without a printer function, I think it makes sense at least to
provide some kind of "type" information so that it's obvious when two
"user objects" are "of the same type".  The method table can play the
role of a "type".

Also, it might make sense to let Emacs build this "method table" object.
So instead of exposing an emacs_user_ptr_ops structure, export
a function which creates a pointer to such an object.  This way the
actual emacs_user_ptr_ops structure can more easily evolve, and Emacs
can more easily fill the various slots with default values where
applicable (e.g. one slot might point to the module that owns the object
and might better be filled by the Emacs core than by the module itself).


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

I wrote what we agreed on...

On Mon, Sep 28, 2015 at 5:01 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> What means "implemented signal()"?  You mean, you export Fsignal to
> the modules?

yes

>> Except for signal() all error related functions are unimplemented
>> (they are not useful because funcall() doesn't return to module code
>> on error right now).
>
> I'm not sure what that means.  What other error-related functions?

Look at error_* function in the API (src/emacs_module.h [1]).
error_get() lets you get the signal symbol & data. error_check()
simply tells you if there was an error and error_clear() clears
current error. They will be implemented once we have a safe funcall.
The safe funcall() will have a catch-all condition-case and will set a
global error state than will be accessible via the API. (Daniel's
design)

> on error right now"?  What means "returns to module code"?

"Returns to module code" means when a function (funcall in this case)
exits normally via return (no longjmp). Since funcall() is called by
modules, when it returns it returns to module code. Hence "returns to
module code".

> Do you provide "cdr"?

no, not directly. You have to intern "cdr" and call funcall().

>  If so, what does it do when a module calls it on
> a string object?  I'd expect it to signal the error and longjmp to the
> nearest condition-case that catches it (presumably from outside of the
> module code, unless the module code setup condition_case as well).

Correct. condition-case is not part of the API though. Like for cdr
you have to intern "condition-case", setup the lisp forms as arguments
yourself and funcall() it. Haven't tested it yet.

> You're talking about how to get docstrings from the source code, but
> I think we should expect Emacs to need/fetch them during "load" at which
> point we'll be looking at the compiled code.  So the first question is
> where should those docstrings (regardless of whether they were found and
> prepared by make-docfile or by some new tool) be located?  Inside the
> .so file?  Is the source code which should be placed nearby?  In some
> kind of separate DOC file?

Like with my old module branch, I was thinking of having multiple DOC files.
When a module is built a DOC file is also generated from the source of
the module. When a module is loaded it looks for the module DOC file
and uses it. This involves changing the doc field of a symbol from int
to Lisp_Object that stores ("path/to/doc" . POSITION), iirc. See
previous discussions we had on this.

We can always do things differently, its not too late. The pro of
docfile is that you can also store the position of the definition in
the source code, which is always helpful.

> Even without a printer function, I think it makes sense at least to
> provide some kind of "type" information so that it's obvious when two
> "user objects" are "of the same type".  The method table can play the
> role of a "type".
>
> Also, it might make sense to let Emacs build this "method table" object.
> So instead of exposing an emacs_user_ptr_ops structure, export
> a function which creates a pointer to such an object.  This way the
> actual emacs_user_ptr_ops structure can more easily evolve, and Emacs
> can more easily fill the various slots with default values where

Good idea.

> applicable (e.g. one slot might point to the module that owns the object
> and might better be filled by the Emacs core than by the module itself).

Currently each instance of the emacs_env (the struct that holds
function pointers to the API that is created upon request from
modules) has a unique id used for module identification. That way the
module API implementation can know which module is using each API
function.

1: https://github.com/aaptel/emacs-dynamic-module/blob/dynamic-modules-2/src/emacs_module.h

Re: Dynamic loading progress

[Davis Herring]

> Correct. condition-case is not part of the API though. Like for cdr
> you have to intern "condition-case", setup the lisp forms as arguments
> yourself and funcall() it. Haven't tested it yet.

You can't funcall special forms; thus in part the need for things like
internal_condition_case.  You'd have to use eval.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Dynamic loading progress

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> * pick a new syntax and teach make-docfile how to parse it. This is
>> the clean approach IMO, but it requires more work. We can use a
>> sufficiently simple syntax so that it can be embedded in comments in
>> other languages than C.
>> * define DEFVAR/DEFUN macro in module API header file to a noop so
>> that make-docfile can parse it as it is.
>
> You're talking about how to get docstrings from the source code, but
> I think we should expect Emacs to need/fetch them during "load" at which
> point we'll be looking at the compiled code.  So the first question is
> where should those docstrings (regardless of whether they were found and
> prepared by make-docfile or by some new tool) be located?  Inside the
> .so file?  Is the source code which should be placed nearby?  In some
> kind of separate DOC file?

At one point, we agreed that the module C code doc strings only need to be
accessible after the module is loaded. That can be done with a function
call in the module initialize function, along with the defun call. In
that case, the doc strings are in the .so file.

If the module author wants some doc strings to be accessible before load
time, they can be on autoloaded functions in elisp code.

-- 
-- Stephe

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2091 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Sa., 26. Sep.
2015 um 16:56 Uhr:

> How should module docstring be written? We either:
>
> * pick a new syntax and teach make-docfile how to parse it. This is
> the clean approach IMO, but it requires more work. We can use a
> sufficiently simple syntax so that it can be embedded in comments in
> other languages than C.
> * define DEFVAR/DEFUN macro in module API header file to a noop so
> that make-docfile can parse it as it is.
>

Library headers should respect namespace conventions. Because in C all
symbols are global, it's important to pick names that are unlikely to
clash. Usually this is done by prefixing all names with a string that is
thought to be sufficiently unique, like "emacs" in your case. But
DEFVAR/DEFUN don't follow that convention, so they can't be used in the
header file.
There are some documentation frameworks available, e.g. Doxygen. It might
be a good idea to use one of them instead of trying to define your own
syntax.


>
> As for the finalizer, I wanted to add a function in the API:
>
> /* Finalizer prototype */
> typedef int (*emacs_finalizer_function)(void *ptr);
>
> emacs_value make_user_ptr (emacs_env *env,
>                void *ptr, emacs_finalizer_function *fin);
>

I still don't understand why we need finalizers.  Using finalizers in
languages without deterministic garbage collection is generally an
antipattern.


>
>
> Or we can have a full "method table" with printer function and other
> things we might need in the future. Although It's a bit overkill with
> just finalizer/printer...
>
> /* Printer prototype */
> typedef emacs_value (*emacs_printer_function)(void *ptr);
>
> /* User-ptr operations */
> typedef struct {
>   emacs_finalizer_function *fin;
>   emacs_printer_function *print;
>   /* ... */
> } emacs_user_ptr_ops;
>
> emacs_value make_user_ptr (emacs_env *env,
>                void *ptr, emacs_user_ptr_ops *ops);
>

What is the use case for this? Is the "user_ptr" functionality really
needed?

[-- Attachment #2: Type: text/html, Size: 2800 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

On Mon, Sep 28, 2015 at 5:42 PM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> What is the use case for this? Is the "user_ptr" functionality really
> needed?

Most libraries creates opaque pointers type you re-use for each calls.
See some libarchive samples for example [1]

Imagine porting libarchive to emacs as a module. You would use a
struct archive pointer stored as a user pointer in an Emacs value. And
the new lisp functions would be very similar to the libarchive API
i.e. taking the struct archive handle as a parameter.

We might not have deteminist finalizer but we could come up with a
lisp macro similar to Python 'with' keyword [2] that would set up an
unwind-protect that calls the finalizer.

1: https://github.com/libarchive/libarchive/wiki/Examples#List_contents_of_Archive_stored_in_File
2: http://effbot.org/zone/python-with-statement.htm

Re: Dynamic loading progress

[Stefan Monnier]

> There are some documentation frameworks available, e.g. Doxygen.
> It might be a good idea to use one of them instead of trying to define
> your own syntax.

Using doxygen sounds like a good idea.

Since the only thing we really need to formalize is the place and format
of the "compiled" docstrings (e.g. "a DOC file"), there could even be
several tools available which allow placing the docstring in various
ways in the source code.  And each module could use the tool it prefers.

> I still don't understand why we need finalizers.  Using finalizers in
> languages without deterministic garbage collection is generally an
> antipattern.

Maybe that's "generally" the case, but when you need to interface a GC'd
language with a manually-managed library, it's the standard technique
(and pretty much only/best).

The finalizer of course has to be "invisible".  Typically, it will do
nothing more than call "free" (or something equivalent) on a few objects.

> What is the use case for this? Is the "user_ptr" functionality really
> needed?

Think of a module which tries to provide bignums to Elisp via GMP.
It'll need to provide the new "bignum" objects, and they'll be
internally represented as GMP objects (which are manually
allocated/freed).  So it has to define a new Lisp_Object type, and some
of its fields are not Lisp_Objects but some pointer to an internal
thingy that needs to be freed iff the Lisp_Object is reclaimed.


        Stefan

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

On 09/28/2015 12:39 PM, Stefan Monnier wrote:
>> There are some documentation frameworks available, e.g. Doxygen.
>> It might be a good idea to use one of them instead of trying to define
>> your own syntax.
> 
> Using doxygen sounds like a good idea.
> 
> Since the only thing we really need to formalize is the place and format
> of the "compiled" docstrings (e.g. "a DOC file"), there could even be
> several tools available which allow placing the docstring in various
> ways in the source code.  And each module could use the tool it prefers.

Do we even need this optimization in the modern world? Life would be
simpler if we could just get rid of the lazy-loaded documentation string
machinery.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> Do we even need this optimization in the modern world? Life would be
> simpler if we could just get rid of the lazy-loaded documentation string
> machinery.

I don't have an opinion on that.  Regardless of whether it's lazy-loaded
(from a DOC file or from a special section in the .so file or ...) or
not, we need to define specifically where the doc will be found.

Maybe if non-lazy-loading is used the doc can be provided in the source
code without any special processing.  That'd be nice.


        Stefan

Re: Dynamic loading progress

[Tom Tromey]

>>>>> "Aurélien" == Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

[ module progress ]

Tonight I tried to use the module code to write an FFI module using
libffi.  I found some holes that make this difficult to do.

Aurélien> As for the finalizer, I wanted to add a function in the API:

Aurélien> Or we can have a full "method table" with printer function and other
Aurélien> things we might need in the future. Although It's a bit overkill with
Aurélien> just finalizer/printer...

It would be great to be able to make new types from C.  That would let
me wrap up various C objects I'd like to return opaquely to Lisp.
Wrapping pointers would be good enough (for now I am using bit vectors)
but having more control would be much better.

Overkill here is better.  You'll want to extend this vtable over time,
for example to add a GC mark function.  I don't particularly care if the
vtable is exposed or private.

Of course, if you can add types from C, why not from Lisp?  But I guess
I could write a module to allow that :)


Here's my list of other things that I noticed.
Note that my module isn't working yet, so take it with a grain of salt.

* I found the emacs_runtime / emacs_env split strange.  I don't know
  what the purpose of this is.  Also there doesn't seem to be a way to
  free an emacs_env, which I suppose makes sense in a way; though
  shouldn't they simply be GC'd as well?

* module-call never frees "args".

* For error handling for now I ended up just calling Emacs stuff
  directly.

Tom

Re: Dynamic loading progress

[Aurélien Aptel]

On Sun, Oct 4, 2015 at 6:30 AM, Tom Tromey <tom@tromey.com> wrote:
> Tonight I tried to use the module code to write an FFI module using
> libffi.  I found some holes that make this difficult to do.

Cool :)

> It would be great to be able to make new types from C.  That would let
> me wrap up various C objects I'd like to return opaquely to Lisp.
> Wrapping pointers would be good enough (for now I am using bit vectors)
> but having more control would be much better.
>
> Overkill here is better.  You'll want to extend this vtable over time,
> for example to add a GC mark function.  I don't particularly care if the
> vtable is exposed or private.
>
> Of course, if you can add types from C, why not from Lisp?  But I guess
> I could write a module to allow that :)

We have to be careful with this vtable thing. I don't want to end up
with C++-level complexity with operator overloading, copy/move
constructor, etc. To store pointers you can currently use bit-vectors
as you did or also have a table that maps integers to pointers on the
module side. For example, a module could return new "types" using a
cons ('my-type . ID) where ID is an integer index into a pointer table
handled by the module author. It's more work for module writers but it
works and keeps the complexity of the module interface low.

The only thing we cannot do rigth now is finalizers. But looking
through the commits I saw that Daniel added support for "finalizer" in
emacs core but I haven't looked into what it actually does. Maybe we
can re-use this feature instead of doing it again basically.

> * I found the emacs_runtime / emacs_env split strange.  I don't know
>   what the purpose of this is.  Also there doesn't seem to be a way to

Daniel's design is modeled after the JNI spec where the env must be
used from the thread that created the runtime the env came from, IIRC.
I have added a "private" module_id field in the env structure which is
used to index a module global refs. I was also thinking that you could
have for example a module that lets you write sub-modules in python,
each with a different module_id but the same runtime.

>   free an emacs_env, which I suppose makes sense in a way; though
>   shouldn't they simply be GC'd as well?

Nothing can be unloaded in elisp but I didn't made that assumption for
modules and I've indexed global refs by module so that you can clear
them per module if we ever implement that.

> * module-call never frees "args".

Fixed, thanks.

> * For error handling for now I ended up just calling Emacs stuff
>   directly.

Ok.

If you have a public repo somewhere I'd be interested to have a look
at your experiements.

Re: Dynamic loading progress

[Aurélien Aptel]

On Sun, Oct 4, 2015 at 4:11 PM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:
> We have to be careful with this vtable thing. I don't want to end up
> with C++-level complexity with operator overloading, copy/move
><snip>
> module side. For example, a module could return new "types" using a
> cons ('my-type . ID) where ID is an integer index into a pointer table
> handled by the module author. It's more work for module writers but it
> works and keeps the complexity of the module interface low.

To be clear, I still prefer the opaque pointer type with the vtable
thingy solution. I was just showing alternatives.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Do., 24. Sep.
2015 um 14:45 Uhr:

> Some updates.
>
> I haven't made any progress on errors. I was reading on
> condition-case, setjmp/longjmp and how it's used for
> signaling/throwing. I actually knew very little about it. I shouldnt
> be the one writing this stuff I think... Anyway I was looking into how
> I would set up a "catch-all" wrapping code so that we can implement a
> funcall that always returns and I still haven't figured it out.


I have implemented this in a private fork:
https://github.com/phst/emacs-dynamic-module/commit/b14073ef8a522bdb90a8d7f0edc575edfd75da26
Note that code is only a suggestion how it could be implemented. It
deviates from Daniel's design by using return parameters instead of TLS,
but that's only a minor difference. The important part is the addition of
protected_call_n to eval.c.


> And realisticly if we
> want this in emacs 25 it's the only way (I don't feel confident enough
> to do that bit properly and I'm the only one working on this).


I feel relatively strongly about this, so I'd work on implementing this if
you'd like.


> We can
> always fix it afterwards or simply have a "safe" and an "unsafe"
> version of funcall in the API.


Going from the safe to the unsafe direction is easy if it's really
required, but going the other direction is hard. I'd suggest to only
implement the safe version for now and only add the unsafe version if
there's some data that shows it's required.


> I think changing the API in the early stage is fine and I don't worry
> too much about binary API changes because it's only going to be used
> with GPL plugins (i.e. modules can be fixed and recompiled, most of
> the time).
>
>
True, however once there's an API it's a good idea to keep it stable. I
expect people to start writing modules relatively soon once the interface
is available, so it's better to put a bit more effort into the interface.

[-- Attachment #2: Type: text/html, Size: 2953 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am Mo., 28. Sep. 2015 um
17:35 Uhr:

> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Do., 24. Sep.
> 2015 um 14:45 Uhr:
>
>> Some updates.
>>
>> I haven't made any progress on errors. I was reading on
>> condition-case, setjmp/longjmp and how it's used for
>> signaling/throwing. I actually knew very little about it. I shouldnt
>> be the one writing this stuff I think... Anyway I was looking into how
>> I would set up a "catch-all" wrapping code so that we can implement a
>> funcall that always returns and I still haven't figured it out.
>
>
> I have implemented this in a private fork:
> https://github.com/phst/emacs-dynamic-module/commit/b14073ef8a522bdb90a8d7f0edc575edfd75da26
> Note that code is only a suggestion how it could be implemented. It
> deviates from Daniel's design by using return parameters instead of TLS,
> but that's only a minor difference. The important part is the addition of
> protected_call_n to eval.c.
>

Here's a more minimal implementation using TLS (but we only have one
thread, the thread checking needs to be implemented as well, but that's
unrelated):
https://github.com/phst/emacs-dynamic-module/commit/41a8cb8a1d29346331fcaf1f28b6c5bee8c8f6ad
This is less than minimal because throw isn't dealt with.

[-- Attachment #2: Type: text/html, Size: 2219 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> Going from the safe to the unsafe direction is easy if it's really
> required, but going the other direction is hard.

You have it backward: it's easy to add a wrapper that catches signals,
but it's *impossible* to "uncatch" a signal that was caught (you can at
best re-throw that signal, which is a close-enough approximation in
many cases, but is still not the same).

> I'd suggest to only implement the safe version for now and only add
> the unsafe version if there's some data that shows it's required.

The only version that is required is the one that lets signals walk up
the stack freely.  I'm not opposed to also providing a "safe" version
that catches signals and turns them into something else, but the other
one is indispensable.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

First of all thank you for actually contributing with code, Philip.
Very much appreciated.

On Mon, Sep 28, 2015 at 5:35 PM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> I have implemented this in a private fork:
> https://github.com/phst/emacs-dynamic-module/commit/b14073ef8a522bdb90a8d7f0edc575edfd75da26
> Note that code is only a suggestion how it could be implemented. It deviates
> from Daniel's design by using return parameters instead of TLS, but that's
> only a minor difference. The important part is the addition of
> protected_call_n to eval.c.

I've looked at both branch. The one using an in-out return parameter
looks really annoying to use though. I think I prefer the "minimal"
implementation [1]. Using that version, we could add an
unsafe_funcall() to the API that doesnt do any signal checking so that
we have both.

Stefan, is this enough? Or should we actually expose `condition-case'
for finer-grained control?

1: https://github.com/phst/emacs-dynamic-module/commit/41a8cb8a1d29346331fcaf1f28b6c5bee8c8f6ad

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am So., 4. Okt.
2015 um 15:25 Uhr:

> First of all thank you for actually contributing with code, Philip.
> Very much appreciated.
>
> On Mon, Sep 28, 2015 at 5:35 PM, Philipp Stephani <p.stephani2@gmail.com>
> wrote:
> > I have implemented this in a private fork:
> >
> https://github.com/phst/emacs-dynamic-module/commit/b14073ef8a522bdb90a8d7f0edc575edfd75da26
> > Note that code is only a suggestion how it could be implemented. It
> deviates
> > from Daniel's design by using return parameters instead of TLS, but
> that's
> > only a minor difference. The important part is the addition of
> > protected_call_n to eval.c.
>
> I've looked at both branch. The one using an in-out return parameter
> looks really annoying to use though. I think I prefer the "minimal"
> implementation [1].


That's fine, it's Daniel's design, after all, and using TLS for error
reporting is common (e.g. errno). I've added a pull request for this:
https://github.com/aaptel/emacs-dynamic-module/pull/7
Another pull request implements checking whether calls are only made on the
main thread: https://github.com/aaptel/emacs-dynamic-module/pull/8


> Using that version, we could add an
> unsafe_funcall() to the API that doesnt do any signal checking so that
> we have both.
>

Daniel, what do you think about that?

[-- Attachment #2: Type: text/html, Size: 2239 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Di., 15. Sep. 2015 um
02:55 Uhr:

> Just, FWIW we can keep talking about it, but my opinion is quite firmly
> made.  I want the first API to provide access to the "naked" primitives
> with no condition-case wrapping.
>
>
This is wrong and dangerous. It makes dynamic loading far less useful than
it could be. Essentially modules e.g. written in C++ will have to consist
of a small C shim that makes IPC calls to some out-of-process server.

[-- Attachment #2: Type: text/html, Size: 793 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> This is wrong and dangerous.  It makes dynamic loading far less useful than
> it could be.  Essentially modules e.g. written in C++ will have to consist
> of a small C shim that makes IPC calls to some out-of-process server.

That is again nonsense: what is being requested (i.e. avoid non-local
exits) can be performed on top of the API I advocte, just as efficiently
as if it were done "natively" inside that API.


        Stefan

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

On 09/28/2015 12:26 PM, Stefan Monnier wrote:
>> This is wrong and dangerous.  It makes dynamic loading far less useful than
>> it could be.  Essentially modules e.g. written in C++ will have to consist
>> of a small C shim that makes IPC calls to some out-of-process server.
> 
> That is again nonsense: what is being requested (i.e. avoid non-local
> exits) can be performed on top of the API I advocte, just as efficiently
> as if it were done "natively" inside that API.

That's true, but only intuitively true for people with a great deal of
experience with finicky C code. I still think you're ignoring human
factors in making this decision.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

On 09/28/2015 08:25 AM, Philipp Stephani wrote:
> 
> 
> Stefan Monnier <monnier@iro.umontreal.ca
> <mailto:monnier@iro.umontreal.ca>> schrieb am Di., 15. Sep. 2015 um
> 02:55 Uhr:
> 
>     Just, FWIW we can keep talking about it, but my opinion is quite firmly
>     made.  I want the first API to provide access to the "naked" primitives
>     with no condition-case wrapping.
> 
> 
> This is wrong and dangerous. It makes dynamic loading far less useful
> than it could be. Essentially modules e.g. written in C++ will have to
> consist of a small C shim that makes IPC calls to some out-of-process
> server.

While I agree that making non-local exits part of the public API is a
titanic footgun and a mistake we'll regret for decades, it's not the
case that it forces non-C modules to use IPC shims. Emacs non-local
unwinding stops at the first matching condition_case on the stack, so as
long there's a condition-case frame between a stack frame that can't be
unwound with longjmp (say, one with C++ destructors) and the Emacs core,
the program will work correctly if Emacs longjmps.

Non-local returns are awful, yes, but they do not make it impossible to
integrate non-C languages with Emacs.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am Mo., 28. Sep. 2015 um
21:29 Uhr:

> On 09/28/2015 08:25 AM, Philipp Stephani wrote:
> >
> >
> > Stefan Monnier <monnier@iro.umontreal.ca
> > <mailto:monnier@iro.umontreal.ca>> schrieb am Di., 15. Sep. 2015 um
> > 02:55 Uhr:
> >
> >     Just, FWIW we can keep talking about it, but my opinion is quite
> firmly
> >     made.  I want the first API to provide access to the "naked"
> primitives
> >     with no condition-case wrapping.
> >
> >
> > This is wrong and dangerous. It makes dynamic loading far less useful
> > than it could be. Essentially modules e.g. written in C++ will have to
> > consist of a small C shim that makes IPC calls to some out-of-process
> > server.
>
> While I agree that making non-local exits part of the public API is a
> titanic footgun and a mistake we'll regret for decades, it's not the
> case that it forces non-C modules to use IPC shims. Emacs non-local
> unwinding stops at the first matching condition_case on the stack, so as
> long there's a condition-case frame between a stack frame that can't be
> unwound with longjmp (say, one with C++ destructors) and the Emacs core,
> the program will work correctly if Emacs longjmps.
>

The issue is that it's impossible for module authors to write such a
`condition-case' in the general case. While individual signals can be
caught, it's not possible to catch all signals using the condition-case
Lisp function. Similarly, there is no way to write a `catch' block that
would catch all tags; `catch' only allows catching individual tag symbols.


>
> Non-local returns are awful, yes, but they do not make it impossible to
> integrate non-C languages with Emacs.
>
>

[-- Attachment #2: Type: text/html, Size: 2412 bytes --]

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

On 09/28/2015 01:09 PM, Philipp Stephani wrote:
> The issue is that it's impossible for module authors to write such a
> `condition-case' in the general case. While individual signals can be
> caught, it's not possible to catch all signals using the condition-case
> Lisp function. Similarly, there is no way to write a `catch' block that
> would catch all tags; `catch' only allows catching individual tag symbols.

I believe Stefan is proposing adding a true catch-all to the module API,
like internal_condition_case.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> I believe Stefan is proposing adding a true catch-all to the module API,
> like internal_condition_case.

Right.  And/or a "safe_funcall" (which does both the catching and the
funcall).


        Stefan

Re: Dynamic loading progress

[Stephen J. Turnbull]

Daniel Colascione writes:

 > Calls back to Emacs functions will be fairly common, because
 > modules use these functions to manipulate Emacs data on behalf of
 > their own callers.

Real examples, please.  S?XEmacs has two decades of experience with
wrapping random libraries, Emacs has some too, and there just aren't
any that I can think of offhand.  I see no reason why modules in Emacs
will change that pattern; S?XEmacs has a 1.5 decades of experience
with require'able modules, and it hasn't changed there.

 > Who will be common, because the entire point of writing modules is
 > to work with the _non_-Emacs world. I can imagine one module that
 > has both non-trivial calls back into Emacs and quite a bit of
 > internal state in suitable for management with Emacs
 > primitives. It's called Python.

You have a serious licensing problem, I think, which is one of the
important reasons why we haven't seen very much in the way of complex
integration in the past.  But be that as it may....

You're kidding, right?  I can't imagine that somebody with the skills
in both Python and Emacs and the energy and time needed to write the
kind of deep integration you're talking about would be deterred by the
need to adapt a couple score lines of "suggested code" to their
preferred conception of error handling.

Of course it will be hard to write the boilerplate the first time, but
by the same token the first try will undoubtedly suck.  So the
implementation will *change* to fix the suckiness.  If the boilerplate
is embedded in the module API it will likely force the modules to
update to work with the new rules.  I think a module that seems to be
working well is very unlikely to be pleased by such churn.

So, you could be right in the long run, but I think it's the better
part of valor to not restrict modules in that way now.  Instead, put
the responsibility for error handling the callback itself first.
Modules that want to can provide their own child-proof wrappers.
After experience incorporate the best (plural, if appropriate) of
those wrappers in Emacs itself, either embedded in the module API or
as an optional standard wrapper.

 > safe_call blindly suppresses errors. Errors from module-invoked
 > functions to propagate normally from the perspective of surrounding
 > Lisp code.

Show us the code to "propagate errors normally to surrounding Lisp
code".  XEmacs has its own safe_call-type functions that do somewhat
better than merely suppressing all errors, but they're heavy and hard
to use correctly -- and there we know at least that the calling side
of the call is XEmacs (the other side is typically low-level I/O code
that uses Lisp to format output or parse input).  But for modules,
since callbacks can be anything and the callback will be separated
from the "surrounding Lisp code" by arbitrary amounts of non-Lisp code
doing unknown things, I doubt you can do "normal".  But let's see how
close you can get.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

On 09/14/2015 07:56 PM, Stephen J. Turnbull wrote:
> Daniel Colascione writes:
>  > primitives. It's called Python.
> 
> You have a serious licensing problem, I think, which is one of the
> important reasons why we haven't seen very much in the way of complex
> integration in the past.

The rest of the discussion aside: Python's current license is
GPLv3-compatible, isn't it? gnu.org and Python's documentation both
claim it is.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stephen J. Turnbull]

Daniel Colascione writes:
 > On 09/14/2015 07:56 PM, Stephen J. Turnbull wrote:

 > The rest of the discussion aside: Python's current license is
 > GPLv3-compatible, isn't it? gnu.org and Python's documentation both
 > claim it is.

AFAIK IANAL TINLA.  That's not the issue I have in mind.

Pragmatically speaking, even if it's not possible to legally prohibit
you from distributing such a module, I think Richard would veto
incorporation of such a module in Emacs, because it would make it
possible to silently link Emacs to proprietary code via a Python
plugin.

Re: Dynamic loading progress

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Daniel Colascione writes:
>  > On 09/14/2015 07:56 PM, Stephen J. Turnbull wrote:
>
>  > The rest of the discussion aside: Python's current license is
>  > GPLv3-compatible, isn't it? gnu.org and Python's documentation both
>  > claim it is.
>
> AFAIK IANAL TINLA.  That's not the issue I have in mind.
>
> Pragmatically speaking, even if it's not possible to legally prohibit
> you from distributing such a module, I think Richard would veto
> incorporation of such a module in Emacs, because it would make it
> possible to silently link Emacs to proprietary code via a Python
> plugin.

Well, regarding the "proprietary plugin" issue it would seem most
expedient to have the plugins run in the Emacs GC and exception model
and, where third-party independently developed libraries are to be
integrated, to have to compile/byte-compile/assemble some
library-specific wrapper encapsulating/mapping the exceptions/callbacks
to the Emacs model.

I have no idea regarding the technical feasibility of various different
approaches.  But regarding the "circumvent GPLv3" angle, having a
default model of completely separate exception/memory handling and FFI
that allows a drop-in direct binary use of libraries not specifically
developed for use with Emacs, while certainly offering the maximum of
convenience in one regard, forms a reasonably clearly defined license
barrier for "application as a whole" forming a clear stop for the reach
of the GPLv3.

So having our binary interfaces and calling conventions and
memory/exception handling default to "like Emacs does" is not just
Lisp-friendly but is also keeping our license enforcement options more
conservative.  While my personal opinion is that licensing
considerations should not justify complications amounting to
uselessness, I see nothing wrong with letting them serve as tie-breaker.

-- 
David Kastrup

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1082 bytes --]

On 09/15/2015 01:45 AM, David Kastrup wrote:
> So having our binary interfaces and calling conventions and
> memory/exception handling default to "like Emacs does" is not just
> Lisp-friendly but is also keeping our license enforcement options more
> conservative. 

I don't see how. Once we have a module that allows general-purpose
loading of code not intended for use in Emacs (no matter how hard that
initial module is to write), there will be literally nothing preventing
users making arbitrary GPLv3-compatible modules that allow users to
chain-load non-GPLv3-compatible code. There's no way to require that
everything in the Emacs address space be licensed under the GPL. And
that's a good thing, because equating sharing an address space with a
"derivative work" boundary is ridiculous.

Even if Emacs were to look for some I_LOAD_ONLY_GPLV3=1 export from
loaded modules, the GPL does not prohibit a module simply lying. The
GPLv3 does not require the implementation of technical license
enforcement mechanisms. (If it did, it would violate freedom 0.)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 09/15/2015 01:45 AM, David Kastrup wrote:
>> So having our binary interfaces and calling conventions and
>> memory/exception handling default to "like Emacs does" is not just
>> Lisp-friendly but is also keeping our license enforcement options more
>> conservative. 
>
> I don't see how.

Maybe you should have read more than a single paragraph then.

-- 
David Kastrup

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On 09/15/2015 06:33 AM, David Kastrup wrote:
> Daniel Colascione <dancol@dancol.org> writes:
> 
>> On 09/15/2015 01:45 AM, David Kastrup wrote:
>>> So having our binary interfaces and calling conventions and
>>> memory/exception handling default to "like Emacs does" is not just
>>> Lisp-friendly but is also keeping our license enforcement options more
>>> conservative. 
>>
>> I don't see how.
> 
> Maybe you should have read more than a single paragraph then.

Maybe you shouldn't snip the part where I explain how the rest of your
argument is incorrect.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 09/15/2015 06:33 AM, David Kastrup wrote:
>> Daniel Colascione <dancol@dancol.org> writes:
>> 
>>> On 09/15/2015 01:45 AM, David Kastrup wrote:
>>>> So having our binary interfaces and calling conventions and
>>>> memory/exception handling default to "like Emacs does" is not just
>>>> Lisp-friendly but is also keeping our license enforcement options more
>>>> conservative. 
>>>
>>> I don't see how.
>> 
>> Maybe you should have read more than a single paragraph then.
>
> Maybe you shouldn't snip the part where I explain how the rest of your
> argument is incorrect.

Your "explanation" did not fit the "rest of my argument" and discussed a
different scenario.

-- 
David Kastrup

Re: Dynamic loading progress

[Stephen J. Turnbull]

Daniel Colascione writes:

 > There's no way to require that everything in the Emacs address
 > space be licensed under the GPL.

That is true (for example, the data segment is not covered by Emacs's
GPL unless it contains Emacs code), but it's a useful guide to
reasoning about code.

 > And that's a good thing, because equating sharing an address space
 > with a "derivative work" boundary is ridiculous.

I'm pretty sure Richard doesn't consider it "ridiculous", though I
expect he'd prefer a phrase like "as a first approximation" to "equate".

 > Even if Emacs were to look for some I_LOAD_ONLY_GPLV3=1 export from
 > loaded modules, the GPL does not prohibit a module simply lying.

The GPL doesn't prohibit any behavior of software.  However, it
arguably would prohibit distribution of a module that lies in that
way.  Not because of the falsehood of the lie, but because from the
fact of the lie I deduce the module actually violates the terms of the
GPL by creating a derivative work, not all of which is licensed under
the GPL.  (I understand you disagree with that argument in principle.
I'm just saying that it can be argued, and I expect it *would* be
argued by the FSF, all the way to court if necessary.)

Re: Dynamic loading progress

[Michael Albinus]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>  > Calls back to Emacs functions will be fairly common, because
>  > modules use these functions to manipulate Emacs data on behalf of
>  > their own callers.
>
> Real examples, please.  S?XEmacs has two decades of experience with
> wrapping random libraries, Emacs has some too, and there just aren't
> any that I can think of offhand.  I see no reason why modules in Emacs
> will change that pattern; S?XEmacs has a 1.5 decades of experience
> with require'able modules, and it hasn't changed there.

For the records, years ago I had ported Emacs' dbusbind.c to SXEmacs,
using their ffi interface. It was almost running; I forgot why it wasn't
finished and included in SXEmacs. A corner-case problem with events maybe.

Once Emacs' dynamic modules are usable, I plan to write a scuccessor of
dbusbind.c for kdbus as dynamic module.

The D-Bus machinery is based on (Lisp level) callbacks.

Best regards, Michael.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1707 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Mo., 14. Sep. 2015 um
03:58 Uhr:

> >>> It's not possible to skip frames in module code using longjmp, so
> >> Why not?
> > Because most C code isn't expecting to be unwound.
> > Forcing non-local flow control on module code is completely
> > unacceptable.  Emacs needs to return with an error indication set.
>
> Hmm... I'm still not sure what's the issue.
>
> AFAICT, the only odd case I can think of is the following:
>
>    1- Emacs calls some module code via the new API.
>    2- This code (which is Emacs-specific) will know about Fthrow.
>

Even if it knows about non-local jumps, in many cases it can't do anything
about them except wrapping them. For example, longjmp is illegal in most
C++ programs (and probably in most other languages as well).


>    3- It may call some non-Emacs-specific code from some other library.

   4- This non-Emacs-specific code calls back to some Emacs-specific
> function.
>

Step 3 and 4 can be left out. The issue only requires that the plugin
contains some code that is not suited for longjmp; e.g., all C++ code, no
matter whether written specifically for Emacs or not.


>    5- This Emacs-specific function calls Fthrow/Fsignal.
>
> Where the problem only shows if/when we reach point 5.
>
> This problem can be handled between 4 and 5 by using an appropriate
> internal_condition_case.
>

Partially yes. internal_condition_case with a Qt handler can indeed catch
all signals, but this invocation is only possible from Emacs C code (using
internal_condition_case), and the fact that signal and throw use longjmp is
an implementation detail anyway, so the catch-all code needs to be inside
the module code.

[-- Attachment #2: Type: text/html, Size: 2510 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

>> 1- Emacs calls some module code via the new API.
>> 2- This code (which is Emacs-specific) will know about Fthrow.
> Even if it knows about non-local jumps, in many cases it can't do anything
> about them except wrapping them.

There's nothing else none can ever do about them other than
wrapping them.  So the fact that they can only wrap them is not
a downside of some alternative.

> For example, longjmp is illegal in most C++ programs (and probably in
> most other languages as well).

Most Emacs modules will be written in C, methinks.  For those written in
other languages, since the API is expressed in C, they'll have to
accommodate the interaction with this C API, yes.

If they want to add a whole layer that wraps every function call in an
internal_condition_case, that's fine by me.

> The issue only requires that the plugin contains some code that is not
> suited for longjmp; e.g., all C++ code, no matter whether written
> specifically for Emacs or not.

That's a problem for the C++ code, not for Emacs's C-level module API.

> Partially yes. internal_condition_case with a Qt handler can indeed catch
> all signals, but this invocation is only possible from Emacs C code (using
> internal_condition_case),

Clearly, we'll want to export it to the modules as well.


        Stefan

Re: Dynamic loading progress

[Stephen J. Turnbull]

Daniel Colascione writes:
 > On 09/13/2015 01:31 PM, Stefan Monnier wrote:

 > >> It's not possible to skip frames in module code using longjmp, so
 > > 
 > > Why not?
 > 
 > Because most C code isn't expecting to be unwound. Forcing non-local
 > flow control on module code is completely unacceptable.

"Completely unacceptable" is nonsense.  Module code needs to be
written the same way any other Emacs code is written, using various
unwind-protect constructs.  If the module wraps external code that
has state that persists past the unwinding, you also need to use the
underlying code's own "unwind protection".

For example, XEmacs has had a PostgreSQL interface module based on
libpq for about 15 years, and nobody has ever reported any trouble of
that kind.  The Lisp API provides transactions, and the unwind code in
XEmacs knows to rollback if there's a pending transaction on nonlocal
exit.  Sure, all of XEmacs, PostgreSQL, and the module have had bugs
in them -- but nobody has blamed an XEmacs crash or database
corruption on that module yet.  OK, there may not be all that much
experience, but at least two people used it heavily in mission
critical roles for 5 years (they were supporting a Fortune 500
company's trouble ticket database).  I acknowledge that the risk is
probably somewhat greater than if the code were built-in to XEmacs.

Sure, it's likely that module programmers will not be as well-versed
in Emacs unwinding as Emacs core developers are, but by that same
token they're probably quite well-versed in the target code's
features.  Overall, modules are likely to be less stable than core
Emacs, but presumably that's a tradeoff that module users are willing
to accept.

And I'm sure that there's non-robust code in libraries out there that
will be totally horked on a non-local exit.  But AFAICS you have the
same problem there if you link statically or at Emacs initialization
-- that code just shouldn't be linked into Emacs period.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

On 09/13/2015 08:43 PM, Stephen J. Turnbull wrote:
> Daniel Colascione writes:
>  > On 09/13/2015 01:31 PM, Stefan Monnier wrote:
> 
>  > >> It's not possible to skip frames in module code using longjmp, so
>  > > 
>  > > Why not?
>  > 
>  > Because most C code isn't expecting to be unwound. Forcing non-local
>  > flow control on module code is completely unacceptable.
> 
> "Completely unacceptable" is nonsense.  Module code needs to be
> written the same way any other Emacs code is written, using various
> unwind-protect constructs.

Why? There is no positive reason to make Emacs module code resemble
Emacs core code. In fact, there's a strong reason to make Emacs module
code more conventional, because it'll improve the accessibility of the
API, which in turn will increase the quantity and quality of Emacs modules.

The Emacs core code has a lot of baggage. It's not practical to clean it
up today, but the module system is a greenfield project. There is no
reason to burden it the same way.

> If the module wraps external code that
> has state that persists past the unwinding, you also need to use the
> underlying code's own "unwind protection".

We shouldn't break this perfectly reasonable code:

void
foo()
{
  std::string x = something();
  emacs_bar(x.c_str());
}

longjmp()ing out of emacs_bar prevents any stack unwinding happening,
which means that any code that requires stack unwinding for correctness
needs to swap every Emacs API call in the equivalent of condition-case.
It's cumbersome, especially for any binding that isn't pure C.

Can you name an extension system for any other program that longjmps out
of its public API functions?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> Why? There is no positive reason to make Emacs module code resemble
> Emacs core code.

Of course, there is.


        Stefan

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Mo., 14. Sep. 2015 um
06:24 Uhr:

> > Why? There is no positive reason to make Emacs module code resemble
> > Emacs core code.
>
> Of course, there is.
>
>
>
Which one? Given the overall state of the Emacs core code (lots of global
mutable state, thread-hostile, lots of macros...) I'd rather want modules
to be as different from the Emacs core as possible.
The module interface is an interface. No good interface prescribes how its
implementations have to look like.

[-- Attachment #2: Type: text/html, Size: 838 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> Which one? Given the overall state of the Emacs core code (lots of global
> mutable state, thread-hostile, lots of macros...) I'd rather want modules
> to be as different from the Emacs core as possible.
> The module interface is an interface. No good interface prescribes how its
> implementations have to look like.

The interface just says "funcall may exit non-locally".  It doesn't have
to say that it uses longjmp for that.  But the end result is the same:
if you want to write your module in a language other than C, you'll have
to deal with interactions between C (including longjmp) and that other
language.


        Stefan

Re: Dynamic loading progress

[Stephen J. Turnbull]

Daniel Colascione writes:
 > On 09/13/2015 08:43 PM, Stephen J. Turnbull wrote:
 > > Daniel Colascione writes:
 > >  > On 09/13/2015 01:31 PM, Stefan Monnier wrote:
 > > 
 > >  > >> It's not possible to skip frames in module code using longjmp, so
 > >  > > 
 > >  > > Why not?
 > >  > 
 > >  > Because most C code isn't expecting to be unwound. Forcing non-local
 > >  > flow control on module code is completely unacceptable.
 > > 
 > > "Completely unacceptable" is nonsense.  Module code needs to be
 > > written the same way any other Emacs code is written, using various
 > > unwind-protect constructs.
 > 
 > Why? There is no positive reason to make Emacs module code resemble
 > Emacs core code.

Er, because it *is* Emacs code?  If a function in a module simply
wraps a library call, then you need to use that library's facilities
to handle problems.  But if it manipulates any Emacs data using Emacs
APIs, you'll use Emacs facilities to handle all that ugly stuff like
GC and Fsignal.

 > In fact, there's a strong reason to make Emacs module code more
 > conventional, because it'll improve the accessibility of the API,
 > which in turn will increase the quantity and quality of Emacs
 > modules.

I think that's wishful thinking.  In my experience with the XEmacs
module code, what basically happens most of the time is that you write
some Lisp to beat the data into the right structure, then you write a
little bit of C to marshal Lisp primitive types into the underlying
library's format, then you call the library API, and marshal the
return into Lisp data types.  Often it's more convenient to call the
Lisp APIs from C than to return to Lisp and do it there, but it's
equivalent.  Sure, there's a lot of boring boilerplate, but it ain't
rocket science by that very token.

In the relatively rare case that you have a way of passing a callback
into Lisp to the library, you either program very defensively in the
Lisp, or just wrap the whole thing in a condition-case (such as
ignore-errors) before passing it to the library.

 > The Emacs core code has a lot of baggage. It's not practical to
 > clean it up today, but the module system is a greenfield
 > project. There is no reason to burden it the same way.

AFAICS, no, it's not a greenfield project.  It's a second story on the
same foundations.  You're welcome to prove otherwise, but I don't have
a clue how you propose to do it (without disallowing callbacks into
Lisp entirely, and I don't see how you can enforce that without some
very pervasive nannying code).

 > Can you name an extension system for any other program that
 > longjmps out of its public API functions?

No, not with any confidence.  XEmacs is the only program whose
internal implementation of nonlocal exit I know well.  But I would
guess that if the language implements nonlocal exits, then any
callback that can be an arbitrary function in the language can
longjmp.

Re: Dynamic loading progress

[Stephen Leake]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Daniel Colascione writes:
>  > On 09/13/2015 08:43 PM, Stephen J. Turnbull wrote:
>  > > Daniel Colascione writes:
>  > >  > On 09/13/2015 01:31 PM, Stefan Monnier wrote:
>  > > 
>  > >  > >> It's not possible to skip frames in module code using longjmp, so
>  > >  > > 
>  > >  > > Why not?
>  > >  > 
>  > >  > Because most C code isn't expecting to be unwound. Forcing non-local
>  > >  > flow control on module code is completely unacceptable.
>  > > 
>  > > "Completely unacceptable" is nonsense.  Module code needs to be
>  > > written the same way any other Emacs code is written, using various
>  > > unwind-protect constructs.
>  > 
>  > Why? There is no positive reason to make Emacs module code resemble
>  > Emacs core code.
>
> Er, because it *is* Emacs code?  

That was my initial view as well, because I new I needed fast
performance for my particular use case.

But I suspect some (most?) modules will be added just to provide a lisp
interface to some external library; the authors will see that as part of
their lisp code, not part of Emacs core.

One reason Emacs succeeds is because lisp is so forgiving; module
developers will expect the Emacs module API to be similarly forgiving. I
don't think we need to accept that as a hard requirement, but anything
we can do towards that goal that has reasonable performance penalties
could be a win. 

>  > In fact, there's a strong reason to make Emacs module code more
>  > conventional, because it'll improve the accessibility of the API,
>  > which in turn will increase the quantity and quality of Emacs
>  > modules.
>
> I think that's wishful thinking.  In my experience with the XEmacs
> module code, what basically happens most of the time is that you write
> some Lisp to beat the data into the right structure, then you write a
> little bit of C to marshal Lisp primitive types into the underlying
> library's format, then you call the library API, and marshal the
> return into Lisp data types.  Often it's more convenient to call the
> Lisp APIs from C than to return to Lisp and do it there, but it's
> equivalent.  Sure, there's a lot of boring boilerplate, but it ain't
> rocket science by that very token.

I see that as "conventional code"; you certainly didn't mention
callbacks and longjmp here.

> In the relatively rare case that you have a way of passing a callback
> into Lisp to the library, you either program very defensively in the
> Lisp, 

Always a good idea anyway, at least until thorough testing is done.

> or just wrap the whole thing in a condition-case (such as
> ignore-errors) before passing it to the library.

What is the downside of having the Emacs module API do that for you?

-- 
-- Stephe

Re: Dynamic loading progress

[Stephen J. Turnbull]

Stephen Leake writes:

 > > Er, because it *is* Emacs code?  
 > 
 > That was my initial view as well, because I new I needed fast
 > performance for my particular use case.
 > 
 > But I suspect some (most?) modules will be added just to provide a lisp
 > interface to some external library; the authors will see that as part of
 > their lisp code, not part of Emacs core.

Which is what I described and you called "conventional".  But that's
*not* what Daniel is exercised about.

 > One reason Emacs succeeds is because lisp is so forgiving; module
 > developers will expect the Emacs module API to be similarly
 > forgiving.

Yeah, and they expect a pony too.  The module API will be C, which is
not a forgiving language.  Lisp forgiveness is possible because it's
backed up by the use of longjmp, you know.

 > > In the relatively rare case that you have a way of passing a callback
 > > into Lisp to the library, you either program very defensively in the
 > > Lisp, 
 > 
 > Always a good idea anyway, at least until thorough testing is done.
 > 
 > > or just wrap the whole thing in a condition-case (such as
 > > ignore-errors) before passing it to the library.
 > 
 > What is the downside of having the Emacs module API do that for you?

Well, the very existence of `ignore-errors' is an abomination, you can
hardly expect me to condone its automatic use. :-)  Suppressing errors
screws exactly the non-Emacsy developers it's supposed to help -- the
callback code is nearly impossible to debug without the Emacs
expertise that we're assuming the module developer *doesn't have*.

Also, the expense of an unnecessary condition-case in a tight loop is
not to be ignored, when the Lisp-level "no brain required" safety belt
for code that might signal but isn't performance-critical is so simple
and stylized.

But in most modules, we're just using library code to do a
calculation, and then we can pass the result (including signaling an
error status) back out to Lisp.  XEmacs (and even more so SXEmacs)
have been doing this a lot over the years.  The necessary code is
pretty boring, and I've never seen one that involves the nonlocal exit
issue.  The Lisp APIs that are most frequently used are hard to reduce
(DEFUN, DEFVAR).  GCPROs (S?XEmacs still uses them) are very rare, and
even those can generally be avoided by sticking strictly to the
marshalling/unmarshalling in C, and doing Lisp manipulations in Lisp.

The problem that Daniel is worried about is real, but I don't see a
good way of solving it in Emacs.

In any case, it doesn't matter to encouraging use of modules.  The
vast majority of modules simply won't have callbacks to deal with.
People who *want* to write a callback in Lisp and pass it to C are
going to be rare, and likely to be experienced Emacs developers, I
would think.  If not, they're surely multilingual.  They probably can
deal with it.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Stephen J. Turnbull <turnbull@sk.tsukuba.ac.jp> schrieb am Mo., 14. Sep.
2015 um 05:43 Uhr:

> Daniel Colascione writes:
>  > On 09/13/2015 01:31 PM, Stefan Monnier wrote:
>
>  > >> It's not possible to skip frames in module code using longjmp, so
>  > >
>  > > Why not?
>  >
>  > Because most C code isn't expecting to be unwound. Forcing non-local
>  > flow control on module code is completely unacceptable.
>
> "Completely unacceptable" is nonsense.  Module code needs to be
> written the same way any other Emacs code is written,


Why? The module interface defines an interface, not an implementation
strategy. The observable behavior of Emacs is described in the Emacs
manual, and that contains nothing about longjmp (unsurprisingly, given that
it's an implementation detail).


> using various
> unwind-protect constructs.


It is not possible to write such constructs in general. Stack frames that
don't originate from C programs simply can't be part of a non-local jump.
unwind-protect doesn't help at all because it doesn't stop non-local jumps.


>
>

> And I'm sure that there's non-robust code in libraries out there that
> will be totally horked on a non-local exit.  But AFAICS you have the
> same problem there if you link statically or at Emacs initialization
> -- that code just shouldn't be linked into Emacs period.
>
>
>
Such a decision would massively reduce the usefulness of modules. It would
prevent dynamic modules in any language other than C, and even for C
modules implementors would need to be careful not to write longjmp-hosting
code. This is indeed unacceptable.

[-- Attachment #2: Type: text/html, Size: 2441 bytes --]

Re: Dynamic loading progress

[Stephen J. Turnbull]

Philipp Stephani writes:

 > > using various unwind-protect constructs.
 > 
 > It is not possible to write such constructs in general. Stack
 > frames that don't originate from C programs simply can't be part of
 > a non-local jump.  unwind-protect doesn't help at all because it
 > doesn't stop non-local jumps.

Er, what does "stop non-local jumps"?  What Emacs can do is catch its
own non-local jumps so as not to orphan non-Emacs stack frames.  It
can't catch (or prevent!) anybody else's.

 > > And I'm sure that there's non-robust code in libraries out there
 > > that will be totally horked on a non-local exit.  But AFAICS you
 > > have the same problem there if you link statically or at Emacs
 > > initialization -- that code just shouldn't be linked into Emacs
 > > period.

 > Such a decision would massively reduce the usefulness of
 > modules.

I don't see how.  Please explain which use cases you have in mind.

 > It would prevent dynamic modules in any language other
 > than C,

Why?  There's a long history of embedding other languages in Emacs
(CCL, perlmacs, pymacs).  Runtime optional loading aka dynamic modules
involves problems new to GNU Emacs, but the longjmp problem is not
new.  It just wasn't a problem in practice because those called
languages don't provide a callback interface (as implemented in those
projects).  They were used purely as subroutines to Emacs: call in,
get result or error, forget until next time.  No longjmps to Emacs out
of the embedded interpreter.

 > and even for C modules implementors would need to be careful not to
 > write longjmp-hosting code.  This is indeed unacceptable.

Heh.  The GPL and RMS's historic no-dynamic-modules policy are each
far more restrictive of code you might want to link with Emacs, yet
you accept them.

Politics aside, I wish you luck (and backups, early and often) when
using code that longjmps past Emacs stack frames, such as any embedded
language that implements its own exception handling.  You'll need it.

But that's not what is at issue here.  What is at issue is how to
handle the case where the module calls into foreign code and provides
a callback that calls Lisp, and thus might do

  [main Lisp] -> [module] -> [callback Lisp] --Fsignal--> [main Lisp]

leaving the module in an inconsistent state because its stack has been
blown away.  As far as I can see, this is a very rare case, limited to
(eg) embedding another full-blown interpreter in Emacs.  Modules like
libz, libcanna, and libpq (in XEmacs) simply don't provide callbacks,
so this scenario is impossible.  OTOH, somebody who really wants to
provide more access to Python internals than `comint' does is not
going to need or want Fcondition_case_for_dummies.

The way I propose to handle the issue is by providing a sample wrapper
based on condition-case for Lisp callbacks that handles longjmps out
of Fsignal (which I hope is the only way to longjmp in Emacs).  Thus:

                                    +-----------Fsignal-----------+
                                    |                             |
                                    V                             |
  [main Lisp] -> [module] -> [condition-case] -> [callback Lisp] -+
       A           |  A            | A                            |
       |           |  |            | |                            |
       +--return---+  +------------+ +-------normal return--------+

protecting the module.

I don't see what else can possibly work, since if you set a condition
case in the API calling into the module (ie, in [main Lisp]), you've
already hosed the called code by the time you handle it.  I don't see
how a *generic* (and mandatory, per Daniel's proposal) wrapper can be
provided, since the callback API will be defined by the called code,
not by Emacs.

The sample wrapper is likely to be quite generic, however, as many
typical callbacks are going to reduce to Fapply(lisp_code, lisp_data).
But it can't be fully generic.  Some callback APIs are going to supply
module-defined data structures and perhaps a different number of
arguments, and in those cases you'll need to write your own callback
wrapper that marshalls the arguments into Lisp.  Finally, you may
anticipate that certain kinds of errors will be common in the Lisp
code called, and you would like to handle those in your own way.  A
generic wrapper provided by the module API is either going to provide
only a generic "you are hosed" error return, or it's going to have to
reimplement condition-case.  Why not just use Fcondition_case itself?

And, of course, some users may prefer to write their own
condition-case in Lisp.

Bottom line on a mandatory API enforcing a condition-case: Mostly
YAGNI, most of the rest YCUIWYNI[1].

Footnotes: 
[1]  You Can't Use It Where You Need It.

Re: Dynamic loading progress

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Philipp Stephani writes:
>
>  > > using various unwind-protect constructs.
>  > 
>  > It is not possible to write such constructs in general. Stack
>  > frames that don't originate from C programs simply can't be part of
>  > a non-local jump.  unwind-protect doesn't help at all because it
>  > doesn't stop non-local jumps.
>
> Er, what does "stop non-local jumps"?  What Emacs can do is catch its
> own non-local jumps so as not to orphan non-Emacs stack frames.  It
> can't catch (or prevent!) anybody else's.

Oh, it could provide its own versions of setjmp/longjmp overriding the C
library ones.  That would actually put the hooks into quite a few
non-local jump implementations.  The downside being that Emacs would
need to provide binary-compatible setjmp/longjmp implementations for all
platforms it supports.  Or some trickery for getting at the library ones
without others seeing them.  And nobody else trying similar trickery.

So while it stinks like trouble all over, "can't" is too strong a
qualifier.

-- 
David Kastrup

Re: Dynamic loading progress

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:

 > > Er, what does "stop non-local jumps"?  What Emacs can do is catch its
 > > own non-local jumps so as not to orphan non-Emacs stack frames.  It
 > > can't catch (or prevent!) anybody else's.

[Dubious :-) proof-of-concept which likely fails in some important
case or other omitted.]

 > So while it stinks like trouble all over, "can't" is too strong a
 > qualifier.

What was that you were saying to Paul about arguing the wrong side of
your own point?  As usual, we are in violent agreement.

Re: Dynamic loading progress

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>  > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>
>  > > Er, what does "stop non-local jumps"?  What Emacs can do is catch its
>  > > own non-local jumps so as not to orphan non-Emacs stack frames.  It
>  > > can't catch (or prevent!) anybody else's.
>
> [Dubious :-) proof-of-concept which likely fails in some important
> case or other omitted.]
>
>  > So while it stinks like trouble all over, "can't" is too strong a
>  > qualifier.
>
> What was that you were saying to Paul about arguing the wrong side of
> your own point?  As usual, we are in violent agreement.

Not really I think.  Conservative garbage collection stinks like trouble
all over as well and Emacs does it.  It's sort of like the Sherlock
Holmes principle.  Once you've ruled out all the possibilities that
don't stink, it has to be one of the stinky ones.  So the decisive
factor is not whether a solution stinks or not, but whether there is
already one that stinks less and will work.

I don't see we are there yet.

-- 
David Kastrup

Re: Dynamic loading progress

[Davis Herring]

> I don't see what else can possibly work, since if you set a condition
> case in the API calling into the module (ie, in [main Lisp]), you've
> already hosed the called code by the time you handle it.

What if we avoid the longjmp call rather than catching it?  Define a
(thread-local) signal-raising hook which is also a specpdl barrier.

A C++ client, for example, could write (perhaps with some added 'extern
"C"')

namespace {
  struct emacs_exception : std::exception {
    emacs_exception(emacs_value e,emacs_value d) : error(e),data(d) {}
    emacs_value error,data;
  };

  void die(emacs_value error,emacs_value data) {
    throw emacs_exception(error,data);
  }

  emacs_value my_subr(emacs_env *env,int nargs,emacs_value args[]) {
    env->error_handle(env,die);  // creates barrier
    try {
      std::vector</*...*/> data; // to be cleaned up
      env->funcall(env,/*...*/);
    } catch(const emacs_exception &e) {
      // various variables assumed to be already obtained elsewhere
      if(eq(e.error,Qfile_error)) {
        /* handle file-not-found */;
        return Qnil;
      } else env->error_signal(env,e.error,e.data);
    }
    return Qt;
  }
}

Then Fsignal/Fthrow can do its cleanup down to the barrier, then call
the hook.  The exception unwinds the stack back into the C++ module,
does appropriate destructor things, and then is caught before the module
returns to Emacs.  There, the module can choose to return normally (thus
acting like condition-case) or propagate the error (thus having its
destructors act like unwind-protect).

Here we have the overhead of error_handle only once per call from Emacs
into the module rather than on each call from the module into Emacs
(probably larger).  We might be able to avoid even that by putting the
handler into the emacs_env, but that would lose the flexible barrier
placement and presumably Fsignal still has to consult a how-to-signal
variable that would have to be set at least once per Emacs->module call.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Dynamic loading progress

[Stephen J. Turnbull]

Davis Herring writes:

 > What if we avoid the longjmp call rather than catching it?  Define a
 > (thread-local) signal-raising hook which is also a specpdl barrier.

We still have to catch the longjmp.  The setjmp *is* the barrier as
far as Fsignal and Fthrow are concerned.  See unwind_to_catch().

Unless you're proposing to replace unwind_to_catch itself with
something that signals "locally" for the duration of the callback.
That seems like a very dubious idea to me.  Rather, as Daniel has been
at pains to say, it's a shame Emacs didn't do that in the first place,
as more recently designed languages have managed to do.[1]  So if
we're going to provide a "no nonlocal exit" unwind_to_catch() for
modules, let's use it for Emacs itself, too!

 > A C++ client, for example, could write

AIUI, that's exactly what Daniel and Philipp *don't* want!  They want
to avoid having the client contain error handling code for Lisp at
all, instead managing it behind the API in the Emacs module
implementation, and thus making it idiot-proof.  Perhaps with some
concessions for setting handlers to the DIYers, but there would alway
be a safety net.

I think this is an illusion, and a dangerous one, for code that is
complicated enough to provide a callback interface that accepts Lisp
functions.  *Especially* if written in crash-prone languages like C
and C++.

 > We might be able to avoid even that by putting the handler into the
 > emacs_env,

Which is what Daniel and Philipp want, and which is what I think is a
horrible idea, akin to using `ignore-errors' in basic Lisp functions
like `car'.


Footnotes: 
[1]  Amusingly enough, even those languages don't always manage to
avoid longjmp entirely: Python 3.5 still has two instances of longjmp,
one in Modules/fpectlmodule.c, and the other in Modules/readline.c.

RE: Dynamic loading progress

[Herring, Davis]

> We still have to catch the longjmp.  The setjmp *is* the barrier as
> far as Fsignal and Fthrow are concerned.  See unwind_to_catch().

The "struct handler" is the barrier: it's a detail of the current implementation that it contains a jmp_buf (that is always used).

> Unless you're proposing to replace unwind_to_catch itself with
> something that signals "locally" for the duration of the callback.

I think by "locally" you mean "by normal function returns only".  That's not my idea, which is basically to (conditionally) replace
  sys_longjmp (catch->jmp, 1);
with
  catch->hook ();

where (according to my straw-man code) catch->hook was populated by the call to env->error_handle.  It's still a non-local transfer, as is necessary for the existing calls to unwind_to_catch to possibly work.

> AIUI, that's exactly what Daniel and Philipp *don't* want!  They want
> to avoid having the client contain error handling code for Lisp at
> all, instead managing it behind the API in the Emacs module
> implementation, and thus making it idiot-proof.

I'm aiming more for idiot-resistant, without the complexity (and overhead) of wrapping every call into Emacs to protect the caller.  In other words, it's a way to recover control (like they want) without significantly extending the "unsafe" interface (which Stefan wants).

> Which is what Daniel and Philipp want, and which is what I think is a
> horrible idea, akin to using `ignore-errors' in basic Lisp functions
> like `car'.

But it's not ignore-errors: in my example, the error does propagate, but as a C++ exception rather than a longjmp.  Some easy RAII would let you reliably get the error_handle and the matching error_unhandle in one shot (avoiding the possibility of forgetting it, as I did in my example!).  In the trivial case that you just want destructors to be called and the error to propagate back into Lisp, you could avoid an explicit try-catch on each module entry point with a wrapper function template.

Davis

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2767 bytes --]

On 09/29/2015 11:56 PM, Herring, Davis wrote:
>> We still have to catch the longjmp.  The setjmp *is* the barrier as
>> far as Fsignal and Fthrow are concerned.  See unwind_to_catch().
> 
> The "struct handler" is the barrier: it's a detail of the current implementation that it contains a jmp_buf (that is always used).
> 
>> Unless you're proposing to replace unwind_to_catch itself with
>> something that signals "locally" for the duration of the callback.
> 
> I think by "locally" you mean "by normal function returns only".  That's not my idea, which is basically to (conditionally) replace
>   sys_longjmp (catch->jmp, 1);
> with
>   catch->hook ();
> 
> where (according to my straw-man code) catch->hook was populated by the call to env->error_handle.  It's still a non-local transfer, as is necessary for the existing calls to unwind_to_catch to possibly work.
> 
>> AIUI, that's exactly what Daniel and Philipp *don't* want!  They want
>> to avoid having the client contain error handling code for Lisp at
>> all, instead managing it behind the API in the Emacs module
>> implementation, and thus making it idiot-proof.
> 
> I'm aiming more for idiot-resistant, without the complexity (and overhead) of wrapping every call into Emacs to protect the caller.  In other words, it's a way to recover control (like they want) without significantly extending the "unsafe" interface (which Stefan wants).
> 
>> Which is what Daniel and Philipp want, and which is what I think is a
>> horrible idea, akin to using `ignore-errors' in basic Lisp functions
>> like `car'.
> 
> But it's not ignore-errors: in my example, the error does propagate,
> but as a C++ exception rather than a longjmp.  Some easy RAII would
> let you reliably get the error_handle and the matching error_unhandle
> in one shot (avoiding the possibility of forgetting it, as I did in
> my example!).  In the trivial case that you just want destructors to
> be called and the error to propagate back into Lisp, you could avoid
> an explicit try-catch on each module entry point with a wrapper
> function template.

One practical difficulty of this scheme is that we would have to build
Emacs itself with exception unwind information. (Since the default
Itanium ABI behavior is to abort if we try to unwind through a stack
frame for which we don't have unwind information.) I don't think that
would be easy to pull off in the general case, and I don't think it's
possible in advance to predict what information we might need to allow
stack unwinding to proceed. This scheme leads us further into
ABI-dependence. I'd prefer longjmp to a system that required us to let
C++ exceptions (with what libstdc++?) propagate through arbitrary Emacs
frames.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

BTW, all this discussion about C++ modules seems to miss the point:
Emacs's own API will be a C API.  When you want to use it from a C++
module, you'll have to convert it to a C++ API.  At that point, you
have various options:

- do a minimalist binding which keeps using naked longjmp, and just
  let the C++ code use explicit internal_condition_case wrappers where
  it thinks it's needed.
- do an "idiot-proof" binding which only provides a "safe_funcall" which
  catches longjmps and turns them into local exits.
- do an "idiot-resistant" binding which only provides a "safe_funcall" that
  catches longjmps and turns them into C++ exceptions, and then back
  into Elisp longjmps (and back-and-forth ad nauseum if needed when
  doing nested mutual callbacks).
- ...younameit...

I'm fine with any of those things, and AFAIK they can all be done on top
of the plain C API with the usual non-local exits.
All I care about at this point is the plain C API.


        Stefan

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2439 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Mi., 30. Sep. 2015 um
10:52 Uhr:

> BTW, all this discussion about C++ modules seems to miss the point:
> Emacs's own API will be a C API.  When you want to use it from a C++
> module, you'll have to convert it to a C++ API.


True, and I think nobody expects otherwise. However, as with other
interfaces, the Emacs module interface should use not all of C, but the
intersection of C and C++ (which is pretty large compared to full C).


>   At that point, you
> have various options:
>
> - do a minimalist binding which keeps using naked longjmp, and just
>   let the C++ code use explicit internal_condition_case wrappers where
>   it thinks it's needed.
>

This is currently impossible because the necessary code is not available to
module authors (and partially does not even exist, as in the case of catch).


> - do an "idiot-proof" binding which only provides a "safe_funcall" which
>   catches longjmps and turns them into local exits.
>

This is what Daniel and I prefer, however I'd like to avoid the negative
spin of the word "idiot" here. To use the first option, a module author has
to know all of the following:
- that Emacs uses longjmp for non-local exits in Lisp (currently an
implementation detail_;
- that longjmp is in general unsafe in C++ code;
- which subset exactly of C++ is safe for longjmp;
- how to correctly set up the necessary traps to reliably catch all
longjmps.
I think it's a stretch to call everyone who doesn't know all of these
things an "idiot". Implementing this option is relatively straightforward;
implementing the first option makes it much harder for module authors, with
negligible benefit to the Emacs core.


> - do an "idiot-resistant" binding which only provides a "safe_funcall" that
>   catches longjmps and turns them into C++ exceptions, and then back
>   into Elisp longjmps (and back-and-forth ad nauseum if needed when
>   doing nested mutual callbacks).
>

No, this is what a C++ binding to the module interface would have to
provide.


> I'm fine with any of those things, and AFAIK they can all be done on top
> of the plain C API with the usual non-local exits.
> All I care about at this point is the plain C API.
>
>
Daniel has designed a great API in
https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00960.html. Let's
just implement this API as designed and check later if there's any need for
the other options.

[-- Attachment #2: Type: text/html, Size: 3598 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> True, and I think nobody expects otherwise. However, as with other
> interfaces, the Emacs module interface should use not all of C, but the
> intersection of C and C++ (which is pretty large compared to full C).

That would be desirable, but less so than the fact that the API just
exposes the functionality provided by Emacs's core code.
So as long as Emacs's signaling facility uses longjmp, that's going to
be exposed in the core API.

>> - do a minimalist binding which keeps using naked longjmp, and just
>> let the C++ code use explicit internal_condition_case wrappers where
>> it thinks it's needed.
> This is currently impossible because the necessary code is not available to
> module authors (and partially does not even exist, as in the case of catch).

Irrelevant detail: we're talking about design of some upcoming API,
which will require coding.  Some of that coding may involve writing new
core code, of course.

Of course, by that measure, some of that coding could involve replacing
longjmp with some other mechanism.  My gut feeling is that this would be
a very large undertaking and that the result would probably not be
palatable to the vast majority of Emacs's C contributors, so I won't
hold my breath.

> Daniel has designed a great API in
> https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00960.html.  Let's
> just implement this API as designed and check later if there's any need for
> the other options.

If you want:

    Modules can call Lisp functions using `funcall', which does the
    obvious thing.  If Lisp signals or throws, `funcall' returns NULL.

that's OK, but it should then be called `safe_funcall', and we should
provide another `funcall' which does not catch the error.

And if `safe_funcall' cannot be defined "by hand" in a module using
other API methods, that would mean that the API is still lacking
functionality (e.e. lacking condition_case and things like that).

You're not going to convince me otherwise.


        Stefan

RE: Dynamic loading progress

[Stephen J. Turnbull]

Herring, Davis writes:

 > > We still have to catch the longjmp.  The setjmp *is* the barrier
 > > as far as Fsignal and Fthrow are concerned.  See
 > > unwind_to_catch().
 > 
 > The "struct handler" is the barrier: it's a detail of the current
 > implementation that it contains a jmp_buf (that is always used).

 > where (according to my straw-man code) catch->hook was populated by
 > the call to env->error_handle.  It's still a non-local transfer, as
 > is necessary for the existing calls to unwind_to_catch to possibly
 > work.

So what's the advantage over unwind_to_catch as implemented?  You
still need to put the barrier in place, only now you're using mixed
idioms (Lisp for the pitcher, C++ for the catcher).  That kind of
thing always gives me hives.

 > > Which [...] is what I think is a horrible idea, akin to using
 > > `ignore-errors' in basic Lisp functions like `car'.
 > 
 > But it's not ignore-errors: in my example,

A-ah, that's *your* example.  I was referring to what I believe Daniel
and Philipp want, which was the idea you pretty much dismissed as
"[too] inflexible": a handler hidden behind the API.

Re: Dynamic loading progress

[Davis Herring]

> So what's the advantage over unwind_to_catch as implemented?  You
> still need to put the barrier in place, only now you're using mixed
> idioms (Lisp for the pitcher, C++ for the catcher).  That kind of
> thing always gives me hives.

As I think I said, it lets you avoid reinstalling the handler every time
you call into Emacs (and thus having to wrap every Emacs API function).

>  > But it's not ignore-errors: in my example,
> 
> A-ah, that's *your* example.  I was referring to what I believe Daniel
> and Philipp want, which was the idea you pretty much dismissed as
> "[too] inflexible": a handler hidden behind the API.

(I'm still talking about my example here, but only because it serves to
demonstrate what properties a hidden handler might have.  Another
implementation of hidden handlers could of course behave differently,
but I haven't seen any particular implementation firmly chosen.)

Even if the handler is hidden, the signal isn't swallowed (or even
turned into a famously-easy-to-ignore return code): it's turned into an
exception and thrown.  It's up to the module code what to do about it,
as always.  My follow-on template-wrapper idea didn't squelch exceptions
either, but turned them back into propagating Lisp signals after having
run whatever C++ destructors.

The inflexible part of the automatic/hidden handler, as I saw it, is
that every entry point must perform the try/catch even if it has no
destructors to run because Emacs will cause the exception to be thrown
anytime control is within the module.  It also prevents doing any
preparatory work before installing the handler, or making a dynamic
choice of handler -- but for neither of those do I have a use case at
the moment.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Dynamic loading progress

[Aurélien Aptel]

On Wed, Mar 4, 2015 at 11:39 PM, Daniel Colascione <dancol@dancol.org> wrote:
> As I'm imagining the system, no. Access is indirected. That insulates C
> code from Emacs GC requirements and lets us easily support global
> references.

Can you explain how global/local references work in your design?

Re: Dynamic loading progress

[Stefan Monnier]

>>> * fixnum <-> int64_t is the only type conversion supported
>> Not sure I understand: can the C side see "Lisp_Object" (presumably as
>> an opaque type)?
> As I'm imagining the system, no. Access is indirected. That insulates C
> code from Emacs GC requirements and lets us easily support global
> references.

But that forces a completely manual management of Lisp_Object
references, making the interface a lot more painful to use.
I must prefer exposing staticpro and conservative stack scanning.


        Stefan

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1182 bytes --]

On 03/05/2015 09:14 PM, Stefan Monnier wrote:
>>>> * fixnum <-> int64_t is the only type conversion supported
>>> Not sure I understand: can the C side see "Lisp_Object" (presumably as
>>> an opaque type)?
>> As I'm imagining the system, no. Access is indirected. That insulates C
>> code from Emacs GC requirements and lets us easily support global
>> references.
> 
> But that forces a completely manual management of Lisp_Object
> references, making the interface a lot more painful to use.
> I must prefer exposing staticpro and conservative stack scanning.

Reasonable people can differ on the conservative scanning issue (JSC
does it; V8 and Java don't).  If we bake the assumption of conservative
scanning into the interface, we're going to regret it later. In
practice, manually managing references is no trouble at all since most
Lisp->C code is call-response, and we automatically unwind references
when C functions return.

I have to insist on not requiring staticpro, at least in its present
form. Registration of new GC roots (i.e., global references) must be
dynamic. You can't possibly predict the reference models modules might
want to use.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stefan Monnier]

> Reasonable people can differ on the conservative scanning issue (JSC
> does it; V8 and Java don't).  If we bake the assumption of conservative
> scanning into the interface, we're going to regret it later.

Yes, if we define the API as using conservative stack scanning, then
when we change to a GC that doesn't use conservative stack scanning, all
the plugins will need to be changed correspondingly.

Of course, when/if we make that change, we'll also have to make the same
changes to Emacs's own code.  My take on it is that making the changes
to Emacs itself will be a crapload more painful than fixing the few
affected plugins, so I'm really not worried about this eventuality.

> I have to insist on not requiring staticpro, at least in its present
> form.  Registration of new GC roots (i.e., global references) must be
> dynamic.

Of course it should be.  I didn't mean to take the current
implementation of staticpro and expose it as is.


        Stefan

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> I've tried to implement modules using Daniel's plan. 
> The code is available on my github repo on the dynamic-modules-2 branch.
> The interesting parts are the API header, the runtime and the sample module.
>
> git repo: https://github.com/aaptel/emacs-dynamic-module.git

Why do we need both emacs_runtime and emacs_env types? If the function
pointers in emacs_env were in emacs_runtime instead, what would we lose?


It seems every module will need 'bind_function', so perhaps that should
be in emacs_module.h.


I'll work on implementing the ada-mode parser with this, to illuminate
other issues.

-- 
-- Stephe

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

>     $ ./src/emacs -Q
>     (module-load "./modules/basic/basic.so")
>     => t
>
>     (basic-sum 2 2)
>     => 4
>
> git repo: https://github.com/aaptel/emacs-dynamic-module.git

This compiles and runs for me on 64 bit Debian and 32 bit Mingw.

-- 
-- Stephe

Re: Dynamic loading progress

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>
>>     $ ./src/emacs -Q
>>     (module-load "./modules/basic/basic.so")
>>     => t
>>
>>     (basic-sum 2 2)
>>     => 4
>>
>> git repo: https://github.com/aaptel/emacs-dynamic-module.git
>
> This compiles and runs for me on 64 bit Debian and 32 bit Mingw.

However, when I try to access an elisp vector, it fails:

---- test_module_vector.c
#include <emacs_module.h>

int plugin_is_GPL_compatible;

/* functions */
static emacs_value length;


/* standard Emacs module init */
int emacs_module_init (struct emacs_runtime *ert)
{
  emacs_env *env = ert->get_environment (ert);

  length = env->intern (env, "length");

  emacs_value names = env->intern (env, "ada_grammar-names");
  emacs_value args[] = { names };
  int64_t names_length = env->fixnum_to_int (env, env->funcall (env, length, 1, args));

  return 0;
}
---- test_module_vector.c


(setq ada_grammar-names [0 1 2 3 4])
(sequencep ada_grammar-names)
(length ada_grammar-names)
(module-load "test_module_vector.dll")

=> module-load: Wrong type argument: sequencep, ada_grammar-names


This happens on both Windows Mingw32 and Debian x86_64


--
-- Stephe

Re: Dynamic loading progress

[Ivan Shmakov]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

[…]

 > However, when I try to access an elisp vector, it fails:

[…]

 > emacs_value names = env->intern (env, "ada_grammar-names");
 > emacs_value args[] = { names };
 > int64_t names_length = env->fixnum_to_int (env, env->funcall (env, length, 1, args));

	Isn’t the above essentially (length 'ada_grammar-names)?  Or,
	verbosely speaking:

  (let* ((length (intern "length"))
         (names (intern "ada_grammar-names"))
     	 (names-length (funcall length names)))
    …)

	(That is, the code above wants to apply 'length to the /value/
	of a variable – not its /name/.)

[…]

 > => module-load: Wrong type argument: sequencep, ada_grammar-names

 > This happens on both Windows Mingw32 and Debian x86_64

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

Re: Dynamic loading progress

[Stephen Leake]

Ivan Shmakov <ivan@siamics.net> writes:

>>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
> […]
>
>  > However, when I try to access an elisp vector, it fails:
>
> […]
>
>  > emacs_value names = env->intern (env, "ada_grammar-names");
>  > emacs_value args[] = { names };
>  > int64_t names_length = env->fixnum_to_int (env, env->funcall (env, length, 1, args));
>
> 	Isn’t the above essentially (length 'ada_grammar-names)?  Or,
> 	verbosely speaking:
>
>   (let* ((length (intern "length"))
>          (names (intern "ada_grammar-names"))
>      	 (names-length (funcall length names)))
>     …)
>
> 	(That is, the code above wants to apply 'length to the /value/
> 	of a variable – not its /name/.)

Yes, that's it. Fixed by adding a call to 'symbol-value'.

It might be good to add a C-level env->symbol_value for this purpose.

Thanks,

-- 
-- Stephe

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> I've tried to implement modules using Daniel's plan. It turned out to
> be pretty easy, mostly because I'm more familiar with the Emacs
> codebase now and because it's a lot less intrusive than my previous
> attempt.
>
> It's not finished but I have a basic module working on linux. It's a
> proof of concept, basically.

I've implemented a test module that calls two Ada mode parser actions
(see below).

The real module will call the actual parser and build the action args
from the parse results.

I did not have to modify anything in the Emacs C code to make this work,
so it shows this module design is adquate for my use case so far.

I still need to implement the lexer level, which will either need to
call the current elisp lexer, or access the buffer text directly.

Any comments on the coding style? I don't think I need GCPRO on any of
the static emacs_values, because they are all declared at global level
in elisp.

--
-- Stephe


/*

  Emacs module for Ada mode, providing a fast generalized LALR parser

*/

#include <emacs_module.h>

int plugin_is_GPL_compatible;

/* elisp functions */
static emacs_value aref;
static emacs_value cons;
static emacs_value length;
static emacs_value nil;
static emacs_value set;
static emacs_value symbol_value;
static emacs_value vector;
static emacs_value wisi_containing_action;
static emacs_value wisi_statement_action;

/* variables */
static emacs_value wisi_tokens;

/* array of symbols used in grammar actions */
static emacs_value* ada_grammar_names;

static emacs_value Fcons (emacs_env *env, emacs_value a, emacs_value b)
{
  emacs_value args[] = { a, b };

  return env->funcall (env, cons, 2, args);
}

/* Parse BUFFER, using parser in current module. */
static void parse (emacs_env *env, emacs_value buffer)
{
  /* For now, just execute the actions from parsing second line in:
     procedure Foo is
     begin
     A := 1;
     end Foo;

     ada_grammar.adb assignment_statement_0
     wisi-tokens: [(name 24 . 25) (COLON_EQUAL 26 . 28) (expression 29 . 30) (SEMICOLON 30 . 31)]
     action: (wisi-statement-action [1 statement-start 2 statement-other 4 statement-end])
  */

  /* Set `wisi-tokens'; wisi-tokens is let-bound in calling elisp
     function. */

  emacs_value tokens[] =
    {  /* name "A" */
      Fcons (env, ada_grammar_names[245], Fcons (env, env->make_fixnum (env, 24), env->make_fixnum (env, 25))),

       /* COLON_EQUAL ":=" */
      Fcons (env, ada_grammar_names[ 81], Fcons (env, env->make_fixnum (env, 26), env->make_fixnum (env, 28))),

       /* expression "1" */
      Fcons (env, ada_grammar_names[194], Fcons (env, env->make_fixnum (env, 29), env->make_fixnum (env, 30))),

       /* SEMICOLON ";" */
      Fcons (env, ada_grammar_names[ 95], Fcons (env, env->make_fixnum (env, 30), env->make_fixnum (env, 31)))
    };

  emacs_value args_1[] = { wisi_tokens, env->funcall (env, vector, 4, tokens) };

  env->funcall (env, set, 2, args_1);

  emacs_value pairs[] =
    { env->make_fixnum (env, 1), ada_grammar_names[341], /* statement-start */
      env->make_fixnum (env, 2), ada_grammar_names[342], /* statement-other */
      env->make_fixnum (env, 4), ada_grammar_names[343]  /* statement-end */
    };

  emacs_value args_2[] = { env->funcall (env, vector, 6, pairs) };

  env->funcall (env, wisi_statement_action, 1, args_2);

  emacs_value args_3[] = { env->make_fixnum (env, 2), env->make_fixnum (env, 3) };

  env->funcall (env, wisi_containing_action, 2, args_3);

}

/* elisp-callable wrapper for `parse'
   args: BUFFER
*/
static emacs_value Fparse (emacs_env *env, int nargs, emacs_value args[])
{
  parse (env, args[0]);

  return nil;
}

/* Binds NAME to FUN */
static void bind_function (emacs_env *env, const char *name, emacs_value Ffun)
{
  emacs_value Qfset  = env->intern (env, "fset");
  emacs_value Qsym   = env->intern (env, name);
  emacs_value args[] = { Qsym, Ffun };

  env->funcall (env, Qfset, 2, args);
}

/* standard Emacs module init */
int emacs_module_init (struct emacs_runtime *ert)
{
  emacs_env *env = ert->get_environment (ert);

  aref                   = env->intern (env, "aref");
  cons                   = env->intern (env, "cons");
  length                 = env->intern (env, "length");
  nil                    = env->intern (env, "nil");
  set                    = env->intern (env, "set");
  symbol_value           = env->intern (env, "symbol-value");
  vector                 = env->intern (env, "vector");
  wisi_statement_action  = env->intern (env, "wisi-statement-action");
  wisi_containing_action = env->intern (env, "wisi-containing-action");

  wisi_tokens = env->intern (env, "wisi-tokens");

  emacs_value names_1 = env->intern (env, "ada_grammar-names");
  emacs_value args_1[] = { names_1 };
  emacs_value names_2 = env->funcall (env, symbol_value, 1, args_1);
  emacs_value args_2[] = { names_2 };
  int64_t names_length = env->fixnum_to_int (env, env->funcall (env, length, 1, args_2));

  ada_grammar_names = malloc (names_length * sizeof (int64_t));

  for (int i = 0; i < names_length; i++)
    {
      emacs_value args[] = {names_2, env->make_fixnum (env, i)};
      ada_grammar_names[i] = env->funcall (env, aref, 2, args);
    };

  bind_function (env, "wisi-module-parse", env->make_function (env, 1, 1, Fparse));

  return 0;
}

Re: Dynamic loading progress

[Aurélien Aptel]

On Mon, Mar 16, 2015 at 7:02 PM, Stephen Leake
<stephen_leake@stephe-leake.org> wrote:
> I've implemented a test module that calls two Ada mode parser actions
> (see below).
>
> The real module will call the actual parser and build the action args
> from the parse results.
>
> I did not have to modify anything in the Emacs C code to make this work,
> so it shows this module design is adquate for my use case so far.

Good! I'd be interested in performance comparison between the pure
elisp version and this module version. I suspect it's not going to be
much unfortunately. You module doesn't seem to be resource intensive
(as of now) anyway.

> I still need to implement the lexer level, which will either need to
> call the current elisp lexer, or access the buffer text directly.

No buffer or string access yet, sorry. I guess you can work around it
by retrieving each character as an int but that's neither practical
nor efficient.

> Any comments on the coding style? I don't think I need GCPRO on any of

There's no GC protection in the module API yet. I think there are some
place of your module that might not be safe. See below.

> the static emacs_values, because they are all declared at global level
> in elisp.

I had not thought about that: global refs can be defined in pure
elisp. That indeed removes the need for registering global values via
the module API.
I'm still not sure I understand what Daniel had in mind for
global/local value registration, I'm kind of waiting for his reply to
move forward. I'll try reading more thoroughly the JNI document he
linked in a previous message in the meantime.


>   emacs_value tokens[] =
>     {  /* name "A" */
>       Fcons (env, ada_grammar_names[245], Fcons (env, env->make_fixnum (env, 24), env->make_fixnum (env, 25))),
>
>        /* COLON_EQUAL ":=" */
>       Fcons (env, ada_grammar_names[ 81], Fcons (env, env->make_fixnum (env, 26), env->make_fixnum (env, 28))),
>
>        /* expression "1" */
>       Fcons (env, ada_grammar_names[194], Fcons (env, env->make_fixnum (env, 29), env->make_fixnum (env, 30))),
>
>        /* SEMICOLON ";" */
>       Fcons (env, ada_grammar_names[ 95], Fcons (env, env->make_fixnum (env, 30), env->make_fixnum (env, 31)))
>     };

^^^^ all those conses can be GC'd if any of the code afterwards can
call the GC. It might not be the case right now put it probably will
at some point. I'm not sure about this but if you call a pure-elisp
function the GC will trigger.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Tue, 17 Mar 2015 10:28:13 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> > I still need to implement the lexer level, which will either need to
> > call the current elisp lexer, or access the buffer text directly.
> 
> No buffer or string access yet, sorry. I guess you can work around it
> by retrieving each character as an int but that's neither practical
> nor efficient.

Actually, most accesses to buffer text do precisely that: access one
character at a time.  The API typically accepts the buffer position.
Why do you think this is inefficient?

If we think modules will need to access lexical units, we might
provide APIs for that, although I expect this to be complicated, due
to the different semantics of what a "word" etc. is, buffer-local
syntax tables, and other complications.

Direct access to buffer text as a C array is also a possibility, but
modules that do that will have to handle the gap, and also cope with
buffer relocation (on platforms that use ralloc.c).

Re: Dynamic loading progress

[Aurélien Aptel]

On Tue, Mar 17, 2015 at 10:52 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> Actually, most accesses to buffer text do precisely that: access one
> character at a time.  The API typically accepts the buffer position.
> Why do you think this is inefficient?

Iterating on a buffer content will need, for each character (I think)
2 funcalls (char-after, forward-char) and a fixnum -> int64_t
conversion. It seems like a non-negligible overhead, but maybe it
isn't, I don't know...

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Tue, 17 Mar 2015 11:51:44 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> On Tue, Mar 17, 2015 at 10:52 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > Actually, most accesses to buffer text do precisely that: access one
> > character at a time.  The API typically accepts the buffer position.
> > Why do you think this is inefficient?
> 
> Iterating on a buffer content will need, for each character (I think)
> 2 funcalls (char-after, forward-char) and a fixnum -> int64_t
> conversion. It seems like a non-negligible overhead, but maybe it
> isn't, I don't know...

Why do we need to go through char-after, with the induced fixnum
conversion?  Why not simply return a character, which we know in
advance will fit into a C 'int'?

Once again, what I had in mind is a function that gets buffer position
and returns the character at that position, as a C 'int'.  That's
all.  I see no reason to go through Lisp primitives for that job.
Maybe I'm missing something.

Re: Dynamic loading progress

[Stefan Monnier]

> Actually, most accesses to buffer text do precisely that: access one
> character at a time.  The API typically accepts the buffer position.
> Why do you think this is inefficient?

For efficient parsing (as done by forward-word, forward-sexp and
friends), I think you need to keep both the current "charpos" and
"bytepos", but Elisp interfaces don't give access to this level.


        Stefan

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>,
>   stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Tue, 17 Mar 2015 12:37:27 -0400
> 
> > Actually, most accesses to buffer text do precisely that: access one
> > character at a time.  The API typically accepts the buffer position.
> > Why do you think this is inefficient?
> 
> For efficient parsing (as done by forward-word, forward-sexp and
> friends), I think you need to keep both the current "charpos" and
> "bytepos", but Elisp interfaces don't give access to this level.

The API we provide for accessing buffer text should handle that as
well.  For example, we could model it on FETCH_CHAR_ADVANCE.

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Mon, Mar 16, 2015 at 7:02 PM, Stephen Leake
> <stephen_leake@stephe-leake.org> wrote:
>> I've implemented a test module that calls two Ada mode parser actions
>> (see below).
>>
>> The real module will call the actual parser and build the action args
>> from the parse results.
>>
>> I did not have to modify anything in the Emacs C code to make this work,
>> so it shows this module design is adquate for my use case so far.
>
> Good! I'd be interested in performance comparison between the pure
> elisp version and this module version. I suspect it's not going to be
> much unfortunately. You module doesn't seem to be resource intensive
> (as of now) anyway.

It doesn't currently do anything; it just explors how to call elisp from
the module. It will eventually call the full lexer and parser.

>> I still need to implement the lexer level, which will either need to
>> call the current elisp lexer, or access the buffer text directly.
>
> No buffer or string access yet, sorry. I guess you can work around it
> by retrieving each character as an int but that's neither practical
> nor efficient.

Right. But I'd like to try calling the ada-mode elisp lexer function
anyway; less code to port to the module, and I suspect it's faster.

>>   emacs_value tokens[] =
>>     {  /* name "A" */
>>       Fcons (env, ada_grammar_names[245], Fcons (env, env->make_fixnum (env, 24), env->make_fixnum (env, 25))),
>>
>>        /* COLON_EQUAL ":=" */
>>       Fcons (env, ada_grammar_names[ 81], Fcons (env, env->make_fixnum (env, 26), env->make_fixnum (env, 28))),
>>
>>        /* expression "1" */
>>       Fcons (env, ada_grammar_names[194], Fcons (env, env->make_fixnum (env, 29), env->make_fixnum (env, 30))),
>>
>>        /* SEMICOLON ";" */
>>       Fcons (env, ada_grammar_names[ 95], Fcons (env, env->make_fixnum (env, 30), env->make_fixnum (env, 31)))
>>     };
>
> ^^^^ all those conses can be GC'd if any of the code afterwards can
> call the GC. It might not be the case right now put it probably will
> at some point. I'm not sure about this but if you call a pure-elisp
> function the GC will trigger.

Right. This is a temporary list built as an argument to an elisp call. I
want it to be garbage collected after the call, but not during.

module_funcall in module.c copies the args (shallow copy only) and does
GCPRO1 on the copy (actually on the first element of the copy), then
does the actual funcall, followed by GCUNPRO. Is that sufficient? 

On the other hand, can we eliminate the copy to save time? I think all
we need to do is include 'fun' in 'args', as the standard lisp syntax does.

-- 
-- Stephe

Re: Dynamic loading progress

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>
>> I've tried to implement modules using Daniel's plan. It turned out to
>> be pretty easy, mostly because I'm more familiar with the Emacs
>> codebase now and because it's a lot less intrusive than my previous
>> attempt.
>>
>> It's not finished but I have a basic module working on linux. It's a
>> proof of concept, basically.
>
> I've implemented a test module that calls two Ada mode parser actions
> (see below).

I've now translated that test module into Ada, and that version also
works on both Windows and Debian.

It is far easier to import this emacs_module.h into Ada than the subset
of lisp.h in the other dynamic modules approach. It will also be easier
for other module implementation languages to use.

So I suggest we adopt this approach as the way forward.

-- 
-- Stephe

Re: Dynamic loading progress; funcall of goto-char fails

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>
>> I've tried to implement modules using Daniel's plan. It turned out to
>> be pretty easy, mostly because I'm more familiar with the Emacs
>> codebase now and because it's a lot less intrusive than my previous
>> attempt.
>>
>> It's not finished but I have a basic module working on linux. It's a
>> proof of concept, basically.
>
> I've implemented a test module that calls two Ada mode parser actions
> (see below).
>
> The real module will call the actual parser and build the action args
> from the parse results.
>
> I did not have to modify anything in the Emacs C code to make this work,
> so it shows this module design is adquate for my use case so far.

I've made more progress; I've got a module that should run the full Ada
parser on a buffer.

However, when I run it, it crashes on a call like:

emacs_value goto_char_sym = env.intern (env, "goto-char");
env.funcall (env, goto_char_sym, env.make_fixnum (env, 1));

When I trace the 'funcall' (in eval.c DEFUN funcall), it hits this code:

  fun = original_fun;
  if (SYMBOLP (fun) && !NILP (fun)
      && (fun = XSYMBOL (fun)->function, SYMBOLP (fun)))
    fun = indirect_function (fun);

and sets fun to 0, which then signals a failure.

Is this because 'goto-char' is implemented in C?

If so, I need a way to call elisp functions implemented in C.

Hmm. I guess for now I can provide elisp wrappers for all the functions
I need.

-- 
-- Stephe

Re: Dynamic loading progress; funcall of goto-char fails

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1687 bytes --]

On 05/05/2015 09:11 PM, Stephen Leake wrote:
> Stephen Leake <stephen_leake@stephe-leake.org> writes:
> 
>> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>>
>>> I've tried to implement modules using Daniel's plan. It turned out to
>>> be pretty easy, mostly because I'm more familiar with the Emacs
>>> codebase now and because it's a lot less intrusive than my previous
>>> attempt.
>>>
>>> It's not finished but I have a basic module working on linux. It's a
>>> proof of concept, basically.
>>
>> I've implemented a test module that calls two Ada mode parser actions
>> (see below).
>>
>> The real module will call the actual parser and build the action args
>> from the parse results.
>>
>> I did not have to modify anything in the Emacs C code to make this work,
>> so it shows this module design is adquate for my use case so far.
> 
> I've made more progress; I've got a module that should run the full Ada
> parser on a buffer.
> 
> However, when I run it, it crashes on a call like:
> 
> emacs_value goto_char_sym = env.intern (env, "goto-char");
> env.funcall (env, goto_char_sym, env.make_fixnum (env, 1));
> 
> When I trace the 'funcall' (in eval.c DEFUN funcall), it hits this code:
> 
>   fun = original_fun;
>   if (SYMBOLP (fun) && !NILP (fun)
>       && (fun = XSYMBOL (fun)->function, SYMBOLP (fun)))
>     fun = indirect_function (fun);
> 
> and sets fun to 0, which then signals a failure.
> 
> Is this because 'goto-char' is implemented in C?

That's odd. Why would it matter? For me, (symbol-function 'goto-char) is
a subr, so the second SYMBOLP above should fail and `fun' should be a
subr object, not nil.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress; funcall of goto-char fails

[Stephen Leake]

Daniel Colascione <dancol@dancol.org> writes:

> On 05/05/2015 09:11 PM, Stephen Leake wrote:
>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>> 
>>> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>>>
>>>> I've tried to implement modules using Daniel's plan. It turned out to
>>>> be pretty easy, mostly because I'm more familiar with the Emacs
>>>> codebase now and because it's a lot less intrusive than my previous
>>>> attempt.
>>>>
>>>> It's not finished but I have a basic module working on linux. It's a
>>>> proof of concept, basically.
>>>
>>> I've implemented a test module that calls two Ada mode parser actions
>>> (see below).
>>>
>>> The real module will call the actual parser and build the action args
>>> from the parse results.
>>>
>>> I did not have to modify anything in the Emacs C code to make this work,
>>> so it shows this module design is adquate for my use case so far.
>> 
>> I've made more progress; I've got a module that should run the full Ada
>> parser on a buffer.
>> 
>> However, when I run it, it crashes on a call like:
>> 
>> emacs_value goto_char_sym = env.intern (env, "goto-char");
>> env.funcall (env, goto_char_sym, env.make_fixnum (env, 1));
>> 
>> When I trace the 'funcall' (in eval.c DEFUN funcall), it hits this code:
>> 
>>   fun = original_fun;
>>   if (SYMBOLP (fun) && !NILP (fun)
>>       && (fun = XSYMBOL (fun)->function, SYMBOLP (fun)))
>>     fun = indirect_function (fun);
>> 
>> and sets fun to 0, which then signals a failure.
>> 
>> Is this because 'goto-char' is implemented in C?
>
> That's odd. Why would it matter? For me, (symbol-function 'goto-char) is
> a subr, so the second SYMBOLP above should fail and `fun' should be a
> subr object, not nil.

Never mind; I found the problem. Due to a cut-and-paste error, I spelled
"goto-char" as "goto_char". And since intern-soft is not currently in
emacs_module.h, I used intern, which of course returned a non-function
symbol instead of an error.

All better now.

Sorry for the noise.

I will have a list of suggestions to add to emacs_module.h once I get
this working.

-- 
-- Stephe

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Tue, 17 Feb 2015 18:46:33 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>, Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> We should call `provide' in the module (like in my module branch)
> using the API. Problem solved?

It will be solved if there's a provide function available in the
interface.  Daniel's suggestion didn't include it, which is why I
asked about that.  Daniel seems to be unconvinced that it's needed.

> Concerning docstrings, we just have to teach the makedoc utility how
> to extract them from this new module system.

Extract them to which place?

Besides, make-docfile is not installed with Emacs, and so will
probably be unavailable when the module is built.

Re: Dynamic loading progress

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

>> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>

>> We should call `provide' in the module (like in my module branch)
>> using the API. Problem solved?

> It will be solved if there's a provide function available in the
> interface.  Daniel's suggestion didn't include it, which is why I
> asked about that.  Daniel seems to be unconvinced that it's needed.

Perhaps like so...?
 emacs_eval(env, "(provide 'mymodule)");

Re: Dynamic loading progress

[Stephen J. Turnbull]

Aurélien Aptel writes:

 > I'm with Eli on the versionning choice

Versioning is a poor choice here for the same reasons it almost always
is.  Dan's design means that the module doesn't need to care about the
implementation, except for the usual reasons of breaking promises
about semantics.  It's equivalent to the recommended design of using
feature tests rather than version tests.

 > but it's just bike shedding at this point.

With Dan's design, there's nothing left *but* bikeshedding, Brother
Aurélien.  It solves problems I didn't know XEmacs's design has.

 > Interning all the time seems tedious and probably (a bit?) slow but
 > moving the interning on Emacs side won't change the slow part.

It's not detectably slow, especially if you cache the result, and the
intern-and-cache process can be performed by a macro in module_api.h
or by a function exported from Emacs.  A little bit of elisp for
collecting and decorating the names you need in a module init
function, and it's bearable tedium.  This is based on a decade and a
half of experience with XEmacs's "ELL" module system.

 > At least on the module side you can do it once and reuse the
 > result. Or maybe you shouldn't if the user changes the binding in
 > the meantime?

I'm not sure about Dan's "one more indirection" design, but in the
case where you use the result of intern directly, this has the
expected semantics: the user's change wins.  Dan's design should have
the same semantics, I'm guessing.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am So., 15. Feb. 2015 um
21:21 Uhr:

>   typedef struct emacs_value_tag* emacs_value;
>

Would it make sense to not use a typedef here? Using a typedef means that
the type including its size is opaque and subject to change, which can
break ABI compatibility. I'd rather have something like:

struct emacs_value {
  // contains private fields
};

and then pass *struct emacs_value** around.

[-- Attachment #2: Type: text/html, Size: 780 bytes --]

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On 09/13/2015 06:04 AM, Philipp Stephani wrote:
> Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>> schrieb
> am So., 15. Feb. 2015 um 21:21 Uhr:
> 
>       typedef struct emacs_value_tag* emacs_value;
> 
> 
> Would it make sense to not use a typedef here? Using a typedef means
> that the type including its size is opaque and subject to change, which
> can break ABI compatibility. I'd rather have something like:
> 
> struct emacs_value {
>   // contains private fields
> };
> 
> and then pass /struct emacs_value*/ around.

You may have missed the "*" in the typedef. The difference is stylistic.
There's no difference between foo and bar here.

typedef struct valuex* value;
void foo(struct valuex* x);
void bar(value y);

I find the typedef much more readable, however.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am So., 13. Sep. 2015 um
16:15 Uhr:

> On 09/13/2015 06:04 AM, Philipp Stephani wrote:
> > Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>> schrieb
> > am So., 15. Feb. 2015 um 21:21 Uhr:
> >
> >       typedef struct emacs_value_tag* emacs_value;
> >
> >
> > Would it make sense to not use a typedef here? Using a typedef means
> > that the type including its size is opaque and subject to change, which
> > can break ABI compatibility. I'd rather have something like:
> >
> > struct emacs_value {
> >   // contains private fields
> > };
> >
> > and then pass /struct emacs_value*/ around.
>
> You may have missed the "*" in the typedef. The difference is stylistic.
> There's no difference between foo and bar here.
>
> typedef struct valuex* value;
> void foo(struct valuex* x);
> void bar(value y);
>
> I find the typedef much more readable, however.
>
>
There's no difference in your design, but using a typedef makes it possible
to use a non-pointer type without changing the API in obvious ways.
E.g. Linus is strongly against such typedefs:
http://lkml.iu.edu/hypermail/linux/kernel/0206.1/0402.html

[-- Attachment #2: Type: text/html, Size: 1799 bytes --]

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

On 09/13/2015 07:27 AM, Philipp Stephani wrote:
> Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>> schrieb
> am So., 13. Sep. 2015 um 16:15 Uhr:
> 
>     On 09/13/2015 06:04 AM, Philipp Stephani wrote:
>     > Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>
>     <mailto:dancol@dancol.org <mailto:dancol@dancol.org>>> schrieb
>     > am So., 15. Feb. 2015 um 21:21 Uhr:
>     >
>     >       typedef struct emacs_value_tag* emacs_value;
>     >
>     >
>     > Would it make sense to not use a typedef here? Using a typedef means
>     > that the type including its size is opaque and subject to change,
>     which
>     > can break ABI compatibility. I'd rather have something like:
>     >
>     > struct emacs_value {
>     >   // contains private fields
>     > };
>     >
>     > and then pass /struct emacs_value*/ around.
> 
>     You may have missed the "*" in the typedef. The difference is stylistic.
>     There's no difference between foo and bar here.
> 
>     typedef struct valuex* value;
>     void foo(struct valuex* x);
>     void bar(value y);
> 
>     I find the typedef much more readable, however.
> 
> 
> There's no difference in your design, but using a typedef makes it
> possible to use a non-pointer type without changing the API in obvious
> ways. 
> E.g. Linus is strongly against such
> typedefs: http://lkml.iu.edu/hypermail/linux/kernel/0206.1/0402.html

Linus is also against integer overflow checking. So what? I can't stand
argumentum ad Linus.

I still find the typedef more readable, because to users, emacs_value is
an opaque type, and the fact that we implement it as a pointer is
irrelevant.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2130 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am So., 13. Sep. 2015 um
16:31 Uhr:

> On 09/13/2015 07:27 AM, Philipp Stephani wrote:
> > Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>> schrieb
> > am So., 13. Sep. 2015 um 16:15 Uhr:
> >
> >     On 09/13/2015 06:04 AM, Philipp Stephani wrote:
> >     > Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>
> >     <mailto:dancol@dancol.org <mailto:dancol@dancol.org>>> schrieb
> >     > am So., 15. Feb. 2015 um 21:21 Uhr:
> >     >
> >     >       typedef struct emacs_value_tag* emacs_value;
> >     >
> >     >
> >     > Would it make sense to not use a typedef here? Using a typedef
> means
> >     > that the type including its size is opaque and subject to change,
> >     which
> >     > can break ABI compatibility. I'd rather have something like:
> >     >
> >     > struct emacs_value {
> >     >   // contains private fields
> >     > };
> >     >
> >     > and then pass /struct emacs_value*/ around.
> >
> >     You may have missed the "*" in the typedef. The difference is
> stylistic.
> >     There's no difference between foo and bar here.
> >
> >     typedef struct valuex* value;
> >     void foo(struct valuex* x);
> >     void bar(value y);
> >
> >     I find the typedef much more readable, however.
> >
> >
> > There's no difference in your design, but using a typedef makes it
> > possible to use a non-pointer type without changing the API in obvious
> > ways.
> > E.g. Linus is strongly against such
> > typedefs: http://lkml.iu.edu/hypermail/linux/kernel/0206.1/0402.html
>
> Linus is also against integer overflow checking. So what? I can't stand
> argumentum ad Linus.
>
> I still find the typedef more readable, because to users, emacs_value is
> an opaque type, and the fact that we implement it as a pointer is
> irrelevant.
>
>
Fair enough, this is really just a minor nitpick. I was worried a bit to
see it typedef'd to void* and cast to Lisp_Object in the implementation,
which decreases type safety a bit and assumes that a Lisp object is always
the size of a pointer, but probably that's minor and I worry too much.

[-- Attachment #2: Type: text/html, Size: 3276 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3974 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am So., 15. Feb. 2015 um
21:21 Uhr:

> Here's a broad outline of what I have in mind.
>

Thanks, that looks really good. Just a few minor issues that I encountered
over the last couple of weeks.


> Thread-local environments
> -------------------------
>
> The `get_environment' member lets us do anything else interesting. As
> in Java, environments are thread-local. We only support one thread for
> the moment, so this constraint is easy to enforce. (Just abort if we
> use an emacs_env off the main thread.)
>

Would you really abort, or rather use the error handling functions? We
should be able to make the error values thread-local so that calling a
function from the wrong thread would be the equivalent of raising a signal,
giving the caller a chance to react. Otherwise the burden of remembering
the correct thread would be on the caller's side.


>   typedef struct emacs_value_tag* emacs_value;
>

I think it's important that this is a pointer to a struct (for type safety
and size correctness) rather than just an arbitrary type.


>
>   typedef emacs_value (*emacs_subr)(
>     emacs_env* env,
>     int nargs,
>     emacs_value args[]);
>

Could we give this a void* data parameter for storing arbitrary user data?
This is common for callbacks in C interfaces and allows C++ users to store
C++ objects in the pointer. This can be implemented using another save
pointer.


>    emacs_value (*make_function)(
>       emacs_env* env,
>       int min_arity,
>       int max_arity,
>       emacs_subr function);
>

Similary, here void* data should be passed to be retrieved later.


>
>     emacs_value (*funcall)(
>       emacs_env* env,
>       emacs_value function,
>       int nargs,
>       emacs_value args[]);
>

Does function have to be a function object, or can it be a symbol? I.e. is
the user supposed to call symbol-function first?


>     int64_t (*fixnum_to_int)(
>       emacs_env* env,
>       emacs_value value);
>
>     emacs_value (*make_fixnum)(
>       emacs_env* env,
>       int64_t value);
>
>
What's the behavior of these functions if the source would not fit into the
result? Undefined behavior, abort(), raising a signal?


> Modules can use make_global_reference to allocate a global reference
> (i.e., a GC root) for any emacs_value; modules must then free these
> references explicitly.
>
> All routines (except make_global_reference) that return emacs_value
> values return local references.  It's up to modules to register
> long-lived references explicitly.
>

In which cases would global references be necessary?


> Like JNI, we can just declare that it's illegal to call all but a few
> specially-marked functions (like global reference deregistration) with
> a pending error.
>

What's the behavior if other functions are called? abort()?


> If Lisp signals or throws, `funcall' returns NULL.
>

Hmm, with the current implementation emacs_value is just the same as
Lisp_Object, i.e. not a real pointer, so NULL doesn't have specific
semantics. Should it return Qnil instead and force the user to use
check_error?


> 1) Do we need JNI-style local variable frames and functions that
>    release local references?
>
> 2) Maybe we want a separate, non-emacs_value type for global references?
>

No idea about these.


>
> 3) How exactly do we represent catch/throw values?
>

I've thought about this a bit, and I think it would be simplest to add a
new function env->set_throw and have get_error and check_error return an
enum { normal, signal, throw }. One could come up with something like
creating a list (error-kind error-tag error-value), but it looks like the
module implementation would create such lists only for the module code to
convert them back, so it's simpler to represent the two kinds of non-local
exits directly in the interface.


>
> 4) Do we need to use int64_t for integers?
>

Maybe just intmax_t and a static check that that is larger than an Elisp
integer?

[-- Attachment #2: Type: text/html, Size: 6266 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 04 Oct 2015 08:57:36 +0000
> Cc: stephen_leake@stephe-leake.org, emacs-devel@gnu.org, 
> 	Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> 
>     4) Do we need to use int64_t for integers?
>     
> Maybe just intmax_t and a static check that that is larger than an Elisp
> integer?

AFAIU, that will always fail in a 32-bit build --with-wide-int, right?

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 5513 bytes --]

On 10/04/2015 01:57 AM, Philipp Stephani wrote:
> Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>> schrieb
> am So., 15. Feb. 2015 um 21:21 Uhr:
> 
>     Here's a broad outline of what I have in mind.
> 
> 
> Thanks, that looks really good. Just a few minor issues that I
> encountered over the last couple of weeks.
>  
> 
>     Thread-local environments
>     -------------------------
> 
>     The `get_environment' member lets us do anything else interesting. As
>     in Java, environments are thread-local. We only support one thread for
>     the moment, so this constraint is easy to enforce. (Just abort if we
>     use an emacs_env off the main thread.)
> 
> 
> Would you really abort, or rather use the error handling functions? We
> should be able to make the error values thread-local so that calling a
> function from the wrong thread would be the equivalent of raising a
> signal, giving the caller a chance to react. Otherwise the burden of
> remembering the correct thread would be on the caller's side.

If we abort, thread mismatch is a programming error, and we can omit
optionally the check for performance. If we fail in some recoverable
way, we have to perform the thread check every time, since it's now
contractual.

>  
> 
>       typedef struct emacs_value_tag* emacs_value;
> 
> 
> I think it's important that this is a pointer to a struct (for type
> safety and size correctness) rather than just an arbitrary type.

A typedef is exactly as typesafe. The question of whether to use a
struct or a typedef is aesthetic. I strongly prefer a typedef, just like
pthreads, and I believe that the people who advocate using structs
directly are simply wrong.


>       typedef emacs_value (*emacs_subr)(
>         emacs_env* env,
>         int nargs,
>         emacs_value args[]);
> 
> 
> Could we give this a void* data parameter for storing arbitrary user
> data? This is common for callbacks in C interfaces and allows C++ users
> to store C++ objects in the pointer. This can be implemented using
> another save pointer.

Yes, of course.

>  
> 
> 
>         emacs_value (*funcall)(
>           emacs_env* env,
>           emacs_value function,
>           int nargs,
>           emacs_value args[]);
> 
> 
> Does function have to be a function object, or can it be a symbol? I.e.
> is the user supposed to call symbol-function first?

It'll have the same semantics as Ffuncall, so we'll be able to call a
symbol or a function.

>         int64_t (*fixnum_to_int)(
>           emacs_env* env,
>           emacs_value value);
> 
>         emacs_value (*make_fixnum)(
>           emacs_env* env,
>           int64_t value);
> 
> 
> What's the behavior of these functions if the source would not fit into
> the result? Undefined behavior, abort(), raising a signal?

I think we can do a range check fairly inexpensively, so just failing is
probably fine. Code that ignores the possibility of failure will just
abort later when it tries to misuse the API. If that's too expensive or
error prone, we can just convert to an unspecified value when out of
range (in practice, lopping off the extra bits).

>     Modules can use make_global_reference to allocate a global reference
>     (i.e., a GC root) for any emacs_value; modules must then free these
>     references explicitly.
> 
>     All routines (except make_global_reference) that return emacs_value
>     values return local references.  It's up to modules to register
>     long-lived references explicitly.
> 
> 
> In which cases would global references be necessary?

Any time you want to hold onto a lisp value outside the dynamic extent
of your emacs_env.

>     Like JNI, we can just declare that it's illegal to call all but a few
>     specially-marked functions (like global reference deregistration) with
>     a pending error.
> 
> 
> What's the behavior if other functions are called? abort()?

abort in check mode; unspecified when optimizing for performance.

>  
> 
>     If Lisp signals or throws, `funcall' returns NULL.
> 
> 
> Hmm, with the current implementation emacs_value is just the same as
> Lisp_Object, i.e. not a real pointer, so NULL doesn't have specific
> semantics. Should it return Qnil instead and force the user to use
> check_error?

I thought we make Qnil equal zero. In any case, I *don't* like the idea
of Lisp_Object being emacs_value. I'd much rather emacs_value be a
pointer to a Lisp_Object, solving this problem completely.

>     3) How exactly do we represent catch/throw values?
> 
> 
> I've thought about this a bit, and I think it would be simplest to add a
> new function env->set_throw and have get_error and check_error return an
> enum { normal, signal, throw }. One could come up with something like
> creating a list (error-kind error-tag error-value), but it looks like
> the module implementation would create such lists only for the module
> code to convert them back, so it's simpler to represent the two kinds of
> non-local exits directly in the interface.

I'm fine with a list; keep in mind that we'll need to handle OOM
somehow, so I'd suggest an opaque type.

>     4) Do we need to use int64_t for integers?
> 
> 
> Maybe just intmax_t and a static check that that is larger than an Elisp
> integer?

Why make the behavior vary depending on what intmax_t is? At least
int64_t is nice and explicit.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 5174 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am So., 4. Okt. 2015 um
11:10 Uhr:

> On 10/04/2015 01:57 AM, Philipp Stephani wrote:
> > Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>> schrieb
> > am So., 15. Feb. 2015 um 21:21 Uhr:
> >
> >     Here's a broad outline of what I have in mind.
> >
> >
> > Thanks, that looks really good. Just a few minor issues that I
> > encountered over the last couple of weeks.
> >
> >
> >     Thread-local environments
> >     -------------------------
> >
> >     The `get_environment' member lets us do anything else interesting. As
> >     in Java, environments are thread-local. We only support one thread
> for
> >     the moment, so this constraint is easy to enforce. (Just abort if we
> >     use an emacs_env off the main thread.)
> >
> >
> > Would you really abort, or rather use the error handling functions? We
> > should be able to make the error values thread-local so that calling a
> > function from the wrong thread would be the equivalent of raising a
> > signal, giving the caller a chance to react. Otherwise the burden of
> > remembering the correct thread would be on the caller's side.
>
> If we abort, thread mismatch is a programming error, and we can omit
> optionally the check for performance. If we fail in some recoverable
> way, we have to perform the thread check every time, since it's now
> contractual.
>

OK.


>
> >
> >
> >       typedef struct emacs_value_tag* emacs_value;
> >
> >
> > I think it's important that this is a pointer to a struct (for type
> > safety and size correctness) rather than just an arbitrary type.
>
> A typedef is exactly as typesafe. The question of whether to use a
> struct or a typedef is aesthetic. I strongly prefer a typedef, just like
> pthreads, and I believe that the people who advocate using structs
> directly are simply wrong.
>

Ah, I'm not against the typedef, I'm just asking whether you would make it
part of the API contract that it's a typedef of a struct pointer, or
whether it can be any type.


> >     Modules can use make_global_reference to allocate a global reference
> >     (i.e., a GC root) for any emacs_value; modules must then free these
> >     references explicitly.
> >
> >     All routines (except make_global_reference) that return emacs_value
> >     values return local references.  It's up to modules to register
> >     long-lived references explicitly.
> >
> >
> > In which cases would global references be necessary?
>
> Any time you want to hold onto a lisp value outside the dynamic extent
> of your emacs_env.
>

Isn't the dynamic extent of the emacs_env the whole program, starting from
the module initializer?


>
> >     Like JNI, we can just declare that it's illegal to call all but a few
> >     specially-marked functions (like global reference deregistration)
> with
> >     a pending error.
> >
> >
> > What's the behavior if other functions are called? abort()?
>
> abort in check mode; unspecified when optimizing for performance.
>

OK, makes sense. Probably it should be abort() in all cases right now, the
test should be really fast.


>
> >
> >
> >     If Lisp signals or throws, `funcall' returns NULL.
> >
> >
> > Hmm, with the current implementation emacs_value is just the same as
> > Lisp_Object, i.e. not a real pointer, so NULL doesn't have specific
> > semantics. Should it return Qnil instead and force the user to use
> > check_error?
>
> I thought we make Qnil equal zero. In any case, I *don't* like the idea
> of Lisp_Object being emacs_value. I'd much rather emacs_value be a
> pointer to a Lisp_Object, solving this problem completely.
>

I agree with you, but that's how it's currently implemented.


>
> >     3) How exactly do we represent catch/throw values?
> >
> >
> > I've thought about this a bit, and I think it would be simplest to add a
> > new function env->set_throw and have get_error and check_error return an
> > enum { normal, signal, throw }. One could come up with something like
> > creating a list (error-kind error-tag error-value), but it looks like
> > the module implementation would create such lists only for the module
> > code to convert them back, so it's simpler to represent the two kinds of
> > non-local exits directly in the interface.
>
> I'm fine with a list; keep in mind that we'll need to handle OOM
> somehow, so I'd suggest an opaque type.
>

I think OOM is currently handled by doing the equivalent of

(apply #'signal memory-signal-data)

so it would be part of the normal signal handling. Using a list would be
possible, but then the distinction between error tag and data could be
removed from the module code. But I think the 'magic lists' idiom common in
Emacs is more of an antipattern and we should try to avoid it in the module
interface, thus the suggestion to use an enum.


>
> >     4) Do we need to use int64_t for integers?
> >
> >
> > Maybe just intmax_t and a static check that that is larger than an Elisp
> > integer?
>
> Why make the behavior vary depending on what intmax_t is? At least
> int64_t is nice and explicit.
>
>
True. The time when Emacs integers will be larger than 64 bits is probably
far in the future.

[-- Attachment #2: Type: text/html, Size: 7201 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am So., 4. Okt. 2015 um
11:41 Uhr:

>
>> >     3) How exactly do we represent catch/throw values?
>> >
>> >
>> > I've thought about this a bit, and I think it would be simplest to add a
>> > new function env->set_throw and have get_error and check_error return an
>> > enum { normal, signal, throw }. One could come up with something like
>> > creating a list (error-kind error-tag error-value), but it looks like
>> > the module implementation would create such lists only for the module
>> > code to convert them back, so it's simpler to represent the two kinds of
>> > non-local exits directly in the interface.
>>
>> I'm fine with a list; keep in mind that we'll need to handle OOM
>> somehow, so I'd suggest an opaque type.
>>
>
> I think OOM is currently handled by doing the equivalent of
>
> (apply #'signal memory-signal-data)
>
> so it would be part of the normal signal handling. Using a list would be
> possible, but then the distinction between error tag and data could be
> removed from the module code. But I think the 'magic lists' idiom common in
> Emacs is more of an antipattern and we should try to avoid it in the module
> interface, thus the suggestion to use an enum.
>
>

I've amended the PR with basic handling of throw. It's treated as if it
were at the top level, i.e. it generates a signal with symbol 'no-catch and
data (tag value). There is currently no functionality available to "throw"
from module code, and probably none is needed because "throw" is generally
used as a local control flow mechanism, and modules can use the control
flow mechanisms available in their languages instead.

[-- Attachment #2: Type: text/html, Size: 2406 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
>> Why make the behavior vary depending on what intmax_t is? At least
>> >int64_t is nice and explicit.
>> >
>> >
> True. The time when Emacs integers will be larger than 64 bits is probably
> far in the future.

No, I've already been toying with integers wider than 64 bits in my own private 
copy of emacs.  It's not ready for publication yet, but you should be assuming 
that bignums are possible and even desirable, and foreign-function APIs should 
not preclude their use.

Also, the C standard does not require support for int64_t.  It's OK to use 
int64_t in platform-specific code where you know the platform has int64_t, but 
otherwise it's better to avoid it.  The Emacs source code largely avoids int64_t 
now, and there's no good reason to require it here.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1420 bytes --]

On 10/04/2015 12:20 PM, Paul Eggert wrote:
> Philipp Stephani wrote:
>>> Why make the behavior vary depending on what intmax_t is? At least
>>> >int64_t is nice and explicit.
>>> >
>>> >
>> True. The time when Emacs integers will be larger than 64 bits is
>> probably
>> far in the future.
> 
> No, I've already been toying with integers wider than 64 bits in my own
> private copy of emacs.  It's not ready for publication yet, but you
> should be assuming that bignums are possible and even desirable, and
> foreign-function APIs should not preclude their use.

First of all, that's awesome.

Anyway, bignums may be larger than 64-bits, but fixnums never will be.
Nor will C ever have an integer type wide enough to accommodate all
possible numeric types. I'm happy with letting C use int64_t for fixnums
and describing larger numbers through strings or GMP bindings or something.

> Also, the C standard does not require support for int64_t.  It's OK to
> use int64_t in platform-specific code where you know the platform has
> int64_t, but otherwise it's better to avoid it.  The Emacs source code
> largely avoids int64_t now, and there's no good reason to require it here.

We already require C99. Any compiler modern enough to support C99 also
supports synthesizing 64-bit integers from whatever the platform makes
available. I am strongly opposed to catering to obsolete systems.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Daniel Colascione wrote:
> Any compiler modern enough to support C99 also
> supports synthesizing 64-bit integers from whatever the platform makes
> available.

No, all that C99 requires is support for integers *at least* 64 bits.  C99 does 
not require support for int64_t, i.e., integers that are *exactly* 64 bits.

If we need a type that is at least 64 bits wide, we can use int_fast64_t or 
int_least64_t.  These types are required by C99 and will work even on the rare 
platforms that lack native 64-bit words.  But really, I'm hoping we can avoid 
the *int*64* stuff.  What's the point of putting a 64-bit limit on an API that 
is supposed to be long-lived and portable?

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

On 10/04/2015 12:55 PM, Paul Eggert wrote:
> Daniel Colascione wrote:
>> Any compiler modern enough to support C99 also
>> supports synthesizing 64-bit integers from whatever the platform makes
>> available.
> 
> No, all that C99 requires is support for integers *at least* 64 bits. 
> C99 does not require support for int64_t, i.e., integers that are
> *exactly* 64 bits.
> 
> If we need a type that is at least 64 bits wide, we can use int_fast64_t
> or int_least64_t.  These types are required by C99 and will work even on
> the rare platforms that lack native 64-bit words.  But really, I'm
> hoping we can avoid the *int*64* stuff.  What's the point of putting a
> 64-bit limit on an API that is supposed to be long-lived and portable?

Can you really imagine a general-purpose machine with 128-bit words? Why
would you build such a thing?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Daniel Colascione wrote:
> Can you really imagine a general-purpose machine with 128-bit words?

I don't have to imagine it.  I have one on my desk right now, running Ubuntu 
15.04 on a Xeon E3-1225.  If I take this program:

   __int128
   mulbig (long a, long b)
   {
     __int128 aa = a;
     return aa * b;
   }

and compile it with gcc -O2, I get the following machine instructions:

	mulbig:
		movq	%rdi, %rax
		imulq	%rsi
		ret

and this yields a 128-bit value with the mathematically-correct answer (integer 
overflow is not possible for this example).

Admittedly my desktop doesn't support 128-bit words as well as I'd like.  But 
the support should be good enough for Emacs (of course, it should be enabled 
only on platforms that have it).

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2491 bytes --]

On 10/04/2015 02:41 AM, Philipp Stephani wrote:
>     >
>     >
>     >       typedef struct emacs_value_tag* emacs_value;
>     >
>     >
>     > I think it's important that this is a pointer to a struct (for type
>     > safety and size correctness) rather than just an arbitrary type.
> 
>     A typedef is exactly as typesafe. The question of whether to use a
>     struct or a typedef is aesthetic. I strongly prefer a typedef, just like
>     pthreads, and I believe that the people who advocate using structs
>     directly are simply wrong.
> 
> 
> Ah, I'm not against the typedef, I'm just asking whether you would make
> it part of the API contract that it's a typedef of a struct pointer, or
> whether it can be any type.

The problem with defining it as a pointer type is that NULL is now the
invalid sentinel value, which seems incompatible with both making this
thing literally a Lisp_Object and Qnil having all zero bits.

That's why I strongly prefer making emacs_value a _pointer_ to a
Lisp_Object, where we store the Lisp_Object in an array owned by the
emacs_env. This way, allocating local values is very cheap.

>     >     Modules can use make_global_reference to allocate a global
>     reference
>     >     (i.e., a GC root) for any emacs_value; modules must then free
>     these
>     >     references explicitly.
>     >
>     >     All routines (except make_global_reference) that return
>     emacs_value
>     >     values return local references.  It's up to modules to register
>     >     long-lived references explicitly.
>     >
>     >
>     > In which cases would global references be necessary?
> 
>     Any time you want to hold onto a lisp value outside the dynamic extent
>     of your emacs_env.
> 
> 
> Isn't the dynamic extent of the emacs_env the whole program, starting
> from the module initializer?

Not necessarily. An emacs_env is valid only for the current call into
module code on the current thread. So if we call module-function-1, wait
for it to return, then call module-function-2, the two calls can have
different environments, and using local references from one might not be
valid on the other. (This approach opens up very important optimizations
in JNI, and it'd be good for us to use the same approach.) A global
reference is just an emacs_env in a heap-allocated object we register as
a GC root.

(To answer the question another way: global references are GC roots.)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3209 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am So., 4. Okt. 2015 um
21:34 Uhr:

> On 10/04/2015 02:41 AM, Philipp Stephani wrote:
> >     >
> >     >
> >     >       typedef struct emacs_value_tag* emacs_value;
> >     >
> >     >
> >     > I think it's important that this is a pointer to a struct (for type
> >     > safety and size correctness) rather than just an arbitrary type.
> >
> >     A typedef is exactly as typesafe. The question of whether to use a
> >     struct or a typedef is aesthetic. I strongly prefer a typedef, just
> like
> >     pthreads, and I believe that the people who advocate using structs
> >     directly are simply wrong.
> >
> >
> > Ah, I'm not against the typedef, I'm just asking whether you would make
> > it part of the API contract that it's a typedef of a struct pointer, or
> > whether it can be any type.
>
> The problem with defining it as a pointer type is that NULL is now the
> invalid sentinel value, which seems incompatible with both making this
> thing literally a Lisp_Object and Qnil having all zero bits.
>
> That's why I strongly prefer making emacs_value a _pointer_ to a
> Lisp_Object, where we store the Lisp_Object in an array owned by the
> emacs_env. This way, allocating local values is very cheap.
>

Aurélien, is that something you agree with and could implement?


>
> >     >     Modules can use make_global_reference to allocate a global
> >     reference
> >     >     (i.e., a GC root) for any emacs_value; modules must then free
> >     these
> >     >     references explicitly.
> >     >
> >     >     All routines (except make_global_reference) that return
> >     emacs_value
> >     >     values return local references.  It's up to modules to register
> >     >     long-lived references explicitly.
> >     >
> >     >
> >     > In which cases would global references be necessary?
> >
> >     Any time you want to hold onto a lisp value outside the dynamic
> extent
> >     of your emacs_env.
> >
> >
> > Isn't the dynamic extent of the emacs_env the whole program, starting
> > from the module initializer?
>
> Not necessarily. An emacs_env is valid only for the current call into
> module code on the current thread. So if we call module-function-1, wait
> for it to return, then call module-function-2, the two calls can have
> different environments, and using local references from one might not be
> valid on the other. (This approach opens up very important optimizations
> in JNI, and it'd be good for us to use the same approach.) A global
> reference is just an emacs_env in a heap-allocated object we register as
> a GC root.
>
> (To answer the question another way: global references are GC roots.)
>
>
OK, IIUC the examples in Aurélien's branch are then invalid because they
store the result of intern("nil") in a static variable, right?

One more thing: is equality of two emacs_value objects defined? Right now
it's `eq', but that's probably also an implementation detail. If equality
is not defined as `eq', we also need a "null" function in the environment
that returns bool, otherwise there would be no way to implement conditions.

[-- Attachment #2: Type: text/html, Size: 4085 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

On Sun, Oct 4, 2015 at 9:47 PM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> Aurélien, is that something you agree with and could implement?

I'm not sure I understood what you wanted.. I've commited this, but it
assumes Lisp_Object are the same size as pointers...

commit 4c2813950d14fa2348e30ee94a4f3b022263e36d
Author: Aurélien Aptel <aurelien.aptel@gmail.com>
Date:   Sun Oct 4 23:04:56 2015 +0200

    use opaque struct emacs_value_tag instead of void* for emacs_value.

diff --git a/src/emacs_module.h b/src/emacs_module.h
index c5ec347..b055547 100644
--- a/src/emacs_module.h
+++ b/src/emacs_module.h
@@ -27,7 +27,7 @@

 /* Current environement */
 typedef struct emacs_env_25 emacs_env;
-typedef void* emacs_value;
+typedef struct emacs_value_tag* emacs_value;

 enum emacs_type {
   EMACS_FIXNUM,
diff --git a/src/module.c b/src/module.c
index 9bbb832..ab058bb 100644
--- a/src/module.c
+++ b/src/module.c
@@ -24,6 +24,8 @@
 #include "dynlib.h"
 #include "coding.h"

+struct emacs_value_tag { Lisp_Object v; };
+
 void syms_of_module (void);
 static struct emacs_runtime* module_get_runtime (void);
 static emacs_env* module_get_environment (struct emacs_runtime *ert);

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Sun, 4 Oct 2015 23:12:16 +0200
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>, Eli Zaretskii <eliz@gnu.org>, 
> 	Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> On Sun, Oct 4, 2015 at 9:47 PM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> > Aurélien, is that something you agree with and could implement?
> 
> I'm not sure I understood what you wanted.. I've commited this, but it
> assumes Lisp_Object are the same size as pointers...

That assumption is false in an Emacs built --with-wide-int.

If you need to wrap a pointer in a Lisp_Object, we have XIL and XLI
macros for that.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Mo., 5. Okt. 2015 um 07:50 Uhr:

> > Date: Sun, 4 Oct 2015 23:12:16 +0200
> > From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> > Cc: Daniel Colascione <dancol@dancol.org>, Eli Zaretskii <eliz@gnu.org>,
> >       Stephen Leake <stephen_leake@stephe-leake.org>,
> >       Emacs development discussions <emacs-devel@gnu.org>
> >
> > On Sun, Oct 4, 2015 at 9:47 PM, Philipp Stephani <p.stephani2@gmail.com>
> wrote:
> > > Aurélien, is that something you agree with and could implement?
> >
> > I'm not sure I understood what you wanted.. I've commited this, but it
> > assumes Lisp_Object are the same size as pointers...
>
> That assumption is false in an Emacs built --with-wide-int.
>
> If you need to wrap a pointer in a Lisp_Object, we have XIL and XLI
> macros for that.
>

These are no-ops in my version of lisp.h:

# define lisp_h_XLI(o) (o)
# define lisp_h_XIL(i) (i)

Unless I'm misunderstanding something, these macros can't be used to wrap
pointers in the general case.

[-- Attachment #2: Type: text/html, Size: 1862 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Wed, 14 Oct 2015 22:28:01 +0000
> Cc: dancol@dancol.org, stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
>     > I'm not sure I understood what you wanted.. I've commited this, but it
>     > assumes Lisp_Object are the same size as pointers...
>     
>     That assumption is false in an Emacs built --with-wide-int.
>     
>     If you need to wrap a pointer in a Lisp_Object, we have XIL and XLI
>     macros for that.
>     
> 
> These are no-ops in my version of lisp.h: 
> 
> # define lisp_h_XLI(o) (o)
> # define lisp_h_XIL(i) (i)
> 
> Unless I'm misunderstanding something, these macros can't be used to wrap
> pointers in the general case.

We do that all the time, so I don't understand why you understand
that.  Please elaborate.

Re: Dynamic loading progress

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philipp Stephani <p.stephani2@gmail.com>
>> Date: Wed, 14 Oct 2015 22:28:01 +0000
>> Cc: dancol@dancol.org, stephen_leake@stephe-leake.org, emacs-devel@gnu.org
>> 
>>     > I'm not sure I understood what you wanted.. I've commited this, but it
>>     > assumes Lisp_Object are the same size as pointers...
>>     
>>     That assumption is false in an Emacs built --with-wide-int.
>>     
>>     If you need to wrap a pointer in a Lisp_Object, we have XIL and XLI
>>     macros for that.
>>     
>> 
>> These are no-ops in my version of lisp.h: 
>> 
>> # define lisp_h_XLI(o) (o)
>> # define lisp_h_XIL(i) (i)
>> 
>> Unless I'm misunderstanding something, these macros can't be used to wrap
>> pointers in the general case.
>
> We do that all the time, so I don't understand why you understand
> that.  Please elaborate.

You can only use them for pointers that are known to fit in the range of
Lisp_Int, ie. all tag bits clear.  This is true for all lisp object
pointers (obviously), but not for other pointers.  For boxing arbitrary
pointers we have Lisp_Save_Value.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Oct 15, 2015 at 9:00 AM, Andreas Schwab <schwab@suse.de> wrote:
> You can only use them for pointers that are known to fit in the range of
> Lisp_Int, ie. all tag bits clear.  This is true for all lisp object
> pointers (obviously), but not for other pointers.  For boxing arbitrary
> pointers we have Lisp_Save_Value.

I've used Save_Value for storing a module function context but a new
type has to be defined for module pointers in order to handle
user-provided finalizers.

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Philipp Stephani <p.stephani2@gmail.com>,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Thu, 15 Oct 2015 09:00:35 +0200
> 
> >>     If you need to wrap a pointer in a Lisp_Object, we have XIL and XLI
> >>     macros for that.
> >>     
> >> 
> >> These are no-ops in my version of lisp.h: 
> >> 
> >> # define lisp_h_XLI(o) (o)
> >> # define lisp_h_XIL(i) (i)
> >> 
> >> Unless I'm misunderstanding something, these macros can't be used to wrap
> >> pointers in the general case.
> >
> > We do that all the time, so I don't understand why you understand
> > that.  Please elaborate.
> 
> You can only use them for pointers that are known to fit in the range of
> Lisp_Int, ie. all tag bits clear.  This is true for all lisp object
> pointers (obviously), but not for other pointers.  For boxing arbitrary
> pointers we have Lisp_Save_Value.

Right, for pointers to simple data types, like 'char', 'short', 'int',
etc., one needs to use Lisp_Save_Value.  For pointers to struct's, XLI
and XIL will do the job on the supported platforms.

Re: Dynamic loading progress

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Right, for pointers to simple data types, like 'char', 'short', 'int',
> etc., one needs to use Lisp_Save_Value.  For pointers to struct's, XLI
> and XIL will do the job on the supported platforms.

No, it doesn't.  Structures are not special in any way.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: p.stephani2@gmail.com,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Thu, 15 Oct 2015 17:56:36 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Right, for pointers to simple data types, like 'char', 'short', 'int',
> > etc., one needs to use Lisp_Save_Value.  For pointers to struct's, XLI
> > and XIL will do the job on the supported platforms.
> 
> No, it doesn't.

It does in Emacs.

Re: Dynamic loading progress

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@suse.de>
>> Cc: p.stephani2@gmail.com,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
>> Date: Thu, 15 Oct 2015 17:56:36 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Right, for pointers to simple data types, like 'char', 'short', 'int',
>> > etc., one needs to use Lisp_Save_Value.  For pointers to struct's, XLI
>> > and XIL will do the job on the supported platforms.
>> 
>> No, it doesn't.
>
> It does in Emacs.

In which way?

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: p.stephani2@gmail.com,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Thu, 15 Oct 2015 18:07:03 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@suse.de>
> >> Cc: p.stephani2@gmail.com,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> >> Date: Thu, 15 Oct 2015 17:56:36 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > Right, for pointers to simple data types, like 'char', 'short', 'int',
> >> > etc., one needs to use Lisp_Save_Value.  For pointers to struct's, XLI
> >> > and XIL will do the job on the supported platforms.
> >> 
> >> No, it doesn't.
> >
> > It does in Emacs.
> 
> In which way?

In the Right Way.  The bits that need to be reset are reset.

Re: Dynamic loading progress

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@suse.de>
>> Cc: p.stephani2@gmail.com,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
>> Date: Thu, 15 Oct 2015 18:07:03 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Andreas Schwab <schwab@suse.de>
>> >> Cc: p.stephani2@gmail.com,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
>> >> Date: Thu, 15 Oct 2015 17:56:36 +0200
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> > Right, for pointers to simple data types, like 'char', 'short', 'int',
>> >> > etc., one needs to use Lisp_Save_Value.  For pointers to struct's, XLI
>> >> > and XIL will do the job on the supported platforms.
>> >> 
>> >> No, it doesn't.
>> >
>> > It does in Emacs.
>> 
>> In which way?
>
> In the Right Way.  The bits that need to be reset are reset.

This is not guaranteed.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: aurelien.aptel+emacs@gmail.com,  p.stephani2@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Thu, 15 Oct 2015 19:03:49 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@suse.de>
> >> Cc: p.stephani2@gmail.com,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> >> Date: Thu, 15 Oct 2015 18:07:03 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> From: Andreas Schwab <schwab@suse.de>
> >> >> Cc: p.stephani2@gmail.com,  aurelien.aptel+emacs@gmail.com,  dancol@dancol.org,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> >> >> Date: Thu, 15 Oct 2015 17:56:36 +0200
> >> >> 
> >> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> >> 
> >> >> > Right, for pointers to simple data types, like 'char', 'short', 'int',
> >> >> > etc., one needs to use Lisp_Save_Value.  For pointers to struct's, XLI
> >> >> > and XIL will do the job on the supported platforms.
> >> >> 
> >> >> No, it doesn't.
> >> >
> >> > It does in Emacs.
> >> 
> >> In which way?
> >
> > In the Right Way.  The bits that need to be reset are reset.
> 
> This is not guaranteed.

I never saw it fail to work on any modern platform.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Do., 15. Okt. 2015 um 04:44 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Wed, 14 Oct 2015 22:28:01 +0000
> > Cc: dancol@dancol.org, stephen_leake@stephe-leake.org,
> emacs-devel@gnu.org
> >
> >     > I'm not sure I understood what you wanted.. I've commited this,
> but it
> >     > assumes Lisp_Object are the same size as pointers...
> >
> >     That assumption is false in an Emacs built --with-wide-int.
> >
> >     If you need to wrap a pointer in a Lisp_Object, we have XIL and XLI
> >     macros for that.
> >
> >
> > These are no-ops in my version of lisp.h:
> >
> > # define lisp_h_XLI(o) (o)
> > # define lisp_h_XIL(i) (i)
> >
> > Unless I'm misunderstanding something, these macros can't be used to wrap
> > pointers in the general case.
>
> We do that all the time, so I don't understand why you understand
> that.  Please elaborate.
>
>
I got it backwards: we need to wrap Lisp_Objects in pointers, not the other
way round.

[-- Attachment #2: Type: text/html, Size: 1680 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 15 Oct 2015 23:07:41 +0000
> Cc: aurelien.aptel+emacs@gmail.com, dancol@dancol.org, 
> 	stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
>     > # define lisp_h_XLI(o) (o)
>     > # define lisp_h_XIL(i) (i)
>     >
>     > Unless I'm misunderstanding something, these macros can't be used to wrap
>     > pointers in the general case.
>     
>     We do that all the time, so I don't understand why you understand
>     that. Please elaborate.
> 
> I got it backwards: we need to wrap Lisp_Objects in pointers, not the other way
> round. 

That you cannot do portably.  So in that case, I suggest to use a
pointer to the Lisp object instead.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am So., 4. Okt.
2015 um 23:12 Uhr:

> On Sun, Oct 4, 2015 at 9:47 PM, Philipp Stephani <p.stephani2@gmail.com>
> wrote:
> > Aurélien, is that something you agree with and could implement?
>
> I'm not sure I understood what you wanted.. I've commited this, but it
> assumes Lisp_Object are the same size as pointers...
>
> commit 4c2813950d14fa2348e30ee94a4f3b022263e36d
> Author: Aurélien Aptel <aurelien.aptel@gmail.com>
> Date:   Sun Oct 4 23:04:56 2015 +0200
>
>     use opaque struct emacs_value_tag instead of void* for emacs_value.
>
> diff --git a/src/emacs_module.h b/src/emacs_module.h
> index c5ec347..b055547 100644
> --- a/src/emacs_module.h
> +++ b/src/emacs_module.h
> @@ -27,7 +27,7 @@
>
>  /* Current environement */
>  typedef struct emacs_env_25 emacs_env;
> -typedef void* emacs_value;
> +typedef struct emacs_value_tag* emacs_value;
>
>  enum emacs_type {
>    EMACS_FIXNUM,
> diff --git a/src/module.c b/src/module.c
> index 9bbb832..ab058bb 100644
> --- a/src/module.c
> +++ b/src/module.c
> @@ -24,6 +24,8 @@
>  #include "dynlib.h"
>  #include "coding.h"
>
> +struct emacs_value_tag { Lisp_Object v; };
> +
>  void syms_of_module (void);
>  static struct emacs_runtime* module_get_runtime (void);
>  static emacs_env* module_get_environment (struct emacs_runtime *ert);
>


This is good (thanks!), but still assumes that sizeof(Lisp_Object) <=
sizeof(void*) and that (Lisp_Object)((void *)0) is never a valid Lisp
object, both of which are not provably true. To implement Daniel's approach
you'll also have to change:

static Lisp_Object value_to_lisp (emacs_value value) {
  return value->v;
}

static emacs_value lisp_to_value (Lisp_Object object) {
  emacs_value v = (emacs_value) malloc(sizeof v);
  // The allocation needs to be smarter, this example leaks memory.
  if (!v) return 0;
  v->v = object;
  return v;
}

[-- Attachment #2: Type: text/html, Size: 2648 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Oct 15, 2015 at 12:25 AM, Philipp Stephani
<p.stephani2@gmail.com> wrote:
> static Lisp_Object value_to_lisp (emacs_value value) {
>   return value->v;
> }
>
> static emacs_value lisp_to_value (Lisp_Object object) {
>   emacs_value v = (emacs_value) malloc(sizeof v);
>   // The allocation needs to be smarter, this example leaks memory.
>   if (!v) return 0;
>   v->v = object;
>   return v;
> }

I'm not a big fan of a dynamically allocating memory for each
conversion, it's too expensive. Daniel solution seems more manageable.

By the way, I've merged your tls-errors branch in mine. But the API
doesn't have an unsafe funcall iiuc. That's fine with me but Stefan
was insistent on that.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Do., 15. Okt.
2015 um 01:48 Uhr:

> On Thu, Oct 15, 2015 at 12:25 AM, Philipp Stephani
> <p.stephani2@gmail.com> wrote:
> > static Lisp_Object value_to_lisp (emacs_value value) {
> >   return value->v;
> > }
> >
> > static emacs_value lisp_to_value (Lisp_Object object) {
> >   emacs_value v = (emacs_value) malloc(sizeof v);
> >   // The allocation needs to be smarter, this example leaks memory.
> >   if (!v) return 0;
> >   v->v = object;
> >   return v;
> > }
>
> I'm not a big fan of a dynamically allocating memory for each
> conversion, it's too expensive. Daniel solution seems more manageable.
>

Agreed. I've implemented Daniel's suggestion on the tls-error branch, it
seems to work fine (but note that environments are now no longer global, so
storing away emacs_values without global references now will lead to
undefined behavior).


>
> By the way, I've merged your tls-errors branch in mine.


Cool, thanks!


> But the API
> doesn't have an unsafe funcall iiuc. That's fine with me but Stefan
> was insistent on that.
>

Let's see what will get accepted upstream.

[-- Attachment #2: Type: text/html, Size: 1929 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Oct 15, 2015 at 2:25 AM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> Agreed. I've implemented Daniel's suggestion on the tls-error branch, it
> seems to work fine (but note that environments are now no longer global, so
> storing away emacs_values without global references now will lead to
> undefined behavior).

Yeah I just realized that. The nice thing with just casting is its the
same memory object and the stack-scanning GC just works. Having the GC
is nice...
Again, IIUC, the only problem is on 32bit with wide-int, which is off
by default.

> Let's see what will get accepted upstream.

That's sneaky... but this is how the free software world works I guess...

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Thu, 15 Oct 2015 12:44:00 +0200
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>, Eli Zaretskii <eliz@gnu.org>, 
> 	Stephen Leake <stephen_leake@stephe-leake.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> Again, IIUC, the only problem is on 32bit with wide-int, which is off
> by default.

Please don't assume that --with-wide-int is some rare marginal use
case.  I configure Emacs with it all the time, and intend to suggest
that 32-bit builds at least on MS-Windows always use it.

So I think we should have that use case covered, if we want the
dynamic loading hit the streets.

Thanks.

Re: Dynamic loading progress

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 347 bytes --]

On Thu, Oct 15, 2015 at 5:41 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Please don't assume that --with-wide-int is some rare marginal use
> case.  I configure Emacs with it all the time, and intend to suggest
> that 32-bit builds at least on MS-Windows always use it.

Why isn't it the default, then (for 32-bit builds)? Are there any downsides?

[-- Attachment #2: Type: text/html, Size: 482 bytes --]

Re: Dynamic loading progress

[martin rudalics]

 >> Please don't assume that --with-wide-int is some rare marginal use
 >> case.  I configure Emacs with it all the time, and intend to suggest
 >> that 32-bit builds at least on MS-Windows always use it.
 >
 > Why isn't it the default, then (for 32-bit builds)? Are there any downsides?

Here emacs.exe grows from 60 to 70 MB when configuring --with-wide-int.

martin

Re: Making --with-wide-int the default (was: Dynamic loading progress)

[Eli Zaretskii]

> Date: Fri, 16 Oct 2015 07:42:00 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>, 
>  p.stephani2@gmail.com, Daniel Colascione <dancol@dancol.org>, 
>  Stephen Leake <stephen_leake@stephe-leake.org>,
>  Emacs developers <emacs-devel@gnu.org>
> 
>  >> Please don't assume that --with-wide-int is some rare marginal use
>  >> case.  I configure Emacs with it all the time, and intend to suggest
>  >> that 32-bit builds at least on MS-Windows always use it.
>  >
>  > Why isn't it the default, then (for 32-bit builds)? Are there any downsides?
> 
> Here emacs.exe grows from 60 to 70 MB when configuring --with-wide-int.

You should compare stripped binaries, not unstripped ones.  85% of
those 70MB is DWARF debug info.

Re: Making --with-wide-int the default

[martin rudalics]

 >> Here emacs.exe grows from 60 to 70 MB when configuring --with-wide-int.
 >
 > You should compare stripped binaries, not unstripped ones.

A 15% increase for unstripped binaries strikes me as remarkable though.

 > 85% of
 > those 70MB is DWARF debug info.

Obviously.  Does that info grow by 8.5 MB when configuring
--with-wide-int?

martin

Re: Making --with-wide-int the default

[Eli Zaretskii]

> Date: Fri, 16 Oct 2015 09:34:24 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: lekktu@gmail.com, emacs-devel@gnu.org
> 
>  >> Here emacs.exe grows from 60 to 70 MB when configuring --with-wide-int.
>  >
>  > You should compare stripped binaries, not unstripped ones.
> 
> A 15% increase for unstripped binaries strikes me as remarkable though.

15% of 10MB is much less than 15% of 60MB.

>  > 85% of
>  > those 70MB is DWARF debug info.
> 
> Obviously.  Does that info grow by 8.5 MB when configuring
> --with-wide-int?

I don't know.  I never bothered to find out.  I generally consider the
size of the binary on disk uninteresting.

The size of the Emacs memory footprint after startup is 17MB vs 15MB
for the non-wide-int build.

Re: Making --with-wide-int the default (was: Dynamic loading progress)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 16 Oct 2015 02:55:47 +0200
> Cc: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>, 
> 	p.stephani2@gmail.com, Daniel Colascione <dancol@dancol.org>, 
> 	Stephen Leake <stephen_leake@stephe-leake.org>, Emacs developers <emacs-devel@gnu.org>
> 
> On Thu, Oct 15, 2015 at 5:41 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Please don't assume that --with-wide-int is some rare marginal use
> > case. I configure Emacs with it all the time, and intend to suggest
> > that 32-bit builds at least on MS-Windows always use it.
> 
> Why isn't it the default, then (for 32-bit builds)? Are there any downsides?

If there is a downside, I haven't found it yet (on MS-Windows).
There's some slowdown, but it all but disappears in an optimized
build.  And being able to have a 2GB buffer instead of 512MB maximum
is IMO worth that.  61-bit integer arithmetics is also nothing to
disregard easily.

To make it the default for all 32-bit builds, we should make sure it
works well on the other supported platforms.  Perhaps Paul could tell
which platforms he found this to work on, and then we could decide.

In any case, the time to make the decision is NOW, because this will
need time for us to be sure any serious bugs are fixed before Emacs
25.1 is released.

Re: Making --with-wide-int the default (was: Dynamic loading progress)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

On Fri, Oct 16, 2015 at 9:09 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> There's some slowdown, but it all but disappears in an optimized
> build.

Hmm, I just built with --with-wide-int, and I don't notice the slowdown,
perhaps because I also build with -O0 and --enable-checking=yes,glyphs, so
it's already a bit slow.

> In any case, the time to make the decision is NOW, because this will
> need time for us to be sure any serious bugs are fixed before Emacs
> 25.1 is released.

Then perhaps we should enable it in all 32-bit platforms and just wait for
the fallout. We can always fall back to the current default if things don't
look good.

[-- Attachment #2: Type: text/html, Size: 851 bytes --]

Re: Making --with-wide-int the default (was: Dynamic loading progress)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 16 Oct 2015 09:26:23 +0200
> Cc: Paul Eggert <eggert@cs.ucla.edu>, Emacs developers <emacs-devel@gnu.org>
> 
> > There's some slowdown, but it all but disappears in an optimized
> > build.
> 
> Hmm, I just built with --with-wide-int, and I don't notice the slowdown,
> perhaps because I also build with -O0 and --enable-checking=yes,glyphs, so it's
> already a bit slow.

You need to run measurable benchmarks, in order to see it.

> Then perhaps we should enable it in all 32-bit platforms and just wait for the
> fallout. We can always fall back to the current default if things don't look
> good.

That'd be my tendency, yes.

Re: Making --with-wide-int the default

[Paul Eggert]

Eli Zaretskii wrote:
> To make it the default for all 32-bit builds, we should make sure it
> works well on the other supported platforms.  Perhaps Paul could tell
> which platforms he found this to work on, and then we could decide.

--with-wide-int is not the default because Stefan was worried about its 
performance implications on older, slower 32-bit machines (such as Stefan's 
and/or RMS's laptops at the time, if I recall correctly). There shouldn't be any 
correctness problem with it; it's a performance issue.

I typically use 64-bit platforms nowadays, where the issue is moot. When I do 
use 32-bit platforms, I normally configure --with-wide-int. There is roughly a 
30% CPU hit and maybe a 60% hit on virtual memory, but it's worth it to me (I 
normally don't notice the difference).

Re: Making --with-wide-int the default

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 16 Oct 2015 01:03:15 -0700
> 
> Eli Zaretskii wrote:
> > To make it the default for all 32-bit builds, we should make sure it
> > works well on the other supported platforms.  Perhaps Paul could tell
> > which platforms he found this to work on, and then we could decide.
> 
> --with-wide-int is not the default because Stefan was worried about its 
> performance implications on older, slower 32-bit machines (such as Stefan's 
> and/or RMS's laptops at the time, if I recall correctly). There shouldn't be any 
> correctness problem with it; it's a performance issue.
> 
> I typically use 64-bit platforms nowadays, where the issue is moot. When I do 
> use 32-bit platforms, I normally configure --with-wide-int. There is roughly a 
> 30% CPU hit and maybe a 60% hit on virtual memory, but it's worth it to me (I 
> normally don't notice the difference).

Thanks, that about summarizes what I see here as well.

Would you say it's safe to make this the default for Emacs 25?

(If Richard reports annoying slowdown, we could suggest him to disable
that at configure time.  Or maybe there should be a configure-time
test for CPU speed?)

Re: Making --with-wide-int the default

[Paul Eggert]

Eli Zaretskii wrote:
> Would you say it's safe to make this the default for Emacs 25?

I'd say it's safe, yes.  --with-wide-int assumes 64-bit int but since we're 
requiring C99-or-later now this should be safe.  And as you say, if there are 
performance problems people can configure --with-wide-int=no.

I guess the main problem will be people configuring with --enable-checking and 
with CFLAGS=-g0 on old slow 32-bit machines.  However, people using non-default 
flags can just add --with-wide-int=no to their mix of flags, so this should be OK.

Re: Making --with-wide-int the default

[David Kastrup]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Eli Zaretskii wrote:
>> Would you say it's safe to make this the default for Emacs 25?
>
> I'd say it's safe, yes.  --with-wide-int assumes 64-bit int but since
> we're requiring C99-or-later now this should be safe.  And as you say,
> if there are performance problems people can configure
> --with-wide-int=no.

I don't it's sensible to configure a non-native default turning
everything into multiple-register operations and obliterating compact
data structures matching the architecture's choices.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  lekktu@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 10:31:32 +0200
> 
> I don't it's sensible to configure a non-native default turning
> everything into multiple-register operations and obliterating compact
> data structures matching the architecture's choices.

We only use 64-bit types in EMACS_INT, i.e. variables that reference
buffer and string positions.  All the rest is kept intact.

What is "non-native" about 'long long int'?

Do you really use 32-bit GNU/Linux machines?  I cannot even easily
find such beasts.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  lekktu@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 10:31:32 +0200
>> 
>> I don't it's sensible to configure a non-native default turning
>> everything into multiple-register operations and obliterating compact
>> data structures matching the architecture's choices.
>
> We only use 64-bit types in EMACS_INT, i.e. variables that reference
> buffer and string positions.  All the rest is kept intact.
>
> What is "non-native" about 'long long int'?
>
> Do you really use 32-bit GNU/Linux machines?  I cannot even easily
> find such beasts.

Uh, that's not hard.  I don't say that my CPU is incapable of going
64bit.  But the system is 32bit mainly for disk space reasons (SSD).
For a while, I even had a 64bit kernel running with a 32bit userland
(which allowed me to create 64-bit executables and test them as well as
the default 32-bit executables), but since this computer was given to me
as a replacement for a computer where the hardware gave up, I am stuck
to now having an NVidia card and the stupid binary-blob driver fails to
figure out how to compile in a 64-bit kernel when the userland is
32-bit.

So I am currently back again to a 32-bit kernel as well as to a 32-bit
system.

-- 
David Kastrup

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  lekktu@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 10:31:32 +0200
>> 
>> I don't it's sensible to configure a non-native default turning
>> everything into multiple-register operations and obliterating compact
>> data structures matching the architecture's choices.
>
> We only use 64-bit types in EMACS_INT, i.e. variables that reference
> buffer and string positions.  All the rest is kept intact.
>
> What is "non-native" about 'long long int'?

Register lengths and opcodes.  You need two registers for every
operation, and several opcodes.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: lekktu@gmail.com,  eggert@cs.ucla.edu,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 11:26:27 +0200
> 
> > What is "non-native" about 'long long int'?
> 
> Register lengths and opcodes.  You need two registers for every
> operation, and several opcodes.

Yes, and the compiler already implements that in the most efficient
way possible.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: lekktu@gmail.com,  eggert@cs.ucla.edu,  emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 11:26:27 +0200
>> 
>> > What is "non-native" about 'long long int'?
>> 
>> Register lengths and opcodes.  You need two registers for every
>> operation, and several opcodes.
>
> Yes, and the compiler already implements that in the most efficient
> way possible.

And GMP makes use of that.  "The most efficient way possible" is still
less efficient than using 32bit.  Particularly regarding storage
efficiency.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: lekktu@gmail.com,  eggert@cs.ucla.edu,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 12:28:10 +0200
> 
> >> > What is "non-native" about 'long long int'?
> >> 
> >> Register lengths and opcodes.  You need two registers for every
> >> operation, and several opcodes.
> >
> > Yes, and the compiler already implements that in the most efficient
> > way possible.
> 
> And GMP makes use of that.

But at the cost of a function call, which brings its own overhead.

> "The most efficient way possible" is still less efficient than using
> 32bit.

You cannot implement 32-bit buffer positions using a 32-bit C data
type.  You need a few more bits.  But you know this, so I have a
feeling we are talking past each other.

Re: Making --with-wide-int the default

[Paul Eggert]

David Kastrup wrote:
> I don't it's sensible to configure a non-native default turning
> everything into multiple-register operations and obliterating compact
> data structures matching the architecture's choices.

That's what I was worried about too, before I implemented --with-wide-int. But 
it turned out to not be a problem. Performance is a bit worse, but I have to 
measure it to see it. This can be surprising, until you try it.

For example, on 32-bit x86, the hot path (cdr of a cons) for the Fcdr function 
is 9 instructions with a narrow int, and 11 instructions --with-wide-int. Most 
of those 9 instructions are call overhead and bit-twiddling for runtime tests, 
and these are the same either way. --with-wide-int causes Fcdr to need one extra 
instruction to load the extra word of the argument, and one extra instruction to 
load the extra word of the result, and that's it.

If you're interested in squeezing out more performance on a --with-wide-int 
configuration, you can try the x32 ABI. E.g., see 
<https://wiki.debian.org/X32Port> for Debian or Ubuntu. I haven't bothered, 
though, as x86 is good enough and works everywhere.

Of course I'd rather have something with GMP and bignums. But that's 
considerably more work than --with-wide-int.

> a GMP number should be
> converted back to a native LISP integer whenever it's small enough
> again.

Obviously. And one can even work around the = vs eq problem for larger integers. 
But these are things that are still on the drawing board. --with-wide-int works now.

Re: Making --with-wide-int the default

[Eli Zaretskii]

> Cc: lekktu@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 16 Oct 2015 01:27:38 -0700
> 
> Eli Zaretskii wrote:
> > Would you say it's safe to make this the default for Emacs 25?
> 
> I'd say it's safe, yes.  --with-wide-int assumes 64-bit int but since we're 
> requiring C99-or-later now this should be safe.  And as you say, if there are 
> performance problems people can configure --with-wide-int=no.
> 
> I guess the main problem will be people configuring with --enable-checking and 
> with CFLAGS=-g0 on old slow 32-bit machines.  However, people using non-default 
> flags can just add --with-wide-int=no to their mix of flags, so this should be OK.

Based on this, I'd like to turn --with-wide-int on by default.  Any
last-minute objections?

Re: Making --with-wide-int the default

[martin rudalics]

 > Based on this, I'd like to turn --with-wide-int on by default.  Any
 > last-minute objections?

With a very visible description of how to turn it off, please.  Mine is
apparently still one of these "old slow 32-bit machines" :-(

martin

Re: Making --with-wide-int the default

[Eli Zaretskii]

> Date: Thu, 12 Nov 2015 09:23:02 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org
> 
>  > Based on this, I'd like to turn --with-wide-int on by default.  Any
>  > last-minute objections?
> 
> With a very visible description of how to turn it off, please.

I intended to document --without-wide-int in NEWS.  Is that visible
enough?

> Mine is apparently still one of these "old slow 32-bit machines" :-(

What kind of CPU do you have there?

Re: Making --with-wide-int the default

[martin rudalics]

 > I intended to document --without-wide-int in NEWS.  Is that visible
 > enough?

Let's hope so.

 >> Mine is apparently still one of these "old slow 32-bit machines" :-(
 >
 > What kind of CPU do you have there?

1.8 GHz Dual Pentium.

martin

Re: Making --with-wide-int the default

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >  > Based on this, I'd like to turn --with-wide-int on by default.  Any
  >  > last-minute objections?

  > With a very visible description of how to turn it off, please.  Mine is
  > apparently still one of these "old slow 32-bit machines" :-(

People who build Emacs should not _have_ to know anything about
--with-wide-int.  We should make configure automatically do the right
thing on machines that don't support wide ints.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, jwiegley@gmail.com, emacs-devel@gnu.org
> Date: Thu, 12 Nov 2015 17:31:51 -0500
> 
>   >  > Based on this, I'd like to turn --with-wide-int on by default.  Any
>   >  > last-minute objections?
> 
>   > With a very visible description of how to turn it off, please.  Mine is
>   > apparently still one of these "old slow 32-bit machines" :-(
> 
> People who build Emacs should not _have_ to know anything about
> --with-wide-int.  We should make configure automatically do the right
> thing on machines that don't support wide ints.

There are no machines we know of that cannot support wide ints.  The
only problem is that using wide ints produces a somewhat slower Emacs
(Paul reported a 30% slowdown), so on slow machines this could be
annoying.

However, the annoyance is in the eyes of the beholder, and the
decision whether the slowdown is worth the 4-fold increase in the size
of the largest buffer you can have (respectively, the largest file you
can edit) is something only humans can make.  E.g., many people here
routinely run Emacs configured with run-time checks and compiled
without optimizations, which makes it 2 - 3 times slower than the
optimized version.  So evidently a somewhat slower Emacs is mostly
bearable, at least to people on this list.

The Emacs build procedure provides about 40 --with-SOMETHING switch,
see "./configure --help".  The above-mentioned one is just one of
them.

Re: Making --with-wide-int the default

[Paul Eggert]

Eli Zaretskii wrote:
> There are no machines we know of that cannot support wide ints.

Yes, Emacs has assumed C99-or-better since May 2014 (see Bug#17487), and every 
such platform supports long long, used to implement wide ints on 32-bit hosts.

Re: Making --with-wide-int the default

[martin rudalics]

 > E.g., many people here
 > routinely run Emacs configured with run-time checks and compiled
 > without optimizations, which makes it 2 - 3 times slower than the
 > optimized version.

Unfortunately I can't do that any more since last year.  With run-time
checks and without optimization Emacs has become so slow that it was
virtually impossible to use it for editing.  Display of text I typed
lagged behind by up to three or four characters.

martin

Re: Making --with-wide-int the default

[Eli Zaretskii]

> Date: Fri, 13 Nov 2015 09:05:25 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org, jwiegley@gmail.com
> 
>  > E.g., many people here
>  > routinely run Emacs configured with run-time checks and compiled
>  > without optimizations, which makes it 2 - 3 times slower than the
>  > optimized version.
> 
> Unfortunately I can't do that any more since last year.  With run-time
> checks and without optimization Emacs has become so slow that it was
> virtually impossible to use it for editing.  Display of text I typed
> lagged behind by up to three or four characters.

My point is that the configure script cannot reasonably be expected to
find out that this is what will happen on a particular machine.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> CC: eliz@gnu.org, jwiegley@gmail.com, emacs-devel@gnu.org
>> Date: Thu, 12 Nov 2015 17:31:51 -0500
>> 
>>   >  > Based on this, I'd like to turn --with-wide-int on by default.  Any
>>   >  > last-minute objections?
>> 
>>   > With a very visible description of how to turn it off, please.  Mine is
>>   > apparently still one of these "old slow 32-bit machines" :-(
>> 
>> People who build Emacs should not _have_ to know anything about
>> --with-wide-int.  We should make configure automatically do the right
>> thing on machines that don't support wide ints.
>
> There are no machines we know of that cannot support wide ints.  The
> only problem is that using wide ints produces a somewhat slower Emacs
> (Paul reported a 30% slowdown), so on slow machines this could be
> annoying.

Also the cell size doubles, so the Lisp data structures take more
memory.  I think it is quite reasonable _not_ to use wide ints by
default on architectures with a 32-bit address space.  Everything takes
up more memory, the maximum of virtual address space is something like
3GB anyway so it's not like you could hope to increase the amount of
stuff you could be editing all that much: it is more likely that with
several buffers loaded at once, you'll _decrease_ the overall amount of
stuff you can keep loaded into buffers at the same time.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: rms@gnu.org,  rudalics@gmx.at,  emacs-devel@gnu.org,  jwiegley@gmail.com
> Date: Fri, 13 Nov 2015 10:11:49 +0100
> 
> Also the cell size doubles, so the Lisp data structures take more
> memory.  I think it is quite reasonable _not_ to use wide ints by
> default on architectures with a 32-bit address space.  Everything takes
> up more memory, the maximum of virtual address space is something like
> 3GB anyway so it's not like you could hope to increase the amount of
> stuff you could be editing all that much: it is more likely that with
> several buffers loaded at once, you'll _decrease_ the overall amount of
> stuff you can keep loaded into buffers at the same time.

I think the much large buffer size limit is more important than these
considerations.  (Your fears about decreasing the buffer size are
unfounded: I actually tried using 1.5GB buffers, and it worked every
time).  And there's a configure-time switch to not use wide ints for
those who think otherwise.

So on balance, I think it's an improvement.  If I'm wrong, we have the
pretest to indicate that, and we can always turn it back off by
default if needed before the release.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: rms@gnu.org,  rudalics@gmx.at,  emacs-devel@gnu.org,  jwiegley@gmail.com
>> Date: Fri, 13 Nov 2015 10:11:49 +0100
>> 
>> Also the cell size doubles, so the Lisp data structures take more
>> memory.  I think it is quite reasonable _not_ to use wide ints by
>> default on architectures with a 32-bit address space.  Everything takes
>> up more memory, the maximum of virtual address space is something like
>> 3GB anyway so it's not like you could hope to increase the amount of
>> stuff you could be editing all that much: it is more likely that with
>> several buffers loaded at once, you'll _decrease_ the overall amount of
>> stuff you can keep loaded into buffers at the same time.
>
> I think the much large buffer size limit is more important than these
> considerations.  (Your fears about decreasing the buffer size are
> unfounded: I actually tried using 1.5GB buffers, and it worked every
> time).

"with several buffers loaded at once".  If buffer size was so much of a
concern, we could represent integers like XEmacs does (basically with a
single tag bit), allowing for buffers of 1GB size.  When the maximal
virtual memory size is something like 3GB, I don't think we'd be missing
out on much.  People wanting to edit files of several gigabyte size
regularly will not be using a 32-bit userland these days anyway.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: rudalics@gmx.at,  jwiegley@gmail.com,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 13 Nov 2015 12:52:30 +0100
> 
> > I think the much large buffer size limit is more important than these
> > considerations.  (Your fears about decreasing the buffer size are
> > unfounded: I actually tried using 1.5GB buffers, and it worked every
> > time).
> 
> "with several buffers loaded at once".

Yes, with several buffers.

> If buffer size was so much of a concern, we could represent integers
> like XEmacs does (basically with a single tag bit), allowing for
> buffers of 1GB size.

We could, but we didn't.  Instead, we have --with-wide-ints.

> People wanting to edit files of several gigabyte size regularly will
> not be using a 32-bit userland these days anyway.

If you need to do that on a 32-bit system, you have no other choice.

Re: Making --with-wide-int the default

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   I think it is quite reasonable _not_ to use wide ints by
  > default on architectures with a 32-bit address space.

+1.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, rudalics@gmx.at, jwiegley@gmail.com,
> 	emacs-devel@gnu.org
> Date: Fri, 13 Nov 2015 17:03:10 -0500
> 
>   >   I think it is quite reasonable _not_ to use wide ints by
>   > default on architectures with a 32-bit address space.
> 
> +1.

Faced with such powerful opposition, I guess this now becomes your
decision, John.  I can always build my own Emacs with that option; the
question is what we want for the majority of our users on 32-bit
systems.

Re: Making --with-wide-int the default

[martin rudalics]

 > Faced with such powerful opposition, I guess this now becomes your
 > decision, John.  I can always build my own Emacs with that option; the
 > question is what we want for the majority of our users on 32-bit
 > systems.

Please note that I'm not part of that opposition.  I just wanted to
mention that my current configuration leaves me no other choice than to
build without that option.  And if it were the first time I ran into the
problem that my build is unbearably slow, I'd like to have some guidance
on how to possibly get out of it.

martin

Re: Making --with-wide-int the default

[Ulrich Mueller]

>>>>> On Sat, 14 Nov 2015, Eli Zaretskii wrote:

> Faced with such powerful opposition, I guess this now becomes your
> decision, John.  I can always build my own Emacs with that option; the
> question is what we want for the majority of our users on 32-bit
> systems.

In case you want any feedback from distros, Gentoo makes the option
available to users as the "wide-int" use flag, and the default is off.
I've not seen a single complaint from users that we should change that
default. (So most likely the Gentoo default will stay off, regardless
of what you decide to do with the upstream default.)

Ulrich

Re: Making --with-wide-int the default

[Eli Zaretskii]

> Date: Sat, 14 Nov 2015 18:38:37 +0100
> From: Ulrich Mueller <ulm@gentoo.org>
> Cc: rudalics@gmx.at, jwiegley@gmail.com, dak@gnu.org, rms@gnu.org,
> 	emacs-devel@gnu.org
> 
> In case you want any feedback from distros, Gentoo makes the option
> available to users as the "wide-int" use flag, and the default is off.
> I've not seen a single complaint from users that we should change that
> default. (So most likely the Gentoo default will stay off, regardless
> of what you decide to do with the upstream default.)

Did the option you offer mention the fact that using it enlarges the
maximum buffer and string size to (almost) 2GB?  If not, it's quite
possible that your users simply did not realize what this option would
give them in user-level functionality, and treated it as yet another
obscure build feature.

Also, I must say it sounds strange to me that you wait for user
complaints before you decide that some option should be on by default.
Do you act the same with all the optional Emacs features, like GnuTLS
or libxml2 or file notifications (or even image support)?  Surely, you
and your team have enough insight on the importance of each feature to
make your own decisions?  But I know nothing about maintaining
distributions, so perhaps I'm missing something here.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 14 Nov 2015 18:38:37 +0100
>> From: Ulrich Mueller <ulm@gentoo.org>
>> Cc: rudalics@gmx.at, jwiegley@gmail.com, dak@gnu.org, rms@gnu.org,
>> 	emacs-devel@gnu.org
>> 
>> In case you want any feedback from distros, Gentoo makes the option
>> available to users as the "wide-int" use flag, and the default is off.
>> I've not seen a single complaint from users that we should change that
>> default. (So most likely the Gentoo default will stay off, regardless
>> of what you decide to do with the upstream default.)
>
> Did the option you offer mention the fact that using it enlarges the
> maximum buffer and string size to (almost) 2GB?  If not, it's quite
> possible that your users simply did not realize what this option would
> give them in user-level functionality, and treated it as yet another
> obscure build feature.
>
> Also, I must say it sounds strange to me that you wait for user
> complaints before you decide that some option should be on by default.

A performance hit by 30% and noticeable increase of memory usage are not
exactly a bargain for being able to load ridiculously large files into
an editor on a 32-bit system.

If people had wagonloads of memory to spare, they'd be running 64-bit
systems in the first place.

> Do you act the same with all the optional Emacs features, like GnuTLS
> or libxml2 or file notifications (or even image support)?  Surely, you
> and your team have enough insight on the importance of each feature to
> make your own decisions?  But I know nothing about maintaining
> distributions, so perhaps I'm missing something here.

Gentoo _also_ is about squeezing the most performance from your hardware
by compiling with system-specific optimizations.  A 30% performance hit
for your text desktop affecting every editing task does not exactly fit
well with that objective.

I really don't get what the clamor for wide ints on a 32 bit system is
supposed to be about.  It's rather few people that you are doing an
actual favor here.  Stuff like GnuTLS or libxml2 or file notifications
or image support don't bog your system down when you don't use them.
Wide ints do.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Ulrich Mueller <ulm@gentoo.org>,  rudalics@gmx.at,  jwiegley@gmail.com,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 15 Nov 2015 21:50:27 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Date: Sat, 14 Nov 2015 18:38:37 +0100
> >> From: Ulrich Mueller <ulm@gentoo.org>
> >> Cc: rudalics@gmx.at, jwiegley@gmail.com, dak@gnu.org, rms@gnu.org,
> >> 	emacs-devel@gnu.org
> >> 
> >> In case you want any feedback from distros, Gentoo makes the option
> >> available to users as the "wide-int" use flag, and the default is off.
> >> I've not seen a single complaint from users that we should change that
> >> default. (So most likely the Gentoo default will stay off, regardless
> >> of what you decide to do with the upstream default.)
> >
> > Did the option you offer mention the fact that using it enlarges the
> > maximum buffer and string size to (almost) 2GB?  If not, it's quite
> > possible that your users simply did not realize what this option would
> > give them in user-level functionality, and treated it as yet another
> > obscure build feature.
> >
> > Also, I must say it sounds strange to me that you wait for user
> > complaints before you decide that some option should be on by default.
> 
> A performance hit by 30% and noticeable increase of memory usage are not
> exactly a bargain for being able to load ridiculously large files into
> an editor on a 32-bit system.
> 
> If people had wagonloads of memory to spare, they'd be running 64-bit
> systems in the first place.

How is this related to what I wrote, may I ask?  Ulrich never
mentioned these factors, and I replied to what he wrote.

> I really don't get what the clamor for wide ints on a 32 bit system is
> supposed to be about.  It's rather few people that you are doing an
> actual favor here.  Stuff like GnuTLS or libxml2 or file notifications
> or image support don't bog your system down when you don't use them.
> Wide ints do.

Once again, how is that relevant to what I wrote in my message to
Ulrich?

Look, it's clear that you are against this option.  You made that
clear several times already; repeating it time and again doesn't add
weight to your opinions.  Especially when those opinions are plugged
with no relation whatsoever to what I wrote.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Ulrich Mueller <ulm@gentoo.org>, rudalics@gmx.at,
>> jwiegley@gmail.com, rms@gnu.org, emacs-devel@gnu.org
>> Date: Sun, 15 Nov 2015 21:50:27 +0100

>> 
>> A performance hit by 30% and noticeable increase of memory usage are not
>> exactly a bargain for being able to load ridiculously large files into
>> an editor on a 32-bit system.
>> 
>> If people had wagonloads of memory to spare, they'd be running 64-bit
>> systems in the first place.
>
> How is this related to what I wrote, may I ask?  Ulrich never
> mentioned these factors, and I replied to what he wrote.

So?  You attack Ulrich because he did not sufficiently stress the
advantages:

>> > Did the option you offer mention the fact that using it enlarges
>> > the maximum buffer and string size to (almost) 2GB?  If not, it's
>> > quite possible that your users simply did not realize what this
>> > option would give them in user-level functionality, and treated it
>> > as yet another obscure build feature.
>> >
>> > Also, I must say it sounds strange to me that you wait for user
>> > complaints before you decide that some option should be on by
>> > default.

and more or less state that apparently the only reason for nobody
complaining about that setting is that he has omitted to properly
describe the option's advantages, not because there would be any actual
justification for the chosen default.  At the same time you are
completely unworried about the _disadvantages_ (which affect every user
editing any file, not just the ones loading files larger than 512MByte
into memory) not getting mentioned at all.

What's wrong with an actual qualified choice based on knowledge of both
advantages and disadvantages?

> Once again, how is that relevant to what I wrote in my message to
> Ulrich?

Because you are only interested in having the advantages of wide ints
listed.

> Look, it's clear that you are against this option.

Not at all.  I am against enabling it by default on unsuspecting users
of 32bit systems.  I am fine with enabling it by default on 64bit
systems since on those, the gained window of usefulness between 512MB
buffers and a reasonable fraction of the virtual memory available to
Emacs is potentially quite larger.  Also the performance cost of using
64bit entities for everything should be quite less noticeable.  I am
also fine with making this option an announced and properly described
user option to users of a compile-your-own system like Gentoo.

If you want a detailed mention of the reason for setting such an option,
it does not make sense to omit the drawbacks of doing so: that seems
only tailored towards having users make complaints (the absence of which
you do not like) rather than an actual informed choice.

The desired complaints would likely have the form "why is this option
not enabled by default?" and of course there is a valid answer for that
as well as a valid answer for why some user would want it enabled by
default.

> You made that clear several times already; repeating it time and again
> doesn't add weight to your opinions.  Especially when those opinions
> are plugged with no relation whatsoever to what I wrote.

Shrug.  You don't see a relation, I see one.  Others are reading this
exchange, so there is a chance that it contributes to forming opinions
and eventually may help with making and explaining decisions.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: ulm@gentoo.org,  rudalics@gmx.at,  jwiegley@gmail.com,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 15 Nov 2015 23:19:00 +0100
> 
> > How is this related to what I wrote, may I ask?  Ulrich never
> > mentioned these factors, and I replied to what he wrote.
> 
> So?  You attack Ulrich because he did not sufficiently stress the
> advantages:

If Ulrich feels attacked, I apologize: I never meant anything even
close.  He said he had another data point related to this decision, so
I thought it was fair to ask the questions that I asked about that
data point.

> and more or less state that apparently the only reason for nobody
> complaining about that setting is that he has omitted to properly
> describe the option's advantages, not because there would be any actual
> justification for the chosen default.

That's not what I said.  What I actually said was this:

> Fair enough, but then we must agree that the lack of complaints
> probably says nothing at all about your users' preferences in this
> matter.

IOW, "the data doesn't support any conclusions".  No accusations, no
attack, nothing of the kind.

> At the same time you are
> completely unworried about the _disadvantages_ (which affect every user
> editing any file, not just the ones loading files larger than 512MByte
> into memory) not getting mentioned at all.

IMO, the advantages far outweigh the disadvantages.  In your opinion,
it's the other way around.  Nothing wrong here, and no reasons to
imply that my opinions are somehow inferior to yours.  they are just
different.

> What's wrong with an actual qualified choice based on knowledge of both
> advantages and disadvantages?

Nothing.  But when you reply to what I wrote, I'd appreciate that you
actually reply to the quoted parts, or else don't quote me at all.

> > Once again, how is that relevant to what I wrote in my message to
> > Ulrich?
> 
> Because you are only interested in having the advantages of wide ints
> listed.

??? I think now you owe me an apology.

> Others are reading this exchange, so there is a chance that it
> contributes to forming opinions and eventually may help with making
> and explaining decisions.

Indeed.

Re: Making --with-wide-int the default

[Ulrich Mueller]

>>>>> On Sun, 15 Nov 2015, Eli Zaretskii wrote:

>> Date: Sat, 14 Nov 2015 18:38:37 +0100
>> From: Ulrich Mueller <ulm@gentoo.org>
>> Cc: rudalics@gmx.at, jwiegley@gmail.com, dak@gnu.org, rms@gnu.org,
>> emacs-devel@gnu.org
>> 
>> In case you want any feedback from distros, Gentoo makes the option
>> available to users as the "wide-int" use flag, and the default is off.
>> I've not seen a single complaint from users that we should change that
>> default. (So most likely the Gentoo default will stay off, regardless
>> of what you decide to do with the upstream default.)

> Did the option you offer mention the fact that using it enlarges the
> maximum buffer and string size to (almost) 2GB? If not, it's quite
> possible that your users simply did not realize what this option would
> give them in user-level functionality, and treated it as yet another
> obscure build feature.

This is what we have as description:

   app-editors/emacs:wide-int - Prefer wide Emacs integers (typically
   62-bit). This option has an effect only on architectures where
   "long" and "long long" types have different size.

Seems that we copied the first sentence from Emacs' configure --help
output, which also doesn't say anything about buffer size or memory
footprint.

> Also, I must say it sounds strange to me that you wait for user
> complaints before you decide that some option should be on by default.

Well, Gentoo may be a bit special there, as we leave the choice
largely to the users. There are of course defaults (set in so-called
"profiles") for the most important flags. But users _will_ complain if
a default is completely unreasonable.

Unrelated question: Are the "62 bit" in the description above correct,
or should it rather be 61 bit?

Ulrich

Re: Making --with-wide-int the default

[Eli Zaretskii]

> Date: Sun, 15 Nov 2015 22:04:46 +0100
> Cc: rudalics@gmx.at, jwiegley@gmail.com, dak@gnu.org, rms@gnu.org,
>         emacs-devel@gnu.org
> From: Ulrich Mueller <ulm@gentoo.org>
> 
> >>>>> On Sun, 15 Nov 2015, Eli Zaretskii wrote:
> 
> >> Date: Sat, 14 Nov 2015 18:38:37 +0100
> >> From: Ulrich Mueller <ulm@gentoo.org>
> >> Cc: rudalics@gmx.at, jwiegley@gmail.com, dak@gnu.org, rms@gnu.org,
> >> emacs-devel@gnu.org
> >> 
> >> In case you want any feedback from distros, Gentoo makes the option
> >> available to users as the "wide-int" use flag, and the default is off.
> >> I've not seen a single complaint from users that we should change that
> >> default. (So most likely the Gentoo default will stay off, regardless
> >> of what you decide to do with the upstream default.)
> 
> > Did the option you offer mention the fact that using it enlarges the
> > maximum buffer and string size to (almost) 2GB? If not, it's quite
> > possible that your users simply did not realize what this option would
> > give them in user-level functionality, and treated it as yet another
> > obscure build feature.
> 
> This is what we have as description:
> 
>    app-editors/emacs:wide-int - Prefer wide Emacs integers (typically
>    62-bit). This option has an effect only on architectures where
>    "long" and "long long" types have different size.
> 
> Seems that we copied the first sentence from Emacs' configure --help
> output, which also doesn't say anything about buffer size or memory
> footprint.

Fair enough, but then we must agree that the lack of complaints
probably says nothing at all about your users' preferences in this
matter.

> Unrelated question: Are the "62 bit" in the description above correct,
> or should it rather be 61 bit?

62, including the sign bit.  Try this:

  M-: (- most-positive-fixnum) RET

and then look at the hex representation.

Re: Making --with-wide-int the default

[David Kastrup]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Sun, 15 Nov 2015, Eli Zaretskii wrote:
>
>>> Date: Sat, 14 Nov 2015 18:38:37 +0100
>>> From: Ulrich Mueller <ulm@gentoo.org>
>>> Cc: rudalics@gmx.at, jwiegley@gmail.com, dak@gnu.org, rms@gnu.org,
>>> emacs-devel@gnu.org
>>> 
>>> In case you want any feedback from distros, Gentoo makes the option
>>> available to users as the "wide-int" use flag, and the default is off.
>>> I've not seen a single complaint from users that we should change that
>>> default. (So most likely the Gentoo default will stay off, regardless
>>> of what you decide to do with the upstream default.)
>
>> Did the option you offer mention the fact that using it enlarges the
>> maximum buffer and string size to (almost) 2GB? If not, it's quite
>> possible that your users simply did not realize what this option would
>> give them in user-level functionality, and treated it as yet another
>> obscure build feature.
>
> This is what we have as description:
>
>    app-editors/emacs:wide-int - Prefer wide Emacs integers (typically
>    62-bit). This option has an effect only on architectures where
>    "long" and "long long" types have different size.
>
> Seems that we copied the first sentence from Emacs' configure --help
> output, which also doesn't say anything about buffer size or memory
> footprint.

It's not really "Prefer".  It makes _all_ Lisp data 64 bit (allowing for
62 bit integers instead of the usual 30 bit ones), at the obvious cost
of doubling the size of Lisp cells (and immediate expressions and
symbols and a few other things) to a value not natively supported by
instructions and registers.

> Unrelated question: Are the "62 bit" in the description above correct,
> or should it rather be 61 bit?

Yes.  The basic type tagging is 3 bits, but two tag values are reserved
for integers.  So effectively only 2 bits are missing from the integer
range.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Paul Eggert]

Eli Zaretskii wrote:
> I can always build my own Emacs with that option

When I build and test 32-bit code, I typically use --with-wide-int as well.

My impression is that --with-wide-int is typically better for production use 
these days.  For older, slower platforms and with debugging turned on and/or 
optimization turned off, --with-wide-int may well cost more than it's worth. 
That being said, typical 32-bit use nowadays would probably benefit from 
--with-wide-int.

Re: Making --with-wide-int the default

[David Kastrup]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Eli Zaretskii wrote:
>> I can always build my own Emacs with that option
>
> When I build and test 32-bit code, I typically use --with-wide-int as well.
>
> My impression is that --with-wide-int is typically better for
> production use these days.  For older, slower platforms and with
> debugging turned on and/or optimization turned off, --with-wide-int
> may well cost more than it's worth. That being said, typical 32-bit
> use nowadays would probably benefit from --with-wide-int.

At the risk of repeating myself: newer, faster platforms tend to run
64-bit systems anyway.  Running a 32-bit system on new hardware these
days implies making a _choice_ for such systems (as the hardware for
memory sizes in excess of 1GB is rather consistently prepared for
running 64-bit systems), for smaller code size and memory and disk
impact at the cost of not being able to address humongous amounts of
data in virtual memory.  The rationale for such a choice is quite
related to the rationale for not using wide Lisp data structures in a
compilation of Emacs.

A sensible system default becomes more important as Emacs gains the
ability of loading modules compiled with C and accessing Elisp data
structures at runtime because, as opposed to options like
--without-toolkit-scroll-bars, "just compile your own binaries if you
don't like the default" stops being an option when you want to make use
of precompiled binary modules.

It's nice that --with-wide-int is available for special needs.  But on
32-bit systems, it comes at a cost that is not consistent with choosing
a 32-bit system in the first place.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 16 Nov 2015 10:00:34 +0100
> 
> It's nice that --with-wide-int is available for special needs.  But on
> 32-bit systems, it comes at a cost that is not consistent with choosing
> a 32-bit system in the first place.

FWIW, in my testing, the cost is mostly insignificant in any real-life
session.  I found Paul's estimates of 30% slowdown to be rather
conservative, I can only get around 30% - 35% in a tight loop that
increments a single integer variable and does nothing else.  In any
real-time scenario I tried, I see between 4% and 13% slowdown.  And
memory footprint of a large session (more than 500 buffers) is less
than 5% larger with wide ints.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> CC: eliz@gnu.org, rudalics@gmx.at, jwiegley@gmail.com,
>> 	emacs-devel@gnu.org
>> Date: Fri, 13 Nov 2015 17:03:10 -0500
>> 
>>   >   I think it is quite reasonable _not_ to use wide ints by
>>   > default on architectures with a 32-bit address space.
>> 
>> +1.
>
> Faced with such powerful opposition, I guess this now becomes your
> decision, John.  I can always build my own Emacs with that option; the
> question is what we want for the majority of our users on 32-bit
> systems.

Here is another data point: the code base seems woefully unprepared to
actually deal with the resulting ranges.

Lisp integers are only converted with macros XINT and XFASTINT as far as
I can tell, and

git grep XINT

shows lots of assignments to variables of type int rather than
EMACS_INT.  I think that there should be macros XINT, XUINT, XEINT (for
EMACS_INT) which trigger a range error when the value does not fit the
respective range.  Possibly also XLONG and XULONG but I'm not sure we
actually have platforms where long/int are different.

Some of those uses may well be security problems.  Using EMACS_INT
everywhere might be inappropriate (where's the point in having array
indexes longer than the addressable memory space?) but then we should
likely get range errors.

Of course, doing this properly might also mean less work for moving to
arbitrary-size integers eventually (probably using the Lisp_Int1 tag
when the 28-bit range does not suffice), though that one will likely
need a lot of converting `eq' -> `eql' in the Lisp code base.

But first we should get wide ints behave sensibly when their range
exceeds the expected target range.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: rms@gnu.org,  rudalics@gmx.at,  emacs-devel@gnu.org,  jwiegley@gmail.com
> Date: Sun, 15 Nov 2015 17:01:40 +0100
> 
> Here is another data point: the code base seems woefully unprepared to
> actually deal with the resulting ranges.
> 
> Lisp integers are only converted with macros XINT and XFASTINT as far as
> I can tell, and
> 
> git grep XINT
> 
> shows lots of assignments to variables of type int rather than
> EMACS_INT.

If these are real problems, they should have bitten us long ago, in
the 64-bit builds, no?

There were, indeed, such problems, which prevented using buffers
larger than 2GB, but they were solved long ago, when a significant
portion of our users moved to 64-bit machines.

> I think that there should be macros XINT, XUINT, XEINT (for
> EMACS_INT) which trigger a range error when the value does not fit the
> respective range.

Could be a valuable debugging aid, I think.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: rms@gnu.org,  rudalics@gmx.at,  emacs-devel@gnu.org,  jwiegley@gmail.com
>> Date: Sun, 15 Nov 2015 17:01:40 +0100
>> 
>> Here is another data point: the code base seems woefully unprepared to
>> actually deal with the resulting ranges.
>> 
>> Lisp integers are only converted with macros XINT and XFASTINT as far as
>> I can tell, and
>> 
>> git grep XINT
>> 
>> shows lots of assignments to variables of type int rather than
>> EMACS_INT.
>
> If these are real problems, they should have bitten us long ago, in
> the 64-bit builds, no?

Why?  They'll only trigger for cases that exceed a range of 32 bits, and
if those cases were other than rare, they'd have rendered the 32-bit
builds (which work with 30-bit precision) unusable.

However, we never previously had the case that a number which looks fine
in Lisp does not transfer into C properly.

> There were, indeed, such problems, which prevented using buffers
> larger than 2GB, but they were solved long ago, when a significant
> portion of our users moved to 64-bit machines.
>
>> I think that there should be macros XINT, XUINT, XEINT (for
>> EMACS_INT) which trigger a range error when the value does not fit
>> the respective range.
>
> Could be a valuable debugging aid, I think.

Well, it's probably worth checking what Guilemacs' idea about XINT is
since Guile _does_ seamlessly degrade into arbitrary precision integers
when exceeding integer precision.  So much of the work for supporting
arbitrary integers (and converting from Guile/Elisp integers to C
integers and back) should probably be in the Guilemacs branch already
and there'd be little point in reinventing it separately.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: rms@gnu.org,  rudalics@gmx.at,  emacs-devel@gnu.org,  jwiegley@gmail.com
> Date: Sun, 15 Nov 2015 21:42:24 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: David Kastrup <dak@gnu.org>
> >> Cc: rms@gnu.org,  rudalics@gmx.at,  emacs-devel@gnu.org,  jwiegley@gmail.com
> >> Date: Sun, 15 Nov 2015 17:01:40 +0100
> >> 
> >> Here is another data point: the code base seems woefully unprepared to
> >> actually deal with the resulting ranges.
> >> 
> >> Lisp integers are only converted with macros XINT and XFASTINT as far as
> >> I can tell, and
> >> 
> >> git grep XINT
> >> 
> >> shows lots of assignments to variables of type int rather than
> >> EMACS_INT.
> >
> > If these are real problems, they should have bitten us long ago, in
> > the 64-bit builds, no?
> 
> Why?

Because an int is a 32-bit data type in 64-bit builds as well.  So if
we use XINT to assign to an int, and the value exceeds INT_MAX, the
same problems will happen in both 64-bit builds and in 32-bit builds
with wide ints.

> They'll only trigger for cases that exceed a range of 32 bits, and
> if those cases were other than rare, they'd have rendered the 32-bit
> builds (which work with 30-bit precision) unusable.

The same considerations apply to 64-bit builds.

My point is that these issues, if they exist, are not unique to the
32-bit builds with wide ints, they are with us even without wide ints.

> However, we never previously had the case that a number which looks fine
> in Lisp does not transfer into C properly.

Once again, if we assign an Emacs integer to a C int, we will have the
same problems in 64-bit builds.

Btw, it's not like we live dangerously there: these issues were
extensively scrutinized several years ago, when we lifted the 2GB
limit on buffers in 64-bit builds, and all the relevant code was
carefully audited and fixed.  It is, of course, possible that some new
problems crept in, but the basis is sound, both for the 64-bit build
and the 32-bit build with wide ints.

Bottom line: I don't think there are new problems the 32-bit build
with wide ints introduces that aren't with us in the 64-bit builds.

Re: Making --with-wide-int the default

[Paul Eggert]

On 11/15/2015 08:01 AM, David Kastrup wrote:
> I can tell, and
>
> git grep XINT
>
> shows lots of assignments to variables of type int rather than
> EMACS_INT.  I think that there should be macros XINT, XUINT, XEINT (for
> EMACS_INT) which trigger a range error when the value does not fit the
> respective range.  Possibly also XLONG and XULONG but I'm not sure we
> actually have platforms where long/int are different.

There's no need XEINT etc., as this work has already been done, using 
macros like CHECK_TYPE_RANGED_INTEGER to make sure that XINT returns 
values in range for a C integer type.  Also, I sometimes build Emacs 
with C integer overflow checking enabled, and it runs fine.

Really, integer overflow is not a significant problem with 
--with-wide-int.  As Eli says, if it were a problem it'd also be a 
problem with 64-bit platforms, which it's not.

Re: Making --with-wide-int the default

[Random832]

Paul Eggert <eggert@cs.ucla.edu> writes:
> Really, integer overflow is not a significant problem with
> --with-wide-int.  As Eli says, if it were a problem it'd also be a
> problem with 64-bit platforms, which it's not.

I think the issue some people are bringing up is that the concrete cases
that would actually trigger the bugs, if there are any (very large
files, very large strings, very large buffers) are rare even on 64-bit
machines. Sure, it's nice to be _able_ to edit 2GB files, but how many
of those does the average user have lying around?

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Mon, 16 Nov 2015 20:34:39 -0500
> 
> Paul Eggert <eggert@cs.ucla.edu> writes:
> > Really, integer overflow is not a significant problem with
> > --with-wide-int.  As Eli says, if it were a problem it'd also be a
> > problem with 64-bit platforms, which it's not.
> 
> I think the issue some people are bringing up is that the concrete cases
> that would actually trigger the bugs, if there are any (very large
> files, very large strings, very large buffers) are rare even on 64-bit
> machines.

They are rare, but they have been tested a few years ago, and the bugs
found fixed.  The people who did that specifically made a point of
editing such large files for the purposes of finding and fixing any
such bugs.

> Sure, it's nice to be _able_ to edit 2GB files, but how many
> of those does the average user have lying around?

Those who need to look at large log files have to do this every day.

Re: Making --with-wide-int the default

[Paul Eggert]

Random832 wrote:
> Sure, it's nice to be_able_  to edit 2GB files, but how many
> of those does the average user have lying around?

Without --with-wide-int the limit is roughly 0.5 GB, not 2 GB. Whether this is a 
problem depends on what sorts of files one typically looks at. These days a 
goodly number of users do run into larger files. Certainly I do -- usually log 
or data files.

Although I try to keep these files less than 0.5 GB, this can be a pain. And 
although I can switch to some other text editor to look at an 0.6 GB file, 
that's awkward, and gives users a bad impression about Emacs's robustness and 
ease of use.

The GNU Coding Standards give advice about whether 32-bit Emacs should default 
to mishandling 0.6 GB files. They say, "Avoid arbitrary limits on the length or 
number of _any_ data structure" (emphasis in original). Emacs buffers are a data 
structure, and --with-wide-int removes a significant and arbitrary limit on 
their length, so the path forward is reasonably clear here.

Re: Making --with-wide-int the default

[Ulrich Mueller]

>>>>> On Mon, 16 Nov 2015, Paul Eggert wrote:

> The GNU Coding Standards give advice about whether 32-bit Emacs
> should default to mishandling 0.6 GB files. They say, "Avoid
> arbitrary limits on the length or number of _any_ data structure"
> (emphasis in original).

But this is about avoiding static allocation. The complete sentence
reads: "Avoid arbitrary limits on the length or number of /any/ data
structure, including file names, lines, files, and symbols, by
allocating all data structures dynamically."

> Emacs buffers are a data structure, and --with-wide-int removes a
> significant and arbitrary limit on their length, so the path forward
> is reasonably clear here.

A limit imposed by the machine's word size doesn't seem so arbitrary,
if it can only be surpassed with a significant performance penalty.

Ulrich

Re: Making --with-wide-int the default

[Paul Eggert]

On 11/17/2015 01:08 AM, Ulrich Mueller wrote:
> A limit imposed by the machine's word size doesn't seem so arbitrary

Yes, quite so, and that's what --with-wide-int does: it changes Emacs so 
that buffer sizes are limited by the machine word size, instead of being 
limited by some arbitrary truncation of that word size, a truncation not 
imposed by the machine.

> if it can only be surpassed with a significant performance penalty

That's too strict and we have never been that strict.  On the contrary 
we regularly make changes to Emacs that impose a significant penalty -- 
i.e., a penalty that can be measured in a statistically significant way. 
Even the GNU Coding Standard's guideline about allocating data 
dynamically can impose a significant performance penalty compared to 
static allocation, in many cases. But that's OK. We can impose a small 
(albeit significant) performance penalty if this removes an important 
arbitrary limit on a data structure size.

Re: Making --with-wide-int the default

[Ulrich Mueller]

>>>>> On Tue, 17 Nov 2015, Paul Eggert wrote:

> On 11/17/2015 01:08 AM, Ulrich Mueller wrote:
>> A limit imposed by the machine's word size doesn't seem so arbitrary

> Yes, quite so, and that's what --with-wide-int does: it changes Emacs so 
> that buffer sizes are limited by the machine word size, instead of being 
> limited by some arbitrary truncation of that word size, a truncation not 
> imposed by the machine.

The maximum sizes aren't much increased though, as the limit of
process address space will be hit soon after. Which for a 32-bit Linux
typically is somewhere between 2 and 3 GiB. So yes, there are some
file sizes which can be edited in addition, but that range is rather
narrow.

I still think that --without-wide-int is the more reasonable default.
Users who need those big buffers or files can easily flip the option.

Ulrich

Re: Making --with-wide-int the default

[Achim Gratz]

Ulrich Mueller writes:
> I still think that --without-wide-int is the more reasonable default.

I beg to differ.

> Users who need those big buffers or files can easily flip the option.

That's not a good argument to base that decision on anyway.  The trigger
for making that switch on all my old 32bit machines was that you can't
represent an awful lot of dates on any modern system otherwise.  So the
decision should really be based on what size time_t is, IMHO.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Making --with-wide-int the default

[David Kastrup]

Achim Gratz <Stromeko@nexgo.de> writes:

> Ulrich Mueller writes:
>> I still think that --without-wide-int is the more reasonable default.
>
> I beg to differ.
>
>> Users who need those big buffers or files can easily flip the option.
>
> That's not a good argument to base that decision on anyway.  The
> trigger for making that switch on all my old 32bit machines was that
> you can't represent an awful lot of dates on any modern system
> otherwise.

Huh?

current-time is a built-in function in ‘C source code’.

(current-time)

Return the current time, as the number of seconds since 1970-01-01 00:00:00.
The time is returned as a list of integers (HIGH LOW USEC PSEC).
HIGH has the most significant bits of the seconds, while LOW has the
least significant 16 bits.  USEC and PSEC are the microsecond and
picosecond counts.

[back]

float-time is a built-in function in ‘C source code’.

(float-time &optional SPECIFIED-TIME)

Return the current time, as a float number of seconds since the epoch.
If SPECIFIED-TIME is given, it is the time to convert to float
instead of the current time.  The argument should have the form
(HIGH LOW) or (HIGH LOW USEC) or (HIGH LOW USEC PSEC).  Thus,
you can use times from ‘current-time’ and from ‘file-attributes’.
SPECIFIED-TIME can also have the form (HIGH . LOW), but this is
considered obsolete.

WARNING: Since the result is floating point, it may not be exact.
If precise time stamps are required, use either ‘current-time’,
or (if you need time as a string) ‘format-time-string’.

[back]

> So the decision should really be based on what size time_t is, IMHO.

I don't see the point since all time representations of Emacs are long
established in a form that works fine with 24 bits of precision.

So time_t is a "solved problem" at the current point of time.  I don't
see that tampering with it makes any sense before Emacs gains arbitrary
precision integers: it seems pointless to tie time manipulations to some
particular machine precision.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Wed, 18 Nov 2015 17:32:10 +0100
> 
> The trigger for making that switch on all my old 32bit machines was
> that you can't represent an awful lot of dates on any modern system
> otherwise.  So the decision should really be based on what size
> time_t is, IMHO.

Maybe I misunderstand you, but the --with-wide-int option doesn't (and
cannot) change anything wrt the time_t type.  If it was a 32-bit type
without wide ints, it will still be a 32-bit type with them.

Or did you mean something else?

Re: Making --with-wide-int the default

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > A limit imposed by the machine's word size doesn't seem so arbitrary,
  > if it can only be surpassed with a significant performance penalty.

That is right.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Making --with-wide-int the default

[David Kastrup]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 11/15/2015 08:01 AM, David Kastrup wrote:
>> I can tell, and
>>
>> git grep XINT
>>
>> shows lots of assignments to variables of type int rather than
>> EMACS_INT.  I think that there should be macros XINT, XUINT, XEINT (for
>> EMACS_INT) which trigger a range error when the value does not fit the
>> respective range.  Possibly also XLONG and XULONG but I'm not sure we
>> actually have platforms where long/int are different.
>
> There's no need XEINT etc., as this work has already been done, using
> macros like CHECK_TYPE_RANGED_INTEGER to make sure that XINT returns
> values in range for a C integer type.

This has to be called explicitly.  There are several consequences of the
current choices: for one thing, one cannot enable C compiler warnings
about too small integer types since any assignment of XINT to an int
will trigger such a warning, making them useless for detecting actual
problems.  As one consequence, every single use of XINT needs manual
code review.

As another consequence, we'll have to redo all of the code review when
moving to Guile and/or arbitrary precision integers since the current
code obviously assumes that EMACS_INT is large enough for any integer
value representable in Lisp.

Guile has all of

-- C Function: char scm_to_char (SCM x)
 -- C Function: signed char scm_to_schar (SCM x)
 -- C Function: unsigned char scm_to_uchar (SCM x)
 -- C Function: short scm_to_short (SCM x)
 -- C Function: unsigned short scm_to_ushort (SCM x)
 -- C Function: int scm_to_int (SCM x)
 -- C Function: unsigned int scm_to_uint (SCM x)
 -- C Function: long scm_to_long (SCM x)
 -- C Function: unsigned long scm_to_ulong (SCM x)
 -- C Function: long long scm_to_long_long (SCM x)
 -- C Function: unsigned long long scm_to_ulong_long (SCM x)
 -- C Function: size_t scm_to_size_t (SCM x)
 -- C Function: ssize_t scm_to_ssize_t (SCM x)
 -- C Function: scm_t_ptrdiff scm_to_ptrdiff_t (SCM x)
 -- C Function: scm_t_int8 scm_to_int8 (SCM x)
 -- C Function: scm_t_uint8 scm_to_uint8 (SCM x)
 -- C Function: scm_t_int16 scm_to_int16 (SCM x)
 -- C Function: scm_t_uint16 scm_to_uint16 (SCM x)
 -- C Function: scm_t_int32 scm_to_int32 (SCM x)
 -- C Function: scm_t_uint32 scm_to_uint32 (SCM x)
 -- C Function: scm_t_int64 scm_to_int64 (SCM x)
 -- C Function: scm_t_uint64 scm_to_uint64 (SCM x)
 -- C Function: scm_t_intmax scm_to_intmax (SCM x)
 -- C Function: scm_t_uintmax scm_to_uintmax (SCM x)
     When X represents an exact integer that fits into the indicated C
     type, return that integer.  Else signal an error, either a
     ‘wrong-type’ error when X is not an exact integer, or an
     ‘out-of-range’ error when it doesn’t fit the given range.

     The functions ‘scm_to_long_long’, ‘scm_to_ulong_long’,
     ‘scm_to_int64’, and ‘scm_to_uint64’ are only available when the
     corresponding types are.

What does that mean?  For one thing it means that the C compiler can
have all of its warnings for losing significant bits enabled
meaningfully.  For another it means that converting a Lisp integer to
something used is a C routine will either succeed or throw an error.

The reverse conversions all succeed, and naturally without ever losing
any precision (as opposed to our current conversion which, I think,
silently drops a few bits from the top of any EMACS_INT when converting
to Lisp_Object .

Integer overflows are a rather popular source of security
vulnerabilities and/or crashes, so having them under control by default
is a good idea and means that we don't need to start rereviewing
everything whenever our integer representation changes.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Paul Eggert]

On 11/17/2015 04:13 AM, David Kastrup wrote:
> Integer overflows are a rather popular source of security
> vulnerabilities and/or crashes, so having them under control by default
> is a good idea

Yes, yes, all that's good, but that set of primitives (scm_t_uint16 
scm_to_uint16 etc.), while an impressively long laundry list, doesn't 
solve the problem or even (to be honest) inspire much confidence that 
the problem is even understood. How does one convert a Guile integer to 
a time_t?  Or to an off_t? Or to a nonnegative ptrdiff_t? None of the 
primitives you mention seem to address the typical problems I run into 
when auditing Emacs source code.

Far more useful is a small set of generic primitives that one can use to 
convert a Lisp integer to any system integer type, checking for overflow 
in the process. Emacs has that already. I suppose something like that 
could be built in Guile too. If so, then all we'd need to do is port the 
existing Emacs generic macros to run atop Guile, and no further code 
review should be needed.

Re: Making --with-wide-int the default

[Tom Tromey]

David> Of course, doing this properly might also mean less work for moving to
David> arbitrary-size integers eventually (probably using the Lisp_Int1 tag
David> when the 28-bit range does not suffice), though that one will likely
David> need a lot of converting `eq' -> `eql' in the Lisp code base.

Preserving the eq/eql distinction seems pointless.  It's a leaky
abstraction with no real gain for Emacs Lisp developers.  I think it's
better to just make `eq' treat bignums as values, and then don't bother
changing all the exist elisp.

Tom

Re: Making --with-wide-int the default

[Stefan Monnier]

>> I think it is quite reasonable _not_ to use wide ints by
>> default on architectures with a 32-bit address space.
> +1.

Another +1 here.

My impression from here is that 32bit userlands are becoming the
minority anyway, so they're mostly used on machines with
limited resources (CPU, cache, RAM).

In this context, I think it's wrong to use up more CPU, RAM and cache by
default all the time, just on the off-chance that the user will want to
view a file that happens to fall just within the 512MB to 2GB window.

Would it have ever helped me to build with wide-int?  Yes, maybe a couple
of time over the last few years.  But those are dwarfed by the tens of
times that this would have made no difference because the file was
greater than 2GB anyway.

For those rare >512MB cases, there's GNU ELPA's "vlf" package.  If we
care about users being able to open very large files on 32bit systems,
we should improve that package rather than enable wide-ints by default.


        Stefan

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 19 Nov 2015 21:20:16 -0500
> 
> >> I think it is quite reasonable _not_ to use wide ints by
> >> default on architectures with a 32-bit address space.
> > +1.
> 
> Another +1 here.

John, it's time to make the decision.  You have all the opinions and
their rationales.  If you have more questions, please ask.  But we
should conclude this longish discussion at some point; now sounds as
good time as any.

Thanks.

Re: Making --with-wide-int the default

[John Wiegley]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> For those rare >512MB cases, there's GNU ELPA's "vlf" package. If we care
> about users being able to open very large files on 32bit systems, we should
> improve that package rather than enable wide-ints by default.

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> John, it's time to make the decision. You have all the opinions and their
> rationales. If you have more questions, please ask. But we should conclude
> this longish discussion at some point; now sounds as good time as any.

I hold with Stefan's reasoning. We should not change --with-wide-int to become
the default.

--with-wide-int as a default benefits one group (most users on 32-bit
platforms) at the cost of another (users on 32-bit platforms whom it would
slow down, and who would need to disable it). Also, it does not affect those
who are becoming the predominant audience (users on 64-bit platforms). Thus, I
don't see it as a necessary change, especially if "vlf" is a viable
alternative. Lastly, 32-bit power users can configure using --with-wide-int.

Making this change requires further tests, documentation on how to disable
wide-ints (and why), and changes the status quo without demonstrating a
compelling benefit.

If it had no costs at all, I'd say it was the "right thing to do"; but it's
not zero cost, and the benefit does not appear to justify the change.

John

Re: Making --with-wide-int the default

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 330 bytes --]

On Wed, Nov 11, 2015 at 7:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Based on this, I'd like to turn --with-wide-int on by default.  Any
> last-minute objections?

I've been building with --with-wide-int for the past few weeks with no ill
effect. My Emacs is slow, but as I build with many checks enabled, it is
always slow...

[-- Attachment #2: Type: text/html, Size: 459 bytes --]

Re: Making --with-wide-int the default

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

On Fri, Oct 16, 2015 at 10:15 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > Cc: emacs-devel@gnu.org
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > Date: Fri, 16 Oct 2015 01:03:15 -0700

> > I typically use 64-bit platforms nowadays, where the issue is moot.
When I do
> > use 32-bit platforms, I normally configure --with-wide-int. There is
roughly a
> > 30% CPU hit and maybe a 60% hit on virtual memory, but it's worth it to
me (I
> > normally don't notice the difference).
>
> Thanks, that about summarizes what I see here as well.

Are we talking of a 30% CPU hit in non-optimized builds, right?

[-- Attachment #2: Type: text/html, Size: 869 bytes --]

Re: Making --with-wide-int the default

[Paul Eggert]

Juanma Barranquero wrote:
> Are we talking of a 30% CPU hit in non-optimized builds, right?

No, I was measuring with -O2.

Re: Making --with-wide-int the default

[David Kastrup]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Eli Zaretskii wrote:
>> To make it the default for all 32-bit builds, we should make sure it
>> works well on the other supported platforms.  Perhaps Paul could tell
>> which platforms he found this to work on, and then we could decide.
>
> --with-wide-int is not the default because Stefan was worried about
> its performance implications on older, slower 32-bit machines (such as
> Stefan's and/or RMS's laptops at the time, if I recall
> correctly). There shouldn't be any correctness problem with it; it's a
> performance issue.
>
> I typically use 64-bit platforms nowadays, where the issue is
> moot. When I do use 32-bit platforms, I normally configure
> --with-wide-int. There is roughly a 30% CPU hit and maybe a 60% hit on
> virtual memory, but it's worth it to me (I normally don't notice the
> difference).

Ugh.  That would be quite noticeable to me.  While I build my own Emacs,
having to drag around a bundle of homebrewn options just to get sensible
behavior does not appeal to me.

Instead of going to 64-bit unilaterally it would seem to make more sense
to me to degrade gracefully into gmp.  GUILE does that, I think that
XEmacs or SXEmacs can do it, and it's usual for Lisp implementations.
That leaves the memory cost for the 0.01% of uses (if at all) that need
it.  Of course, with a CPU cost hit (which in the common case of
addition/subtraction at natural width would amount to checking the
overflow bit afterwards).

Would also make for a nice speedup of calc (which implements everything
the hard way via small ints and lists, the Emacs 18 way: David Gillespie
dropped off the Earth before I could manage to have him dig up his Emacs
19 modifications using floating point when available).  If one can
disentangle the internals appropriately.  But that's probably easier for
using gmp transparently than for using FP as first approximation.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Paul Eggert]

David Kastrup wrote:
> Instead of going to 64-bit unilaterally it would seem to make more sense
> to me to degrade gracefully into gmp.  GUILE does that, I think that
> XEmacs or SXEmacs can do it, and it's usual for Lisp implementations.

In my spare time I've been working on something along those lines, but it's not 
at all ready for prime time. Some Emacs code assumes that integer overflow 
silently wraps around, and another common assumption is that integers are = if 
and only if they are eq. Bignums violate both assumptions.

Re: Making --with-wide-int the default

[David Kastrup]

Paul Eggert <eggert@cs.ucla.edu> writes:

> David Kastrup wrote:
>> Instead of going to 64-bit unilaterally it would seem to make more sense
>> to me to degrade gracefully into gmp.  GUILE does that, I think that
>> XEmacs or SXEmacs can do it, and it's usual for Lisp implementations.
>
> In my spare time I've been working on something along those lines, but
> it's not at all ready for prime time. Some Emacs code assumes that
> integer overflow silently wraps around, and another common assumption
> is that integers are = if and only if they are eq. Bignums violate
> both assumptions.

Yes.  The LilyPond code base (running on GUILE) is riddled with code
that assumes integers are eq.  Fixing that is slow work.  On the other
hand, the unfixed code does not really work worse than Emacs' code works
now: neither cannot handle values larger than standard integers.

In order to get the same semantics as now, a GMP number should be
converted back to a native LISP integer whenever it's small enough
again.  That provides some slight overhead for arithmetic that had to
happen in GMP, but then we are outside of the main performance
characteristic anyway.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Juanma Barranquero <lekktu@gmail.com>,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 10:18:26 +0200
> 
> Instead of going to 64-bit unilaterally it would seem to make more sense
> to me to degrade gracefully into gmp.

How can GMP help extend the maximum size of buffers and strings beyond
what a 32-bit EMACS_INT allows?

And how can GMP be faster than (or even close to) the native 64-bit
integral types supported by the C compiler?

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, Juanma Barranquero
>> <lekktu@gmail.com>, emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 10:18:26 +0200
>> 
>> Instead of going to 64-bit unilaterally it would seem to make more sense
>> to me to degrade gracefully into gmp.
>
> How can GMP help extend the maximum size of buffers and strings beyond
> what a 32-bit EMACS_INT allows?

By choosing an appropriate data type for representing buffer/string
sizes in C and converting back-and-forth from the Lisp type as needed.
Pretty much the same way we do it now.  I think it would be a reasonable
restriction to keep to 2GB size of strings and buffers when working with
a 32bit executable.  That's what people expect on a 32-bit architecture.

> And how can GMP be faster than (or even close to) the native 64-bit
> integral types supported by the C compiler?

GMP will not get to see the vast majority of numbers passing through
Emacs, and on a 32-bit system 32-bit integral types will be faster than
64-bit integral types.

When you are editing gigabyte files, at some point of time, the Lisp
representation of the respective offsets in the high part of the buffer
will become the responsibility of GMP, yes.  I'm not worried about that.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: eggert@cs.ucla.edu,  lekktu@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 11:18:31 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: David Kastrup <dak@gnu.org>
> >> Cc: Eli Zaretskii <eliz@gnu.org>, Juanma Barranquero
> >> <lekktu@gmail.com>, emacs-devel@gnu.org
> >> Date: Fri, 16 Oct 2015 10:18:26 +0200
> >> 
> >> Instead of going to 64-bit unilaterally it would seem to make more sense
> >> to me to degrade gracefully into gmp.
> >
> > How can GMP help extend the maximum size of buffers and strings beyond
> > what a 32-bit EMACS_INT allows?
> 
> By choosing an appropriate data type for representing buffer/string
> sizes in C and converting back-and-forth from the Lisp type as needed.
> Pretty much the same way we do it now.

Sorry, I don't foollow: what "appropriate data type"?  Would that be
'long long'?  If so, that's exactly what we do now, which you say is
"going to 64-bit unilaterally".  What am I missing?

> I think it would be a reasonable restriction to keep to 2GB size of
> strings and buffers when working with a 32bit executable.  That's
> what people expect on a 32-bit architecture.

That's what we have now in 32-bit builds --with-wide-int.  So I'm not
sure why you mention that as some kind of change related to this
discussion.

> > And how can GMP be faster than (or even close to) the native 64-bit
> > integral types supported by the C compiler?
> 
> GMP will not get to see the vast majority of numbers passing through
> Emacs, and on a 32-bit system 32-bit integral types will be faster than
> 64-bit integral types.
> 
> When you are editing gigabyte files, at some point of time, the Lisp
> representation of the respective offsets in the high part of the buffer
> will become the responsibility of GMP, yes.  I'm not worried about that.

I don't understand.  C doesn't have dynamic types.  If the variable
that holds buffer positions needs to support 61-bit offsets, it will
have to be a 64-bit integral data type from the get-go.

And having 61-bit integers for integer arithmetics is also a valuable
feature.  So the EMACS_INT type will have to be able to support that.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: eggert@cs.ucla.edu,  lekktu@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 11:18:31 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: David Kastrup <dak@gnu.org>
>> >> Cc: Eli Zaretskii <eliz@gnu.org>, Juanma Barranquero
>> >> <lekktu@gmail.com>, emacs-devel@gnu.org
>> >> Date: Fri, 16 Oct 2015 10:18:26 +0200
>> >> 
>> >> Instead of going to 64-bit unilaterally it would seem to make more sense
>> >> to me to degrade gracefully into gmp.
>> >
>> > How can GMP help extend the maximum size of buffers and strings beyond
>> > what a 32-bit EMACS_INT allows?
>> 
>> By choosing an appropriate data type for representing buffer/string
>> sizes in C and converting back-and-forth from the Lisp type as needed.
>> Pretty much the same way we do it now.
>
> Sorry, I don't foollow: what "appropriate data type"?  Would that be
> 'long long'?  If so, that's exactly what we do now, which you say is
> "going to 64-bit unilaterally".  What am I missing?

A change in the size of the Lisp data type.

>> I think it would be a reasonable restriction to keep to 2GB size of
>> strings and buffers when working with a 32bit executable.  That's
>> what people expect on a 32-bit architecture.
>
> That's what we have now in 32-bit builds --with-wide-int.  So I'm not
> sure why you mention that as some kind of change related to this
> discussion.

Isn't that also changing the size of a Lisp cell?  And of integer
arithmetic?

>> When you are editing gigabyte files, at some point of time, the Lisp
>> representation of the respective offsets in the high part of the
>> buffer will become the responsibility of GMP, yes.  I'm not worried
>> about that.
>
> I don't understand.  C doesn't have dynamic types.

But Lisp does.

> If the variable that holds buffer positions needs to support 61-bit
> offsets, it will have to be a 64-bit integral data type from the
> get-go.

I repeat: we are talking about a 32-bit binary where restricting buffer
and string size to 32bit offsets would be reasonable and expected.  But
even if we assume that we want to support some larger size in the C
parts, that's just the matter of choosing a different type and a
different conversion from Lisp to C.

GUILE has all of

 -- C Function: char scm_to_char (SCM x)
 -- C Function: signed char scm_to_schar (SCM x)
 -- C Function: unsigned char scm_to_uchar (SCM x)
 -- C Function: short scm_to_short (SCM x)
 -- C Function: unsigned short scm_to_ushort (SCM x)
 -- C Function: int scm_to_int (SCM x)
 -- C Function: unsigned int scm_to_uint (SCM x)
 -- C Function: long scm_to_long (SCM x)
 -- C Function: unsigned long scm_to_ulong (SCM x)
 -- C Function: long long scm_to_long_long (SCM x)
 -- C Function: unsigned long long scm_to_ulong_long (SCM x)
 -- C Function: size_t scm_to_size_t (SCM x)
 -- C Function: ssize_t scm_to_ssize_t (SCM x)
 -- C Function: scm_t_ptrdiff scm_to_ptrdiff_t (SCM x)
 -- C Function: scm_t_int8 scm_to_int8 (SCM x)
 -- C Function: scm_t_uint8 scm_to_uint8 (SCM x)
 -- C Function: scm_t_int16 scm_to_int16 (SCM x)
 -- C Function: scm_t_uint16 scm_to_uint16 (SCM x)
 -- C Function: scm_t_int32 scm_to_int32 (SCM x)
 -- C Function: scm_t_uint32 scm_to_uint32 (SCM x)
 -- C Function: scm_t_int64 scm_to_int64 (SCM x)
 -- C Function: scm_t_uint64 scm_to_uint64 (SCM x)
 -- C Function: scm_t_intmax scm_to_intmax (SCM x)
 -- C Function: scm_t_uintmax scm_to_uintmax (SCM x)
     When X represents an exact integer that fits into the indicated C
     type, return that integer.  Else signal an error, either a
     ‘wrong-type’ error when X is not an exact integer, or an
     ‘out-of-range’ error when it doesn’t fit the given range.

     The functions ‘scm_to_long_long’, ‘scm_to_ulong_long’,
     ‘scm_to_int64’, and ‘scm_to_uint64’ are only available when the
     corresponding types are.

Choose a C type of your choosing for dealing with buffer offsets, create
aliases for its conversions, and you are good to go.  The Lisp side
supports arbitrary sizes, and the actual C-side limits (rather than
something like 26 bits or 29 bits or 30 bits or 61 bits) are established
when converting from Lisp types.

> And having 61-bit integers for integer arithmetics is also a valuable
> feature.

61-bit is some arbitrary junk number.  Transparently degrading from
29-bits to gmp numbers means that there are no arbitrary limits (or at
least you are unlikely to hit them) and the performance for the vast
majority of cases will be 29-bit performance.

> So the EMACS_INT type will have to be able to support that.

I don't see why when one can use GMP (which effectively uses the same
kind arithmetic for 61-bit numbers as C does but does not stop there).

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: eggert@cs.ucla.edu,  lekktu@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 12:27:04 +0200
> 
> >> > How can GMP help extend the maximum size of buffers and strings beyond
> >> > what a 32-bit EMACS_INT allows?
> >> 
> >> By choosing an appropriate data type for representing buffer/string
> >> sizes in C and converting back-and-forth from the Lisp type as needed.
> >> Pretty much the same way we do it now.
> >
> > Sorry, I don't foollow: what "appropriate data type"?  Would that be
> > 'long long'?  If so, that's exactly what we do now, which you say is
> > "going to 64-bit unilaterally".  What am I missing?
> 
> A change in the size of the Lisp data type.

We are talking about the _implementation_ of Lisp data types,
i.e. about the underlying C data types.  What am I missing on that
level?

> >> I think it would be a reasonable restriction to keep to 2GB size of
> >> strings and buffers when working with a 32bit executable.  That's
> >> what people expect on a 32-bit architecture.
> >
> > That's what we have now in 32-bit builds --with-wide-int.  So I'm not
> > sure why you mention that as some kind of change related to this
> > discussion.
> 
> Isn't that also changing the size of a Lisp cell?  And of integer
> arithmetic?

Part of it, yes.  But since a Lisp cell can hold buffer or string
position, what else can you expect?

> >> When you are editing gigabyte files, at some point of time, the Lisp
> >> representation of the respective offsets in the high part of the
> >> buffer will become the responsibility of GMP, yes.  I'm not worried
> >> about that.
> >
> > I don't understand.  C doesn't have dynamic types.
> 
> But Lisp does.

I was talking about C implementation of Lisp types.

> > If the variable that holds buffer positions needs to support 61-bit
> > offsets, it will have to be a 64-bit integral data type from the
> > get-go.
> 
> I repeat: we are talking about a 32-bit binary where restricting buffer
> and string size to 32bit offsets would be reasonable and expected.

You cannot have full 32-bit offsets without enlarging the width of a
Lisp integer.  We could use 34 or 35-bit numbers, but that would be
slower than using a 64-bit 'long long'.

> Choose a C type of your choosing for dealing with buffer offsets, create
> aliases for its conversions, and you are good to go.

We _have_ chosen: it's a 64-bit 'long long'.  What do you suggest to
choose instead?  Any suggested type should be representable as an
Emacs integer.

> > And having 61-bit integers for integer arithmetics is also a valuable
> > feature.
> 
> 61-bit is some arbitrary junk number.  Transparently degrading from
> 29-bits to gmp numbers means that there are no arbitrary limits (or at
> least you are unlikely to hit them) and the performance for the vast
> majority of cases will be 29-bit performance.

I don't understand how you intend to "transparently degrade" in the
implementation of '+', for example.  Are you going to test each
argument for whether it fits into 32-bit limits, and if so, invoke
32-bit arithmetics, else 64-bit arithmetics?  If so, I'm quite sure
the test will steal cycles that will make this slower than just always
using 64-bit arithmetics.

If you mean something else, can you show some C code that would
demonstrate your ideas?

> > So the EMACS_INT type will have to be able to support that.
> 
> I don't see why when one can use GMP (which effectively uses the same
> kind arithmetic for 61-bit numbers as C does but does not stop there).

Because calling a GMP function will most probably yield slower code
than if we let the compiler emit a few instructions required for
64-bit arithmetics.

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: eggert@cs.ucla.edu,  lekktu@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 12:27:04 +0200
>> 
>> >> > How can GMP help extend the maximum size of buffers and strings beyond
>> >> > what a 32-bit EMACS_INT allows?
>> >> 
>> >> By choosing an appropriate data type for representing buffer/string
>> >> sizes in C and converting back-and-forth from the Lisp type as needed.
>> >> Pretty much the same way we do it now.
>> >
>> > Sorry, I don't foollow: what "appropriate data type"?  Would that be
>> > 'long long'?  If so, that's exactly what we do now, which you say is
>> > "going to 64-bit unilaterally".  What am I missing?
>> 
>> A change in the size of the Lisp data type.
>
> We are talking about the _implementation_ of Lisp data types,
> i.e. about the underlying C data types.  What am I missing on that
> level?

You are assuming that the C data type used for working with
buffer/string position is the same as the type of a Lisp cell.

>> >> I think it would be a reasonable restriction to keep to 2GB size of
>> >> strings and buffers when working with a 32bit executable.  That's
>> >> what people expect on a 32-bit architecture.
>> >
>> > That's what we have now in 32-bit builds --with-wide-int.  So I'm not
>> > sure why you mention that as some kind of change related to this
>> > discussion.
>> 
>> Isn't that also changing the size of a Lisp cell?  And of integer
>> arithmetic?
>
> Part of it, yes.  But since a Lisp cell can hold buffer or string
> position, what else can you expect?

That buffer or string positions not representable in the 29 bits or so
available for integers in a Lisp cell are instead represented using a
gpm number for which there is a reference in the Lisp cell.

>> >> When you are editing gigabyte files, at some point of time, the
>> >> Lisp representation of the respective offsets in the high part of
>> >> the buffer will become the responsibility of GMP, yes.  I'm not
>> >> worried about that.
>> >
>> > I don't understand.  C doesn't have dynamic types.
>> 
>> But Lisp does.
>
> I was talking about C implementation of Lisp types.

You are apparently unable to deal with the concept of a Lisp type being
represented by fewer bits than the integral type used in C for things
like buffer/string positions outside of the 29-bit integer range.

How do you think Emacs managed 64-bit doubles "inside" of a 32-bit
integral type used for representing Lisp cells?

>> > If the variable that holds buffer positions needs to support 61-bit
>> > offsets, it will have to be a 64-bit integral data type from the
>> > get-go.
>> 
>> I repeat: we are talking about a 32-bit binary where restricting buffer
>> and string size to 32bit offsets would be reasonable and expected.
>
> You cannot have full 32-bit offsets without enlarging the width of a
> Lisp integer.

You most certainly can by transparent degradation to GMP numbers.
That's exactly how GUILE does it.

>> Choose a C type of your choosing for dealing with buffer offsets,
>> create aliases for its conversions, and you are good to go.
>
> We _have_ chosen: it's a 64-bit 'long long'.  What do you suggest to
> choose instead?  Any suggested type should be representable as an
> Emacs integer.

An "Emacs integer" is a Lisp data type.  You are confusing this with the
C type we use for Lisp cells, and with the C types we might use for
buffer and/or string offsets.

I can't believe we are having this conversation.  Emacs integers so far
are 29 bits, Emacs floats are 64 bits, Emacs cell types are 32 bits and
I don't understand how one can imagine that they are somehow all the
same.

>> > And having 61-bit integers for integer arithmetics is also a
>> > valuable feature.
>> 
>> 61-bit is some arbitrary junk number.  Transparently degrading from
>> 29-bits to gmp numbers means that there are no arbitrary limits (or
>> at least you are unlikely to hit them) and the performance for the
>> vast majority of cases will be 29-bit performance.
>
> I don't understand how you intend to "transparently degrade" in the
> implementation of '+', for example.

If the numbers have different signs and are single-cell integers, the
result fits.

Otherwise you add them and if the result (after conversion to 29 bits)
has a different sign than the starting numbers, you convert the result
instead to a GMP number.

> Are you going to test each argument for whether it fits into 32-bit
> limits,

29-bit limits.  I don't need to test the arguments since they will be
single-cell integers only _when_ they fit.  I only need to test the
result.

> and if so, invoke 32-bit arithmetics, else 64-bit arithmetics?  If so,
> I'm quite sure the test will steal cycles that will make this slower
> than just always using 64-bit arithmetics.

Disregarding the memory requirements and bandwidth for always using
64-bit.  Why don't we use 256 bit for everything?  It makes even more
things work at the cost of eating even more memory and speed.

One has to choose a sensible cutoff point for the default number
representation, and whether one will admit more than that.  A sensible
cutoff point is what fits into a Lisp cell of natural processor size.
GMP allows to admit lots more in the rare case where it is needed.

This is what GUILE does and most Lisp implementations with "arbitrary
precision" arithmetic do.

> If you mean something else, can you show some C code that would
> demonstrate your ideas?

Any application using GUILE and its C API.  Read the GUILE manual for
all the details.  This is not hypothetical, and as I already said, there
is as far as I know a compile time option for SXEmacs at least and
possibly also standard XEmacs to get this behavior for integers.

This is not a pipe dream as you appear to insinuate.

>> > So the EMACS_INT type will have to be able to support that.
>> 
>> I don't see why when one can use GMP (which effectively uses the same
>> kind arithmetic for 61-bit numbers as C does but does not stop there).
>
> Because calling a GMP function will most probably yield slower code
> than if we let the compiler emit a few instructions required for
> 64-bit arithmetics.

GMP needs to be called only when leaving the range of "small integers"
(which is all we even have right now).  64-bit arithmetic in your plan
would be required for every single operation.  Yes, when GMP kicks in,
it will be slower than operations using exactly 64-bit (not more, not
less).  But it's the exception rather than the rule.  So much the
exception that we could entirely make do without it so far.  It _will_
occur frequently when editing files larger than 1GB or so.  But only in
the _Lisp_ representations of those buffer positions.  Everything
implemented in C will use the integral data type we choose for that,
throwing an error when it gets exceeded.  And for a standard 32-bit
compilation, 32-bit integers seem like a reasonable cutoff point: you'll
not be able to load much more than 2G characters into a 32-bit address
space anyway.  Yes, not all of the possible offsets will be
representable in one Lisp cell.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: eggert@cs.ucla.edu,  lekktu@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 16:03:09 +0200
> 
> >> Isn't that also changing the size of a Lisp cell?  And of integer
> >> arithmetic?
> >
> > Part of it, yes.  But since a Lisp cell can hold buffer or string
> > position, what else can you expect?
> 
> That buffer or string positions not representable in the 29 bits or so
> available for integers in a Lisp cell are instead represented using a
> gpm number for which there is a reference in the Lisp cell.

But a GPM number is nothing more or less that some C data type.  It's
not some magic wand that can solve problems just by being there.

> You are apparently unable to deal with the concept of a Lisp type being
> represented by fewer bits than the integral type used in C for things
> like buffer/string positions outside of the 29-bit integer range.

Let's agree to keep such comments out of this discussion, OK?
Otherwise, I will simply end it.

> How do you think Emacs managed 64-bit doubles "inside" of a 32-bit
> integral type used for representing Lisp cells?

So you are suggesting to make a Lisp integer represented like a Lisp
float, i.e. accessible via a pointer?  But that in itself will slow
down integer arithmetics, due to the need to dereference the pointer,
won't it?

> GMP needs to be called only when leaving the range of "small integers"
> (which is all we even have right now).  64-bit arithmetic in your plan
> would be required for every single operation.  Yes, when GMP kicks in,
> it will be slower than operations using exactly 64-bit (not more, not
> less).  But it's the exception rather than the rule.

But there has to be a test for when the exception happens, and that
test is going to exert its price on every operation, whether it
succeeds or fails the test.  I'm not at all sure it will be a net win.

> So much the
> exception that we could entirely make do without it so far.  It _will_
> occur frequently when editing files larger than 1GB or so.  But only in
> the _Lisp_ representations of those buffer positions.  Everything
> implemented in C will use the integral data type we choose for that,
> throwing an error when it gets exceeded.

So you are suggesting to give up the transparent exposure of members
of Lisp objects that are integers?

> Yes, not all of the possible offsets will be representable in one
> Lisp cell.

So you are suggesting that a buffer position will not always be a
simple number, but will sometimes be a cons cell?

Re: Making --with-wide-int the default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: eggert@cs.ucla.edu,  lekktu@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 16:03:09 +0200
>> 
>> >> Isn't that also changing the size of a Lisp cell?  And of integer
>> >> arithmetic?
>> >
>> > Part of it, yes.  But since a Lisp cell can hold buffer or string
>> > position, what else can you expect?
>> 
>> That buffer or string positions not representable in the 29 bits or so
>> available for integers in a Lisp cell are instead represented using a
>> gpm number for which there is a reference in the Lisp cell.
>
> But a GPM number is nothing more or less that some C data type.  It's
> not some magic wand that can solve problems just by being there.
>
>> How do you think Emacs managed 64-bit doubles "inside" of a 32-bit
>> integral type used for representing Lisp cells?
>
> So you are suggesting to make a Lisp integer represented like a Lisp
> float, i.e. accessible via a pointer?

For values not fitting into 29 bits, yes.

> But that in itself will slow down integer arithmetics, due to the need
> to dereference the pointer, won't it?

For values not fitting into 29 bits, yes.

>> GMP needs to be called only when leaving the range of "small
>> integers" (which is all we even have right now).  64-bit arithmetic
>> in your plan would be required for every single operation.  Yes, when
>> GMP kicks in, it will be slower than operations using exactly 64-bit
>> (not more, not less).  But it's the exception rather than the rule.
>
> But there has to be a test for when the exception happens, and that
> test is going to exert its price on every operation, whether it
> succeeds or fails the test.  I'm not at all sure it will be a net win.

With regard to memory requirements and processing speed for values
fitting easily into 32Bit (including booleans, indirect types, symbols,
various other stuff), it will be a net win.  Not everything in Lisp is
an integer.

>> So much the exception that we could entirely make do without it so
>> far.  It _will_ occur frequently when editing files larger than 1GB
>> or so.  But only in the _Lisp_ representations of those buffer
>> positions.  Everything implemented in C will use the integral data
>> type we choose for that, throwing an error when it gets exceeded.
>
> So you are suggesting to give up the transparent exposure of members
> of Lisp objects that are integers?

I have no idea what you are talking about.  Integers and Lisp cells are
not the same.  Treating them as the same is a bug.  Just ask Stefan.  So
I have no idea what you call "transparent exposure" here.

>> Yes, not all of the possible offsets will be representable in one
>> Lisp cell.
>
> So you are suggesting that a buffer position will not always be a
> simple number, but will sometimes be a cons cell?

No.  It will always be a Lisp integer, but Lisp integers will not always
be stored in a single cell.  But GMP does not store its numbers in cons
cells.

Really, I have no idea what your problem here is.  Smooth degradation to
fit rare large values compactly is not a particularly uncommon concept.
Emacs uses it, for example, in UTF-8 representations of 8-bit encodings.
It represents character codes 0-127 as-is, then switches to non-byte
representation of 128-255.  Why don't we use 32-bit characters from the
get-go?  Because they are rarely needed and we don't want to always pay
the price in memory for the rarely needed cases.

Doubling the size of Lisp cells does not just (roughly) double the size
of all integers.

-- 
David Kastrup

Re: Making --with-wide-int the default

[Stephen J. Turnbull]

David Kastrup writes:

 > Instead of going to 64-bit unilaterally it would seem to make more
 > sense to me to degrade gracefully into gmp.  GUILE does that, I
 > think that XEmacs or SXEmacs can do it,

XEmacs can't handle greater than 1 GB buffers on a 32-bit machine.
Yes, when bignums are enabled Lisp arithmetic will automatically
overflow to bignums (and bigfloats), but internal accounting for text
objects uses EMACS_INTs, not Lisp integers.  At one time there were
several users who preferred XEmacs to Emacs because they were
routinely manipulating 512MB+ buffers and files.

If you just want big Lisp integers, yes, bignums are nice.

To be honest, I don't think it's a good idea to use bignums for
internal indexes into text objects.  1 exabyte buffers should be
enough to keep even the full archives of emacs-devel with room to
spare for a while.

Re: Making --with-wide-int the default

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > Instead of going to 64-bit unilaterally it would seem to make more
>  > sense to me to degrade gracefully into gmp.  GUILE does that, I
>  > think that XEmacs or SXEmacs can do it,
>
> XEmacs can't handle greater than 1 GB buffers on a 32-bit machine.
> Yes, when bignums are enabled Lisp arithmetic will automatically
> overflow to bignums (and bigfloats), but internal accounting for text
> objects uses EMACS_INTs, not Lisp integers.  At one time there were
> several users who preferred XEmacs to Emacs because they were
> routinely manipulating 512MB+ buffers and files.
>
> If you just want big Lisp integers, yes, bignums are nice.
>
> To be honest, I don't think it's a good idea to use bignums for
> internal indexes into text objects.  1 exabyte buffers should be
> enough to keep even the full archives of emacs-devel with room to
> spare for a while.

If a particular application warrants a particular C integral type, you
can always use the appropriate conversion routine for extracting it and
it will throw an error when the result does not fit the type.

That's the way one works with GUILE, and it's perfectly usable.  When
stuff blows up, it does so with an error message and at the limits of
the values that the C code can deal with rather than something like a
30-bit integer.

-- 
David Kastrup

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Do., 15. Okt.
2015 um 12:44 Uhr:

> On Thu, Oct 15, 2015 at 2:25 AM, Philipp Stephani <p.stephani2@gmail.com>
> wrote:
> > Agreed. I've implemented Daniel's suggestion on the tls-error branch, it
> > seems to work fine (but note that environments are now no longer global,
> so
> > storing away emacs_values without global references now will lead to
> > undefined behavior).
>
> Yeah I just realized that. The nice thing with just casting is its the
> same memory object and the stack-scanning GC just works. Having the GC
> is nice...
>

Until you change the GC algorithm to a precise GC or reference counting and
cause all modules to crash, leak memory, or worse. This is exactly the kind
of dependency on implementation details that we want to avoid with a clean
module interface.


> Again, IIUC, the only problem is on 32bit with wide-int, which is off
> by default.
>
>
Even if sizeof(Lisp_Object) == sizeof(void*), you still need the guarantee
that (Lisp_Object)((void*)0) is an invalid Lisp object.


> > Let's see what will get accepted upstream.
>
> That's sneaky... but this is how the free software world works I guess...
>

I don't think it's sneaky. The opinions of everyone are known and have been
discussed ad nauseam in public. Either it gets accepted or it doesn't.

[-- Attachment #2: Type: text/html, Size: 2088 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 256 bytes --]

I just realized that we need (at least) one more public environment
function, either "null" or "eq". Otherwise conditions cannot be written in
module code.
I'd vote for bool is_not_null, it seems to be the most basic one, and
doesn't invert the condition.

[-- Attachment #2: Type: text/html, Size: 327 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am Fr., 16. Okt. 2015 um
01:15 Uhr:

> I just realized that we need (at least) one more public environment
> function, either "null" or "eq". Otherwise conditions cannot be written in
> module code.
> I'd vote for bool is_not_null, it seems to be the most basic one, and
> doesn't invert the condition.
>

There are still a few open questions:

   1.  How to represent throw/catch. Currently it's not possible to throw
   from modules, but that should change because otherwise modules can't be
   transparent. For example, the following couldn't work:

   emacs_value top_level = env->intern(env, "top-level");
   if (!top_level) return NULL;
   return env->funcall(env, top_level, 0, NULL);

   because `top-level' uses `throw'.
   There are several ways to represent `throw'. My suggestion would still
   be to use an enum instead of the bool value for get_error and check_error.
   Otherwise the fact that a `throw' has happened has to be encoded somehow in
   the error symbol, which to me seems rather hackish and not future-proof.
   (What happens if Emacs would ever allow to signal a list?)
   2. Whether type_of should return an enum or a symbol. I'd vote for the
   symbol, as in Daniel's original design.
   3. Can runtime->get_environment ever fail?
   4. Some of the function (free_global_ref, the int/float extractors)
   don't return emacs_value and have no clear way to signal an error. For
   these function the caller always has to use check_error/get_error. Is that
   good enough?

[-- Attachment #2: Type: text/html, Size: 1936 bytes --]

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2532 bytes --]

On 10/19/2015 03:38 PM, Philipp Stephani wrote:
> Philipp Stephani <p.stephani2@gmail.com <mailto:p.stephani2@gmail.com>>
> schrieb am Fr., 16. Okt. 2015 um 01:15 Uhr:
> 
>     I just realized that we need (at least) one more public environment
>     function, either "null" or "eq". Otherwise conditions cannot be
>     written in module code.
>     I'd vote for bool is_not_null, it seems to be the most basic one,
>     and doesn't invert the condition.

Yes, we do. I'd prefer eq, since the definition will never change, and
having a nice convenient way to access eq is useful in more situations
than a specialized test for eq nil.

> There are still a few open questions:
> 
>  1.  How to represent throw/catch. Currently it's not possible to throw
>     from modules, but that should change because otherwise modules can't
>     be transparent. For example, the following couldn't work:
> 
>     emacs_value top_level = env->intern(env, "top-level");
>     if (!top_level) return NULL;
>     return env->funcall(env, top_level, 0, NULL);
> 
>     because `top-level' uses `throw'.
>     There are several ways to represent `throw'. My suggestion would
>     still be to use an enum instead of the bool value for get_error and
>     check_error. Otherwise the fact that a `throw' has happened has to
>     be encoded somehow in the error symbol, which to me seems rather
>     hackish and not future-proof. (What happens if Emacs would ever
>     allow to signal a list?)

An enum to indicate the current thread's non-local error propagation
state is probably fine.

>  2. Whether type_of should return an enum or a symbol. I'd vote for the
>     symbol, as in Daniel's original design.

A symbol means that we can define as many types as we want.

>  3. Can runtime->get_environment ever fail?

Maybe we should actually get rid of get_environment and rely entirely on
passed-in environment objects. It simplifies the API and avoids
definitional problems.

>  4. Some of the function (free_global_ref, the int/float extractors)
>     don't return emacs_value and have no clear way to signal an error.
>     For these function the caller always has to use
>     check_error/get_error. Is that good enough?

free_global_ref, like all resource-releasing functions, should be
infallible. For other functions, asking users to check the error state
after a call is fine, although we should also return reasonable dummy
values (0 and NaN, probably) as well in case they don't.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Tom Tromey]

Daniel> Yes, we do. I'd prefer eq, since the definition will never change, and
Daniel> having a nice convenient way to access eq is useful in more situations
Daniel> than a specialized test for eq nil.

Both are useful and it is cheap to provide a method, so why not have
both eq and nilp?

Daniel> Maybe we should actually get rid of get_environment and rely entirely on
Daniel> passed-in environment objects. It simplifies the API and avoids
Daniel> definitional problems.

Yeah, I brought this up earlier and I still don't understand the reason
for the runtime/environment split.

>> 4. Some of the function (free_global_ref, the int/float extractors)
>> don't return emacs_value and have no clear way to signal an error.
>> For these function the caller always has to use
>> check_error/get_error. Is that good enough?

Daniel> free_global_ref, like all resource-releasing functions, should be
Daniel> infallible. For other functions, asking users to check the error state
Daniel> after a call is fine, although we should also return reasonable dummy
Daniel> values (0 and NaN, probably) as well in case they don't.

It might be handy if there was a special emacs_value meaning that an
error occurred.

Tom

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]

On 10/19/2015 08:05 PM, Tom Tromey wrote:
> Daniel> Yes, we do. I'd prefer eq, since the definition will never change, and
> Daniel> having a nice convenient way to access eq is useful in more situations
> Daniel> than a specialized test for eq nil.
> 
> Both are useful and it is cheap to provide a method, so why not have
> both eq and nilp?

I have no objection to including both.

> Daniel> Maybe we should actually get rid of get_environment and rely entirely on
> Daniel> passed-in environment objects. It simplifies the API and avoids
> Daniel> definitional problems.
> 
> Yeah, I brought this up earlier and I still don't understand the reason
> for the runtime/environment split.

A runtime describes Emacs as a whole. It lives forever and has no thread
affinity. An environment represents a specific stack activation of an
Emacs module function. It has thread affinity. Because environments have
thread affinity, we can very cheaply store local references, error
context, and whatever else we want in them. We could probably get away
with only defining environments, but then we'd have no way to allow
routines to call Emacs routines at any time.

(Useful future runtime routines might be "attach this thread to the Lisp
engine", "send QUIT to the main thread", "abort spectacularly", "get
Emacs version", and so on.)

>>> 4. Some of the function (free_global_ref, the int/float extractors)
>>> don't return emacs_value and have no clear way to signal an error.
>>> For these function the caller always has to use
>>> check_error/get_error. Is that good enough?
> 
> Daniel> free_global_ref, like all resource-releasing functions, should be
> Daniel> infallible. For other functions, asking users to check the error state
> Daniel> after a call is fine, although we should also return reasonable dummy
> Daniel> values (0 and NaN, probably) as well in case they don't.
> 
> It might be handy if there was a special emacs_value meaning that an
> error occurred.

We have a perfectly good sentinel: NULL. NULL will never be a valid
local or global reference, so it's safe to use it to indicate that
something went wrong.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Tom Tromey]

Daniel> A runtime describes Emacs as a whole. It lives forever and has
Daniel> no thread affinity.

Thanks.

Daniel> We have a perfectly good sentinel: NULL. NULL will never be a valid
Daniel> local or global reference, so it's safe to use it to indicate that
Daniel> something went wrong.

Ok, I see, it really is JNI.  I was confused because the module API
doesn't implement this.

This design seems a bit odd for Emacs in that "immediate" things like
ints will still have to be wrapped in references.

Tom

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]

On 10/19/2015 08:32 PM, Tom Tromey wrote:
> Daniel> A runtime describes Emacs as a whole. It lives forever and has
> Daniel> no thread affinity.
> 
> Thanks.
> 
> Daniel> We have a perfectly good sentinel: NULL. NULL will never be a valid
> Daniel> local or global reference, so it's safe to use it to indicate that
> Daniel> something went wrong.
> 
> Ok, I see, it really is JNI.  I was confused because the module API
> doesn't implement this.

Sure, although I'd like it to. If we use the currently-implemented
approach, it's not a disaster. It'd just be a shame to have to cart
around long long values on 32-bit platforms even when Emacs is compiled
with plain 32-bit values.

> This design seems a bit odd for Emacs in that "immediate" things like
> ints will still have to be wrapped in references.

Right, because we're not strongly typed like Java. The references are
cheap enough though, and there's no awkardness around pointer width or
type tagging. Once we get a numeric tower, it'll be possible to store a
full 32 bits in an emacs_value without having to be aware that there's
an indirection. This approach also completely insulates us from GC
changes: we could even implement a compacting collector without modules
being aware of it.

FWIW, it's not just JNI. V8 works the same way.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Stephen Leake]

Daniel Colascione <dancol@dancol.org> writes:

> A runtime describes Emacs as a whole. It lives forever and has no thread
> affinity. An environment represents a specific stack activation of an
> Emacs module function. It has thread affinity. Because environments have
> thread affinity, we can very cheaply store local references, error
> context, and whatever else we want in them. 

Please add this to comments in the API, so we don't forget.

or add a module_design.texi, for this and similar design info.

-- 
-- Stephe

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

Stephen Leake <stephen_leake@stephe-leake.org> schrieb am Di., 20. Okt.
2015 um 17:48 Uhr:

> Daniel Colascione <dancol@dancol.org> writes:
>
> > A runtime describes Emacs as a whole. It lives forever and has no thread
> > affinity. An environment represents a specific stack activation of an
> > Emacs module function. It has thread affinity. Because environments have
> > thread affinity, we can very cheaply store local references, error
> > context, and whatever else we want in them.
>
> Please add this to comments in the API, so we don't forget.
>
>
 Or implement it incorrectly. For example, the runtime object in my pull
request is local to the init function.

If the runtime object is static, does its get_environment function always
return the same environment? In other words, is that environment also
static?

[-- Attachment #2: Type: text/html, Size: 1240 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

I've added user-pointer objects. Some feedback I had off-list:

* add functions to the API for fast vector access as it currently
requires too much boxing/unboxing of integers for doing
length/indexing stuff.
* expose fast read-only string access (not sure how do-able this is
considering the encoding problem)
* consider NULL as a no-op finalizer.

Philip, you obviously have more experienced in this memory handling
stuff so I'm going to trust you and merge what you did (but there are
merge conflicts on the last pull requests atm, havent looked into it).
Thanks again for your work :)

This module API is slowly but surely materializing. The interface is
lot more clean and user-friendly than my initial plan. I hope we can
have good performances where it matters to make it really useful :)

Re: Dynamic loading progress

[Aurélien Aptel]

I've finally compiled emacs on a Windows VM using msys2 & mingw-w64!

After doing basically nothing (adding the proper $opsys check and
setting HAVE_MODULES to "yes" in the configure script...) I was able
to build and run successfully all test modules on Windows. Note that
the modules makefile still hardcode the .so suffix for shared
libraries (even though gcc ouputs regular PE DLLs) and loading them
as-is works anyway. Neat. I'll change that, eventually.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Thu, 22 Oct 2015 19:56:00 +0200
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, Daniel Colascione <dancol@dancol.org>, 
> 	Tom Tromey <tom@tromey.com>, Eli Zaretskii <eliz@gnu.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> I've finally compiled emacs on a Windows VM using msys2 & mingw-w64!
> 
> After doing basically nothing (adding the proper $opsys check and
> setting HAVE_MODULES to "yes" in the configure script...) I was able
> to build and run successfully all test modules on Windows. Note that
> the modules makefile still hardcode the .so suffix for shared
> libraries (even though gcc ouputs regular PE DLLs) and loading them
> as-is works anyway. Neat. I'll change that, eventually.

Thanks.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Do., 22. Okt.
2015 um 14:57 Uhr:

> I've added user-pointer objects. Some feedback I had off-list:
>
> * add functions to the API for fast vector access as it currently
> requires too much boxing/unboxing of integers for doing
> length/indexing stuff.
>

This would make sense. Same with things like car and cdr etc.
However, we should be careful not to overload the module interface. It
might make sense to do some basic measurements using a real-life example
and optimize only the hotspots.


> * expose fast read-only string access (not sure how do-able this is
> considering the encoding problem)
>

If the internal Emacs encoding is not defined, then we probably shouldn't
expose it to modules. What should modules do with such string access? They
could only treat the contents as opaque byte arrays.


> * consider NULL as a no-op finalizer.
>
> Philip, you obviously have more experienced in this memory handling
> stuff so I'm going to trust you and merge what you did (but there are
> merge conflicts on the last pull requests atm, havent looked into it).
> Thanks again for your work :)
>

You're welcome, thanks for implementing the module stuff in the first
place! I'm not really experienced either, I've simply implemented what
Daniel has proposed (a simple list of blocks).
I've merged your branch and pushed to my fork, so you should be able to
merge one of the PRs now.


>
> This module API is slowly but surely materializing. The interface is
> lot more clean and user-friendly than my initial plan. I hope we can
> have good performances where it matters to make it really useful :)
>

[-- Attachment #2: Type: text/html, Size: 2469 bytes --]

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

On 10/22/2015 03:49 PM, Philipp Stephani wrote:
> 
> 
> Aurélien Aptel <aurelien.aptel+emacs@gmail.com
> <mailto:aurelien.aptel%2Bemacs@gmail.com>> schrieb am Do., 22. Okt. 2015
> um 14:57 Uhr:
> 
>     I've added user-pointer objects. Some feedback I had off-list:
> 
>     * add functions to the API for fast vector access as it currently
>     requires too much boxing/unboxing of integers for doing
>     length/indexing stuff.
> 
> 
> This would make sense. Same with things like car and cdr etc.
> However, we should be careful not to overload the module interface. It
> might make sense to do some basic measurements using a real-life example
> and optimize only the hotspots.

Agreed.

>  
> 
>     * expose fast read-only string access (not sure how do-able this is
>     considering the encoding problem)
> 
> 
> If the internal Emacs encoding is not defined, then we probably
> shouldn't expose it to modules. What should modules do with such string
> access? They could only treat the contents as opaque byte arrays.

Agreed, although there may be a reasonable case for an API that provides
direct access to strings that happen to be ASCII (or UTF-8) and that
runs a conversion otherwise, all transparently.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
>  Tom Tromey <tom@tromey.com>, Eli Zaretskii <eliz@gnu.org>,
>  Emacs development discussions <emacs-devel@gnu.org>
> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 22 Oct 2015 15:52:58 -0700
> 
> >     * expose fast read-only string access (not sure how do-able this is
> >     considering the encoding problem)
> > 
> > 
> > If the internal Emacs encoding is not defined, then we probably
> > shouldn't expose it to modules. What should modules do with such string
> > access? They could only treat the contents as opaque byte arrays.
> 
> Agreed, although there may be a reasonable case for an API that provides
> direct access to strings that happen to be ASCII (or UTF-8) and that
> runs a conversion otherwise, all transparently.

Not sure what you mean here.  First, the internal representation is a
superset of UTF-8, so if we can provide direct access to UTF-8, why
not let modules access the internal representation?

Second, if by "UTF-8" you mean a unibyte string encoded in strict
UTF-8, then why limit that to UTF-8?  Standard C runtime can handle
many different encodings, including multibyte ones, it is not limited
to UTF-8 (although UTF-8 is easier for certain tasks).  Therefore, I
think access to unibyte strings (a.k.a. "byte streams") should not be
limited at all, and letting modules access multibyte strings in their
internal representation would be a significant bonus we shouldn't
easily give up on.

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

On 10/23/2015 12:05 AM, Eli Zaretskii wrote:
>> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
>>  Tom Tromey <tom@tromey.com>, Eli Zaretskii <eliz@gnu.org>,
>>  Emacs development discussions <emacs-devel@gnu.org>
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Thu, 22 Oct 2015 15:52:58 -0700
>>
>>>     * expose fast read-only string access (not sure how do-able this is
>>>     considering the encoding problem)
>>>
>>>
>>> If the internal Emacs encoding is not defined, then we probably
>>> shouldn't expose it to modules. What should modules do with such string
>>> access? They could only treat the contents as opaque byte arrays.
>>
>> Agreed, although there may be a reasonable case for an API that provides
>> direct access to strings that happen to be ASCII (or UTF-8) and that
>> runs a conversion otherwise, all transparently.
> 
> Not sure what you mean here.  First, the internal representation is a
> superset of UTF-8, so if we can provide direct access to UTF-8, why
> not let modules access the internal representation?


You're right. There are few downsides to providing direct access
provided we document that strings are usually, but not always, strict UTF-8.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 22 Oct 2015 22:49:01 +0000
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, Daniel Colascione <dancol@dancol.org>, 
> 	Tom Tromey <tom@tromey.com>, Eli Zaretskii <eliz@gnu.org>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
>     * expose fast read-only string access (not sure how do-able this is
>     considering the encoding problem)
>     
> If the internal Emacs encoding is not defined, then we probably shouldn't
> expose it to modules.

It is well defined.

> What should modules do with such string access? They could only
> treat the contents as opaque byte arrays.

That'd mean high-performance string processing in modules will be
impossible, because you'd need either to use Lisp interfaces (and
create a Lisp string for them each time you call them), or
encode/decode each string before using it with libc functions.  That'd
be a pity, I think.

Re: Dynamic loading progress

[Aurélien Aptel]

Philip, I've tried to merge you latest PR but I have a problem with
module compiled with -fsanitize-address. This compiles the module with
ASan aka AddressSanitizer on GCC. You've enable it to debug memory
problems I guess and it looks useful but I cannot load them.

[*] ./modt-error: running make
gcc -fsanitize=address -ggdb3 -Wall -I../../src -fPIC -c modt-error.c
gcc -shared -fsanitize=address -o modt-error.so modt-error.o
../../lib-src/make-docfile modt-error.c > modt-error.doc
rm modt-error.o
[*] ./modt-error: running test
../src/emacs -batch -L . -l ert -l ./modt-error/test.el -f
ert-run-tests-batch-and-exit
==24402==ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it
with LD_PRELOAD.
[E] ./modt-error: test failed

I'm sure I'm missing something obvious...

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am So., 25. Okt.
2015 um 17:26 Uhr:

> Philip, I've tried to merge you latest PR but I have a problem with
> module compiled with -fsanitize-address. This compiles the module with
> ASan aka AddressSanitizer on GCC. You've enable it to debug memory
> problems I guess and it looks useful but I cannot load them.
>
> [*] ./modt-error: running make
> gcc -fsanitize=address -ggdb3 -Wall -I../../src -fPIC -c modt-error.c
> gcc -shared -fsanitize=address -o modt-error.so modt-error.o
> ../../lib-src/make-docfile modt-error.c > modt-error.doc
> rm modt-error.o
> [*] ./modt-error: running test
> ../src/emacs -batch -L . -l ert -l ./modt-error/test.el -f
> ert-run-tests-batch-and-exit
> ==24402==ASan runtime does not come first in initial library list; you
> should either link runtime to your application or manually preload it
> with LD_PRELOAD.
> [E] ./modt-error: test failed
>
> I'm sure I'm missing something obvious...
>

I mainly use Clang, which is almost, but not entirely compatible with GCC.
If the flag causes problems for you, just drop it, of course it's not
required.

[-- Attachment #2: Type: text/html, Size: 1506 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

On Sun, Oct 25, 2015 at 5:31 PM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> I mainly use Clang, which is almost, but not entirely compatible with GCC.
> If the flag causes problems for you, just drop it, of course it's not
> required.

Sure, but it looks useful so I'd like to make it work.

In any case I've merged you PR and I've finished the `load'
integration, which makes `require' works now.

`Fload' does a lot of stuff and I'm short-circuiting a lot of it for
modules. I would like some review on this [1] specifically src/lread.c
changes. Do we need to do things before the function returns? I'm
guessing we should close the fd returned by `openp' but if you grep
for openp usage in the emacs codebase you'll notice some occurence
where they don't even store the fd...

https://github.com/aaptel/emacs-dynamic-module/commit/dc5cc38a50ea31b0d64c6664352c3bce88121411

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am So., 25. Okt.
2015 um 19:45 Uhr:

> On Sun, Oct 25, 2015 at 5:31 PM, Philipp Stephani <p.stephani2@gmail.com>
> wrote:
> > I mainly use Clang, which is almost, but not entirely compatible with
> GCC.
> > If the flag causes problems for you, just drop it, of course it's not
> > required.
>
> Sure, but it looks useful so I'd like to make it work.
>

I haven't investigated in detail what the problem is, but since newer GCCs
support ASan, that should be doable.


>
> In any case I've merged you PR and I've finished the `load'
> integration, which makes `require' works now.
>

Cool!

[-- Attachment #2: Type: text/html, Size: 1201 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

One more thing: I'd suggest to unintern module-call and module-refs-hash,
like we do with internal-interpreter-environment. They are implementation
details that users can't meaningfully work with; in the case of
module-call, users have no way to pass the needed environment pointer, and
modifying module-refs-hash would be quite harmful.

[-- Attachment #2: Type: text/html, Size: 368 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

On Sun, Oct 25, 2015 at 9:40 PM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> One more thing: I'd suggest to unintern module-call and module-refs-hash,
> like we do with internal-interpreter-environment. They are implementation
> details that users can't meaningfully work with; in the case of module-call,
> users have no way to pass the needed environment pointer, and modifying
> module-refs-hash would be quite harmful.

Good idea.

Re: Dynamic loading progress

[Aurélien Aptel]

I was thinking of doing a small talk on Emacs module support at FOSDEM
in january. I hope by then we'll be close to done... *cross body
appendices*

Re: Dynamic loading progress

[Ted Zlatanov]

On Wed, 28 Oct 2015 14:47:42 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> I was thinking of doing a small talk on Emacs module support at FOSDEM
AA> in january. I hope by then we'll be close to done... *cross body
AA> appendices*

I've tried to review the discussion so far and believe the code is
"functional enough." I think we got agreement from Stefan (probably John
too) to exempt dynamic modules from the feature freeze.

If you agree, can you please push your branch into Emacs but turn tests
off, so failures don't break the build? Or maybe mark them as soft
failures, if that's possible. I am concerned that otherwise we'll make
it harder and harder to accept the work for the upcoming release, and
that would delay dynamic modules further.  Developing in a feature
branch is good up to a point, but now it's become an impediment because
your work is not visible to a wider audience.

I hope we all agree on this. I can push it if you don't have commit
access; just let me know here or off-list when it's ready.  The commit
message formatting may need adjusting and we may want to squash into
just a few commits, but these are minor details.

Ted

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 06 Nov 2015 10:52:04 -0500
> 
> I think we got agreement from Stefan (probably John too) to exempt
> dynamic modules from the feature freeze.

There's no feature freeze yet.

Re: Dynamic loading progress

[Ted Zlatanov]

On Fri, 06 Nov 2015 17:55:22 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Fri, 06 Nov 2015 10:52:04 -0500
>> 
>> I think we got agreement from Stefan (probably John too) to exempt
>> dynamic modules from the feature freeze.

EZ> There's no feature freeze yet.

I could swear I saw a message about it.  Well, it's good to know, but
the rest of my message still applies :)

Ted

Feature freezes and Emacs 25 (was: Dynamic loading progress)

[John Wiegley]

Since Emacs 25 is not yet released, the transfer of maintainership comes at a
slightly awkward time. I haven't fully caught my breath yet, but don't want to
impede Stefan's legacy from getting out the door.

Let us officially freeze in one weeks, next Friday, Nov 13 at 11:59 PM UTC, so
that any ready, last minute features can get in before 25.1 is closed. Then
we'll take a pause and review whether anything should be allowed after the
freeze. Once we're comfortable with the bug situation, we release.

I'd like to take a somewhat generous period for bug fixes, with feature work
happening in the meanwhile on topic branches. All severe bugs should either be
closed, or consciously deferred, before letting 25 out the door.

John

Re: Feature freezes and Emacs 25

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

> Since Emacs 25 is not yet released, the transfer of maintainership comes at a
> slightly awkward time. I haven't fully caught my breath yet, but don't want to
> impede Stefan's legacy from getting out the door.
>
> Let us officially freeze in one weeks, next Friday, Nov 13 at 11:59 PM
> UTC, so that any ready, last minute features can get in before 25.1 is
> closed.

That sounds like a free-for-all.  With the development cycles of Emacs,
rushing in last-minute features might come at costs of half a year.

I'd rather suggest putting down a "soft freeze" right now where anything
significantly featuresome requires list/maintainer approval.  In
essence, this is vaguely the state Stefan handed over: I considered this
somewhat fuzzily the prevailing mood to be picked up from discussions on
the list.

-- 
David Kastrup

Re: Feature freezes and Emacs 25

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 07 Nov 2015 09:32:36 +0100
> 
> > Let us officially freeze in one weeks, next Friday, Nov 13 at 11:59 PM
> > UTC, so that any ready, last minute features can get in before 25.1 is
> > closed.
> 
> That sounds like a free-for-all.  With the development cycles of Emacs,
> rushing in last-minute features might come at costs of half a year.

Why is that a problem?  We are in no rush to have another major
release.  On the contrary, IMO having a major release with no major
new features is not a good idea.  Emacs 25.1 shouldn't be anything
like 24.6 could have been.

Re: Feature freezes and Emacs 25 (was: Dynamic loading progress)

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 06 Nov 2015 20:53:19 -0500
> 
> Let us officially freeze in one weeks, next Friday, Nov 13 at 11:59 PM UTC, so
> that any ready, last minute features can get in before 25.1 is closed. Then
> we'll take a pause and review whether anything should be allowed after the
> freeze. Once we're comfortable with the bug situation, we release.

I think we should first decide whether new features that are "almost
ready" should or should not be in 25.1.  Two examples that immediately
come to mind are dynamic loading and xwidgets; I don't know if there
are others.  After we make the decision, we could see if one week's
time is enough.

> I'd like to take a somewhat generous period for bug fixes, with feature work
> happening in the meanwhile on topic branches. All severe bugs should either be
> closed, or consciously deferred, before letting 25 out the door.

We used to cut a release branch fairly early in the pretest ,
certainly at least when the first pretest tarball is put on
alpha.gnu.org.  Does the above mean you want the pretest to be worked
on on master instead?

Re: Feature freezes and Emacs 25

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I think we should first decide whether new features that are "almost ready"
> should or should not be in 25.1. Two examples that immediately come to mind
> are dynamic loading and xwidgets; I don't know if there are others. After we
> make the decision, we could see if one week's time is enough.

I'd like to see both of those in 25.1.

Let's freeze this Friday, and take a step back. We have several courses open:

 1. Decide that what we have now should become 24.6.

 2. Decide it will become 25.1.

 3. Decide that we want to bring in more features that aren't yet complete,
    and so push the freeze to a future date.

> We used to cut a release branch fairly early in the pretest , certainly at
> least when the first pretest tarball is put on alpha.gnu.org. Does the above
> mean you want the pretest to be worked on on master instead?

I'd actually like to know more about our current release flow, before having
that discussion here. I don't know yet what I want in that respect.

John

Re: Feature freezes and Emacs 25

[Dmitry Gutov]

On 11/09/2015 11:55 PM, John Wiegley wrote:
> Let's freeze this Friday, and take a step back. We have several courses open:
>
>   1. Decide that what we have now should become 24.6.

That doesn't seem to be an option, simply considering that master has 
been 25.0.25 for quite a while.

>   2. Decide it will become 25.1.
>
>   3. Decide that we want to bring in more features that aren't yet complete,
>      and so push the freeze to a future date.

I'd like to see this happen, maybe for a few weeks, at least for xref.el 
(and maybe project.el, too). There are still unresolved questions and 
issues there, and not enough people participating in the design.

By the way, John, would you join the "project.el semantics" discussion? 
I've pinged you there.

Re: Feature freezes and Emacs 25

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Let's freeze this Friday, and take a step back. We have several courses open:
  > >
  > >   1. Decide that what we have now should become 24.6.

  > That doesn't seem to be an option, simply considering that master has 
  > been 25.0.25 for quite a while.

We don't have to be governed by that.
It's easy enough to make a branch for 24.6.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Feature freezes and Emacs 25

[Dmitry Gutov]

On 11/10/2015 08:17 PM, Richard Stallman wrote:

> We don't have to be governed by that.
> It's easy enough to make a branch for 24.6.

Yes, if you're prepared to ignore the third-party code that has Emacs 
version checks, declared dependencies on Emacs 25.1 in certain packages, 
and existing blog posts "what's going to be in Emacs 25".

Which would be a bad decision, IMO.

Re: Feature freezes and Emacs 25

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, if you're prepared to ignore the third-party code that has Emacs 
  > version checks, declared dependencies on Emacs 25.1 in certain packages, 

It won't be harder to change those than it was to insert them.

  > and existing blog posts "what's going to be in Emacs 25".

They won't be wrong -- anything installed already will still be in
Emacs 25.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Feature freezes and Emacs 25

[Xue Fuqiao]

On Tue, Nov 10, 2015 at 5:55 AM, John Wiegley <jwiegley@gmail.com> wrote:

> Let's freeze this Friday, and take a step back. We have several courses open:
>
>  1. Decide that what we have now should become 24.6.

There is no plan for a new Emacs 24 release, unless some critical
security vulnerabilities arise and need urgent fixing.

>  2. Decide it will become 25.1.
>
>  3. Decide that we want to bring in more features that aren't yet complete,
>     and so push the freeze to a future date.
>
>> We used to cut a release branch fairly early in the pretest , certainly at
>> least when the first pretest tarball is put on alpha.gnu.org. Does the above
>> mean you want the pretest to be worked on on master instead?
>
> I'd actually like to know more about our current release flow, before having
> that discussion here. I don't know yet what I want in that respect.

In case you don't know, here is the new versioning scheme for GNU Emacs:

https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00872.html

Actually, I think we should document our release process more
explicitly.  `admin/FOR-RELEASE' and `admin/notes/versioning' are not
enough.

Re: Feature freezes and Emacs 25

[John Wiegley]

>>>>> Xue Fuqiao <xfq.free@gmail.com> writes:

> In case you don't know, here is the new versioning scheme for GNU Emacs:
> https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00872.html

Ok, the freeze is for 25.1.

> Actually, I think we should document our release process more
> explicitly.  `admin/FOR-RELEASE' and `admin/notes/versioning' are not
> enough.

I agree. Would you be willing to contribute some drafting? I myself am still
unfamiliar with all of our procedures in this regard.

John

Re: Feature freezes and Emacs 25

[Xue Fuqiao]

On Tue, Nov 10, 2015 at 8:07 AM, John Wiegley <jwiegley@gmail.com> wrote:
>>>>>> Xue Fuqiao <xfq.free@gmail.com> writes:
>
>> In case you don't know, here is the new versioning scheme for GNU Emacs:
>> https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00872.html
>
> Ok, the freeze is for 25.1.
>
>> Actually, I think we should document our release process more
>> explicitly.  `admin/FOR-RELEASE' and `admin/notes/versioning' are not
>> enough.
>
> I agree. Would you be willing to contribute some drafting? I myself am still
> unfamiliar with all of our procedures in this regard.

OK.  I'll write a draft tomorrow (or later today).

Re: Feature freezes and Emacs 25

[Xue Fuqiao]

[-- Attachment #1: Type: text/plain, Size: 3806 bytes --]

>>> Actually, I think we should document our release process more
>>> explicitly.  `admin/FOR-RELEASE' and `admin/notes/versioning' are not
>>> enough.
>>
>> I agree. Would you be willing to contribute some drafting? I myself am still
>> unfamiliar with all of our procedures in this regard.

I just wrote a draft document for our release process.  Please see the
attached patches.  I'm not quite familiar with the release process
either, and the process was written merely by impression.  I think
Nicolas and Glenn are more familiar with the release process, so I just
added them to the Cc list.  Comments are warmly welcomed.

The first patch renamed `admin/FOR-RELEASE' to `admin/RELEASE-PROCESS',
and removed the trailing whitespace[1].  The second patch documents the
release process in `admin/RELEASE-PROCESS'.

There are things I would like to write, but didn't write for various
reasons:

1. Document when to remove obsolete features.  For example, "if a
   feature is declared obsolete in version `major.minor', it will
   continue to work in versions `major.minor' and `major+1.minor' but
   raise warnings, and it will be removed in version `major+2.minor'".
   I don't think we have a policy for removing obsolete features
   currently, but IMHO it would be good to have one.

2. Document what to do if a bug needs to be addressed in the next
   release (a.k.a. release-critical bugs).  For example, how to set the
   bug meta-data for release-critical bugs, what meta-data to set, the
   criteria for release-critical bugs, etc.  I didn't write this because
   I don't know how.  Patches/suggestions are very welcome.

3. Document the number of pretest and RC releases we need to make for a
   release, or how to decide the number pretest/RC releases for a
   release.  I didn't document this because I don't know the policy
   about this (if any).

4. Document how to coordinate with the release schedule of packages like
   Org and Gnus.  I didn't write this because I'm not familiar with the
   process.  Again, patches/suggestions are very welcome.

5. Add information about uploading the Windows binary to
   https://ftp.gnu.org/gnu/emacs/windows/, in `admin/make-tarball.txt'.
   I didn't write this because I'm not familiar with the process.
   Patches are welcome.

6. Document the criteria for feature freeze, i.e., when should we freeze
   the master branch?  I didn't write this because we don't have a
   policy for this currently.

7. Things related to people.  Such as:
   * Who is in charge of a release?
   * Who may make a good candidate for an RM (release manager)?

In addition, I have some suggestions about our current release process
and versioning scheme:

* IMHO the current status of Emacs development should be present
  somewhere, for those who don't follow emacs-devel closely.  Something
  like this would be a good start:
  https://wiki.documentfoundation.org/ReleasePlan/5.1#5.1.0_release

* I think we need to encourage developers to prioritize bugs during
  phase two and/or phase three in the release cycle (see my patches for
  the three phases).  It will make the quality of the new release
  better, or at least give developers an overview of the general
  development (bugfix) direction.

* Currently, the version of an RC release (returned by
  `(emacs-version)') is the same as a stable release, but as a user,
  sometimes I can't tell if I'm using a stable version of Emacs or a
  release candidate.  I still think something like 25.1-rc1 is better
  than 25.1, because it's clearer.

The release process I wrote applies mainly to a new major release (such
as 25.1) instead of a minor release (such as 24.5).  So the documents
still need expanding.  Comments?

[1] My `pre-commit' hook won't let me commit if I don't remove the
trailing whitespace ;-)

[-- Attachment #2: 0001-admin-RELEASE-PROCESS-Rename-from-admin-FOR-RELEASE.patch --]
[-- Type: application/octet-stream, Size: 18088 bytes --]

From 912b6e01bcea785928642fc6240fbb7d4bc1ea31 Mon Sep 17 00:00:00 2001
From: Xue Fuqiao <xfq.free@gmail.com>
Date: Tue, 10 Nov 2015 16:55:05 +0800
Subject: [PATCH 1/2] * admin/RELEASE-PROCESS: Rename from admin/FOR-RELEASE.

---
 admin/FOR-RELEASE     | 282 --------------------------------------------------
 admin/RELEASE-PROCESS | 282 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 282 insertions(+), 282 deletions(-)
 delete mode 100644 admin/FOR-RELEASE
 create mode 100644 admin/RELEASE-PROCESS

diff --git a/admin/FOR-RELEASE b/admin/FOR-RELEASE
deleted file mode 100644
index 6ecec89..0000000
--- a/admin/FOR-RELEASE
+++ /dev/null
@@ -1,282 +0,0 @@
-Tasks needed before the next release.
-
-* TO BE DONE SHORTLY BEFORE RELEASE
-
-** Make sure the necessary sources and scripts for any generated files
-are included in the source tarfile.  (They don't need to be installed,
-so eg admin/ is fine.)
-
-** Manuals
-Check for node names using problematic characters:
-  find doc -name '*.texi' -exec grep '^@node[^,]*[:.()]' {} +
-Sadly makeinfo does not warn about such characters.
-
-Check for major new features added since the last release (e.g. new
-lisp files), and add the relevant authors to the Acknowledgments in
-doc/emacs/ack.texi and emacs.texi.
-
-Check cross-references between the manuals (eg from emacs to elisp)
-are correct.  You can use something like the following in the info
-directory in the Emacs build tree:
-
-emacs -Q --eval "(progn (require 'info) (setq Info-directory-list '(\".\")))" \
-  -f info-xref-check-all
-
-Setting Info-directory-list avoids having system info pages confuse
-things.  References to external manuals will be flagged as
-uncheckable.  You should still check these, and also that each
-external manual has an appropriate redirect in the file manual/.htaccess
-in the web pages repository.  E.g.:
-Redirect /software/emacs/manual/html_mono/automake.html /software/automake/manual/automake.html
-Redirect /software/emacs/manual/html_node/automake/ /software/automake/manual/html_node/
-
-Another tool you can use to check links is gnu.org's linc.py:
-http://www.gnu.org/server/source/
-
-You run this something like:
-
-cd /path/to/cvs/emacs-www
-linc.py -o /path/to/output-dir --url http://www.gnu.org/software/emacs/ .
-
-Be warned that it is really, really slow (as in, can take ~ a full day
-to check the manual/ directory).  It is probably best to run it on a
-single directory at a time from eg manual/html_node.  It is very
-inefficient, but may reveal a few things that info-xref does not.
-
-
-make emacs.dvi, elisp.dvi, and deal with any errors (undefined
-references etc) in the output.  Break any overfull lines.
-Underfull hboxes are not serious, but it can be nice to get rid of
-them if a simple rephrasing or rearrangement will work.
-
-Update the master menu and detailed menu (eg the antinews version).
-The command texinfo-multiple-files-update can do this, but you
-probably want to apply the results selectively (eg the current master
-menu has better line-breaks than the automatic version).  It includes
-the menu-entry name (if there is one) as well as the node name - using
-only the latter looks better.  Also, it doesn't seem to handle nested
-includes, so will miss edebug.texi etc.
-
-Check for widow and orphan lines in the printed manual; make sure all
-the pages really look ok in the manual as formatted.  Orphans/widows
-are cases where the first/last line of a paragraph is on its own at
-the end/start of a page, or where the last word in a paragraph is on
-its own at the start of a line.  It looks better if you reword/respace
-things to avoid these.  (AFAIK, there is no way to find these except
-paging through the whole manual.)  This should be the very last thing
-you do, since any change can alter the layout.
-(Actually, there is probably little point in trying to do this.
-It's only really relevant if printed versions of the manuals are going
-to be published.  End-users are not likely to print out all 1000+
-pages of the manuals, and even if they do, the resulting page breaks
-depend on what paper and font size they use.  This also means that if
-you _are_ going to do this, it should be done with the paper and font
-size that the GNU Press are going to use when they print the manuals.
-I think this is different to what you get if you just use eg 'make
-emacs.pdf' (e.g., enable "smallbook").
-
-** Check the keybindings in the refcards are correct, and add any new ones.
-What paper size are the English versions supposed to be on?
-On Debian testing, the packages texlive-lang-czechslovak and
-texlive-lang-polish will let you generate the cs-* and sk-* pdfs.
-(You may need texlive-lang-cyrillic, texlive-lang-german for others.)
-The Makefile rules did not work for me, I had to use something like:
-csplain -output-format=pdf cs-refcard
-
-** Ask maintainers of refcard translations to update them.
-
-Emacs 22 translators:
-
-LANG	Translator            Status
-cs	Pavel Janík           
-de	Sven Joachim          
-fr	Eric Jacoboni         
-pl	Włodek Bzyl           
-pt-br	Rodrigo Real          
-ru	Alex Ott              
-sk	Miroslav Vaško        
-
-** For a major release, add a "New in Emacs XX" section to faq.texi.
-
-** Remove temporary +++/--- lines in NEWS.
-
-** Try to reorder NEWS: most important things first, related items together.
-
-** Consider bumping customize-changed-options-previous-release.
-
-** cusver-check from admin.el can help find new defcustoms missing
-:version tags.
-
-** Add a line to etc/HISTORY for the release version number and date.
-
-* BUGS
-
-** Check for modes which bind M-s that conflicts with a new global binding M-s
-and change key bindings where necessary.  The current list of modes:
-
-1. Gnus binds 'M-s' to 'gnus-summary-search-article-forward'.
-
-2. Minibuffer binds 'M-s' to 'next-matching-history-element'
-   (not useful any more since C-s can now search in the history).
-
-3. 'center-line' in Text mode was already moved to the text formatting
-   keymap as 'M-o M-s' (thus this binding is not necessary any more
-   in 'nroff-mode-map' too and can be removed now from the nroff mode
-   because it can now use the global key binding 'M-o M-s' 'center-line').
-
-4. PCL-CVS binds 'M-s' to 'cvs-status', and log-edit-mode binds it to
-   'log-edit-comment-search-forward'.  Perhaps search commands
-   on the global key binding 'M-s' are useless in these modes.
-
-5. Rmail binds '\es' to 'rmail-search'/'rmail-summary-search'.
-
-
-* DOCUMENTATION
-
-** Check the Emacs Tutorial.
-
-The first line of every tutorial must begin with text ending in a
-period (".", ASCII 0x2E) saying "Emacs Tutorial" in the respective
-language. This should be followed by "See end for copying conditions",
-likewise in the respective language.
-
-After each file name, on the same line or the following line, come the
-names of the people who have checked it.
-
-SECTION                  READERS
-----------------------------------
-TUTORIAL             cyd
-TUTORIAL.bg          ogi
-TUTORIAL.cn          xfq
-TUTORIAL.cs          
-TUTORIAL.de          wl
-TUTORIAL.eo
-TUTORIAL.es          
-TUTORIAL.fr          
-TUTORIAL.he          eliz
-TUTORIAL.it          
-TUTORIAL.ja          
-TUTORIAL.ko
-TUTORIAL.nl	     Pieter Schoenmakers
-TUTORIAL.pl          
-TUTORIAL.pt_BR       
-TUTORIAL.ro
-TUTORIAL.ru          Alex Ott
-TUTORIAL.sk          
-TUTORIAL.sl          Primoz PETERLIN
-TUTORIAL.sv          Mats Lidell
-TUTORIAL.th          
-TUTORIAL.zh
-
-** Check the manual.
-
-abbrevs.texi      
-ack.texi          
-anti.texi         
-arevert-xtra.texi 
-basic.texi        
-buffers.texi      
-building.texi     
-calendar.texi     
-cal-xtra.texi     
-cmdargs.texi      
-commands.texi     
-custom.texi       
-dired.texi        
-dired-xtra.texi   
-display.texi      
-emacs.texi        
-emacs-xtra.texi   
-emerge-xtra.texi  
-entering.texi     
-files.texi        
-fixit.texi        
-fortran-xtra.texi 
-frames.texi       
-glossary.texi     
-help.texi         
-indent.texi       
-killing.texi      
-kmacro.texi       
-macos.texi        
-maintaining.texi  
-mark.texi         
-mini.texi         
-misc.texi         
-modes.texi        
-msdos.texi        
-msdos-xtra.texi   
-mule.texi         
-m-x.texi          
-package.texi      
-picture-xtra.texi 
-programs.texi     
-regs.texi         
-rmail.texi        
-screen.texi       
-search.texi       
-sending.texi      
-text.texi         
-trouble.texi      
-vc-xtra.texi      
-vc1-xtra.texi     
-windows.texi      
-xresources.texi   
-
-** Check the Lisp manual.
-
-abbrevs.texi      
-anti.texi         
-back.texi         
-backups.texi      
-buffers.texi      
-commands.texi     
-compile.texi      
-control.texi      
-customize.texi    
-debugging.texi    
-display.texi      
-edebug.texi       
-elisp.texi
-errors.texi       
-eval.texi         
-files.texi        
-frames.texi       
-functions.texi    
-hash.texi         
-help.texi         
-hooks.texi        
-index.texi
-internals.texi    
-intro.texi        
-keymaps.texi      
-lists.texi        
-loading.texi      
-macros.texi       
-maps.texi         
-markers.texi      
-minibuf.texi      
-modes.texi        
-nonascii.texi     
-numbers.texi      Paul Eggert (24.4)
-objects.texi      
-os.texi           
-package.texi      
-positions.texi    
-processes.texi    
-searching.texi    
-sequences.texi    
-streams.texi      
-strings.texi      
-symbols.texi      
-syntax.texi       
-text.texi         
-tips.texi         
-variables.texi    
-windows.texi      
-
-\f
-Local variables:
-mode: outline
-coding: utf-8
-end:
diff --git a/admin/RELEASE-PROCESS b/admin/RELEASE-PROCESS
new file mode 100644
index 0000000..286b90e
--- /dev/null
+++ b/admin/RELEASE-PROCESS
@@ -0,0 +1,282 @@
+Tasks needed before the next release.
+
+* TO BE DONE SHORTLY BEFORE RELEASE
+
+** Make sure the necessary sources and scripts for any generated files
+are included in the source tarfile.  (They don't need to be installed,
+so eg admin/ is fine.)
+
+** Manuals
+Check for node names using problematic characters:
+  find doc -name '*.texi' -exec grep '^@node[^,]*[:.()]' {} +
+Sadly makeinfo does not warn about such characters.
+
+Check for major new features added since the last release (e.g. new
+lisp files), and add the relevant authors to the Acknowledgments in
+doc/emacs/ack.texi and emacs.texi.
+
+Check cross-references between the manuals (eg from emacs to elisp)
+are correct.  You can use something like the following in the info
+directory in the Emacs build tree:
+
+emacs -Q --eval "(progn (require 'info) (setq Info-directory-list '(\".\")))" \
+  -f info-xref-check-all
+
+Setting Info-directory-list avoids having system info pages confuse
+things.  References to external manuals will be flagged as
+uncheckable.  You should still check these, and also that each
+external manual has an appropriate redirect in the file manual/.htaccess
+in the web pages repository.  E.g.:
+Redirect /software/emacs/manual/html_mono/automake.html /software/automake/manual/automake.html
+Redirect /software/emacs/manual/html_node/automake/ /software/automake/manual/html_node/
+
+Another tool you can use to check links is gnu.org's linc.py:
+http://www.gnu.org/server/source/
+
+You run this something like:
+
+cd /path/to/cvs/emacs-www
+linc.py -o /path/to/output-dir --url http://www.gnu.org/software/emacs/ .
+
+Be warned that it is really, really slow (as in, can take ~ a full day
+to check the manual/ directory).  It is probably best to run it on a
+single directory at a time from eg manual/html_node.  It is very
+inefficient, but may reveal a few things that info-xref does not.
+
+
+make emacs.dvi, elisp.dvi, and deal with any errors (undefined
+references etc) in the output.  Break any overfull lines.
+Underfull hboxes are not serious, but it can be nice to get rid of
+them if a simple rephrasing or rearrangement will work.
+
+Update the master menu and detailed menu (eg the antinews version).
+The command texinfo-multiple-files-update can do this, but you
+probably want to apply the results selectively (eg the current master
+menu has better line-breaks than the automatic version).  It includes
+the menu-entry name (if there is one) as well as the node name - using
+only the latter looks better.  Also, it doesn't seem to handle nested
+includes, so will miss edebug.texi etc.
+
+Check for widow and orphan lines in the printed manual; make sure all
+the pages really look ok in the manual as formatted.  Orphans/widows
+are cases where the first/last line of a paragraph is on its own at
+the end/start of a page, or where the last word in a paragraph is on
+its own at the start of a line.  It looks better if you reword/respace
+things to avoid these.  (AFAIK, there is no way to find these except
+paging through the whole manual.)  This should be the very last thing
+you do, since any change can alter the layout.
+(Actually, there is probably little point in trying to do this.
+It's only really relevant if printed versions of the manuals are going
+to be published.  End-users are not likely to print out all 1000+
+pages of the manuals, and even if they do, the resulting page breaks
+depend on what paper and font size they use.  This also means that if
+you _are_ going to do this, it should be done with the paper and font
+size that the GNU Press are going to use when they print the manuals.
+I think this is different to what you get if you just use eg 'make
+emacs.pdf' (e.g., enable "smallbook").
+
+** Check the keybindings in the refcards are correct, and add any new ones.
+What paper size are the English versions supposed to be on?
+On Debian testing, the packages texlive-lang-czechslovak and
+texlive-lang-polish will let you generate the cs-* and sk-* pdfs.
+(You may need texlive-lang-cyrillic, texlive-lang-german for others.)
+The Makefile rules did not work for me, I had to use something like:
+csplain -output-format=pdf cs-refcard
+
+** Ask maintainers of refcard translations to update them.
+
+Emacs 22 translators:
+
+LANG	Translator            Status
+cs	Pavel Janík
+de	Sven Joachim
+fr	Eric Jacoboni
+pl	Włodek Bzyl
+pt-br	Rodrigo Real
+ru	Alex Ott
+sk	Miroslav Vaško
+
+** For a major release, add a "New in Emacs XX" section to faq.texi.
+
+** Remove temporary +++/--- lines in NEWS.
+
+** Try to reorder NEWS: most important things first, related items together.
+
+** Consider bumping customize-changed-options-previous-release.
+
+** cusver-check from admin.el can help find new defcustoms missing
+:version tags.
+
+** Add a line to etc/HISTORY for the release version number and date.
+
+* BUGS
+
+** Check for modes which bind M-s that conflicts with a new global binding M-s
+and change key bindings where necessary.  The current list of modes:
+
+1. Gnus binds 'M-s' to 'gnus-summary-search-article-forward'.
+
+2. Minibuffer binds 'M-s' to 'next-matching-history-element'
+   (not useful any more since C-s can now search in the history).
+
+3. 'center-line' in Text mode was already moved to the text formatting
+   keymap as 'M-o M-s' (thus this binding is not necessary any more
+   in 'nroff-mode-map' too and can be removed now from the nroff mode
+   because it can now use the global key binding 'M-o M-s' 'center-line').
+
+4. PCL-CVS binds 'M-s' to 'cvs-status', and log-edit-mode binds it to
+   'log-edit-comment-search-forward'.  Perhaps search commands
+   on the global key binding 'M-s' are useless in these modes.
+
+5. Rmail binds '\es' to 'rmail-search'/'rmail-summary-search'.
+
+
+* DOCUMENTATION
+
+** Check the Emacs Tutorial.
+
+The first line of every tutorial must begin with text ending in a
+period (".", ASCII 0x2E) saying "Emacs Tutorial" in the respective
+language. This should be followed by "See end for copying conditions",
+likewise in the respective language.
+
+After each file name, on the same line or the following line, come the
+names of the people who have checked it.
+
+SECTION                  READERS
+----------------------------------
+TUTORIAL             cyd
+TUTORIAL.bg          ogi
+TUTORIAL.cn          xfq
+TUTORIAL.cs
+TUTORIAL.de          wl
+TUTORIAL.eo
+TUTORIAL.es
+TUTORIAL.fr
+TUTORIAL.he          eliz
+TUTORIAL.it
+TUTORIAL.ja
+TUTORIAL.ko
+TUTORIAL.nl	     Pieter Schoenmakers
+TUTORIAL.pl
+TUTORIAL.pt_BR
+TUTORIAL.ro
+TUTORIAL.ru          Alex Ott
+TUTORIAL.sk
+TUTORIAL.sl          Primoz PETERLIN
+TUTORIAL.sv          Mats Lidell
+TUTORIAL.th
+TUTORIAL.zh
+
+** Check the manual.
+
+abbrevs.texi
+ack.texi
+anti.texi
+arevert-xtra.texi
+basic.texi
+buffers.texi
+building.texi
+calendar.texi
+cal-xtra.texi
+cmdargs.texi
+commands.texi
+custom.texi
+dired.texi
+dired-xtra.texi
+display.texi
+emacs.texi
+emacs-xtra.texi
+emerge-xtra.texi
+entering.texi
+files.texi
+fixit.texi
+fortran-xtra.texi
+frames.texi
+glossary.texi
+help.texi
+indent.texi
+killing.texi
+kmacro.texi
+macos.texi
+maintaining.texi
+mark.texi
+mini.texi
+misc.texi
+modes.texi
+msdos.texi
+msdos-xtra.texi
+mule.texi
+m-x.texi
+package.texi
+picture-xtra.texi
+programs.texi
+regs.texi
+rmail.texi
+screen.texi
+search.texi
+sending.texi
+text.texi
+trouble.texi
+vc-xtra.texi
+vc1-xtra.texi
+windows.texi
+xresources.texi
+
+** Check the Lisp manual.
+
+abbrevs.texi
+anti.texi
+back.texi
+backups.texi
+buffers.texi
+commands.texi
+compile.texi
+control.texi
+customize.texi
+debugging.texi
+display.texi
+edebug.texi
+elisp.texi
+errors.texi
+eval.texi
+files.texi
+frames.texi
+functions.texi
+hash.texi
+help.texi
+hooks.texi
+index.texi
+internals.texi
+intro.texi
+keymaps.texi
+lists.texi
+loading.texi
+macros.texi
+maps.texi
+markers.texi
+minibuf.texi
+modes.texi
+nonascii.texi
+numbers.texi      Paul Eggert (24.4)
+objects.texi
+os.texi
+package.texi
+positions.texi
+processes.texi
+searching.texi
+sequences.texi
+streams.texi
+strings.texi
+symbols.texi
+syntax.texi
+text.texi
+tips.texi
+variables.texi
+windows.texi
+
+\f
+Local variables:
+mode: outline
+coding: utf-8
+end:
-- 
2.6.3


[-- Attachment #3: 0002-Document-the-release-process.patch --]
[-- Type: application/octet-stream, Size: 4943 bytes --]

From 56de66dfc2547b58ec97829cea873de9653736e3 Mon Sep 17 00:00:00 2001
From: Xue Fuqiao <xfq.free@gmail.com>
Date: Tue, 10 Nov 2015 17:46:51 +0800
Subject: [PATCH 2/2] Document the release process

* admin/notes/versioning: Add information about RC releases.

* admin/RELEASE-PROCESS: Document the release process.

* admin/authors.el (authors-ignored-files):
* admin/README: Change FOR-RELEASE to RELEASE-PROCESS.
---
 admin/README           |  4 ++--
 admin/RELEASE-PROCESS  | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 admin/authors.el       |  2 +-
 admin/notes/versioning | 11 +++++++----
 4 files changed, 54 insertions(+), 8 deletions(-)

diff --git a/admin/README b/admin/README
index 2286e35..22c72f4 100644
--- a/admin/README
+++ b/admin/README
@@ -12,9 +12,9 @@ what you do when using them.
 
 * Instructions and scripts used to prepare an Emacs release.
 
-** FOR-RELEASE
+** RELEASE-PROCESS
 
-Living list of activities that must be completed before the next release.
+The release process used by GNU Emacs.
 
 ** make-tarball.txt
 
diff --git a/admin/RELEASE-PROCESS b/admin/RELEASE-PROCESS
index 286b90e..672cab4 100644
--- a/admin/RELEASE-PROCESS
+++ b/admin/RELEASE-PROCESS
@@ -1,7 +1,43 @@
-Tasks needed before the next release.
+This document describes the release process used by GNU Emacs.
+
+* RELEASE CYCLE
+
+Each release cycle will be split into three periods.
+
+** Phase one: development
+
+The first phase of the release schedule is the "heads-down" working
+period for new features.
+
+** Phase two: feature freeze
+
+Shortly before the feature freeze, Emacs developers will be devoted to
+figuring out what features to include in the next release and what
+features to defer to a later release.
+
+During the feature freeze, no new features will be accepted in
+`master'.  It ensures that sufficient share of the release cycle is
+dedicated to QA.
+
+When the feature freeze is over, a release branch called "emacs-NN"
+("NN" represents the major version number of the new Emacs release)
+will be cut from `master'.
+
+** Phase three: bugfixes
+
+The last phase of a release cycle is spent fixing bugs and eliminating
+undocumented new features on the "emacs-NN" branch.
+
+In parallel to this phase, `master' can receive new features, to be
+released in the next release cycle.  From time to time, the master
+branches merges bugfix commits from the "emacs-NN" branch.
 
 * TO BE DONE SHORTLY BEFORE RELEASE
 
+** Make sure the Copyright date reflects the current year in the source
+files.  See `admin/notes/years' for information about maintaining
+copyright years for GNU Emacs.
+
 ** Make sure the necessary sources and scripts for any generated files
 are included in the source tarfile.  (They don't need to be installed,
 so eg admin/ is fine.)
@@ -275,6 +311,13 @@ tips.texi
 variables.texi
 windows.texi
 
+* OTHER INFORMATION
+
+For Emacs's versioning scheme, see `admin/notes/versioning'.
+
+For instructions to create pretest or release tarballs, announcements,
+etc., see `admin/make-tarball.txt'.
+
 \f
 Local variables:
 mode: outline
diff --git a/admin/authors.el b/admin/authors.el
index 3d7850a..f5d23be 100644
--- a/admin/authors.el
+++ b/admin/authors.el
@@ -267,7 +267,7 @@ authors-ignored-files
   '("external-lisp"
     "lock" "share-lib" "local-lisp"
     "noleim-Makefile.in"
-    "NEWS" "ORDERS" "PROBLEMS" "FAQ" "AUTHORS" "FOR-RELEASE" "TODO" "todo"
+    "NEWS" "ORDERS" "PROBLEMS" "FAQ" "AUTHORS" "RELEASE-PROCESS" "TODO" "todo"
     "MACHINES" "SERVICE"
     "README.unicode" "README.multi-tty" "TUTORIAL.translators"
     "NEWS.unicode" "COPYING.DJ" "Makefile.old" "Makefile.am"
diff --git a/admin/notes/versioning b/admin/notes/versioning
index e422b22..e956a68 100644
--- a/admin/notes/versioning
+++ b/admin/notes/versioning
@@ -9,16 +9,19 @@ Emacs version numbers have the form
 "build" increments each time Emacs is built in the same location
 (without cleaning) and isn't really part of the version.
 
-bugfix releases increase "minor" by 1.
-non-bugfix releases increase "major" by 1, and reset "minor" to 1.
+Bugfix releases increase "minor" by 1.
+Non-bugfix releases increase "major" by 1, and reset "minor" to 1.
 (The division between bugfix and non-bugfix has not always been clear
 historically.)
 
 Unreleased (development) versions have an extra "devel" component.
 This is a fairly meaningless number that may be unchanged for a long time.
 It is normally 50.
-When the release process starts, it changes to 90, 91, ...
-When the actual release is made, this component is removed.
+
+After we cut the release branch, we’ll make pretest and release
+candidate (RC) releases.  For pretest releases, the "devel" component
+changes to 90, 91, ...  When the first RC release is made, this
+component is removed.
 
 The development version for a new major release has "minor" = 0.
 The development version for a new minor release has "minor" = that of
-- 
2.6.3

Re: Feature freezes and Emacs 25

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

Xue Fuqiao <xfq.free@gmail.com> writes:

> I just wrote a draft document for our release process.  Please see the
> attached patches.  I'm not quite familiar with the release process
> either, and the process was written merely by impression.  I think
> Nicolas and Glenn are more familiar with the release process, so I just
> added them to the Cc list.  Comments are warmly welcomed.

Thanks, I'll take the time to review your patch shortly.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Feature freezes and Emacs 25

[Eli Zaretskii]

> Date: Tue, 10 Nov 2015 21:25:33 +0800
> From: Xue Fuqiao <xfq.free@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, nicolas@petton.fr, rgm@gnu.org
> 
> I just wrote a draft document for our release process.  Please see the
> attached patches.

Thanks!

> The first patch renamed `admin/FOR-RELEASE' to `admin/RELEASE-PROCESS',

Please grep all the repository for references to FOR-RELEASE.  At
least admin/notes/bugtracker still does.

> 1. Document when to remove obsolete features.  For example, "if a
>    feature is declared obsolete in version `major.minor', it will
>    continue to work in versions `major.minor' and `major+1.minor' but
>    raise warnings, and it will be removed in version `major+2.minor'".
>    I don't think we have a policy for removing obsolete features
>    currently, but IMHO it would be good to have one.

I'm not sure a fixed policy of this kind is possible.  Minor features
can be removed quickly, not-so-minor ones not so quickly.

> 2. Document what to do if a bug needs to be addressed in the next
>    release (a.k.a. release-critical bugs).  For example, how to set the
>    bug meta-data for release-critical bugs, what meta-data to set, the
>    criteria for release-critical bugs, etc.  I didn't write this because
>    I don't know how.  Patches/suggestions are very welcome.

  http://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00226.html

> 3. Document the number of pretest and RC releases we need to make for a
>    release, or how to decide the number pretest/RC releases for a
>    release.  I didn't document this because I don't know the policy
>    about this (if any).

There is no policy, and I don't think there could be.  It depends on
the number of bugs reported during the pretest, which in turn depends
on how many radically new features were added.  E.g., Emacs 21.1 had
something like 15 pretests, IIRC.

> * IMHO the current status of Emacs development should be present
>   somewhere, for those who don't follow emacs-devel closely.  Something
>   like this would be a good start:
>   https://wiki.documentfoundation.org/ReleasePlan/5.1#5.1.0_release

That's not status, that's plan.  See the discussion about cadence
model for what I think about that in conjunction with Emacs.

> * I think we need to encourage developers to prioritize bugs during
>   phase two and/or phase three in the release cycle (see my patches for
>   the three phases).  It will make the quality of the new release
>   better, or at least give developers an overview of the general
>   development (bugfix) direction.

Not sure what you mean by "prioritize bugs".  Do you mean priorities
among bugs?  That will only help if there are enough people who work
on fixing bugs, and we want to employ those resources more
efficiently.

> * Currently, the version of an RC release (returned by
>   `(emacs-version)') is the same as a stable release, but as a user,
>   sometimes I can't tell if I'm using a stable version of Emacs or a
>   release candidate.  I still think something like 25.1-rc1 is better
>   than 25.1, because it's clearer.

Actually, we almost never made release candidates.

> [1] My `pre-commit' hook won't let me commit if I don't remove the
> trailing whitespace ;-)

There's the --no-verify switch to "git commit".

> +** Phase two: feature freeze

I think we should discuss this and understand better why this is
needed and what are its entry and exit conditions.

> +Shortly before the feature freeze, Emacs developers will be devoted to
> +figuring out what features to include in the next release and what
> +features to defer to a later release.

If the feature is already on master, we don't have any easy way of
removing it.  Nor should we, IMO.

> +After we cut the release branch, we’ll make pretest and release
> +candidate (RC) releases.  For pretest releases, the "devel" component
> +changes to 90, 91, ...

The number of the first pretest version is not carved in stone.  It
depends on the developers' estimation of how close we are to the
release.  E.g., Emacs 21.1 had its first pretest called 21.0.70,
AFAIR.

Thanks again for working on this.

RE: Feature freezes and Emacs 25

[Drew Adams]

> > 1. Document when to remove obsolete features.  For example, "if a
> >    feature is declared obsolete in version `major.minor', it will
> >    continue to work in versions `major.minor' and `major+1.minor' but
> >    raise warnings, and it will be removed in version `major+2.minor'".
> >    I don't think we have a policy for removing obsolete features
> >    currently, but IMHO it would be good to have one.
> 
> I'm not sure a fixed policy of this kind is possible.  Minor features
> can be removed quickly, not-so-minor ones not so quickly.

There should _never_ be such a policy to remove an obsolete feature.
Each deprecation, and each subsequent removal of its support, should
be handled on a case-by-case basis.  Anything else is madness.

Re: Feature freezes and Emacs 25

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I think we should wait years to remove obsolete features.  Hurrying
would increase the hassle this causes for users who like them,
and we can afford to wait.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Feature freezes and Emacs 25

[Xue Fuqiao]

[-- Attachment #1: Type: text/plain, Size: 4351 bytes --]

On Wed, Nov 11, 2015 at 1:50 AM, Eli Zaretskii <eliz@gnu.org> wrote:

Hi Eli,

> Please grep all the repository for references to FOR-RELEASE.  At
> least admin/notes/bugtracker still does.

Fixed, thanks.

I removed the reference to FOR-RELEASE in admin/notes/bugtracker,
because I think all bugs found should be in the bug tracker, instead of
having some of them in the RELEASE-PROCESS file.

>> 1. Document when to remove obsolete features.  For example, "if a
>>    feature is declared obsolete in version `major.minor', it will
>>    continue to work in versions `major.minor' and `major+1.minor' but
>>    raise warnings, and it will be removed in version `major+2.minor'".
>>    I don't think we have a policy for removing obsolete features
>>    currently, but IMHO it would be good to have one.
>
> I'm not sure a fixed policy of this kind is possible.  Minor features
> can be removed quickly, not-so-minor ones not so quickly.

OK.

>> 2. Document what to do if a bug needs to be addressed in the next
>>    release (a.k.a. release-critical bugs).  For example, how to set the
>>    bug meta-data for release-critical bugs, what meta-data to set, the
>>    criteria for release-critical bugs, etc.  I didn't write this because
>>    I don't know how.  Patches/suggestions are very welcome.
>
>   http://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00226.html

Thanks.  I've made some changes about this in the new patch (attached).
Furthermore, I hope we can get rid of the BUGS section in
RELEASE-PROCESS and move it entirely to Debbugs.

(BTW, I hope we can file a meta-bug, listing all release-tracking bugs
like #19758 and #19759, and update it in each release, so that we don't
have to update the RELEASE-CRITICAL BUGS section in every future
release, and we can see our previous release-tracking bugs more
conveniently.)

>> * IMHO the current status of Emacs development should be present
>>   somewhere, for those who don't follow emacs-devel closely.  Something
>>   like this would be a good start:
>>   https://wiki.documentfoundation.org/ReleasePlan/5.1#5.1.0_release
>
> That's not status, that's plan.  See the discussion about cadence
> model for what I think about that in conjunction with Emacs.

OK, I'll see.

>> * I think we need to encourage developers to prioritize bugs during
>>   phase two and/or phase three in the release cycle (see my patches for
>>   the three phases).  It will make the quality of the new release
>>   better, or at least give developers an overview of the general
>>   development (bugfix) direction.
>
> Not sure what you mean by "prioritize bugs".

I mean, adding severity level for bugs and adding blocking bugs.  So
the chance of important bugs missed in the new release will be reduced.
This needs to be done in every phase, but is even more important in the
bugfix phase.

>> [1] My `pre-commit' hook won't let me commit if I don't remove the
>> trailing whitespace ;-)
>
> There's the --no-verify switch to "git commit".

Thanks for the hint Eli - you are right as usual.

>> +** Phase two: feature freeze
>
> I think we should discuss this and understand better why this is
> needed and what are its entry and exit conditions.

I agree.  John (and others)?  Any comment on this?

>> +Shortly before the feature freeze, Emacs developers will be devoted to
>> +figuring out what features to include in the next release and what
>> +features to defer to a later release.
>
> If the feature is already on master, we don't have any easy way of
> removing it.  Nor should we, IMO.

That's not what I meant.  I meant the long-term feature branches
(concurrency/dynamic-modules/xwidget), and features still under
discussion (xref/project?).

>> +After we cut the release branch, we’ll make pretest and release
>> +candidate (RC) releases.  For pretest releases, the "devel" component
>> +changes to 90, 91, ...
>
> The number of the first pretest version is not carved in stone.  It
> depends on the developers' estimation of how close we are to the
> release.  E.g., Emacs 21.1 had its first pretest called 21.0.70,
> AFAIR.

But IMHO having a consistent versioning scheme for pretests should
better in the future.  The estimation is somewhat subjective.

Thanks for your review!

[-- Attachment #2: 0002-Document-the-release-process.patch --]
[-- Type: application/octet-stream, Size: 6437 bytes --]

From 43f3db250aeea5c81556f7f624dcab87889c20e5 Mon Sep 17 00:00:00 2001
From: Xue Fuqiao <xfq.free@gmail.com>
Date: Tue, 10 Nov 2015 17:46:51 +0800
Subject: [PATCH 2/2] Document the release process

* admin/notes/versioning: Add information about RC releases.

* admin/RELEASE-PROCESS: Document the release process.

* admin/authors.el (authors-ignored-files):
* admin/README:
* CONTRIBUTE: Change FOR-RELEASE to RELEASE-PROCESS.
---
 CONTRIBUTE             |  2 +-
 admin/README           |  4 ++--
 admin/RELEASE-PROCESS  | 58 +++++++++++++++++++++++++++++++++++++++++++++++++-
 admin/authors.el       |  2 +-
 admin/notes/bugtracker |  3 +--
 admin/notes/versioning | 11 ++++++----
 6 files changed, 69 insertions(+), 11 deletions(-)

diff --git a/CONTRIBUTE b/CONTRIBUTE
index 0ca5d0d..db1e35c 100644
--- a/CONTRIBUTE
+++ b/CONTRIBUTE
@@ -145,7 +145,7 @@ messages:
   will suffice.
 
 - There is no need to mention files such as NEWS, MAINTAINERS, and
-  FOR-RELEASE, or to indicate regeneration of files such as
+  RELEASE-PROCESS, or to indicate regeneration of files such as
   'configure', in the ChangeLog entry.  "There is no need" means you
   don't have to, but you can if you want to.
 
diff --git a/admin/README b/admin/README
index 2286e35..22c72f4 100644
--- a/admin/README
+++ b/admin/README
@@ -12,9 +12,9 @@ what you do when using them.
 
 * Instructions and scripts used to prepare an Emacs release.
 
-** FOR-RELEASE
+** RELEASE-PROCESS
 
-Living list of activities that must be completed before the next release.
+The release process used by GNU Emacs.
 
 ** make-tarball.txt
 
diff --git a/admin/RELEASE-PROCESS b/admin/RELEASE-PROCESS
index 286b90e..cf59184 100644
--- a/admin/RELEASE-PROCESS
+++ b/admin/RELEASE-PROCESS
@@ -1,7 +1,56 @@
-Tasks needed before the next release.
+This document describes the release process used by GNU Emacs.
+
+* RELEASE CYCLE
+
+Each release cycle will be split into three periods.
+
+** Phase one: development
+
+The first phase of the release schedule is the "heads-down" working
+period for new features.
+
+** Phase two: feature freeze
+
+Shortly before the feature freeze, Emacs developers will be devoted to
+figuring out what features to include in the next release and what
+features to defer to a later release.
+
+During the feature freeze, no new features will be accepted in
+`master'.  It ensures that sufficient share of the release cycle is
+dedicated to QA.
+
+When the feature freeze is over, a release branch called "emacs-NN"
+("NN" represents the major version number of the new Emacs release)
+will be cut from `master'.
+
+** Phase three: bugfixes
+
+The last phase of a release cycle is spent fixing bugs and eliminating
+undocumented new features on the "emacs-NN" branch.
+
+In parallel to this phase, `master' can receive new features, to be
+released in the next release cycle.  From time to time, the master
+branches merges bugfix commits from the "emacs-NN" branch.
+
+* RELEASE-CRITICAL BUGS
+
+Emacs uses the "blocking bug(s)" feature of Debbugs for bugs need to
+be addressed in the next release.
+
+Currently, bug#19759 is the tracking bug for release of 25.1.  Say
+bug#123 needs to be fixed for Emacs 25.1.  Send a message to
+control@debbugs.gnu.org that says:
+
+   block 19759 by 123
+
+Change "block" to "unblock" to unblock the bug.
 
 * TO BE DONE SHORTLY BEFORE RELEASE
 
+** Make sure the Copyright date reflects the current year in the source
+files.  See `admin/notes/years' for information about maintaining
+copyright years for GNU Emacs.
+
 ** Make sure the necessary sources and scripts for any generated files
 are included in the source tarfile.  (They don't need to be installed,
 so eg admin/ is fine.)
@@ -275,6 +324,13 @@ tips.texi
 variables.texi
 windows.texi
 
+* OTHER INFORMATION
+
+For Emacs's versioning scheme, see `admin/notes/versioning'.
+
+For instructions to create pretest or release tarballs, announcements,
+etc., see `admin/make-tarball.txt'.
+
 \f
 Local variables:
 mode: outline
diff --git a/admin/authors.el b/admin/authors.el
index 3d7850a..f5d23be 100644
--- a/admin/authors.el
+++ b/admin/authors.el
@@ -267,7 +267,7 @@ authors-ignored-files
   '("external-lisp"
     "lock" "share-lib" "local-lisp"
     "noleim-Makefile.in"
-    "NEWS" "ORDERS" "PROBLEMS" "FAQ" "AUTHORS" "FOR-RELEASE" "TODO" "todo"
+    "NEWS" "ORDERS" "PROBLEMS" "FAQ" "AUTHORS" "RELEASE-PROCESS" "TODO" "todo"
     "MACHINES" "SERVICE"
     "README.unicode" "README.multi-tty" "TUTORIAL.translators"
     "NEWS.unicode" "COPYING.DJ" "Makefile.old" "Makefile.am"
diff --git a/admin/notes/bugtracker b/admin/notes/bugtracker
index fb65bbe..3d6df03 100644
--- a/admin/notes/bugtracker
+++ b/admin/notes/bugtracker
@@ -140,8 +140,7 @@ you can add an element to gnus-posting-styles to do this automatically, eg:
 
 ** To record a bug in the tracker without sending mail to the bug list.
 This can be useful to make a note of something discussed on
-emacs-devel that needs fixing.  In other words, this can be the
-equivalent of adding something to FOR-RELEASE.
+emacs-devel that needs fixing.
 
 To: quiet@debbugs.gnu.org
 [headers end]
diff --git a/admin/notes/versioning b/admin/notes/versioning
index e422b22..e956a68 100644
--- a/admin/notes/versioning
+++ b/admin/notes/versioning
@@ -9,16 +9,19 @@ Emacs version numbers have the form
 "build" increments each time Emacs is built in the same location
 (without cleaning) and isn't really part of the version.
 
-bugfix releases increase "minor" by 1.
-non-bugfix releases increase "major" by 1, and reset "minor" to 1.
+Bugfix releases increase "minor" by 1.
+Non-bugfix releases increase "major" by 1, and reset "minor" to 1.
 (The division between bugfix and non-bugfix has not always been clear
 historically.)
 
 Unreleased (development) versions have an extra "devel" component.
 This is a fairly meaningless number that may be unchanged for a long time.
 It is normally 50.
-When the release process starts, it changes to 90, 91, ...
-When the actual release is made, this component is removed.
+
+After we cut the release branch, we’ll make pretest and release
+candidate (RC) releases.  For pretest releases, the "devel" component
+changes to 90, 91, ...  When the first RC release is made, this
+component is removed.
 
 The development version for a new major release has "minor" = 0.
 The development version for a new minor release has "minor" = that of
-- 
2.6.3

Re: Feature freezes and Emacs 25

[Glenn Morris]

Xue Fuqiao wrote:

> I just wrote a draft document for our release process.  Please see the
> attached patches.  I'm not quite familiar with the release process
> either, and the process was written merely by impression.  I think
> Nicolas and Glenn are more familiar with the release process, so I just
> added them to the Cc list.  Comments are warmly welcomed.

I haven't read the draft, so the following comments are only on your email.

> The first patch renamed `admin/FOR-RELEASE' to `admin/RELEASE-PROCESS',
> and removed the trailing whitespace[1].

Does the (long-standing) name of the file really make a difference to anything?
If you are going to rename it, you don't have to stick with an
old-school SHOUTY name, you can use lower-case.


In the following, there is no existing policy in many cases.
IMO trying to document existing policy and create policy should be
separate activities. Also I would resist the temptation to have a policy
for _everything_.

> 1. Document when to remove obsolete features.

There's no existing policy, and this seems separate from making releases?
The unofficial one is something like "there should normally be at least
one major release between making something obsolete and removing it".
(Personally I go with two.)

> 2. Document what to do if a bug needs to be addressed in the next
>    release (a.k.a. release-critical bugs). 

Fix it?

>  For example, how to set the bug meta-data for release-critical bugs,
>  what meta-data to set,

In the past we increased the severity of such bugs.
Personally I felt this was a bad idea,

http://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00196.html

so I unilaterally started using "block" for this.
That thread explains how it works.

No one else has shown any interest in tracking such things AFAIK.

> 3. Document the number of pretest and RC releases we need to make for a
>    release, or how to decide the number pretest/RC releases for a
>    release.

The "policy" is that there are as many pretests as needed until the code
is of sufficient quality to release. A new pretest is made whenever
there have been sufficient changes from the last one to warrant it.

Normally, there is one RC, unless an unexpected last-minute problem occurs.

> 4. Document how to coordinate with the release schedule of packages like
>    Org and Gnus.

There is no policy. Personally I don't think one is needed.

> 5. Add information about uploading the Windows binary to
>    https://ftp.gnu.org/gnu/emacs/windows/, in `admin/make-tarball.txt'.

In exactly the same way anything else is uploaded to ftp.gnu.org?

> 6. Document the criteria for feature freeze, i.e., when should we freeze
>    the master branch?

This is up the lead maintainer(s).

> 7. Things related to people.  Such as:
>    * Who is in charge of a release?

Whoever the lead maintainer(s) says is.

>    * Who may make a good candidate for an RM (release manager)?

The one person who volunteers?

Sorry, I'm more cynical than you! :)

> * I think we need to encourage developers to prioritize bugs during
>   phase two and/or phase three in the release cycle (see my patches for
>   the three phases).  It will make the quality of the new release
>   better, or at least give developers an overview of the general
>   development (bugfix) direction.

Obviously we've always tried to do that. Good luck!

> * Currently, the version of an RC release (returned by
>   `(emacs-version)') is the same as a stable release, but as a user,
>   sometimes I can't tell if I'm using a stable version of Emacs or a
>   release candidate.  I still think something like 25.1-rc1 is better
>   than 25.1, because it's clearer.

The whole point of an RC is that the tarfile is literally identical to
the release. If all goes as planned, it IS the release. Therefore it has
to have the same version number. Otherwise it's just another pretest.
Personally I have never found this a source of confusion.

Re: Feature freezes and Emacs 25

[Xue Fuqiao]

On Thu, Nov 12, 2015 at 3:59 AM, Glenn Morris <rgm@gnu.org> wrote:

Hi Glenn,

> Does the (long-standing) name of the file really make a difference to anything?

Hmm...  Personally, I think "for release" is just a list of activities
that must be completed before the next release, but "release process" or
"release cycle" is throughout Emacs development.  Since English is not
my native language, my feeling could be wrong.

> If you are going to rename it, you don't have to stick with an
> old-school SHOUTY name, you can use lower-case.

OK.  I can change it if no one disagrees.

>>  For example, how to set the bug meta-data for release-critical bugs,
>>  what meta-data to set,
>
> In the past we increased the severity of such bugs.
> Personally I felt this was a bad idea,
>
> http://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00196.html
>
> so I unilaterally started using "block" for this.
> That thread explains how it works.
>
> No one else has shown any interest in tracking such things AFAIK.

Perhaps many developers aren't aware of it.  (I'm an example.)

>> 5. Add information about uploading the Windows binary to
>>    https://ftp.gnu.org/gnu/emacs/windows/, in `admin/make-tarball.txt'.
>
> In exactly the same way anything else is uploaded to ftp.gnu.org?

Well, I mean _build and upload_ the binary.  Maybe Someone™ familiar
with Windows should write a script to build a Windows binary for release
automatically.

Re: Feature freezes and Emacs 25

[Glenn Morris]

Xue Fuqiao wrote:

> Hmm...  Personally, I think "for release" is just a list of activities
> that must be completed before the next release, but "release process" or
> "release cycle" is throughout Emacs development. 

Fair enough.

>> No one else has shown any interest in tracking such things AFAIK.
>
> Perhaps many developers aren't aware of it.  (I'm an example.)

AFAIK, nobody thought to even ask "how shall we track the bugs we need
to fix for the next release". Till now! :)

> Well, I mean _build and upload_ the binary.  Maybe Someone™ familiar
> with Windows should write a script to build a Windows binary for release
> automatically.

Perhaps. Personally I wouldn't like to see "provide pre-built binaries
for MS Windows users" becomes an "official" responsibility of Emacs.
It's just something that some people have been doing on a best effort basis.
But of course the people who do the work get to make the decisions.

Re: Feature freezes and Emacs 25

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

On Sun, Nov 15, 2015 at 3:42 AM, Glenn Morris <rgm@gnu.org> wrote:

> Perhaps. Personally I wouldn't like to see "provide pre-built binaries
> for MS Windows users" becomes an "official" responsibility of Emacs.
> It's just something that some people have been doing on a best effort
basis.
> But of course the people who do the work get to make the decisions.

Which just reminds me...

I've been building 32-bit, non-opt, checking-and-debugging-enabled binary
snapshots from trunk (and will do now from emacs-25), but as I put then in
my Dropbox and I don't want the hassle to have to put source tarballs too,
I haven't publicised them at all, just to interested parties.

I'd be willing to upload them to some place a bit more "official", if that
would be useful.

[-- Attachment #2: Type: text/html, Size: 1000 bytes --]

Re: Feature freezes and Emacs 25

[John Wiegley]

>>>>> Glenn Morris <rgm@gnu.org> writes:

> AFAIK, nobody thought to even ask "how shall we track the bugs we need to
> fix for the next release". Till now! :)

I had suggested using a release tag for 25.1 in debbugs. Is that not possible?
At some time soon we'll need to start assessing existing bugs and applying
that tag to them.

John

Re: Feature freezes and Emacs 25

[Glenn Morris]

John Wiegley wrote:

[I see you use Mail-Followup-To, which results in looking like I'm
writing to someone else.]

>>>>>> Glenn Morris <rgm@gnu.org> writes:
>
>> AFAIK, nobody thought to even ask "how shall we track the bugs we need to
>> fix for the next release". Till now! :)
>
> I had suggested using a release tag for 25.1 in debbugs. Is that not
> possible? At some time soon we'll need to start assessing existing
> bugs and applying that tag to them.

I started tracking 25.1 bugs in February, since we've always known the
next release is coming, and the regressions and security issues were
already piling up.

http://debbugs.gnu.org/19759 is my (partial) selection,
I don't claim it's exhaustive.

You can use a usertag if you prefer, but personally I think
block/unblock is better.

IMO now is the time to start tracking 25.2 (or whatever master is to
be released as) issues as well. The "list of issues that need to be
fixed before release" should always be being updated, else it's easy
for items to get lost.

Re: Feature freezes and Emacs 25

[Glenn Morris]

Glenn Morris wrote:

> http://debbugs.gnu.org/19759 is my (partial) selection,

PS does not include standard tasks (see whatever FOR-RELEASE is called
now), like reviewing NEWS and documenting everything that needs it, and
proof-reading every chapter of the manual and lispref.

Re: Feature freezes and Emacs 25

[John Wiegley]

>>>>> Glenn Morris <rgm@gnu.org> writes:

> [I see you use Mail-Followup-To, which results in looking like I'm
> writing to someone else.]

What is the proper method to use?

> You can use a usertag if you prefer, but personally I think block/unblock is
> better.

Ok, block/unblock sounds fine.

> IMO now is the time to start tracking 25.2 (or whatever master is to be
> released as) issues as well. The "list of issues that need to be fixed
> before release" should always be being updated, else it's easy for items to
> get lost.

Agreed.

John

Re: Feature freezes and Emacs 25

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

John Wiegley <jwiegley@gmail.com> writes:

>> Actually, I think we should document our release process more
>> explicitly.  `admin/FOR-RELEASE' and `admin/notes/versioning' are not
>> enough.

admin/make-tarball.txt is also very useful.

>
> I agree. Would you be willing to contribute some drafting? I myself am still
> unfamiliar with all of our procedures in this regard.

I can take care of building and publishing the various pretest, release
candidate and stable tarballs, as well as updating the website for the
release, like I did for Emacs 24.5.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Feature freezes and Emacs 25

[John Wiegley]

>>>>> Nicolas Petton <nicolas@petton.fr> writes:

> I can take care of building and publishing the various pretest, release
> candidate and stable tarballs, as well as updating the website for the
> release, like I did for Emacs 24.5.

Thank you, Nicolas! As you do so, can you keep a journal of the exact steps
you follow, so I can learn to repeat what you're about to do?

John

Re: Feature freezes and Emacs 25

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Xue Fuqiao <xfq.free@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,  Emacs-devel <emacs-devel@gnu.org>
> Date: Tue, 10 Nov 2015 06:22:37 -0800
> 
> >>>>> Nicolas Petton <nicolas@petton.fr> writes:
> 
> > I can take care of building and publishing the various pretest, release
> > candidate and stable tarballs, as well as updating the website for the
> > release, like I did for Emacs 24.5.
> 
> Thank you, Nicolas! As you do so, can you keep a journal of the exact steps
> you follow, so I can learn to repeat what you're about to do?

I think the steps are already documented in admin/make-tarball.txt and
admin/FOR-RELEASE.

Re: Feature freezes and Emacs 25

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Thank you, Nicolas! As you do so, can you keep a journal of the exact steps
>> you follow, so I can learn to repeat what you're about to do?
>
> I think the steps are already documented in admin/make-tarball.txt and
> admin/FOR-RELEASE.

These are exactly the steps I'm following.  It's all very well
documented. With the initial help of Glenn last time I was able to
learned very quickly how to build the tarballs and test them.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Feature freezes and Emacs 25

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Nicolas Petton <nicolas@petton.fr> writes:
>
>> I can take care of building and publishing the various pretest, release
>> candidate and stable tarballs, as well as updating the website for the
>> release, like I did for Emacs 24.5.
>
> Thank you, Nicolas!

You're welcome!

> As you do so, can you keep a journal of the exact steps
> you follow, so I can learn to repeat what you're about to do?

Sure, and as I said it's for the most part already well documented :)

BTW, even if we freeze Friday, I guess there won't be a pretest this
week? (I will be offline for a few days).

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Feature freezes and Emacs 25

[Eli Zaretskii]

> From: Nicolas Petton <nicolas@petton.fr>
> Cc: Xue Fuqiao <xfq.free@gmail.com>, Eli Zaretskii <eliz@gnu.org>, Emacs-devel <emacs-devel@gnu.org>
> Date: Tue, 10 Nov 2015 18:39:43 +0100
> 
> BTW, even if we freeze Friday, I guess there won't be a pretest this
> week? (I will be offline for a few days).

IMO, the pretest should be tarred something like 2 weeks after either
the freeze or the branch, to let people who didn't actively use the
master version start using it and fix any grave bugs.

Re: Feature freezes and Emacs 25

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 266 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> IMO, the pretest should be tarred something like 2 weeks after either
> the freeze or the branch, to let people who didn't actively use the
> master version start using it and fix any grave bugs.

Ok, that works for me.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Feature freezes and Emacs 25

[Eli Zaretskii]

> From: Nicolas Petton <nicolas@petton.fr>
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-devel <emacs-devel@gnu.org>
> Date: Tue, 10 Nov 2015 10:41:56 +0100
> 
> I can take care of building and publishing the various pretest, release
> candidate and stable tarballs, as well as updating the website for the
> release, like I did for Emacs 24.5.

Thank you!

Re: Feature freezes and Emacs 25

[Juri Linkov]

>> I think we should first decide whether new features that are "almost ready"
>> should or should not be in 25.1. Two examples that immediately come to mind
>> are dynamic loading and xwidgets; I don't know if there are others. After we
>> make the decision, we could see if one week's time is enough.
>
> I'd like to see both of those in 25.1.
>
> Let's freeze this Friday, and take a step back. We have several courses open:
>
>  1. Decide that what we have now should become 24.6.
>
>  2. Decide it will become 25.1.
>
>  3. Decide that we want to bring in more features that aren't yet complete,
>     and so push the freeze to a future date.

John, could you please extend the deadline of the feature freeze a little
since I have no chance to commit all changes until Friday.

Basically what I'm doing is the same kind of work as Alan's efforts in bug#17453.
While Alan is designing a framework to support multiple windows
by adding a new arg with a group of windows, I'm designing a framework
to support multiple regions by adding a new arg with a group of regions.

In bug#19829 I've collected the evidence of commands that need this
feature as a basis for the new design, but the deadlock was caused by the
same question that backpedaled the Alan's design: whether to add a new arg
to the existing functions or to try stuffing it into the existing ones?

Re: Feature freezes and Emacs 25

[John Wiegley]

>>>>> Juri Linkov <juri@linkov.net> writes:

> John, could you please extend the deadline of the feature freeze a little
> since I have no chance to commit all changes until Friday.
> 
> Basically what I'm doing is the same kind of work as Alan's efforts in
> bug#17453. While Alan is designing a framework to support multiple windows
> by adding a new arg with a group of windows, I'm designing a framework to
> support multiple regions by adding a new arg with a group of regions.
> 
> In bug#19829 I've collected the evidence of commands that need this feature
> as a basis for the new design, but the deadlock was caused by the same
> question that backpedaled the Alan's design: whether to add a new arg to the
> existing functions or to try stuffing it into the existing ones?

Given the discussion Alan and I are having, maybe we should step back the
three of us and make sure that we're taking a consistent approach.

If your work is similar to Alan's in nature, I'd be willing to extend its
deadline beyond the freeze date as well -- if it doesn't come in too late. I'd
rather not extend the freeze overall, since otherwise we might keep doing it.
Too many cool ideas out there. :)

The freeze is motivation to get 25.1 out the door, so 25.2 can come sooner.

John

Re: Feature freezes and Emacs 25

[Juri Linkov]

>> John, could you please extend the deadline of the feature freeze a little
>> since I have no chance to commit all changes until Friday.
>>
>> Basically what I'm doing is the same kind of work as Alan's efforts in
>> bug#17453. While Alan is designing a framework to support multiple windows
>> by adding a new arg with a group of windows, I'm designing a framework to
>> support multiple regions by adding a new arg with a group of regions.
>>
>> In bug#19829 I've collected the evidence of commands that need this feature
>> as a basis for the new design, but the deadlock was caused by the same
>> question that backpedaled the Alan's design: whether to add a new arg to the
>> existing functions or to try stuffing it into the existing ones?
>
> Given the discussion Alan and I are having, maybe we should step back the
> three of us and make sure that we're taking a consistent approach.
>
> If your work is similar to Alan's in nature, I'd be willing to extend its
> deadline beyond the freeze date as well -- if it doesn't come in too late. I'd
> rather not extend the freeze overall, since otherwise we might keep doing it.
> Too many cool ideas out there. :)
>
> The freeze is motivation to get 25.1 out the door, so 25.2 can come sooner.

Actually I'm happy that you are pushing for the sooner release,
so now we have to concentrate on both these almost finished features.

Also in the next release I'd like to see improvements in the Isearch prompt
listed in the proposal Artur published on EmacsWiki, but it seems it's
impossible to reach consensus in a reasonable time before the next release
due to a wide spectrum of opinions, so perhaps Isearch prompt changes need
to wait for 25.2.

Re: Feature freezes and Emacs 25

[John Wiegley]

>>>>> Juri Linkov <juri@linkov.net> writes:

> Also in the next release I'd like to see improvements in the Isearch prompt
> listed in the proposal Artur published on EmacsWiki, but it seems it's
> impossible to reach consensus in a reasonable time before the next release
> due to a wide spectrum of opinions, so perhaps Isearch prompt changes need
> to wait for 25.2.

I agree. 25.2 won't be that long after 25.1, so this shouldn't sound like a
kiss of death.

John

Re: Feature freezes and Emacs 25

[Artur Malabarba]

Juri Linkov <juri@linkov.net> writes:

> Actually I'm happy that you are pushing for the sooner release,
> so now we have to concentrate on both these almost finished features.
>
> Also in the next release I'd like to see improvements in the Isearch prompt
> listed in the proposal Artur published on EmacsWiki, but it seems it's
> impossible to reach consensus in a reasonable time before the next release
> due to a wide spectrum of opinions, so perhaps Isearch prompt changes need
> to wait for 25.2.

Even if we did postpone the feature-freeze, I'd probably still not apply
the isearch patch until after that.
Call me a buzzkill, but I actually like to know that “sensitive”
features are going to get a good amount of testing before being called
“stable”. 😄

That's twice as valid when the code involves some decisions that we're
going to have to live with afterwards (yes window*-start, I'm looking at
you 😉).

Re: Feature freezes and Emacs 25

[John Wiegley]

>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Call me a buzzkill, but I actually like to know that “sensitive” features
> are going to get a good amount of testing before being called “stable”. 😄

If it's something you care about, consider that motivation to do more testing
during the pre-release cycle. :)

Whatever feature is introduced, until it is released the only ones who will
test it are the developers building from Emacs.git. A pre-release tarball
actually has a much larger audience. And we always have the option of backing
out the change if it receives too much bad feedback.

> That's twice as valid when the code involves some decisions that we're going
> to have to live with afterwards (yes window*-start, I'm looking at you 😉).

Agreed. When it comes to modifying core functionality, we need to proceed very
cautiously.

John

Re: Feature freezes and Emacs 25

[Eli Zaretskii]

> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Date: Thu, 12 Nov 2015 04:01:29 +0000
> Cc: emacs-devel@gnu.org
> 
> > Also in the next release I'd like to see improvements in the Isearch prompt
> > listed in the proposal Artur published on EmacsWiki, but it seems it's
> > impossible to reach consensus in a reasonable time before the next release
> > due to a wide spectrum of opinions, so perhaps Isearch prompt changes need
> > to wait for 25.2.
> 
> Even if we did postpone the feature-freeze, I'd probably still not apply
> the isearch patch until after that.
> Call me a buzzkill, but I actually like to know that “sensitive”
> features are going to get a good amount of testing before being called
> “stable”. 😄
> 
> That's twice as valid when the code involves some decisions that we're
> going to have to live with afterwards (yes window*-start, I'm looking at
> you 😉).

These are all valid considerations, but we should take them in their
correct perspective: with the release some 8 months away (according to
the previous statistics), we have enough time to make sure it's
stable.

Re: Feature freezes and Emacs 25

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 547 bytes --]

> > From: Artur Malabarba <bruce.connor.am@gmail.com>
> > Even if we did postpone the feature-freeze, I'd probably still not apply
> > the isearch patch until after that.
> > Call me a buzzkill, but I actually like to know that “sensitive”
> > features are going to get a good amount of testing before being called
> > “stable”. 😄

Actually, this came out a bit wrong. I meant to say "the sensitive features
that I write". As in, it makes me feel safer. I didn't mean to imply that
other people are doing anything wrong here.

[-- Attachment #2: Type: text/html, Size: 711 bytes --]

Re: Feature freezes and Emacs 25

[Xue Fuqiao]

[-- Attachment #1: Type: text/plain, Size: 194 bytes --]

Hi,

I've changed RELEASE-PROCESS to lower-case and removed the feature
freeze section of my patches.  I'll push it to `master' later.  It can
be tweaked in the repository afterwards if needed.

[-- Attachment #2: Type: text/html, Size: 333 bytes --]

Re: Feature freezes and Emacs 25

[Eli Zaretskii]

> Date: Sun, 15 Nov 2015 09:51:40 +0800
> From: Xue Fuqiao <xfq.free@gmail.com>
> 
> I've changed RELEASE-PROCESS to lower-case and removed the feature
> freeze section of my patches. I'll push it to `master' later.

Thanks, but why to 'master'?  Doesn't the document describe the
process we want to use during the release of Emacs 25.1?  If so, then
its place is on the emacs-25 branch, and since this is documentation
that cannot possibly break anything, there are no dilemmas regarding
its effect on stability.

I thought I wrote a day or two ago that documentation changes should
go to emacs-25, not to master.  (Actually, the default branch for
pushing any work should now be emacs-25, except for new features.)

TIA

Re: Feature freezes and Emacs 25

[Xue Fuqiao]

On Tue, Nov 17, 2015 at 12:12 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sun, 15 Nov 2015 09:51:40 +0800
>> From: Xue Fuqiao <xfq.free@gmail.com>
>>
>> I've changed RELEASE-PROCESS to lower-case and removed the feature
>> freeze section of my patches. I'll push it to `master' later.
>
> Thanks, but why to 'master'?  Doesn't the document describe the
> process we want to use during the release of Emacs 25.1?  If so, then
> its place is on the emacs-25 branch, and since this is documentation
> that cannot possibly break anything, there are no dilemmas regarding
> its effect on stability.

Mea culpa.

I remembered that the admin/ directory was not in the release tarball,
but I just found that I was wrong (it _is_ in my Emacs 24.5 tarball).
And yes, I'll push my not-new-feature commits to the "emacs-25" branch
before the 25.1 release.

What should I do now?  Cherry-pick my commits and add “Backport:” to the
commit log?

Re: Feature freezes and Emacs 25

[Eli Zaretskii]

> Date: Tue, 17 Nov 2015 07:59:57 +0800
> From: Xue Fuqiao <xfq.free@gmail.com>
> Cc: Emacs-devel <emacs-devel@gnu.org>
> 
> > Thanks, but why to 'master'?  Doesn't the document describe the
> > process we want to use during the release of Emacs 25.1?  If so, then
> > its place is on the emacs-25 branch, and since this is documentation
> > that cannot possibly break anything, there are no dilemmas regarding
> > its effect on stability.
> 
> Mea culpa.

No sweat.

> I remembered that the admin/ directory was not in the release tarball,
> but I just found that I was wrong (it _is_ in my Emacs 24.5 tarball).

That shouldn't be a factor.

> What should I do now?  Cherry-pick my commits and add “Backport:” to the
> commit log?

Yes, please.

Thanks.

Re: Feature freezes and Emacs 25

[Xue Fuqiao]

On Tue, Nov 17, 2015 at 11:43 AM, Eli Zaretskii <eliz@gnu.org> wrote:

>> What should I do now?  Cherry-pick my commits and add “Backport:” to the
>> commit log?
>
> Yes, please.

Done.  Thank you.

Design of commands operating on rectangular regions (was: Feature freezes and Emacs 25)

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 221 bytes --]

> If your work is similar to Alan's in nature, I'd be willing to extend its
> deadline beyond the freeze date as well -- if it doesn't come in too late.

Thanks, here is a complete patch from bug#19829 ready to install:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: rectangular_regions.patch --]
[-- Type: text/x-diff, Size: 18390 bytes --]

diff --git a/lisp/simple.el b/lisp/simple.el
index 1f2f4fe..3d09a54 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -970,15 +970,34 @@ (defcustom delete-active-region t
 (defvar region-extract-function
   (lambda (delete)
     (when (region-beginning)
-      (if (eq delete 'delete-only)
-          (delete-region (region-beginning) (region-end))
-        (filter-buffer-substring (region-beginning) (region-end) delete))))
+      (cond
+       ((eq delete 'bounds)
+        (list (cons (region-beginning) (region-end))))
+       ((eq delete 'delete-only)
+        (delete-region (region-beginning) (region-end)))
+       (t
+        (filter-buffer-substring (region-beginning) (region-end) delete)))))
   "Function to get the region's content.
 Called with one argument DELETE.
 If DELETE is `delete-only', then only delete the region and the return value
 is undefined.  If DELETE is nil, just return the content as a string.
+If DELETE is `bounds', then don't delete, but just return the
+bounds of the region as a list of (START . END) boundaries.
 If anything else, delete the region and return its content as a string.")
 
+(defvar region-insert-function
+  (lambda (lines)
+    (let ((first t))
+      (while lines
+        (or first
+            (insert ?\n))
+        (insert-for-yank (car lines))
+        (setq lines (cdr lines)
+              first nil))))
+  "Function to insert the region's content.
+Called with one argument LINES.
+Insert the region as a list of lines.")
+
 (defun delete-backward-char (n &optional killflag)
   "Delete the previous N characters (following if N is negative).
 If Transient Mark mode is enabled, the mark is active, and N is 1,
@@ -3282,7 +3306,8 @@ (defun shell-command-sentinel (process signal)
 
 (defun shell-command-on-region (start end command
 				      &optional output-buffer replace
-				      error-buffer display-error-buffer)
+				      error-buffer display-error-buffer
+				      region-noncontiguous-p)
   "Execute string COMMAND in inferior shell with region as input.
 Normally display output (if any) in temp buffer `*Shell Command Output*';
 Prefix arg means replace the region with it.  Return the exit code of
@@ -3345,7 +3370,8 @@ (defun shell-command-on-region (start end command
 		       current-prefix-arg
 		       current-prefix-arg
 		       shell-command-default-error-buffer
-		       t)))
+		       t
+		       (region-noncontiguous-p))))
   (let ((error-file
 	 (if error-buffer
 	     (make-temp-file
@@ -3354,6 +3380,19 @@ (defun shell-command-on-region (start end command
 				    temporary-file-directory)))
 	   nil))
 	exit-status)
+    ;; Unless a single contiguous chunk is selected, operate on multiple chunks.
+    (if region-noncontiguous-p
+        (let ((input (concat (funcall region-extract-function 'delete) "\n"))
+              output)
+          (with-temp-buffer
+            (insert input)
+            (call-process-region (point-min) (point-max)
+                                 shell-file-name t t
+                                 nil shell-command-switch
+                                 command)
+            (setq output (split-string (buffer-string) "\n")))
+          (goto-char start)
+          (funcall region-insert-function output))
     (if (or replace
 	    (and output-buffer
 		 (not (or (bufferp output-buffer) (stringp output-buffer)))))
@@ -3443,7 +3482,7 @@ (defun shell-command-on-region (start end command
 			      exit-status output))))
 	    ;; Don't kill: there might be useful info in the undo-log.
 	    ;; (kill-buffer buffer)
-	    ))))
+	    )))))
 
     (when (and error-file (file-exists-p error-file))
       (if (< 0 (nth 7 (file-attributes error-file)))
@@ -5038,6 +5077,8 @@ (defun region-active-p ()
        ;; region is active when there's no mark.
        (progn (cl-assert (mark)) t)))
 
+(defun region-noncontiguous-p ()
+  (> (length (funcall region-extract-function 'bounds)) 1))
 
 (defvar redisplay-unhighlight-region-function
   (lambda (rol) (when (overlayp rol) (delete-overlay rol))))
diff --git a/lisp/rect.el b/lisp/rect.el
index acd3a48..560fbc2 100644
--- a/lisp/rect.el
+++ b/lisp/rect.el
@@ -257,6 +257,19 @@ (defun extract-rectangle (start end)
     (apply-on-rectangle 'extract-rectangle-line start end lines)
     (nreverse (cdr lines))))
 
+(defun extract-rectangle-bounds (start end)
+  "Return the bounds of the rectangle with corners at START and END.
+Return it as a list of (START . END) boundaries, one for each line of
+the rectangle."
+  (let (bounds)
+    (apply-on-rectangle
+     (lambda (startcol endcol)
+       (move-to-column startcol)
+       (push (cons (prog1 (point) (move-to-column endcol)) (point))
+	     bounds))
+     start end)
+    (nreverse bounds)))
+
 (defvar killed-rectangle nil
   "Rectangle for `yank-rectangle' to insert.")
 
@@ -563,6 +576,8 @@ (add-function :around redisplay-unhighlight-region-function
               #'rectangle--unhighlight-for-redisplay)
 (add-function :around region-extract-function
               #'rectangle--extract-region)
+(add-function :around region-insert-function
+              #'rectangle--insert-region)
 
 (defvar rectangle-mark-mode-map
   (let ((map (make-sparse-keymap)))
@@ -681,8 +696,12 @@ (defun rectangle-previous-line (&optional n)
 
 
 (defun rectangle--extract-region (orig &optional delete)
-  (if (not rectangle-mark-mode)
-      (funcall orig delete)
+  (cond
+   ((not rectangle-mark-mode)
+    (funcall orig delete))
+   ((eq delete 'bounds)
+    (extract-rectangle-bounds (region-beginning) (region-end)))
+   (t
     (let* ((strs (funcall (if delete
                               #'delete-extract-rectangle
                             #'extract-rectangle)
@@ -696,7 +715,14 @@ (defun rectangle--extract-region (orig &optional delete)
         (put-text-property 0 (length str) 'yank-handler
                            `(rectangle--insert-for-yank ,strs t)
                            str)
-        str))))
+        str)))))
+
+(defun rectangle--insert-region (orig strings)
+  (cond
+   ((not rectangle-mark-mode)
+    (funcall orig strings))
+   (t
+    (funcall #'insert-rectangle strings))))
 
 (defun rectangle--insert-for-yank (strs)
   (push (point) buffer-undo-list)
diff --git a/lisp/emulation/cua-rect.el b/lisp/emulation/cua-rect.el
index ea8b524..d389f6e 100644
--- a/lisp/emulation/cua-rect.el
+++ b/lisp/emulation/cua-rect.el
@@ -666,6 +666,22 @@ (defun cua--extract-rectangle ()
     	     (setq rect (cons row rect))))))
     (nreverse rect)))
 
+(defun cua--extract-rectangle-bounds ()
+  (let (rect)
+    (if (not (cua--rectangle-virtual-edges))
+        (cua--rectangle-operation nil nil nil nil nil ; do not tabify
+          (lambda (s e _l _r)
+             (setq rect (cons (cons s e) rect))))
+      (cua--rectangle-operation nil 1 nil nil nil ; do not tabify
+        (lambda (s e l r _v)
+           (goto-char s)
+           (move-to-column l)
+           (setq s (point))
+           (move-to-column r)
+           (setq e (point))
+           (setq rect (cons (cons s e) rect)))))
+    (nreverse rect)))
+
 (defun cua--insert-rectangle (rect &optional below paste-column line-count)
   ;; Insert rectangle as insert-rectangle, but don't set mark and exit with
   ;; point at either next to top right or below bottom left corner
@@ -1394,6 +1410,8 @@ (defun cua--rectangle-post-command ()
 
 (add-function :around region-extract-function
               #'cua--rectangle-region-extract)
+(add-function :around region-insert-function
+              #'cua--insert-rectangle)
 (add-function :around redisplay-highlight-region-function
               #'cua--rectangle-highlight-for-redisplay)
 
@@ -1405,8 +1423,12 @@ (defun cua--rectangle-highlight-for-redisplay (orig &rest args)
 
 (defun cua--rectangle-region-extract (orig &optional delete)
   (cond
-   ((not cua--rectangle) (funcall orig delete))
-   ((eq delete 'delete-only) (cua--delete-rectangle))
+   ((not cua--rectangle)
+    (funcall orig delete))
+   ((eq delete 'bounds)
+    (cua--extract-rectangle-bounds))
+   ((eq delete 'delete-only)
+    (cua--delete-rectangle))
    (t
     (let* ((strs (cua--extract-rectangle))
            (str (mapconcat #'identity strs "\n")))
diff --git a/lisp/replace.el b/lisp/replace.el
index d6590c5..a06e363 100644
--- a/lisp/replace.el
+++ b/lisp/replace.el
@@ -284,7 +284,7 @@ (defun query-replace-read-args (prompt regexp-flag &optional noerror)
 	  (and current-prefix-arg (not (eq current-prefix-arg '-)))
 	  (and current-prefix-arg (eq current-prefix-arg '-)))))
 
-(defun query-replace (from-string to-string &optional delimited start end backward)
+(defun query-replace (from-string to-string &optional delimited start end backward region-noncontiguous-p)
   "Replace some occurrences of FROM-STRING with TO-STRING.
 As each match is found, the user must type a character saying
 what to do with it.  For directions, type \\[help-command] at that time.
@@ -328,22 +328,21 @@ (defun query-replace (from-string to-string &optional delimited start end backwa
 		   (if current-prefix-arg
 		       (if (eq current-prefix-arg '-) " backward" " word")
 		     "")
-		   (if (and transient-mark-mode mark-active) " in region" ""))
+		   (if (use-region-p) " in region" ""))
 	   nil)))
      (list (nth 0 common) (nth 1 common) (nth 2 common)
 	   ;; These are done separately here
 	   ;; so that command-history will record these expressions
 	   ;; rather than the values they had this time.
-	   (if (and transient-mark-mode mark-active)
-	       (region-beginning))
-	   (if (and transient-mark-mode mark-active)
-	       (region-end))
-	   (nth 3 common))))
-  (perform-replace from-string to-string t nil delimited nil nil start end backward))
+	   (if (use-region-p) (region-beginning))
+	   (if (use-region-p) (region-end))
+	   (nth 3 common)
+	   (if (use-region-p) (region-noncontiguous-p)))))
+  (perform-replace from-string to-string t nil delimited nil nil start end backward region-noncontiguous-p))
 
 (define-key esc-map "%" 'query-replace)
 
-(defun query-replace-regexp (regexp to-string &optional delimited start end backward)
+(defun query-replace-regexp (regexp to-string &optional delimited start end backward region-noncontiguous-p)
   "Replace some things after point matching REGEXP with TO-STRING.
 As each match is found, the user must type a character saying
 what to do with it.  For directions, type \\[help-command] at that time.
@@ -408,18 +407,17 @@ (defun query-replace-regexp (regexp to-string &optional delimited start end back
 		       (if (eq current-prefix-arg '-) " backward" " word")
 		     "")
 		   " regexp"
-		   (if (and transient-mark-mode mark-active) " in region" ""))
+		   (if (use-region-p) " in region" ""))
 	   t)))
      (list (nth 0 common) (nth 1 common) (nth 2 common)
 	   ;; These are done separately here
 	   ;; so that command-history will record these expressions
 	   ;; rather than the values they had this time.
-	   (if (and transient-mark-mode mark-active)
-	       (region-beginning))
-	   (if (and transient-mark-mode mark-active)
-	       (region-end))
-	   (nth 3 common))))
-  (perform-replace regexp to-string t t delimited nil nil start end backward))
+	   (if (use-region-p) (region-beginning))
+	   (if (use-region-p) (region-end))
+	   (nth 3 common)
+	   (if (use-region-p) (region-noncontiguous-p)))))
+  (perform-replace regexp to-string t t delimited nil nil start end backward region-noncontiguous-p))
 
 (define-key esc-map [?\C-%] 'query-replace-regexp)
 
@@ -485,9 +483,9 @@ (defun query-replace-regexp-eval (regexp to-expr &optional delimited start end)
        ;; and the user might enter a single token.
        (replace-match-string-symbols to)
        (list from (car to) current-prefix-arg
-	     (if (and transient-mark-mode mark-active)
+	     (if (use-region-p)
 		 (region-beginning))
-	     (if (and transient-mark-mode mark-active)
+	     (if (use-region-p)
 		 (region-end))))))
   (perform-replace regexp (cons 'replace-eval-replacement to-expr)
 		   t 'literal delimited nil nil start end))
@@ -523,9 +521,9 @@ (defun map-query-replace-regexp (regexp to-strings &optional n start end)
      (list from to
 	   (and current-prefix-arg
 		(prefix-numeric-value current-prefix-arg))
-	   (if (and transient-mark-mode mark-active)
+	   (if (use-region-p)
 	       (region-beginning))
-	   (if (and transient-mark-mode mark-active)
+	   (if (use-region-p)
 	       (region-end)))))
   (let (replacements)
     (if (listp to-strings)
@@ -587,12 +585,12 @@ (defun replace-string (from-string to-string &optional delimited start end backw
 		       (if (eq current-prefix-arg '-) " backward" " word")
 		     "")
 		   " string"
-		   (if (and transient-mark-mode mark-active) " in region" ""))
+		   (if (use-region-p) " in region" ""))
 	   nil)))
      (list (nth 0 common) (nth 1 common) (nth 2 common)
-	   (if (and transient-mark-mode mark-active)
+	   (if (use-region-p)
 	       (region-beginning))
-	   (if (and transient-mark-mode mark-active)
+	   (if (use-region-p)
 	       (region-end))
 	   (nth 3 common))))
   (perform-replace from-string to-string nil nil delimited nil nil start end backward))
@@ -661,12 +659,12 @@ (defun replace-regexp (regexp to-string &optional delimited start end backward)
 		       (if (eq current-prefix-arg '-) " backward" " word")
 		     "")
 		   " regexp"
-		   (if (and transient-mark-mode mark-active) " in region" ""))
+		   (if (use-region-p) " in region" ""))
 	   t)))
      (list (nth 0 common) (nth 1 common) (nth 2 common)
-	   (if (and transient-mark-mode mark-active)
+	   (if (use-region-p)
 	       (region-beginning))
-	   (if (and transient-mark-mode mark-active)
+	   (if (use-region-p)
 	       (region-end))
 	   (nth 3 common))))
   (perform-replace regexp to-string nil t delimited nil nil start end backward))
@@ -832,7 +830,7 @@ (defun keep-lines (regexp &optional rstart rend interactive)
 		  (unless (or (bolp) (eobp))
 		    (forward-line 0))
 		  (point-marker)))))
-    (if (and interactive transient-mark-mode mark-active)
+    (if (and interactive (use-region-p))
 	(setq rstart (region-beginning)
 	      rend (progn
 		     (goto-char (region-end))
@@ -901,7 +899,7 @@ (defun flush-lines (regexp &optional rstart rend interactive)
       (progn
 	(goto-char (min rstart rend))
 	(setq rend (copy-marker (max rstart rend))))
-    (if (and interactive transient-mark-mode mark-active)
+    (if (and interactive (use-region-p))
 	(setq rstart (region-beginning)
 	      rend (copy-marker (region-end)))
       (setq rstart (point)
@@ -951,7 +949,7 @@ (defun how-many (regexp &optional rstart rend interactive)
               (setq rend (max rstart rend)))
           (goto-char rstart)
           (setq rend (point-max)))
-      (if (and interactive transient-mark-mode mark-active)
+      (if (and interactive (use-region-p))
 	  (setq rstart (region-beginning)
 		rend (region-end))
 	(setq rstart (point)
@@ -2068,7 +2066,7 @@ (defun replace-dehighlight ()
 
 (defun perform-replace (from-string replacements
 		        query-flag regexp-flag delimited-flag
-			&optional repeat-count map start end backward)
+			&optional repeat-count map start end backward region-noncontiguous-p)
   "Subroutine of `query-replace'.  Its complexity handles interactive queries.
 Don't use this in your own program unless you want to query and set the mark
 just as `query-replace' does.  Instead, write a simple loop like this:
@@ -2115,6 +2113,9 @@ (defun perform-replace (from-string replacements
 
          ;; If non-nil, it is marker saying where in the buffer to stop.
          (limit nil)
+         ;; Use local binding in add-function below.
+         (isearch-filter-predicate isearch-filter-predicate)
+         (region-bounds nil)
 
          ;; Data for the next match.  If a cons, it has the same format as
          ;; (match-data); otherwise it is t if a match is possible at point.
@@ -2127,6 +2128,24 @@ (defun perform-replace (from-string replacements
                       "Query replacing %s with %s: (\\<query-replace-map>\\[help] for help) ")
                      minibuffer-prompt-properties))))
 
+    ;; Unless a single contiguous chunk is selected, operate on multiple chunks.
+    (when region-noncontiguous-p
+      (setq region-bounds
+            (mapcar (lambda (position)
+                      (cons (copy-marker (car position))
+                            (copy-marker (cdr position))))
+                    (funcall region-extract-function 'bounds)))
+      (add-function :after-while isearch-filter-predicate
+                    (lambda (start end)
+                      (delq nil (mapcar
+                                 (lambda (bounds)
+                                   (and
+                                    (>= start (car bounds))
+                                    (<= start (cdr bounds))
+                                    (>= end   (car bounds))
+                                    (<= end   (cdr bounds))))
+                                 region-bounds)))))
+
     ;; If region is active, in Transient Mark mode, operate on region.
     (if backward
 	(when end
diff --git a/src/casefiddle.c b/src/casefiddle.c
index b94ea8e..7064d9d 100644
--- a/src/casefiddle.c
+++ b/src/casefiddle.c
@@ -306,14 +306,30 @@
   return Qnil;
 }
 
-DEFUN ("downcase-region", Fdowncase_region, Sdowncase_region, 2, 2, "r",
+DEFUN ("downcase-region", Fdowncase_region, Sdowncase_region, 2, 3,
+       "(list (region-beginning) (region-end) (region-noncontiguous-p))",
        doc: /* Convert the region to lower case.  In programs, wants two arguments.
 These arguments specify the starting and ending character numbers of
 the region to operate on.  When used as a command, the text between
 point and the mark is operated on.  */)
-  (Lisp_Object beg, Lisp_Object end)
+  (Lisp_Object beg, Lisp_Object end, Lisp_Object region_noncontiguous_p)
 {
-  casify_region (CASE_DOWN, beg, end);
+  Lisp_Object bounds = Qnil;
+
+  if (!NILP (region_noncontiguous_p))
+    {
+      bounds = call1 (Fsymbol_value (intern ("region-extract-function")),
+			 intern ("bounds"));
+
+      while (CONSP (bounds))
+	{
+	  casify_region (CASE_DOWN, XCAR (XCAR (bounds)), XCDR (XCAR (bounds)));
+	  bounds = XCDR (bounds);
+	}
+    }
+  else
+    casify_region (CASE_DOWN, beg, end);
+
   return Qnil;
 }
 

Re: Design of commands operating on rectangular regions

[John Wiegley]

>>>>> Juri Linkov <juri@linkov.net> writes:

>  (defun shell-command-on-region (start end command
>  				      &optional output-buffer replace
> -				      error-buffer display-error-buffer)
> +				      error-buffer display-error-buffer
> +				      region-noncontiguous-p)
>    "Execute string COMMAND in inferior shell with region as input.

Juri,

Can you tell me more about the meaning of `region-noncontiguous-p' here? How
does the caller know when this option is required?

My takeaway is that this patch conflates user-level indications (selected
regions) too closely with the programmatically provided entities (the region
specified by START END). Perhaps it points to the need for an `interactive'
specifier to request "the current region, though if multiple sub-regions
exist, the set of spans indicated by those sub-regions". A function like
`shell-command-on-region' should be determined by its arguments. This patch
introduces a much tighter coupling with its context of use than what we had
previously.

To make that last statement clearer: Right now, `shell-command-on-region' acts
on a buffer, and a region of text within that buffer. This region is provided
to the command through its arguments by (interactive "r"). The only thing that
`shell-command-on-region' needs to do beyond that is to query the current
buffer, in order to feed text to the executed process. So, our "context" is
the current buffer, while everything else is provided by its arguments.

Your patch introduces a more hybridized behavior: (interactive "r") provides
the total extent of the region, but now there's an additional parameter,
`region-noncontiguous-p', that alters the behavior of shell-command-on-region
so that it pays attention not only to the current buffer, but also to
user-specified regions when drawing text from the buffer. And these regions
are only somewhat related to the region provided by "interactive 'r'".

I feel that the semantic "level" of these three arguments -- START, END and
REGION-NONCONTIGUOUS-P -- are now different, which I find confusing.
`region-noncontiguous-p' doesn't relate to the `interactive' form at all, but
rather solely to this new behavior, internal to shell-command-on-region.

I'd really like to solve this in a different manner for 25.1.

John

RE: Design of commands operating on rectangular regions

[Drew Adams]

In order to follow all of this, especially since you now
seem to be talking about user-level changes, could someone
please specify (describe) the non-contiguous region feature,
including how users can create and modify such a region
interactively?

That would be really helpful, for me at least.

And could you specify clearly the relation between this
non-contiguous region feature and rectangles?  Presumably
the former is more general but also allows some handling
of rectangles (?).

It sounds like there has been a fair amount of implementation
already, without really any discussion of the design and how
users are affected (interactive behavior - wherever).  But
let me know if I'm mistaken and am jumping the gun, here.

Re: Design of commands operating on rectangular regions

[Juri Linkov]

> In order to follow all of this, especially since you now
> seem to be talking about user-level changes, could someone
> please specify (describe) the non-contiguous region feature,
> including how users can create and modify such a region
> interactively?

First you can select the non-contiguous region with
'C-x SPC', then extend it with the usual point motion keys,
then run one of the commands supporting such regions, e.g.
'C-x C-l' to downcase, 'M-%' to replace occurrences, 'C-u M-|'
to replace the region with the command output.

> And could you specify clearly the relation between this
> non-contiguous region feature and rectangles?  Presumably
> the former is more general but also allows some handling
> of rectangles (?).

Rectangles are a subset of non-contiguous regions that can be
expanded with other shapes, e.g. circular regions.

RE: Design of commands operating on rectangular regions

[Drew Adams]

> > In order to follow all of this, especially since you now
> > seem to be talking about user-level changes, could someone
> > please specify (describe) the non-contiguous region feature,
> > including how users can create and modify such a region
> > interactively?

Thanks for describing it.  I don't want to belabor this,
but if you have the time, here are some questions.  If not,
I'll see it when I use it. ;-)

> First you can select the non-contiguous region with
> 'C-x SPC', then extend it with the usual point motion keys,

A non-contiguous region with N pieces has 2*N ends.  Which
end(s) get extended with the motion keys?  And does `C-x SPC'
do what it does now, i.e., set a mark at point, so that point
is one end of at least one of the pieces?

Can the pieces (zones) overlap?

> then run one of the commands supporting such regions, e.g.
> 'C-x C-l' to downcase, 'M-%' to replace occurrences, 'C-u M-|'
> to replace the region with the command output.
> 
> > And could you specify clearly the relation between this
> > non-contiguous region feature and rectangles?  Presumably
> > the former is more general but also allows some handling
> > of rectangles (?).
> 
> Rectangles are a subset of non-contiguous regions that can be
> expanded with other shapes, e.g. circular regions.

OK, so I guess from another post by you that the non-contig
region info includes two properties/components: (1) a shape,
specifying, e.g., whether the zones form one or more
rectangles (or circles or whatever) and (2) the zone limits.

Is that right?

Re: Design of commands operating on rectangular regions

[Juri Linkov]

> Can you tell me more about the meaning of `region-noncontiguous-p' here?
> How does the caller know when this option is required?

To be able to process a non-contiguous region differently than the ordinary
region, a command needs to know either whether it's operating on the special
region (so the command itself can acquire all additional information about
the region and its shape), or receive all this information through its arg.

I provided the patches for both variants in bug#19829, and it seems that you
prefer the latter, if I'm not mistaken.  Please confirm do you think that
it would be better to add a 'region' arg such that when used programmatically
it's required to be in this form, or when used interactively it takes a form
with fine-grained boundaries:

  (region
   (shape . rectangle)
   (positions . (1 . 2) (3 . 4) (5 . 6) ...)
   ...

to accompany coarse-grained boundaries in START and END args.

> My takeaway is that this patch conflates user-level indications (selected
> regions) too closely with the programmatically provided entities (the region
> specified by START END). Perhaps it points to the need for an `interactive'
> specifier to request "the current region, though if multiple sub-regions
> exist, the set of spans indicated by those sub-regions". A function like
> `shell-command-on-region' should be determined by its arguments. This patch
> introduces a much tighter coupling with its context of use than what we had
> previously.
>
> To make that last statement clearer: Right now, `shell-command-on-region' acts
> on a buffer, and a region of text within that buffer. This region is provided
> to the command through its arguments by (interactive "r"). The only thing that
> `shell-command-on-region' needs to do beyond that is to query the current
> buffer, in order to feed text to the executed process. So, our "context" is
> the current buffer, while everything else is provided by its arguments.
>
> Your patch introduces a more hybridized behavior: (interactive "r") provides
> the total extent of the region, but now there's an additional parameter,
> `region-noncontiguous-p', that alters the behavior of shell-command-on-region
> so that it pays attention not only to the current buffer, but also to
> user-specified regions when drawing text from the buffer. And these regions
> are only somewhat related to the region provided by "interactive 'r'".
>
> I feel that the semantic "level" of these three arguments -- START, END and
> REGION-NONCONTIGUOUS-P -- are now different, which I find confusing.
> `region-noncontiguous-p' doesn't relate to the `interactive' form at all, but
> rather solely to this new behavior, internal to shell-command-on-region.

RE: Feature freezes and Emacs 25

[Drew Adams]

> I'm designing a framework
> to support multiple regions by adding a new arg with a group of regions.
> 
> In bug#19829 I've collected the evidence of commands that need this
> feature as a basis for the new design, but the deadlock was caused by the
> same question that backpedaled the Alan's design: whether to add a new arg
> to the existing functions or to try stuffing it into the existing ones?

FWIW, with Isearch+ you can search or query-replace multiple regions
(which can be defined in several ways, including just by their limits),
including across multiple buffers.

But I'm glad that you're adding this to isearch.el and also providing
rectangle search, even if it means additional Isearch+ work. ;-)

Re: Feature freezes and Emacs 25

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

> Since Emacs 25 is not yet released, the transfer of maintainership comes at a
> slightly awkward time. I haven't fully caught my breath yet, but don't want to
> impede Stefan's legacy from getting out the door.
>
> Let us officially freeze in one weeks, next Friday, Nov 13 at 11:59 PM UTC, so
> that any ready, last minute features can get in before 25.1 is closed. Then
> we'll take a pause and review whether anything should be allowed after the
> freeze. Once we're comfortable with the bug situation, we release.
>
> I'd like to take a somewhat generous period for bug fixes, with feature work
> happening in the meanwhile on topic branches. All severe bugs should either be
> closed, or consciously deferred, before letting 25 out the door.


If possible, I'd like to get my branch
fix/no-undo-boundary-on-secondary-buffer-change in as well. It's not a
huge change (in terms of size), although it's quite significant in that
it affects nearly every keypress in Emacs because it changes the undo
system.

It changes a behaviour which is breaking my current package, and which
cannot be worked around in lisp (at least not sanely).

Phil

kqueue in Emacs 25.1? (was: Feature freezes and Emacs 25)

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

> Let us officially freeze in one weeks, next Friday, Nov 13 at 11:59 PM UTC, so
> that any ready, last minute features can get in before 25.1 is closed. Then
> we'll take a pause and review whether anything should be allowed after the
> freeze. Once we're comfortable with the bug situation, we release.

The topic itself has been discussed a while ago: shall kqueue support be
added to the file notification backends? kqueue is the native file
notification library for all BSD-like systems, including Mac OS.

The answer that time was NO, because

- kqueue is already supported implicitely by gfilenotify, and
- it is said to have a horrible API.

Time passes, and at least the first argument is not valid any more. Due
to Bug#20280 we know, that gfilenotify cannot be used on systems which
have defined HAVE_NS. No file notification support on Mac OS, therefore.

The second argument is still valid, but ...

I have started a while ago to write kqueue.c, in order to have also
kqueue support in Emacs. Just my little pet project, looking whether
it's possible. Slow progress, 'tho.

The question is: assumed I succeed with this (point 2 of the arguments
is still valid), would kqueue support be accepted in Emacs? And in case
of yes, shall I concentrate on this task next time, in order to finish
before feature freeze?

Note, that I haven't anything to present yet, I'm still fighting with
the API. And I develop under Ubuntu, *real* tests must be performed by
somebody else.

> John

Best regards, Michael.

Re: kqueue in Emacs 25.1? (was: Feature freezes and Emacs 25)

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Sat, 07 Nov 2015 15:20:23 +0100
> 
> I have started a while ago to write kqueue.c, in order to have also
> kqueue support in Emacs. Just my little pet project, looking whether
> it's possible. Slow progress, 'tho.
> 
> The question is: assumed I succeed with this (point 2 of the arguments
> is still valid), would kqueue support be accepted in Emacs? And in case
> of yes, shall I concentrate on this task next time, in order to finish
> before feature freeze?
> 
> Note, that I haven't anything to present yet, I'm still fighting with
> the API. And I develop under Ubuntu, *real* tests must be performed by
> somebody else.

Can you estimate how soon can you have a test-ready implementation?

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> Note, that I haven't anything to present yet, I'm still fighting with
>> the API. And I develop under Ubuntu, *real* tests must be performed by
>> somebody else.
>
> Can you estimate how soon can you have a test-ready implementation?

Don't know. Maybe end of this week I could push something into a scratch
branch. It depends, whether there is a chance to be accepted; I would
give it priority then.

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: emacs-devel@gnu.org
> Date: Sat, 07 Nov 2015 15:53:40 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Note, that I haven't anything to present yet, I'm still fighting with
> >> the API. And I develop under Ubuntu, *real* tests must be performed by
> >> somebody else.
> >
> > Can you estimate how soon can you have a test-ready implementation?
> 
> Don't know. Maybe end of this week I could push something into a scratch
> branch. It depends, whether there is a chance to be accepted; I would
> give it priority then.

IMO, if it's just one week away, we should wait for it.  John?

Re: kqueue in Emacs 25.1?

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> IMO, if it's just one week away, we should wait for it.  John?

Agreed, let's exempt this from our feature freeze if it's that close.

John

Re: kqueue in Emacs 25.1?

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

> Agreed, let's exempt this from our feature freeze if it's that close.

Unfortunately, I am blocked. There seems to be an error in Ubuntu's
libkqueue0 package, which prevents me from further testing. I could
reproduce the problem with a very small test program outside Emacs, so I
have reported this as Ubuntu bug
<https://bugs.launchpad.net/ubuntu/+source/libkqueue/+bug/1514837>.

Unless somebody shows me that I made an error, or this bug is fixed in
Ubuntu, I cannot continue to work on this. I have no *BSD or Mac OS
machine which I could use.

I will collect everything what I have done so far, und push it as
"scratch/kqueue" branch to git.

> John

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Tue, Nov 10 2015, Michael Albinus wrote:

> I could
> reproduce the problem with a very small test program outside Emacs, so I
> have reported this as Ubuntu bug
> <https://bugs.launchpad.net/ubuntu/+source/libkqueue/+bug/1514837>.

I compiled your test program (after adding #include <sys/types.h> at the
top, see kqueue(2)) on FreeBSD 10 and it seems to work as expected?


[1 ~]$ cd /tmp/
[2 /tmp]$ cc -Wall julipedia.c -o julipedia
[3 /tmp]$ touch foo
[4 /tmp]$ ./julipedia >/dev/pts/7 2>&1
[5 /tmp]$ 

And on the other terminal:

[1 ~]$ tty
/dev/pts/7
[2 ~]$ cd /tmp/
[3 /tmp]$ echo >foo
[4 /tmp]$ File written
File extended
File attributes modified

[4 /tmp]$ echo >>foo
[5 /tmp]$ File written
File extended

[5 /tmp]$ chmod a+x foo 
File attributes modified
[6 /tmp]$ rm foo 
File deleted
[7 /tmp]$ 

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Wolfgang Jenkner <wjenkner@inode.at> writes:

Hi Wolfgang,

> I compiled your test program (after adding #include <sys/types.h> at the
> top, see kqueue(2)) on FreeBSD 10 and it seems to work as expected?

Yes. That's what I expect under BSD!

> [1 ~]$ cd /tmp/
> [2 /tmp]$ cc -Wall julipedia.c -o julipedia
> [3 /tmp]$ touch foo
> [4 /tmp]$ ./julipedia >/dev/pts/7 2>&1
> [5 /tmp]$ 
>
> And on the other terminal:
>
> [1 ~]$ tty
> /dev/pts/7
> [2 ~]$ cd /tmp/
> [3 /tmp]$ echo >foo
> [4 /tmp]$ File written
> File extended
> File attributes modified
>
> [4 /tmp]$ echo >>foo
> [5 /tmp]$ File written
> File extended
>
> [5 /tmp]$ chmod a+x foo 
> File attributes modified
> [6 /tmp]$ rm foo 
> File deleted
> [7 /tmp]$ 

Whatever I do with /tmp/foo, I see always

File written
File extended
File attributes modified
File link count changed
File renamed
File deleted

No chance to find out which action was triggered :-(

I'm just polishing my code a little bit, and then I will push it as
branch scratch/kqueue. Then you have something to play with it :-)

First of all, I would like to know whether it compiles and links under
BSD the way I have changed it in configure.ac and src/Makefile.in.

Thanks for your testing, and best regards, Michael.

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Wolfgang,

> I'm just polishing my code a little bit, and then I will push it as
> branch scratch/kqueue. Then you have something to play with it :-)

I've pushed this branch to savannah.

> First of all, I would like to know whether it compiles and links under
> BSD the way I have changed it in configure.ac and src/Makefile.in.

Could you, pls, check in your environment?

You could also play a little bit with `kqueue-add-watch', but be aware
that watching directories is not implemented yet. This must be emulated,
kqueue has no native support for watching directories.

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Wed, Nov 11 2015, Michael Albinus wrote:

> I've pushed this branch to savannah.

Thank you very much!
>
>> First of all, I would like to know whether it compiles and links under
>> BSD the way I have changed it in configure.ac and src/Makefile.in.
>
> Could you, pls, check in your environment?

Please, see the patch with trivial fixes below.

> You could also play a little bit with `kqueue-add-watch', but be aware
> that watching directories is not implemented yet. This must be emulated,
> kqueue has no native support for watching directories.

Doing the same tests as with julipedia:

echo >foo
(7 (attrib) "/tmp/foo") (7 (extend write) "/tmp/foo")

echo >>foo
(7 (extend write) "/tmp/foo")

chmod a+x foo
(7 (attrib) "/tmp/foo")

rm foo
(7 (delete) "/tmp/foo")

-- >8 --
Subject: [PATCH] Build fixes for kqueue support.

* src/kqueue.c (Fkqueue_add_watch): O_BINARY is not a POSIX open(3)
flag.

* configure.ac (HAVE_KQUEUE): There is no pkg-config module for native
kqueue on *BSD.
---
 configure.ac | 7 +++++++
 src/kqueue.c | 2 +-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 9c6db56..f9274d7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2732,6 +2732,13 @@ case $with_file_notification,$NOTIFY_OBJ in
        NOTIFY_LIBS=$KQUEUE_LIBS
        NOTIFY_OBJ=kqueue.o
        NOTIFY_SUMMARY="yes -lkqueue"
+    else
+       AC_SEARCH_LIBS(kqueue, [])
+       if test "$ac_cv_search_kqueue" != no; then
+         AC_DEFINE(HAVE_KQUEUE, 1, [Define to 1 to use kqueue.])
+	 NOTIFY_OBJ=kqueue.o
+	 NOTIFY_SUMMARY="yes (kqueue)"
+       fi
     fi ;;
 esac
 
diff --git a/src/kqueue.c b/src/kqueue.c
index c2e859f..d2f3d37 100644
--- a/src/kqueue.c
+++ b/src/kqueue.c
@@ -163,7 +163,7 @@ will be reported only in case of the `rename' event.  */)
 
   /* Open file.  */
   file = ENCODE_FILE (file);
-  fd = emacs_open (SSDATA (file), O_NONBLOCK | O_BINARY | O_RDONLY, 0);
+  fd = emacs_open (SSDATA (file), O_NONBLOCK | O_RDONLY, 0);
   if (fd == -1)
     report_file_error ("File cannot be opened", file);
 
-- 
2.6.2

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Wolfgang Jenkner <wjenkner@inode.at> writes:

Hi Wolfgang,

> Please, see the patch with trivial fixes below.

Feel free to commit them to the branch.

>> You could also play a little bit with `kqueue-add-watch', but be aware
>> that watching directories is not implemented yet. This must be emulated,
>> kqueue has no native support for watching directories.
>
> Doing the same tests as with julipedia:
>
> echo >foo
> (7 (attrib) "/tmp/foo") (7 (extend write) "/tmp/foo")
>
> echo >>foo
> (7 (extend write) "/tmp/foo")
>
> chmod a+x foo
> (7 (attrib) "/tmp/foo")
>
> rm foo
> (7 (delete) "/tmp/foo")

All of this looks good to me.

I will see whether I could downgrade libkqueue0 on my Ubuntu system to
an earlier version, which works correctly. Then I could continue to work
on this.

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Wed, Nov 11 2015, Michael Albinus wrote:

>> Please, see the patch with trivial fixes below.
>
> Feel free to commit them to the branch.

Done.

Wolfgang

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Wolfgang Jenkner <wjenkner@inode.at> writes:

>> Feel free to commit them to the branch.
>
> Done.

Thanks.

> Wolfgang

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Wed, Nov 11 2015, Michael Albinus wrote:

> This must be emulated,
> kqueue has no native support for watching directories.

In the sense that a directory can be watched (it's a vnode type) but it
is not so straightforward to find out what exactly has changed when,
say, a `write' event arrives for it?

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Wolfgang Jenkner <wjenkner@inode.at> writes:

>> This must be emulated,
>> kqueue has no native support for watching directories.
>
> In the sense that a directory can be watched (it's a vnode type) but it
> is not so straightforward to find out what exactly has changed when,
> say, a `write' event arrives for it?

Yes. I've studied the code in libinotify-kqueue and in the
glib/gio/kqueue backend. Both seem to scan the directory for changes,
once there is a "something has changed" kqueue event for a directory.

I plan to implement it based on directory_files_internal (in
dired.c). This is the working horse of directory-files-and-attributes.

Btw, in case you don't know it yet:
<http://people.freebsd.org/~jlemon/papers/kqueue.pdf> gives a pretty
good overview about kqueue.

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Wolfgang,

>> In the sense that a directory can be watched (it's a vnode type) but it
>> is not so straightforward to find out what exactly has changed when,
>> say, a `write' event arrives for it?
>
> Yes. I've studied the code in libinotify-kqueue and in the
> glib/gio/kqueue backend. Both seem to scan the directory for changes,
> once there is a "something has changed" kqueue event for a directory.
>
> I plan to implement it based on directory_files_internal (in
> dired.c). This is the working horse of directory-files-and-attributes.

Could you pls run a test for me? In julipedia.c, pls change "/tmp/foo"
to "/tmp". Start it, and do some file operations in /tmp. What happens?

Here under Ubuntu, it crashes immediately. Seems to be another bug :-(

I've tried to downgrade libkqueue0 from 2.0.3 to 1.0.4, but same problems ...

TIA, and best regards, Michael.

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Thu, Nov 12 2015, Michael Albinus wrote:

> Could you pls run a test for me? In julipedia.c, pls change "/tmp/foo"
> to "/tmp". Start it, and do some file operations in /tmp. What happens?

[1 ~]$ cd /tmp/
[2 /tmp]$ ./julipedia &
[1] 13728
[3 /tmp]$ touch x
File written
[4 /tmp]$ mv x y
File written
[5 /tmp]$ rm y
[6 /tmp]$ File written

[6 /tmp]$ 

I've seen only `write' events here.

Wolfgang

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Wolfgang Jenkner <wjenkner@inode.at> writes:

>> Could you pls run a test for me? In julipedia.c, pls change "/tmp/foo"
>> to "/tmp". Start it, and do some file operations in /tmp. What happens?
>
> [1 ~]$ cd /tmp/
> [2 /tmp]$ ./julipedia &
> [1] 13728
> [3 /tmp]$ touch x
> File written
> [4 /tmp]$ mv x y
> File written
> [5 /tmp]$ rm y
> [6 /tmp]$ File written
>
> [6 /tmp]$ 
>
> I've seen only `write' events here.

Well, that's as expected.

Do you know a *BSD VM I could download? It shall be prepared to compile
and run Emacs; I don't know too much how to install a proper environment
for Emacs under *BSD.

> Wolfgang

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Fri, Nov 13 2015, Michael Albinus wrote:

> Do you know a *BSD VM I could download? It shall be prepared to compile
> and run Emacs; I don't know too much how to install a proper environment
> for Emacs under *BSD.

The official FreeBSD VM images contain a complete base system, including
a working (clang) toolchain, see

https://www.freebsd.org/where.html

I'd suggest one of

ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.2-RELEASE/amd64/Latest/

see

ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/README.txt

(I've just tried the .raw disk image under qemu and compiled a simple
hello world program, which works fine.  Also, storage is persistent in
this case, i.e., the test program was still there after rebooting the
VM.)

Almost everything should be there to compile a non-X emacs from source,
with the exception of GNU make (gmake) and pkg-config (actually pkgconf,
a compatible replacement without glib dependency).

If you can bring up the network in the emulator (with DHCP everything
should be automatic), `pkg install gmake pkgconf' should do the trick.

Mounting ext2fs loopback (in GNU/Linux parlance) disk images should also
work r/w.

For any questions, please feel free to email me (I think they would be
slightly off topic for the list...).

Wolfgang

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Fri, Nov 13 2015, Wolfgang Jenkner wrote:

> If you can bring up the network in the emulator (with DHCP everything
> should be automatic),

Ahem, sorry, not quite automatic

$ cat /etc/rc.conf

hostname="xx.yy.zz"
ifconfig_em0="DHCP"

where em0 must be replaced by the name of the network device as shown by
ifconfig (or dmesg), see

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-dhcp.html

Wolfgang

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Wolfgang Jenkner <wjenkner@inode.at> writes:

> I'd suggest one of
>
> ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.2-RELEASE/amd64/Latest/

Thanks for this, I'll try it.

> For any questions, please feel free to email me (I think they would be
> slightly off topic for the list...).

For sure :-)

> Wolfgang

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> Wolfgang Jenkner <wjenkner@inode.at> writes:
>
>> I'd suggest one of
>>
>> ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.2-RELEASE/amd64/Latest/
>
> Thanks for this, I'll try it.

Thanks to the support by Wolfgang, I could continue implementing kqueue
integration for Emacs. The code in branch scratch/kqueue is working
pretty well on FreeBSD 10.2. The example in the Elisp manual runs proper
except the notification of file attribute changes. This is a minor
problem only; other file notification backends have also problems with
this.

test/automated/auto-revert-tests.el passes.

test/automated/file-notify-tests.el shows two failed test
cases. According to my debugging, it is not a problem of the kqueue
backend itself, but a problem how this test suite tries to catch file
notification events. Also a minor problem I would say.

How to continue? There are several options:

* Merge the branch into the emacs-25 branch.

* Merge the branch into the master branch.

* Keep the code still in this branch for a while.

Personally, I'm in favor of the first option. In any case: People using
*BSD or Mac OS are encouraged to run their own tests.

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[John Wiegley]

>>>>> Michael Albinus <michael.albinus@gmx.de> writes:

> Thanks to the support by Wolfgang, I could continue implementing kqueue
> integration for Emacs. The code in branch scratch/kqueue is working pretty
> well on FreeBSD 10.2. The example in the Elisp manual runs proper except the
> notification of file attribute changes. This is a minor problem only; other
> file notification backends have also problems with this.

Hi Michael,

I think this work should continue on its branch until you at least have the
tests working, and then merge it into master.

John

Re: kqueue in Emacs 25.1?

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

> Hi Michael,

Hi John,

> I think this work should continue on its branch until you at least have the
> tests working, and then merge it into master.

Will do.

> John

Best regards, Michael.

Re: kqueue in Emacs 25?

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

> Hi Michael,

Hi everybody,

> I think this work should continue on its branch until you at least have the
> tests working, and then merge it into master.

I have merged branch scratch/kqueue into master. This adds the kqueue
file notification library for Emacsen running under *BSD or Mac OS.

Running "./configure; make" shall enable it on such machines. The
configure output reports, whether it has been enabled. If this doesn't
happen automatically, pls report.

Alternatively, you can enable it by calling

    ./configure --with-file-notification=kqueue

The file notification test suite runs via

    make -C test/automated file-notify-tests

Be patient, some of the tests take their time. Pls report also, if it
doesn't work for you.

> John

Best regards, Michael.

Re: kqueue in Emacs 25?

[John Wiegley]

>>>>> Michael Albinus <michael.albinus@gmx.de> writes:

> I have merged branch scratch/kqueue into master. This adds the kqueue file
> notification library for Emacsen running under *BSD or Mac OS.

This is great news, Michael, thank you! I'd been wanting file notifications on
OS X for a while.

John

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Sat, Nov 07 2015, Michael Albinus wrote:

> The topic itself has been discussed a while ago: shall kqueue support be
> added to the file notification backends?

I'd say it depends on how you implement it: if you are trying to emulate
the inotify API it might be better to just link with libinotify-kqueue

https://github.com/dmatveev/libinotify-kqueue

On the other hand, if you are writing "native" support for kqueue, that
would be awesome :-)

As for emacs with libinotify-kqueue, I've tried this on FreeBSD
10-STABLE (before your recent changes to the notify stuff, though) and
it seems to work well when watching small directories, but emacs somehow
locks up for directories with more than 1500 files or so (while using,
say, inotifywait from inotify-tools seems to work without problems for
larger directories).

Here's the trivial patch for libinotify support, by the way:

-- >8 --
Subject: [PATCH] Tentative libinotify support.

---
 configure.ac    | 13 +++++++++----
 src/Makefile.in |  4 +++-
 2 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/configure.ac b/configure.ac
index 83d45a9..4216821 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2696,19 +2696,24 @@ case $with_file_notification,$opsys in
     fi ;;
 esac
 
-dnl inotify is available only on GNU/Linux.
+dnl inotify is available on GNU/Linux, and on *BSD via libinotify.
 case $with_file_notification,$NOTIFY_OBJ in
   inotify, | yes,)
     AC_CHECK_HEADER(sys/inotify.h)
     if test "$ac_cv_header_sys_inotify_h" = yes ; then
-	AC_CHECK_FUNC(inotify_init1)
-	if test "$ac_cv_func_inotify_init1" = yes; then
+       AC_SEARCH_LIBS(inotify_init1, inotify)
+       if test "$ac_cv_search_inotify_init1" != no; then
 	  AC_DEFINE(HAVE_INOTIFY, 1, [Define to 1 to use inotify.])
 	  NOTIFY_OBJ=inotify.o
-	  NOTIFY_SUMMARY="yes -lglibc (inotify)"
+	  LIB_INOTIFY=
+	  if test "$ac_cv_search_inotify_init1" != "none required"; then
+	     LIB_INOTIFY="$ac_cv_search_inotify_init1"
+	  fi
+	  NOTIFY_SUMMARY="yes $LIB_INOTIFY (inotify)"
        fi
     fi ;;
 esac
+AC_SUBST(LIB_INOTIFY)
 
 dnl g_file_monitor exists since glib 2.18.  G_FILE_MONITOR_EVENT_MOVED
 dnl has been added in glib 2.24.  It has been tested under
diff --git a/src/Makefile.in b/src/Makefile.in
index 9c0a3bb..bb9c003 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -153,6 +153,8 @@ DBUS_OBJ = @DBUS_OBJ@
 
 LIB_EXECINFO=@LIB_EXECINFO@
 
+LIB_INOTIFY=@LIB_INOTIFY@
+
 SETTINGS_CFLAGS = @SETTINGS_CFLAGS@
 SETTINGS_LIBS = @SETTINGS_LIBS@
 
@@ -468,7 +470,7 @@ LIBES = $(LIBS) $(W32_LIBS) $(LIBS_GNUSTEP) $(LIBX_BASE) $(LIBIMAGE) \
    $(LIBS_TERMCAP) $(GETLOADAVG_LIBS) $(SETTINGS_LIBS) $(LIBSELINUX_LIBS) \
    $(FREETYPE_LIBS) $(FONTCONFIG_LIBS) $(LIBOTF_LIBS) $(M17N_FLT_LIBS) \
    $(LIBGNUTLS_LIBS) $(LIB_PTHREAD) \
-   $(GFILENOTIFY_LIBS) $(LIB_MATH) $(LIBZ)
+   $(GFILENOTIFY_LIBS) $(LIB_INOTIFY) $(LIB_MATH) $(LIBZ)
 
 $(leimdir)/leim-list.el: bootstrap-emacs$(EXEEXT)
 	$(MAKE) -C ../leim leim-list.el EMACS="$(bootstrap_exe)"
-- 
2.6.2

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Sat, Nov 07 2015, Wolfgang Jenkner wrote:

> I'd say it depends on how you implement it: if you are trying to emulate
> the inotify API it might be better to just link with libinotify-kqueue

I should add that one difference between libinotify-kqueue and native
GNU/Linux inotify is that libinotify uses a separate worker-thread.
This is a difference that can become visible sometimes (at least in the
current implementation), e.g., the worker thread doesn't block most
signals, so the effect of signal handlers can be delayed until control
returns to the main thread.

Re: kqueue in Emacs 25.1?

[Paul Eggert]

Wolfgang Jenkner wrote:
> one difference between libinotify-kqueue and native
> GNU/Linux inotify is that libinotify uses a separate worker-thread.
> This is a difference that can become visible sometimes (at least in the
> current implementation), e.g., the worker thread doesn't block most
> signals, so the effect of signal handlers can be delayed until control
> returns to the main thread.

Can you elaborate on why this is an issue?  If the worker thread doesn't block 
(say) SIGCHLD, the thread should call deliver_child_signal, which calls 
deliver_process_signal, which (a) calls pthread_sigmask to arrange for the 
worker thread to block SIGCHLD indefinitely so that the failure-to-block problem 
doesn't recur, and (b) calls pthread_kill (main_thread, SIGCHLD) to forward the 
signal to the main thread just this once. When you write "can be delayed" are 
you referring to the delay in (b), or to some other delay? And what are the 
symptoms of the delay?

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Sat, Nov 07 2015, Paul Eggert wrote:

> Can you elaborate on why this is an issue?  

I wasn't talking of emacs here.  I just meant that applications in
general don't expect an extra thread after calling inotify_init(),
because there is none on GNU/Linux (I think).  The special situation
that I had in mind was a glitch in inotifywatch due to this, see

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204366

Re: kqueue in Emacs 25.1?

[Michael Albinus]

Wolfgang Jenkner <wjenkner@inode.at> writes:

> On Sat, Nov 07 2015, Michael Albinus wrote:
>
>> The topic itself has been discussed a while ago: shall kqueue support be
>> added to the file notification backends?
>
> I'd say it depends on how you implement it: if you are trying to emulate
> the inotify API it might be better to just link with libinotify-kqueue
>
> https://github.com/dmatveev/libinotify-kqueue

Thanks for the link, it will be helpful to study that code.

> On the other hand, if you are writing "native" support for kqueue, that
> would be awesome :-)

That's the plan.

Would you be willing to help me with this, while coding, and (more
important) testing on *BSD?

Best regards, Michael.

Re: kqueue in Emacs 25.1?

[Wolfgang Jenkner]

On Sat, Nov 07 2015, Michael Albinus wrote:

> Would you be willing to help me with this, while coding, and (more
> important) testing on *BSD?

I'll gladly test this on my FreeBSD 10 system, but I'm afraid I've yet
to learn most about kqueue(2) by reading your code...

Re: Dynamic loading progress

[John Wiegley]

>>>>> Ted Zlatanov <tzz@lifelogs.com> writes:

> I've tried to review the discussion so far and believe the code is
> "functional enough." I think we got agreement from Stefan (probably John
> too) to exempt dynamic modules from the feature freeze.

If Eli and Stefan agree, you have my go ahead to integrate the code. If anyone
else disagrees, please say so now.

John

Re: Dynamic loading progress

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Fri, 06 Nov 2015 16:52:53 -0500
> 
> >>>>> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
> > I've tried to review the discussion so far and believe the code is
> > "functional enough." I think we got agreement from Stefan (probably John
> > too) to exempt dynamic modules from the feature freeze.
> 
> If Eli and Stefan agree, you have my go ahead to integrate the code. If anyone
> else disagrees, please say so now.

I don't see any reasons to disagree.  But adding such a feature, IME,
needs more than just one week before we shake it up enough.

Re: Dynamic loading progress

[Ted Zlatanov]

On Sat, 07 Nov 2015 10:35:39 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: John Wiegley <jwiegley@gmail.com>
>> Date: Fri, 06 Nov 2015 16:52:53 -0500
>> 
>> >>>>> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
>> > I've tried to review the discussion so far and believe the code is
>> > "functional enough." I think we got agreement from Stefan (probably John
>> > too) to exempt dynamic modules from the feature freeze.
>> 
>> If Eli and Stefan agree, you have my go ahead to integrate the code. If anyone
>> else disagrees, please say so now.

EZ> I don't see any reasons to disagree.  But adding such a feature, IME,
EZ> needs more than just one week before we shake it up enough.

I agree, but please recall it's been brewing for a very long time, so
it's not something we just started.

Ted

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Sat, 07 Nov 2015 08:33:12 -0500
> 
> EZ> I don't see any reasons to disagree.  But adding such a feature, IME,
> EZ> needs more than just one week before we shake it up enough.
> 
> I agree, but please recall it's been brewing for a very long time, so
> it's not something we just started.

I know.  But if you mean to imply that a week might be enough, then I
don't think so.  As long as the feature is on a separate branch, it
doesn't get enough exposure to users, neither wrt possible build-time
complications, nor wrt run-time problems.  There are just too many
corner cases which are only visible once the feature is on master.

Re: Dynamic loading progress

[Ted Zlatanov]

On Sat, 07 Nov 2015 15:48:08 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Sat, 07 Nov 2015 08:33:12 -0500
>> 
EZ> I don't see any reasons to disagree.  But adding such a feature, IME,
EZ> needs more than just one week before we shake it up enough.
>> 
>> I agree, but please recall it's been brewing for a very long time, so
>> it's not something we just started.

EZ> I know.  But if you mean to imply that a week might be enough, then I
EZ> don't think so.  As long as the feature is on a separate branch, it
EZ> doesn't get enough exposure to users, neither wrt possible build-time
EZ> complications, nor wrt run-time problems.  There are just too many
EZ> corner cases which are only visible once the feature is on master.

Yes, and that's exactly why we need it in master.

I think 1 week is the time to feature freeze, but we have lots of time
to fix bugs and issues.  Changing the module ABI/API for instance is
IMHO perfectly OK during the feature freeze.  The feature is "dynamic
modules" not a specific version thereof, until Emacs is released.  Then
it becomes "dynamic modules as defined by 25.1" and we have to live with
them :)

Ted

Re: Dynamic loading progress

[Aurélien Aptel]

Well the repo is on github, dynamic-modules-2 branch. The changelog
might need some work and we might want to move around some files but
it's ready to merged in master, I think.

https://github.com/aaptel/emacs-dynamic-module/tree/dynamic-modules-2

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Sa., 7. Nov.
2015 um 18:47 Uhr:

> Well the repo is on github, dynamic-modules-2 branch. The changelog
> might need some work and we might want to move around some files but
> it's ready to merged in master, I think.
>
> https://github.com/aaptel/emacs-dynamic-module/tree/dynamic-modules-2
>
>
In order to produce a clean patch, I tried merging the master (from
emacs-mirror) or emacs-24 branch, but both produced too many merge
conflicts (even in files untouched by the module code). Which upstream
branch did you use to create the dynamic-modules-2 branch originally?

[-- Attachment #2: Type: text/html, Size: 1054 bytes --]

Re: Dynamic loading progress

[Steinar Bang]

>>>>> Philipp Stephani <p.stephani2@gmail.com>:
> In order to produce a clean patch, I tried merging the master (from
> emacs-mirror) or emacs-24 branch, but both produced too many merge
> conflicts (even in files untouched by the module code).

What kind of conflicts in what files? Could you provide some detail?

What was the command line sequence you used to do the merges from master
and emacs-24?

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Steinar Bang <sb@dod.no> schrieb am So., 8. Nov. 2015 um 13:55 Uhr:

> >>>>> Philipp Stephani <p.stephani2@gmail.com>:
> > In order to produce a clean patch, I tried merging the master (from
> > emacs-mirror) or emacs-24 branch, but both produced too many merge
> > conflicts (even in files untouched by the module code).
>
> What kind of conflicts in what files? Could you provide some detail?
>
>
Basically everywhere (more details probably wouldn't help).


> What was the command line sequence you used to do the merges from master
> and emacs-24?
>
>
>
I've added the GitHub emacs-mirror as origin, merged the to the master of
Aurélien's fork, and then tried to merge that into the dynamics-module-2
branch. All with plain 'git merge'.
However, I now managed to produce a two-step merge that didn't produce
additional conflicts. I've now also merged in the Savannah master and
pushed the result to my fork.

[-- Attachment #2: Type: text/html, Size: 1493 bytes --]

Re: Dynamic loading progress

[Ted Zlatanov]

On Sun, 08 Nov 2015 11:16:53 +0000 Philipp Stephani <p.stephani2@gmail.com> wrote: 

PS> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Sa., 7. Nov.
PS> 2015 um 18:47 Uhr:

>> Well the repo is on github, dynamic-modules-2 branch. The changelog
>> might need some work and we might want to move around some files but
>> it's ready to merged in master, I think.
>> 
>> https://github.com/aaptel/emacs-dynamic-module/tree/dynamic-modules-2
>> 
>> 
PS> In order to produce a clean patch, I tried merging the master (from
PS> emacs-mirror) or emacs-24 branch, but both produced too many merge
PS> conflicts (even in files untouched by the module code). Which upstream
PS> branch did you use to create the dynamic-modules-2 branch originally?

Agreed. Aurélien, the diff is huge and there are many conflicts. It
looks like the fork from master happened in February 2015 and you're
simply out of date. Can you please rebase against the emacs.git
origin/master?  Last push...

Ted

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

Ted Zlatanov <tzz@lifelogs.com> schrieb am So., 8. Nov. 2015 um 14:08 Uhr:

> On Sun, 08 Nov 2015 11:16:53 +0000 Philipp Stephani <p.stephani2@gmail.com>
> wrote:
>
> PS> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Sa., 7.
> Nov.
> PS> 2015 um 18:47 Uhr:
>
> >> Well the repo is on github, dynamic-modules-2 branch. The changelog
> >> might need some work and we might want to move around some files but
> >> it's ready to merged in master, I think.
> >>
> >> https://github.com/aaptel/emacs-dynamic-module/tree/dynamic-modules-2
> >>
> >>
> PS> In order to produce a clean patch, I tried merging the master (from
> PS> emacs-mirror) or emacs-24 branch, but both produced too many merge
> PS> conflicts (even in files untouched by the module code). Which upstream
> PS> branch did you use to create the dynamic-modules-2 branch originally?
>
> Agreed. Aurélien, the diff is huge and there are many conflicts. It
> looks like the fork from master happened in February 2015 and you're
> simply out of date. Can you please rebase against the emacs.git
> origin/master?  Last push...
>

Please check
https://github.com/phst/emacs-dynamic-module/tree/dynamic-modules-2, it
should have the current master merged in.
Unfortunately I can't test that right now because for some reason make is
incredibly slow on my machine after the merge.

[-- Attachment #2: Type: text/html, Size: 2102 bytes --]

Re: Dynamic loading progress

[Ted Zlatanov]

On Sun, 08 Nov 2015 13:42:21 +0000 Philipp Stephani <p.stephani2@gmail.com> wrote: 

PS> Ted Zlatanov <tzz@lifelogs.com> schrieb am So., 8. Nov. 2015 um 14:08 Uhr:

>> Agreed. Aurélien, the diff is huge and there are many conflicts. It
>> looks like the fork from master happened in February 2015 and you're
>> simply out of date. Can you please rebase against the emacs.git
>> origin/master?  Last push...

PS> Please check
PS> https://github.com/phst/emacs-dynamic-module/tree/dynamic-modules-2, it
PS> should have the current master merged in.
PS> Unfortunately I can't test that right now because for some reason make is
PS> incredibly slow on my machine after the merge.

Sorry, Philipp, I don't know how you and Aurélien have collaborated in
the past, and I've considered Aurélien's code authoritative. I don't
want to bypass his review and approval, so maybe you can work with him
on that?  It will save us time instead of reviewing several times.

Thank you!
Ted

Re: Dynamic loading progress

[Aurélien Aptel]

On Sun, Nov 8, 2015 at 9:38 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>> Agreed. Aurélien, the diff is huge and there are many conflicts. It
>>> looks like the fork from master happened in February 2015 and you're
>>> simply out of date. Can you please rebase against the emacs.git
>>> origin/master?  Last push...

I already merged master in my branch at some point, and did so with
`merge --squash` IIRC, note to self: never do that again. Turns out it
made any subsequent master merge a lot harder, sorry for that. The
weird thing is most conflicting changes where actually identical but
git couldn't tell. The diff were the same on both side but git couldnt
tell they were from the same commits since they were squashed.

> PS> Please check
> PS> https://github.com/phst/emacs-dynamic-module/tree/dynamic-modules-2, it
> PS> should have the current master merged in.
> PS> Unfortunately I can't test that right now because for some reason make is
> PS> incredibly slow on my machine after the merge.
>
> Sorry, Philipp, I don't know how you and Aurélien have collaborated in
> the past, and I've considered Aurélien's code authoritative. I don't
> want to bypass his review and approval, so maybe you can work with him
> on that?  It will save us time instead of reviewing several times.

We've been working together on github, I've just merged his code.

Re: Dynamic loading progress

[Aurélien Aptel]

Oh and also, I have an account on savannah (aaptel). Can I be added to
the Emacs group and all other stuff commiters have access to?

Re: Dynamic loading progress

[Yuri Khan]

On Mon, Nov 9, 2015 at 4:40 PM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:

> I already merged master in my branch at some point, and did so with
> `merge --squash` IIRC, note to self: never do that again.

Merge --squash is history rewriting, with all of the consequences thereof.

> Turns out it
> made any subsequent master merge a lot harder, sorry for that. The
> weird thing is most conflicting changes where actually identical but
> git couldn't tell. The diff were the same on both side but git couldnt
> tell they were from the same commits since they were squashed.

Typical symptoms of merging a branch with a rebased version of itself.

Re: Dynamic loading progress

[Ted Zlatanov]

On Mon, 9 Nov 2015 11:40:06 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Sun, Nov 8, 2015 at 9:38 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Sorry, Philipp, I don't know how you and Aurélien have collaborated in
>> the past, and I've considered Aurélien's code authoritative. I don't
>> want to bypass his review and approval, so maybe you can work with him
>> on that?  It will save us time instead of reviewing several times.

AA> We've been working together on github, I've just merged his code.

I checked this branch: `git fetch git@github.com:aaptel/emacs-dynamic-module.git dynamic-modules-2`?

It's not been rebased against origin/master and there are many small
commits. I don't actually know what's the policy on merging such a major
branch from a non-committer[1]: should I rebase, squash, or preserve the
history? Does anyone know?

I also don't know if Philipp, who has many changes in that branch, has
signed the contributor papers.

Thanks
Ted

[1] I know Aurélien asked for the commit permissions on Savannah but I'm
working with today's state :)

Re: Dynamic loading progress

[Aurélien Aptel]

On Mon, Nov 9, 2015 at 12:46 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> I checked this branch: `git fetch git@github.com:aaptel/emacs-dynamic-module.git dynamic-modules-2`?

Correct

> It's not been rebased against origin/master and there are many small
> commits. I don't actually know what's the policy on merging such a major
> branch from a non-committer[1]: should I rebase, squash, or preserve the
> history? Does anyone know?

Whatever you do, I don't think its useful to preserve the history of
my branch. I would suggest maybe splitting commits for the most
invasive changes (throw/signal, Fload/load-suffixes, new user_ptr
type) and import the completely new stuff in a single commit (all the
content of modules/ basically). I know Philip tried to write Changelog
entries for some commits in the commit message that can be re-used. If
can try to make a new branch that can be re-based on master with the
commits I had in mind if you want.

> I also don't know if Philipp, who has many changes in that branch, has
> signed the contributor papers.

I thought he did, but I cannot find any commits on master from him so
I guess he didn't.

Re: Dynamic loading progress

[Steinar Bang]

>>>>> Aurélien Aptel <aurelien.aptel+emacs@gmail.com>:

> Whatever you do, I don't think its useful to preserve the history of
> my branch. I would suggest maybe splitting commits for the most
> invasive changes (throw/signal, Fload/load-suffixes, new user_ptr
> type) and import the completely new stuff in a single commit (all the
> content of modules/ basically). I know Philip tried to write Changelog
> entries for some commits in the commit message that can be re-used. If
> can try to make a new branch that can be re-based on master with the
> commits I had in mind if you want.

If Ted is going to do the job he has basically two choices:
 - Preserve all history of the branch
 - Squash all history of the branch into a single commit

Everything else will be too hard, I think.

Re: Dynamic loading progress

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> If Ted is going to do the job he has basically two choices:
>  - Preserve all history of the branch
>  - Squash all history of the branch into a single commit

(And I personally wouldn't have tried squashing after I have merged in
from master)

Re: Dynamic loading progress

[Aurélien Aptel]

On Mon, Nov 9, 2015 at 2:47 PM, Steinar Bang <sb@dod.no> wrote:
>> If Ted is going to do the job he has basically two choices:
>>  - Preserve all history of the branch
>>  - Squash all history of the branch into a single commit
>
> (And I personally wouldn't have tried squashing after I have merged in
> from master)

After merging master in the branch i can simply use a diff between the
2 (which shouldnt include conflicting/redundant changes this time) as
what I'll use to make the new clean branch.

Re: Dynamic loading progress

[Aurélien Aptel]

On Mon, Nov 9, 2015 at 3:26 PM, Aurélien Aptel
<aurelien.aptel+emacs@gmail.com> wrote:
> After merging master in the branch i can simply use a diff between the
> 2 (which shouldnt include conflicting/redundant changes this time) as
> what I'll use to make the new clean branch.

So assuming master was successfully merged in dynamic-modules-2:

git diff master dynamic-modules-2 > dyn.patch
git checkout -b dynamic-modules-3 master
git apply dyn.patch

# now make a series of commit splitted in a way that makes sense
git add -p FILE..
git commit

etc.

Re: Dynamic loading progress

[Aurélien Aptel]

I've made a clean new branch 'dynamic-modules-clean-1' on top of a
recent master which you should be able to rebase easily. I've split my
"working" branch into 6 Changelog-ready commits. I'm the author of the
commits but a significant part of the code was also written by Philip
Stefani (big thanks to him). Don't really know how to we can reflect
that since it's all mixed up now.

It's on github. You can rework the commit messages if you want as I'm
not very good at this.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Mo., 9. Nov.
2015 um 20:59 Uhr:

> I've made a clean new branch 'dynamic-modules-clean-1' on top of a
> recent master which you should be able to rebase easily.


Thanks! Since the freeze is only days away, can we now start merging these
changes into the Savannah master?

[-- Attachment #2: Type: text/html, Size: 629 bytes --]

Re: Dynamic loading progress

[Ted Zlatanov]

On Mon, 9 Nov 2015 13:27:00 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> On Mon, Nov 9, 2015 at 12:46 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> I checked this branch: `git fetch git@github.com:aaptel/emacs-dynamic-module.git dynamic-modules-2`?

AA> Correct

>> It's not been rebased against origin/master and there are many small
>> commits. I don't actually know what's the policy on merging such a major
>> branch from a non-committer[1]: should I rebase, squash, or preserve the
>> history? Does anyone know?

AA> Whatever you do, I don't think its useful to preserve the history of
AA> my branch. I would suggest maybe splitting commits for the most
AA> invasive changes (throw/signal, Fload/load-suffixes, new user_ptr
AA> type) and import the completely new stuff in a single commit (all the
AA> content of modules/ basically). I know Philip tried to write Changelog
AA> entries for some commits in the commit message that can be re-used. If
AA> can try to make a new branch that can be re-based on master with the
AA> commits I had in mind if you want.

Yes, please go ahead if you can. You know best which commits are
"milestones" and which are not worth preserving. The rebasing can be
part of your work.

>> I also don't know if Philipp, who has many changes in that branch, has
>> signed the contributor papers.

AA> I thought he did, but I cannot find any commits on master from him so
AA> I guess he didn't.

That could be a problem, unfortunately.  Let's wait for Philipp to
comment.

Ted

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

Ted Zlatanov <tzz@lifelogs.com> schrieb am Mo., 9. Nov. 2015 um 14:38 Uhr:

> On Mon, 9 Nov 2015 13:27:00 +0100 Aurélien Aptel <
> aurelien.aptel+emacs@gmail.com> wrote:
>
>
> >> I also don't know if Philipp, who has many changes in that branch, has
> >> signed the contributor papers.
>
> AA> I thought he did, but I cannot find any commits on master from him so
> AA> I guess he didn't.
>
> That could be a problem, unfortunately.  Let's wait for Philipp to
> comment.
>
>

I don't need to sign the papers because my employer (Google) has a special
agreement with the FSF. Please use my work address phst@google.com and
"Copyright Google Inc." for my contributions. I already have a few patches
in master under that address.

[-- Attachment #2: Type: text/html, Size: 1207 bytes --]

Re: Dynamic loading progress

[Ted Zlatanov]

On Mon, 09 Nov 2015 16:16:31 +0000 Philipp Stephani <p.stephani2@gmail.com> wrote: 

PS> I don't need to sign the papers because my employer (Google) has a special
PS> agreement with the FSF. Please use my work address phst@google.com and
PS> "Copyright Google Inc." for my contributions. I already have a few patches
PS> in master under that address.

Thank you.  I can see that your existing patches didn't need the
"Copyright Google Inc." in the commit message, so I think we can skip it here.

On Mon, 9 Nov 2015 20:58:59 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> I've made a clean new branch 'dynamic-modules-clean-1' on top of a
AA> recent master which you should be able to rebase easily. I've split my
AA> "working" branch into 6 Changelog-ready commits. I'm the author of the
AA> commits but a significant part of the code was also written by Philip
AA> Stefani (big thanks to him). Don't really know how to we can reflect
AA> that since it's all mixed up now.

AA> It's on github. You can rework the commit messages if you want as I'm
AA> not very good at this.

Aurélien, it looks great.  Thank you for the work.

I have only two requests before I merge:

1) Can you add this header line to the commits that need it (it may
require some checking, unfortunately):

Co-authored-by: Philipp Stephani <phst@google.com>

I *think* that's sufficient.

2) Since you're modifying some commits, can you remove the final period
from the first line?  It's a suggestion in CONTRIBUTE but not required.

Otherwise I think it's ready to merge.

On Tue, 10 Nov 2015 20:07:36 +0000 Philipp Stephani <p.stephani2@gmail.com> wrote: 

PS> Thanks! Since the freeze is only days away, can we now start merging these
PS> changes into the Savannah master?

Yes, I'll do the merge as soon as (1) above is resolved. Technically the
patch looks good.

Ted

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2113 bytes --]

Ted Zlatanov <tzz@lifelogs.com> schrieb am Mi., 11. Nov. 2015 um 12:22 Uhr:

> On Mon, 09 Nov 2015 16:16:31 +0000 Philipp Stephani <p.stephani2@gmail.com>
> wrote:
>
> PS> I don't need to sign the papers because my employer (Google) has a
> special
> PS> agreement with the FSF. Please use my work address phst@google.com and
> PS> "Copyright Google Inc." for my contributions. I already have a few
> patches
> PS> in master under that address.
>
> Thank you.  I can see that your existing patches didn't need the
> "Copyright Google Inc." in the commit message, so I think we can skip it
> here.
>
> On Mon, 9 Nov 2015 20:58:59 +0100 Aurélien Aptel <
> aurelien.aptel+emacs@gmail.com> wrote:
>
> AA> I've made a clean new branch 'dynamic-modules-clean-1' on top of a
> AA> recent master which you should be able to rebase easily. I've split my
> AA> "working" branch into 6 Changelog-ready commits. I'm the author of the
> AA> commits but a significant part of the code was also written by Philip
> AA> Stefani (big thanks to him). Don't really know how to we can reflect
> AA> that since it's all mixed up now.
>
> AA> It's on github. You can rework the commit messages if you want as I'm
> AA> not very good at this.
>
> Aurélien, it looks great.  Thank you for the work.
>
> I have only two requests before I merge:
>
> 1) Can you add this header line to the commits that need it (it may
> require some checking, unfortunately):
>
> Co-authored-by: Philipp Stephani <phst@google.com>
>
> I *think* that's sufficient.
>
> 2) Since you're modifying some commits, can you remove the final period
> from the first line?  It's a suggestion in CONTRIBUTE but not required.
>
> Otherwise I think it's ready to merge.
>
> On Tue, 10 Nov 2015 20:07:36 +0000 Philipp Stephani <p.stephani2@gmail.com>
> wrote:
>
> PS> Thanks! Since the freeze is only days away, can we now start merging
> these
> PS> changes into the Savannah master?
>
> Yes, I'll do the merge as soon as (1) above is resolved. Technically the
> patch looks good.
>
>
Wonderful, thanks a lot!

[-- Attachment #2: Type: text/html, Size: 2920 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

Ted, we're still working some things out on github. Hopefully I can
give a v2 merge-ready branch by this weekend.

Re: Dynamic loading progress

[Ted Zlatanov]

On Fri, 13 Nov 2015 02:29:09 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> Ted, we're still working some things out on github. Hopefully I can
AA> give a v2 merge-ready branch by this weekend.

John, can the dynamic modules branch be pushed then or do you need it
today?  I'm OK with the delay.

Ted

Freeze is almost here (was: Dynamic loading progress)

[John Wiegley]

>>>>> Ted Zlatanov <tzz@lifelogs.com> writes:

> John, can the dynamic modules branch be pushed then or do you need it today?
> I'm OK with the delay.

Sure, it can be pushed then. In fact, anytime next week is OK for that branch.

I've created the "emacs-25" branch, and will merge from master one last time
in 8.4 hours. After that, only bug fixes should be applied to that branch, or
feature work for those with an exemption.

I encourage everyone without pressing feature work to make that branch their
primary working area for the next few months, and to help me categorize the
bugs appropriately.

John

Re: Freeze is almost here (was: Dynamic loading progress)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 198 bytes --]

On Fri, Nov 13, 2015 at 4:39 PM, John Wiegley <jwiegley@gmail.com> wrote:

> After that, only bug fixes should be applied to that branch, or
> feature work for those with an exemption.

Or tests...

[-- Attachment #2: Type: text/html, Size: 324 bytes --]

Re: Freeze is almost here (was: Dynamic loading progress)

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Fri, 13 Nov 2015 07:39:27 -0800
> 
> >>>>> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
> > John, can the dynamic modules branch be pushed then or do you need it today?
> > I'm OK with the delay.
> 
> Sure, it can be pushed then. In fact, anytime next week is OK for that branch.
> 
> I've created the "emacs-25" branch, and will merge from master one last time
> in 8.4 hours. After that, only bug fixes should be applied to that branch, or
> feature work for those with an exemption.

We don't merge from master to the release branch, we merge in the
opposite direction.  See gitmerge.el.  So I think the dynamic modules
changes should be applied to emacs-25, and will then be merged to
master in due time.

> I encourage everyone without pressing feature work to make that branch their
> primary working area for the next few months, and to help me categorize the
> bugs appropriately.

Seconded.

in any case, once the release branch is cut, any bugfixes that fix
problems visible on emacs-25 should be applied there, not to master
(they will be later merged to master via gitmerge.el).  The master
branch should only get direct commits of fixes for bugs that don't
exist on the release branch.

Re: Freeze is almost here (was: Dynamic loading progress)

[Eli Zaretskii]

> Date: Fri, 13 Nov 2015 21:24:36 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> We don't merge from master to the release branch, we merge in the
> opposite direction.  See gitmerge.el.  So I think the dynamic modules
> changes should be applied to emacs-25, and will then be merged to
> master in due time.

I see that you branched emacs-25 not from the latest master.  This
means all the commits since 9463abf will have to be cherry-picked from
master to the emacs-25 branch.

Re: Freeze is almost here

[Achim Gratz]

Eli Zaretskii writes:
> I see that you branched emacs-25 not from the latest master.  This
> means all the commits since 9463abf will have to be cherry-picked from
> master to the emacs-25 branch.

That's likely why John plans to merge master into the release branch
again at some later time.  This postpones the actual freeze and avoids
the superfluous cherry-picks.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Freeze is almost here

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Fri, 13 Nov 2015 21:05:14 +0100
> 
> Eli Zaretskii writes:
> > I see that you branched emacs-25 not from the latest master.  This
> > means all the commits since 9463abf will have to be cherry-picked from
> > master to the emacs-25 branch.
> 
> That's likely why John plans to merge master into the release branch
> again at some later time.  This postpones the actual freeze and avoids
> the superfluous cherry-picks.

Merging from master to the release branch will likely make a mess when
we later merge back the bugfixes.

Anyway, I already pushed all the cherry-picks (with a special marker
that should exempt them from merging back via gitmerge.el).

Re: Freeze is almost here

[Achim Gratz]

Eli Zaretskii writes:
> Merging from master to the release branch will likely make a mess when
> we later merge back the bugfixes.

No, not any more than merging master multiple times into a feature
branch does before you push it back to master.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Freeze is almost here

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Fri, 13 Nov 2015 21:36:59 +0100
> 
> Eli Zaretskii writes:
> > Merging from master to the release branch will likely make a mess when
> > we later merge back the bugfixes.
> 
> No, not any more than merging master multiple times into a feature
> branch does before you push it back to master.

Which indeed makes a mess, see "git log --graph" on master.

Re: Freeze is almost here

[Dmitry Gutov]

On 11/13/2015 10:17 PM, Eli Zaretskii wrote:

> Merging from master to the release branch will likely make a mess when
> we later merge back the bugfixes.

Like others mentioned, not at all.

> Anyway, I already pushed all the cherry-picks (with a special marker
> that should exempt them from merging back via gitmerge.el).

Yeah, that was premature. John was clearly planning a fast-forward merge 
from master later today.

On the other hand, I don't know why we needed emacs-25 to be created in 
advance.

Re: Freeze is almost here

[Artur Malabarba]

2015-11-13 20:17 GMT+00:00 Eli Zaretskii <eliz@gnu.org>:
> Anyway, I already pushed all the cherry-picks (with a special marker
> that should exempt them from merging back via gitmerge.el).

What is the marker you use? I need to do push a commit to emacs-25
that is not to be merged back to master.

Re: Freeze is almost here

[Eli Zaretskii]

> Date: Sat, 14 Nov 2015 11:27:48 +0000
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> 2015-11-13 20:17 GMT+00:00 Eli Zaretskii <eliz@gnu.org>:
> > Anyway, I already pushed all the cherry-picks (with a special marker
> > that should exempt them from merging back via gitmerge.el).
> 
> What is the marker you use?

"Backport".  This is described in CONTRIBUTE; I just made it a tad
more clear.

Re: Freeze is almost here

[Artur Malabarba]

2015-11-14 11:55 GMT+00:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Sat, 14 Nov 2015 11:27:48 +0000
>> From: Artur Malabarba <bruce.connor.am@gmail.com>
>> Cc: emacs-devel <emacs-devel@gnu.org>
>>
>> 2015-11-13 20:17 GMT+00:00 Eli Zaretskii <eliz@gnu.org>:
>> > Anyway, I already pushed all the cherry-picks (with a special marker
>> > that should exempt them from merging back via gitmerge.el).
>>
>> What is the marker you use?
>
> "Backport".  This is described in CONTRIBUTE; I just made it a tad
> more clear.

Thanks.

Re: Freeze is almost here

[Dmitry Gutov]

On 11/14/2015 01:27 PM, Artur Malabarba wrote:

> What is the marker you use? I need to do push a commit to emacs-25
> that is not to be merged back to master.

As you can see, each of those cherry-picked commits has "Backport" in 
its message.

For other possible markers, see gitmerge-skip-regexp in admin/gitmerge.el.

Re: Freeze is almost here (was: Dynamic loading progress)

[Eli Zaretskii]

> Date: Fri, 13 Nov 2015 21:37:53 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: jwiegley@gmail.com, emacs-devel@gnu.org
> 
> I see that you branched emacs-25 not from the latest master.  This
> means all the commits since 9463abf will have to be cherry-picked from
> master to the emacs-25 branch.

Done.

Re: Freeze is almost here (was: Dynamic loading progress)

[Eli Zaretskii]

> Date: Fri, 13 Nov 2015 22:05:47 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Date: Fri, 13 Nov 2015 21:37:53 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: jwiegley@gmail.com, emacs-devel@gnu.org
> > 
> > I see that you branched emacs-25 not from the latest master.  This
> > means all the commits since 9463abf will have to be cherry-picked from
> > master to the emacs-25 branch.
> 
> Done.

We should now change master to identify itself as version 25.1.50 or
maybe even 26.0.50.  OK?

Re: Freeze is almost here

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> We should now change master to identify itself as version 25.1.50 or maybe
> even 26.0.50. OK?

Let's go with 25.1.50 for now.

Next time we'll coordinate better to avoid the cherry-picking. The merge
commit you'll see today is because I didn't read these messages until after
doing it, but it should have had very little impact. We also have about 12
commits of replicated history, but that's the price of learning.

FWIW, I don't believe "directionality" matters to Git, until you have changes
on one branch you don't ever want to see on another. Since the freeze hadn't
happened yet, it should have been OK to unify the branches in either
direction.

John

Re: Freeze is almost here

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 13 Nov 2015 14:57:56 -0800
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We should now change master to identify itself as version 25.1.50 or maybe
> > even 26.0.50. OK?
> 
> Let's go with 25.1.50 for now.

Done.

> FWIW, I don't believe "directionality" matters to Git

It matters to us, though.

> until you have changes on one branch you don't ever want to see on
> another. Since the freeze hadn't happened yet, it should have been
> OK to unify the branches in either direction.

I don't understand how freeze is relevant to this.  Or maybe I don't
understand what "freeze" means in terms of Git operations.  Can you
explain?

Re: Freeze is almost here

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> FWIW, I don't believe "directionality" matters to Git
> It matters to us, though.

All I meant is that until the actual freeze, master and emacs-25 were
equivalent. Now they no longer are.

I'm sorry to have created so much confusion, I assumed too much about our
collective Git habits. Next time I'll "cut once", and it will be frozen from
that moment forward. I had feared I wouldn't be available to do it at the
right time, so I did it early, expecting to allow changes from master until
the cutoff point.

Now that all that's done, we're back on the right footing. I'll make a wider
announcement later today.

Thanks for your patience, Eli,
  John

Re: Freeze is almost here

[Steinar Bang]

>>>>> "John Wiegley" <jwiegley@gmail.com>:

> I'm sorry to have created so much confusion, I assumed too much about our
> collective Git habits. Next time I'll "cut once", and it will be frozen from
> that moment forward. I had feared I wouldn't be available to do it at the
> right time, so I did it early, expecting to allow changes from master until
> the cutoff point.

Note that you can branch off from any commit on master after the fact.

Ie. you can look down the history list of master and decide "Ah!
There!", and then do
 git checkout -b emacs-26.1 <sha1-has-of-the-commit-where-the-branch-should-start>
 git push -u origin HEAD

Re: Freeze is almost here

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> "John Wiegley" <jwiegley@gmail.com>:
>> I'm sorry to have created so much confusion, I assumed too much about our
>> collective Git habits. Next time I'll "cut once", and it will be frozen from
>> that moment forward. I had feared I wouldn't be available to do it at the
>> right time, so I did it early, expecting to allow changes from master until
>> the cutoff point.

> Note that you can branch off from any commit on master after the fact.

> Ie. you can look down the history list of master and decide "Ah!
> There!", and then do
>  git checkout -b emacs-26.1 <sha1-has-of-the-commit-where-the-branch-should-start>
>  git push -u origin HEAD

(note: my use of "emacs-26.1" as an example is for the next release
branch off master.  "emacs-25" is already a done deal)

Re: Freeze is almost here

[John Wiegley]

>>>>> Steinar Bang <sb@dod.no> writes:

> Note that you can branch off from any commit on master after the fact. Ie.
> you can look down the history list of master and decide "Ah! There!", and
> then do git checkout -b emacs-26.1
> <sha1-has-of-the-commit-where-the-branch-should-start> git push -u origin
> HEAD

That is such a good point, thanks Steinar.

John

Re: Freeze is almost here

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> I see that you branched emacs-25 not from the latest master.  This
> means all the commits since 9463abf will have to be cherry-picked from
> master to the emacs-25 branch.

??? Just fast forward it.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Freeze is almost here

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 13 Nov 2015 22:34:50 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I see that you branched emacs-25 not from the latest master.  This
> > means all the commits since 9463abf will have to be cherry-picked from
> > master to the emacs-25 branch.
> 
> ??? Just fast forward it.

I didn't know how to do that (can you tell?).  In any case, by the
time I got to that, someone already made one commit to the branch, so
I think fast-forwarding would have been impossible.

Re: Freeze is almost here

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: jwiegley@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 13 Nov 2015 22:34:50 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I see that you branched emacs-25 not from the latest master.  This
>> > means all the commits since 9463abf will have to be cherry-picked from
>> > master to the emacs-25 branch.
>> 
>> ??? Just fast forward it.
>
> I didn't know how to do that (can you tell?).

git merge

> In any case, by the time I got to that, someone already made one
> commit to the branch, so I think fast-forwarding would have been
> impossible.

No, there is no such commit.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Freeze is almost here

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 14 Nov 2015 09:16:40 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> ??? Just fast forward it.
> >
> > I didn't know how to do that (can you tell?).
> 
> git merge

Right, thanks.

> > In any case, by the time I got to that, someone already made one
> > commit to the branch, so I think fast-forwarding would have been
> > impossible.
> 
> No, there is no such commit.

AFAIU, there is: 26e7624b234f350a1f483d6fe0be8c30cca03008.

Re: Freeze is almost here

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAIU, there is: 26e7624b234f350a1f483d6fe0be8c30cca03008.

Right, I missed that.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Freeze is almost here

[David Engster]

Eli Zaretskii writes:
>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: jwiegley@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 13 Nov 2015 22:34:50 +0100
>
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I see that you branched emacs-25 not from the latest master.  This
>> > means all the commits since 9463abf will have to be cherry-picked from
>> > master to the emacs-25 branch.
>> 
>> ??? Just fast forward it.
>
> I didn't know how to do that (can you tell?).

By merging master into emacs-25. Git will do a fast-forward by default
if it is possible (use --ff-only to abort otherwise).

Regarding your 'Backport' messages in the cherry-picks: You only need to
do that when there was a conflict. Otherwise, gitmerge.el should detect
cherry-picks through the cherry-mark and automatically mark them as to
be skipped. However, I realize now that I'm calling the wrong log
command for that; I'll fix that over the weekend.

-David

Re: Freeze is almost here

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Andreas Schwab <schwab@linux-m68k.org>,  jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 14 Nov 2015 09:45:10 +0100
> 
> >> ??? Just fast forward it.
> >
> > I didn't know how to do that (can you tell?).
> 
> By merging master into emacs-25. Git will do a fast-forward by default
> if it is possible (use --ff-only to abort otherwise).

Thanks, will do in the future.

> Regarding your 'Backport' messages in the cherry-picks: You only need to
> do that when there was a conflict. Otherwise, gitmerge.el should detect
> cherry-picks through the cherry-mark and automatically mark them as to
> be skipped. However, I realize now that I'm calling the wrong log
> command for that; I'll fix that over the weekend.

Thanks.

Please also consider whether we need some changes in gitmerge.el now
that we produce ChangeLog from the Git log.

Re: Freeze is almost here

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: Andreas Schwab <schwab@linux-m68k.org>, jwiegley@gmail.com,
>> emacs-devel@gnu.org
>
>> Date: Sat, 14 Nov 2015 09:45:10 +0100

>> Regarding your 'Backport' messages in the cherry-picks: You only need to
>> do that when there was a conflict. Otherwise, gitmerge.el should detect
>> cherry-picks through the cherry-mark and automatically mark them as to
>> be skipped. However, I realize now that I'm calling the wrong log
>> command for that; I'll fix that over the weekend.
>
> Thanks.
>
> Please also consider whether we need some changes in gitmerge.el now
> that we produce ChangeLog from the Git log.

I see now that master was merged into emacs-25. Is this now the new
workflow? If so, there's not much point in keeping gitmerge.el anyway.

-David

Re: Freeze is almost here

[David Engster]

David Engster writes:
> I see now that master was merged into emacs-25. Is this now the new
> workflow? If so, there's not much point in keeping gitmerge.el anyway.

Sorry, forget what I said. I forgot the we don't freeze master anymore,
so I guess this was a one-time merge just to forward emacs-25. I was
just confused because git couldn't detect the cherry-picks, but this is
because of this merge from master.

-David

Re: Freeze is almost here

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: schwab@linux-m68k.org,  jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 14 Nov 2015 13:57:34 +0100
> 
> David Engster writes:
> > I see now that master was merged into emacs-25. Is this now the new
> > workflow? If so, there's not much point in keeping gitmerge.el anyway.
> 
> Sorry, forget what I said. I forgot the we don't freeze master anymore,
> so I guess this was a one-time merge just to forward emacs-25. I was
> just confused because git couldn't detect the cherry-picks, but this is
> because of this merge from master.

Could you describe the conditions necessary for gitmerge to detect
cherry-picks?  I see that it uses --cherry-mark, but would that mark
cherry-picks whose commit message was edited by "cherry-pick -e"?
(That editing is necessary to add "Backported", at least.)

Also, what about the effect of the "cherry-pick -x" switch, does
affect gitmerge in any way?

Re: Freeze is almost here

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: schwab@linux-m68k.org,  jwiegley@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 14 Nov 2015 13:57:34 +0100
>
>> 
>> David Engster writes:
>> > I see now that master was merged into emacs-25. Is this now the new
>> > workflow? If so, there's not much point in keeping gitmerge.el anyway.
>> 
>> Sorry, forget what I said. I forgot the we don't freeze master anymore,
>> so I guess this was a one-time merge just to forward emacs-25. I was
>> just confused because git couldn't detect the cherry-picks, but this is
>> because of this merge from master.
>
> Could you describe the conditions necessary for gitmerge to detect
> cherry-picks?  I see that it uses --cherry-mark, but would that mark
> cherry-picks whose commit message was edited by "cherry-pick -e"?

Yes. Cherry-picks are detected through the "patch-id", which is not
affected by the commit message but only by the diff (see 'git help
patch-id'). This is why it's fine to change the commit message, but when
you resolve a conflict, the patch-id will usually change.

But when you later *merge* the branch from which you cherry-picked (like
we did with 'master'), git of course sees the cherry-picked commits as
merged, and hence does not check the patch-id later as well (most likely
because that would be slow). At least that's how I understand it.

> Also, what about the effect of the "cherry-pick -x" switch, does
> affect gitmerge in any way?

We could of course add "cherry picked" to the gitmerge-skip-regexp, but
otherwise, the '-x' has no affect on how git itself detects
cherry-picks.

-David

Re: Freeze is almost here

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: schwab@linux-m68k.org,  jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 14 Nov 2015 15:37:51 +0100
> 
> But when you later *merge* the branch from which you cherry-picked (like
> we did with 'master'), git of course sees the cherry-picked commits as
> merged, and hence does not check the patch-id later as well (most likely
> because that would be slow). At least that's how I understand it.

I hope we won't merge from master to emacs-25.

> > Also, what about the effect of the "cherry-pick -x" switch, does
> > affect gitmerge in any way?
> 
> We could of course add "cherry picked" to the gitmerge-skip-regexp, but
> otherwise, the '-x' has no affect on how git itself detects
> cherry-picks.

If detection of cherry-picks is reliable, I see no need to add that to
gitmerge-skip-regexp.

Thanks.

Re: Freeze is almost here

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: schwab@linux-m68k.org,  jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 14 Nov 2015 13:54:02 +0100
> 
> I see now that master was merged into emacs-25. Is this now the new
> workflow?

I hope not.  I tried to prevent that, but evidently failed.

Sorry.

Re: Freeze is almost here

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: schwab@linux-m68k.org,  jwiegley@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 14 Nov 2015 13:54:02 +0100
>> 
>> I see now that master was merged into emacs-25. Is this now the new
>> workflow?
>
> I hope not.  I tried to prevent that, but evidently failed.

Well, I think everyone agrees that we cannot merge 'master' into
'emacs-25' as soon as we have commits 'master' that introduce new
features. The problem is that everyone must know that bug fixes
etc. should from now on land in 'emacs-25'. Didn't we have an 'announce'
list for things like that?

-David

Re: Freeze is almost here

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: schwab@linux-m68k.org,  jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 14 Nov 2015 15:42:18 +0100
> 
> Well, I think everyone agrees that we cannot merge 'master' into
> 'emacs-25' as soon as we have commits 'master' that introduce new
> features.

Yes, and I thought this rule starts as soon as emacs-25 branch was
cut.

> Didn't we have an 'announce' list for things like that?

Not a list, a keyword:

  http://lists.gnu.org/archive/html/emacs-devel/2015-10/msg00062.html

AFAIU, this means the message should have a "Keywords: emacs-announce"
header, or have a "Keywords: emacs-announce" line in the first 5 lines
of its body.  Then list subscribers who specified they only want to
get messages for that topic will receive such a message.

Re: Freeze is almost here (was: Dynamic loading progress)

[Xue Fuqiao]

On Fri, Nov 13, 2015 at 11:39 PM, John Wiegley <jwiegley@gmail.com> wrote:

Hi John,

> I encourage everyone without pressing feature work to make that branch their
> primary working area for the next few months, and to help me categorize the
> bugs appropriately.

Will `master' will be frozen?  If not, I'll make changes to my patch in
the "Feature freezes and Emacs 25" thread.

Re: Freeze is almost here

[John Wiegley]

>>>>> Xue Fuqiao <xfq.free@gmail.com> writes:

> Will `master' will be frozen? If not, I'll make changes to my patch in the
> "Feature freezes and Emacs 25" thread.

Thanks for asking; to clarify: 'master' is not going to be frozen at any
point. Rather, emacs-25 now represents the candidate for 25.1, and is where
work should proceed for fixing bugs.

John

Re: Freeze is almost here

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Fri, 13 Nov 2015 22:06:43 -0800
> Cc: Emacs-devel <emacs-devel@gnu.org>
> 
> >>>>> Xue Fuqiao <xfq.free@gmail.com> writes:
> 
> > Will `master' will be frozen? If not, I'll make changes to my patch in the
> > "Feature freezes and Emacs 25" thread.
> 
> Thanks for asking; to clarify: 'master' is not going to be frozen at any
> point. Rather, emacs-25 now represents the candidate for 25.1, and is where
> work should proceed for fixing bugs.

A further clarification, about documentation: documentation changes
that are relevant to Emacs 25.1 should be committed to the emacs-25
branch, even if they don't technically "fix bugs".

Re: Dynamic loading progress

[Aurélien Aptel]

As promised, you can find a new merge-ready branch called
dynamic-modules-clean-2 on my github repo.

I've incorporated our latest changes (mainly API renaming & merge of
all test modules into one) and edited the commits messages (typos,
removed final period, added Co-authored-by tag when appropriate).

You should be able to cleanly merge or rebase it on master.

Re: Dynamic loading progress

[raman]

Nice! Would be interesting to see if something like json parsing can be
implemented via this FFI  to achieve speed-up when parsing large JSON
responses. 
-- 

Re: Dynamic loading progress

[Steinar Bang]

>>>>> Aurélien Aptel <aurelien.aptel+emacs@gmail.com>:

> You should be able to cleanly merge or rebase it on master.

Hm... wasn't dynamic loading intended as an emacs 25 feature?

Is the starting point of this branch before or after the branch point of
the emacs-25 branch?

Note that if the starting point is a commit later than the branch point,
the branch will need to be rebased to emacs-25.

Re: Dynamic loading progress

[Aurélien Aptel]

I havent followed all the discussions on that, merge it wherever it
needs to so it will be included in emacs 25 :)

Re: Dynamic loading progress

[Ted Zlatanov]

On Tue, 17 Nov 2015 12:08:37 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> I havent followed all the discussions on that, merge it wherever it
AA> needs to so it will be included in emacs 25 :)

It's merged.  I made a separate post about the process, but I hope we
can continue the technical discussion here:

* what planned features remain unfinished?
* are there any known bugs?
* when do we make `--with-modules' the default?
* should there be a separate manual or at least a section in the developer manual?

Thanks
Ted

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 18 Nov 2015 14:38:35 -0500
> 
> * what planned features remain unfinished?
> * are there any known bugs?

Not for me to answer, but in any case new features should go to
master, I think.  Therefore, I suggest to wait with them until what we
have on the release branch stabilizes.  (E.g., I just had to make some
fixes there in the Windows-specific code, and also the code is
formatted not according to our conventions.)

> * when do we make `--with-modules' the default?

Not in Emacs 25.1, I think.

> * should there be a separate manual or at least a section in the developer manual?

Of course!  Every feature should be documented, and this one is no
exception.  I suggest to start with NEWS.

Thanks.

Re: Dynamic loading progress

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> * what planned features remain unfinished?
>> * are there any known bugs?

> Not for me to answer, but in any case new features should go to master, I
> think. Therefore, I suggest to wait with them until what we have on the
> release branch stabilizes. (E.g., I just had to make some fixes there in the
> Windows-specific code, and also the code is formatted not according to our
> conventions.)

>> * when do we make `--with-modules' the default?

> Not in Emacs 25.1, I think.

+1 on both points.

John

Re: Dynamic loading progress

[Ted Zlatanov]

On Wed, 18 Nov 2015 22:58:16 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Wed, 18 Nov 2015 14:38:35 -0500
>> 
>> * what planned features remain unfinished?
>> * are there any known bugs?

EZ> Not for me to answer, but in any case new features should go to
EZ> master, I think.  Therefore, I suggest to wait with them until what we
EZ> have on the release branch stabilizes.  (E.g., I just had to make some
EZ> fixes there in the Windows-specific code, and also the code is
EZ> formatted not according to our conventions.)

I agree that new features should go to master. I wanted to find out, for
when we write the NEWS entries and other docs, and for general tracking
of progress, what deficiencies (unfinished features and bugs) remain.

Sorry for letting bad formatting slip through.  I checked the code but I
guess wasn't thorough.  Would you recommend a specific tool for the
formatting review?

>> * should there be a separate manual or at least a section in the developer manual?

EZ> Of course!  Every feature should be documented, and this one is no
EZ> exception.  I suggest to start with NEWS.

My goal with that question was actually to find out *where* the
developer docs should go, not whether they are necessary.  I think we
all agree on the necessity.  I can try a first stab at them when I know
in what context they'll exist.

Regarding NEWS specifically, Aurélien is absent at the moment and will
be back next week. Maybe Philipp would like to provide the NEWS entries?
I will write them if neither of them is able or if it can't wait until
next week.

Ted

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 18 Nov 2015 21:19:23 -0500
> 
> Sorry for letting bad formatting slip through.  I checked the code but I
> guess wasn't thorough.  Would you recommend a specific tool for the
> formatting review?

I just use my eyes.  Maybe others can recommend some tools.

> >> * should there be a separate manual or at least a section in the developer manual?
> 
> EZ> Of course!  Every feature should be documented, and this one is no
> EZ> exception.  I suggest to start with NEWS.
> 
> My goal with that question was actually to find out *where* the
> developer docs should go, not whether they are necessary.

Sorry, it wasn't clear to me.

I think the place is the ELisp manual.  I see no reason to have a
separate manual, since I don't expect the documentation to be large
enough to justify that.

Thanks.

Re: Dynamic loading progress

[Eli Zaretskii]

Thanks for making this possible.

Please allow me a few questions/comments about the dynamic modules
code, based on some initial reading:

Why module.c, but confusingly emacs_module.h?  Are there any reasons
not to use the same base name, like we do for other C sources?

This comment in module.c:

  /* If checking is enabled, abort if the current thread is not the
     Emacs main thread. */
  static void check_main_thread (void);

confused me, because a later comment says:

  void module_init (void)
  {
    /* It is not guaranteed that dynamic initializers run in the main thread,
       therefore we detect the main thread here. */

If dynamic initializers might run in a thread different from the Emacs
main thread, then the code in module_init will record that other
thread, not the Emacs main thread, right?

Also, another comment:

  /* On Windows, we store both a handle to the main thread and the
     thread ID because the latter can be reused when a thread
     terminates. */

seems to imply that 'main_thread' here is not the Emacs's main thread,
because that thread never terminates as long as the Emacs session is
alive.

So what's the deal here? what does this thread checking supposed to
detect?

In this code from module.c:

      Lisp_Object value = HASH_VALUE (h, i);
      eassert (NATNUMP (value));
      const EMACS_UINT refcount = XFASTINT (value);
      if (refcount >= MOST_POSITIVE_FIXNUM)
        {
          module_non_local_exit_signal_1 (env, Qoverflow_error, Qnil);
          return NULL;
        }

how can the 'if' clause ever be true?  refcount is an Emacs integer,
as you have just verified, no?  And if this somehow can happen, then
why isn't there a similar check in the other functions?

Re this fragment from module.c:

  Lisp_Object ret = list4 (Qlambda,
                           list2 (Qand_rest, Qargs),
                           documentation ? build_string (documentation) : Qnil,
                           list3 (module_call_func,
                                  envobj,
                                  Qargs));

Thou shalt not use build_string, except when you _know_ the argument
will always be a pure-ASCII string.  Practically, this means the
argument must be a constant ASCII string.  See these messages (and the
preceding discussion, if you are interested) for the gory details:

  http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00955.html
  http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00976.html
  http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00979.html

The above should call make_multibyte_string instead.

Again in module.c:

  /*
   * Emacs internal encoding is more-or-less UTF8, let's assume utf8
   * encoded emacs string are the same byte size.
   */

  if (!buffer || length == 0 || *length-1 < raw_size)
    {
      *length = raw_size + 1;
      return false;
    }

I don't understand why you need this assumption.  You are going to
encode the string in a moment, so why not test 'length' against the
size you actually obtain there?  (The above test will misfire when the
original string includes characters above the Unicode codespace, which
require 5 bytes internally, but their encoding maps them to Unicode
codepoints which cannot take more than 4 bytes.  So you might reject
perfectly valid calls.)

In module_make_string you have:

  /* Assume STR is utf8 encoded */
  return lisp_to_value (env, make_string (str, length));

The discussion I pointed to above concluded that <quote>make_string is
a bug</quote>.  So please use make_multibyte_string here instead.

It looks like XUSER_PTR is used both as an lvalue and an rvalue.  This
is different from any other object, where we have separate Xfoo and
XSETfoo macros.  Suggest to follow suit.

  static void module_set_user_finalizer (emacs_env *env,
					     emacs_value uptr,
					     emacs_finalizer_function fin)
  {
    check_main_thread ();
    eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
    const Lisp_Object lisp = value_to_lisp (uptr);
    if (! USER_PTRP (lisp)) module_wrong_type (env, Quser_ptr, lisp);
    XUSER_PTR (lisp)->finalizer = fin; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
  }

No validity checks on 'fin'?

In module_vec_get:

  /* Prevent error-prone comparison between types of different signedness. */
  const size_t size = ASIZE (lvec);
  eassert (size >= 0);

How can the assertion be ever violated?

In module-load:

  CHECK_STRING (file);
  handle = dynlib_open (SDATA (file));

Every Lisp primitive that accepts file arguments _must_ call
expand-file-name on the file name, before using it.  Otherwise,
relative file names will produce subtle and hard-to-debug problems
when the Lisp program calling them involves changing the current
directory of the Emacs process.

The other mandatory thing absent from the above is ENCODE_FILE.  You
cannot pass unencoded file names to C runtime functions.

  struct {
    struct emacs_runtime pub;
    struct emacs_runtime_private priv;
  } runtime = {
    .pub = {
      .size = sizeof runtime.pub,
      .get_environment = module_get_environment,
      .private_members = &runtime.priv
    }
  };

Is this portable enough?

  int r = module_init (&runtime.pub);

I think calling this function "module_init" is sub-optimal: you have a
void function by the same name in this source file.  How about
renaming it to something else?

Also, it seems that once a module is loaded, it cannot be unloaded
(i.e., no unload-feature equivalent is provided).  Is that on purpose?
An Emacs process can live for a very long time, so keeping all of the
modules open in it at all times is not necessarily TRT.  E.g., how do
I update to a newer version of a module?

Shouldn't module-init call dynlib_close before it returns?  Otherwise
we are leaking descriptors here, no?

In module-call:

  const EMACS_INT len = XINT (Flength (arglist));
  eassert (len >= 0);
  if (len > MOST_POSITIVE_FIXNUM)
    xsignal0 (Qoverflow_error);

How can the 'if' clause ever be true?  XINT by definition cannot
produce anything but a valid EMACS_INT, can it?

  if (len > INT_MAX || len < envptr->min_arity || (envptr->max_arity >= 0 && len > envptr->max_arity))
    xsignal2 (Qwrong_number_of_arguments, module_format_fun_env (envptr), make_number (len));

Why the test against INT_MAX?  EMACS_INT can legitimately be a 64-bit
data type with values far exceeding INT_MAX.  Why impose this
limitation here?

allocate_emacs_value calls malloc; shouldn't it call xmalloc instead,
or at least conform to the XMALLOC_BLOCK_INPUT_CHECK protocol?

In module_format_fun_env, you produce a unibyte string, and then use
that in calls to functions like xsignal1, which expect Lisp strings in
their internal multibyte representation.  You should instead decode
the unibyte string (using UTF-8) before you return it.

Btw, I wonder whether we should provide a more friendly capabilities
for retrieving the function name and its module.  dladdr is not
portable enough, and we have all the information at our fingertips
in the module's init function, we just need to keep it instead of
throwing it away.  I envision debugging module-related problems will
not be a very rare situation, so we need any help we can get.  WDYT?

In syms_of_module:

  /* Unintern `module-environments' because it is only used
     internally. */
  Funintern (Qmodule_environments, Qnil);

What if some Lisp defines a (interned) symbol by that name?  Won't
they clash?

The Windows-specific parts of dynlib.c need work, e.g. you cannot pass
UTF-8 encoded file names to Windows APIs.  And there are some other
issues.  I'll take care of that.

About the tests: The Makefile in mod-test is Unix-specific: it uses a
literal .so extension.  I also think the Python script should be
rewritten in Emacs Lisp, so that Python installation is not required.
Finally, all of the module tests and associated files should be moved
into test/, preferably even test/automated/ and made part of the "make
check" run.

Re: Dynamic loading progress

[Paul Eggert]

Eli, thanks very much for doing that review. It's clear we have a lot of 
work to do in the new module code. I have also been reviewing it and 
have some changes sort-of-prepared, and I'll try to start folding them 
in soon.

The main thing I noticed (that was not already in your list or in my 
upcoming changes) are some problems in memory allocation. The module 
code calls malloc directly, instead of xmalloc, for reasons I don't 
understand. This leads to problems, e.g., the memory is not properly 
accounted for by the memory profiler. Why can't the new module code 
invoke xmalloc? More generally, how should modules obtain memory in an 
Emacs-friendly way?

Also, the file name "emacs_module.h" doesn't use the usual Emacs style 
rules; hyphen not underscore. But as you say, "module.h" would be better 
anyway. I'll change it to "module.h" for now; if there's some reason it 
needs an "emacs" prefix we can change it to "emacs-module.h" later.

There are some other things I noticed which I'll try to summarize after 
fixing the stuff that's easy to fix (missing copyright notices, 
formatting, etc.).

Re: Dynamic loading progress

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 19 Nov 2015 07:55:34 -0800
> 
> The main thing I noticed (that was not already in your list or in my 
> upcoming changes) are some problems in memory allocation. The module 
> code calls malloc directly, instead of xmalloc, for reasons I don't 
> understand.

I suspect the reason is the same that it doesn't call CHECK_NUMBER
etc., but instead emulates that in its own code: the authors wanted to
refrain from causing a non-local exit directly, they wanted such exits
to go through their own non-local exit routine.  But that's a guess.

If this is indeed the reason, we should probably look for a better
solution.

> More generally, how should modules obtain memory in an
> Emacs-friendly way?

The code in module.c runs in Emacs, so I think it's friendly enough
(apart of the xmalloc issue).  Or did I misunderstand what you meant?

> There are some other things I noticed which I'll try to summarize after 
> fixing the stuff that's easy to fix (missing copyright notices, 
> formatting, etc.).

Thanks.

Re: Dynamic loading progress

[Paul Eggert]

On 11/19/2015 08:27 AM, Eli Zaretskii wrote:
> The code in module.c runs in Emacs, so I think it's friendly enough
> (apart of the xmalloc issue).  Or did I misunderstand what you meant?

It's the xmalloc issue. That is, if a module contains random C code 
inside a library outside of your control that invokes malloc, I suppose 
it's a lost cause, we can't memory-profile that. But if as part of a 
module you write a wrapper for Emacs, the wrapper code should use 
xmalloc rather than malloc, so that at least we can account for its 
memory and fail in a standard way if it's not available. That sort of thing.

Re: Dynamic loading progress

[Eli Zaretskii]

> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 19 Nov 2015 08:37:53 -0800
> 
> On 11/19/2015 08:27 AM, Eli Zaretskii wrote:
> > The code in module.c runs in Emacs, so I think it's friendly enough
> > (apart of the xmalloc issue).  Or did I misunderstand what you meant?
> 
> It's the xmalloc issue. That is, if a module contains random C code 
> inside a library outside of your control that invokes malloc, I suppose 
> it's a lost cause, we can't memory-profile that. But if as part of a 
> module you write a wrapper for Emacs, the wrapper code should use 
> xmalloc rather than malloc, so that at least we can account for its 
> memory and fail in a standard way if it's not available. That sort of thing.

Right.  I think we should be able to provide an xmalloc-compatible
function for module.c to use, once we know exactly why xmalloc was
avoided.

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 19 Nov 2015 07:55:34 -0800
> Cc: emacs-devel@gnu.org
> 
> Eli, thanks very much for doing that review. It's clear we have a lot of 
> work to do in the new module code. I have also been reviewing it and 
> have some changes sort-of-prepared, and I'll try to start folding them 
> in soon.

Do any of those changes touch modules/mod-test/Makefile (apart of
those in the copyright notice you've already pushed)?  I have a few
changes there, to allow running the tests on MS-Windows, at least
manually, but if you are working on that file, I will wait until you
push.

Thanks.

Re: Dynamic loading progress

[Paul Eggert]

On 11/19/2015 09:26 AM, Eli Zaretskii wrote:
> Do any of those changes touch modules/mod-test/Makefile

Not yet :-).  Please go ahead and install the changes you have in mind, 
I'll merge anything else that comes up.

Re: Dynamic loading progress

[Eli Zaretskii]

> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 19 Nov 2015 09:32:39 -0800
> 
> Please go ahead and install the changes you have in mind, 

Done, thanks.

Re: Dynamic loading progress

[Stephen Leake]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Also, the file name "emacs_module.h" doesn't use the usual Emacs style
> rules; hyphen not underscore. But as you say, "module.h" would be
> better anyway. I'll change it to "module.h" for now; if there's some
> reason it needs an "emacs" prefix we can change it to "emacs-module.h"
> later.

"emacs_module.h" is the only Emacs file that client code (that defines
modules) includes. Therefore, it should have "emacs" in the name for
namespace management; client code might have "module.h" for some other
purpose.

Changing to "emacs-module.h" makes sense.

-- 
-- Stephe

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Ted Zlatanov <tzz@lifelogs.com>,
>   Aurélien Aptel <aurelien.aptel+emacs@gmail.com>,
>   emacs-devel@gnu.org
> Date: Thu, 19 Nov 2015 11:57:48 -0600
> 
> "emacs_module.h" is the only Emacs file that client code (that defines
> modules) includes. Therefore, it should have "emacs" in the name for
> namespace management; client code might have "module.h" for some other
> purpose.
> 
> Changing to "emacs-module.h" makes sense.

But then module.c should be renamed to emacs-module.c.  Or even
emodule.[ch], which is shorter.

Alternatively, we could have an internal and an external header files;
the latter will have to be installed by "make install" in the include
tree, and so doesn't have to be used for internal compilation.

Re: Dynamic loading progress

[Paul Eggert]

On 11/19/2015 10:06 AM, Eli Zaretskii wrote:
> But then module.c should be renamed to emacs-module.c.  Or even
> emodule.[ch], which is shorter.

A quick Google search suggests that Enlightenment uses emodule.h, so if 
we want the name to be globally unique then we'll need a more-specific 
name like emacs-module.h or gemodule.h (and likewise for .c).

> Alternatively, we could have an internal and an external header files;
> the latter will have to be installed by "make install" in the include
> tree, and so doesn't have to be used for internal compilation.

If possible I'd rather use the same .h file for both internal and 
external. Simpler, more-often checked, that sort of thing.

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 19 Nov 2015 10:53:44 -0800
> Cc: aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com, emacs-devel@gnu.org
> 
> On 11/19/2015 10:06 AM, Eli Zaretskii wrote:
> > But then module.c should be renamed to emacs-module.c.  Or even
> > emodule.[ch], which is shorter.
> 
> A quick Google search suggests that Enlightenment uses emodule.h, so if 
> we want the name to be globally unique then we'll need a more-specific 
> name like emacs-module.h or gemodule.h (and likewise for .c).
> 
> > Alternatively, we could have an internal and an external header files;
> > the latter will have to be installed by "make install" in the include
> > tree, and so doesn't have to be used for internal compilation.
> 
> If possible I'd rather use the same .h file for both internal and 
> external. Simpler, more-often checked, that sort of thing.

I'm okay with emacs-module.[ch].

Thanks.

Re: Dynamic loading progress

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I'm okay with emacs-module.[ch].

Me too.

John

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1807 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Do., 19. Nov. 2015 um 16:55 Uhr:

> The main thing I noticed (that was not already in your list or in my
> upcoming changes) are some problems in memory allocation. The module
> code calls malloc directly, instead of xmalloc, for reasons I don't
> understand. This leads to problems, e.g., the memory is not properly
> accounted for by the memory profiler. Why can't the new module code
> invoke xmalloc? More generally, how should modules obtain memory in an
> Emacs-friendly way?
>

The problem with xmalloc is that it can signal (i.e. call longjmp). Modules
are generally not prepared for long jumps, so it's important that no
longjmp escapes into module code. The easiest way to ensure this is to make
code not call longjmp in the first place.
Some of the code in module.c does call longjmp; that is protected by
appropriate calls to setjmp. We could do that with all code, but for cases
like malloc it seemed simple enough to avoid the calls to longjmp
altogether.
Maybe we should add a variant that behaves like xmalloc but is guaranteed
to never call longjmp?


>
> Also, the file name "emacs_module.h" doesn't use the usual Emacs style
> rules; hyphen not underscore.


If a hyphen is preferred, feel free to rename.


> But as you say, "module.h" would be better
> anyway. I'll change it to "module.h" for now; if there's some reason it
> needs an "emacs" prefix we can change it to "emacs-module.h" later.
>
>
Please don't use module.h; this header file is the only one that is
intended for module authors to include, so it needs a globally unique name
which should start with 'emacs'.


> There are some other things I noticed which I'll try to summarize after
> fixing the stuff that's easy to fix (missing copyright notices,
> formatting, etc.).
>
>

[-- Attachment #2: Type: text/html, Size: 2664 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

On 11/19/2015 02:45 PM, Philipp Stephani wrote:
> Maybe we should add a variant that behaves like xmalloc but is 
> guaranteed to never call longjmp?

What should this new xmalloc variant do when it runs out of memory? 
xmalloc never returns NULL, so presumably that should be true for the 
variant as well.

>
> Please don't use module.h; this header file is the only one that is 
> intended for module authors to include, so it needs a globally unique 
> name which should start with 'emacs'.

Yes, that's already done, it's renamed to emacs-module.h now.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 756 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Do., 19. Nov. 2015 um 23:56 Uhr:

> On 11/19/2015 02:45 PM, Philipp Stephani wrote:
> > Maybe we should add a variant that behaves like xmalloc but is
> > guaranteed to never call longjmp?
>
> What should this new xmalloc variant do when it runs out of memory?
> xmalloc never returns NULL, so presumably that should be true for the
> variant as well.
>

Returning NULL is fine. malloc also returns NULL on OOM, and module.c
handles that.


>
> >
> > Please don't use module.h; this header file is the only one that is
> > intended for module authors to include, so it needs a globally unique
> > name which should start with 'emacs'.
>
> Yes, that's already done, it's renamed to emacs-module.h now.
>

Thanks.

[-- Attachment #2: Type: text/html, Size: 1256 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

On 11/19/2015 03:08 PM, Philipp Stephani wrote:
> Returning NULL is fine. malloc also returns NULL on OOM, and module.c 
> handles that.

I'm not so much worried about emacs-module.c. I'm worried about what 
modules will do. If we tell programmers 'just call xmalloc' things will 
be easy for them. If not, they'll be more likely to make mistakes.

Is there some way xmalloc (actually, memory_full) can detect that it was 
invoked from a module rather than from ordinary Emacs code? If so, we 
could have memory_full do something different than call xsignal, in that 
case.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 00:50 Uhr:

> On 11/19/2015 03:08 PM, Philipp Stephani wrote:
> > Returning NULL is fine. malloc also returns NULL on OOM, and module.c
> > handles that.
>
> I'm not so much worried about emacs-module.c. I'm worried about what
> modules will do. If we tell programmers 'just call xmalloc' things will
> be easy for them. If not, they'll be more likely to make mistakes.
>
> Is there some way xmalloc (actually, memory_full) can detect that it was
> invoked from a module rather than from ordinary Emacs code? If so, we
> could have memory_full do something different than call xsignal, in that
> case.
>

Modules cannot (legally) call xmalloc. They need to obtain memory using
malloc or any other standard function. In general, modules cannot call any
functions except those declared in emacs-module.h.

[-- Attachment #2: Type: text/html, Size: 1200 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

On 11/19/2015 03:55 PM, Philipp Stephani wrote:
> Modules cannot (legally) call xmalloc. They need to obtain memory 
> using malloc or any other standard function. In general, modules 
> cannot call any functions except those declared in emacs-module.h.

I'm suggesting that we declare a memory allocator in emacs-module.h. 
Sorry if that wasn't clear.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 00:58 Uhr:

> On 11/19/2015 03:55 PM, Philipp Stephani wrote:
> > Modules cannot (legally) call xmalloc. They need to obtain memory
> > using malloc or any other standard function. In general, modules
> > cannot call any functions except those declared in emacs-module.h.
>
> I'm suggesting that we declare a memory allocator in emacs-module.h.
> Sorry if that wasn't clear.
>

We can do that, but I don't expect modules to use it. Modules will
typically be used to wrap existing C or C++ libraries or make heavy use of
such libraries, and those libraries won't use an Emacs-specific allocator.
What could go wrong if modules use malloc for memory allocation?

[-- Attachment #2: Type: text/html, Size: 1050 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> Modules will
> typically be used to wrap existing C or C++ libraries or make heavy use of
> such libraries, and those libraries won't use an Emacs-specific allocator.

It's quite likely that wrappers will need to allocate memory to marshal data 
back and forth between C and Elisp.  If I wanted to build a module to use GMP, 
for example, I'd want to use GMP's low-level primitives that don't do memory 
management, and have the wrapper manage memory itself.

I suppose some wrappers won't need to manage memory at all, but I'd say most 
libraries need memory, and the overall system will benefit from having a single 
memory manager rather than having each library manage its own memory independently.

Have you used the Emacs memory profiler?

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 22:04 Uhr:

> Philipp Stephani wrote:
> > Modules will
> > typically be used to wrap existing C or C++ libraries or make heavy use
> of
> > such libraries, and those libraries won't use an Emacs-specific
> allocator.
>
> It's quite likely that wrappers will need to allocate memory to marshal
> data
> back and forth between C and Elisp.  If I wanted to build a module to use
> GMP,
> for example, I'd want to use GMP's low-level primitives that don't do
> memory
> management, and have the wrapper manage memory itself.
>
> I suppose some wrappers won't need to manage memory at all, but I'd say
> most
> libraries need memory, and the overall system will benefit from having a
> single
> memory manager rather than having each library manage its own memory
> independently.
>

That sounds reasonable, and since there is no other way for modules to use
Emacs's allocator it would make sense to add it to the module interface
(the env object), as long as modules remain free to pick any allocator.


>
> Have you used the Emacs memory profiler?
>

I think once, but not regularly. Still sounds like a useful thing to have.

[-- Attachment #2: Type: text/html, Size: 1691 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 19 Nov 2015 23:55:57 +0000
> Cc: emacs-devel@gnu.org
> 
> Modules cannot (legally) call xmalloc. They need to obtain memory using malloc
> or any other standard function.

How is calling malloc different from calling xmalloc?  Neither is in
the interface defined in emacs-module.h, right?

> In general, modules cannot call any functions except those declared
> in emacs-module.h.

Can they call functions from external libraries?  (I thought allowing
that was one of the main design goals.)  If so, what's the difference?

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 941 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Fr., 20. Nov. 2015 um 11:11 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Thu, 19 Nov 2015 23:55:57 +0000
> > Cc: emacs-devel@gnu.org
> >
> > Modules cannot (legally) call xmalloc. They need to obtain memory using
> malloc
> > or any other standard function.
>
> How is calling malloc different from calling xmalloc?  Neither is in
> the interface defined in emacs-module.h, right?
>

malloc is a standard C function, xmalloc is an internal Emacs function.


>
> > In general, modules cannot call any functions except those declared
> > in emacs-module.h.
>
> Can they call functions from external libraries?  (I thought allowing
> that was one of the main design goals.)  If so, what's the difference?
>

yes, sorry, I was being sloppy. Modules cannot call any Emacs C functions
except those declared in emacs-module.h. Of course they can call arbitrary
other library functions.

[-- Attachment #2: Type: text/html, Size: 1562 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 12401 bytes --]

Thanks for the thorough review; I'll prepare patches for all of these
issues.

Eli Zaretskii <eliz@gnu.org> schrieb am Do., 19. Nov. 2015 um 16:28 Uhr:

> Thanks for making this possible.
>
> Please allow me a few questions/comments about the dynamic modules
> code, based on some initial reading:
>
> Why module.c, but confusingly emacs_module.h?  Are there any reasons
> not to use the same base name, like we do for other C sources?
>
>
emacs_module.h is intended to be included by module authors. Therefore its
name needs to be globally unique, which in practice means it needs to start
with 'emacs_'.
module.c could be renamed accordingly, if you prefer that. However,
Aurélien picked the simple name because it's not required to be globally
unique.


> This comment in module.c:
>
>   /* If checking is enabled, abort if the current thread is not the
>      Emacs main thread. */
>   static void check_main_thread (void);
>
> confused me, because a later comment says:
>
>   void module_init (void)
>   {
>     /* It is not guaranteed that dynamic initializers run in the main
> thread,
>        therefore we detect the main thread here. */
>
> If dynamic initializers might run in a thread different from the Emacs
> main thread, then the code in module_init will record that other
> thread, not the Emacs main thread, right?
>

No, because module_init is guaranteed to be called from the main thread
because main calls it explicitly.


>
> Also, another comment:
>
>   /* On Windows, we store both a handle to the main thread and the
>      thread ID because the latter can be reused when a thread
>      terminates. */
>
> seems to imply that 'main_thread' here is not the Emacs's main thread,
> because that thread never terminates as long as the Emacs session is
> alive.
>
> So what's the deal here? what does this thread checking supposed to
> detect?
>

This guards against the Emacs main thread having exited while module code
in some other thread is still running and attempting to call Emacs
functions. This is undefined behavior, but we included an explicit check if
checking is enabled because that case is somewhat subtle.


>
> In this code from module.c:
>
>       Lisp_Object value = HASH_VALUE (h, i);
>       eassert (NATNUMP (value));
>       const EMACS_UINT refcount = XFASTINT (value);
>       if (refcount >= MOST_POSITIVE_FIXNUM)
>         {
>           module_non_local_exit_signal_1 (env, Qoverflow_error, Qnil);
>           return NULL;
>         }
>
> how can the 'if' clause ever be true?  refcount is an Emacs integer,
> as you have just verified, no?  And if this somehow can happen, then
> why isn't there a similar check in the other functions?
>

refcount can be MOST_POSITIVE_FIXNUM because that's an inclusive bound.
It's important to check that case because later refcount is incremented by
one, and if it's equal to MOST_POSITIVE_FIXNUM it would be outside the
allowed range afterwards. No other function increments numbers, thus no
other functions need this check.


>
> Re this fragment from module.c:
>
>   Lisp_Object ret = list4 (Qlambda,
>                            list2 (Qand_rest, Qargs),
>                            documentation ? build_string (documentation) :
> Qnil,
>                            list3 (module_call_func,
>                                   envobj,
>                                   Qargs));
>
> Thou shalt not use build_string, except when you _know_ the argument
> will always be a pure-ASCII string.  Practically, this means the
> argument must be a constant ASCII string.  See these messages (and the
> preceding discussion, if you are interested) for the gory details:
>
>   http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00955.html
>   http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00976.html
>   http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00979.html
>
> The above should call make_multibyte_string instead.
>

We had a discussion about encodings in
https://github.com/aaptel/emacs-dynamic-module/issues/37. Sorry that this
didn't get resolved earlier; it's an important point. My suggestion would
be to always mandate specifying an encoding whenever a char* is passed, and
limit that to two or three functions dealing with creating strings and
accessing string contents. Would that address your concerns?


>
> Again in module.c:
>
>   /*
>    * Emacs internal encoding is more-or-less UTF8, let's assume utf8
>    * encoded emacs string are the same byte size.
>    */
>
>   if (!buffer || length == 0 || *length-1 < raw_size)
>     {
>       *length = raw_size + 1;
>       return false;
>     }
>
> I don't understand why you need this assumption.  You are going to
> encode the string in a moment, so why not test 'length' against the
> size you actually obtain there?  (The above test will misfire when the
> original string includes characters above the Unicode codespace, which
> require 5 bytes internally, but their encoding maps them to Unicode
> codepoints which cannot take more than 4 bytes.  So you might reject
> perfectly valid calls.)
>
> In module_make_string you have:
>
>   /* Assume STR is utf8 encoded */
>   return lisp_to_value (env, make_string (str, length));
>
> The discussion I pointed to above concluded that <quote>make_string is
> a bug</quote>.  So please use make_multibyte_string here instead.
>

See above; my suggestion would be to change the string handling code by
limiting encoding and decoding to a small set of functions where the
encoding would have to be specified explicitly.


>
> It looks like XUSER_PTR is used both as an lvalue and an rvalue.  This
> is different from any other object, where we have separate Xfoo and
> XSETfoo macros.  Suggest to follow suit.
>

Agreed.


>
>   static void module_set_user_finalizer (emacs_env *env,
>                                              emacs_value uptr,
>                                              emacs_finalizer_function fin)
>   {
>     check_main_thread ();
>     eassert (module_non_local_exit_check (env) ==
> emacs_funcall_exit_return);
>     const Lisp_Object lisp = value_to_lisp (uptr);
>     if (! USER_PTRP (lisp)) module_wrong_type (env, Quser_ptr, lisp);
>     XUSER_PTR (lisp)->finalizer = fin; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>   }
>
> No validity checks on 'fin'?
>

How should it be validated? In C an arbitrary (invalid) pointer could be
passed. I think we just have to accept that this is UB.


>
> In module_vec_get:
>
>   /* Prevent error-prone comparison between types of different signedness.
> */
>   const size_t size = ASIZE (lvec);
>   eassert (size >= 0);
>
> How can the assertion be ever violated?
>

Yeah, that's a bug. I probably meant size to be declared as ptrdiff_t,
which is what ASIZE returns.


>
> In module-load:
>
>   CHECK_STRING (file);
>   handle = dynlib_open (SDATA (file));
>
> Every Lisp primitive that accepts file arguments _must_ call
> expand-file-name on the file name, before using it.  Otherwise,
> relative file names will produce subtle and hard-to-debug problems
> when the Lisp program calling them involves changing the current
> directory of the Emacs process.
>
> The other mandatory thing absent from the above is ENCODE_FILE.  You
> cannot pass unencoded file names to C runtime functions.
>

OK, will send a patch.


>
>   struct {
>     struct emacs_runtime pub;
>     struct emacs_runtime_private priv;
>   } runtime = {
>     .pub = {
>       .size = sizeof runtime.pub,
>       .get_environment = module_get_environment,
>       .private_members = &runtime.priv
>     }
>   };
>
> Is this portable enough?
>

According to http://en.cppreference.com/w/c/language/struct_initialization
 and https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html designated
initializers are part of C99, which is required for building Emacs.


>
>   int r = module_init (&runtime.pub);
>
> I think calling this function "module_init" is sub-optimal: you have a
> void function by the same name in this source file.  How about
> renaming it to something else?
>

Agree


>
> Also, it seems that once a module is loaded, it cannot be unloaded
> (i.e., no unload-feature equivalent is provided).  Is that on purpose?
> An Emacs process can live for a very long time, so keeping all of the
> modules open in it at all times is not necessarily TRT.  E.g., how do
> I update to a newer version of a module?
>

Unloading is important (
https://github.com/aaptel/emacs-dynamic-module/issues/36), but for now we
decided to delay it to a later version because we expect much discussion
about the precise semantics.


>
> Shouldn't module-init call dynlib_close before it returns?  Otherwise
> we are leaking descriptors here, no?
>

My impression from reading dlclose(3) is that modules shouldn't be unloaded
while they are still used.


>
> In module-call:
>
>   const EMACS_INT len = XINT (Flength (arglist));
>   eassert (len >= 0);
>   if (len > MOST_POSITIVE_FIXNUM)
>     xsignal0 (Qoverflow_error);
>
> How can the 'if' clause ever be true?  XINT by definition cannot
> produce anything but a valid EMACS_INT, can it?
>

True. Not sure what I was thinking. (Could be replaced by an eassert, to
document the assumption.)


>
>   if (len > INT_MAX || len < envptr->min_arity || (envptr->max_arity >= 0
> && len > envptr->max_arity))
>     xsignal2 (Qwrong_number_of_arguments, module_format_fun_env (envptr),
> make_number (len));
>
> Why the test against INT_MAX?  EMACS_INT can legitimately be a 64-bit
> data type with values far exceeding INT_MAX.  Why impose this
> limitation here?
>

Because the nargs argument in the module interface is an int.
If you think functions with more than INT_MAX arguments should be
supported, the type for nargs should be changed to int64.


>
> allocate_emacs_value calls malloc; shouldn't it call xmalloc instead,
> or at least conform to the XMALLOC_BLOCK_INPUT_CHECK protocol?
>

If xmalloc is called, then we need to make sure that no signals (longjmps)
can escape to module code. If appropriate setjmps are in place that should
be doable, but I need to check whether there are edge cases.


>
> In module_format_fun_env, you produce a unibyte string, and then use
> that in calls to functions like xsignal1, which expect Lisp strings in
> their internal multibyte representation.  You should instead decode
> the unibyte string (using UTF-8) before you return it.
>

OK, will send a patch.


>
> Btw, I wonder whether we should provide a more friendly capabilities
> for retrieving the function name and its module.  dladdr is not
> portable enough, and we have all the information at our fingertips
> in the module's init function, we just need to keep it instead of
> throwing it away.  I envision debugging module-related problems will
> not be a very rare situation, so we need any help we can get.  WDYT?
>

Hmm, I don't know whether we have access to the function name without using
dladdr. The user just passes a pointer to the module, not its name.


>
> In syms_of_module:
>
>   /* Unintern `module-environments' because it is only used
>      internally. */
>   Funintern (Qmodule_environments, Qnil);
>
> What if some Lisp defines a (interned) symbol by that name?  Won't
> they clash?
>

I followed the lead of internal-interpreter-environment in eval.c, which
uses the same pattern.


>
> The Windows-specific parts of dynlib.c need work, e.g. you cannot pass
> UTF-8 encoded file names to Windows APIs.  And there are some other
> issues.  I'll take care of that.
>

I can also do that, I should be able to set up a build environment in a VM.


>
> About the tests: The Makefile in mod-test is Unix-specific: it uses a
> literal .so extension.  I also think the Python script should be
> rewritten in Emacs Lisp, so that Python installation is not required.
> Finally, all of the module tests and associated files should be moved
> into test/, preferably even test/automated/ and made part of the "make
> check" run.
>
>
Yes, tracked in https://github.com/aaptel/emacs-dynamic-module/issues/34

[-- Attachment #2: Type: text/html, Size: 17434 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

On 11/19/2015 02:41 PM, Philipp Stephani wrote:
>
> Yeah, that's a bug. I probably meant size to be declared as ptrdiff_t, 
> which is what ASIZE returns.

I've fixed this by changing size_t to ptrdiff_t in the new code (or to 
int, if it's guaranteed to fit in INT_MAX).

> According to 
> http://en.cppreference.com/w/c/language/struct_initialization and 
> https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html designated 
> initializers are part of C99, which is required for building Emacs.

I think designated initializers are pretty safe nowadays. (We don't 
assume all of C99 even now, by the way, just a reasonably portable 
subset; but this should be in the subset).

That particular initializer is overkill, though. It's not obvious to the 
casual reader that it initializes the private members to zero, and I 
doubt whether any such initialization was intended.  (This is one of the 
downsides of C99 initializers.) I simplified it.


I have more work to do in this area and plan to follow up later.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 00:51 Uhr:

> On 11/19/2015 02:41 PM, Philipp Stephani wrote:
> >
> > Yeah, that's a bug. I probably meant size to be declared as ptrdiff_t,
> > which is what ASIZE returns.
>
> I've fixed this by changing size_t to ptrdiff_t in the new code (or to
> int, if it's guaranteed to fit in INT_MAX).
>
>
Thanks. I saw you changed some other members to ptrdiff_t (the public size
members of emacs_runtime and emacs_env), is that intentional, and if so,
what is the reason? Anyway, emacs-module.h now lacks an #include <stddef.h>.

[-- Attachment #2: Type: text/html, Size: 911 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

On 11/19/2015 03:57 PM, Philipp Stephani wrote:
> Thanks. I saw you changed some other members to ptrdiff_t (the public 
> size members of emacs_runtime and emacs_env), is that intentional, and 
> if so, what is the reason?

As a general rule, in Emacs source code we prefer signed arithmetic to 
unsigned, because the latter is so error prone when it comes to 
comparisons. Also, signed arithmetic allows for better low-level 
checking, e.g., with -fsanitize=undefined. (There are a few exceptions, 
e.g., hash values, but they're relatively rare.)  So the module 
interface should encourage the use of signed integer arithmetic when 
possible.

It might also be useful to support modules that, for whatever reason, 
cannot deal with signed integers and must use unsigned integers. That 
could be something we add later, if necessary.

> Anyway, emacs-module.h now lacks an #include <stddef.h>.

Thanks, fixed now.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 01:03 Uhr:

> On 11/19/2015 03:57 PM, Philipp Stephani wrote:
> > Thanks. I saw you changed some other members to ptrdiff_t (the public
> > size members of emacs_runtime and emacs_env), is that intentional, and
> > if so, what is the reason?
>
> As a general rule, in Emacs source code we prefer signed arithmetic to
> unsigned, because the latter is so error prone when it comes to
> comparisons. Also, signed arithmetic allows for better low-level
> checking, e.g., with -fsanitize=undefined. (There are a few exceptions,
> e.g., hash values, but they're relatively rare.)  So the module
> interface should encourage the use of signed integer arithmetic when
> possible.
>

This is generally a good practice.
However for the size members I think there's still a slight advantage in
using size_t. This member will typically be used as follows:

if (env->size >= sizeof *env) {
  // good, object is at least as big as expected
} else {
  // Emacs is too old
}

The return type of sizeof is size_t. Using ptrdiff_t for the size member
means that this is now a comparison between types of different signedness.
Such comparisons are rather subtle, and compilers warn about them (e.g.
clang with -Wextra). So I'd suggest th avoid such comparisons. As we never
do any arithmetic with the size members, the wrap-around behavior is not an
issue in this case.


>
> It might also be useful to support modules that, for whatever reason,
> cannot deal with signed integers and must use unsigned integers. That
> could be something we add later, if necessary.
>

I think we always expect a reasonably standard C or C++ compiler, which has
to support signed integers.


>
> > Anyway, emacs-module.h now lacks an #include <stddef.h>.
>
> Thanks, fixed now.
>

[-- Attachment #2: Type: text/html, Size: 2554 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> This member will typically be used as follows:
>
> if (env->size >= sizeof *env) {
>    // good, object is at least as big as expected
> } else {
>    // Emacs is too old
> }


I'm leery of recommending this sort of checking.  In practice incompatible 
structure layout changes so often involve something more than merely extending a 
structure, that a size check is not worth the trouble; you really want a version 
check.  I might be persuaded that it's a good idea if I saw some code doing it 
effectively, but currently there is no such code in the Emacs sources (the 
module test function doesn't check sizes -- why not?).

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 21:48 Uhr:

> Philipp Stephani wrote:
> > This member will typically be used as follows:
> >
> > if (env->size >= sizeof *env) {
> >    // good, object is at least as big as expected
> > } else {
> >    // Emacs is too old
> > }
>
>
> I'm leery of recommending this sort of checking.  In practice incompatible
> structure layout changes so often involve something more than merely
> extending a
> structure, that a size check is not worth the trouble;


No such incompatible changes can be allowed to the emacs_env and
emacs_runtime structures. Only additions at the end of the structures are
allowed. This is independent of how the version checking is made.


> you really want a version
> check.  I might be persuaded that it's a good idea


See Daniel's initial design (
https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00960.html) for
rationale.


> if I saw some code doing it
> effectively, but currently there is no such code in the Emacs sources


Not inside Emacs, but e.g. the Windows API uses this approach extensively
for version checking, and has been forward- and backward-compatible for
decades with it.


> (the
> module test function doesn't check sizes -- why not?).
>

 Just an omission, we'll fix that. All modules need to do version checks so
they don't run into UB if they use a more recent emacs-module.h than the
Emacs in which they are loaded.

[-- Attachment #2: Type: text/html, Size: 2340 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> the Windows API uses this approach extensively
> for version checking, and has been forward- and backward-compatible for
> decades with it.

This sort of approach might work in the MS-Windows world, which has a single 
dominant supplier.  It's not clear that it would work in the GNU/Linux world, 
where different suppliers have different ABIs.  And even if one takes exactly 
the same Emacs source and module source and runs on the same operating system, 
one can easily build Emacs executables that won't interoperate with modules, 
simply by using different compiler flags.  The sizes might match, but the code 
won't.  So I don't think sizes will suffice, at least, not outside of MS-Windows.

Daniel's message pointed to JNI for an inspiration. JNI uses version numbers 
here (its GetVersion method), not sizes, I assume partly for the reasons 
discussed above. Shouldn't we do the same?

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2510 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Sa., 21. Nov. 2015 um 00:06 Uhr:

> Philipp Stephani wrote:
> > the Windows API uses this approach extensively
> > for version checking, and has been forward- and backward-compatible for
> > decades with it.
>
> This sort of approach might work in the MS-Windows world, which has a
> single
> dominant supplier.  It's not clear that it would work in the GNU/Linux
> world,
> where different suppliers have different ABIs.  And even if one takes
> exactly
> the same Emacs source and module source and runs on the same operating
> system,
> one can easily build Emacs executables that won't interoperate with
> modules,
> simply by using different compiler flags.


Do you refer to different structure layouts due to padding differences?
That would need to be ruled out somehow, maybe by making the padding
explicit in emacs-module.h.
I also don't see how version checks should be able to protect against this.
If a module expects a structure member at a certain offset, it has to be
placed at that offset. There is no way in C how to check for that at
runtime.


>   The sizes might match, but the code
> won't.  So I don't think sizes will suffice, at least, not outside of
> MS-Windows.
>
>
I honestly can't think of a situation where version checking would work but
size checking wouldn't. Given that users will use modules compiled with
arbitrary versions of emacs-module.h together with arbitrary Emacsen, the
following compatibility constraints have to be fulfilled:

   - Types of structure members may never change.
   - Structure members may never be reordered.
   - Structure members may never be removed.
   - New structure members can only be added at the end.

Otherwise we'd need to export different versions of the structures and
module authors would have to pick the correct version at runtime, which is
more error-prone. Given these constraints, version number checks are
equivalent to size checks. This isn't really related to Windows; these are
ABI compatibility constraints that always occur when passing structures.


> Daniel's message pointed to JNI for an inspiration. JNI uses version
> numbers
> here (its GetVersion method), not sizes, I assume partly for the reasons
> discussed above. Shouldn't we do the same?


Not necessarily. Daniel had good reasons to deviate from the JNI design in
this respect: size checks are simpler and less error-prone because users
don't need to remember or look up which member is present in which version.

[-- Attachment #2: Type: text/html, Size: 3220 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> I honestly can't think of a situation where version checking would work but
> size checking wouldn't.

For example, a structure contains an off_t value, and time_t grows from 32 to 
64-bits.  The designer knew this might be a problem because of Y2038 issues, and 
so created padding for the time_t to grow into.

Another example: suppose we change ptrdiff_t back to int.

Another example: suppose we want the same module to work in both narrow and wide 
int Emacs, so we allocate storage for wide integers even though half of them are 
not used in narrow platforms.

What you're saying, if I understand it, is that we promise that we'll never make 
any changes like that.  This sounds overly constraining, at least for now. 
While the interface is experimental we'll retain the freedom to make arbitrary 
changes to it. When it settles down we can think about promises about never ever 
making changes in the future. In the meantime the size field is experimental and 
maybe we'll think of uses for it other than sizes.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2239 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am So., 22. Nov. 2015 um 00:25 Uhr:

> Philipp Stephani wrote:
> > I honestly can't think of a situation where version checking would work
> but
> > size checking wouldn't.
>
> For example, a structure contains an off_t value, and time_t grows from 32
> to
> 64-bits.  The designer knew this might be a problem because of Y2038
> issues, and
> so created padding for the time_t to grow into.
>
> Another example: suppose we change ptrdiff_t back to int.
>

Such issues are why we generally prefer fixed-size integers.
Currently the structures contain only pointers and pointer-sized integers.
If we are careful to keep it that way, sizes can only ever increase.


>
> Another example: suppose we want the same module to work in both narrow
> and wide
> int Emacs, so we allocate storage for wide integers even though half of
> them are
> not used in narrow platforms.
>

The module interface never exposes Lisp_Objects directly, so it's immune to
such considerations.


>
> What you're saying, if I understand it, is that we promise that we'll
> never make
> any changes like that.  This sounds overly constraining, at least for now.
>

Not really, because there are no cases where we'd need to e.g. expose an
off_t field. We only need three types of fields:
- A field for version checking (size should be reasonably fixed)
- A pointer-sized field pointing to private data (size is fixed within a
single address space)
- Function pointer fields for interacting with Emacs (size is fixed within
the address space)
If we ever need to expose an off_t value, it will be in the form of another
function pointer.
Even if, for whatever reason, we must expose an off_t value directly, we
can insert appropriate padding to make sure the size increases.


> While the interface is experimental we'll retain the freedom to make
> arbitrary
> changes to it. When it settles down we can think about promises about
> never ever
> making changes in the future. In the meantime the size field is
> experimental and
> maybe we'll think of uses for it other than sizes.
>

The module interface is supposed to be stable at least for the lifetime of
Emacs 25. At least from our side it's not meant to be experimental.

[-- Attachment #2: Type: text/html, Size: 3100 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

I'm not going to belabor the point. If you want to use sizes as version numbers 
it's not worth fighting over. If we ever change a function API without changing 
the structure layout, I guess we'll have to insert a dummy structure member to 
make the size grow. Sounds like a kludge, but there are worse kludges in Emacs.

Getting back to the original issue, your worry was that a ptrdiff_t size would 
lead to unnecessary warnings.  I didn't get any such warning when compiling this 
with gcc -Wall:

#include <stddef.h>
ptrdiff_t size;

int main (void) { return size < sizeof size; }

I expect the warnings you're worried about occur when comparing an unknown 
ptrdiff_t with an unknown size_t; they should not occur when comparing an 
unknown ptrdiff_t with a size_t constant.  If that's the case, let's leave it 
ptrdiff_t.  And even if it's not the case, I'm inclined to leave it ptrdiff_t, 
as any module code will run into similar issues with the other ptrdiff_t 
components, so why make an exception for this one?

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1592 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am So., 22. Nov. 2015 um 19:38 Uhr:

> I'm not going to belabor the point. If you want to use sizes as version
> numbers
> it's not worth fighting over. If we ever change a function API without
> changing
> the structure layout, I guess we'll have to insert a dummy structure
> member to
> make the size grow. Sounds like a kludge, but there are worse kludges in
> Emacs.
>


Adding back Daniel. Not sure whether he follows this discussion, but he
should definitely participate.


>
> Getting back to the original issue, your worry was that a ptrdiff_t size
> would
> lead to unnecessary warnings.  I didn't get any such warning when
> compiling this
> with gcc -Wall:
>
> #include <stddef.h>
> ptrdiff_t size;
>
> int main (void) { return size < sizeof size; }
>
> I expect the warnings you're worried about occur when comparing an unknown
> ptrdiff_t with an unknown size_t; they should not occur when comparing an
> unknown ptrdiff_t with a size_t constant.


I get a warning with both Clang and GCC when compiling with -Wextra.


>   If that's the case, let's leave it
> ptrdiff_t.  And even if it's not the case, I'm inclined to leave it
> ptrdiff_t,
> as any module code will run into similar issues with the other ptrdiff_t
> components, so why make an exception for this one?
>

What other ptrdiff_t components are there? The only other occurrences I see
are the argument counts (which are never compared to sizes), and the string
lengths (where callers might run into trouble if a buffer is larger than
PTRDIFF_MAX, but that seems unlikely).

[-- Attachment #2: Type: text/html, Size: 2265 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 19 Nov 2015 22:41:09 +0000
> Cc: emacs-devel@gnu.org
> 
> Thanks for the thorough review; I'll prepare patches for all of these issues.

Thank you.

>     This comment in module.c:
>     
>     /* If checking is enabled, abort if the current thread is not the
>     Emacs main thread. */
>     static void check_main_thread (void);
>     
>     confused me, because a later comment says:
>     
>     void module_init (void)
>     {
>     /* It is not guaranteed that dynamic initializers run in the main thread,
>     therefore we detect the main thread here. */
>     
>     If dynamic initializers might run in a thread different from the Emacs
>     main thread, then the code in module_init will record that other
>     thread, not the Emacs main thread, right?
> 
> No, because module_init is guaranteed to be called from the main thread because
> main calls it explicitly.

So you are saying that module_init runs in the main thread, but
dynamic initializers might run in another thread?  How's that
possible?

>     Also, another comment:
>     
>     /* On Windows, we store both a handle to the main thread and the
>     thread ID because the latter can be reused when a thread
>     terminates. */
>     
>     seems to imply that 'main_thread' here is not the Emacs's main thread,
>     because that thread never terminates as long as the Emacs session is
>     alive.
>     
>     So what's the deal here? what does this thread checking supposed to
>     detect?
> 
> This guards against the Emacs main thread having exited while module code in
> some other thread is still running and attempting to call Emacs functions.

On what OS can this happen?  It cannot happen on Windows, AFAIK, so
complicating the code by using a handle is not necessary on Windows,
because the thread ID of the main Emacs thread will never be reused as
long as the Emacs session is alive.  Moreover, we already have that
thread's ID in a global variable called dwMainThreadId, computed
during the session startup (albeit after module_init, so it would need
to be moved to an earlier stage), so we could simply compare against
its value in check_main_thread.

> This is undefined behavior, but we included an explicit check if
> checking is enabled because that case is somewhat subtle.

I'm surprised this could happen on some OS.  Can you point to any
details about this?

>     In this code from module.c:
>     
>     Lisp_Object value = HASH_VALUE (h, i);
>     eassert (NATNUMP (value));
>     const EMACS_UINT refcount = XFASTINT (value);
>     if (refcount >= MOST_POSITIVE_FIXNUM)
>     {
>     module_non_local_exit_signal_1 (env, Qoverflow_error, Qnil);
>     return NULL;
>     }
>     
>     how can the 'if' clause ever be true? refcount is an Emacs integer,
>     as you have just verified, no? And if this somehow can happen, then
>     why isn't there a similar check in the other functions?
> 
> refcount can be MOST_POSITIVE_FIXNUM because that's an inclusive bound. It's
> important to check that case because later refcount is incremented by one, and
> if it's equal to MOST_POSITIVE_FIXNUM it would be outside the allowed range
> afterwards. No other function increments numbers, thus no other functions need
> this check.

OK, but then either there should have been a comment explaining this,
or, better, the test should have been after the addition.  (Which, by
some strange luck -- or maybe something else -- is just what Paul
already did ;-)

>     Re this fragment from module.c:
>     
>     Lisp_Object ret = list4 (Qlambda,
>     list2 (Qand_rest, Qargs),
>     documentation ? build_string (documentation) : Qnil,
>     list3 (module_call_func,
>     envobj,
>     Qargs));
>     
>     Thou shalt not use build_string, except when you _know_ the argument
>     will always be a pure-ASCII string. Practically, this means the
>     argument must be a constant ASCII string. See these messages (and the
>     preceding discussion, if you are interested) for the gory details:
>     
>     http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00955.html
>     http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00976.html
>     http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00979.html
>     
>     The above should call make_multibyte_string instead.
>     
> 
> We had a discussion about encodings in
> https://github.com/aaptel/emacs-dynamic-module/issues/37. Sorry that this
> didn't get resolved earlier; it's an important point. My suggestion would be to
> always mandate specifying an encoding whenever a char* is passed, and limit
> that to two or three functions dealing with creating strings and accessing
> string contents. Would that address your concerns?

No, this is not about encoding at all.  This is about calling
build_string: you should only call it when the argument is a fixed
ASCII string.  If the argument is not fixed or might include non-ASCII
characters, you should call make_multibyte_string instead.  That's
because build_string might decide on its own whether to produce a
unibyte or a multibyte string, out of your control, whereas we always
want a multibyte string in this context.

build_string doesn't encode or decode its argument, it creates a Lisp
object whose text is taken from the argument.  It's similar to
make_number in that respect.

>     Again in module.c:
>     
>     /*
>     * Emacs internal encoding is more-or-less UTF8, let's assume utf8
>     * encoded emacs string are the same byte size.
>     */
>     
>     if (!buffer || length == 0 || *length-1 < raw_size)
>     {
>     *length = raw_size + 1;
>     return false;
>     }
>     
>     I don't understand why you need this assumption. You are going to
>     encode the string in a moment, so why not test 'length' against the
>     size you actually obtain there? (The above test will misfire when the
>     original string includes characters above the Unicode codespace, which
>     require 5 bytes internally, but their encoding maps them to Unicode
>     codepoints which cannot take more than 4 bytes. So you might reject
>     perfectly valid calls.)
>     
>     In module_make_string you have:
>     
>     /* Assume STR is utf8 encoded */
>     return lisp_to_value (env, make_string (str, length));
>     
>     The discussion I pointed to above concluded that <quote>make_string is
>     a bug</quote>. So please use make_multibyte_string here instead.
>     
> 
> See above; my suggestion would be to change the string handling code by
> limiting encoding and decoding to a small set of functions where the encoding
> would have to be specified explicitly.

Once again, that's not the issue.  AFAIU, you have already specified
(albeit implicitly) that all strings passed as arguments to and from
the module functions are UTF-8 encoded.  And that's fine by me, that's
not what these comments are about.

The first comment is about a minor issue in
module_copy_string_contents.  Consider this fragment from that
function:

  ptrdiff_t raw_size = SBYTES (lisp_str);

  /* Emacs internal encoding is more-or-less UTF8, let's assume utf8
     encoded emacs string are the same byte size.  */

  if (!buffer || length == 0 || *length-1 < raw_size)
    {
      *length = raw_size + 1;
      return false;
    }

  Lisp_Object lisp_str_utf8 = ENCODE_UTF_8 (lisp_str);
  eassert (raw_size == SBYTES (lisp_str_utf8));
  *length = raw_size + 1;
  memcpy (buffer, SDATA (lisp_str_utf8), SBYTES (lisp_str_utf8));
  buffer[raw_size] = 0;

The comment and the 'if' clause after it in effect tell that you want
to check whether 'buffer' has enough space to hold the encoded string,
but you don't know how many bytes will be needed for that.  So you
make some assumption (which, as I pointed out, could be wrong in rare
cases), and reject the strings that fail your test.  What I'm sating
is that after the call to ENCODE_UTF_8, you can compute the exact
length in bytes of the UTF-8 encoded string, so if you move the test
there, you won't have to make any assumptions and you won't err.

The second comment is again about calling make_string (which
build_string calls internally): use make_multibyte_string instead.

>     static void module_set_user_finalizer (emacs_env *env,
>     emacs_value uptr,
>     emacs_finalizer_function fin)
>     {
>     check_main_thread ();
>     eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
>     const Lisp_Object lisp = value_to_lisp (uptr);
>     if (! USER_PTRP (lisp)) module_wrong_type (env, Quser_ptr, lisp);
>     XUSER_PTR (lisp)->finalizer = fin; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>     }
>     
>     No validity checks on 'fin'?
>     
> How should it be validated? In C an arbitrary (invalid) pointer could be
> passed. I think we just have to accept that this is UB.

Given the extensive checks elsewhere in the file, this surprised me,
that's all.  How to test it? calling valid_pointer_p would be one
thing I'd suggest.  Maybe there are other (perhaps platform-dependent)
checks possible here, I would have to look around.  For example, it
would be good to make sure this points into executable code.

>     Shouldn't module-init call dynlib_close before it returns? Otherwise
>     we are leaking descriptors here, no?
> 
> My impression from reading dlclose(3) is that modules shouldn't be unloaded
> while they are still used.

Yes, it was my mistake in understanding how the dynamic linking works
in this case.  Sorry.

>     if (len > INT_MAX || len < envptr->min_arity || (envptr->max_arity >= 0 &&
>     len > envptr->max_arity))
>     xsignal2 (Qwrong_number_of_arguments, module_format_fun_env (envptr),
>     make_number (len));
>     
>     Why the test against INT_MAX? EMACS_INT can legitimately be a 64-bit
>     data type with values far exceeding INT_MAX. Why impose this
>     limitation here?
> 
> Because the nargs argument in the module interface is an int.

Shouldn't it be EMACS_INT instead?

> If you think functions with more than INT_MAX arguments should be supported,
> the type for nargs should be changed to int64.

I thought your design decision was already to use int64 for any
integer argument, no?

Anyway, something that comes from Lisp should allow EMACS_INT, IMO.

>     allocate_emacs_value calls malloc; shouldn't it call xmalloc instead,
>     or at least conform to the XMALLOC_BLOCK_INPUT_CHECK protocol?
> 
> If xmalloc is called, then we need to make sure that no signals (longjmps) can
> escape to module code. If appropriate setjmps are in place that should be
> doable, but I need to check whether there are edge cases.

Paul already addressed that: xmalloc in Emacs does double or even
triple duty, and it would be a shame to lose that here.  So I think we
should try to find a way to modify xmalloc such that it satisfies the
module code as well.

>     Btw, I wonder whether we should provide a more friendly capabilities
>     for retrieving the function name and its module. dladdr is not
>     portable enough, and we have all the information at our fingertips
>     in the module's init function, we just need to keep it instead of
>     throwing it away. I envision debugging module-related problems will
>     not be a very rare situation, so we need any help we can get. WDYT?
> 
> Hmm, I don't know whether we have access to the function name without using
> dladdr.

I thought about recording their names when you bind them to Lisp
symbols.

>     About the tests: The Makefile in mod-test is Unix-specific: it uses a
>     literal .so extension. I also think the Python script should be
>     rewritten in Emacs Lisp, so that Python installation is not required.
>     Finally, all of the module tests and associated files should be moved
>     into test/, preferably even test/automated/ and made part of the "make
>     check" run.
> 
> Yes, tracked in https://github.com/aaptel/emacs-dynamic-module/issues/34 

You will see that I put there some preliminary stuff, so at least
these tests can be run manually on Windows, by overriding a single
Make variable from the command line.

Thanks.

Re: Dynamic loading progress

[Eli Zaretskii]

One more comment:

  #elif defined HAVE_UNISTD_H

  /* POSIX systems.  */

  #include <dlfcn.h>

This should probably use HAVE_DLFCN_H instead (computed by configure),
instead of relying on unistd.h.

Btw, 2 of the mod-test tests fail for me today; yesterday they all
passed.  I cannot see what could cause this, since all the relevant
changes seem to me to be only reformatting and more or less trivial
changes in data types.

Re: Dynamic loading progress

[Eli Zaretskii]

> Date: Fri, 20 Nov 2015 13:59:05 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com, emacs-devel@gnu.org
> 
> Btw, 2 of the mod-test tests fail for me today; yesterday they all
> passed.  I cannot see what could cause this, since all the relevant
> changes seem to me to be only reformatting and more or less trivial
> changes in data types.

I suspect the push_handlers changes in aa7dac8.  It looks like
signaling and throwing no longer works in a module.

Paul, could you have a look?  You should be able to see the problem by
running the test in modules/mod-test/.  Two tests there fail.

Re: Dynamic loading progress

[Paul Eggert]

Eli Zaretskii wrote:
> I suspect the push_handlers changes in aa7dac8.

No, it was the reindenting change! I got confused by the do-while indenting in a 
complicated macro. Fixed in commit 95f69f2c5999be4b9444861b6d4ae1bd3ab87f83.  I 
also noticed some int/ptrdiff_t issues in mod-test.c, and fixed them in commit 
39d13206f3b37261bf4a2ebadfc1103414f4fd3e.  Plus, I assume emacs_module_init is 
part of the module API?  If so, it should be declared in emacs-module.h, which I 
did in commit d0e07e0e2239771fd21b9525ea421cf7ba1cc97c.

I didn't notice all these problems because I wasn't running that test.  I was 
just doing 'make check'. Philipp, shouldn't mod-test.c part of 'make check', 
when Emacs is configured --with-modules? Also, shouldn't the mod-test.c Makefile 
use the same GCC warning flags etc. that everyone else does?  That would help 
catch these bugs better.

Re: Dynamic loading progress

[Eli Zaretskii]

> Cc: p.stephani2@gmail.com, aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com,
>  emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 20 Nov 2015 12:39:01 -0800
> 
> Eli Zaretskii wrote:
> > I suspect the push_handlers changes in aa7dac8.
> 
> No, it was the reindenting change! I got confused by the do-while indenting in a 
> complicated macro. Fixed in commit 95f69f2c5999be4b9444861b6d4ae1bd3ab87f83.

Thanks, the tests now pass here as well.

> I didn't notice all these problems because I wasn't running that test.  I was 
> just doing 'make check'. Philipp, shouldn't mod-test.c part of 'make check', 
> when Emacs is configured --with-modules? Also, shouldn't the mod-test.c Makefile 
> use the same GCC warning flags etc. that everyone else does?  That would help 
> catch these bugs better.

Yes, on both counts.  It's a matter of time to settle all that,
though.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Fr., 20. Nov. 2015 um 22:22 Uhr:

> > Cc: p.stephani2@gmail.com, aurelien.aptel+emacs@gmail.com,
> tzz@lifelogs.com,
> >  emacs-devel@gnu.org
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > Date: Fri, 20 Nov 2015 12:39:01 -0800
> >
> > Eli Zaretskii wrote:
> > > I suspect the push_handlers changes in aa7dac8.
> >
> > No, it was the reindenting change! I got confused by the do-while
> indenting in a
> > complicated macro. Fixed in commit
> 95f69f2c5999be4b9444861b6d4ae1bd3ab87f83.
>
> Thanks, the tests now pass here as well.
>
> > I didn't notice all these problems because I wasn't running that test.
> I was
> > just doing 'make check'. Philipp, shouldn't mod-test.c part of 'make
> check',
> > when Emacs is configured --with-modules? Also, shouldn't the mod-test.c
> Makefile
> > use the same GCC warning flags etc. that everyone else does?  That would
> help
> > catch these bugs better.
>
> Yes, on both counts.  It's a matter of time to settle all that,
> though.
>

Absolutely! The tests should be no different from any other tests. There is
https://github.com/aaptel/emacs-dynamic-module/issues/34, but we haven't
gotten around to it yet.

[-- Attachment #2: Type: text/html, Size: 1973 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Eli Zaretskii wrote:
> I thought your design decision was already to use int64 for any
> integer argument, no?

I later changed that to use intmax_t for any integers that might be converted to 
or from Emacs fixnums.  Function argument counts always must fit into ptrdiff_t, 
though, since the arg vector can't be larger than that, so it's OK to limit them 
to ptrdiff_t.

Re: Dynamic loading progress

[Eli Zaretskii]

> Cc: aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 20 Nov 2015 10:44:01 -0800
> 
> Eli Zaretskii wrote:
> > I thought your design decision was already to use int64 for any
> > integer argument, no?
> 
> I later changed that to use intmax_t for any integers that might be converted to 
> or from Emacs fixnums.

Yes, okay.  But intmax_t is a 64-bit type on most modern platforms,
right?

Re: Dynamic loading progress

[Paul Eggert]

Eli Zaretskii wrote:
> intmax_t is a 64-bit type on most modern platforms,
> right?

Yes, this is a no-op change for the platforms I know of.  It's more 
future-proof, that's all.  Neither C99 nor C11 nor even POSIX guarantee that 
int64_t is supported (this is for portability to mainframes with 72-bit longs, 
and I suppose to hypothetical future machines).  Of course it's fine to use 
int64_t in platform-specific code where the platform guarantees that int64_t exists.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 19:44 Uhr:

> Eli Zaretskii wrote:
> > I thought your design decision was already to use int64 for any
> > integer argument, no?
>
> I later changed that to use intmax_t for any integers that might be
> converted to
> or from Emacs fixnums.  Function argument counts always must fit into
> ptrdiff_t,
> though, since the arg vector can't be larger than that, so it's OK to
> limit them
> to ptrdiff_t.
>

Daniel felt pretty strongly about using int64 for fixnums. Is there any
reason to change that? Right now the compile will just fail if Emacs
fixnums don't fit into an int64.

[-- Attachment #2: Type: text/html, Size: 952 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> Daniel felt pretty strongly about using int64 for fixnums.

As I recall, he felt that we shouldn't use EMACS_INT in the module API, and 
int64_t was merely a means to that goal. If so, that's not a reason to prefer 
int64_t over intmax_t; it's merely a reason to make sure that intmax_t is at 
least as portable as int64_t is in this area. Which it is.

 > Is there any reason to change that?

Yes, int64_t is not required by POSIX, C99, etc.  That is, int64_t is an 
optional type.  In contrast, intmax_t is required on all C99 platforms, and it 
has better support (e.g., there's a printf format specifier for it), so there 
are advantages to intmax_t over int64_t.  I don't know of any advantage int64_t 
would have over intmax_t on any platform that Emacs supports.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 21:32 Uhr:

> Philipp Stephani wrote:
> > Daniel felt pretty strongly about using int64 for fixnums.
>
> As I recall, he felt that we shouldn't use EMACS_INT in the module API, and
> int64_t was merely a means to that goal. If so, that's not a reason to
> prefer
> int64_t over intmax_t; it's merely a reason to make sure that intmax_t is
> at
> least as portable as int64_t is in this area. Which it is.
>
>  > Is there any reason to change that?
>
> Yes, int64_t is not required by POSIX, C99, etc.  That is, int64_t is an
> optional type.  In contrast, intmax_t is required on all C99 platforms,
> and it
> has better support (e.g., there's a printf format specifier for it), so
> there
> are advantages to intmax_t over int64_t.  I don't know of any advantage
> int64_t
> would have over intmax_t on any platform that Emacs supports.
>

It would be guaranteed to always have the same size, if available. My
understanding is that intmax_t could differ between compilers even on the
same machine, which would silently break compatibility.

[-- Attachment #2: Type: text/html, Size: 1444 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> intmax_t could differ between compilers even on the
> same machine, which would silently break compatibility.

Doesn't sound very likely.  intmax_t is hardwired into the C library.  If 
compilers disagree about its size, printf would stop working.  Any 
incompatibility this extreme would require treating the different compilers like 
we treat any other differences in platform: you need to recompile your module.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Fr., 20. Nov. 2015 um 21:59 Uhr:

> Philipp Stephani wrote:
> > intmax_t could differ between compilers even on the
> > same machine, which would silently break compatibility.
>
> Doesn't sound very likely.


Maybe not likely, but not impossible.


>   intmax_t is hardwired into the C library.  If
> compilers disagree about its size, printf would stop working.


It would still be possible to link against different C libraries. (However,
that would probably break in other places.)


>   Any
> incompatibility this extreme would require treating the different
> compilers like
> we treat any other differences in platform: you need to recompile your
> module.
>

That might be true, though I'd like to wait for Daniel's assessment: as
said, he felt quite strongly about not using intmax_t.

[-- Attachment #2: Type: text/html, Size: 1440 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> Maybe not likely, but not impossible.

I've never run into it, and have never heard of anyone running into it. In 
contrast, there are a few machines without int64_t -- though admittedly rare, we 
might as well not impose obstacles to porting to them if it's easy, which it is 
here.

intmax_t is not the only alternative. We could also use 'long long int'. That's 
also portable to any C99 host, and should be just as well supported as intmax_t 
is (it has printf formats too). 'long long int' has been around for longer than 
intmax_t has, so in that sense it's a more-conservative choice.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Sa., 21. Nov. 2015 um 00:44 Uhr:

> Philipp Stephani wrote:
> > Maybe not likely, but not impossible.
>
> I've never run into it, and have never heard of anyone running into it. In
> contrast, there are a few machines without int64_t -- though admittedly
> rare, we
> might as well not impose obstacles to porting to them if it's easy, which
> it is
> here.
>
> intmax_t is not the only alternative. We could also use 'long long int'.
> That's
> also portable to any C99 host, and should be just as well supported as
> intmax_t
> is (it has printf formats too). 'long long int' has been around for longer
> than
> intmax_t has, so in that sense it's a more-conservative choice.
>

I'm not feeling strongly about the exact type, but I'd suggest to wait for
Daniel's input before making a final decision.

[-- Attachment #2: Type: text/html, Size: 1191 bytes --]

Re: Dynamic loading progress

[Ted Zlatanov]

On Sat, 21 Nov 2015 08:34:16 +0000 Philipp Stephani <p.stephani2@gmail.com> wrote: 

PS> I'm not feeling strongly about the exact type, but I'd suggest to wait for
PS> Daniel's input before making a final decision.

I would ask John Wiegley to make the final decisions in those cases or
to assign someone to make them for dynamic modules. Daniel is not
available immediately (unless someone has a more direct way to contact
him) and we clearly have enough expertise here.

Ted

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 14612 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Fr., 20. Nov. 2015 um 10:54 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Thu, 19 Nov 2015 22:41:09 +0000
> > Cc: emacs-devel@gnu.org
> >
> > Thanks for the thorough review; I'll prepare patches for all of these
> issues.
>
> Thank you.
>
> >     This comment in module.c:
> >
> >     /* If checking is enabled, abort if the current thread is not the
> >     Emacs main thread. */
> >     static void check_main_thread (void);
> >
> >     confused me, because a later comment says:
> >
> >     void module_init (void)
> >     {
> >     /* It is not guaranteed that dynamic initializers run in the main
> thread,
> >     therefore we detect the main thread here. */
> >
> >     If dynamic initializers might run in a thread different from the
> Emacs
> >     main thread, then the code in module_init will record that other
> >     thread, not the Emacs main thread, right?
> >
> > No, because module_init is guaranteed to be called from the main thread
> because
> > main calls it explicitly.
>
> So you are saying that module_init runs in the main thread, but
> dynamic initializers might run in another thread?  How's that
> possible?
>

According to http://stackoverflow.com/a/29776416 it is not guaranteed that
dynamic initializers run in the main thread.


>
> >     Also, another comment:
> >
> >     /* On Windows, we store both a handle to the main thread and the
> >     thread ID because the latter can be reused when a thread
> >     terminates. */
> >
> >     seems to imply that 'main_thread' here is not the Emacs's main
> thread,
> >     because that thread never terminates as long as the Emacs session is
> >     alive.
> >
> >     So what's the deal here? what does this thread checking supposed to
> >     detect?
> >
> > This guards against the Emacs main thread having exited while module
> code in
> > some other thread is still running and attempting to call Emacs
> functions.
>
> On what OS can this happen?  It cannot happen on Windows, AFAIK,


http://blogs.msdn.com/b/oldnewthing/archive/2010/08/27/10054832.aspx seems
to indicate that it is possible that other threads continue running after
the main thread has exited.


> so
> complicating the code by using a handle is not necessary on Windows,
> because the thread ID of the main Emacs thread will never be reused as
> long as the Emacs session is alive.  Moreover, we already have that
> thread's ID in a global variable called dwMainThreadId, computed
> during the session startup (albeit after module_init, so it would need
> to be moved to an earlier stage), so we could simply compare against
> its value in check_main_thread.
>

My initial code didn't use a thread identifier at all because it could have
been reused after the main thread ended, introducing a subtle bug.


>
> > This is undefined behavior, but we included an explicit check if
> > checking is enabled because that case is somewhat subtle.
>
> I'm surprised this could happen on some OS.  Can you point to any
> details about this?
>

It's not ruled out by the C standard or the APIs of operating systems. If
Emacs doesn't call ExitThread or manipulates the C runtime, then indeed it
can't happen (at least on Windows), but I think it's brittle to rely on
such subtleties without explicitly checking for them (somebody might
introduce a call to ExitThread in the future without looking at module
code).


>
> >     In this code from module.c:
> >
> >     Lisp_Object value = HASH_VALUE (h, i);
> >     eassert (NATNUMP (value));
> >     const EMACS_UINT refcount = XFASTINT (value);
> >     if (refcount >= MOST_POSITIVE_FIXNUM)
> >     {
> >     module_non_local_exit_signal_1 (env, Qoverflow_error, Qnil);
> >     return NULL;
> >     }
> >
> >     how can the 'if' clause ever be true? refcount is an Emacs integer,
> >     as you have just verified, no? And if this somehow can happen, then
> >     why isn't there a similar check in the other functions?
> >
> > refcount can be MOST_POSITIVE_FIXNUM because that's an inclusive bound.
> It's
> > important to check that case because later refcount is incremented by
> one, and
> > if it's equal to MOST_POSITIVE_FIXNUM it would be outside the allowed
> range
> > afterwards. No other function increments numbers, thus no other
> functions need
> > this check.
>
> OK, but then either there should have been a comment explaining this,
> or, better, the test should have been after the addition.  (Which, by
> some strange luck -- or maybe something else -- is just what Paul
> already did ;-)
>

If the test gets moved after the addition, then we should have a
verify(MAX_EMACS_UINT - 1 > MOST_POSITIVE_FIXNUM) or use
__builtin_add_overflow to make sure the addition doesn't cause UB.


>
> >     Re this fragment from module.c:
> >
> >     Lisp_Object ret = list4 (Qlambda,
> >     list2 (Qand_rest, Qargs),
> >     documentation ? build_string (documentation) : Qnil,
> >     list3 (module_call_func,
> >     envobj,
> >     Qargs));
> >
> >     Thou shalt not use build_string, except when you _know_ the argument
> >     will always be a pure-ASCII string. Practically, this means the
> >     argument must be a constant ASCII string. See these messages (and the
> >     preceding discussion, if you are interested) for the gory details:
> >
> >
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00955.html
> >
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00976.html
> >
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00979.html
> >
> >     The above should call make_multibyte_string instead.
> >
> >
> > We had a discussion about encodings in
> > https://github.com/aaptel/emacs-dynamic-module/issues/37. Sorry that
> this
> > didn't get resolved earlier; it's an important point. My suggestion
> would be to
> > always mandate specifying an encoding whenever a char* is passed, and
> limit
> > that to two or three functions dealing with creating strings and
> accessing
> > string contents. Would that address your concerns?
>
> No, this is not about encoding at all.  This is about calling
> build_string: you should only call it when the argument is a fixed
> ASCII string.  If the argument is not fixed or might include non-ASCII
> characters, you should call make_multibyte_string instead.  That's
> because build_string might decide on its own whether to produce a
> unibyte or a multibyte string, out of your control, whereas we always
> want a multibyte string in this context.
>
> build_string doesn't encode or decode its argument, it creates a Lisp
> object whose text is taken from the argument.  It's similar to
> make_number in that respect.
>

I'll send a patch with my changes to explain my intention.


>
> >     Again in module.c:
> >
> >     /*
> >     * Emacs internal encoding is more-or-less UTF8, let's assume utf8
> >     * encoded emacs string are the same byte size.
> >     */
> >
> >     if (!buffer || length == 0 || *length-1 < raw_size)
> >     {
> >     *length = raw_size + 1;
> >     return false;
> >     }
> >
> >     I don't understand why you need this assumption. You are going to
> >     encode the string in a moment, so why not test 'length' against the
> >     size you actually obtain there? (The above test will misfire when the
> >     original string includes characters above the Unicode codespace,
> which
> >     require 5 bytes internally, but their encoding maps them to Unicode
> >     codepoints which cannot take more than 4 bytes. So you might reject
> >     perfectly valid calls.)
> >
> >     In module_make_string you have:
> >
> >     /* Assume STR is utf8 encoded */
> >     return lisp_to_value (env, make_string (str, length));
> >
> >     The discussion I pointed to above concluded that <quote>make_string
> is
> >     a bug</quote>. So please use make_multibyte_string here instead.
> >
> >
> > See above; my suggestion would be to change the string handling code by
> > limiting encoding and decoding to a small set of functions where the
> encoding
> > would have to be specified explicitly.
>
> Once again, that's not the issue.  AFAIU, you have already specified
> (albeit implicitly) that all strings passed as arguments to and from
> the module functions are UTF-8 encoded.  And that's fine by me, that's
> not what these comments are about.
>
> The first comment is about a minor issue in
> module_copy_string_contents.  Consider this fragment from that
> function:
>
>   ptrdiff_t raw_size = SBYTES (lisp_str);
>
>   /* Emacs internal encoding is more-or-less UTF8, let's assume utf8
>      encoded emacs string are the same byte size.  */
>
>   if (!buffer || length == 0 || *length-1 < raw_size)
>     {
>       *length = raw_size + 1;
>       return false;
>     }
>
>   Lisp_Object lisp_str_utf8 = ENCODE_UTF_8 (lisp_str);
>   eassert (raw_size == SBYTES (lisp_str_utf8));
>   *length = raw_size + 1;
>   memcpy (buffer, SDATA (lisp_str_utf8), SBYTES (lisp_str_utf8));
>   buffer[raw_size] = 0;
>
> The comment and the 'if' clause after it in effect tell that you want
> to check whether 'buffer' has enough space to hold the encoded string,
> but you don't know how many bytes will be needed for that.  So you
> make some assumption (which, as I pointed out, could be wrong in rare
> cases), and reject the strings that fail your test.  What I'm sating
> is that after the call to ENCODE_UTF_8, you can compute the exact
> length in bytes of the UTF-8 encoded string, so if you move the test
> there, you won't have to make any assumptions and you won't err.
>
> The second comment is again about calling make_string (which
> build_string calls internally): use make_multibyte_string instead.
>
> >     static void module_set_user_finalizer (emacs_env *env,
> >     emacs_value uptr,
> >     emacs_finalizer_function fin)
> >     {
> >     check_main_thread ();
> >     eassert (module_non_local_exit_check (env) ==
> emacs_funcall_exit_return);
> >     const Lisp_Object lisp = value_to_lisp (uptr);
> >     if (! USER_PTRP (lisp)) module_wrong_type (env, Quser_ptr, lisp);
> >     XUSER_PTR (lisp)->finalizer = fin; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> >     }
> >
> >     No validity checks on 'fin'?
> >
> > How should it be validated? In C an arbitrary (invalid) pointer could be
> > passed. I think we just have to accept that this is UB.
>
> Given the extensive checks elsewhere in the file, this surprised me,
> that's all.  How to test it? calling valid_pointer_p would be one
> thing I'd suggest.  Maybe there are other (perhaps platform-dependent)
> checks possible here, I would have to look around.  For example, it
> would be good to make sure this points into executable code.
>

You can't validate pointers in C. For example, is the following pointer
valid?

long a;
double *p = (double *)(&a);

The answer is no; this is an aliasing violation, and dereferencing p is UB,
but you won't detect this by trying to read the process's address space.
See also http://blogs.msdn.com/b/oldnewthing/archive/2006/09/27/773741.aspx.
The best way to detect such issues is running under asan.


>
> >     if (len > INT_MAX || len < envptr->min_arity || (envptr->max_arity
> >= 0 &&
> >     len > envptr->max_arity))
> >     xsignal2 (Qwrong_number_of_arguments, module_format_fun_env (envptr),
> >     make_number (len));
> >
> >     Why the test against INT_MAX? EMACS_INT can legitimately be a 64-bit
> >     data type with values far exceeding INT_MAX. Why impose this
> >     limitation here?
> >
> > Because the nargs argument in the module interface is an int.
>
> Shouldn't it be EMACS_INT instead?
>

EMACS_INT is an internal type that isn't exposed to module authors. It
should be either int or int64.


>
> > If you think functions with more than INT_MAX arguments should be
> supported,
> > the type for nargs should be changed to int64.
>
> I thought your design decision was already to use int64 for any
> integer argument, no?
>

Daniel's initial design (
https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00960.html) had
int, but I guess he wouldn't mind changing this to int64.


>
> Anyway, something that comes from Lisp should allow EMACS_INT, IMO.
>

It can't use EMACS_INT because that's an internal type defined in lisp.h,
and emacs-module.h can't include lisp.h. Furthermore is should be a type
with a static size to allow for ABI stability. Mismatches should be dealt
with by explicit bounds checks.


>
> >     allocate_emacs_value calls malloc; shouldn't it call xmalloc instead,
> >     or at least conform to the XMALLOC_BLOCK_INPUT_CHECK protocol?
> >
> > If xmalloc is called, then we need to make sure that no signals
> (longjmps) can
> > escape to module code. If appropriate setjmps are in place that should be
> > doable, but I need to check whether there are edge cases.
>
> Paul already addressed that: xmalloc in Emacs does double or even
> triple duty, and it would be a shame to lose that here.  So I think we
> should try to find a way to modify xmalloc such that it satisfies the
> module code as well.
>

Could xmalloc grow a variant that is guaranteed not to call longjmp?


>
> >     Btw, I wonder whether we should provide a more friendly capabilities
> >     for retrieving the function name and its module. dladdr is not
> >     portable enough, and we have all the information at our fingertips
> >     in the module's init function, we just need to keep it instead of
> >     throwing it away. I envision debugging module-related problems will
> >     not be a very rare situation, so we need any help we can get. WDYT?
> >
> > Hmm, I don't know whether we have access to the function name without
> using
> > dladdr.
>
> I thought about recording their names when you bind them to Lisp
> symbols.
>

That would be nice, however it would require set_symbol_function,
Fdefalias, etc. to explicitly check for module functions in their RHS. Not
sure whether that's worth it.


>
> >     About the tests: The Makefile in mod-test is Unix-specific: it uses a
> >     literal .so extension. I also think the Python script should be
> >     rewritten in Emacs Lisp, so that Python installation is not required.
> >     Finally, all of the module tests and associated files should be moved
> >     into test/, preferably even test/automated/ and made part of the
> "make
> >     check" run.
> >
> > Yes, tracked in https://github.com/aaptel/emacs-dynamic-module/issues/34
>
> You will see that I put there some preliminary stuff, so at least
> these tests can be run manually on Windows, by overriding a single
> Make variable from the command line.
>
> Thanks.
>

[-- Attachment #2: Type: text/html, Size: 19777 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Fri, 20 Nov 2015 19:58:25 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     So you are saying that module_init runs in the main thread, but
>     dynamic initializers might run in another thread? How's that
>     possible?
> 
> According to http://stackoverflow.com/a/29776416 it is not guaranteed that
> dynamic initializers run in the main thread.

Ah, you are talking about C++ dynamic initializers!  So the model is
that someone writes a module in C++, starts a thread there, and then
inside some dynamic initializer calls emacs-module interface
functions, is that it?  If that's the situation, I'd suggest a
prominent commentary describing this in the source.

>     On what OS can this happen? It cannot happen on Windows, AFAIK,
> 
> http://blogs.msdn.com/b/oldnewthing/archive/2010/08/27/10054832.aspx seems to
> indicate that it is possible that other threads continue running after the main
> thread has exited.

Not in Emacs: we use the C runtime, and never call ExitThread, let
alone ExitProcess.  Doing that would be a silly bug.  There's no need
to make a single component of Emacs, and not the most important one,
protected to such extreme degree against calamities that are only
possible if Emacs developers will lose their senses.

> My initial code didn't use a thread identifier at all because it could have
> been reused after the main thread ended, introducing a subtle bug.

No, it cannot be reused, because we hold a handle on the main thread,
see w32term.c around line 6985.  As long as a single handle is open on
a thread, it still exists from the OS POV, and its ID cannot be
reused.

>     > This is undefined behavior, but we included an explicit check if
>     > checking is enabled because that case is somewhat subtle.
>     
>     I'm surprised this could happen on some OS. Can you point to any
>     details about this?
>     
> 
> It's not ruled out by the C standard or the APIs of operating systems. If Emacs
> doesn't call ExitThread or manipulates the C runtime, then indeed it can't
> happen (at least on Windows), but I think it's brittle to rely on such
> subtleties without explicitly checking for them (somebody might introduce a
> call to ExitThread in the future without looking at module code).

I think these precautions are unnecessary.

Anyway, thanks for explaining this, I now know how to change the code
to DTRT on MS-Windows wrt to the thread checks.

>     OK, but then either there should have been a comment explaining this,
>     or, better, the test should have been after the addition. (Which, by
>     some strange luck -- or maybe something else -- is just what Paul
>     already did ;-)
> 
> If the test gets moved after the addition, then we should have a verify
> (MAX_EMACS_UINT - 1 > MOST_POSITIVE_FIXNUM) or use __builtin_add_overflow to
> make sure the addition doesn't cause UB.

I don't think so.  Please take a look at the code Paul wrote there, I
don't see a problem with it.

>     > No validity checks on 'fin'?
>     >
>     > How should it be validated? In C an arbitrary (invalid) pointer could be
>     > passed. I think we just have to accept that this is UB.
>     
>     Given the extensive checks elsewhere in the file, this surprised me,
>     that's all. How to test it? calling valid_pointer_p would be one
>     thing I'd suggest. Maybe there are other (perhaps platform-dependent)
>     checks possible here, I would have to look around. For example, it
>     would be good to make sure this points into executable code.
>     
> You can't validate pointers in C. For example, is the following pointer valid?
> 
> long a;
> double *p = (double *)(&a);
> 
> The answer is no; this is an aliasing violation, and dereferencing p is UB, but
> you won't detect this by trying to read the process's address space.

The function valid_pointer_p validates a pointer in the sense that it
points into the program's address space.  That's not a full
validation, and won't catch unaligned pointers or pointers into
non-executable portions of memory, but it's better than nothing.  I'd
certainly consider it under "#ifdef ENABLE_CHECKING" at least.

> See also
> http://blogs.msdn.com/b/oldnewthing/archive/2006/09/27/773741.aspx.

We don't use IsBadWritePtr on Windows to check this, see
w32_valid_pointer_p for how this is actually implemented.

Anyway, I'm surprised by this extreme POV: even if we cannot validate
a pointer 100%, surely it doesn't mean we cannot or shouldn't do some
partial job?  Why this "all or nothing" approach?

>     > if (len > INT_MAX || len < envptr->min_arity || (envptr->max_arity >= 0
>     &&
>     > len > envptr->max_arity))
>     > xsignal2 (Qwrong_number_of_arguments, module_format_fun_env (envptr),
>     > make_number (len));
>     >
>     > Why the test against INT_MAX? EMACS_INT can legitimately be a 64-bit
>     > data type with values far exceeding INT_MAX. Why impose this
>     > limitation here?
>     >
>     > Because the nargs argument in the module interface is an int.
>     
>     Shouldn't it be EMACS_INT instead?
> 
> EMACS_INT is an internal type that isn't exposed to module authors.

OK, but the type of nargs is ptrdiff_t, so the test should be against
the maximum of that type.  On 64-bit hosts, ptrdiff_t is a 64-bit
type, while an int is still 32-bit wide.

> Could xmalloc grow a variant that is guaranteed not to call longjmp?

We need to devise a way for it to detect that it was called from
emacs-module.c, the rest is simple, I think.

Thanks.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 7962 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Fr., 20. Nov. 2015 um 22:56 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Fri, 20 Nov 2015 19:58:25 +0000
> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
> emacs-devel@gnu.org
> >
> >     So you are saying that module_init runs in the main thread, but
> >     dynamic initializers might run in another thread? How's that
> >     possible?
> >
> > According to http://stackoverflow.com/a/29776416 it is not guaranteed
> that
> > dynamic initializers run in the main thread.
>
> Ah, you are talking about C++ dynamic initializers!  So the model is
> that someone writes a module in C++, starts a thread there, and then
> inside some dynamic initializer calls emacs-module interface
> functions, is that it?  If that's the situation, I'd suggest a
> prominent commentary describing this in the source.
>

The SO post talks about C++ but the issue is the same in C. AFAIK with C11
and C++11 the execution models are harmonized.

It seems the comment is overly confusing. It is supposed to warn about the
following. Naively, if you wanted to test whether you are in the main
thread, you would do (module types and naming):

static thread_id main_thread = get_current_thread();
bool in_main_thread() { return get_current_thread() == main_thread; }

The dynamic initializer here is the first "get_current_thread()"; it is not
guaranteed to run in the main thread, so "main_thread" is not guaranteed to
contain the main thread ID. Therefore you have to do:

static thread_id main_thread;  // initialized later
int main() {
  // guaranteed to be in the main thread
  main_thread = get_current_thread();
}

That's all. I'm not aware of any runtime that would run dynamic
initializers outside of the main thread, but it's not impossible and easy
to protect against.



>
> >     On what OS can this happen? It cannot happen on Windows, AFAIK,
> >
> > http://blogs.msdn.com/b/oldnewthing/archive/2010/08/27/10054832.aspx
> seems to
> > indicate that it is possible that other threads continue running after
> the main
> > thread has exited.
>
> Not in Emacs: we use the C runtime, and never call ExitThread, let
> alone ExitProcess.  Doing that would be a silly bug.  There's no need
> to make a single component of Emacs, and not the most important one,
> protected to such extreme degree against calamities that are only
> possible if Emacs developers will lose their senses.
>

Fair enough. We can replace this with a comment ("assume no more threads
are running after main has exited") if you prefer that.


>
> > My initial code didn't use a thread identifier at all because it could
> have
> > been reused after the main thread ended, introducing a subtle bug.
>
> No, it cannot be reused, because we hold a handle on the main thread,
> see w32term.c around line 6985.  As long as a single handle is open on
> a thread, it still exists from the OS POV, and its ID cannot be
> reused.
>
> >     > This is undefined behavior, but we included an explicit check if
> >     > checking is enabled because that case is somewhat subtle.
> >
> >     I'm surprised this could happen on some OS. Can you point to any
> >     details about this?
> >
> >
> > It's not ruled out by the C standard or the APIs of operating systems.
> If Emacs
> > doesn't call ExitThread or manipulates the C runtime, then indeed it
> can't
> > happen (at least on Windows), but I think it's brittle to rely on such
> > subtleties without explicitly checking for them (somebody might
> introduce a
> > call to ExitThread in the future without looking at module code).
>
> I think these precautions are unnecessary.
>
> Anyway, thanks for explaining this, I now know how to change the code
> to DTRT on MS-Windows wrt to the thread checks.
>

This is unfortunately all surprisingly subtle and vaguely defined. See e.g.
http://stackoverflow.com/q/19744250/178761 (apparently the standards are
vague about what happens to detached threads after main has exited).


>
> >     OK, but then either there should have been a comment explaining this,
> >     or, better, the test should have been after the addition. (Which, by
> >     some strange luck -- or maybe something else -- is just what Paul
> >     already did ;-)
> >
> > If the test gets moved after the addition, then we should have a verify
> > (MAX_EMACS_UINT - 1 > MOST_POSITIVE_FIXNUM) or use
> __builtin_add_overflow to
> > make sure the addition doesn't cause UB.
>
> I don't think so.  Please take a look at the code Paul wrote there, I
> don't see a problem with it.
>

There is no problem with the code, but reading it requires knowledge that
there are tag bits, which prevent a possible signed overflow. A verify
would be a compiler-checked comment for that.


>
> >     > No validity checks on 'fin'?
> >     >
> >     > How should it be validated? In C an arbitrary (invalid) pointer
> could be
> >     > passed. I think we just have to accept that this is UB.
> >
> >     Given the extensive checks elsewhere in the file, this surprised me,
> >     that's all. How to test it? calling valid_pointer_p would be one
> >     thing I'd suggest. Maybe there are other (perhaps platform-dependent)
> >     checks possible here, I would have to look around. For example, it
> >     would be good to make sure this points into executable code.
> >
> > You can't validate pointers in C. For example, is the following pointer
> valid?
> >
> > long a;
> > double *p = (double *)(&a);
> >
> > The answer is no; this is an aliasing violation, and dereferencing p is
> UB, but
> > you won't detect this by trying to read the process's address space.
>
> The function valid_pointer_p validates a pointer in the sense that it
> points into the program's address space.  That's not a full
> validation, and won't catch unaligned pointers or pointers into
> non-executable portions of memory, but it's better than nothing.  I'd
> certainly consider it under "#ifdef ENABLE_CHECKING" at least.
>

We can add that for consistency, though I'm not sure whether such checks
don't do more harm than good.


>
> > See also
> > http://blogs.msdn.com/b/oldnewthing/archive/2006/09/27/773741.aspx.
>
> We don't use IsBadWritePtr on Windows to check this, see
> w32_valid_pointer_p for how this is actually implemented.
>
>
Much of this applies generally.
> "But what should I do, then, if somebody passes me a bad pointer?"
> You should crash.


> Anyway, I'm surprised by this extreme POV: even if we cannot validate
> a pointer 100%, surely it doesn't mean we cannot or shouldn't do some
> partial job?  Why this "all or nothing" approach?
>

We can check whether it's NULL. Apart from that, everything else is outside
of the C standard.


>
> >     > if (len > INT_MAX || len < envptr->min_arity || (envptr->max_arity
> >= 0
> >     &&
> >     > len > envptr->max_arity))
> >     > xsignal2 (Qwrong_number_of_arguments, module_format_fun_env
> (envptr),
> >     > make_number (len));
> >     >
> >     > Why the test against INT_MAX? EMACS_INT can legitimately be a
> 64-bit
> >     > data type with values far exceeding INT_MAX. Why impose this
> >     > limitation here?
> >     >
> >     > Because the nargs argument in the module interface is an int.
> >
> >     Shouldn't it be EMACS_INT instead?
> >
> > EMACS_INT is an internal type that isn't exposed to module authors.
>
> OK, but the type of nargs is ptrdiff_t, so the test should be against
> the maximum of that type.  On 64-bit hosts, ptrdiff_t is a 64-bit
> type, while an int is still 32-bit wide.
>

Yes, the type is *now* ptrdiff_t. It was int when I wrote the code :-)


>
> > Could xmalloc grow a variant that is guaranteed not to call longjmp?
>
> We need to devise a way for it to detect that it was called from
> emacs-module.c, the rest is simple, I think.
>
>
Hmm, why does it need to detect anything? Can't it just be a different
function that doesn't signal, similar to push_handler and
push_handler_nosignal?

[-- Attachment #2: Type: text/html, Size: 11254 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> why does it need to detect anything? Can't it just be a different
> function that doesn't signal, similar to push_handler and
> push_handler_nosignal?

Yes, that should work and should be easy to implement. But how would it behave 
to the caller? Will the caller have to check that it doesn't return NULL? I hope 
not; that would be a pain and error-prone.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Sa., 21. Nov. 2015 um 00:29 Uhr:

> Philipp Stephani wrote:
> > why does it need to detect anything? Can't it just be a different
> > function that doesn't signal, similar to push_handler and
> > push_handler_nosignal?
>
> Yes, that should work and should be easy to implement. But how would it
> behave
> to the caller? Will the caller have to check that it doesn't return NULL?
> I hope
> not; that would be a pain and error-prone.
>

Why, that's how malloc also behaves.

[-- Attachment #2: Type: text/html, Size: 833 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> Why, that's how malloc also behaves.

Yes, of course. But this part of malloc's behavior is so painful that Emacs code 
typically never invokes malloc directly; it uses xmalloc, which is guaranteed to 
return non-null. Many modules will have needs similar to Emacs itself in that 
respect, and it would be better if they didn't have to reinvent this so-common 
wheel.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Sa., 21. Nov. 2015 um 01:28 Uhr:

> Philipp Stephani wrote:
> > Why, that's how malloc also behaves.
>
> Yes, of course. But this part of malloc's behavior is so painful that
> Emacs code
> typically never invokes malloc directly; it uses xmalloc, which is
> guaranteed to
> return non-null. Many modules will have needs similar to Emacs itself in
> that
> respect, and it would be better if they didn't have to reinvent this
> so-common
> wheel.
>

But what should the interface be? There are only three possibilities in C:
return a sentinel value, crash, longjmp. Crashing prevents modules from
doing their own error handling or degrading gracefully. longjmp is not
possible in general because arbitrary C (and especially C++) code isn't
prepared for it. So the sentinel value is the only option. This is also
consistent with all the other environment functions that return a pointer.

[-- Attachment #2: Type: text/html, Size: 1236 bytes --]

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
>> Many modules will have needs similar to Emacs itself in
>> >that
>> >respect, and it would be better if they didn't have to reinvent this
>> >so-common
>> >wheel.
>> >
> But what should the interface be? There are only three possibilities in C:
> return a sentinel value, crash, longjmp.

There's a fourth possibility: do what Emacs does. Emacs defines its own 
throw-catch mechanism built atop C. C modules that need to cleanup when unwound 
can do so by registering via the equivalent of record_unwind_protect, and Emacs 
won't longjmp through them without cleaning up.  This already works for xmalloc 
memory-exhaustion failures, and modules can just use the already-existing mechanism.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1465 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am So., 22. Nov. 2015 um 00:10 Uhr:

> Philipp Stephani wrote:
> >> Many modules will have needs similar to Emacs itself in
> >> >that
> >> >respect, and it would be better if they didn't have to reinvent this
> >> >so-common
> >> >wheel.
> >> >
> > But what should the interface be? There are only three possibilities in
> C:
> > return a sentinel value, crash, longjmp.
>
> There's a fourth possibility: do what Emacs does. Emacs defines its own
> throw-catch mechanism built atop C. C modules that need to cleanup when
> unwound
> can do so by registering via the equivalent of record_unwind_protect, and
> Emacs
> won't longjmp through them without cleaning up.  This already works for
> xmalloc
> memory-exhaustion failures, and modules can just use the already-existing
> mechanism.
>

We have already discussed longjmp at length in this thread, and Daniel,
Aurélien and I are all strongly against it. Using longjmp makes it
effectively impossible to write modules in any language but C, and even C
code has to be very careful (e.g. it can't use __attribute__((cleanup))).
longjmp is only possible within a monolithic system that is able to make
closed-world assumptions such as the Emacs binary itself, it is impossible
in generic and portable library code. We have worked hard to remove the
possibility of longjmp leaking into module code, and I won't put that
possibility back in.

[-- Attachment #2: Type: text/html, Size: 1815 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 09:03:39 +0000
> Cc: aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com, emacs-devel@gnu.org
> 
> We have already discussed longjmp at length in this thread, and Daniel,
> Aurélien and I are all strongly against it. Using longjmp makes it effectively
> impossible to write modules in any language but C, and even C code has to be
> very careful (e.g. it can't use __attribute__((cleanup))). longjmp is only
> possible within a monolithic system that is able to make closed-world
> assumptions such as the Emacs binary itself, it is impossible in generic and
> portable library code. We have worked hard to remove the possibility of longjmp
> leaking into module code, and I won't put that possibility back in. 

I don't understand: we already have in emacs-module.c machinery that
handles non-local exits.  Why cannot xmalloc do the same when it's
called from emacs-module.c?

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 17:23 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 22 Nov 2015 09:03:39 +0000
> > Cc: aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com,
> emacs-devel@gnu.org
> >
> > We have already discussed longjmp at length in this thread, and Daniel,
> > Aurélien and I are all strongly against it. Using longjmp makes it
> effectively
> > impossible to write modules in any language but C, and even C code has
> to be
> > very careful (e.g. it can't use __attribute__((cleanup))). longjmp is
> only
> > possible within a monolithic system that is able to make closed-world
> > assumptions such as the Emacs binary itself, it is impossible in generic
> and
> > portable library code. We have worked hard to remove the possibility of
> longjmp
> > leaking into module code, and I won't put that possibility back in.
>
> I don't understand: we already have in emacs-module.c machinery that
> handles non-local exits.  Why cannot xmalloc do the same when it's
> called from emacs-module.c?
>

My impression was that Paul talked about code in modules, not code in
emacs-module.c. The latter can use xmalloc just fine, if appropriate
protection is installed first.

[-- Attachment #2: Type: text/html, Size: 1836 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 16:27:19 +0000
> Cc: eggert@cs.ucla.edu, aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com, 
> 	emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 17:23 Uhr:
> 
>     I don't understand: we already have in emacs-module.c machinery that
>     handles non-local exits. Why cannot xmalloc do the same when it's
>     called from emacs-module.c?
> 
> My impression was that Paul talked about code in modules, not code in
> emacs-module.c. The latter can use xmalloc just fine, if appropriate protection
> is installed first. 

I'll let Paul explain what he was worried about.  What I was worried
about is that memory allocation done by emacs-module.c on behalf of
modules doesn't behave like xmalloc does, doesn't comply to the
MALLOC_BLOCK_INPUT protocol, and cannot be probed by MALLOC_PROBE.  I
think this should be easy to fix.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 1087 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 17:56 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 22 Nov 2015 16:27:19 +0000
> > Cc: eggert@cs.ucla.edu, aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com
> ,
> >       emacs-devel@gnu.org
> >
> > Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 17:23 Uhr:
> >
> >     I don't understand: we already have in emacs-module.c machinery that
> >     handles non-local exits. Why cannot xmalloc do the same when it's
> >     called from emacs-module.c?
> >
> > My impression was that Paul talked about code in modules, not code in
> > emacs-module.c. The latter can use xmalloc just fine, if appropriate
> protection
> > is installed first.
>
> I'll let Paul explain what he was worried about.  What I was worried
> about is that memory allocation done by emacs-module.c on behalf of
> modules doesn't behave like xmalloc does, doesn't comply to the
> MALLOC_BLOCK_INPUT protocol, and cannot be probed by MALLOC_PROBE.  I
> think this should be easy to fix.
>
>
Yes, I've attached a patch.

[-- Attachment #1.2: Type: text/html, Size: 1850 bytes --]

[-- Attachment #2: 0001-Use-xmalloc-in-module-code.patch --]
[-- Type: application/octet-stream, Size: 3862 bytes --]

From 1511fab973d0c054f4279f857412ae0f3dee28ae Mon Sep 17 00:00:00 2001
From: Philipp Stephani <phst@google.com>
Date: Sun, 22 Nov 2015 18:14:22 +0100
Subject: [PATCH] Use xmalloc in module code

---
 src/emacs-module.c | 31 ++++++++++++++++---------------
 1 file changed, 16 insertions(+), 15 deletions(-)

diff --git a/src/emacs-module.c b/src/emacs-module.c
index 84072b9..8705280 100644
--- a/src/emacs-module.c
+++ b/src/emacs-module.c
@@ -136,7 +136,7 @@ struct module_fun_env;
 
 static Lisp_Object module_format_fun_env (const struct module_fun_env *);
 static Lisp_Object value_to_lisp (emacs_value);
-static emacs_value allocate_emacs_value (emacs_env *, struct emacs_value_storage *, Lisp_Object);
+static emacs_value allocate_emacs_value (struct emacs_value_storage *, Lisp_Object);
 static emacs_value lisp_to_value (emacs_env *, Lisp_Object);
 static enum emacs_funcall_exit module_non_local_exit_check (emacs_env *);
 static void check_main_thread (void);
@@ -284,7 +284,7 @@ module_make_global_ref (emacs_env *env, emacs_value ref)
       hash_put (h, new_obj, make_natnum (1), hashcode);
     }
 
-  return allocate_emacs_value (env, &global_storage, new_obj);
+  return allocate_emacs_value (&global_storage, new_obj);
 }
 
 static void
@@ -440,6 +440,7 @@ module_type_of (emacs_env *env, emacs_value value)
 {
   check_main_thread ();
   eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
+  MODULE_HANDLE_SIGNALS;
   return lisp_to_value (env, Ftype_of (value_to_lisp (value)));
 }
 
@@ -478,6 +479,7 @@ module_make_integer (emacs_env *env, intmax_t n)
 {
   check_main_thread ();
   eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
+  MODULE_HANDLE_SIGNALS;
   if (! (MOST_NEGATIVE_FIXNUM <= n && n <= MOST_POSITIVE_FIXNUM))
     {
       module_non_local_exit_signal_1 (env, Qoverflow_error, Qnil);
@@ -850,18 +852,21 @@ module_args_out_of_range (emacs_env *env, Lisp_Object a1, Lisp_Object a2)
 static Lisp_Object
 value_to_lisp (emacs_value v)
 {
+  eassert (v != NULL);
   return v->v;
 }
 
 /* Convert an internal object to an `emacs_value'.  Allocate storage
-   from the environment; return NULL if allocation fails.  */
+   from the environment; raise a signal if allocation fails.  */
 static emacs_value
 lisp_to_value (emacs_env *env, Lisp_Object o)
 {
+  eassert (env != NULL);
   struct emacs_env_private *p = env->private_members;
+  // TODO: Remove the next statement.
   if (p->pending_non_local_exit != emacs_funcall_exit_return)
     return NULL;
-  return allocate_emacs_value (env, &p->storage, o);
+  return allocate_emacs_value (&p->storage, o);
 }
 
 \f
@@ -898,23 +903,19 @@ finalize_storage (struct emacs_value_storage *storage)
     }
 }
 
-/* Allocate a new value from STORAGE and stores OBJ in it.  Return
-   NULL if allocations fails and use ENV for non local exit reporting.  */
+/* Allocate a new value from STORAGE and stores OBJ in it.  Signal an
+   error if allocation fails.  */
 static emacs_value
-allocate_emacs_value (emacs_env *env, struct emacs_value_storage *storage,
+allocate_emacs_value (struct emacs_value_storage *storage,
 		      Lisp_Object obj)
 {
-  eassert (storage->current);
+  eassert (storage != NULL);
+  eassert (storage->current != NULL);
   eassert (storage->current->offset < value_frame_size);
-  eassert (! storage->current->next);
+  eassert (storage->current->next == NULL);
   if (storage->current->offset == value_frame_size - 1)
     {
-      storage->current->next = malloc (sizeof *storage->current->next);
-      if (! storage->current->next)
-        {
-          module_out_of_memory (env);
-          return NULL;
-        }
+      storage->current->next = xmalloc (sizeof *storage->current->next);
       initialize_frame (storage->current->next);
       storage->current = storage->current->next;
     }
-- 
2.6.3

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 1249 bytes --]

Another one containing changelog entries.

Philipp Stephani <p.stephani2@gmail.com> schrieb am So., 22. Nov. 2015 um
18:18 Uhr:

> Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 17:56 Uhr:
>
>> > From: Philipp Stephani <p.stephani2@gmail.com>
>> > Date: Sun, 22 Nov 2015 16:27:19 +0000
>> > Cc: eggert@cs.ucla.edu, aurelien.aptel+emacs@gmail.com,
>> tzz@lifelogs.com,
>> >       emacs-devel@gnu.org
>> >
>> > Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 17:23
>> Uhr:
>> >
>> >     I don't understand: we already have in emacs-module.c machinery that
>> >     handles non-local exits. Why cannot xmalloc do the same when it's
>> >     called from emacs-module.c?
>> >
>> > My impression was that Paul talked about code in modules, not code in
>> > emacs-module.c. The latter can use xmalloc just fine, if appropriate
>> protection
>> > is installed first.
>>
>> I'll let Paul explain what he was worried about.  What I was worried
>> about is that memory allocation done by emacs-module.c on behalf of
>> modules doesn't behave like xmalloc does, doesn't comply to the
>> MALLOC_BLOCK_INPUT protocol, and cannot be probed by MALLOC_PROBE.  I
>> think this should be easy to fix.
>>
>>
> Yes, I've attached a patch.
>

[-- Attachment #1.2: Type: text/html, Size: 2285 bytes --]

[-- Attachment #2: 0001-Use-xmalloc-in-module-code.patch --]
[-- Type: application/octet-stream, Size: 4057 bytes --]

From 81b711aaf0d967f7c20ce8453afb0cfbef5e58e5 Mon Sep 17 00:00:00 2001
From: Philipp Stephani <phst@google.com>
Date: Sun, 22 Nov 2015 18:14:22 +0100
Subject: [PATCH] Use xmalloc in module code

* emacs-module.c (allocate_emacs_value): Use xmalloc instead of malloc.
(module_type_of, module_make_integer): Catch signals.
(lisp_to_value, module_make_global_ref): Adapt to changed interface.
---
 src/emacs-module.c | 31 ++++++++++++++++---------------
 1 file changed, 16 insertions(+), 15 deletions(-)

diff --git a/src/emacs-module.c b/src/emacs-module.c
index 84072b9..8705280 100644
--- a/src/emacs-module.c
+++ b/src/emacs-module.c
@@ -136,7 +136,7 @@ struct module_fun_env;
 
 static Lisp_Object module_format_fun_env (const struct module_fun_env *);
 static Lisp_Object value_to_lisp (emacs_value);
-static emacs_value allocate_emacs_value (emacs_env *, struct emacs_value_storage *, Lisp_Object);
+static emacs_value allocate_emacs_value (struct emacs_value_storage *, Lisp_Object);
 static emacs_value lisp_to_value (emacs_env *, Lisp_Object);
 static enum emacs_funcall_exit module_non_local_exit_check (emacs_env *);
 static void check_main_thread (void);
@@ -284,7 +284,7 @@ module_make_global_ref (emacs_env *env, emacs_value ref)
       hash_put (h, new_obj, make_natnum (1), hashcode);
     }
 
-  return allocate_emacs_value (env, &global_storage, new_obj);
+  return allocate_emacs_value (&global_storage, new_obj);
 }
 
 static void
@@ -440,6 +440,7 @@ module_type_of (emacs_env *env, emacs_value value)
 {
   check_main_thread ();
   eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
+  MODULE_HANDLE_SIGNALS;
   return lisp_to_value (env, Ftype_of (value_to_lisp (value)));
 }
 
@@ -478,6 +479,7 @@ module_make_integer (emacs_env *env, intmax_t n)
 {
   check_main_thread ();
   eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
+  MODULE_HANDLE_SIGNALS;
   if (! (MOST_NEGATIVE_FIXNUM <= n && n <= MOST_POSITIVE_FIXNUM))
     {
       module_non_local_exit_signal_1 (env, Qoverflow_error, Qnil);
@@ -850,18 +852,21 @@ module_args_out_of_range (emacs_env *env, Lisp_Object a1, Lisp_Object a2)
 static Lisp_Object
 value_to_lisp (emacs_value v)
 {
+  eassert (v != NULL);
   return v->v;
 }
 
 /* Convert an internal object to an `emacs_value'.  Allocate storage
-   from the environment; return NULL if allocation fails.  */
+   from the environment; raise a signal if allocation fails.  */
 static emacs_value
 lisp_to_value (emacs_env *env, Lisp_Object o)
 {
+  eassert (env != NULL);
   struct emacs_env_private *p = env->private_members;
+  // TODO: Remove the next statement.
   if (p->pending_non_local_exit != emacs_funcall_exit_return)
     return NULL;
-  return allocate_emacs_value (env, &p->storage, o);
+  return allocate_emacs_value (&p->storage, o);
 }
 
 \f
@@ -898,23 +903,19 @@ finalize_storage (struct emacs_value_storage *storage)
     }
 }
 
-/* Allocate a new value from STORAGE and stores OBJ in it.  Return
-   NULL if allocations fails and use ENV for non local exit reporting.  */
+/* Allocate a new value from STORAGE and stores OBJ in it.  Signal an
+   error if allocation fails.  */
 static emacs_value
-allocate_emacs_value (emacs_env *env, struct emacs_value_storage *storage,
+allocate_emacs_value (struct emacs_value_storage *storage,
 		      Lisp_Object obj)
 {
-  eassert (storage->current);
+  eassert (storage != NULL);
+  eassert (storage->current != NULL);
   eassert (storage->current->offset < value_frame_size);
-  eassert (! storage->current->next);
+  eassert (storage->current->next == NULL);
   if (storage->current->offset == value_frame_size - 1)
     {
-      storage->current->next = malloc (sizeof *storage->current->next);
-      if (! storage->current->next)
-        {
-          module_out_of_memory (env);
-          return NULL;
-        }
+      storage->current->next = xmalloc (sizeof *storage->current->next);
       initialize_frame (storage->current->next);
       storage->current = storage->current->next;
     }
-- 
2.6.3

Re: Dynamic loading progress

[Paul Eggert]

Thanks for the patch, but this handles only part of the problem, namely, the 
memory allocated by emacs-module.c directly. We need a way for modules 
themselves to allocate memory that is properly accounted for, and this 
allocation should be nearly as convenient as xmalloc so that modules don't need 
to check for failure after each allocation.

Since emacs-module.c can invoke xmalloc, surely it can export a memory-allocator 
function implemented via xmalloc, using the same techniques emacs-module.c is 
already using to call xmalloc in your patch. This should work regardless of 
whether the module is written in C or C++.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am So., 22. Nov. 2015 um 19:50 Uhr:

> Thanks for the patch, but this handles only part of the problem, namely,
> the
> memory allocated by emacs-module.c directly. We need a way for modules
> themselves to allocate memory that is properly accounted for, and this
> allocation should be nearly as convenient as xmalloc so that modules don't
> need
> to check for failure after each allocation.
>

This is _impossible_ without using longjmp. We really don't want to allow
longjmp to skip over module frames.
An interface that doesn't use longjmp but returns NULL on error is fine
with me. It's something entirely different though: My patch is only for the
internal Emacs code.

[-- Attachment #2: Type: text/html, Size: 1021 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 19:19:49 +0000
> Cc: aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com, 
> 	Emacs developers <emacs-devel@gnu.org>, Daniel Colascione <dancol@dancol.org>
> 
>     Thanks for the patch, but this handles only part of the problem, namely,
>     the
>     memory allocated by emacs-module.c directly. We need a way for modules
>     themselves to allocate memory that is properly accounted for, and this
>     allocation should be nearly as convenient as xmalloc so that modules don't
>     need
>     to check for failure after each allocation.
> 
> This is _impossible_ without using longjmp.

Why?  Why cannot emacs-module.c expose an allocator, which modules
will then use as a sole source of memory?  And why such an allocator
cannot do exactly what you did in your patch?

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 1045 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 20:26 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 22 Nov 2015 19:19:49 +0000
> > Cc: aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com,
> >       Emacs developers <emacs-devel@gnu.org>, Daniel Colascione <
> dancol@dancol.org>
> >
> >     Thanks for the patch, but this handles only part of the problem,
> namely,
> >     the
> >     memory allocated by emacs-module.c directly. We need a way for
> modules
> >     themselves to allocate memory that is properly accounted for, and
> this
> >     allocation should be nearly as convenient as xmalloc so that modules
> don't
> >     need
> >     to check for failure after each allocation.
> >
> > This is _impossible_ without using longjmp.
>
> Why?  Why cannot emacs-module.c expose an allocator, which modules
> will then use as a sole source of memory?  And why such an allocator
> cannot do exactly what you did in your patch?
>

I've attached a patch for such an allocator, but it doesn't use longjmp.

[-- Attachment #1.2: Type: text/html, Size: 1746 bytes --]

[-- Attachment #2: 0002-Add-functions-to-access-the-Emacs-allocator.patch --]
[-- Type: application/octet-stream, Size: 2767 bytes --]

From 6a39cee30848489b13ddef026a13706353145262 Mon Sep 17 00:00:00 2001
From: Philipp Stephani <phst@google.com>
Date: Sun, 22 Nov 2015 20:52:55 +0100
Subject: [PATCH 2/2] Add functions to access the Emacs allocator

* emacs-module.c (module_allocate_raw_memory)
(module_free_raw_memory): new functions
---
 src/emacs-module.c | 21 +++++++++++++++++++++
 src/emacs-module.h | 11 +++++++++++
 2 files changed, 32 insertions(+)

diff --git a/src/emacs-module.c b/src/emacs-module.c
index 8705280..1f4e3f5 100644
--- a/src/emacs-module.c
+++ b/src/emacs-module.c
@@ -287,6 +287,25 @@ module_make_global_ref (emacs_env *env, emacs_value ref)
   return allocate_emacs_value (&global_storage, new_obj);
 }
 
+static void *
+module_allocate_raw_memory (emacs_env *env, ptrdiff_t size)
+{
+  check_main_thread ();
+  eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
+  MODULE_HANDLE_SIGNALS;
+  return xmalloc (size);
+}
+
+static void
+module_free_raw_memory (emacs_env *env, void *memory)
+{
+  check_main_thread ();
+  eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
+  MODULE_HANDLE_SIGNALS_VOID;
+  eassert (memory != NULL);
+  return xfree (memory);
+}
+
 static void
 module_free_global_ref (emacs_env *env, emacs_value ref)
 {
@@ -953,6 +972,8 @@ initialize_environment (struct env_storage *env)
   env->pub.private_members = &env->priv;
   env->pub.make_global_ref = module_make_global_ref;
   env->pub.free_global_ref = module_free_global_ref;
+  env->pub.allocate_raw_memory = module_allocate_raw_memory;
+  env->pub.free_raw_memory = module_free_raw_memory;
   env->pub.non_local_exit_check = module_non_local_exit_check;
   env->pub.non_local_exit_clear = module_non_local_exit_clear;
   env->pub.non_local_exit_get = module_non_local_exit_get;
diff --git a/src/emacs-module.h b/src/emacs-module.h
index 06fc4c0..ecead23 100644
--- a/src/emacs-module.h
+++ b/src/emacs-module.h
@@ -95,6 +95,17 @@ struct emacs_env_25
   void (*free_global_ref) (emacs_env *env,
 			   emacs_value global_reference);
 
+  /* Uses the Emacs memory allocator to allocate raw memory.  Returns
+     NULL if allocation fails.  Using this function over malloc(3) has
+     the advantage that input is correctly blocked and the memory
+     profiler is notified about the allocation.  Memory allocated with
+     this function must be freed by free_raw_memory.  */
+  void *(*allocate_raw_memory) (emacs_env *env, ptrdiff_t size);
+
+  /* Frees a block of memory allocated by allocate_raw_memory.  The
+     pointer to the memory block must not be NULL.  */
+  void (*free_raw_memory) (emacs_env *env, void *memory);
+
   /* Non-local exit handling.  */
 
   enum emacs_funcall_exit (*non_local_exit_check) (emacs_env *env);
-- 
2.6.3

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 19:58:26 +0000
> Cc: eggert@cs.ucla.edu, aurelien.aptel+emacs@gmail.com, tzz@lifelogs.com, 
> 	emacs-devel@gnu.org, dancol@dancol.org
> 
>     Why? Why cannot emacs-module.c expose an allocator, which modules
>     will then use as a sole source of memory? And why such an allocator
>     cannot do exactly what you did in your patch?
>     
> I've attached a patch for such an allocator, but it doesn't use longjmp. 

Thanks.

Paul, if this is OK with you, please push (Phillip doesn't have write
access to the repository).

Re: Dynamic loading progress

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli> Why?  Why cannot emacs-module.c expose an allocator, which modules
Eli> will then use as a sole source of memory?

Commonly a module will use some other library, which will do whatever it
does.

Tom

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Tom Tromey <tom@tromey.com>
> Cc: Philipp Stephani <p.stephani2@gmail.com>,  aurelien.aptel+emacs@gmail.com,  tzz@lifelogs.com,  eggert@cs.ucla.edu,  dancol@dancol.org,  emacs-devel@gnu.org
> Date: Sun, 22 Nov 2015 19:47:53 -0700
> 
> >>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> Eli> Why?  Why cannot emacs-module.c expose an allocator, which modules
> Eli> will then use as a sole source of memory?
> 
> Commonly a module will use some other library, which will do whatever it
> does.

Yes, of course.  The intent is to provide an allocator for
Emacs-related data.

Re: Dynamic loading progress

[Paul Eggert]

On 11/22/2015 06:47 PM, Tom Tromey wrote:
> Commonly a module will use some other library, which will do whatever it
> does.

Yes, if another library calls the system malloc directly, the module 
author will be stuck with that behavior. However, many libraries let or 
require the caller to allocate memory, or provide ways for the caller to 
specify the memory-allocator functions, and module authors should prefer 
this to having the libraries invoke malloc directly, because Emacs wants 
to do memory accounting etc.

Re: Dynamic loading progress

[Eli Zaretskii]

> Cc: Philipp Stephani <p.stephani2@gmail.com>, aurelien.aptel+emacs@gmail.com,
>  tzz@lifelogs.com, dancol@dancol.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 23 Nov 2015 08:29:16 -0800
> 
> Yes, if another library calls the system malloc directly, the module 
> author will be stuck with that behavior. However, many libraries let or 
> require the caller to allocate memory, or provide ways for the caller to 
> specify the memory-allocator functions, and module authors should prefer 
> this to having the libraries invoke malloc directly, because Emacs wants 
> to do memory accounting etc.

I agree.

This is even more important in the Windows build of Emacs, because we
provide our own implementation of malloc and friends, which should be
less prone to heap fragmentation issues (it is an emulation of
mmap-based malloc).

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Fri, 20 Nov 2015 23:22:49 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     Ah, you are talking about C++ dynamic initializers! So the model is
>     that someone writes a module in C++, starts a thread there, and then
>     inside some dynamic initializer calls emacs-module interface
>     functions, is that it? If that's the situation, I'd suggest a
>     prominent commentary describing this in the source.
> 
> The SO post talks about C++ but the issue is the same in C. AFAIK with C11 and
> C++11 the execution models are harmonized.

It doesn't matter for the issue at hand whether it's C, C++, Java, or
whatever.  My originally incorrect interpretation of what you wrote
was that you are talking about initializers that are part of the
module code, i.e. the emacs_module_init function they implement.

> It seems the comment is overly confusing. It is supposed to warn about the
> following. Naively, if you wanted to test whether you are in the main thread,
> you would do (module types and naming):
> 
> static thread_id main_thread = get_current_thread();
> bool in_main_thread() { return get_current_thread() == main_thread; }
> 
> The dynamic initializer here is the first "get_current_thread()"; it is not
> guaranteed to run in the main thread, so "main_thread" is not guaranteed to
> contain the main thread ID.  Therefore you have to do:
> 
> static thread_id main_thread; // initialized later
> int main() {
> // guaranteed to be in the main thread
> main_thread = get_current_thread();
> }
> 
> That's all. I'm not aware of any runtime that would run dynamic initializers
> outside of the main thread, but it's not impossible and easy to protect
> against.

AFAIK, code that does this:

  static thread_id main_thread = get_current_thread();

is invalid in C, because such initializers cannot call functions.  So
it would surprise me to see code which tried to record its thread ID
before 'main'.

So I think we should reword that comment to be much less mysterious
and confusing than it is now.  (Look how much did we need to talk
about for you to explain to me what was the intent of the comment.)

>     Anyway, thanks for explaining this, I now know how to change the code
>     to DTRT on MS-Windows wrt to the thread checks.
> 
> This is unfortunately all surprisingly subtle and vaguely defined. See e.g.
> http://stackoverflow.com/q/19744250/178761 (apparently the standards are vague
> about what happens to detached threads after main has exited). 

I don't see how that affects the issue at hand.  The issue at hand is
whether a thread ID of the main thread could be reused while some of
the other threads belonging to the Emacs process are still running.
And the answer to that on MS-Windows is AFAIU a sound NO, because as
long as the Emacs process is alive, it holds a handle on the main
thread, which precludes the OS from discarding that thread's kernel
object.  Why? because a thread handle can and is used to query the OS
about that thread's conditions, like its exit code, or wait for its
completion in the likes of WaitForSingleObject.  So the kernel object
that represents the thread must be kept by the OS as long as at least
one open handle for the thread exists, and that prevents the OS from
reusing the thread ID.

>     > See also
>     > http://blogs.msdn.com/b/oldnewthing/archive/2006/09/27/773741.aspx.
>     
>     We don't use IsBadWritePtr on Windows to check this, see
>     w32_valid_pointer_p for how this is actually implemented.
>     
> Much of this applies generally.
> > "But what should I do, then, if somebody passes me a bad pointer?"
> > You should crash.

Which is what we do, since eassert aborts.  We will just do it sooner,
which IME is a Good Thing.

>     Anyway, I'm surprised by this extreme POV: even if we cannot validate
>     a pointer 100%, surely it doesn't mean we cannot or shouldn't do some
>     partial job? Why this "all or nothing" approach?
> 
> We can check whether it's NULL. Apart from that, everything else is outside of
> the C standard.

Emacs is not a Standard C program, far from it.  It uses a lot of
stuff outside of any C standard, and for a very good reason: it is a
large and complicate program with many features that require access to
OS facilities.

IOW, using only Standard C features is not, and cannot be, a
requirement for Emacs code.

>     We need to devise a way for it to detect that it was called from
>     emacs-module.c, the rest is simple, I think.
> 
> Hmm, why does it need to detect anything? Can't it just be a different function
> that doesn't signal, similar to push_handler and push_handler_nosignal? 

I don't think we want each of its callers call the signaling part by
itself.  That would be repeating the problem with malloc itself: many
programs simply neglect to include the code which does TRT when it
returns NULL.  xmalloc solves this, and makes sure the (non-trivial)
error action and message are always the same in that case.

We need a variant of this for memory allocated on behalf of modules, I
think.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 6513 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 21. Nov. 2015 um 09:35 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Fri, 20 Nov 2015 23:22:49 +0000
> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
> emacs-devel@gnu.org
> >
> >     Ah, you are talking about C++ dynamic initializers! So the model is
> >     that someone writes a module in C++, starts a thread there, and then
> >     inside some dynamic initializer calls emacs-module interface
> >     functions, is that it? If that's the situation, I'd suggest a
> >     prominent commentary describing this in the source.
> >
> > The SO post talks about C++ but the issue is the same in C. AFAIK with
> C11 and
> > C++11 the execution models are harmonized.
>
> It doesn't matter for the issue at hand whether it's C, C++, Java, or
> whatever.  My originally incorrect interpretation of what you wrote
> was that you are talking about initializers that are part of the
> module code, i.e. the emacs_module_init function they implement.
>
> > It seems the comment is overly confusing. It is supposed to warn about
> the
> > following. Naively, if you wanted to test whether you are in the main
> thread,
> > you would do (module types and naming):
> >
> > static thread_id main_thread = get_current_thread();
> > bool in_main_thread() { return get_current_thread() == main_thread; }
> >
> > The dynamic initializer here is the first "get_current_thread()"; it is
> not
> > guaranteed to run in the main thread, so "main_thread" is not guaranteed
> to
> > contain the main thread ID.  Therefore you have to do:
> >
> > static thread_id main_thread; // initialized later
> > int main() {
> > // guaranteed to be in the main thread
> > main_thread = get_current_thread();
> > }
> >
> > That's all. I'm not aware of any runtime that would run dynamic
> initializers
> > outside of the main thread, but it's not impossible and easy to protect
> > against.
>
> AFAIK, code that does this:
>
>   static thread_id main_thread = get_current_thread();
>
> is invalid in C, because such initializers cannot call functions.  So
> it would surprise me to see code which tried to record its thread ID
> before 'main'.
>

Indeed, you're right. (Yes, I'm a C++ programmer :))
Sorry for the confusion.


>
> So I think we should reword that comment to be much less mysterious
> and confusing than it is now.  (Look how much did we need to talk
> about for you to explain to me what was the intent of the comment.)
>

Since the alternative isn't even legal, the comment can just go away.


>
> >     Anyway, thanks for explaining this, I now know how to change the code
> >     to DTRT on MS-Windows wrt to the thread checks.
> >
> > This is unfortunately all surprisingly subtle and vaguely defined. See
> e.g.
> > http://stackoverflow.com/q/19744250/178761 (apparently the standards
> are vague
> > about what happens to detached threads after main has exited).
>
> I don't see how that affects the issue at hand.  The issue at hand is
> whether a thread ID of the main thread could be reused while some of
> the other threads belonging to the Emacs process are still running.
> And the answer to that on MS-Windows is AFAIU a sound NO, because as
> long as the Emacs process is alive, it holds a handle on the main
> thread, which precludes the OS from discarding that thread's kernel
> object.  Why? because a thread handle can and is used to query the OS
> about that thread's conditions, like its exit code, or wait for its
> completion in the likes of WaitForSingleObject.  So the kernel object
> that represents the thread must be kept by the OS as long as at least
> one open handle for the thread exists, and that prevents the OS from
> reusing the thread ID.
>

Does it actually hold that handle? It sounds reasonable, but I can't find
it documented.


>
> >     > See also
> >     > http://blogs.msdn.com/b/oldnewthing/archive/2006/09/27/773741.aspx
> .
> >
> >     We don't use IsBadWritePtr on Windows to check this, see
> >     w32_valid_pointer_p for how this is actually implemented.
> >
> > Much of this applies generally.
> > > "But what should I do, then, if somebody passes me a bad pointer?"
> > > You should crash.
>
> Which is what we do, since eassert aborts.  We will just do it sooner,
> which IME is a Good Thing.
>

OK. We should be a bit careful with the current implementation of
valid_pointer_p though, as AFAIK write(2) could cast the pointer it
receives to char* and dereference it, but that's probably no worse than not
checking it at all, and unrelated to modules.


>
> >     Anyway, I'm surprised by this extreme POV: even if we cannot validate
> >     a pointer 100%, surely it doesn't mean we cannot or shouldn't do some
> >     partial job? Why this "all or nothing" approach?
> >
> > We can check whether it's NULL. Apart from that, everything else is
> outside of
> > the C standard.
>
> Emacs is not a Standard C program, far from it.  It uses a lot of
> stuff outside of any C standard, and for a very good reason: it is a
> large and complicate program with many features that require access to
> OS facilities.
>
> IOW, using only Standard C features is not, and cannot be, a
> requirement for Emacs code.
>

Fair enough.


>
> >     We need to devise a way for it to detect that it was called from
> >     emacs-module.c, the rest is simple, I think.
> >
> > Hmm, why does it need to detect anything? Can't it just be a different
> function
> > that doesn't signal, similar to push_handler and push_handler_nosignal?
>
> I don't think we want each of its callers call the signaling part by
> itself.  That would be repeating the problem with malloc itself: many
> programs simply neglect to include the code which does TRT when it
> returns NULL.  xmalloc solves this, and makes sure the (non-trivial)
> error action and message are always the same in that case.
>
> We need a variant of this for memory allocated on behalf of modules, I
> think.
>

But this would require modules to be prepared for handling longjmps, which
in general they aren't.
In an "unsafe" language like C, we can't do without the users' cooperation.
If users ignore NULL returns from malloc, that's a bug and we can't do much
about it. We could make it harder to accidentially ignore the result:
bool emacs_malloc(size_t size, void **out)
__attribute__((warn_unused_result));
but that requires a compiler extension and is not really consistent with
the rest of the module interface, where NULL is regularly returned on
failure.

[-- Attachment #2: Type: text/html, Size: 8752 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 21 Nov 2015 09:19:48 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     > Anyway, thanks for explaining this, I now know how to change the code
>     > to DTRT on MS-Windows wrt to the thread checks.
>     >
>     > This is unfortunately all surprisingly subtle and vaguely defined. See
>     e.g.
>     > http://stackoverflow.com/q/19744250/178761 (apparently the standards are
>     vague
>     > about what happens to detached threads after main has exited).
>     
>     I don't see how that affects the issue at hand. The issue at hand is
>     whether a thread ID of the main thread could be reused while some of
>     the other threads belonging to the Emacs process are still running.
>     And the answer to that on MS-Windows is AFAIU a sound NO, because as
>     long as the Emacs process is alive, it holds a handle on the main
>     thread, which precludes the OS from discarding that thread's kernel
>     object. Why? because a thread handle can and is used to query the OS
>     about that thread's conditions, like its exit code, or wait for its
>     completion in the likes of WaitForSingleObject. So the kernel object
>     that represents the thread must be kept by the OS as long as at least
>     one open handle for the thread exists, and that prevents the OS from
>     reusing the thread ID.
> 
> Does it actually hold that handle? It sounds reasonable, but I can't find it
> documented.

Not sure I understand what you are saying.  We create a handle, and
then never close it.  So it stays open for as long as the process
lives.  If some of the process's threads are still running, the
process is still alive, so the handle is not closed by the OS.

>     I don't think we want each of its callers call the signaling part by
>     itself. That would be repeating the problem with malloc itself: many
>     programs simply neglect to include the code which does TRT when it
>     returns NULL. xmalloc solves this, and makes sure the (non-trivial)
>     error action and message are always the same in that case.
>     
>     We need a variant of this for memory allocated on behalf of modules, I
>     think.
> 
> But this would require modules to be prepared for handling longjmps, which in
> general they aren't.

The jump should be in a module-friendly way, obviously.  IOW, it will
have to call (the moral equivalent of) module_non_local_exit_signal.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2482 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Fr., 20. Nov. 2015 um 10:54 Uhr:

>
> >     Re this fragment from module.c:
> >
> >     Lisp_Object ret = list4 (Qlambda,
> >     list2 (Qand_rest, Qargs),
> >     documentation ? build_string (documentation) : Qnil,
> >     list3 (module_call_func,
> >     envobj,
> >     Qargs));
> >
> >     Thou shalt not use build_string, except when you _know_ the argument
> >     will always be a pure-ASCII string. Practically, this means the
> >     argument must be a constant ASCII string. See these messages (and the
> >     preceding discussion, if you are interested) for the gory details:
> >
> >
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00955.html
> >
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00976.html
> >
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-10/msg00979.html
> >
> >     The above should call make_multibyte_string instead.
> >
> >
> > We had a discussion about encodings in
> > https://github.com/aaptel/emacs-dynamic-module/issues/37. Sorry that
> this
> > didn't get resolved earlier; it's an important point. My suggestion
> would be to
> > always mandate specifying an encoding whenever a char* is passed, and
> limit
> > that to two or three functions dealing with creating strings and
> accessing
> > string contents. Would that address your concerns?
>
> No, this is not about encoding at all.  This is about calling
> build_string: you should only call it when the argument is a fixed
> ASCII string.  If the argument is not fixed or might include non-ASCII
> characters, you should call make_multibyte_string instead.  That's
> because build_string might decide on its own whether to produce a
> unibyte or a multibyte string, out of your control, whereas we always
> want a multibyte string in this context.
>
> build_string doesn't encode or decode its argument, it creates a Lisp
> object whose text is taken from the argument.  It's similar to
> make_number in that respect.
>
>
Let me summarize the issues I see: The internal Emacs encoding can change
between versions (command in mule-conf.el), therefore we shouldn't use it
in the module API. IIUC this rules out make_multibyte_string: it only
accepts the internal encoding. Therefore I proposed to always have users
specify the encoding explicitly and then use code_convert_string_norecord
to create the Lisp string objects. Would that work? (We probably then need
another set of functions for unibyte strings.)

[-- Attachment #2: Type: text/html, Size: 3478 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 21 Nov 2015 09:01:12 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
> Let me summarize the issues I see: The internal Emacs encoding can change
> between versions (command in mule-conf.el), therefore we shouldn't use it in
> the module API. IIUC this rules out make_multibyte_string: it only accepts the
> internal encoding. Therefore I proposed to always have users specify the
> encoding explicitly and then use code_convert_string_norecord to create the
> Lisp string objects. Would that work? (We probably then need another set of
> functions for unibyte strings.)

I'm not sure I'm following, so let's take a step back, okay?

My comments were about using build_string and make_string in 2
functions defined in emacs-module.c: module_make_function and
module_make_string.  Both of these emacs-module.c functions produce
strings for consumption by Emacs, AFAIU: the former produces a doc
string of a function defined by a module, which will be used by
various documentation-related functions and commands within Emacs, the
latter produces a string to be passed to Emacs Lisp code for use as
any other Lisp string.  Do you agree so far?

If you agree, then in both cases the strings these functions return
should be in the internal representation of strings used by Emacs, not
in some encoding like UTF-8 or ISO-8859-1.  (We could also use encoded
strings, but that would require Lisp programs using module functions
to always decode any strings they receive, which is less efficient and
more error-prone.)

(Btw, I don't think we should worry about changing the internal
representation of characters in Emacs, because make_multibyte_string
will be updated as needed.)

This is what my comments were about.  I think that you, by contrast,
are talking about the encoding of the _input_ strings, in this case
the 'documentation' argument to module_make_function and 'str'
argument to module_make_string.  My assumption was that these
arguments will always have to be in UTF-8 encoding; if that assumption
is true, then no decoding via code_convert_string_norecord is
necessary, since make_multibyte_string will DTRT.  We can (and
probably should) document the fact that all non-ASCII strings must be
UTF-8 encoded as a requirement of the emacs-module interface.

If you are thinking about accepting strings encoded in other
encodings, I'd consider this an extension, to be added later if
needed.  After all, a module can easily convert to UTF-8 by itself,
using facilities such as iconv.

In any case, code_convert_string_norecord cannot be the complete
solution, because it accepts Lisp string objects, not C strings.  You
still need to create a Lisp string (but this time using
make_unibyte_string).  The point is to always use either
make_unibyte_string or make_multibyte_string, and never build_string
or make_string; the latter 2 should only be used for fixed ASCII-only
strings.

I hope this clarifies the issue.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 4165 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 21. Nov. 2015 um 10:30 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sat, 21 Nov 2015 09:01:12 +0000
> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
> emacs-devel@gnu.org
> >
> > Let me summarize the issues I see: The internal Emacs encoding can change
> > between versions (command in mule-conf.el), therefore we shouldn't use
> it in
> > the module API. IIUC this rules out make_multibyte_string: it only
> accepts the
> > internal encoding. Therefore I proposed to always have users specify the
> > encoding explicitly and then use code_convert_string_norecord to create
> the
> > Lisp string objects. Would that work? (We probably then need another set
> of
> > functions for unibyte strings.)
>
> I'm not sure I'm following, so let's take a step back, okay?
>
> My comments were about using build_string and make_string in 2
> functions defined in emacs-module.c: module_make_function and
> module_make_string.  Both of these emacs-module.c functions produce
> strings for consumption by Emacs, AFAIU: the former produces a doc
> string of a function defined by a module, which will be used by
> various documentation-related functions and commands within Emacs, the
> latter produces a string to be passed to Emacs Lisp code for use as
> any other Lisp string.  Do you agree so far?
>

Yes.


>
> If you agree, then in both cases the strings these functions return
> should be in the internal representation of strings used by Emacs, not
> in some encoding like UTF-8 or ISO-8859-1.  (We could also use encoded
> strings, but that would require Lisp programs using module functions
> to always decode any strings they receive, which is less efficient and
> more error-prone.)
>

Yes. Just for understanding: there are two types of strings: unibyte (just
a sequence of chars), and multibyte (sequence of chars interpreted in the
internal Emacs encoding), right?


>
> (Btw, I don't think we should worry about changing the internal
> representation of characters in Emacs, because make_multibyte_string
> will be updated as needed.)
>

This is a crucial point. If the internal encoding never changes, then we
can declare that those string parameters are expected to be in the internal
encoding. But see the discussion in
https://github.com/aaptel/emacs-dynamic-module/issues/37: the comment in
mule-conf.el seems to indicate that the internal encoding is not stable.


>
> This is what my comments were about.  I think that you, by contrast,
> are talking about the encoding of the _input_ strings, in this case
> the 'documentation' argument to module_make_function and 'str'
> argument to module_make_string.  My assumption was that these
> arguments will always have to be in UTF-8 encoding; if that assumption
> is true, then no decoding via code_convert_string_norecord is
> necessary, since make_multibyte_string will DTRT.  We can (and
> probably should) document the fact that all non-ASCII strings must be
> UTF-8 encoded as a requirement of the emacs-module interface.
>

Or rather, an extension to UTF-8 capable of encoding surrogate code points
and numbers that are not code points, as described in
https://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Representations.html
.


>
> If you are thinking about accepting strings encoded in other
> encodings, I'd consider this an extension, to be added later if
> needed.  After all, a module can easily convert to UTF-8 by itself,
> using facilities such as iconv.
>

Yes, provided the internal Emacs encoding is stable.


>
> In any case, code_convert_string_norecord cannot be the complete
> solution, because it accepts Lisp string objects, not C strings.  You
> still need to create a Lisp string (but this time using
> make_unibyte_string).  The point is to always use either
> make_unibyte_string or make_multibyte_string, and never build_string
> or make_string; the latter 2 should only be used for fixed ASCII-only
> strings.
>
>
Yes, that's fine, the question is about whether the internal encoding is
stable. If it's stable, we can use make_multibyte_string; if not, we can
only use make_unibyte_string.

[-- Attachment #2: Type: text/html, Size: 5844 bytes --]

Re: Dynamic loading progress

[David Kastrup]

Philipp Stephani <p.stephani2@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 21. Nov. 2015 um 10:30 Uhr:
>
>> This is what my comments were about.  I think that you, by contrast,
>> are talking about the encoding of the _input_ strings, in this case
>> the 'documentation' argument to module_make_function and 'str'
>> argument to module_make_string.  My assumption was that these
>> arguments will always have to be in UTF-8 encoding; if that
>> assumption is true, then no decoding via code_convert_string_norecord
>> is necessary, since make_multibyte_string will DTRT.  We can (and
>> probably should) document the fact that all non-ASCII strings must be
>> UTF-8 encoded as a requirement of the emacs-module interface.
>>
>
> Or rather, an extension to UTF-8 capable of encoding surrogate code
> points and numbers that are not code points, as described in
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Representations.html
> .

That's mostly irrelevant for fixed strings as valid UTF-8 is encoded as
itself and invalid UTF-8 is encoded as well-processable invalid UTF-8.
Apart from strings generated with `string-as-multibyte' (what kind of
terrible idea is that?) or unibyte strings, all Emacs strings are
well-processable (meaning that they are represented by start byte and
extension bytes, with the start byte encoding the byte count 1-6 in the
UTF-8 typical manner even if UTF-8 itself does not go beyond the Unicode
range encodable in 4 bytes).

> Yes, provided the internal Emacs encoding is stable.

It has changed several times in the past.  It's a reasonably good bet
that the basics of its current UTF-8 scheme will stick around.

-- 
David Kastrup

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 21 Nov 2015 10:31:24 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     If you agree, then in both cases the strings these functions return
>     should be in the internal representation of strings used by Emacs, not
>     in some encoding like UTF-8 or ISO-8859-1. (We could also use encoded
>     strings, but that would require Lisp programs using module functions
>     to always decode any strings they receive, which is less efficient and
>     more error-prone.)
> 
> Yes. Just for understanding: there are two types of strings: unibyte (just a
> sequence of chars), and multibyte (sequence of chars interpreted in the
> internal Emacs encoding), right?

Yes.  However, unibyte strings are just streams of bytes; Emacs cannot
interpret them, and they generally appear on display as octal escapes.
They should never be presented to the user, except if the user
explicitly requested that, e.g. by a command such as
find-file-literally.

>     (Btw, I don't think we should worry about changing the internal
>     representation of characters in Emacs, because make_multibyte_string
>     will be updated as needed.)
> 
> This is a crucial point. If the internal encoding never changes, then we can
> declare that those string parameters are expected to be in the internal
> encoding.

No, we cannot, or rather should not.  It is unreasonable to expect
external modules to know the intricacies of the internal
representation.  Most Emacs hackers don't.

> But see the discussion in
> https://github.com/aaptel/emacs-dynamic-module/issues/37: the comment in
> mule-conf.el seems to indicate that the internal encoding is not stable.

That discussion is about zero-copy access to Emacs buffer text and
Emacs strings inside module code.  Such access is indeed impossible
without either knowing _something_ about the internal representation,
or having additional APIs in emacs-module.c that allow modules such
access while hiding the details of the internal representation.  We
could discuss extending the module functionality to include this.

But that is a separate issue from what module_make_function and
module_make_string do.  These two functions are basic, and don't need
to know about the internal representation or use it.  While direct
access to Emacs buffer text will be needed by only some modules,
module_make_function will be used by all of them, and
module_make_string by many.

So I think we shouldn't conflate these two issues; they are separate.

>     This is what my comments were about. I think that you, by contrast,
>     are talking about the encoding of the _input_ strings, in this case
>     the 'documentation' argument to module_make_function and 'str'
>     argument to module_make_string. My assumption was that these
>     arguments will always have to be in UTF-8 encoding; if that assumption
>     is true, then no decoding via code_convert_string_norecord is
>     necessary, since make_multibyte_string will DTRT. We can (and
>     probably should) document the fact that all non-ASCII strings must be
>     UTF-8 encoded as a requirement of the emacs-module interface.
> 
> Or rather, an extension to UTF-8 capable of encoding surrogate code points and
> numbers that are not code points, as described in
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Representations.html.

No, I meant strict UTF-8, not its Emacs extension.

>     If you are thinking about accepting strings encoded in other
>     encodings, I'd consider this an extension, to be added later if
>     needed. After all, a module can easily convert to UTF-8 by itself,
>     using facilities such as iconv.
> 
> Yes, provided the internal Emacs encoding is stable.

That's not what I meant.  (AFAIK, iconv doesn't know about the Emacs
internal representation.)  I meant that a module could convert from
any encoding to UTF-8, and then pass the resulting UTF-8 string to the
emacs-module API.

>     In any case, code_convert_string_norecord cannot be the complete
>     solution, because it accepts Lisp string objects, not C strings. You
>     still need to create a Lisp string (but this time using
>     make_unibyte_string). The point is to always use either
>     make_unibyte_string or make_multibyte_string, and never build_string
>     or make_string; the latter 2 should only be used for fixed ASCII-only
>     strings.
> 
> Yes, that's fine, the question is about whether the internal encoding is
> stable.

With my suggestion, the stability of the internal representation is
not an issue.

> If it's stable, we can use make_multibyte_string; if not, we can
> only use make_unibyte_string.

If the arguments strings are in strict UTF-8, then
make_multibyte_string will DTRT automagically, no matter what the
internal representation is.  That is their contract.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 4102 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 21. Nov. 2015 um 12:10 Uhr:

> >     (Btw, I don't think we should worry about changing the internal
> >     representation of characters in Emacs, because make_multibyte_string
> >     will be updated as needed.)
> >
> > This is a crucial point. If the internal encoding never changes, then we
> can
> > declare that those string parameters are expected to be in the internal
> > encoding.
>
> No, we cannot, or rather should not.  It is unreasonable to expect
> external modules to know the intricacies of the internal
> representation.  Most Emacs hackers don't.
>

Fine with me, but how would we then represent Emacs strings that are not
valid Unicode strings? Just raise an error?


>
> > But see the discussion in
> > https://github.com/aaptel/emacs-dynamic-module/issues/37: the comment in
> > mule-conf.el seems to indicate that the internal encoding is not stable.
>
> That discussion is about zero-copy access to Emacs buffer text and
> Emacs strings inside module code.


Partially, the encoding discussion is also part of that because it's
required to specify the encoding before zero-copy access is even possible.


>   Such access is indeed impossible
> without either knowing _something_ about the internal representation,
> or having additional APIs in emacs-module.c that allow modules such
> access while hiding the details of the internal representation.  We
> could discuss extending the module functionality to include this.
>
>
Yes, there's no need for that in this subthread though.


> But that is a separate issue from what module_make_function and
> module_make_string do.  These two functions are basic, and don't need
> to know about the internal representation or use it.  While direct
> access to Emacs buffer text will be needed by only some modules,
> module_make_function will be used by all of them, and
> module_make_string by many.
>
> So I think we shouldn't conflate these two issues; they are separate.
>

OK.


>
> >     This is what my comments were about. I think that you, by contrast,
> >     are talking about the encoding of the _input_ strings, in this case
> >     the 'documentation' argument to module_make_function and 'str'
> >     argument to module_make_string. My assumption was that these
> >     arguments will always have to be in UTF-8 encoding; if that
> assumption
> >     is true, then no decoding via code_convert_string_norecord is
> >     necessary, since make_multibyte_string will DTRT. We can (and
> >     probably should) document the fact that all non-ASCII strings must be
> >     UTF-8 encoded as a requirement of the emacs-module interface.
> >
> > Or rather, an extension to UTF-8 capable of encoding surrogate code
> points and
> > numbers that are not code points, as described in
> >
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Representations.html
> .
>
> No, I meant strict UTF-8, not its Emacs extension.
>

That would be possible and provide a clean interface. However, Emacs
strings are extended, so we'd need to specify how they interact with UTF-8
strings.

   - If a module passes a char sequence that's not a valid UTF-8 string,
   but a valid Emacs multibyte string, what should happen? Error, undefined
   behavior, silently accepted?
   - If copy_string_contents is passed an Emacs string that is not a valid
   Unicode string, what should happen? Error, or should the internal
   representation be silently leaked?



> > If it's stable, we can use make_multibyte_string; if not, we can
> > only use make_unibyte_string.
>
> If the arguments strings are in strict UTF-8, then
> make_multibyte_string will DTRT automagically, no matter what the
> internal representation is.  That is their contract.
>

OK, then we can use that, of course. The question of handling invalid UTF-8
strings is still open, though, as make_multibyte_string doesn't enforce
valid UTF-8.
If it's the contract of make_multibyte_string that it will always accept
UTF-8, then that should be added as a comment to that function. Currently I
don't see it documented anywhere.

[-- Attachment #2: Type: text/html, Size: 5697 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 21 Nov 2015 12:11:45 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     No, we cannot, or rather should not. It is unreasonable to expect
>     external modules to know the intricacies of the internal
>     representation. Most Emacs hackers don't.
>     
> Fine with me, but how would we then represent Emacs strings that are not valid
> Unicode strings? Just raise an error?

No need to raise an error.  Strings that are returned to modules
should be encoded into UTF-8.  That encoding already takes care of
these situations: it either produces the UTF-8 encoding of the
equivalent Unicode characters, or outputs raw bytes.

We are using this all the time when we save files or send stuff over
the network.

>     No, I meant strict UTF-8, not its Emacs extension.
> 
> That would be possible and provide a clean interface. However, Emacs strings
> are extended, so we'd need to specify how they interact with UTF-8 strings.
> 
> * If a module passes a char sequence that's not a valid UTF-8 string, but a
>   valid Emacs multibyte string, what should happen? Error, undefined behavior,
>   silently accepted?

We are quite capable of quietly accepting such strings, so that is
what I would suggest.  Doing so would be in line with what Emacs does
when such invalid sequences come from other sources, like files.

> * If copy_string_contents is passed an Emacs string that is not a valid Unicode
>   string, what should happen?

How can that happen?  The Emacs string comes from the Emacs bowels, so
it must be "valid" string by Emacs standards.  Or maybe I don't
understand what you mean by "invalid Unicode string".

In any case, we already deal with any such problems when we save a
buffer to a file, or send it over the network.  This isn't some new
problem we need to cope with.

> OK, then we can use that, of course. The question of handling invalid UTF-8
> strings is still open, though, as make_multibyte_string doesn't enforce valid
> UTF-8.

It doesn't enforce valid UTF-8 because it can handle invalid UTF-8 as
well.  That's by design.

> If it's the contract of make_multibyte_string that it will always accept UTF-8,
> then that should be added as a comment to that function. Currently I don't see
> it documented anywhere. 

That part of the documentation is only revealed to veteran Emacs
hackers, subject to swearing not to reveal that to the uninitiated and
to some blood-letting that seals the oath ;-)

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 21. Nov. 2015 um 14:23 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sat, 21 Nov 2015 12:11:45 +0000
> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
> emacs-devel@gnu.org
> >
> >     No, we cannot, or rather should not. It is unreasonable to expect
> >     external modules to know the intricacies of the internal
> >     representation. Most Emacs hackers don't.
> >
> > Fine with me, but how would we then represent Emacs strings that are not
> valid
> > Unicode strings? Just raise an error?
>
> No need to raise an error.  Strings that are returned to modules
> should be encoded into UTF-8.  That encoding already takes care of
> these situations: it either produces the UTF-8 encoding of the
> equivalent Unicode characters, or outputs raw bytes.
>

Then we should document such a situation and give module authors a way to
detect them. For example, what happens if a sequence of such raw bytes
happens to be a valid UTF-8 sequence? Is there a way for module code to
detect this situation?


>
> We are using this all the time when we save files or send stuff over
> the network.
>
> >     No, I meant strict UTF-8, not its Emacs extension.
> >
> > That would be possible and provide a clean interface. However, Emacs
> strings
> > are extended, so we'd need to specify how they interact with UTF-8
> strings.
> >
> > * If a module passes a char sequence that's not a valid UTF-8 string,
> but a
> >   valid Emacs multibyte string, what should happen? Error, undefined
> behavior,
> >   silently accepted?
>
> We are quite capable of quietly accepting such strings, so that is
> what I would suggest.  Doing so would be in line with what Emacs does
> when such invalid sequences come from other sources, like files.
>

If we accept such strings, then we should document what the extensions are.
- Are UTF-8-like sequences encoding surrogate code points accepted?
- Are UTF-8-like sequences encoding integers outside the Unicode codespace
accepted?
- Are non-shortest forms accepted?
- Are other invalid code unit sequences accepted?
If the answer to any of these is "yes", we can't say we accept UTF-8,
because we don't. Rather we should say what is actually accepted.


>
> > * If copy_string_contents is passed an Emacs string that is not a valid
> Unicode
> >   string, what should happen?
>
> How can that happen?  The Emacs string comes from the Emacs bowels, so
> it must be "valid" string by Emacs standards.  Or maybe I don't
> understand what you mean by "invalid Unicode string".
>

A sequence of integers where at least one element is not a Unicode scalar
value.


>
> In any case, we already deal with any such problems when we save a
> buffer to a file, or send it over the network.  This isn't some new
> problem we need to cope with.
>

Yes, but the module interface is new, it doesn't necessarily have to have
the same behavior. If we say we emit only UTF-8, then we should do so.


>
> > OK, then we can use that, of course. The question of handling invalid
> UTF-8
> > strings is still open, though, as make_multibyte_string doesn't enforce
> valid
> > UTF-8.
>
> It doesn't enforce valid UTF-8 because it can handle invalid UTF-8 as
> well.  That's by design.
>

Then whatever it handles needs to be specified.


>
> > If it's the contract of make_multibyte_string that it will always accept
> UTF-8,
> > then that should be added as a comment to that function. Currently I
> don't see
> > it documented anywhere.
>
> That part of the documentation is only revealed to veteran Emacs
> hackers, subject to swearing not to reveal that to the uninitiated and
> to some blood-letting that seals the oath ;-)
>

I see ;-)

[-- Attachment #2: Type: text/html, Size: 5362 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3031 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am So., 22. Nov. 2015 um
10:25 Uhr:

> Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 21. Nov. 2015 um 14:23 Uhr:
>
>> > From: Philipp Stephani <p.stephani2@gmail.com>
>> > Date: Sat, 21 Nov 2015 12:11:45 +0000
>> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
>> emacs-devel@gnu.org
>> >
>> >     No, we cannot, or rather should not. It is unreasonable to expect
>> >     external modules to know the intricacies of the internal
>> >     representation. Most Emacs hackers don't.
>> >
>> > Fine with me, but how would we then represent Emacs strings that are
>> not valid
>> > Unicode strings? Just raise an error?
>>
>> No need to raise an error.  Strings that are returned to modules
>> should be encoded into UTF-8.  That encoding already takes care of
>> these situations: it either produces the UTF-8 encoding of the
>> equivalent Unicode characters, or outputs raw bytes.
>>
>
> Then we should document such a situation and give module authors a way to
> detect them. For example, what happens if a sequence of such raw bytes
> happens to be a valid UTF-8 sequence? Is there a way for module code to
> detect this situation?
>
>

I've thought a bit more about this issue an in the following I'll attempt
to derive the desired behavior from first principles without referring to
internal Emacs functions.
There are two kinds of Emacs strings, unibyte and multibyte.
https://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Representations.html
and
https://www.gnu.org/software/emacs/manual/html_node/elisp/Character-Codes.html
agree
that multibyte strings are sequences of integers (let's avoid the
overloaded and vague term "characters") in the range 0 to #x3FFFFF
(inclusive). It is also clear that within the subset of that range
corresponding the the Unicode codespace the intergers are interpreted as
Unicode code points. Given that new APIs should use UTF-8, the following
approach looks reasonable to me:
- There are two sets of functions for creating and reading strings, unibyte
and multibyte. If a string of the wrong type is passed, a signal is raised.
This way the two types are clearly separated.
- The behavior of the unibyte API is uncontroversial and has no failure
modes apart from generic ones such as wrong type, argument out of range,
OOM.
- The multibyte API should use an extension of UTF-8 to encode Emacs
strings. The extension is the obvious one already in use in multiple places.
- There should be a one-to-one mapping between Emacs multibyte strings and
encoded module API strings. Therefore non-shortest forms, illegal code unit
sequences, and code unit sequences that would encode values outside the
range of Emacs characters are illegal and raise a signal. Likewise, such
sequences will never be returned from Emacs.
I think this is a relatively simple and unsurprising approach. It allows
encoding the documented Emacs character space while still being fully
compatible with UTF-8 and not resorting to undocumented Emacs internals.

[-- Attachment #2: Type: text/html, Size: 4258 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 14:56:12 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
> - The multibyte API should use an extension of UTF-8 to encode Emacs strings.
> The extension is the obvious one already in use in multiple places.

It is only used in one place: the internal representation of
characters in buffers and strings.  Emacs _never_ lets this internal
representation leak outside.  In practice the last sentence means that
text that Emacs encoded in UTF-8 will only include either valid UTF-8
sequences of characters whose codepoints are below #x200000 or single
bytes that don't belong to any UTF-8 sequence.

You are suggesting to expose the internal representation to outside
application code, which predictably will cause that representation to
leak into Lisp.  That'd be a disaster.  We had something like that
back in the Emacs 20 era, and it took many years to plug those leaks.
We would be making a grave mistake to go back there.

What you suggest is also impossible without deep changes in how we
decode and encode text: that process maps codepoints above #1FFFFF to
either codepoints below that mark or to raw bytes.  So it's impossible
to produce these high codes in UTF-8 compatible form while handling
UTF-8 text.  To say nothing about the simple fact that no library
function in any C library will ever be able to do anything useful with
such codepoints, because they are our own invention.

> - There should be a one-to-one mapping between Emacs multibyte strings and
> encoded module API strings.

UTF-8 encoded strings satisfy that requirement.

> Therefore non-shortest forms, illegal code unit sequences, and code
> unit sequences that would encode values outside the range of Emacs
> characters are illegal and raise a signal.

Once again, this was tried in the past and was found to be a bad idea.
Emacs provides features to test the result of converting invalid
sequences, for the purposes of detecting such problems, but it leaves
that to the application.

> Likewise, such sequences will never be returned from Emacs.

Emacs doesn't return invalid sequences, if the original text didn't
include raw bytes.  If there were raw bytes in the original text,
Emacs has no choice but return them back, or else it will violate a
basic expectation from a text-processing program: that it shall never
change the portions of text that were not affected by the processing.

> I think this is a relatively simple and unsurprising approach. It allows
> encoding the documented Emacs character space while still being fully
> compatible with UTF-8 and not resorting to undocumented Emacs internals.

So does the approach I suggested.  The advantage of my suggestion is
that it follows a long Emacs tradition about every aspect of encoding
and decoding text, and doesn't require any changes in the existing
infrastructure.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 5637 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 19:04 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 22 Nov 2015 14:56:12 +0000
> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
> emacs-devel@gnu.org
> >
> > - The multibyte API should use an extension of UTF-8 to encode Emacs
> strings.
> > The extension is the obvious one already in use in multiple places.
>
> It is only used in one place: the internal representation of
> characters in buffers and strings.  Emacs _never_ lets this internal
> representation leak outside.


If I run in scratch:

(with-temp-buffer
  (insert #x3fff40)
  (describe-char (point-min)))

Then the resulting help buffer says "buffer code: #xF8 #x8F #xBF #xBD
#x80", is that not considered a leak?


>   In practice the last sentence means that
> text that Emacs encoded in UTF-8 will only include either valid UTF-8
> sequences of characters whose codepoints are below #x200000 or single
> bytes that don't belong to any UTF-8 sequence.
>

I get the same result as above when running

(with-temp-buffer
  (call-process "echo" nil t nil (string #x3fff40))
  (describe-char (point-min)))

that means the non-UTF sequence is even "leaked" to the external process!


>
> You are suggesting to expose the internal representation to outside
> application code, which predictably will cause that representation to
> leak into Lisp.  That'd be a disaster.  We had something like that
> back in the Emacs 20 era, and it took many years to plug those leaks.
> We would be making a grave mistake to go back there.
>

I don't suggest leaking anything what isn't already leaked. The extension
of the codespace to 22 bits is well documented.


>
> What you suggest is also impossible without deep changes in how we
> decode and encode text: that process maps codepoints above #1FFFFF to
> either codepoints below that mark or to raw bytes.  So it's impossible
> to produce these high codes in UTF-8 compatible form while handling
> UTF-8 text.  To say nothing about the simple fact that no library
> function in any C library will ever be able to do anything useful with
> such codepoints, because they are our own invention.
>

Unless the behavior changed recently, that doesn't seem the case:

(encode-coding-string (string #x3fff40) 'utf-8-unix)
"\370\217\277\275\200"

Or are you talking about something different?


>
> > - There should be a one-to-one mapping between Emacs multibyte strings
> and
> > encoded module API strings.
>
> UTF-8 encoded strings satisfy that requirement.
>

No! UTF-8 can only encode Unicode scalar values. Only the Emacs extension
to UTF-8 (which I think Emacs calls "UTF-8" unfortunately) satisfies this.
If you are talking about this extension, then we talk about the same thing
anyway.


>
> > Therefore non-shortest forms, illegal code unit sequences, and code
> > unit sequences that would encode values outside the range of Emacs
> > characters are illegal and raise a signal.
>
> Once again, this was tried in the past and was found to be a bad idea.
> Emacs provides features to test the result of converting invalid
> sequences, for the purposes of detecting such problems, but it leaves
> that to the application.
>

It's probably OK to accept invalid sequences for consistency with
decode-coding-string and friends. I don't really like it though: the module
API, like decode-coding-string, is not a general-purpose UI for end users,
and accepting invalid sequences is error-prone and can even introduce
security issues (see e.g.
https://blogs.oracle.com/CoreJavaTechTips/entry/the_overhaul_of_java_utf).


>
> > Likewise, such sequences will never be returned from Emacs.
>
> Emacs doesn't return invalid sequences, if the original text didn't
> include raw bytes.  If there were raw bytes in the original text,
> Emacs has no choice but return them back, or else it will violate a
> basic expectation from a text-processing program: that it shall never
> change the portions of text that were not affected by the processing.
>

It seems that Emacs does return invalid sequences for characters such as
#x3ffff40 (or anything else outside of Unicode except the 127 values for
encoding raw bytes).
Returning raw bytes means that encoding and decoding isn't a perfect
roundtrip:

(decode-coding-string (encode-coding-string (string #x3fffc2 #x3fffbb)
'utf-8-unix) 'utf-8-unix)
"»"

We might be able to live with that as it's an extreme edge case.


>
> > I think this is a relatively simple and unsurprising approach. It allows
> > encoding the documented Emacs character space while still being fully
> > compatible with UTF-8 and not resorting to undocumented Emacs internals.
>
> So does the approach I suggested.  The advantage of my suggestion is
> that it follows a long Emacs tradition about every aspect of encoding
> and decoding text, and doesn't require any changes in the existing
> infrastructure.
>

What are the exact difference between the approaches? As far as I can see
differences exist only for the following points:
- Accepting invalid sequences. I consider that a bug in general-purpose
APIs, including decode-coding-string. However, given that Emacs already
extends the Unicode codespace and therefore has to accept some invalid
sequences anyway, it might be OK if it's clearly documented.
- Emitting raw bytes instead of extended sequences. Though I'm not a fan of
this it might be unavoidable to be able to treat strings transparently
(which is desirable).

[-- Attachment #2: Type: text/html, Size: 7815 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 19:10:44 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     It is only used in one place: the internal representation of
>     characters in buffers and strings. Emacs _never_ lets this internal
>     representation leak outside.
> 
> If I run in scratch:
> 
> (with-temp-buffer
> (insert #x3fff40)
> (describe-char (point-min)))

Emacs will never find such "byte" in any text.  So this feature is not
really relevant to the issue at hand.

> Then the resulting help buffer says "buffer code: #xF8 #x8F #xBF #xBD #x80", is
> that not considered a leak?

No.  You created this yourself, and got what you asked for.

More generally, can you imagine a real-life situation where a string
with such "bytes" could be received from a module, as part of a C
'char *' string?

>     You are suggesting to expose the internal representation to outside
>     application code, which predictably will cause that representation to
>     leak into Lisp. That'd be a disaster. We had something like that
>     back in the Emacs 20 era, and it took many years to plug those leaks.
>     We would be making a grave mistake to go back there.
> 
> I don't suggest leaking anything what isn't already leaked. The extension of
> the codespace to 22 bits is well documented.

I don't think it's reasonable to request that module authors read all
that stuff and understand it, before they can write a simple module
that manipulates non-ASCII text.  Writing such modules should be that
hard.

> Returning raw bytes means that encoding and decoding isn't a perfect roundtrip:
> 
> (decode-coding-string (encode-coding-string (string #x3fffc2 #x3fffbb)
> 'utf-8-unix) 'utf-8-unix)
> "»"

If you start with raw bytes, not large integers, then the roundtrip
will be perfect.

> What are the exact difference between the approaches? As far as I can see
> differences exist only for the following points:
> - Accepting invalid sequences. I consider that a bug in general-purpose APIs,
> including decode-coding-string. However, given that Emacs already extends the
> Unicode codespace and therefore has to accept some invalid sequences anyway, it
> might be OK if it's clearly documented.
> - Emitting raw bytes instead of extended sequences. Though I'm not a fan of
> this it might be unavoidable to be able to treat strings transparently (which
> is desirable). 

Then I think we agree after all.

Thanks.

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 09:25:08 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     > Fine with me, but how would we then represent Emacs strings that are not
>     valid
>     > Unicode strings? Just raise an error?
>     
>     No need to raise an error. Strings that are returned to modules
>     should be encoded into UTF-8. That encoding already takes care of
>     these situations: it either produces the UTF-8 encoding of the
>     equivalent Unicode characters, or outputs raw bytes.
> 
> Then we should document such a situation and give module authors a way to
> detect them.

I already suggested what we should say in the documentation: that
these interfaces accept and produce UTF-8 encoded non-ASCII text.

> For example, what happens if a sequence of such raw bytes happens
> to be a valid UTF-8 sequence? Is there a way for module code to detect this
> situation?

How can you detect that if you are only given the byte stream?  You
can't.  You need some additional information to be able to distinguish
between these two alternatives.

Look, an Emacs module _must_ support non-ASCII text, otherwise it
would be severely limited, to say the least.  Having interfaces that
accept and produce UTF-8 encoded strings is the simplest complete
solution to this problem.  So we must at least support that much.

Supporting strings of raw bytes is also possible, probably even
desirable, but it's an extension, something that would be required
much more rarely.  Such strings cannot be meaningfully treated as
text: you cannot ask if some byte is upper-case or lower-case letter,
you cannot display such strings as readable text, you cannot count
characters in it, etc.  Such strings are useful for a limited number
of specialized jobs, and handling them in Lisp requires some caution,
because if you treat them as normal text strings, you get surprises.

So let's solve the more important issues first, and talk about
extensions later.  The more important issue is how can a module pass
to Emacs non-ASCII text and get back non-ASCII text.  And the answer
to that is to use UTF-8 encoded strings.

>     We are quite capable of quietly accepting such strings, so that is
>     what I would suggest. Doing so would be in line with what Emacs does
>     when such invalid sequences come from other sources, like files.
> 
> If we accept such strings, then we should document what the extensions are.
> - Are UTF-8-like sequences encoding surrogate code points accepted?
> - Are UTF-8-like sequences encoding integers outside the Unicode codespace
> accepted?
> - Are non-shortest forms accepted?
> - Are other invalid code unit sequences accepted?

_Anything_ can be accepted.  _Any_ byte sequence.  Emacs will cope.
The perpetrator will probably get back after processing a string that
is not entirely human-readable, or its processing will sometimes
produce surprises, like if the string is lower-cased.  But nothing bad
will happen to Emacs, it won't crash and won't garble its display.
Moreover, just passing such a string to Emacs, then outputting it back
without any changes will produce an exact copy of the input, which is
quite a feat, considering that the input was "invalid".

If you want to see what "bad" things can happen, take a Latin-1
encoded FILE and visit it with "C-x RET c utf-8 RET C-x C-f FILE RET".
Then play with the buffer a while.  This is what happens when Emacs is
told the text is in UTF-8, when it really isn't.  There's no
catastrophe, but the luser who does that might be amply punished, at
the very least she will not see the letters she expects.  However, if
you save such a buffer to a file, using UTF-8, you will get the same
Latin-1 encoded text as was there originally.

Now, given such resilience, why do we need to raise an error?

> If the answer to any of these is "yes", we can't say we accept UTF-8, because
> we don't.

We _expect_ UTF-8, and if given that, will produce known, predictable
results when the string is processed as text.  We can _tolerate_
violations, resulting in somewhat surprising behavior, if such a text
is treated as "normal" human-readable text.  (If the module knows what
it does, and really means to work with raw bytes, then Emacs will do
what the module expects, and produce raw bytes on output, as
expected.)

> Rather we should say what is actually accepted.

Saying that is meaningless in this case, because we can accept
anything.  _If_ the module wants the string it passes to be processed
as human-readable text that consists of recognizable characters, then
the module should _only_ pass valid UTF-8 sequences.  But raising
errors upon detecting violations was discovered long ago a bad idea
that users resented.  So we don't, and neither should the module API.

>     > * If copy_string_contents is passed an Emacs string that is not a valid
>     Unicode
>     > string, what should happen?
>     
>     How can that happen? The Emacs string comes from the Emacs bowels, so
>     it must be "valid" string by Emacs standards. Or maybe I don't
>     understand what you mean by "invalid Unicode string".
> 
> A sequence of integers where at least one element is not a Unicode scalar
> value.

Emacs doesn't store characters as scalar Unicode values, so this
doesn't really explain to me your concept of a "valid Unicode string".

>     In any case, we already deal with any such problems when we save a
>     buffer to a file, or send it over the network. This isn't some new
>     problem we need to cope with.
> 
> Yes, but the module interface is new, it doesn't necessarily have to have the
> same behavior.

Of course, it does!  Modules are Emacs extensions, so the interface
should support the same features that core Emacs does.  Why? because
there's no limits to what creative minds can do with this feature, so
we should not artificially impose such limitations where we have
sound, time-proven infrastructure that doesn't need them.

> If we say we emit only UTF-8, then we should do so.

We emit only valid UTF-8, provided that its source (if it came from a
module) was valid UTF-8.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 8151 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 18:35 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 22 Nov 2015 09:25:08 +0000
> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
> emacs-devel@gnu.org
> >
> >     > Fine with me, but how would we then represent Emacs strings that
> are not
> >     valid
> >     > Unicode strings? Just raise an error?
> >
> >     No need to raise an error. Strings that are returned to modules
> >     should be encoded into UTF-8. That encoding already takes care of
> >     these situations: it either produces the UTF-8 encoding of the
> >     equivalent Unicode characters, or outputs raw bytes.
> >
> > Then we should document such a situation and give module authors a way to
> > detect them.
>
> I already suggested what we should say in the documentation: that
> these interfaces accept and produce UTF-8 encoded non-ASCII text.
>

If the interface accepts UTF-8, then it must signal an error for invalid
sequences; the Unicode standard mandates this.
If the interface produces UTF-8, then it must only ever produce valid
sequences, this is again required by the Unicode standard.


>
> > For example, what happens if a sequence of such raw bytes happens
> > to be a valid UTF-8 sequence? Is there a way for module code to detect
> this
> > situation?
>
> How can you detect that if you are only given the byte stream?  You
> can't.  You need some additional information to be able to distinguish
> between these two alternatives.
>

That's why I propose to not encode raw bytes as bytes, but as the Emacs
integer codes used to represent them.


>
> Look, an Emacs module _must_ support non-ASCII text, otherwise it
> would be severely limited, to say the least.


Absolutely!


> Having interfaces that
> accept and produce UTF-8 encoded strings is the simplest complete
> solution to this problem.  So we must at least support that much.
>

Agreed.


>
> Supporting strings of raw bytes is also possible, probably even
> desirable, but it's an extension, something that would be required
> much more rarely.  Such strings cannot be meaningfully treated as
> text: you cannot ask if some byte is upper-case or lower-case letter,
> you cannot display such strings as readable text, you cannot count
> characters in it, etc.  Such strings are useful for a limited number
> of specialized jobs, and handling them in Lisp requires some caution,
> because if you treat them as normal text strings, you get surprises.
>

Yes. However, without an interface they are awkward to produce.


>
> So let's solve the more important issues first, and talk about
> extensions later.  The more important issue is how can a module pass
> to Emacs non-ASCII text and get back non-ASCII text.  And the answer
> to that is to use UTF-8 encoded strings.
>

Full agreement.


>
> >     We are quite capable of quietly accepting such strings, so that is
> >     what I would suggest. Doing so would be in line with what Emacs does
> >     when such invalid sequences come from other sources, like files.
> >
> > If we accept such strings, then we should document what the extensions
> are.
> > - Are UTF-8-like sequences encoding surrogate code points accepted?
> > - Are UTF-8-like sequences encoding integers outside the Unicode
> codespace
> > accepted?
> > - Are non-shortest forms accepted?
> > - Are other invalid code unit sequences accepted?
>
> _Anything_ can be accepted.  _Any_ byte sequence.  Emacs will cope.
>

Not if they accept UTF-8. The Unicode standard rules out accepting invalid
byte sequences.
If any byte sequence is accepted, then the behavior becomes more complex.
We need to exhaustively describe the behavior for any possible byte
sequence, otherwise module authors cannot make any assumption.


> The perpetrator will probably get back after processing a string that
> is not entirely human-readable, or its processing will sometimes
> produce surprises, like if the string is lower-cased.  But nothing bad
> will happen to Emacs, it won't crash and won't garble its display.
> Moreover, just passing such a string to Emacs, then outputting it back
> without any changes will produce an exact copy of the input, which is
> quite a feat, considering that the input was "invalid".
>
> If you want to see what "bad" things can happen, take a Latin-1
> encoded FILE and visit it with "C-x RET c utf-8 RET C-x C-f FILE RET".
> Then play with the buffer a while.  This is what happens when Emacs is
> told the text is in UTF-8, when it really isn't.  There's no
> catastrophe, but the luser who does that might be amply punished, at
> the very least she will not see the letters she expects.  However, if
> you save such a buffer to a file, using UTF-8, you will get the same
> Latin-1 encoded text as was there originally.
>
> Now, given such resilience, why do we need to raise an error?
>

The Unicode standard says so. If we document that *a superset of UTF-8* is
accepted, then we don't need to raise an error. So I'd suggest we do
exactly that, but describe what that superset is.


>
> > If the answer to any of these is "yes", we can't say we accept UTF-8,
> because
> > we don't.
>
> We _expect_ UTF-8, and if given that, will produce known, predictable
> results when the string is processed as text.  We can _tolerate_
> violations, resulting in somewhat surprising behavior, if such a text
> is treated as "normal" human-readable text.  (If the module knows what
> it does, and really means to work with raw bytes, then Emacs will do
> what the module expects, and produce raw bytes on output, as
> expected.)
>

No matter what we expect or tolerate, we need to state that. If all byte
sequences are accepted, then we also need to state that, but describe what
the behavior is if there are invalid UTF-8 sequences in the input.


>
> > Rather we should say what is actually accepted.
>
> Saying that is meaningless in this case, because we can accept
> anything.  _If_ the module wants the string it passes to be processed
> as human-readable text that consists of recognizable characters, then
> the module should _only_ pass valid UTF-8 sequences.  But raising
> errors upon detecting violations was discovered long ago a bad idea
> that users resented.  So we don't, and neither should the module API.
>

Module authors are not end users. I agree that end users should not see
errors on decoding failure, but modules use only programmatic access, where
we can be more strict.


>
> >     > * If copy_string_contents is passed an Emacs string that is not a
> valid
> >     Unicode
> >     > string, what should happen?
> >
> >     How can that happen? The Emacs string comes from the Emacs bowels, so
> >     it must be "valid" string by Emacs standards. Or maybe I don't
> >     understand what you mean by "invalid Unicode string".
> >
> > A sequence of integers where at least one element is not a Unicode scalar
> > value.
>
> Emacs doesn't store characters as scalar Unicode values, so this
> doesn't really explain to me your concept of a "valid Unicode string".
>

An Emacs string is a sequence of integers. It doesn't have to be a sequence
of scalar values.


>
> >     In any case, we already deal with any such problems when we save a
> >     buffer to a file, or send it over the network. This isn't some new
> >     problem we need to cope with.
> >
> > Yes, but the module interface is new, it doesn't necessarily have to
> have the
> > same behavior.
>
> Of course, it does!  Modules are Emacs extensions, so the interface
> should support the same features that core Emacs does.  Why? because
> there's no limits to what creative minds can do with this feature, so
> we should not artificially impose such limitations where we have
> sound, time-proven infrastructure that doesn't need them.
>

I agree that we shouldn't add such limitations. But I disagree that we
should leave the behavior undocumented in such cases.


>
> > If we say we emit only UTF-8, then we should do so.
>
> We emit only valid UTF-8, provided that its source (if it came from a
> module) was valid UTF-8.
>

Then in turn we shouldn't say we emit only UTF-8.

[-- Attachment #2: Type: text/html, Size: 11250 bytes --]

Re: Dynamic loading progress

[David Kastrup]

Philipp Stephani <p.stephani2@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 18:35 Uhr:
>
>> > From: Philipp Stephani <p.stephani2@gmail.com>
>> > Date: Sun, 22 Nov 2015 09:25:08 +0000
>> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
>> emacs-devel@gnu.org
>> >
>> >     > Fine with me, but how would we then represent Emacs strings that
>> are not
>> >     valid
>> >     > Unicode strings? Just raise an error?
>> >
>> >     No need to raise an error. Strings that are returned to modules
>> >     should be encoded into UTF-8. That encoding already takes care of
>> >     these situations: it either produces the UTF-8 encoding of the
>> >     equivalent Unicode characters, or outputs raw bytes.
>> >
>> > Then we should document such a situation and give module authors a way to
>> > detect them.
>>
>> I already suggested what we should say in the documentation: that
>> these interfaces accept and produce UTF-8 encoded non-ASCII text.
>>
>
> If the interface accepts UTF-8, then it must signal an error for invalid
> sequences; the Unicode standard mandates this.
> If the interface produces UTF-8, then it must only ever produce valid
> sequences, this is again required by the Unicode standard.

The Unicode standard does not mandate Emacs' internal string encodings.
Emacs 22 had an entirely different internal string encoding that was
less convenient for a module interface.

Emacs' internal encoding has the property that valid UTF-8 sequences are
represented by themselves.

Which is a convenience to the programmer.  Not more, not less.

> That's why I propose to not encode raw bytes as bytes, but as the
> Emacs integer codes used to represent them.

So UCS-16 instead of UTF-8?  With conversions for every call?  That
sounds like shuffling the problem around, and shuffling does not come
for free.

It's ok to provide checking sequences that verify that something is a
valid internal Emacs string (which includes more than Unicode) and flag
an error if it isn't.  Also that something is a valid UTF-8 string if
that is desired (and which will in case of error optionally either
convert to the Emacs-internal representation or flag an error).

But making each call gate automatically verify every string for UTF-8
compliance is both wasteful as well as making it impossible to process
generic Emacs strings in an external module.

-- 
David Kastrup

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 18:19:29 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     I already suggested what we should say in the documentation: that
>     these interfaces accept and produce UTF-8 encoded non-ASCII text.
>     
> 
> If the interface accepts UTF-8, then it must signal an error for invalid
> sequences; the Unicode standard mandates this.

The Unicode standard cannot mandate anything for Emacs, because Emacs
is not subject to Unicode standardization.

> If the interface produces UTF-8, then it must only ever produce valid
> sequences

As I explained, this would violate the basic expectation from a text
editing program.

> That's why I propose to not encode raw bytes as bytes, but as the Emacs integer
> codes used to represent them.

If we do that, no external code will be able to do anything useful
with such "bytes".  Module authors will have to write their own
replacements for library functions.  This will never be accepted by
our users.

> If any byte sequence is accepted, then the behavior becomes more complex. We
> need to exhaustively describe the behavior for any possible byte sequence,
> otherwise module authors cannot make any assumption.

We say that we accept valid UTF-8 encoded strings; anything else
might produce invalid UTF-8 on output.

> No matter what we expect or tolerate, we need to state that.

No, we don't.  When the callers violate the contract, they cannot
expect to know in detail what will happen.  If they want to know, they
will have to read the source.

> Module authors are not end users.

They are users like anyone who writes Lisp.  They came to expect that
Emacs behaves in certain ways, and modules should follow suit.

> I agree that end users should not see errors on decoding failure,
> but modules use only programmatic access, where we can be more
> strict.

You cannot be more strict, unless you rewrite the whole
encoding/decoding machinery, or write specialized code to detect and
reject invalid UTF-8 before it is passed to a decoder.  There are no
good reasons to do either, so let's not.

> An Emacs string is a sequence of integers.

No, it's a sequence of bytes.

> I agree that we shouldn't add such limitations. But I disagree that we should
> leave the behavior undocumented in such cases.

OK, so let's agree to disagree.  If that disagreement gets in your way
of fixing the issues related to this discussion, please say so, and I
will fix them myself

Thanks.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3763 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 20:20 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 22 Nov 2015 18:19:29 +0000
> > Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com,
> emacs-devel@gnu.org
> >
> >     I already suggested what we should say in the documentation: that
> >     these interfaces accept and produce UTF-8 encoded non-ASCII text.
> >
> >
> > If the interface accepts UTF-8, then it must signal an error for invalid
> > sequences; the Unicode standard mandates this.
>
> The Unicode standard cannot mandate anything for Emacs, because Emacs
> is not subject to Unicode standardization.
>

True, but I think we shouldn't make the terminology more confusing. If we
say "UTF-8", we should mean "UTF-8 as defined in the Unicode standard", not
the Emacs extension of UTF-8. That's all.


>
> > If the interface produces UTF-8, then it must only ever produce valid
> > sequences
>
> As I explained, this would violate the basic expectation from a text
> editing program.
>
> > That's why I propose to not encode raw bytes as bytes, but as the Emacs
> integer
> > codes used to represent them.
>
> If we do that, no external code will be able to do anything useful
> with such "bytes".  Module authors will have to write their own
> replacements for library functions.  This will never be accepted by
> our users.
>

I wouldn't be so pessimistic, but I was convinced by consistency with
encode-coding-string. So yes, let's use the raw bytes (and document that).


>
> > If any byte sequence is accepted, then the behavior becomes more
> complex. We
> > need to exhaustively describe the behavior for any possible byte
> sequence,
> > otherwise module authors cannot make any assumption.
>
> We say that we accept valid UTF-8 encoded strings; anything else
> might produce invalid UTF-8 on output.
>

Couldn't we just say "it behaves as if encoding and decoding were done
using the utf-8-unix coding system"? Because I think that's what this boils
down to.


>
> > No matter what we expect or tolerate, we need to state that.
>
> No, we don't.  When the callers violate the contract, they cannot
> expect to know in detail what will happen.  If they want to know, they
> will have to read the source.
>

So you want this to be unspecified or undefined behavior? That might be OK
(we already have that in several places), but we still need to state what
the contract is.


>
> > Module authors are not end users.
>
> They are users like anyone who writes Lisp.  They came to expect that
> Emacs behaves in certain ways, and modules should follow suit.
>
> > I agree that end users should not see errors on decoding failure,
> > but modules use only programmatic access, where we can be more
> > strict.
>
> You cannot be more strict, unless you rewrite the whole
> encoding/decoding machinery, or write specialized code to detect and
> reject invalid UTF-8 before it is passed to a decoder.  There are no
> good reasons to do either, so let's not.
>
> > An Emacs string is a sequence of integers.
>
> No, it's a sequence of bytes.
>

From
https://www.gnu.org/software/emacs/manual/html_node/elisp/String-Basics.html
:
"In Emacs Lisp, characters are simply integers ... A string is a fixed
sequence of characters"
How a string is represented internally shouldn't be the concern of module
authors.


>
> > I agree that we shouldn't add such limitations. But I disagree that we
> should
> > leave the behavior undocumented in such cases.
>
> OK, so let's agree to disagree.  If that disagreement gets in your way
> of fixing the issues related to this discussion, please say so, and I
> will fix them myself
>
>
No, I will definitely fix it. I think our disagreement is way smaller than
it might look like.

[-- Attachment #2: Type: text/html, Size: 5521 bytes --]

Re: Dynamic loading progress

[David Kastrup]

Philipp Stephani <p.stephani2@gmail.com> writes:

>> > An Emacs string is a sequence of integers.
>>
>> No, it's a sequence of bytes.
>>
>
> From
> https://www.gnu.org/software/emacs/manual/html_node/elisp/String-Basics.html
> :
> "In Emacs Lisp, characters are simply integers ... A string is a fixed
> sequence of characters"
> How a string is represented internally shouldn't be the concern of module
> authors.

Except that we are not talking about modules written in Emacs Lisp.  The
qualifier "In Emacs Lisp" does not hold here.

-- 
David Kastrup

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 19:37:43 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
> I think we shouldn't make the terminology more confusing. If we say
> "UTF-8", we should mean "UTF-8 as defined in the Unicode standard", not the
> Emacs extension of UTF-8. That's all.

I agree, and that's how I use "UTF-8".  The internal representation
used by Emacs is called "utf-8-emacs" or "emacs-internal".

>     We say that we accept valid UTF-8 encoded strings; anything else
>     might produce invalid UTF-8 on output.
> 
> Couldn't we just say "it behaves as if encoding and decoding were done using
> the utf-8-unix coding system"? Because I think that's what this boils down to.

Not sure what you mean by "utf-8-unix", or why it would be better to
say that.  I think this makes the issue harder to understand, because
it involves a reference to the encoding/decoding stuff, something that
module authors might not be fluent with.

>     > No matter what we expect or tolerate, we need to state that.
>     
>     No, we don't. When the callers violate the contract, they cannot
>     expect to know in detail what will happen. If they want to know, they
>     will have to read the source.
> 
> So you want this to be unspecified or undefined behavior? That might be OK (we
> already have that in several places), but we still need to state what the
> contract is.

You can call it "undefined behavior" if you want.  Personally, I don't
think that's accurate: "undefined" means anything can happen, whereas
Emacs at least promises to output the original bytes unchanged, as
long as the text modifications didn't touch them.

>     > An Emacs string is a sequence of integers.
>     
>     No, it's a sequence of bytes.
> 
> From
> https://www.gnu.org/software/emacs/manual/html_node/elisp/String-Basics.html:
> "In Emacs Lisp, characters are simply integers ... A string is a fixed sequence
> of characters"

That's the _User_ manual, it simplifies things to avoid too much
complexity.

> How a string is represented internally shouldn't be the concern of module
> authors.

Indeed.  But it does concern us, the developers of Emacs internals.

> No, I will definitely fix it.

Thank you.

Re: Dynamic loading progress

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philipp Stephani <p.stephani2@gmail.com>
>
>>     > No matter what we expect or tolerate, we need to state that.
>>     
>>     No, we don't. When the callers violate the contract, they cannot
>>     expect to know in detail what will happen. If they want to know, they
>>     will have to read the source.
>> 
>> So you want this to be unspecified or undefined behavior? That might be OK (we
>> already have that in several places), but we still need to state what the
>> contract is.
>
> You can call it "undefined behavior" if you want.  Personally, I don't
> think that's accurate: "undefined" means anything can happen, whereas
> Emacs at least promises to output the original bytes unchanged, as
> long as the text modifications didn't touch them.

Uh, no?  After converting from utf-8-unix to emacs-internal, Emacs' own
processing will not affect originally bad sequences in utf-8-unix.
However, that would require all string passing to include a
decoding/encoding step so that the nominal encoding used by the module
is utf-8-unix.

I think in the interest of both roundtrip times as well as general
usefulness of modules, it would be better if the default communication
encoding with the module would indeed be emacs-internal.

That way, modules can solve problems outside of Unicode proper.

This is definitely not a no-brainer: choosing utf-8-unix as the
communication encoding for modules would be an interface with less
dependency on Emacs' coding internals but it would pin down UTF-8 as the
only actual useful encoding modules could be made to deal with and would
incur performance penalties.

-- 
David Kastrup

Re: Dynamic loading progress

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Philipp Stephani <p.stephani2@gmail.com>,  aurelien.aptel+emacs@gmail.com,  tzz@lifelogs.com,  emacs-devel@gnu.org
> Date: Sun, 22 Nov 2015 21:59:43 +0100
> 
> > You can call it "undefined behavior" if you want.  Personally, I don't
> > think that's accurate: "undefined" means anything can happen, whereas
> > Emacs at least promises to output the original bytes unchanged, as
> > long as the text modifications didn't touch them.
> 
> Uh, no?  After converting from utf-8-unix to emacs-internal, Emacs' own
> processing will not affect originally bad sequences in utf-8-unix.
> However, that would require all string passing to include a
> decoding/encoding step so that the nominal encoding used by the module
> is utf-8-unix.

That's exactly what I am proposing.

> I think in the interest of both roundtrip times as well as general
> usefulness of modules, it would be better if the default communication
> encoding with the module would indeed be emacs-internal.

No!  Modules cannot convert non-ASCII text into emacs-internal (we
don't expose the coding.c functionality through the API), and
shouldn't have to.  That is the job of the interface.

> That way, modules can solve problems outside of Unicode proper.

What problems are those?

> This is definitely not a no-brainer: choosing utf-8-unix as the
> communication encoding for modules would be an interface with less
> dependency on Emacs' coding internals but it would pin down UTF-8 as the
> only actual useful encoding modules could be made to deal with and would
> incur performance penalties.

That's exactly what's being proposed.  But utf-8-unix is NOT the
internal encoding, it's the Unicode-mandated UTF-8.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 1656 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 22. Nov. 2015 um 20:51 Uhr:

> >     > No matter what we expect or tolerate, we need to state that.
> >
> >     No, we don't. When the callers violate the contract, they cannot
> >     expect to know in detail what will happen. If they want to know, they
> >     will have to read the source.
> >
> > So you want this to be unspecified or undefined behavior? That might be
> OK (we
> > already have that in several places), but we still need to state what the
> > contract is.
>
> You can call it "undefined behavior" if you want.  Personally, I don't
> think that's accurate: "undefined" means anything can happen, whereas
> Emacs at least promises to output the original bytes unchanged, as
> long as the text modifications didn't touch them.
>

"Unspecified" would fit the bill better. Actually for most interesting
inputs (UTF-8 strings) the behavior is well-defined anyway.


>
> >     > An Emacs string is a sequence of integers.
> >
> >     No, it's a sequence of bytes.
> >
> > From
> >
> https://www.gnu.org/software/emacs/manual/html_node/elisp/String-Basics.html
> :
> > "In Emacs Lisp, characters are simply integers ... A string is a fixed
> sequence
> > of characters"
>
> That's the _User_ manual, it simplifies things to avoid too much
> complexity.
>

So where's the programmer's manual then? The source code? ;-)


>
> > How a string is represented internally shouldn't be the concern of module
> > authors.
>
> Indeed.  But it does concern us, the developers of Emacs internals.
>
> > No, I will definitely fix it.
>
> Thank you.
>

Attached a patch that uses make_multibyte_string directly.

[-- Attachment #1.2: Type: text/html, Size: 2604 bytes --]

[-- Attachment #2: 0001-Improvements-to-string-conversions.patch --]
[-- Type: application/octet-stream, Size: 3773 bytes --]

From 6e5b055078fe7155c871a1b68257d3705aeb2ceb Mon Sep 17 00:00:00 2001
From: Philipp Stephani <phst@google.com>
Date: Sun, 22 Nov 2015 22:07:40 +0100
Subject: [PATCH] Improvements to string conversions

* emacs-module.c (module_make_function, module_make_string): Use
make_multibyte_string.
(module_copy_string_contents): Encode before reading the byte
size.  Return false if and only if an error occurred.
---
 src/emacs-module.c | 52 +++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 39 insertions(+), 13 deletions(-)

diff --git a/src/emacs-module.c b/src/emacs-module.c
index c8a0d89..f49802c 100644
--- a/src/emacs-module.c
+++ b/src/emacs-module.c
@@ -25,6 +25,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include <stddef.h>
 #include <stdint.h>
 #include <stdio.h>
+#include <string.h>
 
 #include "lisp.h"
 #include "dynlib.h"
@@ -386,9 +387,20 @@ module_make_function (emacs_env *env, ptrdiff_t min_arity, ptrdiff_t max_arity,
   envptr->data = data;
 
   Lisp_Object envobj = make_save_ptr (envptr);
+  Lisp_Object doc;
+  if (documentation == NULL)
+    doc = Qnil;
+  else
+    {
+      ptrdiff_t nbytes = strlen (documentation);
+      ptrdiff_t nchars, ignored_nbytes;
+      parse_str_as_multibyte (documentation, nbytes, &nchars, &ignored_nbytes);
+      doc = make_multibyte_string (documentation, nchars, nbytes);
+    }
+
   Lisp_Object ret = list4 (Qlambda,
                            list2 (Qand_rest, Qargs),
-                           documentation ? build_string (documentation) : Qnil,
+                           doc,
                            list3 (module_call_func,
                                   envobj,
                                   Qargs));
@@ -515,21 +527,34 @@ module_copy_string_contents (emacs_env *env, emacs_value value, char *buffer,
       return false;
     }
 
-  ptrdiff_t raw_size = SBYTES (lisp_str);
+  Lisp_Object lisp_str_utf8 = ENCODE_UTF_8 (lisp_str);
+  ptrdiff_t raw_size = SBYTES (lisp_str_utf8);
+  if (raw_size == PTRDIFF_MAX)
+    {
+      module_non_local_exit_signal_1 (env, Qoverflow_error, Qnil);
+      return false;
+    }
+  ptrdiff_t required_buf_size = raw_size + 1;
+
+  eassert (length != NULL);
 
-  /* Emacs internal encoding is more-or-less UTF8, let's assume utf8
-     encoded emacs string are the same byte size.  */
+  if (buffer == NULL)
+    {
+      *length = required_buf_size;
+      return true;
+    }
+
+  eassert (*length >= 0);
 
-  if (!buffer || length == 0 || *length-1 < raw_size)
+  if (*length < required_buf_size)
     {
-      *length = raw_size + 1;
+      *length = required_buf_size;
+      module_non_local_exit_signal_1 (env, Qargs_out_of_range, Qnil);
       return false;
     }
 
-  Lisp_Object lisp_str_utf8 = ENCODE_UTF_8 (lisp_str);
-  eassert (raw_size == SBYTES (lisp_str_utf8));
-  *length = raw_size + 1;
-  memcpy (buffer, SDATA (lisp_str_utf8), SBYTES (lisp_str_utf8));
+  *length = required_buf_size;
+  memcpy (buffer, SDATA (lisp_str_utf8), raw_size);
   buffer[raw_size] = 0;
 
   return true;
@@ -541,13 +566,14 @@ module_make_string (emacs_env *env, const char *str, ptrdiff_t length)
   check_main_thread ();
   eassert (module_non_local_exit_check (env) == emacs_funcall_exit_return);
   MODULE_HANDLE_SIGNALS;
-  if (length > PTRDIFF_MAX)
+  if (length > STRING_BYTES_BOUND)
     {
       module_non_local_exit_signal_1 (env, Qoverflow_error, Qnil);
       return NULL;
     }
-  /* Assume STR is utf8 encoded.  */
-  return lisp_to_value (env, make_string (str, length));
+  ptrdiff_t nchars, ignored_nbytes;
+  parse_str_as_multibyte (str, length, &nchars, &ignored_nbytes);
+  return lisp_to_value (env, make_multibyte_string (str, nchars, length));
 }
 
 static emacs_value
-- 
2.6.3

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 21:10:58 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
>     You can call it "undefined behavior" if you want. Personally, I don't
>     think that's accurate: "undefined" means anything can happen, whereas
>     Emacs at least promises to output the original bytes unchanged, as
>     long as the text modifications didn't touch them.
> 
> "Unspecified" would fit the bill better. Actually for most interesting inputs
> (UTF-8 strings) the behavior is well-defined anyway.

Correct.

>     That's the _User_ manual, it simplifies things to avoid too much
>     complexity.
> 
> So where's the programmer's manual then? The source code? ;-)

There's the ELisp manual, yes.  It should be more accurate in this
regard.

> Attached a patch that uses make_multibyte_string directly. 

Looks good to me, thanks.

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 22 Nov 2015 21:10:58 +0000
> Cc: tzz@lifelogs.com, aurelien.aptel+emacs@gmail.com, emacs-devel@gnu.org
> 
> Attached a patch that uses make_multibyte_string directly. 

Thanks, pushed.

On further thought, I concluded that making a unibyte string and then
decoding it explicitly by UTF-8 is more tolerant to deviations from
UTF-8, so I made that change in a follow-up commit.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Do., 19. Nov. 2015 um 04:39 Uhr:

> I think the place is the ELisp manual.  I see no reason to have a
> separate manual, since I don't expect the documentation to be large
> enough to justify that.
>

Agreed. I've started to write some reference manual, but that is more of a
mnemonic device for me to remember what the exact behavior of each function
should be. I hope to come up with some user-level documentation soon.

[-- Attachment #2: Type: text/html, Size: 745 bytes --]

Re: Dynamic loading progress

[Ted Zlatanov]

On Thu, 19 Nov 2015 22:14:15 +0000 Philipp Stephani <p.stephani2@gmail.com> wrote: 

PS> Eli Zaretskii <eliz@gnu.org> schrieb am Do., 19. Nov. 2015 um 04:39 Uhr:
>> I think the place is the ELisp manual.  I see no reason to have a
>> separate manual, since I don't expect the documentation to be large
>> enough to justify that.

PS> Agreed. I've started to write some reference manual, but that is more of a
PS> mnemonic device for me to remember what the exact behavior of each function
PS> should be. I hope to come up with some user-level documentation soon.

Please consider sharing it with me and Aurélien and possibly others so
we can work on it together before merging it. It could live in a
temporary Github branch of emacs.git for instance.

Thanks
Ted

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

Ted Zlatanov <tzz@lifelogs.com> schrieb am Do., 19. Nov. 2015 um 03:19 Uhr:

> EZ> Of course!  Every feature should be documented, and this one is no
> EZ> exception.  I suggest to start with NEWS.
>
> My goal with that question was actually to find out *where* the
> developer docs should go, not whether they are necessary.  I think we
> all agree on the necessity.  I can try a first stab at them when I know
> in what context they'll exist.
>
> Regarding NEWS specifically, Aurélien is absent at the moment and will
> be back next week. Maybe Philipp would like to provide the NEWS entries?
> I will write them if neither of them is able or if it can't wait until
> next week.
>

I can see whether I can come up with something. However as far as I can see
the release of 25.1 is not imminent, so the documentation doesn't need to
happen immediately.

[-- Attachment #2: Type: text/html, Size: 1182 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 19 Nov 2015 22:12:14 +0000
> 
>     Regarding NEWS specifically, Aurélien is absent at the moment and will
>     be back next week. Maybe Philipp would like to provide the NEWS entries?
>     I will write them if neither of them is able or if it can't wait until
>     next week.
>     
> 
> I can see whether I can come up with something. However as far as I can see the
> release of 25.1 is not imminent, so the documentation doesn't need to happen
> immediately.

The documentation in the _manuals_ doesn't have to happen immediately,
although it should be our goal to make that happen more frequently
than it happens today: after all, those who use the development Emacs
should also ideally have the manuals to go with that.  But NEWS should
have some minimal documentation of the new feature _immediately_,
i.e. as part of the last commit of the feature.  Otherwise, users will
have no idea what is being added and how to use it, and frequently
even not that it was in fact added.

Thanks in advance.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 1272 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Fr., 20. Nov. 2015 um 08:50 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Thu, 19 Nov 2015 22:12:14 +0000
> >
> >     Regarding NEWS specifically, Aurélien is absent at the moment and
> will
> >     be back next week. Maybe Philipp would like to provide the NEWS
> entries?
> >     I will write them if neither of them is able or if it can't wait
> until
> >     next week.
> >
> >
> > I can see whether I can come up with something. However as far as I can
> see the
> > release of 25.1 is not imminent, so the documentation doesn't need to
> happen
> > immediately.
>
> The documentation in the _manuals_ doesn't have to happen immediately,
> although it should be our goal to make that happen more frequently
> than it happens today: after all, those who use the development Emacs
> should also ideally have the manuals to go with that.  But NEWS should
> have some minimal documentation of the new feature _immediately_,
> i.e. as part of the last commit of the feature.  Otherwise, users will
> have no idea what is being added and how to use it, and frequently
> even not that it was in fact added.
>
> Thanks in advance.
>


I've attached a preliminary patch for NEWS.

[-- Attachment #1.2: Type: text/html, Size: 1707 bytes --]

[-- Attachment #2: 0001-Add-NEWS-entry-for-modules.patch --]
[-- Type: application/octet-stream, Size: 827 bytes --]

From a9a8d868a04742faf8852a08d183b82eb3894a4b Mon Sep 17 00:00:00 2001
From: Philipp Stephani <phst@google.com>
Date: Fri, 20 Nov 2015 22:06:05 +0100
Subject: [PATCH] Add NEWS entry for modules

---
 etc/NEWS | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/etc/NEWS b/etc/NEWS
index 9793d5e..fe0122a 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -302,6 +302,12 @@ header.
 which specifies an alternative printing method which is faster when
 few or no entries have changed.
 
+** Emacs can now load shared/dynamic libraries (modules) that expose a
+C interface.  Such modules can provide additional functions or
+otherwise interact with Emacs just like Lisp code.  Modules have to
+export a function `emacs_module_init' and conform to the API laid out
+in emacs-module.h.
+
 \f
 * Editing Changes in Emacs 25.1
 
-- 
2.6.3

Re: Dynamic loading progress

[Paul Eggert]

Philipp Stephani wrote:
> I've attached a preliminary patch for NEWS.

That text should also mention the --with-modules option of 'configure', and 
mention that modules are disabled by default, and that modules are currently 
experimental and subject to change.

Re: Dynamic loading progress

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 20 Nov 2015 15:46:56 -0800
> 
> Philipp Stephani wrote:
> > I've attached a preliminary patch for NEWS.
> 
> That text should also mention the --with-modules option of 'configure', and 
> mention that modules are disabled by default, and that modules are currently 
> experimental and subject to change.

Agreed.  The parts about --with-modules and its being optional should
go into the section of NEWS about installation changes.

Thanks.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 380 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Sa., 21. Nov. 2015 um 00:46 Uhr:

> Philipp Stephani wrote:
> > I've attached a preliminary patch for NEWS.
>
> That text should also mention the --with-modules option of 'configure', and
> mention that modules are disabled by default, and that modules are
> currently
> experimental and subject to change.
>

OK, attached a new patch.

[-- Attachment #1.2: Type: text/html, Size: 681 bytes --]

[-- Attachment #2: 0001-Add-NEWS-entry-for-modules.patch --]
[-- Type: application/octet-stream, Size: 1408 bytes --]

From 33a51ab2526f0bce7f0e9bc06693f4db8401cf1a Mon Sep 17 00:00:00 2001
From: Philipp Stephani <phst@google.com>
Date: Fri, 20 Nov 2015 22:06:05 +0100
Subject: [PATCH] Add NEWS entry for modules

---
 etc/NEWS | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/etc/NEWS b/etc/NEWS
index 9793d5e..8ed8133 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -36,6 +36,9 @@ otherwise leave it unmarked.
 This builds Emacs with Cairo drawing.  As a side effect, it provides
 support for built-in printing, when Emacs was built with GTK+.
 
+** New configure option --with-modules.
+This enables support for loading dynamic modules; see below.
+
 ---
 ** By default, Emacs no longer works on IRIX.  We expect that Emacs
 users are not affected by this, as SGI stopped supporting IRIX in
@@ -302,6 +305,14 @@ header.
 which specifies an alternative printing method which is faster when
 few or no entries have changed.
 
+** Emacs can now load shared/dynamic libraries (modules) that expose a
+C interface.  Such modules can provide additional functions or
+otherwise interact with Emacs just like Lisp code.  Modules have to
+export a function `emacs_module_init' and conform to the API laid out
+in emacs-module.h.  Modules are disabled by default and need to be
+enabled using the --with-modules configure flag.  They are
+experimental and subject to change.
+
 \f
 * Editing Changes in Emacs 25.1
 
-- 
2.6.3

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 21 Nov 2015 08:30:11 +0000
> Cc: emacs-devel@gnu.org
> 
> OK, attached a new patch. 

LGTM, please push.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 21. Nov. 2015 um 10:07 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sat, 21 Nov 2015 08:30:11 +0000
> > Cc: emacs-devel@gnu.org
> >
> > OK, attached a new patch.
>
> LGTM, please push.
>

I don't have write access to the Savannah repository, can you please
install the patch for me?

[-- Attachment #2: Type: text/html, Size: 776 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 21 Nov 2015 09:25:09 +0000
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
>     LGTM, please push.
> 
> I don't have write access to the Savannah repository, can you please install
> the patch for me? 

Done.

Btw, I see you don't have a copyright assignment on file.  Would you
like to start the paperwork to do that now, so that your future
contributions won't stall?

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 21. Nov. 2015 um 10:52 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sat, 21 Nov 2015 09:25:09 +0000
> > Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> >
> >     LGTM, please push.
> >
> > I don't have write access to the Savannah repository, can you please
> install
> > the patch for me?
>
> Done.
>
> Btw, I see you don't have a copyright assignment on file.  Would you
> like to start the paperwork to do that now, so that your future
> contributions won't stall?
>

My contributions are covered by the agreement between Google and the FSF.

[-- Attachment #2: Type: text/html, Size: 1133 bytes --]

Re: Dynamic loading progress

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 21 Nov 2015 10:33:15 +0000
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
>     Btw, I see you don't have a copyright assignment on file. Would you
>     like to start the paperwork to do that now, so that your future
>     contributions won't stall?
> 
> My contributions are covered by the agreement between Google and the FSF. 

Thanks, I didn't know that.

Re: Dynamic loading progress

[John Wiegley]

>>>>> Steinar Bang <sb@dod.no> writes:

> Hm... wasn't dynamic loading intended as an emacs 25 feature?

I'd like for this to be an emacs 25 feature, unless it still has lots of
implementation work left; then it should go into master toward 25.2.

Criteria for 25.1 now is: Bug fixes and work that refines/improves code
already in 25.1. No new functionality, or anything that would destabilize
25.1. In particular, no API changes of any kind without discussion.

John

Re: Dynamic loading progress

[Xue Fuqiao]

On Wed, Nov 18, 2015 at 1:10 AM, John Wiegley <jwiegley@gmail.com> wrote:
>>>>>> Steinar Bang <sb@dod.no> writes:
>
>> Hm... wasn't dynamic loading intended as an emacs 25 feature?
>
> I'd like for this to be an emacs 25 feature, unless it still has lots of
> implementation work left; then it should go into master toward 25.2.

25.2 should be a bugfix release.  See admin/notes/versioning.

Re: Dynamic loading progress

[Xue Fuqiao]

On Wed, Nov 18, 2015 at 8:17 AM, Xue Fuqiao <xfq.free@gmail.com> wrote:
> On Wed, Nov 18, 2015 at 1:10 AM, John Wiegley <jwiegley@gmail.com> wrote:
>>>>>>> Steinar Bang <sb@dod.no> writes:
>>
>>> Hm... wasn't dynamic loading intended as an emacs 25 feature?
>>
>> I'd like for this to be an emacs 25 feature, unless it still has lots of
>> implementation work left; then it should go into master toward 25.2.
>
> 25.2 should be a bugfix release.  See admin/notes/versioning.

But the current master is 25.1.50, instead of 26.0.50.  And it accepts
new features.

Do you plan to change the versioning scheme (again), John?

Ref: https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00872.html

Re: Dynamic loading progress

[John Wiegley]

>>>>> Xue Fuqiao <xfq.free@gmail.com> writes:

> But the current master is 25.1.50, instead of 26.0.50. And it accepts new
> features.

> Do you plan to change the versioning scheme (again), John?

> Ref: https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00872.html

To me, version numbers are about delivery to the user. If APIs change, or
Emacs start to look and feel different, it should be a major version bump. If
it instead integrates something that's not really a user-facing feature, that
can go in on a minor bump. In general, the 25 "series" should feel like a
consistent march of refinement.

Concurrency would very much justify 26.x, but I don't think dynamic loading
should. That's much more of a developer-oriented feature. It doesn't change
the overall feel, and while it adds APIs, it doesn't change them. (If I'm
wrong on that last point, then yes, 26 it is).

John

Re: Dynamic loading progress

[Steinar Bang]

>>>>> Aurélien Aptel <aurelien.aptel+emacs@gmail.com>:

> I already merged master in my branch at some point, and did so with
> `merge --squash` IIRC, note to self: never do that again.

Indeed.

What this did was replace the master history merged into your branch
with a single commit containing the diffs.

This resulted in files with the same content, but with a broken history.

The way I would have fixed this, would have been
 - make a copy of the current branch
 - roll back the branch to the commit just before the squash merge
 - do a regular merge with emacs master
 - cherry pick all commits made to the original branch after the squash
   merge

After this I would have had a branch with identical contents, but that
can be merged without problems to emacs master.

Suggestion to enable git rerere by default

[Nicolas Richard]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Sun, Nov 8, 2015 at 9:38 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>>> Agreed. Aurélien, the diff is huge and there are many conflicts. It
>>>> looks like the fork from master happened in February 2015 and you're
>>>> simply out of date. Can you please rebase against the emacs.git
>>>> origin/master?  Last push...
>
> I already merged master in my branch at some point, and did so with
> `merge --squash` IIRC, note to self: never do that again.

I agree with that conclusion. However I want to mention...

> made any subsequent master merge a lot harder, sorry for that. The
> weird thing is most conflicting changes where actually identical but
> git couldn't tell. The diff were the same on both side but git couldnt
> tell they were from the same commits since they were squashed.

...did you enable git rerere ? (you can check with: git config --get
rerere.enabled). If not, I suggest that you do it:
    git config --global rerere.enabled 1

I'm not 100% sure that would have helped in your usecase, but it's
useful anyway. Excerpt from `git help rerere' :
       In a workflow employing relatively long lived topic branches, the
       developer sometimes needs to resolve the same conflicts over and
       over again until the topic branches are done (either merged to
       the "release" branch, or sent out and accepted upstream).

       This command assists the developer in this process by recording
       conflicted automerge results and corresponding hand resolve
       results on the initial manual merge, and applying previously
       recorded hand resolutions to their corresponding automerge
       results.


Perhaps we could add that as a suggestion in admin/notes/git-workflow :

modified   admin/notes/git-workflow
@@ -124,6 +124,11 @@ the resulting merge, but if you really want to, feel free to do that.
 Note you can also resume gitmerge in a new Emacs session, since the
 current state will be saved to disk.
 
+The following is helpful if you're going to merge the same set of
+changes multiple times :
+
+  git config --global rerere.enabled 1
+
 When everything's done, look hard at the resulting merge.  Skipping
 commits requires separate merges, so don't be surprised to see more
 than one merge commit.  If you're happy, push.
 



-- 
Nicolas Richard

Re: Suggestion to enable git rerere by default

[Artur Malabarba]

Nicolas Richard <youngfrog@members.fsf.org> writes:

> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>> made any subsequent master merge a lot harder, sorry for that. The
>> weird thing is most conflicting changes where actually identical but
>> git couldn't tell. The diff were the same on both side but git couldnt
>> tell they were from the same commits since they were squashed.
>
> ...did you enable git rerere ? (you can check with: git config --get
> rerere.enabled). If not, I suggest that you do it:
>     git config --global rerere.enabled 1

I had never heard about it. If it's really as nice as it sounds (no
drawbacks?), lets add this recommendation.

> modified   admin/notes/git-workflow
> @@ -124,6 +124,11 @@ the resulting merge, but if you really want to, feel free to do that.
>  Note you can also resume gitmerge in a new Emacs session, since the
>  current state will be saved to disk.
>  
> +The following is helpful if you're going to merge the same set of
> +changes multiple times :

This sounds strange to me. Would it be better to say: “if you're going
to merge master onto the same (long-lived) feature branch multiple
times”?

Re: Suggestion to enable git rerere by default

[Nicolas Richard]

> I had never heard about it. If it's really as nice as it sounds (no
> drawbacks?), lets add this recommendation.

Here's what I found on the potential drawbacks:
http://stackoverflow.com/questions/5519244/are-there-any-downsides-to-enabling-git-rerere

IIUC the main problem is rerere might resolve conflicts that textually
look the same but were semantically different 

Note that when rerere does its work, it leaves no conflict markers but
the changes are still left uncommitted and you can review them.


> > +The following is helpful if you're going to merge the same set of
> > +changes multiple times :
> This sounds strange to me. Would it be better to say: “if you're going
> to merge master onto the same (long-lived) feature branch multiple
> times”?

I'll trust you. I'm 100% not english native.

Nicolas

Re: Dynamic loading progress

[John Wiegley]

>>>>> Ted Zlatanov <tzz@lifelogs.com> writes:

> I think 1 week is the time to feature freeze, but we have lots of time to
> fix bugs and issues. Changing the module ABI/API for instance is IMHO
> perfectly OK during the feature freeze. The feature is "dynamic modules" not
> a specific version thereof, until Emacs is released. Then it becomes
> "dynamic modules as defined by 25.1" and we have to live with them :)

The feature freeze is just to draw a line between what will and won't be in 25
for sure. If you think this feature still deserves a lot of attention, though,
it should go into master after we've released 25.1.

John

Re: Dynamic loading progress

[Ted Zlatanov]

On Mon, 09 Nov 2015 14:10:39 -0800 John Wiegley <jwiegley@gmail.com> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I think 1 week is the time to feature freeze, but we have lots of time to
>> fix bugs and issues. Changing the module ABI/API for instance is IMHO
>> perfectly OK during the feature freeze. The feature is "dynamic modules" not
>> a specific version thereof, until Emacs is released. Then it becomes
>> "dynamic modules as defined by 25.1" and we have to live with them :)

JW> The feature freeze is just to draw a line between what will and won't be in 25
JW> for sure. If you think this feature still deserves a lot of attention, though,
JW> it should go into master after we've released 25.1.

Based on the discussions and code I think it's solid enough to be the
"25.1 dynamic modules interface and implementation."

Ted

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

John Wiegley <jwiegley@gmail.com> schrieb am Sa., 7. Nov. 2015 um 05:58 Uhr:

> >>>>> Ted Zlatanov <tzz@lifelogs.com> writes:
>
> > I've tried to review the discussion so far and believe the code is
> > "functional enough." I think we got agreement from Stefan (probably John
> > too) to exempt dynamic modules from the feature freeze.
>
> If Eli and Stefan agree,


Stefan disagreed with (a) error handling (longjmp vs. local return) and (b)
local object allocation (direct cast vs. a simple local allocator).


> you have my go ahead to integrate the code.


Would you prefer this as one big patch, or as several smaller ones? A big
patch would make it easier to see the code in context, but splitting it up
might increase review speed.

[-- Attachment #2: Type: text/html, Size: 1274 bytes --]

Re: Dynamic loading progress

[Steinar Bang]

>>>>> Philipp Stephani <p.stephani2@gmail.com>:

> Would you prefer this as one big patch, or as several smaller ones? A big
> patch would make it easier to see the code in context, but splitting it up
> might increase review speed.

Unless you have done strange rebasing on your git branch, but instead
used plain merges, then I would suggest doing a git merge instead of
using patches.

That will preserve the full history of the files, for "git blame"
(aka. annotate) and git log.

The merge commit can also be used to provide the patch diff it that is
desired for review, etc.

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

Steinar Bang <sb@dod.no> schrieb am So., 8. Nov. 2015 um 13:51 Uhr:

> >>>>> Philipp Stephani <p.stephani2@gmail.com>:
>
> > Would you prefer this as one big patch, or as several smaller ones? A big
> > patch would make it easier to see the code in context, but splitting it
> up
> > might increase review speed.
>
> Unless you have done strange rebasing on your git branch, but instead
> used plain merges, then I would suggest doing a git merge instead of
> using patches.
>

Fine with me, though that will likely produce lots of 'noise' commits
because we weren't particularly strict about history cleanliness.

[-- Attachment #2: Type: text/html, Size: 1000 bytes --]

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1829 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am So., 4. Okt. 2015 um
21:34 Uhr:

> On 10/04/2015 02:41 AM, Philipp Stephani wrote:
> >     >
> >     >
> >     >       typedef struct emacs_value_tag* emacs_value;
> >     >
> >     >
> >     > I think it's important that this is a pointer to a struct (for type
> >     > safety and size correctness) rather than just an arbitrary type.
> >
> >     A typedef is exactly as typesafe. The question of whether to use a
> >     struct or a typedef is aesthetic. I strongly prefer a typedef, just
> like
> >     pthreads, and I believe that the people who advocate using structs
> >     directly are simply wrong.
> >
> >
> > Ah, I'm not against the typedef, I'm just asking whether you would make
> > it part of the API contract that it's a typedef of a struct pointer, or
> > whether it can be any type.
>
> The problem with defining it as a pointer type is that NULL is now the
> invalid sentinel value, which seems incompatible with both making this
> thing literally a Lisp_Object and Qnil having all zero bits.
>
> That's why I strongly prefer making emacs_value a _pointer_ to a
> Lisp_Object, where we store the Lisp_Object in an array owned by the
> emacs_env. This way, allocating local values is very cheap.
>

A simple fixed array however puts a fixed upper bound on the number of
local values used. That might be undesirable, especially for plugins
dealing with large data structures. The simplest unbound allocation
mechanism would use an array of emacs_value pointers (emacs_value **)
stored in the environment object; the array itself and its elements would
be allocated using malloc as usual, and the array would be grown using
realloc. However, that requires lots of allocations (at least one per local
object used) and could make a more complex allocator necessary.

[-- Attachment #2: Type: text/html, Size: 2306 bytes --]

Re: Dynamic loading progress

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2390 bytes --]

On 10/14/2015 03:34 PM, Philipp Stephani wrote:
> 
> 
> Daniel Colascione <dancol@dancol.org <mailto:dancol@dancol.org>> schrieb
> am So., 4. Okt. 2015 um 21:34 Uhr:
> 
>     On 10/04/2015 02:41 AM, Philipp Stephani wrote:
>     >     >
>     >     >
>     >     >       typedef struct emacs_value_tag* emacs_value;
>     >     >
>     >     >
>     >     > I think it's important that this is a pointer to a struct
>     (for type
>     >     > safety and size correctness) rather than just an arbitrary type.
>     >
>     >     A typedef is exactly as typesafe. The question of whether to use a
>     >     struct or a typedef is aesthetic. I strongly prefer a typedef,
>     just like
>     >     pthreads, and I believe that the people who advocate using structs
>     >     directly are simply wrong.
>     >
>     >
>     > Ah, I'm not against the typedef, I'm just asking whether you would
>     make
>     > it part of the API contract that it's a typedef of a struct
>     pointer, or
>     > whether it can be any type.
> 
>     The problem with defining it as a pointer type is that NULL is now the
>     invalid sentinel value, which seems incompatible with both making this
>     thing literally a Lisp_Object and Qnil having all zero bits.
> 
>     That's why I strongly prefer making emacs_value a _pointer_ to a
>     Lisp_Object, where we store the Lisp_Object in an array owned by the
>     emacs_env. This way, allocating local values is very cheap.
> 
> 
> A simple fixed array however puts a fixed upper bound on the number of
> local values used. That might be undesirable, especially for plugins
> dealing with large data structures.

Right. That's why we have multiple such arrays, each of a pretty decent
size, linked together.

Something like this:

struct emacs_value_frame {
    Lisp_Object objects[512];
    struct emacs_value_frame* next;
};

struct emacs_env_guts {
    ...
    struct emacs_value_frame initial_frame;
    struct emacs_value_frame* current_frame;
};

We get 512 local references for free just as a side
effect of having an emacs_env, and if we need more references, we can
heap-allocate another emacs_value_frame and make it current. This way, a
reasonable number of local references is free, but if you need more, you
can have them.

This is exactly the approach Java uses.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

Your approach is interesting but it currently works as-is for 64bits
platforms and 32bits platform without wide-int and can be fixed later
without breaking the API, if I understand correctly.

I would rather focus on the remaining blocking things for the merge in
master (and availability in Emacs 25!):

* New opaque Lisp type to embbed C pointers with a vtable (for at
least a finalizer)

Can we re-use the finalizer feature you implemented? I haven't looked into it.

* Loading

I'm currently in the process of having `load' look for modules. For
this I've added the system shared lib extension (.dll, .so, ...) to
Vload_prefixes choosen at configure time (modules are not handled in
Fload at the moment).
I know that on OSX there's both ".dyn" and ".so" but I don't know much
about it. Is supporting both worth it?

Couldn't we simply pick an extension like ".edm" (emacs dynamic
module) and rename the generated lib on each system when a module is
built?

* Doc strings

If we use the doxygen syntax, it won't follow the various elisp
conventions (args in caps, first line must be descriptive, etc).
Jumping to definition is also pretty convenient, should we keep that
feature for modules? Also how should we load the docstrings
themselves?

* Packaging

We need to extend the spec of ELisp packages to take into account
modules. The way I see it a module will have a "core" source file in C
(or something else) and a bunch of helpers implemented in elisp on top
that expose the package features in a more user-friendly way.

As for distribution, only packaging sources is easier. This is a
pretty big undertaking but we could bundle a compiler on platforms
that don't have one easily accessible (like Strawberry Perl on Windows
that bundles mingw-w64).

Re: Dynamic loading progress

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2340 bytes --]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> schrieb am Do., 15. Okt.
2015 um 01:32 Uhr:

> Your approach is interesting but it currently works as-is for 64bits
> platforms and 32bits platform without wide-int and can be fixed later
> without breaking the API, if I understand correctly.
>

I would be more worried about representation of nullptr. With casting
nullptr can stand for a valid Lisp value, so that it's not possible to
return invalid values, which is visible to the user.


> * Loading
>
> I know that on OSX there's both ".dyn" and ".so" but I don't know much
> about it. Is supporting both worth it?
>

According to http://stackoverflow.com/a/2339910/178761 there are two types:
bundles and dynamic libraries. The differences were bigger in the past
(only bundles could be loaded dynamically), but now they are mostly
equivalent. Apparently .so and .bundle are usually used for bundles and
.dylib for dynamic libraries, but e.g. your example modules compile as
dynamic libraries with an .so extension and Emacs is totally happy. So
supporting .so should be enough.


>
> Couldn't we simply pick an extension like ".edm" (emacs dynamic
> module) and rename the generated lib on each system when a module is
> built?
>

I'd prefer using the conventional extensions. A custom extension might
confuse users and clash with existing meanings of that extension.


>
> * Doc strings
>
> If we use the doxygen syntax, it won't follow the various elisp
> conventions (args in caps, first line must be descriptive, etc).
> Jumping to definition is also pretty convenient, should we keep that
> feature for modules? Also how should we load the docstrings
> themselves?
>
>
Can't modules already run the equivalent of

(put 'module-func 'function-documentation "docstring")

? With that the basics should already be there.


> * Packaging
>
> We need to extend the spec of ELisp packages to take into account
> modules. The way I see it a module will have a "core" source file in C
> (or something else) and a bunch of helpers implemented in elisp on top
> that expose the package features in a more user-friendly way.
>

That is one option, but modules consisting entirely of non-Elisp files are
also to be expected, given that the module API is as powerful as Elisp.

[-- Attachment #2: Type: text/html, Size: 3421 bytes --]

Re: Dynamic loading progress

[Aurélien Aptel]

On Thu, Oct 15, 2015 at 3:05 AM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> According to http://stackoverflow.com/a/2339910/178761 there are two types:
> bundles and dynamic libraries. The differences were bigger in the past (only
> bundles could be loaded dynamically), but now they are mostly equivalent.
> Apparently .so and .bundle are usually used for bundles and .dylib for
> dynamic libraries, but e.g. your example modules compile as dynamic
> libraries with an .so extension and Emacs is totally happy. So supporting
> .so should be enough.

OK.

> I'd prefer using the conventional extensions. A custom extension might
> confuse users and clash with existing meanings of that extension.

Fair enough.

> (put 'module-func 'function-documentation "docstring")
>
> ? With that the basics should already be there.

True, but we lose the jump-to-definition feature.

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Thu, Oct 15, 2015 at 3:05 AM, Philipp Stephani <p.stephani2@gmail.com> wrote:
>> According to http://stackoverflow.com/a/2339910/178761 there are two types:
>> bundles and dynamic libraries. The differences were bigger in the past (only
>> bundles could be loaded dynamically), but now they are mostly equivalent.
>> Apparently .so and .bundle are usually used for bundles and .dylib for
>> dynamic libraries, but e.g. your example modules compile as dynamic
>> libraries with an .so extension and Emacs is totally happy. So supporting
>> .so should be enough.
>
> OK.
>
>> I'd prefer using the conventional extensions. A custom extension might
>> confuse users and clash with existing meanings of that extension.
>
> Fair enough.
>
>> (put 'module-func 'function-documentation "docstring")
>>
>> ? With that the basics should already be there.
>
> True, but we lose the jump-to-definition feature.

"jump-to-definition" relies on `load-history' (see elisp-mode.el
elisp--xref-find-definitions and subr.el symbol-file). So module
initialization functions can add the appropriate entries in
`load-history'.

-- 
-- Stephe

Re: Dynamic loading progress

[Andreas Schwab]

Daniel Colascione <dancol@dancol.org> writes:

> Positionally, as fixed in the structure that defines the table.
>
> struct {
>   void (*func1)(int);
>   void (*func2)(double);
>   /* etc. */
> }
>
> This way, the ABI Emacs exports is explicit, and breaking it is unlikely
> to happen accidentally.

You should allow for future extension, by adding a version field.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Dynamic loading progress

[Stefan Monnier]

>> If that doesn't work, then we really need an emacs.h file ASAP, and
>> the module should include only that file, without including <config.h>
>> inside that file.
> Nor <lisp.h>.  And probably not lots of other things.

Of course not.  It's the other way around: lisp.h should include emacs.h.

> Is there an acceptable (ie portable and reliable) way to determine the
> OS in a Makefile? I'm not aware of one (short of Gnu config.guess).

M-x module-make RET should do the trick, since Emacs knows for which OS
it was built and can know which compiler options were used.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

On Wed, Feb 11, 2015 at 10:17 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>>> Symbol's function definition is void: defun

> [*] running test.el...
> Eager macro-expansion failure: (void-variable )
> Running 5 tests (2015-02-11 16:13:26-0500)
> /home/tzz/source/emacs/emacs/src/emacs: symbol lookup error: /home/tzz/source/emacs/emacs/modules/yaml/yaml.so: undefined symbol: yaml_parser_initialize

The branch builds fine on my machine (archlinux), both with and
without ltdl configure options. There is something wrong the
linker/linking parameters for the emacs binary or modules.

Re: Dynamic loading progress

[Stephen Leake]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> Run ./modules/tests.py. It should work no matter the current working
> directory.

This works for me in branch feature/aptel/dynamic-modules-rc3 on Debian
stable (wheezy) 32 bit.

-- 
-- Stephe

Re: Dynamic loading progress

[Ted Zlatanov]

On Sat, 31 Jan 2015 19:57:42 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> I'm just letting everyone know I'm working on my patch again. I've
AA> started by merging the last 2-3 months commits in master but of course
AA> there are merge conflicts... It's taking more time to resolve than
AA> anticipated.

AA> There's a macro conflict with "Qc". I had to rename Qc to Qc_ in
AA> modules/opaque/opaque.c because globals.h defines Qc to something.

AA> I'm also working on a small test suite since I (and maintainers)
AA> quickly want to know when my code breaks.

Great, thanks.  I assume you started with the dynamic-modules-rc2 branch
I pushed to the Emacs repo?  If not, take a look at it.

With the test suite, I'm sure we'll be able to merge it very quickly.

Ted

Re: Dynamic loading progress

[Stefan Monnier]

AA> I'm just letting everyone know I'm working on my patch again. I've
AA> started by merging the last 2-3 months commits in master but of course
AA> there are merge conflicts... It's taking more time to resolve than
AA> anticipated.

If you're not sure what to do with a particular conflict, you can sent
it here: it's likely that someone will be able to tell you very quickly
what's up with it.

Most conflicts are "trivial" to solve to those who know what was the reason
for the change.


        Stefan

Re: Dynamic loading progress

[Aurélien Aptel]

On Wed, Feb 4, 2015 at 1:11 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Sat, 31 Jan 2015 19:57:42 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote:
>
> AA> I'm just letting everyone know I'm working on my patch again. I've
> AA> started by merging the last 2-3 months commits in master but of course
> AA> there are merge conflicts... It's taking more time to resolve than
> AA> anticipated.
>
> AA> There's a macro conflict with "Qc". I had to rename Qc to Qc_ in
> AA> modules/opaque/opaque.c because globals.h defines Qc to something.
>
> AA> I'm also working on a small test suite since I (and maintainers)
> AA> quickly want to know when my code breaks.
>
> Great, thanks.  I assume you started with the dynamic-modules-rc2 branch
> I pushed to the Emacs repo?  If not, take a look at it.

Nope, I've used my dynamic-modules branch... fuck.

Someone keeps messing with small parts of doc.c I'm also working on
and each time I have to re-understand partially how the spaghetti code
in Snarf_documention works to merge. Currently, the wrong prefix
(sibling_xxx) is added to the doc file path which prevent the
docstring to be loaded and makes emacs abort. I'm sure it's not hard
but it's the third time and it tires me. I should have waited to merge
or let someone else do it. Sorry for the rant.

I've tried to keep every changes from master while merging, I hope I
haven't skipped/reverted something.

> With the test suite, I'm sure we'll be able to merge it very quickly.

I've made some tests in several module test.el file using ERT. There's
algo a small script in modules/tests.py which runs all the module
tests. Be warned though, since the loading part is broken I've written
the tests blindly. Some might fail/pass when they should/nt.

The code is on my dynamic-modules branch on my github repo.

$ git clone https://github.com/aaptel/emacs-dynamic-module.git
$ git checkout dynamic-modules

I'm sure there's a lighter way to get it (if you already have the
emacs repo) involving arcane git commands. If you know them I'm
interested.

Re: Dynamic loading progress

[Ted Zlatanov]

On Wed, 4 Feb 2015 22:49:44 +0100 Aurélien Aptel <aurelien.aptel+emacs@gmail.com> wrote: 

AA> Nope, I've used my dynamic-modules branch... fuck.

Yeah, at this point my work has diverged too much from yours.  My work
was mostly adding ChangeLogs and cleaning up, so don't worry, just keep
going with your side.

AA> The code is on my dynamic-modules branch on my github repo.

AA> $ git clone https://github.com/aaptel/emacs-dynamic-module.git
AA> $ git checkout dynamic-modules

AA> I'm sure there's a lighter way to get it (if you already have the
AA> emacs repo) involving arcane git commands. If you know them I'm
AA> interested.

    git fetch git@github.com:aaptel/emacs-dynamic-module.git dynamic-modules
    ... then work against FETCH_HEAD ...

I pushed your work so far to the Emacs repo in branch
feature/aptel/dynamic-modules-rc3 so:

    git clone emacs-repo-url
    git co feature/aptel/dynamic-modules-rc3
    git log master.. 

shows your commits correctly.

We can do a final cleanup with `git rebase -i master' and fix
ChangeLogs, etc. but let's get it in a working state first.

I'll delete the dynamic-modules-rc2 branch eventually but am leaving it
for now in case I want the ChangeLog entries I wrote there.

Ted