* Preview: portable dumper
@ 2016-11-28 19:50 Daniel Colascione
2016-11-28 19:58 ` Burton Samograd
` (3 more replies)
0 siblings, 4 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 19:50 UTC (permalink / raw)
To: Emacs developers
[-- Attachment #1: Type: text/plain, Size: 2898 bytes --]
I've been working on a portable dumper for GNU Emacs. The attached patch
is an early version of this work. I'll write up all the usual NEWS
entries, changelogs, and (in this case necessary) dedicated
documentation before I land it. I want to start getting review comments
now that the code has roughly its final shape.
The point of this gargantuan patch is that we can rip out our unexec
implementations and replace them with loading a data file that contains
an Emacs heap image. There are no dependencies on executable rewriting,
disabling ASLR, or saving and restoring internal malloc state. This
system works with fully position-independent executables and with any
malloc implementation.
Basically, there's a new dump-emacs-portable function that walks the
Emacs heap and writes the data, along with necessary relocation, to a
file called emacs.pdmp. On startup, early in main, we find emacs.pdmp
and load it. Once we've loaded the file, we walk the list of relocations
contained in the dump and adjust it to account for the runtime locations
of Emacs and the dump data (neither of which we know in advance in a PIE
world.)
There are a few subtleties: I've carefully designed the file format to
be mmap-able and to minimize the number of on-demand copies the system
makes while accessing this file. For example, we stick bool-vectors and
string data at the end of the dump in a contiguous block. We follow this
block with the relocations, which we can throw away as soon as we've
used them.
An additional optimization follows, although this part isn't implemented
yet: we can define a "preferred load address" for the dump and write
relocation information such that if the dump and Emacs end up being
where we expect them to be, we don't have to perform any relocations at all.
The system gracefully degrades though. If we can't use mmap or whatever
on a given platform, it's possible to just slurp the whole file into a
malloced region of memory and access it from there. This approach would
benefit from compression, which will reduce IO loads: LZ4 reduces the
dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away
The code isn't even close to optimized yet --- I've only tested it at
-O0, defined GC_CHECK_MARKED_OBJECTS, and not yet inlined
frequently-called functions pdumper.h --- but even so, it's within 100ms
or so of an unexeced Emacs.
It's also possible to dump an already-dumped Emacs, so it should be
possible for users to have their own dump files.
If we want to preserve the current model of a single "emacs" executable
that contains itself, we can embed emacs.pdmp inside the emacs
executable data section pretty easily. There's no behavior change involved.
If you want to try this code, build CANNOT_DUMP=1, run ./temacs -l
loadup pdump, then ./emacs (or if that doesn't work, ./emacs
--dump-file=emacs.pdmp).
[-- Attachment #2: pdumper.diff --]
[-- Type: text/x-patch, Size: 278645 bytes --]
diff --git a/.gitignore b/.gitignore
index 15f9c56..3b7975f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -182,6 +182,7 @@ lib-src/emacsclient
lib-src/etags
lib-src/hexl
lib-src/make-docfile
+lib-src/make-fingerprint
lib-src/movemail
lib-src/profile
lib-src/test-distrib
@@ -195,6 +196,9 @@ src/bootstrap-emacs
src/emacs
src/emacs-[0-9]*
src/temacs
+src/temacs.in
+src/fingerprint.c
+src/*.pdmp
# Character-set info.
admin/charsets/jisx2131-filter
diff --git a/configure.ac b/configure.ac
index c9759e1..3e44716 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2167,6 +2167,10 @@ AC_DEFUN
hybrid_malloc=yes
fi
+## XXX
+hybrid_malloc=no
+system_malloc=yes
+
GMALLOC_OBJ=
HYBRID_MALLOC=
if test "${system_malloc}" = "yes"; then
diff --git a/doc/lispref/loading.texi b/doc/lispref/loading.texi
index 44ce719..901a009 100644
--- a/doc/lispref/loading.texi
+++ b/doc/lispref/loading.texi
@@ -303,10 +303,7 @@ Library Search
initializes @code{load-path} using the @file{lisp}
directory in the directory containing the sources from which it
was built.
-@c Though there should be no *.el files in builddir/lisp, so it's pointless.
-If you built Emacs in a separate directory from the
-sources, it also adds the lisp directories from the build directory.
-(In all cases, elements are represented as absolute file names.)
+Elements are represented as absolute file names.
@cindex site-lisp directories
Unless you start Emacs with the @option{--no-site-lisp} option,
@@ -329,9 +326,9 @@ Library Search
The first one is for locally installed files for a specific Emacs
version; the second is for locally installed files meant for use
with all installed Emacs versions. (If Emacs is running uninstalled,
-it also adds @file{site-lisp} directories from the source and build
-directories, if they exist. Normally these directories do not contain
-@file{site-lisp} directories.)
+it also adds @file{site-lisp} directories from the source
+directory. Normally this directory does not contain a
+@file{site-lisp} directory.)
@cindex @env{EMACSLOADPATH} environment variable
If the environment variable @env{EMACSLOADPATH} is set, it modifies
diff --git a/lib-src/Makefile.in b/lib-src/Makefile.in
index c3693ab..db768eb 100644
--- a/lib-src/Makefile.in
+++ b/lib-src/Makefile.in
@@ -158,7 +158,7 @@ INSTALLABLES =
UTILITIES = profile${EXEEXT} movemail${EXEEXT} hexl${EXEEXT} \
update-game-score${EXEEXT}
-DONT_INSTALL= make-docfile${EXEEXT}
+DONT_INSTALL= make-docfile${EXEEXT} make-fingerprint${EXEEXT}
# Like UTILITIES, but they're not system-dependent, and should not be
# deleted by the distclean target.
@@ -385,6 +385,9 @@ profile${EXEEXT}:
make-docfile${EXEEXT}: ${srcdir}/make-docfile.c $(NTLIB) $(config_h)
$(AM_V_CCLD)$(CC) ${ALL_CFLAGS} $< $(NTLIB) $(LOADLIBES) -o $@
+make-fingerprint${EXEEXT}: ${srcdir}/make-fingerprint.c $(NTLIB) $(config_h)
+ $(AM_V_CCLD)$(CC) ${ALL_CFLAGS} $< $(NTLIB) $(LOADLIBES) -o $@
+
movemail${EXEEXT}: ${srcdir}/movemail.c pop.o $(NTLIB) $(config_h)
$(AM_V_CCLD)$(CC) ${ALL_CFLAGS} ${MOVE_FLAGS} $< pop.o \
$(NTLIB) $(LOADLIBES) $(LIBS_MOVE) -o $@
diff --git a/lib-src/make-fingerprint.c b/lib-src/make-fingerprint.c
new file mode 100644
index 0000000..dd2f093
--- /dev/null
+++ b/lib-src/make-fingerprint.c
@@ -0,0 +1,86 @@
+/* Hash inputs and generate C file with the digest.
+
+Copyright (C) 1985-1986, 1992-1994, 1997, 1999-2016 Free Software
+Foundation, Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
+
+
+/* The arguments given to this program are all the object files that
+ go into building GNU Emacs. There is no special search logic to find
+ the files. */
+
+#include <config.h>
+
+#include <stdarg.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sysstdio.h>
+#include <sha256.h>
+
+#ifdef WINDOWSNT
+/* Defined to be sys_fopen in ms-w32.h, but only #ifdef emacs, so this
+ is really just insurance. */
+#undef fopen
+#include <direct.h>
+#endif /* WINDOWSNT */
+
+int
+main (int argc, char **argv)
+{
+ struct sha256_ctx ctx;
+ sha256_init_ctx (&ctx);
+
+ for (int i = 1; i < argc; ++i)
+ {
+ FILE *f = fopen (argv[i], "r" FOPEN_BINARY);
+ if (!f)
+ {
+ fprintf (stderr, "%s: Error: could not open %s\n",
+ argv[0], argv[i]);
+ return 1;
+ }
+
+ char buf[128*1024];
+ do
+ {
+ size_t chunksz = fread (buf, 1, sizeof (buf), f);
+ if (ferror (f))
+ {
+ fprintf (stderr, "%s: Error: could not read %s\n",
+ argv[0], argv[i]);
+ return 1;
+ }
+ sha256_process_bytes (buf, chunksz, &ctx);
+ } while (!feof (f));
+ fclose (f);
+ }
+
+ uint8_t digest[32];
+ sha256_finish_ctx (&ctx, digest);
+ printf ("#include \"fingerprint.h\"\n");
+ printf ("\n");
+ printf ("const uint8_t fingerprint[32] = { ");
+ for (int i = 0; i < 32; ++i)
+ printf ("%s0x%02X", i ? ", " : "", digest[i]);
+ printf (" };\n");
+
+ return EXIT_SUCCESS;
+}
+
+/* make-fingerprint.c ends here */
diff --git a/lisp/loadup.el b/lisp/loadup.el
index 5c16464..58a1ba9 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -26,6 +26,9 @@
;; This is loaded into a bare Emacs to make a dumpable one.
+;; Emacs injects the variable `dump-mode' to tell us how to dump. We unintern
+;; it before allowing user code to run.
+
;; If you add a file to be loaded here, keep the following points in mind:
;; i) If the file is no-byte-compile, explicitly load the .el version.
@@ -55,28 +58,52 @@
(setq redisplay--inhibit-bidi t)
;; Add subdirectories to the load-path for files that might get
-;; autoloaded when bootstrapping.
+;; autoloaded when bootstrapping or running Emacs normally.
;; This is because PATH_DUMPLOADSEARCH is just "../lisp".
-(if (or (equal (member "bootstrap" command-line-args) '("bootstrap"))
+(if (or (member dump-mode '("bootstrap" "pbootstrap"))
;; FIXME this is irritatingly fragile.
(equal (nth 4 command-line-args) "unidata-gen.el")
- (equal (nth 7 command-line-args) "unidata-gen-files")
+ (equal (nth 7 command-line-args) "unidata-gen-files")
+ ;; XXX figure out why we need this here
(if (fboundp 'dump-emacs)
(string-match "src/bootstrap-emacs" (nth 0 command-line-args))
t))
- (let ((dir (car load-path)))
- ;; We'll probably overflow the pure space.
- (setq purify-flag nil)
- ;; Value of max-lisp-eval-depth when compiling initially.
- ;; During bootstrapping the byte-compiler is run interpreted when
- ;; compiling itself, which uses a lot more stack than usual.
- (setq max-lisp-eval-depth 2200)
- (setq load-path (list (expand-file-name "." dir)
- (expand-file-name "emacs-lisp" dir)
- (expand-file-name "language" dir)
- (expand-file-name "international" dir)
- (expand-file-name "textmodes" dir)
- (expand-file-name "vc" dir)))))
+ ;; Find the entry in load-path that contains Emacs elisp and
+ ;; splice some additional directories in there for the benefit
+ ;; of autoload and regular Emacs use.
+ (let ((subdirs '("emacs-lisp"
+ "language"
+ "international"
+ "textmodes"
+ "vc"))
+ (iter load-path))
+ (while iter
+ (let ((dir (car iter))
+ (subdirs subdirs)
+ esubdirs esubdir)
+ (while subdirs
+ (setq esubdir (expand-file-name (car subdirs) dir))
+ (setq subdirs (cdr subdirs))
+ (if (file-directory-p esubdir)
+ (setq esubdirs (cons esubdir esubdirs))
+ (setq subdirs nil esubdirs nil)))
+ (if esubdirs
+ (progn
+ (setcdr iter (nconc (nreverse esubdirs) (cdr iter)))
+ (setq iter nil))
+ (setq iter (cdr iter))
+ (if (null iter)
+ (signal
+ 'error (list
+ (format-message
+ "Could not find elisp load-path: searched %S"
+ load-path))))))))
+ ;; We'll probably overflow the pure space.
+ (setq purify-flag nil)
+ ;; Value of max-lisp-eval-depth when compiling initially.
+ ;; During bootstrapping the byte-compiler is run interpreted when
+ ;; compiling itself, which uses a lot more stack than usual.
+ (setq max-lisp-eval-depth 2200))
(if (eq t purify-flag)
;; Hash consing saved around 11% of pure space in my tests.
@@ -84,9 +111,7 @@
(message "Using load-path %s" load-path)
-;; This is a poor man's `last', since we haven't loaded subr.el yet.
-(if (or (equal (member "bootstrap" command-line-args) '("bootstrap"))
- (equal (member "dump" command-line-args) '("dump")))
+(if dump-mode
(progn
;; To reduce the size of dumped Emacs, we avoid making huge char-tables.
(setq inhibit-load-charset-map t)
@@ -343,8 +368,7 @@
;; file primitive. So the only workable solution to support building
;; in non-ASCII directories is to manipulate unibyte strings in the
;; current locale's encoding.
-(if (and (member (car (last command-line-args)) '("dump" "bootstrap"))
- (multibyte-string-p default-directory))
+(if (and dump-mode (multibyte-string-p default-directory))
(error "default-directory must be unibyte when dumping Emacs!"))
;; Determine which last version number to use
@@ -443,37 +467,39 @@
;; Make sure we will attempt bidi reordering henceforth.
(setq redisplay--inhibit-bidi nil)
-(if (member (car (last command-line-args)) '("dump" "bootstrap"))
- (progn
- (message "Dumping under the name emacs")
+(if dump-mode
+ (let ((output (cond ((equal dump-mode "pdump") "emacs.pdmp")
+ ((equal dump-mode "dump") "emacs")
+ ((equal dump-mode "bootstrap") "emacs")
+ ((equal dump-mode "pbootstrap") (error "XXX"))
+ (t (error "unrecognized dump mode %s" dump-mode)))))
+ (message "Dumping under the name %s" output)
(condition-case ()
- (delete-file "emacs")
- (file-error nil))
- ;; We used to dump under the name xemacs, but that occasionally
- ;; confused people installing Emacs (they'd install the file
- ;; under the name `xemacs'), and it's inconsistent with every
- ;; other GNU program's build process.
- (dump-emacs "emacs" "temacs")
- (message "%d pure bytes used" pure-bytes-used)
- ;; Recompute NAME now, so that it isn't set when we dump.
- (if (not (or (eq system-type 'ms-dos)
- ;; Don't bother adding another name if we're just
- ;; building bootstrap-emacs.
- (equal (last command-line-args) '("bootstrap"))))
- (let ((name (concat "emacs-" emacs-version))
- (exe (if (eq system-type 'windows-nt) ".exe" "")))
- (while (string-match "[^-+_.a-zA-Z0-9]+" name)
- (setq name (concat (downcase (substring name 0 (match-beginning 0)))
- "-"
- (substring name (match-end 0)))))
- (setq name (concat name exe))
- (message "Adding name %s" name)
- ;; When this runs on Windows, invocation-directory is not
- ;; necessarily the current directory.
- (add-name-to-file (expand-file-name (concat "emacs" exe)
- invocation-directory)
- (expand-file-name name invocation-directory)
- t)))
+ (delete-file output)
+ (file-error nil))
+ (if (equal dump-mode "pdump")
+ (dump-emacs-portable output)
+ (dump-emacs output "temacs")
+ (message "%d pure bytes used" pure-bytes-used)
+ ;; Recompute NAME now, so that it isn't set when we dump.
+ (if (not (or (eq system-type 'ms-dos)
+ ;; Don't bother adding another name if we're just
+ ;; building bootstrap-emacs.
+ (equal dump-mode "bootstrap")))
+ (let ((name (concat "emacs-" emacs-version))
+ (exe (if (eq system-type 'windows-nt) ".exe" "")))
+ (while (string-match "[^-+_.a-zA-Z0-9]+" name)
+ (setq name (concat (downcase (substring name 0 (match-beginning 0)))
+ "-"
+ (substring name (match-end 0)))))
+ (setq name (concat name exe))
+ (message "Adding name %s" name)
+ ;; When this runs on Windows, invocation-directory is not
+ ;; necessarily the current directory.
+ (add-name-to-file (expand-file-name (concat "emacs" exe)
+ invocation-directory)
+ (expand-file-name name invocation-directory)
+ t))))
(kill-emacs)))
;; For machines with CANNOT_DUMP defined in config.h,
@@ -484,6 +510,7 @@
(equal (nth 2 command-line-args) "loadup"))
(setcdr command-line-args (nthcdr 3 command-line-args)))
+(unintern 'dump-mode obarray)
(eval top-level)
\f
diff --git a/src/.gdbinit b/src/.gdbinit
index b0c0dfd..d9820e5 100644
--- a/src/.gdbinit
+++ b/src/.gdbinit
@@ -1215,7 +1215,7 @@ document xwhichsymbols
maximum number of symbols referencing it to produce.
end
-# Show Lisp backtrace after normal backtrace.
+# # Show Lisp backtrace after normal backtrace.
define hookpost-backtrace
set $bt = backtrace_top ()
if backtrace_p ($bt)
diff --git a/src/Makefile.in b/src/Makefile.in
index dc0bfff..d1397b78 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -401,7 +401,7 @@ base_obj =
buffer.o filelock.o insdel.o marker.o \
minibuf.o fileio.o dired.o \
cmds.o casetab.o casefiddle.o indent.o search.o regex.o undo.o \
- alloc.o data.o doc.o editfns.o callint.o \
+ alloc.o pdumper.o data.o doc.o editfns.o callint.o \
eval.o floatfns.o fns.o font.o print.o lread.o $(MODULES_OBJ) \
syntax.o $(UNEXEC_OBJ) bytecode.o \
process.o gnutls.o callproc.o \
@@ -602,14 +602,28 @@ LIBEGNU_ARCHIVE =
$(LIBEGNU_ARCHIVE): $(config_h)
$(MAKE) -C $(lib) all
+EMACS_DEPS_PRE=$(LIBXMENU) $(ALLOBJS)
+EMACS_DEPS_POST=$(LIBEGNU_ARCHIVE) $(EMACSRES) ${charsets} ${charscript}
+BUILD_EMACS_PRE=$(AM_V_CCLD)$(CC) $(ALL_CFLAGS) $(TEMACS_LDFLAGS) $(LDFLAGS) \
+ -o $@ $(ALLOBJS)
+BUILD_EMACS_POST=$(LIBEGNU_ARCHIVE) $(W32_RES_LINK) $(LIBES)
+
+## We hash this file to generate the build fingerprint
+temacs.in$(EXEEXT): $(EMACS_DEPS_PRE) fingerprint-dummy.o $(EMACS_DEPS_POST)
+ $(BUILD_EMACS_PRE) fingerprint-dummy.o $(BUILD_EMACS_POST)
+
+$(libsrc)/make-fingerprint$(EXEEXT): $(libsrc)/make-fingerprint.c $(lib)/libgnu.a
+ $(MAKE) -C $(libsrc) make-fingerprint$(EXEEXT)
+
+fingerprint.c: temacs.in$(EXEEXT) $(libsrc)/make-fingerprint$(EXEEXT)
+ $(libsrc)/make-fingerprint$(EXEEXT) temacs.in$(EEXIST) > fingerprint.c
+
## We have to create $(etc) here because init_cmdargs tests its
## existence when setting Vinstallation_directory (FIXME?).
## This goes on to affect various things, and the emacs binary fails
## to start if Vinstallation_directory has the wrong value.
-temacs$(EXEEXT): $(LIBXMENU) $(ALLOBJS) \
- $(LIBEGNU_ARCHIVE) $(EMACSRES) ${charsets} ${charscript}
- $(AM_V_CCLD)$(CC) $(ALL_CFLAGS) $(TEMACS_LDFLAGS) $(LDFLAGS) \
- -o temacs $(ALLOBJS) $(LIBEGNU_ARCHIVE) $(W32_RES_LINK) $(LIBES)
+temacs$(EXEEXT): $(EMACS_DEPS_PRE) fingerprint.o $(EMACS_DEPS_POST)
+ $(BUILD_EMACS_PRE) fingerprint.o $(BUILD_EMACS_POST)
$(MKDIR_P) $(etc)
ifneq ($(CANNOT_DUMP),yes)
ifneq ($(PAXCTL_notdumped),)
@@ -651,6 +665,7 @@ .PHONY:
mostlyclean:
rm -f temacs$(EXEEXT) core *.core \#* *.o
+ rm -f temacs.in$(EXEEXT) fingerprint.c
rm -f ../etc/DOC
rm -f bootstrap-emacs$(EXEEXT) emacs-$(version)$(EXEEXT)
rm -f buildobj.h
diff --git a/src/alloc.c b/src/alloc.c
index 90c6f94..e31f71d 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -42,6 +42,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "keyboard.h"
#include "frame.h"
#include "blockinput.h"
+#include "pdumper.h"
#include "termhooks.h" /* For struct terminal. */
#ifdef HAVE_WINDOW_SYSTEM
#include TERM_HEADER
@@ -77,6 +78,8 @@ static bool valgrind_p;
/* GC_CHECK_MARKED_OBJECTS means do sanity checks on allocated objects. */
+#define GC_CHECK_MARKED_OBJECTS 1 // XXX
+
/* GC_MALLOC_CHECK defined means perform validity checks of malloc'd
memory. Can do this only if using gmalloc.c and if not checking
marked objects. */
@@ -202,13 +205,13 @@ alloc_unexec_post (void)
/* Mark, unmark, query mark bit of a Lisp string. S must be a pointer
to a struct Lisp_String. */
-#define MARK_STRING(S) ((S)->size |= ARRAY_MARK_FLAG)
-#define UNMARK_STRING(S) ((S)->size &= ~ARRAY_MARK_FLAG)
-#define STRING_MARKED_P(S) (((S)->size & ARRAY_MARK_FLAG) != 0)
+#define XMARK_STRING(S) ((S)->size |= ARRAY_MARK_FLAG)
+#define XUNMARK_STRING(S) ((S)->size &= ~ARRAY_MARK_FLAG)
+#define XSTRING_MARKED_P(S) (((S)->size & ARRAY_MARK_FLAG) != 0)
-#define VECTOR_MARK(V) ((V)->header.size |= ARRAY_MARK_FLAG)
-#define VECTOR_UNMARK(V) ((V)->header.size &= ~ARRAY_MARK_FLAG)
-#define VECTOR_MARKED_P(V) (((V)->header.size & ARRAY_MARK_FLAG) != 0)
+#define XMARK_VECTOR(V) ((V)->header.size |= ARRAY_MARK_FLAG)
+#define XUNMARK_VECTOR(V) ((V)->header.size &= ~ARRAY_MARK_FLAG)
+#define XVECTOR_MARKED_P(V) (((V)->header.size & ARRAY_MARK_FLAG) != 0)
/* Default value of gc_cons_threshold (see below). */
@@ -357,6 +360,36 @@ static void compact_small_strings (void);
static void free_large_strings (void);
extern Lisp_Object which_symbols (Lisp_Object, EMACS_INT) EXTERNALLY_VISIBLE;
+/* Forward declare mark accessor functions: they're used all over the
+ place. */
+
+INLINE static bool vector_marked_p (const struct Lisp_Vector *v);
+INLINE static void set_vector_marked (struct Lisp_Vector *v);
+
+INLINE static bool vectorlike_marked_p (const struct vectorlike_header *v);
+INLINE static void set_vectorlike_marked (struct vectorlike_header *v);
+
+INLINE static bool cons_marked_p (const struct Lisp_Cons *c);
+INLINE static void set_cons_marked (struct Lisp_Cons *c);
+
+INLINE static bool string_marked_p (const struct Lisp_String *s);
+INLINE static void set_string_marked (struct Lisp_String *s);
+
+INLINE static bool symbol_marked_p (const struct Lisp_Symbol *s);
+INLINE static void set_symbol_marked (struct Lisp_Symbol *s);
+
+INLINE static bool misc_any_marked_p (const struct Lisp_Misc_Any *m);
+INLINE static void set_misc_any_marked (struct Lisp_Misc_Any *m);
+
+INLINE static bool marker_marked_p (const struct Lisp_Marker *m);
+INLINE static void set_marker_marked (struct Lisp_Marker *m);
+
+INLINE static bool overlay_marked_p (const struct Lisp_Overlay *m);
+INLINE static void set_overlay_marked (struct Lisp_Overlay *m);
+
+INLINE static bool interval_marked_p (INTERVAL i);
+INLINE static void set_interval_marked (INTERVAL i);
+
/* When scanning the C stack for live Lisp objects, Emacs keeps track of
what memory allocated via lisp_malloc and lisp_align_malloc is intended
for what purpose. This enumeration specifies the type of memory. */
@@ -383,7 +416,10 @@ enum mem_type
/* A unique object in pure space used to make some Lisp objects
on free lists recognizable in O(1). */
-static Lisp_Object Vdead;
+#ifndef ENABLE_CHECKING
+static
+#endif
+Lisp_Object Vdead;
#define DEADP(x) EQ (x, Vdead)
#ifdef GC_MALLOC_CHECK
@@ -465,30 +501,21 @@ static struct mem_node *mem_find (void *);
#endif
/* Addresses of staticpro'd variables. Initialize it to a nonzero
- value; otherwise some compilers put it into BSS. */
+ value if we might dump; otherwise some compilers put it into
+ BSS. */
-enum { NSTATICS = 2048 };
-static Lisp_Object *staticvec[NSTATICS] = {&Vpurify_flag};
+Lisp_Object *staticvec[NSTATICS]
+#ifndef CANNOT_DUMP
+= {&Vpurify_flag}
+#endif
+ ;
/* Index of next unused slot in staticvec. */
-static int staticidx;
+int staticidx;
static void *pure_alloc (size_t, int);
-/* True if N is a power of 2. N should be positive. */
-
-#define POWER_OF_2(n) (((n) & ((n) - 1)) == 0)
-
-/* Return X rounded to the next multiple of Y. Y should be positive,
- and Y - 1 + X should not overflow. Arguments should not have side
- effects, as they are evaluated more than once. Tune for Y being a
- power of 2. */
-
-#define ROUNDUP(x, y) (POWER_OF_2 (y) \
- ? ((y) - 1 + (x)) & ~ ((y) - 1) \
- : ((y) - 1 + (x)) - ((y) - 1 + (x)) % (y))
-
/* Return PTR rounded up to the next multiple of ALIGNMENT. */
static void *
@@ -553,18 +580,18 @@ mmap_lisp_allowed_p (void)
over our address space. We also can't use mmap for lisp objects
if we might dump: unexec doesn't preserve the contents of mmapped
regions. */
- return pointers_fit_in_lispobj_p () && !might_dump;
+ return pointers_fit_in_lispobj_p () && !might_dump_with_unexec;
}
#endif
/* Head of a circularly-linked list of extant finalizers. */
-static struct Lisp_Finalizer finalizers;
+struct Lisp_Finalizer finalizers;
/* Head of a circularly-linked list of finalizers that must be invoked
because we deemed them unreachable. This list must be global, and
not a local inside garbage_collect_1, in case we GC again while
running finalizers. */
-static struct Lisp_Finalizer doomed_finalizers;
+struct Lisp_Finalizer doomed_finalizers;
\f
/************************************************************************
@@ -895,6 +922,8 @@ xfree (void *block)
{
if (!block)
return;
+ if (pdumper_object_p (block))
+ return;
MALLOC_BLOCK_INPUT;
free (block);
MALLOC_UNBLOCK_INPUT;
@@ -1117,6 +1146,9 @@ lisp_malloc (size_t nbytes, enum mem_type type)
static void
lisp_free (void *block)
{
+ if (pdumper_object_p (block))
+ return;
+
MALLOC_BLOCK_INPUT;
free (block);
#ifndef GC_MALLOC_CHECK
@@ -1548,22 +1580,23 @@ make_interval (void)
/* Mark Lisp objects in interval I. */
static void
-mark_interval (register INTERVAL i, Lisp_Object dummy)
+mark_interval_tree_1 (register INTERVAL i, Lisp_Object dummy)
{
/* Intervals should never be shared. So, if extra internal checking is
enabled, GC aborts if it seems to have visited an interval twice. */
- eassert (!i->gcmarkbit);
- i->gcmarkbit = 1;
+ eassert (!interval_marked_p (i));
+ set_interval_marked (i);
mark_object (i->plist);
}
/* Mark the interval tree rooted in I. */
-#define MARK_INTERVAL_TREE(i) \
- do { \
- if (i && !i->gcmarkbit) \
- traverse_intervals_noorder (i, mark_interval, Qnil); \
- } while (0)
+static void
+mark_interval_tree (INTERVAL i)
+{
+ if (i && !interval_marked_p (i))
+ traverse_intervals_noorder (i, mark_interval_tree_1, Qnil);
+}
/***********************************************************************
String Allocation
@@ -1798,7 +1831,9 @@ static void
init_strings (void)
{
empty_unibyte_string = make_pure_string ("", 0, 0, 0);
+ staticpro (&empty_unibyte_string);
empty_multibyte_string = make_pure_string ("", 0, 0, 1);
+ staticpro (&empty_multibyte_string);
}
@@ -2091,10 +2126,10 @@ sweep_strings (void)
if (s->data)
{
/* String was not on free-list before. */
- if (STRING_MARKED_P (s))
+ if (XSTRING_MARKED_P (s))
{
/* String is live; unmark it and its intervals. */
- UNMARK_STRING (s);
+ XUNMARK_STRING (s);
/* Do not use string_(set|get)_intervals here. */
s->intervals = balance_intervals (s->intervals);
@@ -2591,7 +2626,8 @@ make_formatted_string (char *buf, const char *format, ...)
&= ~((bits_word) 1 << ((n) % BITS_PER_BITS_WORD)))
#define FLOAT_BLOCK(fptr) \
- ((struct float_block *) (((uintptr_t) (fptr)) & ~(BLOCK_ALIGN - 1)))
+ (eassert (!pdumper_object_p (fptr)), \
+ ((struct float_block *) (((uintptr_t) (fptr)) & ~(BLOCK_ALIGN - 1))))
#define FLOAT_INDEX(fptr) \
((((uintptr_t) (fptr)) & (BLOCK_ALIGN - 1)) / sizeof (struct Lisp_Float))
@@ -2604,13 +2640,13 @@ struct float_block
struct float_block *next;
};
-#define FLOAT_MARKED_P(fptr) \
+#define XFLOAT_MARKED_P(fptr) \
GETMARKBIT (FLOAT_BLOCK (fptr), FLOAT_INDEX ((fptr)))
-#define FLOAT_MARK(fptr) \
+#define XFLOAT_MARK(fptr) \
SETMARKBIT (FLOAT_BLOCK (fptr), FLOAT_INDEX ((fptr)))
-#define FLOAT_UNMARK(fptr) \
+#define XFLOAT_UNMARK(fptr) \
UNSETMARKBIT (FLOAT_BLOCK (fptr), FLOAT_INDEX ((fptr)))
/* Current float_block. */
@@ -2660,7 +2696,7 @@ make_float (double float_value)
MALLOC_UNBLOCK_INPUT;
XFLOAT_INIT (val, float_value);
- eassert (!FLOAT_MARKED_P (XFLOAT (val)));
+ eassert (!XFLOAT_MARKED_P (XFLOAT (val)));
consing_since_gc += sizeof (struct Lisp_Float);
floats_consed++;
total_free_floats--;
@@ -2685,7 +2721,8 @@ make_float (double float_value)
/ (sizeof (struct Lisp_Cons) * CHAR_BIT + 1))
#define CONS_BLOCK(fptr) \
- ((struct cons_block *) ((uintptr_t) (fptr) & ~(BLOCK_ALIGN - 1)))
+ (eassert (!pdumper_object_p (fptr)), \
+ ((struct cons_block *) ((uintptr_t) (fptr) & ~(BLOCK_ALIGN - 1))))
#define CONS_INDEX(fptr) \
(((uintptr_t) (fptr) & (BLOCK_ALIGN - 1)) / sizeof (struct Lisp_Cons))
@@ -2698,13 +2735,13 @@ struct cons_block
struct cons_block *next;
};
-#define CONS_MARKED_P(fptr) \
+#define XCONS_MARKED_P(fptr) \
GETMARKBIT (CONS_BLOCK (fptr), CONS_INDEX ((fptr)))
-#define CONS_MARK(fptr) \
+#define XMARK_CONS(fptr) \
SETMARKBIT (CONS_BLOCK (fptr), CONS_INDEX ((fptr)))
-#define CONS_UNMARK(fptr) \
+#define XUNMARK_CONS(fptr) \
UNSETMARKBIT (CONS_BLOCK (fptr), CONS_INDEX ((fptr)))
/* Current cons_block. */
@@ -2766,7 +2803,7 @@ DEFUN ("cons", Fcons, Scons, 2, 2, 0,
XSETCAR (val, car);
XSETCDR (val, cdr);
- eassert (!CONS_MARKED_P (XCONS (val)));
+ eassert (!XCONS_MARKED_P (XCONS (val)));
consing_since_gc += sizeof (struct Lisp_Cons);
total_free_conses--;
cons_cells_consed++;
@@ -3089,6 +3126,7 @@ static void
init_vectors (void)
{
zero_vector = make_pure_vector (0);
+ staticpro (&zero_vector);
}
/* Allocate vector from a vector block. */
@@ -3159,7 +3197,7 @@ allocate_vector_from_block (size_t nbytes)
/* Return the memory footprint of V in bytes. */
-static ptrdiff_t
+ptrdiff_t
vector_nbytes (struct Lisp_Vector *v)
{
ptrdiff_t size = v->header.size & ~ARRAY_MARK_FLAG;
@@ -3234,9 +3272,9 @@ sweep_vectors (void)
for (vector = (struct Lisp_Vector *) block->data;
VECTOR_IN_BLOCK (vector, block); vector = next)
{
- if (VECTOR_MARKED_P (vector))
+ if (XVECTOR_MARKED_P (vector))
{
- VECTOR_UNMARK (vector);
+ XUNMARK_VECTOR (vector);
total_vectors++;
nbytes = vector_nbytes (vector);
total_vector_slots += nbytes / word_size;
@@ -3256,7 +3294,7 @@ sweep_vectors (void)
while (VECTOR_IN_BLOCK (next, block))
{
- if (VECTOR_MARKED_P (next))
+ if (XVECTOR_MARKED_P (next))
break;
cleanup_vector (next);
nbytes = vector_nbytes (next);
@@ -3296,9 +3334,9 @@ sweep_vectors (void)
for (lv = large_vectors; lv; lv = *lvprev)
{
vector = large_vector_vec (lv);
- if (VECTOR_MARKED_P (vector))
+ if (XVECTOR_MARKED_P (vector))
{
- VECTOR_UNMARK (vector);
+ XUNMARK_VECTOR (vector);
total_vectors++;
if (vector->header.size & PSEUDOVECTOR_FLAG)
{
@@ -3961,7 +3999,7 @@ mark_finalizer_list (struct Lisp_Finalizer *head)
finalizer != head;
finalizer = finalizer->next)
{
- finalizer->base.gcmarkbit = true;
+ set_misc_any_marked (&finalizer->base);
mark_object (finalizer->function);
}
}
@@ -3978,7 +4016,8 @@ queue_doomed_finalizers (struct Lisp_Finalizer *dest,
while (finalizer != src)
{
struct Lisp_Finalizer *next = finalizer->next;
- if (!finalizer->base.gcmarkbit && !NILP (finalizer->function))
+ if (!misc_any_marked_p (&finalizer->base)
+ && !NILP (finalizer->function))
{
unchain_finalizer (finalizer);
finalizer_insert (dest, finalizer);
@@ -4044,6 +4083,169 @@ FUNCTION. FUNCTION will be run once per finalizer object. */)
\f
/************************************************************************
+ Mark bit access functions
+ ************************************************************************/
+
+/* With the rare excpetion of functions implementing block-based
+ allocation of various types, you should not directly test or set GC
+ mark bits on objects. Some objects might live in special memory
+ regions (e.g., a dump image) and might store their mark bits
+ elsewhere. */
+
+static enum pvec_type find_pvec_type (const struct Lisp_Vector *ptr);
+
+static bool
+vector_marked_p (const struct Lisp_Vector *v)
+{
+ if (pdumper_object_p (v))
+ {
+ /* TODO: look into using a range test against
+ hot_discardable_start instead of loading the pvec header.
+ We'll have already loaded the dump header cache line, after
+ all. */
+ enum pvec_type pvectype = find_pvec_type (v);
+ if (pvectype == PVEC_BOOL_VECTOR)
+ return true;
+ return pdumper_marked_p (v);
+ }
+ return XVECTOR_MARKED_P (v);
+}
+
+static void
+set_vector_marked (struct Lisp_Vector *v)
+{
+ if (pdumper_object_p (v))
+ {
+ eassert (find_pvec_type (v) != PVEC_BOOL_VECTOR);
+ pdumper_set_marked (v);
+ }
+ else
+ XMARK_VECTOR (v);
+}
+
+static bool
+vectorlike_marked_p (const struct vectorlike_header *header)
+{
+ return vector_marked_p ((const struct Lisp_Vector *) header);
+}
+
+static void
+set_vectorlike_marked (struct vectorlike_header *header)
+{
+ set_vector_marked ((struct Lisp_Vector *) header);
+}
+
+static bool
+cons_marked_p (const struct Lisp_Cons *c)
+{
+ return pdumper_object_p (c)
+ ? pdumper_marked_p (c)
+ : XCONS_MARKED_P (c);
+}
+
+static void
+set_cons_marked (struct Lisp_Cons *c)
+{
+ if (pdumper_object_p (c))
+ pdumper_set_marked (c);
+ else
+ XMARK_CONS (c);
+}
+
+static bool
+string_marked_p (const struct Lisp_String *s)
+{
+ return pdumper_object_p (s)
+ ? pdumper_marked_p (s)
+ : XSTRING_MARKED_P (s);
+}
+
+static void
+set_string_marked (struct Lisp_String *s)
+{
+ if (pdumper_object_p (s))
+ pdumper_set_marked (s);
+ else
+ XMARK_STRING (s);
+}
+
+static bool
+symbol_marked_p (const struct Lisp_Symbol *s)
+{
+ return pdumper_object_p (s)
+ ? pdumper_marked_p (s)
+ : s->gcmarkbit;
+}
+
+static void
+set_symbol_marked (struct Lisp_Symbol *s)
+{
+ if (pdumper_object_p (s))
+ pdumper_set_marked (s);
+ else
+ s->gcmarkbit = true;
+}
+
+static bool
+misc_any_marked_p (const struct Lisp_Misc_Any *m)
+{
+ return pdumper_object_p (m)
+ ? pdumper_marked_p (m)
+ : m->gcmarkbit;
+}
+
+static void
+set_misc_any_marked (struct Lisp_Misc_Any *m)
+{
+ if (pdumper_object_p (m))
+ pdumper_set_marked (m);
+ else
+ m->gcmarkbit = true;
+}
+
+static bool
+marker_marked_p (const struct Lisp_Marker *m)
+{
+ return misc_any_marked_p ((struct Lisp_Misc_Any *) m);
+}
+
+static void
+set_marker_marked (struct Lisp_Marker *m)
+{
+ set_misc_any_marked ((struct Lisp_Misc_Any *) m);
+}
+
+static bool
+overlay_marked_p (const struct Lisp_Overlay *m)
+{
+ return misc_any_marked_p ((struct Lisp_Misc_Any *) m);
+}
+
+static void
+set_overlay_marked (struct Lisp_Overlay *m)
+{
+ set_misc_any_marked ((struct Lisp_Misc_Any *) m);
+}
+
+static bool
+interval_marked_p (INTERVAL i)
+{
+ return pdumper_object_p (i)
+ ? pdumper_marked_p (i)
+ : i->gcmarkbit;
+}
+
+static void
+set_interval_marked (INTERVAL i)
+{
+ if (pdumper_object_p (i))
+ pdumper_set_marked (i);
+ else
+ i->gcmarkbit = true;
+}
+
+\f
+/************************************************************************
Memory Full Handling
************************************************************************/
@@ -4712,6 +4914,19 @@ mark_maybe_object (Lisp_Object obj)
return;
void *po = XPNTR (obj);
+
+ /* If the pointer is in the dumped image and the dump has a record
+ of the object starting at the place where the pointerp points, we
+ definitely have an object. If the pointer is in the dumped image
+ and the dump has no idea what the pointer is pointing at, we
+ definitely _don't_ have an object. */
+ if (pdumper_object_p (po))
+ {
+ if (pdumper_object_p_precise (po))
+ mark_object (obj);
+ return;
+ }
+
struct mem_node *m = mem_find (po);
if (m != MEM_NIL)
@@ -4722,11 +4937,11 @@ mark_maybe_object (Lisp_Object obj)
{
case Lisp_String:
mark_p = (live_string_p (m, po)
- && !STRING_MARKED_P ((struct Lisp_String *) po));
+ && !XSTRING_MARKED_P ((struct Lisp_String *) po));
break;
case Lisp_Cons:
- mark_p = (live_cons_p (m, po) && !CONS_MARKED_P (XCONS (obj)));
+ mark_p = (live_cons_p (m, po) && !XCONS_MARKED_P (XCONS (obj)));
break;
case Lisp_Symbol:
@@ -4734,7 +4949,7 @@ mark_maybe_object (Lisp_Object obj)
break;
case Lisp_Float:
- mark_p = (live_float_p (m, po) && !FLOAT_MARKED_P (XFLOAT (obj)));
+ mark_p = (live_float_p (m, po) && !XFLOAT_MARKED_P (XFLOAT (obj)));
break;
case Lisp_Vectorlike:
@@ -4742,9 +4957,9 @@ mark_maybe_object (Lisp_Object obj)
buffer because checking that dereferences the pointer
PO which might point anywhere. */
if (live_vector_p (m, po))
- mark_p = !SUBRP (obj) && !VECTOR_MARKED_P (XVECTOR (obj));
+ mark_p = !SUBRP (obj) && !XVECTOR_MARKED_P (XVECTOR (obj));
else if (live_buffer_p (m, po))
- mark_p = BUFFERP (obj) && !VECTOR_MARKED_P (XBUFFER (obj));
+ mark_p = BUFFERP (obj) && !XVECTOR_MARKED_P (XBUFFER (obj));
break;
case Lisp_Misc:
@@ -4799,6 +5014,17 @@ mark_maybe_pointer (void *p)
p = (void *) ((uintptr_t) p & ~(GCALIGNMENT - 1));
}
+ if (pdumper_object_p (p))
+ {
+ enum Lisp_Type type = pdumper_find_object_type (p);
+ if (type != PDUMPER_NO_OBJECT)
+ mark_object ((type == Lisp_Symbol)
+ ? make_lisp_symbol(p)
+ : make_lisp_ptr(p, type));
+ /* See mark_maybe_object for why we can confidently return. */
+ return;
+ }
+
m = mem_find (p);
if (m != MEM_NIL)
{
@@ -4812,18 +5038,18 @@ mark_maybe_pointer (void *p)
break;
case MEM_TYPE_BUFFER:
- if (live_buffer_p (m, p) && !VECTOR_MARKED_P ((struct buffer *)p))
+ if (live_buffer_p (m, p) && !XVECTOR_MARKED_P ((struct buffer *)p))
XSETVECTOR (obj, p);
break;
case MEM_TYPE_CONS:
- if (live_cons_p (m, p) && !CONS_MARKED_P ((struct Lisp_Cons *) p))
+ if (live_cons_p (m, p) && !XCONS_MARKED_P ((struct Lisp_Cons *) p))
XSETCONS (obj, p);
break;
case MEM_TYPE_STRING:
if (live_string_p (m, p)
- && !STRING_MARKED_P ((struct Lisp_String *) p))
+ && !XSTRING_MARKED_P ((struct Lisp_String *) p))
XSETSTRING (obj, p);
break;
@@ -4838,7 +5064,7 @@ mark_maybe_pointer (void *p)
break;
case MEM_TYPE_FLOAT:
- if (live_float_p (m, p) && !FLOAT_MARKED_P (p))
+ if (live_float_p (m, p) && !XFLOAT_MARKED_P (p))
XSETFLOAT (obj, p);
break;
@@ -4848,7 +5074,7 @@ mark_maybe_pointer (void *p)
{
Lisp_Object tem;
XSETVECTOR (tem, p);
- if (!SUBRP (tem) && !VECTOR_MARKED_P (XVECTOR (tem)))
+ if (!SUBRP (tem) && !XVECTOR_MARKED_P (XVECTOR (tem)))
obj = tem;
}
break;
@@ -5122,6 +5348,9 @@ valid_lisp_object_p (Lisp_Object obj)
if (p == &buffer_defaults || p == &buffer_local_symbols)
return 2;
+ if (pdumper_object_p (p))
+ return pdumper_object_p_precise (p) ? 1 : 0;
+
struct mem_node *m = mem_find (p);
if (m == MEM_NIL)
@@ -5538,7 +5767,7 @@ compact_font_cache_entry (Lisp_Object entry)
/* Consider OBJ if it is (font-spec . [font-entity font-entity ...]). */
if (CONSP (obj) && GC_FONT_SPEC_P (XCAR (obj))
- && !VECTOR_MARKED_P (GC_XFONT_SPEC (XCAR (obj)))
+ && !vectorlike_marked_p (&GC_XFONT_SPEC (XCAR (obj))->header)
/* Don't use VECTORP here, as that calls ASIZE, which could
hit assertion violation during GC. */
&& (VECTORLIKEP (XCDR (obj))
@@ -5554,7 +5783,8 @@ compact_font_cache_entry (Lisp_Object entry)
{
Lisp_Object objlist;
- if (VECTOR_MARKED_P (GC_XFONT_ENTITY (AREF (obj_cdr, i))))
+ if (vectorlike_marked_p (
+ &GC_XFONT_ENTITY (AREF (obj_cdr, i))->header))
break;
objlist = AREF (AREF (obj_cdr, i), FONT_OBJLIST_INDEX);
@@ -5564,7 +5794,7 @@ compact_font_cache_entry (Lisp_Object entry)
struct font *font = GC_XFONT_OBJECT (val);
if (!NILP (AREF (val, FONT_TYPE_INDEX))
- && VECTOR_MARKED_P(font))
+ && vectorlike_marked_p(&font->header))
break;
}
if (CONSP (objlist))
@@ -5633,7 +5863,7 @@ compact_undo_list (Lisp_Object list)
{
if (CONSP (XCAR (tail))
&& MARKERP (XCAR (XCAR (tail)))
- && !XMARKER (XCAR (XCAR (tail)))->gcmarkbit)
+ && !marker_marked_p (XMARKER (XCAR (XCAR (tail)))))
*prev = XCDR (tail);
else
prev = xcdr_addr (tail);
@@ -5659,6 +5889,69 @@ mark_pinned_symbols (void)
}
}
+static void
+visit_vectorlike_root (struct gc_root_visitor visitor,
+ struct Lisp_Vector *ptr,
+ enum gc_root_type type)
+{
+ ptrdiff_t size = ptr->header.size;
+ ptrdiff_t i;
+
+ if (size & PSEUDOVECTOR_FLAG)
+ size &= PSEUDOVECTOR_SIZE_MASK;
+ for (i = 0; i < size; i++)
+ visitor.visit (&ptr->contents[i], type, visitor.data);
+}
+
+static void
+visit_buffer_root (struct gc_root_visitor visitor,
+ struct buffer *buffer,
+ enum gc_root_type type)
+{
+ /* Buffers that are roots don't have intervals, an undo list, or
+ other constructs that real buffers have. */
+ eassert (buffer->base_buffer == NULL);
+ eassert (buffer->overlays_before == NULL);
+ eassert (buffer->overlays_after == NULL);
+
+ /* Visit the buffer-locals. */
+ visit_vectorlike_root (visitor, (struct Lisp_Vector *) buffer, type);
+}
+
+/* Visit GC roots stored in the Emacs data section. Used by both core
+ GC and by the portable dumping code.
+
+ There are other GC roots of course, but these roots are dynamic
+ runtime data structures that pdump doesn't care about and so we can
+ continue to mark those directly in garbage_collect_1. */
+void
+visit_static_gc_roots (struct gc_root_visitor visitor)
+{
+ visit_buffer_root (visitor,
+ &buffer_defaults,
+ GC_ROOT_BUFFER_LOCAL_DEFAULT);
+ visit_buffer_root (visitor,
+ &buffer_local_symbols,
+ GC_ROOT_BUFFER_LOCAL_NAME);
+
+ for (int i = 0; i < ARRAYELTS (lispsym); i++)
+ {
+ Lisp_Object sptr = builtin_lisp_symbol (i);
+ visitor.visit (&sptr, GC_ROOT_C_SYMBOL, visitor.data);
+ }
+
+ for (int i = 0; i < staticidx; i++)
+ visitor.visit (staticvec[i], GC_ROOT_STATICPRO, visitor.data);
+}
+
+static void
+mark_object_root_visitor (Lisp_Object *root_ptr,
+ enum gc_root_type type,
+ void *data)
+{
+ mark_object (*root_ptr);
+}
+
/* Subroutine of Fgarbage_collect that does most of the work. It is a
separate function so that we could limit mark_stack in searching
the stack frames below this function, thus avoiding the rare cases
@@ -5671,7 +5964,6 @@ garbage_collect_1 (void *end)
{
struct buffer *nextb;
char stack_top_variable;
- ptrdiff_t i;
bool message_p;
ptrdiff_t count = SPECPDL_INDEX ();
struct timespec start;
@@ -5752,14 +6044,10 @@ garbage_collect_1 (void *end)
/* Mark all the special slots that serve as the roots of accessibility. */
- mark_buffer (&buffer_defaults);
- mark_buffer (&buffer_local_symbols);
-
- for (i = 0; i < ARRAYELTS (lispsym); i++)
- mark_object (builtin_lisp_symbol (i));
-
- for (i = 0; i < staticidx; i++)
- mark_object (*staticvec[i]);
+ struct gc_root_visitor visitor;
+ memset (&visitor, 0, sizeof (visitor));
+ visitor.visit = mark_object_root_visitor;
+ visit_static_gc_roots (visitor);
mark_pinned_symbols ();
mark_specpdl ();
@@ -5811,10 +6099,6 @@ garbage_collect_1 (void *end)
gc_sweep ();
- /* Clear the mark bits that we set in certain root slots. */
- VECTOR_UNMARK (&buffer_defaults);
- VECTOR_UNMARK (&buffer_local_symbols);
-
check_cons_list ();
gc_in_progress = 0;
@@ -6014,7 +6298,7 @@ mark_glyph_matrix (struct glyph_matrix *matrix)
for (; glyph < end_glyph; ++glyph)
if (STRINGP (glyph->object)
- && !STRING_MARKED_P (XSTRING (glyph->object)))
+ && !string_marked_p (XSTRING (glyph->object)))
mark_object (glyph->object);
}
}
@@ -6035,13 +6319,18 @@ static int last_marked_index;
ptrdiff_t mark_object_loop_halt EXTERNALLY_VISIBLE;
static void
-mark_vectorlike (struct Lisp_Vector *ptr)
+mark_vectorlike (struct vectorlike_header *header)
{
+ struct Lisp_Vector *ptr = (struct Lisp_Vector *) header;
ptrdiff_t size = ptr->header.size;
ptrdiff_t i;
- eassert (!VECTOR_MARKED_P (ptr));
- VECTOR_MARK (ptr); /* Else mark it. */
+ eassert (!vector_marked_p (ptr));
+
+ /* Bool vectors have a different case in mark_object. */
+ eassert (PSEUDOVECTOR_TYPE (&ptr->header) != PVEC_BOOL_VECTOR);
+
+ set_vector_marked (ptr); /* Else mark it. */
if (size & PSEUDOVECTOR_FLAG)
size &= PSEUDOVECTOR_SIZE_MASK;
@@ -6064,17 +6353,18 @@ mark_char_table (struct Lisp_Vector *ptr, enum pvec_type pvectype)
/* Consult the Lisp_Sub_Char_Table layout before changing this. */
int i, idx = (pvectype == PVEC_SUB_CHAR_TABLE ? SUB_CHAR_TABLE_OFFSET : 0);
- eassert (!VECTOR_MARKED_P (ptr));
- VECTOR_MARK (ptr);
+ eassert (!vector_marked_p (ptr));
+ set_vector_marked (ptr);
for (i = idx; i < size; i++)
{
Lisp_Object val = ptr->contents[i];
- if (INTEGERP (val) || (SYMBOLP (val) && XSYMBOL (val)->gcmarkbit))
+ if (INTEGERP (val) ||
+ (SYMBOLP (val) && symbol_marked_p (XSYMBOL (val))))
continue;
if (SUB_CHAR_TABLE_P (val))
{
- if (! VECTOR_MARKED_P (XVECTOR (val)))
+ if (! vector_marked_p (XVECTOR (val)))
mark_char_table (XVECTOR (val), PVEC_SUB_CHAR_TABLE);
}
else
@@ -6088,7 +6378,7 @@ mark_compiled (struct Lisp_Vector *ptr)
{
int i, size = ptr->header.size & PSEUDOVECTOR_SIZE_MASK;
- VECTOR_MARK (ptr);
+ set_vector_marked (ptr);
for (i = 0; i < size; i++)
if (i != COMPILED_CONSTANTS)
mark_object (ptr->contents[i]);
@@ -6100,12 +6390,11 @@ mark_compiled (struct Lisp_Vector *ptr)
static void
mark_overlay (struct Lisp_Overlay *ptr)
{
- for (; ptr && !ptr->gcmarkbit; ptr = ptr->next)
+ for (; ptr && !overlay_marked_p (ptr); ptr = ptr->next)
{
- ptr->gcmarkbit = 1;
- /* These two are always markers and can be marked fast. */
- XMARKER (ptr->start)->gcmarkbit = 1;
- XMARKER (ptr->end)->gcmarkbit = 1;
+ set_overlay_marked (ptr);
+ mark_object (ptr->start);
+ mark_object (ptr->end);
mark_object (ptr->plist);
}
}
@@ -6116,11 +6405,11 @@ static void
mark_buffer (struct buffer *buffer)
{
/* This is handled much like other pseudovectors... */
- mark_vectorlike ((struct Lisp_Vector *) buffer);
+ mark_vectorlike (&buffer->header);
/* ...but there are some buffer-specific things. */
- MARK_INTERVAL_TREE (buffer_intervals (buffer));
+ mark_interval_tree (buffer_intervals (buffer));
/* For now, we just don't mark the undo_list. It's done later in
a special way just before the sweep phase, and after stripping
@@ -6130,7 +6419,8 @@ mark_buffer (struct buffer *buffer)
mark_overlay (buffer->overlays_after);
/* If this is an indirect buffer, mark its base buffer. */
- if (buffer->base_buffer && !VECTOR_MARKED_P (buffer->base_buffer))
+ if (buffer->base_buffer &&
+ !vectorlike_marked_p (&buffer->base_buffer->header))
mark_buffer (buffer->base_buffer);
}
@@ -6149,8 +6439,8 @@ mark_face_cache (struct face_cache *c)
if (face)
{
- if (face->font && !VECTOR_MARKED_P (face->font))
- mark_vectorlike ((struct Lisp_Vector *) face->font);
+ if (face->font && !vectorlike_marked_p (&face->font->header))
+ mark_vectorlike (&face->font->header);
for (j = 0; j < LFACE_VECTOR_SIZE; ++j)
mark_object (face->lface[j]);
@@ -6209,7 +6499,7 @@ mark_discard_killed_buffers (Lisp_Object list)
{
Lisp_Object tail, *prev = &list;
- for (tail = list; CONSP (tail) && !CONS_MARKED_P (XCONS (tail));
+ for (tail = list; CONSP (tail) && !cons_marked_p (XCONS (tail));
tail = XCDR (tail))
{
Lisp_Object tem = XCAR (tail);
@@ -6219,7 +6509,7 @@ mark_discard_killed_buffers (Lisp_Object list)
*prev = XCDR (tail);
else
{
- CONS_MARK (XCONS (tail));
+ set_cons_marked (XCONS (tail));
mark_object (XCAR (tail));
prev = xcdr_addr (tail);
}
@@ -6228,6 +6518,79 @@ mark_discard_killed_buffers (Lisp_Object list)
return list;
}
+static void
+mark_frame (struct Lisp_Vector *ptr)
+{
+ struct frame *f = (struct frame *) ptr;
+ mark_vectorlike (&ptr->header);
+ mark_face_cache (f->face_cache);
+#ifdef HAVE_WINDOW_SYSTEM
+ if (FRAME_WINDOW_P (f) && FRAME_X_OUTPUT (f))
+ {
+ struct font *font = FRAME_FONT (f);
+
+ if (font && !vectorlike_marked_p (&font->header))
+ mark_vectorlike (&font->header);
+ }
+#endif
+}
+
+static void
+mark_window (struct Lisp_Vector *ptr)
+{
+ struct window *w = (struct window *) ptr;
+
+ mark_vectorlike (&ptr->header);
+
+ /* Mark glyph matrices, if any. Marking window
+ matrices is sufficient because frame matrices
+ use the same glyph memory. */
+ if (w->current_matrix)
+ {
+ mark_glyph_matrix (w->current_matrix);
+ mark_glyph_matrix (w->desired_matrix);
+ }
+
+ /* Filter out killed buffers from both buffer lists
+ in attempt to help GC to reclaim killed buffers faster.
+ We can do it elsewhere for live windows, but this is the
+ best place to do it for dead windows. */
+ wset_prev_buffers
+ (w, mark_discard_killed_buffers (w->prev_buffers));
+ wset_next_buffers
+ (w, mark_discard_killed_buffers (w->next_buffers));
+}
+
+static void
+mark_hash_table (struct Lisp_Vector *ptr)
+{
+ struct Lisp_Hash_Table *h = (struct Lisp_Hash_Table *) ptr;
+
+ mark_vectorlike (&h->header);
+ mark_object (h->test.name);
+ mark_object (h->test.user_hash_function);
+ mark_object (h->test.user_cmp_function);
+ /* If hash table is not weak, mark all keys and values. For weak
+ tables, mark only the vector and not its contents --- that's what
+ makes it weak. */
+ if (NILP (h->weak))
+ mark_object (h->key_and_value);
+ else
+ set_vector_marked (XVECTOR (h->key_and_value));
+}
+
+static enum pvec_type
+find_pvec_type (const struct Lisp_Vector *ptr)
+{
+ enum pvec_type pvectype;
+ if (ptr->header.size & PSEUDOVECTOR_FLAG)
+ pvectype = ((ptr->header.size & PVEC_TYPE_MASK)
+ >> PSEUDOVECTOR_AREA_BITS);
+ else
+ pvectype = PVEC_NORMAL_VECTOR;
+ return pvectype;
+}
+
/* Determine type of generic Lisp_Object and mark it accordingly.
This function implements a straightforward depth-first marking
@@ -6266,6 +6629,12 @@ mark_object (Lisp_Object arg)
structure allocated from the heap. */
#define CHECK_ALLOCATED() \
do { \
+ if (pdumper_object_p(po)) \
+ { \
+ if (!pdumper_object_p_precise (po)) \
+ emacs_abort (); \
+ break; \
+ } \
m = mem_find (po); \
if (m == MEM_NIL) \
emacs_abort (); \
@@ -6275,6 +6644,8 @@ mark_object (Lisp_Object arg)
function LIVEP. */
#define CHECK_LIVE(LIVEP) \
do { \
+ if (pdumper_object_p(po)) \
+ break; \
if (!LIVEP (m, po)) \
emacs_abort (); \
} while (0)
@@ -6309,11 +6680,11 @@ mark_object (Lisp_Object arg)
case Lisp_String:
{
register struct Lisp_String *ptr = XSTRING (obj);
- if (STRING_MARKED_P (ptr))
- break;
+ if (string_marked_p (ptr))
+ break;
CHECK_ALLOCATED_AND_LIVE (live_string_p);
- MARK_STRING (ptr);
- MARK_INTERVAL_TREE (ptr->intervals);
+ set_string_marked (ptr);
+ mark_interval_tree (ptr->intervals);
#ifdef GC_CHECK_STRING_BYTES
/* Check that the string size recorded in the string is the
same as the one recorded in the sdata structure. */
@@ -6327,21 +6698,19 @@ mark_object (Lisp_Object arg)
register struct Lisp_Vector *ptr = XVECTOR (obj);
register ptrdiff_t pvectype;
- if (VECTOR_MARKED_P (ptr))
+ if (vector_marked_p (ptr))
break;
#ifdef GC_CHECK_MARKED_OBJECTS
- m = mem_find (po);
- if (m == MEM_NIL && !SUBRP (obj))
- emacs_abort ();
+ if (!pdumper_object_p(po))
+ {
+ m = mem_find (po);
+ if (m == MEM_NIL && !SUBRP (obj))
+ emacs_abort ();
+ }
#endif /* GC_CHECK_MARKED_OBJECTS */
- if (ptr->header.size & PSEUDOVECTOR_FLAG)
- pvectype = ((ptr->header.size & PVEC_TYPE_MASK)
- >> PSEUDOVECTOR_AREA_BITS);
- else
- pvectype = PVEC_NORMAL_VECTOR;
-
+ pvectype = find_pvec_type (ptr);
if (pvectype != PVEC_SUBR && pvectype != PVEC_BUFFER)
CHECK_LIVE (live_vector_p);
@@ -6359,77 +6728,28 @@ mark_object (Lisp_Object arg)
}
#endif /* GC_CHECK_MARKED_OBJECTS */
mark_buffer ((struct buffer *) ptr);
- break;
-
- case PVEC_COMPILED:
- /* Although we could treat this just like a vector, mark_compiled
- returns the COMPILED_CONSTANTS element, which is marked at the
- next iteration of goto-loop here. This is done to avoid a few
- recursive calls to mark_object. */
- obj = mark_compiled (ptr);
- if (!NILP (obj))
- goto loop;
- break;
-
- case PVEC_FRAME:
- {
- struct frame *f = (struct frame *) ptr;
-
- mark_vectorlike (ptr);
- mark_face_cache (f->face_cache);
-#ifdef HAVE_WINDOW_SYSTEM
- if (FRAME_WINDOW_P (f) && FRAME_X_OUTPUT (f))
- {
- struct font *font = FRAME_FONT (f);
-
- if (font && !VECTOR_MARKED_P (font))
- mark_vectorlike ((struct Lisp_Vector *) font);
- }
-#endif
- }
- break;
-
- case PVEC_WINDOW:
- {
- struct window *w = (struct window *) ptr;
-
- mark_vectorlike (ptr);
-
- /* Mark glyph matrices, if any. Marking window
- matrices is sufficient because frame matrices
- use the same glyph memory. */
- if (w->current_matrix)
- {
- mark_glyph_matrix (w->current_matrix);
- mark_glyph_matrix (w->desired_matrix);
- }
-
- /* Filter out killed buffers from both buffer lists
- in attempt to help GC to reclaim killed buffers faster.
- We can do it elsewhere for live windows, but this is the
- best place to do it for dead windows. */
- wset_prev_buffers
- (w, mark_discard_killed_buffers (w->prev_buffers));
- wset_next_buffers
- (w, mark_discard_killed_buffers (w->next_buffers));
- }
- break;
+ break;
+
+ case PVEC_COMPILED:
+ /* Although we could treat this just like a vector, mark_compiled
+ returns the COMPILED_CONSTANTS element, which is marked at the
+ next iteration of goto-loop here. This is done to avoid a few
+ recursive calls to mark_object. */
+ obj = mark_compiled (ptr);
+ if (!NILP (obj))
+ goto loop;
+ break;
+
+ case PVEC_FRAME:
+ mark_frame (ptr);
+ break;
+
+ case PVEC_WINDOW:
+ mark_window (ptr);
+ break;
case PVEC_HASH_TABLE:
- {
- struct Lisp_Hash_Table *h = (struct Lisp_Hash_Table *) ptr;
-
- mark_vectorlike (ptr);
- mark_object (h->test.name);
- mark_object (h->test.user_hash_function);
- mark_object (h->test.user_cmp_function);
- /* If hash table is not weak, mark all keys and values.
- For weak tables, mark only the vector. */
- if (NILP (h->weak))
- mark_object (h->key_and_value);
- else
- VECTOR_MARK (XVECTOR (h->key_and_value));
- }
+ mark_hash_table (ptr);
break;
case PVEC_CHAR_TABLE:
@@ -6438,8 +6758,10 @@ mark_object (Lisp_Object arg)
break;
case PVEC_BOOL_VECTOR:
- /* No Lisp_Objects to mark in a bool vector. */
- VECTOR_MARK (ptr);
+ /* Do not mark bool vectors in a dump image: these objects
+ are "cold" and don't have mark bits. */
+ if (!pdumper_object_p (ptr))
+ set_vector_marked (ptr);
break;
case PVEC_SUBR:
@@ -6449,7 +6771,7 @@ mark_object (Lisp_Object arg)
emacs_abort ();
default:
- mark_vectorlike (ptr);
+ mark_vectorlike (&ptr->header);
}
}
break;
@@ -6458,10 +6780,10 @@ mark_object (Lisp_Object arg)
{
register struct Lisp_Symbol *ptr = XSYMBOL (obj);
nextsym:
- if (ptr->gcmarkbit)
- break;
- CHECK_ALLOCATED_AND_LIVE_SYMBOL ();
- ptr->gcmarkbit = 1;
+ if (symbol_marked_p (ptr))
+ break;
+ CHECK_ALLOCATED_AND_LIVE_SYMBOL ();
+ set_symbol_marked(ptr);
/* Attempt to catch bogus objects. */
eassert (valid_lisp_object_p (ptr->function));
mark_object (ptr->function);
@@ -6488,8 +6810,8 @@ mark_object (Lisp_Object arg)
default: emacs_abort ();
}
if (!PURE_P (XSTRING (ptr->name)))
- MARK_STRING (XSTRING (ptr->name));
- MARK_INTERVAL_TREE (string_intervals (ptr->name));
+ set_string_marked (XSTRING (ptr->name));
+ mark_interval_tree (string_intervals (ptr->name));
/* Inner loop to mark next symbol in this bucket, if any. */
po = ptr = ptr->next;
if (ptr)
@@ -6500,8 +6822,8 @@ mark_object (Lisp_Object arg)
case Lisp_Misc:
CHECK_ALLOCATED_AND_LIVE (live_misc_p);
- if (XMISCANY (obj)->gcmarkbit)
- break;
+ if (misc_any_marked_p (XMISCANY (obj)))
+ break;
switch (XMISCTYPE (obj))
{
@@ -6509,11 +6831,11 @@ mark_object (Lisp_Object arg)
/* DO NOT mark thru the marker's chain.
The buffer's markers chain does not preserve markers from gc;
instead, markers are removed from the chain when freed by gc. */
- XMISCANY (obj)->gcmarkbit = 1;
+ set_misc_any_marked (XMISCANY (obj));
break;
case Lisp_Misc_Save_Value:
- XMISCANY (obj)->gcmarkbit = 1;
+ set_misc_any_marked (XMISCANY (obj));
mark_save_value (XSAVE_VALUE (obj));
break;
@@ -6522,46 +6844,51 @@ mark_object (Lisp_Object arg)
break;
case Lisp_Misc_Finalizer:
- XMISCANY (obj)->gcmarkbit = true;
+ set_misc_any_marked (XMISCANY (obj));
mark_object (XFINALIZER (obj)->function);
break;
#ifdef HAVE_MODULES
case Lisp_Misc_User_Ptr:
- XMISCANY (obj)->gcmarkbit = true;
+ set_misc_any_marked (XMISCANY (obj));
break;
#endif
default:
emacs_abort ();
}
+
break;
case Lisp_Cons:
{
register struct Lisp_Cons *ptr = XCONS (obj);
- if (CONS_MARKED_P (ptr))
- break;
- CHECK_ALLOCATED_AND_LIVE (live_cons_p);
- CONS_MARK (ptr);
- /* If the cdr is nil, avoid recursion for the car. */
- if (EQ (ptr->u.cdr, Qnil))
- {
- obj = ptr->car;
- cdr_count = 0;
- goto loop;
- }
- mark_object (ptr->car);
- obj = ptr->u.cdr;
- cdr_count++;
- if (cdr_count == mark_object_loop_halt)
- emacs_abort ();
- goto loop;
+ if (cons_marked_p (ptr))
+ break;
+ CHECK_ALLOCATED_AND_LIVE (live_cons_p);
+ set_cons_marked (ptr);
+ /* If the cdr is nil, avoid recursion for the car. */
+ if (EQ (ptr->u.cdr, Qnil))
+ {
+ obj = ptr->car;
+ cdr_count = 0;
+ goto loop;
+ }
+ mark_object (ptr->car);
+ obj = ptr->u.cdr;
+ cdr_count++;
+ if (cdr_count == mark_object_loop_halt)
+ emacs_abort ();
+ goto loop;
}
case Lisp_Float:
CHECK_ALLOCATED_AND_LIVE (live_float_p);
- FLOAT_MARK (XFLOAT (obj));
+ /* Do not mark floats stored in a dump image: these floats are
+ "cold" and do not have mark bits. */
+ if (!pdumper_object_p (XFLOAT (obj)) &&
+ !XFLOAT_MARKED_P (XFLOAT (obj)))
+ XFLOAT_MARK (XFLOAT (obj));
break;
case_Lisp_Int:
@@ -6575,6 +6902,7 @@ mark_object (Lisp_Object arg)
#undef CHECK_ALLOCATED
#undef CHECK_ALLOCATED_AND_LIVE
}
+
/* Mark the Lisp pointers in the terminal objects.
Called by Fgarbage_collect. */
@@ -6591,13 +6919,11 @@ mark_terminals (void)
gets marked. */
mark_image_cache (t->image_cache);
#endif /* HAVE_WINDOW_SYSTEM */
- if (!VECTOR_MARKED_P (t))
- mark_vectorlike ((struct Lisp_Vector *)t);
+ if (!vectorlike_marked_p (&t->header))
+ mark_vectorlike (&t->header);
}
}
-
-
/* Value is non-zero if OBJ will survive the current GC because it's
either marked or does not need to be marked to survive. */
@@ -6613,27 +6939,29 @@ survives_gc_p (Lisp_Object obj)
break;
case Lisp_Symbol:
- survives_p = XSYMBOL (obj)->gcmarkbit;
+ survives_p = symbol_marked_p (XSYMBOL (obj));
break;
case Lisp_Misc:
- survives_p = XMISCANY (obj)->gcmarkbit;
+ survives_p = misc_any_marked_p (XMISCANY (obj));
break;
case Lisp_String:
- survives_p = STRING_MARKED_P (XSTRING (obj));
+ survives_p = string_marked_p (XSTRING (obj));
break;
case Lisp_Vectorlike:
- survives_p = SUBRP (obj) || VECTOR_MARKED_P (XVECTOR (obj));
+ survives_p = SUBRP (obj) || vector_marked_p (XVECTOR (obj));
break;
case Lisp_Cons:
- survives_p = CONS_MARKED_P (XCONS (obj));
+ survives_p = cons_marked_p (XCONS (obj));
break;
case Lisp_Float:
- survives_p = FLOAT_MARKED_P (XFLOAT (obj));
+ survives_p =
+ XFLOAT_MARKED_P (XFLOAT (obj)) ||
+ pdumper_object_p (XFLOAT (obj));
break;
default:
@@ -6686,7 +7014,7 @@ sweep_conses (void)
for (pos = start; pos < stop; pos++)
{
- if (!CONS_MARKED_P (&cblk->conses[pos]))
+ if (!XCONS_MARKED_P (&cblk->conses[pos]))
{
this_free++;
cblk->conses[pos].u.chain = cons_free_list;
@@ -6696,7 +7024,7 @@ sweep_conses (void)
else
{
num_used++;
- CONS_UNMARK (&cblk->conses[pos]);
+ XUNMARK_CONS (&cblk->conses[pos]);
}
}
}
@@ -6739,7 +7067,7 @@ sweep_floats (void)
register int i;
int this_free = 0;
for (i = 0; i < lim; i++)
- if (!FLOAT_MARKED_P (&fblk->floats[i]))
+ if (!XFLOAT_MARKED_P (&fblk->floats[i]))
{
this_free++;
fblk->floats[i].u.chain = float_free_list;
@@ -6748,7 +7076,7 @@ sweep_floats (void)
else
{
num_used++;
- FLOAT_UNMARK (&fblk->floats[i]);
+ XFLOAT_UNMARK (&fblk->floats[i]);
}
lim = FLOAT_BLOCK_SIZE;
/* If this block contains only free floats and we have already
@@ -6962,14 +7290,15 @@ sweep_buffers (void)
total_buffers = 0;
for (buffer = all_buffers; buffer; buffer = *bprev)
- if (!VECTOR_MARKED_P (buffer))
+ if (!vectorlike_marked_p (&buffer->header))
{
*bprev = buffer->next;
lisp_free (buffer);
}
else
{
- VECTOR_UNMARK (buffer);
+ if (!pdumper_object_p (buffer))
+ XUNMARK_VECTOR (buffer);
/* Do not use buffer_(set|get)_intervals here. */
buffer->text->intervals = balance_intervals (buffer->text->intervals);
total_buffers++;
@@ -6983,7 +7312,7 @@ gc_sweep (void)
{
/* Remove or mark entries in weak hash tables.
This must be done before any object is unmarked. */
- sweep_weak_hash_tables ();
+ mark_and_sweep_weak_hash_tables ();
sweep_strings ();
check_string_bytes (!noninteractive);
@@ -6994,6 +7323,7 @@ gc_sweep (void)
sweep_misc ();
sweep_buffers ();
sweep_vectors ();
+ pdumper_clear_marks ();
check_string_bytes (!noninteractive);
}
@@ -7267,19 +7597,31 @@ verify_alloca (void)
/* Initialization. */
+static void init_alloc_once_for_pdumper (void);
+
void
init_alloc_once (void)
{
+ gc_cons_threshold = GC_DEFAULT_THRESHOLD;
/* Even though Qt's contents are not set up, its address is known. */
Vpurify_flag = Qt;
- purebeg = PUREBEG;
- pure_size = PURESIZE;
+ /* Call init_alloc_once_for_pdumper now so we run mem_init early.
+ Keep in mind that when we reload from a dump, we'll run _only_
+ init_alloc_once_for_pdumper and not init_alloc_once at all. */
+ pdumper_do_now_and_after_load (init_alloc_once_for_pdumper);
verify_alloca ();
- init_finalizer_list (&finalizers);
- init_finalizer_list (&doomed_finalizers);
+ init_strings ();
+ init_vectors ();
+}
+
+static void
+init_alloc_once_for_pdumper (void)
+{
+ purebeg = PUREBEG;
+ pure_size = PURESIZE;
mem_init ();
Vdead = make_pure_string ("DEAD", 4, 4, 0);
@@ -7288,11 +7630,11 @@ init_alloc_once (void)
mallopt (M_MMAP_THRESHOLD, 64 * 1024); /* Mmap threshold. */
mallopt (M_MMAP_MAX, MMAP_MAX_AREAS); /* Max. number of mmap'ed areas. */
#endif
- init_strings ();
- init_vectors ();
+
+ init_finalizer_list (&finalizers);
+ init_finalizer_list (&doomed_finalizers);
refill_memory_reserve ();
- gc_cons_threshold = GC_DEFAULT_THRESHOLD;
}
void
diff --git a/src/atimer.c b/src/atimer.c
index 64c9e8a..258c61f 100644
--- a/src/atimer.c
+++ b/src/atimer.c
@@ -584,6 +584,7 @@ init_atimer (void)
sigaction (SIGALRM, &action, 0);
#ifdef ENABLE_CHECKING
- defsubr (&Sdebug_timer_check);
+ if (!initialized)
+ defsubr (&Sdebug_timer_check);
#endif
}
diff --git a/src/buffer.c b/src/buffer.c
index aa556b7..58fd64f 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -43,6 +43,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "keymap.h"
#include "frame.h"
#include "xwidget.h"
+#include "pdumper.h"
#ifdef WINDOWSNT
#include "w32heap.h" /* for mmap_* */
@@ -524,6 +525,8 @@ even if it is dead. The return value is never nil. */)
/* No one shows us now. */
b->window_count = 0;
+ memset (&b->local_flags, 0, sizeof (b->local_flags));
+
BUF_GAP_SIZE (b) = 20;
block_input ();
/* We allocate extra 1-byte at the tail and keep it always '\0' for
@@ -776,6 +779,8 @@ CLONE nil means the indirect buffer's state is reset to default values. */)
/* Always -1 for an indirect buffer. */
b->window_count = -1;
+ memset (&b->local_flags, 0, sizeof (b->local_flags));
+
b->pt = b->base_buffer->pt;
b->begv = b->base_buffer->begv;
b->zv = b->base_buffer->zv;
@@ -4959,24 +4964,37 @@ alloc_buffer_text (struct buffer *b, ptrdiff_t nbytes)
void
enlarge_buffer_text (struct buffer *b, ptrdiff_t delta)
{
- void *p;
- ptrdiff_t nbytes = (BUF_Z_BYTE (b) - BUF_BEG_BYTE (b) + BUF_GAP_SIZE (b) + 1
- + delta);
block_input ();
+ void *p;
+ unsigned char *old_beg = b->text->beg;
+ ptrdiff_t old_nbytes =
+ BUF_Z_BYTE (b) - BUF_BEG_BYTE (b) + BUF_GAP_SIZE (b) + 1;
+ ptrdiff_t new_nbytes = old_nbytes + delta;
+
+ if (pdumper_object_p (old_beg))
+ b->text->beg = NULL;
+ else
+ old_beg = NULL;
+
#if defined USE_MMAP_FOR_BUFFERS
- p = mmap_realloc ((void **) &b->text->beg, nbytes);
+ p = mmap_realloc ((void **) &b->text->beg, new_nbytes);
#elif defined REL_ALLOC
- p = r_re_alloc ((void **) &b->text->beg, nbytes);
+ p = r_re_alloc ((void **) &b->text->beg, new_nbytes);
#else
- p = xrealloc (b->text->beg, nbytes);
+ p = xrealloc (b->text->beg, new_nbytes);
#endif
if (p == NULL)
{
+ if (old_beg)
+ b->text->beg = old_beg;
unblock_input ();
- memory_full (nbytes);
+ memory_full (new_nbytes);
}
+ if (old_beg)
+ memcpy (p, old_beg, old_nbytes);
+
BUF_BEG_ADDR (b) = p;
unblock_input ();
}
@@ -4989,13 +5007,16 @@ free_buffer_text (struct buffer *b)
{
block_input ();
+ if (!pdumper_object_p (b->text->beg))
+ {
#if defined USE_MMAP_FOR_BUFFERS
- mmap_free ((void **) &b->text->beg);
+ mmap_free ((void **) &b->text->beg);
#elif defined REL_ALLOC
- r_alloc_free ((void **) &b->text->beg);
+ r_alloc_free ((void **) &b->text->beg);
#else
- xfree (b->text->beg);
+ xfree (b->text->beg);
#endif
+ }
BUF_BEG_ADDR (b) = NULL;
unblock_input ();
@@ -5006,12 +5027,23 @@ free_buffer_text (struct buffer *b)
/***********************************************************************
Initialization
***********************************************************************/
-
void
init_buffer_once (void)
{
+ /* TODO: clean up the buffer-local machinery. Right now,
+ we have:
+
+ buffer_defaults: default values of buffer-locals
+ buffer_local_flags: metadata
+ buffer_permanent_local_flags: metadata
+ buffer_local_symbols: metadata
+
+ There must be a simpler way to store the metadata.
+ */
+
int idx;
+ PDUMPER_REMEMBER_SCALAR (buffer_permanent_local_flags);
memset (buffer_permanent_local_flags, 0, sizeof buffer_permanent_local_flags);
/* 0 means not a lisp var, -1 means always local, else mask. */
@@ -5096,13 +5128,20 @@ init_buffer_once (void)
XSETFASTINT (BVAR (&buffer_local_flags, extra_line_spacing), idx); ++idx;
XSETFASTINT (BVAR (&buffer_local_flags, cursor_in_non_selected_windows), idx); ++idx;
+ /* buffer_local_flags contains no pointers, so it's safe to treat it
+ as a blob for pdumper. */
+ PDUMPER_REMEMBER_SCALAR (buffer_local_flags);
+
/* Need more room? */
if (idx >= MAX_PER_BUFFER_VARS)
emacs_abort ();
last_per_buffer_idx = idx;
+ PDUMPER_REMEMBER_SCALAR (last_per_buffer_idx);
/* Make sure all markable slots in buffer_defaults
are initialized reasonably, so mark_buffer won't choke. */
+ /* TODO: XXX: we no longer call mark_buffer on buffer_defaults, so
+ this work should be unnecessary. */
reset_buffer (&buffer_defaults);
eassert (NILP (BVAR (&buffer_defaults, name)));
reset_buffer_local_variables (&buffer_defaults, 1);
@@ -5192,7 +5231,9 @@ init_buffer_once (void)
Vbuffer_alist = Qnil;
current_buffer = 0;
+ pdumper_remember_lv_raw_ptr (¤t_buffer, Lisp_Vectorlike);
all_buffers = 0;
+ pdumper_remember_lv_raw_ptr (&all_buffers, Lisp_Vectorlike);
QSFundamental = build_pure_c_string ("Fundamental");
@@ -5216,14 +5257,14 @@ init_buffer_once (void)
}
void
-init_buffer (int initialized)
+init_buffer (void)
{
char *pwd;
Lisp_Object temp;
ptrdiff_t len;
#ifdef USE_MMAP_FOR_BUFFERS
- if (initialized)
+ if (dumped_with_unexec)
{
struct buffer *b;
@@ -5264,9 +5305,6 @@ init_buffer (int initialized)
eassert (b->text->beg != NULL);
}
}
-#else /* not USE_MMAP_FOR_BUFFERS */
- /* Avoid compiler warnings. */
- (void) initialized;
#endif /* USE_MMAP_FOR_BUFFERS */
AUTO_STRING (scratch, "*scratch*");
diff --git a/src/callint.c b/src/callint.c
index c0afc7b..6baead9 100644
--- a/src/callint.c
+++ b/src/callint.c
@@ -883,7 +883,8 @@ syms_of_callint (void)
intern_c_string ("region-beginning"),
intern_c_string ("region-end"),
intern_c_string ("point"),
- intern_c_string ("mark"));
+ intern_c_string ("mark"));
+ staticpro (&preserved_fns);
DEFSYM (Qlist, "list");
DEFSYM (Qlet, "let");
diff --git a/src/callproc.c b/src/callproc.c
index dc3ca4a..0db4dc7 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -1577,9 +1577,7 @@ init_callproc (void)
}
}
-#ifndef CANNOT_DUMP
- if (initialized)
-#endif
+ if (!will_dump)
{
tempdir = Fdirectory_file_name (Vexec_directory);
if (! file_accessible_directory_p (tempdir))
diff --git a/src/category.c b/src/category.c
index 8315797..3522ce4 100644
--- a/src/category.c
+++ b/src/category.c
@@ -42,15 +42,6 @@ bset_category_table (struct buffer *b, Lisp_Object val)
b->category_table_ = val;
}
-/* The version number of the latest category table. Each category
- table has a unique version number. It is assigned a new number
- also when it is modified. When a regular expression is compiled
- into the struct re_pattern_buffer, the version number of the
- category table (of the current buffer) at that moment is also
- embedded in the structure.
-
- For the moment, we are not using this feature. */
-static int category_table_version;
\f
/* Category set staff. */
@@ -514,6 +505,4 @@ See the documentation of the variable `word-combining-categories'. */);
defsubr (&Schar_category_set);
defsubr (&Scategory_set_mnemonics);
defsubr (&Smodify_category_entry);
-
- category_table_version = 0;
}
diff --git a/src/charset.c b/src/charset.c
index ff937bc..8a339c2 100644
--- a/src/charset.c
+++ b/src/charset.c
@@ -40,6 +40,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "charset.h"
#include "coding.h"
#include "buffer.h"
+#include "pdumper.h"
/*** GENERAL NOTES on CODED CHARACTER SETS (CHARSETS) ***
@@ -62,9 +63,8 @@ Lisp_Object Vcharset_hash_table;
/* Table of struct charset. */
struct charset *charset_table;
-
-static ptrdiff_t charset_table_size;
-static int charset_table_used;
+ptrdiff_t charset_table_size;
+int charset_table_used;
/* Special charsets corresponding to symbols. */
int charset_ascii;
@@ -843,6 +843,8 @@ usage: (define-charset-internal ...) */)
bool new_definition_p;
int nchars;
+ memset (&charset, 0, sizeof (charset));
+
if (nargs != charset_arg_max)
Fsignal (Qwrong_number_of_arguments,
Fcons (intern ("define-charset-internal"),
@@ -1134,9 +1136,9 @@ usage: (define-charset-internal ...) */)
struct charset *new_table =
xpalloc (0, &new_size, 1,
min (INT_MAX, MOST_POSITIVE_FIXNUM),
- sizeof *charset_table);
- memcpy (new_table, charset_table, old_size * sizeof *new_table);
- charset_table = new_table;
+ sizeof *charset_table);
+ memcpy (new_table, charset_table, old_size * sizeof *new_table);
+ charset_table = new_table;
charset_table_size = new_size;
/* FIXME: This leaks memory, as the old charset_table becomes
unreachable. If the old charset table is charset_table_init
@@ -2303,15 +2305,26 @@ init_charset_once (void)
for (i = 0; i < ISO_MAX_DIMENSION; i++)
for (j = 0; j < ISO_MAX_CHARS; j++)
for (k = 0; k < ISO_MAX_FINAL; k++)
- iso_charset_table[i][j][k] = -1;
+ iso_charset_table[i][j][k] = -1;
+
+ PDUMPER_REMEMBER_SCALAR (iso_charset_table);
for (i = 0; i < 256; i++)
emacs_mule_charset[i] = -1;
+ PDUMPER_REMEMBER_SCALAR (emacs_mule_charset);
+
charset_jisx0201_roman = -1;
+ PDUMPER_REMEMBER_SCALAR (charset_jisx0201_roman);
+
charset_jisx0208_1978 = -1;
+ PDUMPER_REMEMBER_SCALAR (charset_jisx0208_1978);
+
charset_jisx0208 = -1;
+ PDUMPER_REMEMBER_SCALAR (charset_jisx0208);
+
charset_ksc5601 = -1;
+ PDUMPER_REMEMBER_SCALAR (charset_ksc5601);
}
#ifdef emacs
@@ -2400,21 +2413,32 @@ the value may be a list of mnemonics. */);
charset_ascii
= define_charset_internal (Qascii, 1, "\x00\x7F\0\0\0\0\0",
- 0, 127, 'B', -1, 0, 1, 0, 0);
+ 0, 127, 'B', -1, 0, 1, 0, 0);
+ PDUMPER_REMEMBER_SCALAR (charset_ascii);
+
charset_iso_8859_1
= define_charset_internal (Qiso_8859_1, 1, "\x00\xFF\0\0\0\0\0",
- 0, 255, -1, -1, -1, 1, 0, 0);
+ 0, 255, -1, -1, -1, 1, 0, 0);
+ PDUMPER_REMEMBER_SCALAR (charset_iso_8859_1);
+
charset_unicode
= define_charset_internal (Qunicode, 3, "\x00\xFF\x00\xFF\x00\x10\0",
- 0, MAX_UNICODE_CHAR, -1, 0, -1, 1, 0, 0);
+ 0, MAX_UNICODE_CHAR, -1, 0, -1, 1, 0, 0);
+ PDUMPER_REMEMBER_SCALAR (charset_unicode);
+
charset_emacs
= define_charset_internal (Qemacs, 3, "\x00\xFF\x00\xFF\x00\x3F\0",
- 0, MAX_5_BYTE_CHAR, -1, 0, -1, 1, 1, 0);
+ 0, MAX_5_BYTE_CHAR, -1, 0, -1, 1, 1, 0);
+ PDUMPER_REMEMBER_SCALAR (charset_emacs);
+
charset_eight_bit
= define_charset_internal (Qeight_bit, 1, "\x80\xFF\0\0\0\0\0",
128, 255, -1, 0, -1, 0, 1,
- MAX_5_BYTE_CHAR + 1);
+ MAX_5_BYTE_CHAR + 1);
+ PDUMPER_REMEMBER_SCALAR (charset_eight_bit);
+
charset_unibyte = charset_iso_8859_1;
+ PDUMPER_REMEMBER_SCALAR (charset_unibyte);
}
#endif /* emacs */
diff --git a/src/charset.h b/src/charset.h
index 8e77567..00c9b7e 100644
--- a/src/charset.h
+++ b/src/charset.h
@@ -248,6 +248,8 @@ extern Lisp_Object Vcharset_hash_table;
/* Table of struct charset. */
extern struct charset *charset_table;
+extern ptrdiff_t charset_table_size;
+extern int charset_table_used;
#define CHARSET_FROM_ID(id) (charset_table + (id))
diff --git a/src/coding.c b/src/coding.c
index feed9c8..4024bee 100644
--- a/src/coding.c
+++ b/src/coding.c
@@ -298,6 +298,7 @@ encode_coding_XXX (struct coding_system *coding)
#include "composite.h"
#include "coding.h"
#include "termhooks.h"
+#include "pdumper.h"
Lisp_Object Vcoding_system_hash_table;
@@ -10755,6 +10756,9 @@ init_coding_once (void)
coding_priorities[i] = i;
}
+ PDUMPER_REMEMBER_SCALAR (coding_categories);
+ PDUMPER_REMEMBER_SCALAR (coding_priorities);
+
/* ISO2022 specific initialize routine. */
for (i = 0; i < 0x20; i++)
iso_code_class[i] = ISO_control_0;
@@ -10774,6 +10778,8 @@ init_coding_once (void)
iso_code_class[ISO_CODE_SS3] = ISO_single_shift_3;
iso_code_class[ISO_CODE_CSI] = ISO_control_sequence_introducer;
+ PDUMPER_REMEMBER_SCALAR (iso_code_class);
+
for (i = 0; i < 256; i++)
{
emacs_mule_bytes[i] = 1;
@@ -10782,6 +10788,8 @@ init_coding_once (void)
emacs_mule_bytes[EMACS_MULE_LEADING_CODE_PRIVATE_12] = 3;
emacs_mule_bytes[EMACS_MULE_LEADING_CODE_PRIVATE_21] = 4;
emacs_mule_bytes[EMACS_MULE_LEADING_CODE_PRIVATE_22] = 4;
+
+ PDUMPER_REMEMBER_SCALAR (emacs_mule_bytes);
}
#ifdef emacs
@@ -10805,6 +10813,7 @@ syms_of_coding (void)
Vcode_conversion_workbuf_name = build_pure_c_string (" *code-conversion-work*");
reused_workbuf_in_use = 0;
+ PDUMPER_REMEMBER_SCALAR (reused_workbuf_in_use);
DEFSYM (Qcharset, "charset");
DEFSYM (Qtarget_idx, "target-idx");
diff --git a/src/conf_post.h b/src/conf_post.h
index 060b912..833beb0 100644
--- a/src/conf_post.h
+++ b/src/conf_post.h
@@ -265,8 +265,10 @@ extern int emacs_setenv_TZ (char const *);
#if 3 <= __GNUC__
# define ATTRIBUTE_MALLOC __attribute__ ((__malloc__))
+# define ATTRIBUTE_SECTION(name) __attribute__((section (name)))
#else
# define ATTRIBUTE_MALLOC
+#define ATTRIBUTE_SECTION(name)
#endif
#if __has_attribute (alloc_size)
diff --git a/src/data.c b/src/data.c
index d221db4..1de42b7 100644
--- a/src/data.c
+++ b/src/data.c
@@ -740,7 +740,7 @@ The return value is undefined. */)
{
bool autoload = AUTOLOADP (definition);
- if (NILP (Vpurify_flag) || !autoload)
+ if (!will_dump || !autoload)
{ /* Only add autoload entries after dumping, because the ones before are
not useful and else we get loads of them from the loaddefs.el. */
diff --git a/src/dbusbind.c b/src/dbusbind.c
index a0146a3..e7aabbf 100644
--- a/src/dbusbind.c
+++ b/src/dbusbind.c
@@ -1827,6 +1827,8 @@ be called when the D-Bus reply message arrives. */);
xd_registered_buses = Qnil;
staticpro (&xd_registered_buses);
+ // XXX: reset buses on dump load
+
Fprovide (intern_c_string ("dbusbind"), Qnil);
}
diff --git a/src/dispnew.c b/src/dispnew.c
index 70d4de0..b266b6e 100644
--- a/src/dispnew.c
+++ b/src/dispnew.c
@@ -41,6 +41,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "systime.h"
#include "tparam.h"
#include "xwidget.h"
+#include "pdumper.h"
#ifdef HAVE_WINDOW_SYSTEM
#include TERM_HEADER
@@ -5960,9 +5961,7 @@ init_display (void)
with. Otherwise newly opened tty frames will not resize
automatically. */
#ifdef SIGWINCH
-#ifndef CANNOT_DUMP
- if (initialized)
-#endif /* CANNOT_DUMP */
+ if (!will_dump)
{
struct sigaction action;
emacs_sigaction_init (&action, deliver_window_change_signal);
@@ -6026,11 +6025,7 @@ init_display (void)
#endif /* HAVE_NTGUI */
#ifdef HAVE_NS
- if (!inhibit_window_system
-#ifndef CANNOT_DUMP
- && initialized
-#endif
- )
+ if (!inhibit_window_system && !will_dump)
{
Vinitial_window_system = Qns;
Vwindow_system_version = make_number (10);
@@ -6118,10 +6113,8 @@ init_display (void)
calculate_costs (XFRAME (selected_frame));
- /* Set up faces of the initial terminal frame of a dumped Emacs. */
- if (initialized
- && !noninteractive
- && NILP (Vinitial_window_system))
+ /* Set up faces of the initial terminal frame. */
+ if (!noninteractive && NILP (Vinitial_window_system))
{
/* For the initial frame, we don't have any way of knowing what
are the foreground and background colors of the terminal. */
@@ -6168,6 +6161,8 @@ WINDOW nil or omitted means report on the selected window. */)
Initialization
***********************************************************************/
+static void syms_of_display_for_pdumper (void);
+
void
syms_of_display (void)
{
@@ -6275,11 +6270,12 @@ See `buffer-display-table' for more information. */);
beginning of the next redisplay). */
redisplay_dont_pause = true;
-#ifdef CANNOT_DUMP
- if (noninteractive)
-#endif
- {
- Vinitial_window_system = Qnil;
- Vwindow_system_version = Qnil;
- }
+ pdumper_do_now_and_after_load (syms_of_display_for_pdumper);
+}
+
+static void
+syms_of_display_for_pdumper (void)
+{
+ Vinitial_window_system = Qnil;
+ Vwindow_system_version = Qnil;
}
diff --git a/src/doc.c b/src/doc.c
index ce4f89b..c71e393 100644
--- a/src/doc.c
+++ b/src/doc.c
@@ -118,17 +118,15 @@ get_doc_string (Lisp_Object filepos, bool unibyte, bool definition)
Lisp_Object docdir
= NILP (tem) ? ENCODE_FILE (Vdoc_directory) : empty_unibyte_string;
ptrdiff_t docdir_sizemax = SBYTES (docdir) + 1;
-#ifndef CANNOT_DUMP
- docdir_sizemax = max (docdir_sizemax, sizeof sibling_etc);
-#endif
+ if (will_dump)
+ docdir_sizemax = max (docdir_sizemax, sizeof sibling_etc);
name = SAFE_ALLOCA (docdir_sizemax + SBYTES (file));
lispstpcpy (lispstpcpy (name, docdir), file);
fd = emacs_open (name, O_RDONLY, 0);
if (fd < 0)
{
-#ifndef CANNOT_DUMP
- if (!NILP (Vpurify_flag))
+ if (will_dump)
{
/* Preparing to dump; DOC file is probably not installed.
So check in ../etc. */
@@ -136,7 +134,6 @@ get_doc_string (Lisp_Object filepos, bool unibyte, bool definition)
fd = emacs_open (name, O_RDONLY, 0);
}
-#endif
if (fd < 0)
{
if (errno == EMFILE || errno == ENFILE)
@@ -544,12 +541,7 @@ the same file name is found in the `doc-directory'. */)
CHECK_STRING (filename);
- if
-#ifndef CANNOT_DUMP
- (!NILP (Vpurify_flag))
-#else /* CANNOT_DUMP */
- (0)
-#endif /* CANNOT_DUMP */
+ if (will_dump)
{
dirname = sibling_etc;
dirlen = sizeof sibling_etc - 1;
diff --git a/src/editfns.c b/src/editfns.c
index 72c7a9c..3e64473 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -228,7 +228,7 @@ tzlookup (Lisp_Object zone, bool settz)
}
void
-init_editfns (bool dumping)
+init_editfns (void)
{
const char *user_name;
register char *p;
@@ -238,10 +238,9 @@ init_editfns (bool dumping)
/* Set up system_name even when dumping. */
init_and_cache_system_name ();
-#ifndef CANNOT_DUMP
/* When just dumping out, set the time zone to a known unlikely value
and skip the rest of this function. */
- if (dumping)
+ if (will_dump_with_unexec)
{
# ifdef HAVE_TZSET
xputenv (dump_tz_string);
@@ -249,17 +248,17 @@ init_editfns (bool dumping)
# endif
return;
}
-#endif
char *tz = getenv ("TZ");
-#if !defined CANNOT_DUMP && defined HAVE_TZSET
+#if defined HAVE_TZSET
/* If the execution TZ happens to be the same as the dump TZ,
change it to some other value and then change it back,
to force the underlying implementation to reload the TZ info.
This is needed on implementations that load TZ info from files,
since the TZ file contents may differ between dump and execution. */
- if (tz && strcmp (tz, &dump_tz_string[tzeqlen]) == 0)
+ if (dumped_with_unexec &&
+ tz && strcmp (tz, &dump_tz_string[tzeqlen]) == 0)
{
++*tz;
tzset ();
@@ -1341,7 +1340,7 @@ of the user with that uid, or nil if there is no such user. */)
(That can happen if Emacs is dumpable
but you decide to run `temacs -l loadup' and not dump. */
if (NILP (Vuser_login_name))
- init_editfns (false);
+ init_editfns ();
if (NILP (uid))
return Vuser_login_name;
@@ -1364,7 +1363,7 @@ This ignores the environment variables LOGNAME and USER, so it differs from
(That can happen if Emacs is dumpable
but you decide to run `temacs -l loadup' and not dump. */
if (NILP (Vuser_login_name))
- init_editfns (false);
+ init_editfns ();
return Vuser_real_login_name;
}
diff --git a/src/emacs-module.c b/src/emacs-module.c
index 68aeb0c..ac84473 100644
--- a/src/emacs-module.c
+++ b/src/emacs-module.c
@@ -1010,7 +1010,10 @@ void
syms_of_module (void)
{
if (!plain_values)
- ltv_mark = Fcons (Qnil, Qnil);
+ {
+ ltv_mark = Fcons (Qnil, Qnil);
+ staticpro (<v_mark);
+ }
eassert (NILP (value_to_lisp (module_nil)));
DEFSYM (Qmodule_refs_hash, "module-refs-hash");
diff --git a/src/emacs.c b/src/emacs.c
index efd4fa3..0380b5f 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -113,6 +113,9 @@ extern void moncontrol (int mode);
#include <sys/resource.h>
#endif
+#include "pdumper.h"
+#include "epaths.h"
+
static const char emacs_version[] = PACKAGE_VERSION;
static const char emacs_copyright[] = COPYRIGHT;
static const char emacs_bugreport[] = PACKAGE_BUGREPORT;
@@ -125,16 +128,7 @@ Lisp_Object empty_unibyte_string, empty_multibyte_string;
Lisp_Object Vlibrary_cache;
#endif
-/* Set after Emacs has started up the first time.
- Prevents reinitialization of the Lisp world and keymaps
- on subsequent starts. */
-bool initialized;
-
-/* Set to true if this instance of Emacs might dump. */
-#ifndef DOUG_LEA_MALLOC
-static
-#endif
-bool might_dump;
+struct gflags gflags;
#ifdef DARWIN_OS
extern void unexec_init_emacs_zone (void);
@@ -509,8 +503,7 @@ init_cmdargs (int argc, char **argv, int skip_args, char *original_pwd)
etc_exists = Ffile_exists_p (tem);
if (!NILP (etc_exists))
{
- Vinstallation_directory
- = Ffile_name_as_directory (dir);
+ Vinstallation_directory = Ffile_name_as_directory (dir);
break;
}
}
@@ -535,8 +528,7 @@ init_cmdargs (int argc, char **argv, int skip_args, char *original_pwd)
if (!NILP (etc_exists))
{
tem = Fexpand_file_name (build_string (".."), dir);
- Vinstallation_directory
- = Ffile_name_as_directory (tem);
+ Vinstallation_directory = Ffile_name_as_directory (tem);
break;
}
}
@@ -664,6 +656,136 @@ close_output_streams (void)
_exit (EXIT_FAILURE);
}
+static bool
+string_starts_with_p(const char* string, const char* prefix)
+{
+ return strncmp(string, prefix, strlen(prefix)) == 0;
+}
+
+#define DUMP_FILE_ARGUMENT "--dump-file"
+
+static char *
+find_and_remove_dump_file_argument (int *inout_argc, char ***inout_argv)
+{
+ int argc = *inout_argc;
+ char **argv = *inout_argv;
+ char *found = NULL;
+ int i;
+
+ for (i = 1; i < argc; ++i)
+ if (string_starts_with_p (argv[i], DUMP_FILE_ARGUMENT) &&
+ ((argv[i] + strlen (DUMP_FILE_ARGUMENT))[0] == '=' ||
+ (argv[i] + strlen (DUMP_FILE_ARGUMENT))[0] == '\0'))
+ {
+ int j = i;
+ found = argv[j++] + strlen (DUMP_FILE_ARGUMENT);
+ if (*found == '=')
+ ++found;
+ else if (i < argc)
+ found = argv[j++];
+ else
+ {
+ fprintf (stderr, "%s: no argument given for %s\n",
+ argv[0], DUMP_FILE_ARGUMENT);
+ exit (1);
+ }
+
+ memmove (&argv[i], &argv[j], sizeof (*argv) * (argc - j));
+ argc -= (j - i);
+ argv[argc] = NULL;
+ break;
+ }
+ else if (strcmp (argv[i], "--") == 0)
+ break;
+
+ *inout_argc = argc;
+ *inout_argv = argv;
+ return found;
+}
+
+static const char *
+dump_error_to_string (enum pdumper_load_result result)
+{
+ switch (result)
+ {
+ case PDUMPER_LOAD_SUCCESS:
+ return "success";
+ case PDUMPER_LOAD_OOM:
+ return "out of memory";
+ case PDUMPER_NOT_LOADED:
+ return "not loaded";
+ case PDUMPER_LOAD_FILE_NOT_FOUND:
+ return "could not open file";
+ case PDUMPER_LOAD_BAD_FILE_TYPE:
+ return "not a dump file";
+ case PDUMPER_LOAD_VERSION_MISMATCH:
+ return "not built for this Emacs executable";
+ default:
+ return "generic error";
+ }
+}
+
+static const char *
+load_dump (int *inout_argc, char ***inout_argv, char *argv0_base)
+{
+ int argc = *inout_argc;
+ char **argv = *inout_argv;
+ const char *suffix = ".pdmp";
+ enum pdumper_load_result result;
+
+ char *dump_file = find_and_remove_dump_file_argument (&argc, &argv);
+ if (initialized && dump_file)
+ /* TODO: maybe more thoroughly scrub process environment in order
+ to make this use case possible? Right now, we assume that
+ things we don't touch are zero-initialized, and in an unexeced
+ Emacs, this assumption doesn't hold. */
+ fatal ("cannot load dump file into unexeced Emacs");
+
+ if (initialized)
+ return NULL;
+
+ result = PDUMPER_NOT_LOADED;
+ if (dump_file)
+ result = pdumper_load (dump_file);
+
+ if (dump_file && result != PDUMPER_LOAD_SUCCESS)
+ fatal ("could not load dump file \"%s\": %s",
+ dump_file, dump_error_to_string (result));
+
+ if (result == PDUMPER_LOAD_SUCCESS)
+ goto out;
+
+ dump_file = alloca (strlen (argv[0]) + strlen (suffix) + 1);
+ sprintf (dump_file, "%s%s", argv[0], suffix);
+
+ result = pdumper_load (dump_file);
+ if (result == PDUMPER_LOAD_SUCCESS)
+ goto out;
+
+ if (result != PDUMPER_LOAD_FILE_NOT_FOUND)
+ fatal ("could not load dump file \"%s\": %s",
+ dump_file, dump_error_to_string (result));
+
+ /* For searching PATH_EXEC, just use "emacs". Emacs shouldn't break
+ if its binary is renamed. */
+ argv0_base = "emacs";
+ dump_file = alloca (strlen (PATH_EXEC)
+ + 1
+ + strlen (argv0_base)
+ + strlen (suffix)
+ + 1);
+ sprintf (dump_file, "%s%c%s%s",
+ PATH_EXEC, DIRECTORY_SEP, argv0_base, suffix);
+ result = pdumper_load (dump_file);
+ if (result != PDUMPER_LOAD_SUCCESS)
+ dump_file = NULL;
+
+ out:
+ *inout_argc = argc;
+ *inout_argv = argv;
+ return dump_file ? strdup (dump_file) : NULL;
+}
+
/* ARGSUSED */
int
main (int argc, char **argv)
@@ -671,7 +793,6 @@ main (int argc, char **argv)
Lisp_Object dummy;
char stack_bottom_variable;
bool do_initial_setlocale;
- bool dumping;
int skip_args = 0;
bool no_loadup = false;
char *junk = 0;
@@ -686,14 +807,36 @@ main (int argc, char **argv)
stack_base = &dummy;
- dumping = !initialized && (strcmp (argv[argc - 1], "dump") == 0
- || strcmp (argv[argc - 1], "bootstrap") == 0);
+ /* Figure out where we are. Fancy filesystem functions aren't
+ available at this point, so use pure text manipulation. */
+ char *argv0_base = strrchr (argv[0], DIRECTORY_SEP);
+ argv0_base = argv0_base ? argv0_base + 1 : argv[0];
+ bool is_temacs = strcmp ("temacs", argv0_base) == 0;
+ const char *loaded_dump = NULL;
+
+ const char *dump_mode = NULL;
+ if (!initialized && is_temacs)
+ {
+#ifndef CANNOT_DUMP
+ if (strcmp (argv[argc - 1], "dump") == 0 ||
+ strcmp (argv[argc - 1], "bootstrap"))
+ gflags.will_dump_with_unexec_ = true;
+#endif
+ if (strcmp (argv[argc - 1], "pdump") == 0 ||
+ strcmp (argv[argc - 1], "pbootstrap"))
+ gflags.will_dump_with_pdumper_ = true;
+ gflags.will_dump_ = will_dump_with_pdumper || will_dump_with_unexec;
+ if (will_dump)
+ dump_mode = argv[argc - 1];
+ }
+ else if (!is_temacs)
+ loaded_dump = load_dump (&argc, &argv, argv0_base);
/* True if address randomization interferes with memory allocation. */
# ifdef __PPC64__
bool disable_aslr = true;
# else
- bool disable_aslr = dumping;
+ bool disable_aslr = will_dump_with_unexec;
# endif
if (disable_aslr && disable_address_randomization ())
@@ -710,10 +853,6 @@ main (int argc, char **argv)
perror (argv[0]);
}
-#ifndef CANNOT_DUMP
- might_dump = !initialized;
-#endif
-
#ifdef GNU_LINUX
if (!initialized)
{
@@ -841,10 +980,7 @@ main (int argc, char **argv)
/* Extra space to cover what we're likely to use for other reasons. */
int extra = 200000;
- bool try_to_grow_stack = true;
-#ifndef CANNOT_DUMP
- try_to_grow_stack = !noninteractive || initialized;
-#endif
+ bool try_to_grow_stack = !noninteractive || initialized;
if (try_to_grow_stack)
{
@@ -1173,17 +1309,15 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
#if defined HAVE_PTHREAD && !defined SYSTEM_MALLOC \
&& !defined DOUG_LEA_MALLOC && !defined HYBRID_MALLOC
-# ifndef CANNOT_DUMP
/* Do not make gmalloc thread-safe when creating bootstrap-emacs, as
that causes an infinite recursive loop with FreeBSD. See
Bug#14569. The part of this bug involving Cygwin is no longer
relevant, now that Cygwin defines HYBRID_MALLOC. */
- if (!noninteractive || initialized)
-# endif
+ if (!noninteractive || !will_dump)
malloc_enable_thread ();
#endif
- init_signals (dumping);
+ init_signals ();
noninteractive1 = noninteractive;
@@ -1192,7 +1326,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
if (!initialized)
{
init_alloc_once ();
- init_obarray ();
+ init_obarray_once ();
init_eval_once ();
init_charset_once ();
init_coding_once ();
@@ -1230,7 +1364,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
/* Before init_window_once, because it sets up the
Vcoding_system_hash_table. */
syms_of_coding (); /* This should be after syms_of_fileio. */
-
+ init_frame_once (); /* Before init_window_once. */
init_window_once (); /* Init the window system. */
#ifdef HAVE_WINDOW_SYSTEM
init_fringe_once (); /* Swap bitmaps if necessary. */
@@ -1389,7 +1523,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
/* egetenv is a pretty low-level facility, which may get called in
many circumstances; it seems flimsy to put off initializing it
until calling init_callproc. Do not do it when dumping. */
- if (! dumping)
+ if (!will_dump)
set_initial_environment ();
#ifdef WINDOWSNT
@@ -1403,7 +1537,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
variables from the parent process without modifications from
Emacs. */
init_environment (argv);
- init_ntproc (dumping); /* must precede init_editfns. */
+ init_ntproc (will_dump); /* must precede init_editfns. */
#endif
/* AIX crashes are reported in system versions 3.2.3 and 3.2.4
@@ -1415,7 +1549,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
#endif
/* Init buffer storage and default directory of main buffer. */
- init_buffer (initialized);
+ init_buffer ();
init_callproc_1 (); /* Must precede init_cmdargs and init_sys_modes. */
@@ -1581,6 +1715,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
#endif /* WINDOWSNT */
syms_of_profiler ();
+ syms_of_pdumper ();
keys_of_casefiddle ();
keys_of_cmds ();
@@ -1608,7 +1743,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
/* This calls putenv and so must precede init_process_emacs. Also,
it sets Voperating_system_release, which init_process_emacs uses. */
- init_editfns (dumping);
+ init_editfns ();
/* These two call putenv. */
#ifdef HAVE_DBUS
@@ -1679,7 +1814,11 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
#endif
#endif
- initialized = 1;
+ gflags.initialized_ = true;
+
+ Fset (intern ("dump-mode"), dump_mode ? build_string (dump_mode) : Qnil);
+ if (loaded_dump)
+ Vdump_file_name = build_string (loaded_dump); // XXX: filesystem decode
/* Enter editor command loop. This never returns. */
Frecursive_edit ();
@@ -2184,12 +2323,19 @@ You must run Emacs in batch mode in order to dump it. */)
#endif /* not WINDOWSNT */
#endif /* not SYSTEM_MALLOC and not HYBRID_MALLOC */
+ struct gflags old_gflags = gflags;
+ gflags.will_dump_ = false;
+ gflags.will_dump_with_unexec = false;
+ gflags.dumped_with_unexec_ = true;
+
alloc_unexec_pre ();
unexec (SSDATA (filename), !NILP (symfile) ? SSDATA (symfile) : 0);
alloc_unexec_post ();
+ gflags = old_gflags;
+
#ifdef WINDOWSNT
Vlibrary_cache = Qnil;
#endif
@@ -2203,6 +2349,7 @@ You must run Emacs in batch mode in order to dump it. */)
}
#endif /* not CANNOT_DUMP */
+
\f
#if HAVE_SETLOCALE
/* Recover from setlocale (LC_ALL, ""). */
@@ -2531,7 +2678,11 @@ Don't rely on it for testing whether a feature you want to use is available. */
Vsystem_configuration_features = build_string (EMACS_CONFIG_FEATURES);
DEFVAR_BOOL ("noninteractive", noninteractive1,
- doc: /* Non-nil means Emacs is running without interactive terminal. */);
+ doc: /* Non-nil means Emacs is running without interactive terminal. */);
+
+
+ DEFVAR_LISP ("dump-file-name", Vdump_file_name,
+ doc: /* Name of the dump file used to start this Emacs process. */);
DEFVAR_LISP ("kill-emacs-hook", Vkill_emacs_hook,
doc: /* Hook run when `kill-emacs' is called.
diff --git a/src/eval.c b/src/eval.c
index 884e1eb..d3e9435 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -29,6 +29,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "keyboard.h"
#include "dispextern.h"
#include "buffer.h"
+#include "pdumper.h"
/* Chain of condition and catch handlers currently in effect. */
@@ -78,10 +79,6 @@ static EMACS_INT when_entered_debugger;
/* FIXME: We should probably get rid of this! */
Lisp_Object Vsignaling_function;
-/* If non-nil, Lisp code must not be run since some part of Emacs is in
- an inconsistent state. Currently unused. */
-Lisp_Object inhibit_lisp_code;
-
/* These would ordinarily be static, but they need to be visible to GDB. */
bool backtrace_p (union specbinding *) EXTERNALLY_VISIBLE;
Lisp_Object *backtrace_args (union specbinding *) EXTERNALLY_VISIBLE;
@@ -204,18 +201,25 @@ near_C_stack_top (void)
return backtrace_args (backtrace_top ());
}
+static void init_eval_once_for_pdumper (void);
+
void
init_eval_once (void)
{
- enum { size = 50 };
- union specbinding *pdlvec = xmalloc ((size + 1) * sizeof *specpdl);
- specpdl_size = size;
- specpdl = specpdl_ptr = pdlvec + 1;
/* Don't forget to update docs (lispref node "Local Variables"). */
max_specpdl_size = 1300; /* 1000 is not enough for CEDET's c-by.el. */
max_lisp_eval_depth = 800;
-
Vrun_hooks = Qnil;
+ pdumper_do_now_and_after_load (init_eval_once_for_pdumper);
+}
+
+static void
+init_eval_once_for_pdumper (void)
+{
+ enum { size = 50 };
+ union specbinding *pdlvec = malloc ((size + 1) * sizeof *specpdl);
+ specpdl_size = size;
+ specpdl = specpdl_ptr = pdlvec + 1;
}
static struct handler handlerlist_sentinel;
@@ -1954,7 +1958,7 @@ it defines a macro. */)
/* This is to make sure that loadup.el gives a clear picture
of what files are preloaded and when. */
- if (! NILP (Vpurify_flag))
+ if (will_dump)
error ("Attempt to autoload %s while preparing to dump",
SDATA (SYMBOL_NAME (funname)));
@@ -3711,33 +3715,33 @@ mark_specpdl (void)
for (pdl = specpdl; pdl != specpdl_ptr; pdl++)
{
switch (pdl->kind)
- {
- case SPECPDL_UNWIND:
- mark_object (specpdl_arg (pdl));
- break;
-
- case SPECPDL_BACKTRACE:
- {
- ptrdiff_t nargs = backtrace_nargs (pdl);
- mark_object (backtrace_function (pdl));
- if (nargs == UNEVALLED)
- nargs = 1;
- while (nargs--)
- mark_object (backtrace_args (pdl)[nargs]);
- }
- break;
-
- case SPECPDL_LET_DEFAULT:
- case SPECPDL_LET_LOCAL:
- mark_object (specpdl_where (pdl));
- /* Fall through. */
- case SPECPDL_LET:
- mark_object (specpdl_symbol (pdl));
- mark_object (specpdl_old_value (pdl));
- break;
-
- case SPECPDL_UNWIND_PTR:
- case SPECPDL_UNWIND_INT:
+ {
+ case SPECPDL_UNWIND:
+ mark_object (specpdl_arg (pdl));
+ break;
+
+ case SPECPDL_BACKTRACE:
+ {
+ ptrdiff_t nargs = backtrace_nargs (pdl);
+ mark_object (backtrace_function (pdl));
+ if (nargs == UNEVALLED)
+ nargs = 1;
+ while (nargs--)
+ mark_object (backtrace_args (pdl)[nargs]);
+ }
+ break;
+
+ case SPECPDL_LET_DEFAULT:
+ case SPECPDL_LET_LOCAL:
+ mark_object (specpdl_where (pdl));
+ /* Fall through. */
+ case SPECPDL_LET:
+ mark_object (specpdl_symbol (pdl));
+ mark_object (specpdl_old_value (pdl));
+ break;
+
+ case SPECPDL_UNWIND_PTR:
+ case SPECPDL_UNWIND_INT:
case SPECPDL_UNWIND_VOID:
break;
@@ -3924,8 +3928,6 @@ alist of active lexical bindings. */);
staticpro (&Vsignaling_function);
Vsignaling_function = Qnil;
- inhibit_lisp_code = Qnil;
-
defsubr (&Sor);
defsubr (&Sand);
defsubr (&Sif);
diff --git a/src/filelock.c b/src/filelock.c
index a4b742a..5ffdf72 100644
--- a/src/filelock.c
+++ b/src/filelock.c
@@ -171,13 +171,10 @@ get_boot_time (void)
}
#if defined (BOOT_TIME)
-#ifndef CANNOT_DUMP
- /* The utmp routines maintain static state.
- Don't touch that state unless we are initialized,
- since it might not survive dumping. */
- if (! initialized)
+ /* The utmp routines maintain static state. Don't touch that state
+ if we are going to dump, since it might not survive dumping. */
+ if (will_dump)
return boot_time;
-#endif /* not CANNOT_DUMP */
/* Try to get boot time from utmp before wtmp,
since utmp is typically much smaller than wtmp.
@@ -671,7 +668,7 @@ lock_file (Lisp_Object fn)
/* Don't do locking while dumping Emacs.
Uncompressing wtmp files uses call-process, which does not work
in an uninitialized Emacs. */
- if (! NILP (Vpurify_flag))
+ if (will_dump)
return;
orig_fn = fn;
diff --git a/src/fingerprint-dummy.c b/src/fingerprint-dummy.c
new file mode 100644
index 0000000..295654a
--- /dev/null
+++ b/src/fingerprint-dummy.c
@@ -0,0 +1,24 @@
+/* Dummy fingerprint
+
+Copyright (C) 2016 Free Software Foundation,
+Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
+
+#include "fingerprint.h"
+
+/* Dummy fingerprint to use as hash input. */
+const uint8_t fingerprint[32] = { 0 };
diff --git a/src/fingerprint.h b/src/fingerprint.h
new file mode 100644
index 0000000..b48d40f
--- /dev/null
+++ b/src/fingerprint.h
@@ -0,0 +1,32 @@
+/* Header file for the Emacs build fingerprint.
+
+Copyright (C) 2016 Free Software Foundation,
+Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
+
+#ifndef EMACS_FINGERPRINT_H
+#define EMACS_FINGERPRINT_H
+
+#include <stdint.h>
+
+/* We generate fingerprint.c and fingerprint.o from all the sources in
+ Emacs. This way, we have a unique value that we can use to pair
+ data files (like a portable dump image) with a specific build of
+ Emacs. */
+extern const uint8_t fingerprint[32];
+
+#endif
diff --git a/src/fns.c b/src/fns.c
index dfc7842..645c0bf 100644
--- a/src/fns.c
+++ b/src/fns.c
@@ -34,6 +34,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "buffer.h"
#include "intervals.h"
#include "window.h"
+#include "pdumper.h"
static void sort_vector_copy (Lisp_Object, ptrdiff_t,
Lisp_Object *restrict, Lisp_Object *restrict);
@@ -2864,7 +2865,7 @@ suppressed. */)
/* This is to make sure that loadup.el gives a clear picture
of what files are preloaded and when. */
- if (! NILP (Vpurify_flag))
+ if (will_dump)
error ("(require %s) while preparing to dump",
SDATA (SYMBOL_NAME (feature)));
@@ -3735,7 +3736,7 @@ hashfn_eq (struct hash_table_test *ht, Lisp_Object key)
`equal' to compare keys. The hash code returned is guaranteed to fit
in a Lisp integer. */
-static EMACS_UINT
+EMACS_UINT
hashfn_equal (struct hash_table_test *ht, Lisp_Object key)
{
return sxhash (key, 0);
@@ -3745,7 +3746,7 @@ hashfn_equal (struct hash_table_test *ht, Lisp_Object key)
`eql' to compare keys. The hash code returned is guaranteed to fit
in a Lisp integer. */
-static EMACS_UINT
+EMACS_UINT
hashfn_eql (struct hash_table_test *ht, Lisp_Object key)
{
return FLOATP (key) ? hashfn_equal (ht, key) : hashfn_eq (ht, key);
@@ -3980,6 +3981,51 @@ maybe_resize_hash_table (struct Lisp_Hash_Table *h)
}
}
+static bool
+hash_rehash_needed_p (const struct Lisp_Hash_Table *h)
+{
+ return h->count < 0;
+}
+
+static void
+hash_table_rehash (struct Lisp_Hash_Table *h)
+{
+ ptrdiff_t size = ASIZE (h->next);
+ Lisp_Object hash = h->hash;
+ Lisp_Object kv = h->key_and_value;
+ for (ptrdiff_t i = 0; i < size; ++i)
+ if (!NILP (AREF (hash, i)))
+ {
+ Lisp_Object key = AREF (kv, 2*i);
+ EMACS_UINT hash_code = h->test.hashfn (&h->test, key);
+ ASET (hash, i, make_number (hash_code));
+ }
+ Lisp_Object index = h->index;
+ Ffillarray (index, Qnil);
+ Lisp_Object next = h->next;
+ for (ptrdiff_t i = 0; i < size; ++i)
+ if (!NILP (AREF (hash, i)))
+ {
+ EMACS_UINT hash_code = XUINT (AREF (hash, i));
+ ptrdiff_t start_of_bucket = hash_code % ASIZE (index);
+ ASET (next, i, AREF (index, start_of_bucket));
+ ASET (index, start_of_bucket, make_number (i));
+ eassert (!EQ (AREF (next, i), make_number (i))); /* Stop loops. */
+ }
+}
+
+static void
+hash_rehash_if_needed (struct Lisp_Hash_Table *h)
+{
+ if (hash_rehash_needed_p (h))
+ {
+ hash_table_rehash (h);
+ /* Do last so that if we're interrupted, we retry on next access. */
+ eassert (h->count < 0);
+ h->count = -h->count;
+ eassert (!hash_rehash_needed_p (h));
+ }
+}
/* Lookup KEY in hash table H. If HASH is non-null, return in *HASH
the hash code of KEY. Value is the index of the entry in H
@@ -3992,6 +4038,8 @@ hash_lookup (struct Lisp_Hash_Table *h, Lisp_Object key, EMACS_UINT *hash)
ptrdiff_t start_of_bucket;
Lisp_Object idx;
+ hash_rehash_if_needed (h);
+
hash_code = h->test.hashfn (&h->test, key);
eassert ((hash_code & ~INTMASK) == 0);
if (hash)
@@ -4025,6 +4073,8 @@ hash_put (struct Lisp_Hash_Table *h, Lisp_Object key, Lisp_Object value,
{
ptrdiff_t start_of_bucket, i;
+ hash_rehash_if_needed (h);
+
eassert ((hash & ~INTMASK) == 0);
/* Increment count after resizing because resizing may fail. */
@@ -4057,6 +4107,8 @@ hash_remove_from_table (struct Lisp_Hash_Table *h, Lisp_Object key)
ptrdiff_t start_of_bucket;
Lisp_Object idx, prev;
+ hash_rehash_if_needed (h);
+
hash_code = h->test.hashfn (&h->test, key);
eassert ((hash_code & ~INTMASK) == 0);
start_of_bucket = hash_code % ASIZE (h->index);
@@ -4103,6 +4155,9 @@ hash_remove_from_table (struct Lisp_Hash_Table *h, Lisp_Object key)
static void
hash_clear (struct Lisp_Hash_Table *h)
{
+ if (CONSP (h->index))
+ h->index = CDR (h->index);
+
if (h->count > 0)
{
ptrdiff_t i, size = HASH_TABLE_SIZE (h);
@@ -4137,6 +4192,8 @@ hash_clear (struct Lisp_Hash_Table *h)
static bool
sweep_weak_table (struct Lisp_Hash_Table *h, bool remove_entries_p)
{
+ eassert (!hash_rehash_needed_p (h));
+
ptrdiff_t n = gc_asize (h->index);
bool marked = false;
@@ -4221,9 +4278,17 @@ sweep_weak_table (struct Lisp_Hash_Table *h, bool remove_entries_p)
current garbage collection. Remove weak tables that don't survive
from Vweak_hash_tables. Called from gc_sweep. */
+static bool
+ht_marked_p (struct Lisp_Hash_Table *h)
+{
+ return pdumper_object_p (h)
+ ? pdumper_marked_p (h)
+ : !!(h->header.size & ARRAY_MARK_FLAG);
+}
+
NO_INLINE /* For better stack traces */
void
-sweep_weak_hash_tables (void)
+mark_and_sweep_weak_hash_tables (void)
{
struct Lisp_Hash_Table *h, *used, *next;
bool marked;
@@ -4239,7 +4304,7 @@ sweep_weak_hash_tables (void)
marked = 0;
for (h = weak_hash_tables; h; h = h->next_weak)
{
- if (h->header.size & ARRAY_MARK_FLAG)
+ if (ht_marked_p (h))
marked |= sweep_weak_table (h, 0);
}
}
@@ -4250,7 +4315,7 @@ sweep_weak_hash_tables (void)
{
next = h->next_weak;
- if (h->header.size & ARRAY_MARK_FLAG)
+ if (ht_marked_p (h))
{
/* TABLE is marked as used. Sweep its contents. */
if (h->count > 0)
@@ -5066,9 +5131,14 @@ If nil, use the current buffer." */ )
}
\f
+static void syms_of_fns_for_pdumper (void);
+
void
syms_of_fns (void)
{
+ /* Not staticpro! */
+ pdumper_remember_lv_raw_ptr (&weak_hash_tables, Lisp_Vectorlike);
+
DEFSYM (Qmd5, "md5");
DEFSYM (Qsha1, "sha1");
DEFSYM (Qsha224, "sha224");
@@ -5232,4 +5302,20 @@ this variable. */);
defsubr (&Ssecure_hash);
defsubr (&Sbuffer_hash);
defsubr (&Slocale_info);
+
+ pdumper_do_now_and_after_load (syms_of_fns_for_pdumper);
+}
+
+static void
+syms_of_fns_for_pdumper (void)
+{
+ /* Rehash weak tables eagerly on dump load so that we don't have to
+ support running hash_rehash_if_needed during GC. (Rehashing can
+ invoke user-defined code, and we can't support running arbitrary
+ user code during GC. */
+ if (dumped_with_pdumper)
+ for (struct Lisp_Hash_Table *ht = weak_hash_tables;
+ ht != NULL;
+ ht = ht->next_weak)
+ hash_rehash_if_needed (ht);
}
diff --git a/src/font.c b/src/font.c
index ce63233..7612f48 100644
--- a/src/font.c
+++ b/src/font.c
@@ -38,6 +38,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "fontset.h"
#include "font.h"
#include "termhooks.h"
+#include "pdumper.h"
#ifdef HAVE_WINDOW_SYSTEM
#include TERM_HEADER
@@ -5293,9 +5294,10 @@ syms_of_font (void)
sort_shift_bits[FONT_SIZE_INDEX] = 16;
sort_shift_bits[FONT_WIDTH_INDEX] = 23;
/* Note that the other elements in sort_shift_bits are not used. */
+ PDUMPER_REMEMBER_SCALAR (sort_shift_bits);
- staticpro (&font_charset_alist);
font_charset_alist = Qnil;
+ staticpro (&font_charset_alist);
DEFSYM (Qopentype, "opentype");
@@ -5333,13 +5335,13 @@ syms_of_font (void)
DEFSYM (QCuser_spec, ":user-spec");
- staticpro (&scratch_font_spec);
scratch_font_spec = Ffont_spec (0, NULL);
- staticpro (&scratch_font_prefer);
+ staticpro (&scratch_font_spec);
scratch_font_prefer = Ffont_spec (0, NULL);
+ staticpro (&scratch_font_prefer);
- staticpro (&Vfont_log_deferred);
Vfont_log_deferred = Fmake_vector (make_number (3), Qnil);
+ staticpro (&Vfont_log_deferred);
#if 0
#ifdef HAVE_LIBOTF
diff --git a/src/fontset.c b/src/fontset.c
index 38ff780..130293d 100644
--- a/src/fontset.c
+++ b/src/fontset.c
@@ -39,6 +39,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include TERM_HEADER
#endif /* HAVE_WINDOW_SYSTEM */
#include "font.h"
+#include "pdumper.h"
/* FONTSET
@@ -2126,6 +2127,7 @@ syms_of_fontset (void)
build_pure_c_string ("-*-*-*-*-*-*-*-*-*-*-*-*-fontset-default"));
ASET (Vfontset_table, 0, Vdefault_fontset);
next_fontset_id = 1;
+ PDUMPER_REMEMBER_SCALAR (next_fontset_id);
auto_fontset_alist = Qnil;
staticpro (&auto_fontset_alist);
diff --git a/src/frame.c b/src/frame.c
index b1d89f3..6e75dc3 100644
--- a/src/frame.c
+++ b/src/frame.c
@@ -52,6 +52,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#ifdef USE_X_TOOLKIT
#include "widget.h"
#endif
+#include "pdumper.h"
/* The currently selected frame. */
@@ -840,10 +841,7 @@ make_initial_frame (void)
Lisp_Object frame;
eassert (initial_kboard);
-
- /* The first call must initialize Vframe_list. */
- if (! (NILP (Vframe_list) || CONSP (Vframe_list)))
- Vframe_list = Qnil;
+ eassert (NILP (Vframe_list) || CONSP (Vframe_list));
terminal = init_initial_terminal ();
@@ -4904,6 +4902,26 @@ make_monitor_attribute_list (struct MonitorInfo *monitors,
Initialization
***********************************************************************/
+static void init_frame_once_for_pdumper (void);
+
+void
+init_frame_once (void)
+{
+ staticpro (&Vframe_list);
+ staticpro (&selected_frame);
+ PDUMPER_IGNORE (last_nonminibuf_frame);
+ Vframe_list = Qnil;
+ selected_frame = Qnil;
+ pdumper_do_now_and_after_load (init_frame_once_for_pdumper);
+}
+
+static void
+init_frame_once_for_pdumper (void)
+{
+ PDUMPER_RESET_LV (Vframe_list, Qnil);
+ PDUMPER_RESET_LV (selected_frame, Qnil);
+}
+
void
syms_of_frame (void)
{
@@ -5304,8 +5322,6 @@ This variable is effective only with the X toolkit (and there only when
Gtk+ tooltips are not used) and on Windows. */);
tooltip_reuse_hidden_frame = false;
- staticpro (&Vframe_list);
-
defsubr (&Sframep);
defsubr (&Sframe_live_p);
defsubr (&Swindow_system);
diff --git a/src/fringe.c b/src/fringe.c
index 986bde1..4515186 100644
--- a/src/fringe.c
+++ b/src/fringe.c
@@ -29,6 +29,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "buffer.h"
#include "blockinput.h"
#include "termhooks.h"
+#include "pdumper.h"
/* Fringe bitmaps are represented in three different ways:
@@ -1725,12 +1726,18 @@ mark_fringe_data (void)
/* Initialize this module when Emacs starts. */
+static void init_fringe_once_for_pdumper (void);
+
void
init_fringe_once (void)
{
- int bt;
+ pdumper_do_now_and_after_load (init_fringe_once_for_pdumper);
+}
- for (bt = NO_FRINGE_BITMAP + 1; bt < MAX_STANDARD_FRINGE_BITMAPS; bt++)
+static void
+init_fringe_once_for_pdumper (void)
+{
+ for (int bt = NO_FRINGE_BITMAP + 1; bt < MAX_STANDARD_FRINGE_BITMAPS; bt++)
init_fringe_bitmap (bt, &standard_bitmaps[bt], 1);
}
diff --git a/src/ftcrfont.c b/src/ftcrfont.c
index 2676502..4532152 100644
--- a/src/ftcrfont.c
+++ b/src/ftcrfont.c
@@ -26,6 +26,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "blockinput.h"
#include "font.h"
#include "ftfont.h"
+#include "pdumper.h"
/* FTCR font driver. */
@@ -282,6 +283,8 @@ ftcrfont_draw (struct glyph_string *s,
\f
+static void syms_of_ftcrfont_for_pdumper (void);
+
void
syms_of_ftcrfont (void)
{
@@ -289,7 +292,11 @@ syms_of_ftcrfont (void)
abort ();
DEFSYM (Qftcr, "ftcr");
+}
+static void
+syms_of_ftcrfont_for_pdumper (void)
+{
ftcrfont_driver = ftfont_driver;
ftcrfont_driver.type = Qftcr;
ftcrfont_driver.list = ftcrfont_list;
diff --git a/src/ftfont.c b/src/ftfont.c
index 1ae3f88..d1e234f 100644
--- a/src/ftfont.c
+++ b/src/ftfont.c
@@ -34,6 +34,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "composite.h"
#include "font.h"
#include "ftfont.h"
+#include "pdumper.h"
/* Flag to tell if FcInit is already called or not. */
static bool fc_initialized;
@@ -2776,6 +2777,8 @@ ftfont_combining_capability (struct font *font)
#endif
}
+static void syms_of_ftfont_for_pdumper (void);
+
void
syms_of_ftfont (void)
{
@@ -2799,6 +2802,13 @@ syms_of_ftfont (void)
staticpro (&ft_face_cache);
ft_face_cache = Qnil;
+ pdumper_do_now_and_after_load (syms_of_ftfont_for_pdumper);
+}
+
+static void
+syms_of_ftfont_for_pdumper (void)
+{
+ PDUMPER_RESET_LV (ft_face_cache, Qnil);
ftfont_driver.type = Qfreetype;
register_font_driver (&ftfont_driver, NULL);
}
diff --git a/src/ftxfont.c b/src/ftxfont.c
index bfdeb40..ea7f476 100644
--- a/src/ftxfont.c
+++ b/src/ftxfont.c
@@ -28,6 +28,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "frame.h"
#include "blockinput.h"
#include "font.h"
+#include "pdumper.h"
/* FTX font driver. */
@@ -345,10 +346,18 @@ ftxfont_end_for_frame (struct frame *f)
\f
+static void syms_of_ftxfont_for_pdumper (void);
+
void
syms_of_ftxfont (void)
{
DEFSYM (Qftx, "ftx");
+ pdumper_do_now_and_after_load (syms_of_ftxfont_for_pdumper);
+}
+
+static void
+syms_of_ftxfont_for_pdumper (void)
+{
ftxfont_driver = ftfont_driver;
ftxfont_driver.type = Qftx;
diff --git a/src/gnutls.c b/src/gnutls.c
index af2ba52..c7c708b 100644
--- a/src/gnutls.c
+++ b/src/gnutls.c
@@ -24,6 +24,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "process.h"
#include "gnutls.h"
#include "coding.h"
+#include "pdumper.h"
#ifdef HAVE_GNUTLS
@@ -1726,6 +1727,7 @@ syms_of_gnutls (void)
);
#ifdef HAVE_GNUTLS
gnutls_global_initialized = 0;
+ PDUMPER_IGNORE (gnutls_global_initialized);
DEFSYM (Qgnutls_code, "gnutls-code");
DEFSYM (Qgnutls_anon, "gnutls-anon");
diff --git a/src/image.c b/src/image.c
index d82fedb..3c818a2 100644
--- a/src/image.c
+++ b/src/image.c
@@ -45,6 +45,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "coding.h"
#include "termhooks.h"
#include "font.h"
+#include "pdumper.h"
#ifdef HAVE_SYS_STAT_H
#include <sys/stat.h>
@@ -9794,7 +9795,9 @@ void
syms_of_image (void)
{
/* Initialize this only once; it will be reset before dumping. */
+ /* The portable dumper will just leave it NULL, so no need to reset. */
image_types = NULL;
+ PDUMPER_IGNORE (image_types);
/* Must be defined now because we're going to update it below, while
defining the supported image types. */
diff --git a/src/insdel.c b/src/insdel.c
index ed914ec..3b4594e 100644
--- a/src/insdel.c
+++ b/src/insdel.c
@@ -29,6 +29,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "buffer.h"
#include "window.h"
#include "region-cache.h"
+#include "pdumper.h"
static void insert_from_string_1 (Lisp_Object, ptrdiff_t, ptrdiff_t, ptrdiff_t,
ptrdiff_t, bool, bool);
@@ -1912,6 +1913,14 @@ prepare_to_modify_buffer_1 (ptrdiff_t start, ptrdiff_t end,
if (!NILP (BVAR (current_buffer, read_only)))
Fbarf_if_buffer_read_only (temp);
+ /* If we're about to modify a buffer the contents of which come from
+ a dump file, copy the contents to private storage first so we
+ don't take a COW fault on the buffer text and keep it around
+ forever. */
+ if (pdumper_object_p (BEG_ADDR))
+ enlarge_buffer_text (current_buffer, 0);
+ eassert (!pdumper_object_p (BEG_ADDR));
+
run_undoable_change();
bset_redisplay (current_buffer);
diff --git a/src/intervals.h b/src/intervals.h
index 9a38d84..8ea36d9 100644
--- a/src/intervals.h
+++ b/src/intervals.h
@@ -26,7 +26,6 @@ INLINE_HEADER_BEGIN
struct interval
{
/* The first group of entries deal with the tree structure. */
-
ptrdiff_t total_length; /* Length of myself and both children. */
ptrdiff_t position; /* Cache of interval's character position. */
/* This field is usually updated
diff --git a/src/keyboard.c b/src/keyboard.c
index 65938a5..c5a7f4d 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -66,6 +66,8 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include <ignore-value.h>
+#include "pdumper.h"
+
#ifdef HAVE_WINDOW_SYSTEM
#include TERM_HEADER
#endif /* HAVE_WINDOW_SYSTEM */
@@ -10996,6 +10998,8 @@ static const struct event_head head_table[] = {
{SYMBOL_INDEX (Qselect_window), SYMBOL_INDEX (Qswitch_frame)}
};
+static void syms_of_keyboard_for_pdumper (void);
+
void
syms_of_keyboard (void)
{
@@ -11006,9 +11010,11 @@ syms_of_keyboard (void)
staticpro (&Vlispy_mouse_stem);
regular_top_level_message = build_pure_c_string ("Back to top level");
+ staticpro (®ular_top_level_message);
#ifdef HAVE_STACK_OVERFLOW_HANDLING
recover_top_level_message
= build_pure_c_string ("Re-entering top level after C stack overflow");
+ staticpro (&recover_top_level_message);
#endif
DEFVAR_LISP ("internal--top-level-message", Vinternal__top_level_message,
doc: /* Message displayed by `normal-top-level'. */);
@@ -11820,7 +11826,38 @@ otherwise be lost. If nil, crash immediately in response to fatal
signals. */);
attempt_orderly_shutdown_on_fatal_signal = true;
+ pdumper_do_now_and_after_load (syms_of_keyboard_for_pdumper);
+}
+
+static void
+syms_of_keyboard_for_pdumper (void)
+{
+ /* Make sure input state is pristine when restoring from a dump.
+ init_keyboard() also resets some of these, but the duplication
+ doesn't hurt and makes sure that allocate_kboard and subsequent
+ early init functions see the environment they expect. */
+
+ PDUMPER_RESET_LV (pending_funcalls, Qnil);
+ PDUMPER_RESET_LV (unread_switch_frame, Qnil);
+ PDUMPER_RESET_LV (internal_last_event_frame, Qnil);
+ PDUMPER_RESET_LV (last_command_event, Qnil);
+ PDUMPER_RESET_LV (last_nonmenu_event, Qnil);
+ PDUMPER_RESET_LV (last_input_event, Qnil);
+ PDUMPER_RESET_LV (Vunread_command_events, Qnil);
+ PDUMPER_RESET_LV (Vunread_post_input_method_events, Qnil);
+ PDUMPER_RESET_LV (Vunread_input_method_events, Qnil);
+ PDUMPER_RESET_LV (Vthis_command, Qnil);
+ PDUMPER_RESET_LV (Vreal_this_command, Qnil);
+ PDUMPER_RESET_LV (Vthis_command_keys_shift_translated, Qnil);
+ PDUMPER_RESET_LV (Vthis_original_command, Qnil);
+ PDUMPER_RESET (num_input_keys, 0);
+ PDUMPER_RESET (num_nonmacro_input_events, 0);
+ PDUMPER_RESET_LV (Vlast_event_frame, Qnil);
+ PDUMPER_RESET_LV (Vdeferred_action_list, Qnil);
+ PDUMPER_RESET_LV (Vdelayed_warnings_list, Qnil);
+
/* Create the initial keyboard. Qt means 'unset'. */
+ eassert (initial_kboard == NULL);
initial_kboard = allocate_kboard (Qt);
}
@@ -11911,8 +11948,8 @@ mark_kboards (void)
for (kb = all_kboards; kb; kb = kb->next_kboard)
{
if (kb->kbd_macro_buffer)
- for (p = kb->kbd_macro_buffer; p < kb->kbd_macro_ptr; p++)
- mark_object (*p);
+ for (p = kb->kbd_macro_buffer; p < kb->kbd_macro_ptr; p++)
+ mark_object (*p);
mark_object (KVAR (kb, Voverriding_terminal_local_map));
mark_object (KVAR (kb, Vlast_command));
mark_object (KVAR (kb, Vreal_last_command));
@@ -11936,17 +11973,17 @@ mark_kboards (void)
union buffered_input_event *event;
for (event = kbd_fetch_ptr; event != kbd_store_ptr; event++)
{
- if (event == kbd_buffer + KBD_BUFFER_SIZE)
- event = kbd_buffer;
- /* These two special event types has no Lisp_Objects to mark. */
- if (event->kind != SELECTION_REQUEST_EVENT
- && event->kind != SELECTION_CLEAR_EVENT)
- {
- mark_object (event->ie.x);
- mark_object (event->ie.y);
- mark_object (event->ie.frame_or_window);
- mark_object (event->ie.arg);
- }
+ if (event == kbd_buffer + KBD_BUFFER_SIZE)
+ event = kbd_buffer;
+ /* These two special event types has no Lisp_Objects to mark. */
+ if (event->kind != SELECTION_REQUEST_EVENT
+ && event->kind != SELECTION_CLEAR_EVENT)
+ {
+ mark_object (event->ie.x);
+ mark_object (event->ie.y);
+ mark_object (event->ie.frame_or_window);
+ mark_object (event->ie.arg);
+ }
}
}
}
diff --git a/src/lisp.h b/src/lisp.h
index e087828..73ccdee 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -343,9 +343,13 @@ error !;
(struct Lisp_Symbol *) ((intptr_t) XLI (a) - Lisp_Symbol \
+ (char *) lispsym))
# define lisp_h_XTYPE(a) ((enum Lisp_Type) (XLI (a) & ~VALMASK))
+#if 0 /* XXX */
# define lisp_h_XUNTAG(a, type) \
__builtin_assume_aligned ((void *) (intptr_t) (XLI (a) - (type)), \
- GCALIGNMENT)
+ GCALIGNMENT)
+#else
+# define lisp_h_XUNTAG(a, type) ((void *) (intptr_t) (XLI (a) - (type)))
+#endif
#endif
/* When compiling via gcc -O0, define the key operations as macros, as
@@ -604,12 +608,42 @@ extern _Noreturn Lisp_Object wrong_type_argument (Lisp_Object, Lisp_Object);
extern _Noreturn void wrong_choice (Lisp_Object, Lisp_Object);
/* Defined in emacs.c. */
-#ifdef DOUG_LEA_MALLOC
-extern bool might_dump;
+
+extern struct gflags {
+ /* Set after Emacs has started up the first time.
+ Prevents reinitialization of the Lisp world and keymaps
+ on subsequent starts. */
+ bool initialized_ : 1;
+ /* True means this Emacs instance was born to dump. */
+ bool will_dump_ : 1;
+ bool will_dump_with_pdumper_ : 1;
+#ifndef CANNOT_DUMP
+ bool will_dump_with_unexec_ : 1;
+#endif
+#ifndef CANNOT_DUMP
+ /* Set in an Emacs process that has been restored from an unexec
+ dump. */
+ bool dumped_with_unexec_ : 1;
#endif
-/* True means Emacs has already been initialized.
- Used during startup to detect startup of dumped Emacs. */
-extern bool initialized;
+ /* Set in an Emacs process that has been restored from a portable
+ dump. */
+ bool dumped_with_pdumper_ : 1;
+} gflags;
+
+#define CONST_GFLAGS ((const struct gflags *)(&gflags))
+
+#define initialized CONST_GFLAGS->initialized_
+#define will_dump CONST_GFLAGS->will_dump_
+#define will_dump_with_pdumper CONST_GFLAGS->will_dump_with_pdumper_
+#ifdef CANNOT_DUMP
+# define will_dump_with_unexec false
+# define dumped_with_unexec false
+#else
+# define will_dump_with_unexec CONST_GFLAGS->will_dump_with_unexec_
+# define dumped_with_unexec CONST_GFLAGS->dumped_with_unexec_
+#endif
+
+#define dumped_with_pdumper CONST_GFLAGS->dumped_with_pdumper_
/* Defined in floatfns.c. */
extern double extract_float (Lisp_Object);
@@ -744,6 +778,19 @@ struct Lisp_Symbol
# define DEFINE_NON_NIL_Q_SYMBOL_MACROS true
#endif
+/* True if N is a power of 2. N should be positive. */
+
+#define POWER_OF_2(n) (((n) & ((n) - 1)) == 0)
+
+/* Return X rounded to the next multiple of Y. Y should be positive,
+ and Y - 1 + X should not overflow. Arguments should not have side
+ effects, as they are evaluated more than once. Tune for Y being a
+ power of 2. */
+
+#define ROUNDUP(x, y) (POWER_OF_2 (y) \
+ ? ((y) - 1 + (x)) & ~ ((y) - 1) \
+ : ((y) - 1 + (x)) - ((y) - 1 + (x)) % (y))
+
#include "globals.h"
/* Convert a Lisp_Object to the corresponding EMACS_INT and vice versa.
@@ -2279,6 +2326,9 @@ struct Lisp_Finalizer
Lisp_Object function;
};
+extern struct Lisp_Finalizer finalizers;
+extern struct Lisp_Finalizer doomed_finalizers;
+
/* A miscellaneous object, when it's on the free list. */
struct Lisp_Free
{
@@ -2664,8 +2714,16 @@ BUFFER_OBJFWDP (union Lisp_Fwd *a)
return XFWDTYPE (a) == Lisp_Fwd_Buffer_Obj;
}
+INLINE enum pvec_type
+PSEUDOVECTOR_TYPE (const struct vectorlike_header *a)
+{
+ return ((a->size & PSEUDOVECTOR_FLAG)
+ ? (a->size & PVEC_TYPE_MASK) >> PSEUDOVECTOR_AREA_BITS
+ : PVEC_NORMAL_VECTOR);
+}
+
INLINE bool
-PSEUDOVECTOR_TYPEP (struct vectorlike_header *a, int code)
+PSEUDOVECTOR_TYPEP (const struct vectorlike_header *a, int code)
{
return ((a->size & (PSEUDOVECTOR_FLAG | PVEC_TYPE_MASK))
== (PSEUDOVECTOR_FLAG | (code << PSEUDOVECTOR_AREA_BITS)));
@@ -2920,6 +2978,16 @@ CHECK_NUMBER_CDR (Lisp_Object x)
XSETCDR (x, tmp);
}
\f
+
+/* If we're not dumping and we might be using the portable dumper, try
+ to bunch all the subr structures together for more efficient dump
+ loading. */
+#ifdef CANNOT_DUMP
+# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
+#else
+# define SUBR_SECTION_ATTRIBUTE
+#endif
+
/* Define a built-in function for calling from Lisp.
`lname' should be the name to give the function in Lisp,
as a null-terminated C string.
@@ -2950,6 +3018,7 @@ CHECK_NUMBER_CDR (Lisp_Object x)
#ifdef _MSC_VER
#define DEFUN(lname, fnname, sname, minargs, maxargs, intspec, doc) \
Lisp_Object fnname DEFUN_ARGS_ ## maxargs ; \
+ SUBR_SECTION_ATTRIBUTE \
static struct Lisp_Subr alignas (GCALIGNMENT) sname = \
{ { (PVEC_SUBR << PSEUDOVECTOR_AREA_BITS) \
| (sizeof (struct Lisp_Subr) / sizeof (EMACS_INT)) }, \
@@ -2958,6 +3027,7 @@ CHECK_NUMBER_CDR (Lisp_Object x)
Lisp_Object fnname
#else /* not _MSC_VER */
#define DEFUN(lname, fnname, sname, minargs, maxargs, intspec, doc) \
+ SUBR_SECTION_ATTRIBUTE \
static struct Lisp_Subr alignas (GCALIGNMENT) sname = \
{ { PVEC_SUBR << PSEUDOVECTOR_AREA_BITS }, \
{ .a ## maxargs = fnname }, \
@@ -3234,6 +3304,11 @@ extern Lisp_Object Vascii_canon_table;
/* Call staticpro (&var) to protect static variable `var'. */
void staticpro (Lisp_Object *);
+
+enum { NSTATICS = 2048 };
+extern Lisp_Object *staticvec[NSTATICS];
+extern int staticidx;
+
\f
/* Forward declarations for prototypes. */
struct window;
@@ -3426,9 +3501,11 @@ extern void syms_of_syntax (void);
enum { NEXT_ALMOST_PRIME_LIMIT = 11 };
extern EMACS_INT next_almost_prime (EMACS_INT) ATTRIBUTE_CONST;
extern Lisp_Object larger_vector (Lisp_Object, ptrdiff_t, ptrdiff_t);
-extern void sweep_weak_hash_tables (void);
+extern void mark_and_sweep_weak_hash_tables (void);
EMACS_UINT hash_string (char const *, ptrdiff_t);
EMACS_UINT sxhash (Lisp_Object, int);
+EMACS_UINT hashfn_eql (struct hash_table_test *ht, Lisp_Object key);
+EMACS_UINT hashfn_equal (struct hash_table_test *ht, Lisp_Object key);
Lisp_Object make_hash_table (struct hash_table_test, Lisp_Object, Lisp_Object,
Lisp_Object, Lisp_Object);
ptrdiff_t hash_lookup (struct Lisp_Hash_Table *, Lisp_Object, EMACS_UINT *);
@@ -3590,6 +3667,9 @@ extern Lisp_Object *stack_base;
extern EMACS_INT consing_since_gc;
extern EMACS_INT gc_relative_threshold;
extern EMACS_INT memory_full_cons_threshold;
+#ifdef ENABLE_CHECKING
+extern Lisp_Object Vdead;
+#endif
extern Lisp_Object list1 (Lisp_Object);
extern Lisp_Object list2 (Lisp_Object, Lisp_Object);
extern Lisp_Object list3 (Lisp_Object, Lisp_Object, Lisp_Object);
@@ -3599,6 +3679,21 @@ extern Lisp_Object list5 (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object,
enum constype {CONSTYPE_HEAP, CONSTYPE_PURE};
extern Lisp_Object listn (enum constype, ptrdiff_t, Lisp_Object, ...);
+enum gc_root_type {
+ GC_ROOT_STATICPRO,
+ GC_ROOT_BUFFER_LOCAL_DEFAULT,
+ GC_ROOT_BUFFER_LOCAL_NAME,
+ GC_ROOT_C_SYMBOL
+};
+
+struct gc_root_visitor {
+ void (*visit)(Lisp_Object *root_ptr,
+ enum gc_root_type type,
+ void *data);
+ void *data;
+};
+extern void visit_static_gc_roots (struct gc_root_visitor visitor);
+
/* Build a frequently used 2/3/4-integer lists. */
INLINE Lisp_Object
@@ -3627,6 +3722,7 @@ extern Lisp_Object make_string (const char *, ptrdiff_t);
extern Lisp_Object make_formatted_string (char *, const char *, ...)
ATTRIBUTE_FORMAT_PRINTF (2, 3);
extern Lisp_Object make_unibyte_string (const char *, ptrdiff_t);
+extern ptrdiff_t vector_nbytes (struct Lisp_Vector *v);
/* Make unibyte string from C string when the length isn't known. */
@@ -3827,7 +3923,7 @@ extern Lisp_Object string_to_number (char const *, int, bool);
extern void map_obarray (Lisp_Object, void (*) (Lisp_Object, Lisp_Object),
Lisp_Object);
extern void dir_warning (const char *, Lisp_Object);
-extern void init_obarray (void);
+extern void init_obarray_once (void);
extern void init_lread (void);
extern void syms_of_lread (void);
@@ -3847,7 +3943,6 @@ intern_c_string (const char *str)
extern Lisp_Object Vautoload_queue;
extern Lisp_Object Vrun_hooks;
extern Lisp_Object Vsignaling_function;
-extern Lisp_Object inhibit_lisp_code;
extern struct handler *handlerlist;
/* To run a normal hook, use the appropriate function from the list below.
@@ -3945,7 +4040,7 @@ extern _Noreturn void time_overflow (void);
extern Lisp_Object make_buffer_string (ptrdiff_t, ptrdiff_t, bool);
extern Lisp_Object make_buffer_string_both (ptrdiff_t, ptrdiff_t, ptrdiff_t,
ptrdiff_t, bool);
-extern void init_editfns (bool);
+extern void init_editfns (void);
extern void syms_of_editfns (void);
/* Defined in buffer.c. */
@@ -3960,7 +4055,7 @@ extern bool overlay_touches_p (ptrdiff_t);
extern Lisp_Object other_buffer_safely (Lisp_Object);
extern Lisp_Object get_truename_buffer (Lisp_Object);
extern void init_buffer_once (void);
-extern void init_buffer (int);
+extern void init_buffer (void);
extern void syms_of_buffer (void);
extern void keys_of_buffer (void);
@@ -4104,6 +4199,7 @@ extern void store_in_alist (Lisp_Object *, Lisp_Object, Lisp_Object);
extern Lisp_Object do_switch_frame (Lisp_Object, int, int, Lisp_Object);
extern Lisp_Object get_frame_param (struct frame *, Lisp_Object);
extern void frames_discard_buffer (Lisp_Object);
+extern void init_frame_once (void);
extern void syms_of_frame (void);
/* Defined in emacs.c. */
diff --git a/src/lread.c b/src/lread.c
index eab9b8b..9466827 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -42,6 +42,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "systime.h"
#include "termhooks.h"
#include "blockinput.h"
+#include "pdumper.h"
#include <c-ctype.h>
#ifdef MSDOS
@@ -1826,7 +1827,7 @@ readevalloop (Lisp_Object readcharfun,
? Qnil : list1 (Qt)));
/* Try to ensure sourcename is a truename, except whilst preloading. */
- if (NILP (Vpurify_flag)
+ if (!will_dump
&& !NILP (sourcename) && !NILP (Ffile_name_absolute_p (sourcename))
&& !NILP (Ffboundp (Qfile_truename)))
sourcename = call1 (Qfile_truename, sourcename) ;
@@ -3980,13 +3981,6 @@ usage: (unintern NAME OBARRAY) */)
if (SYMBOLP (name) && !EQ (name, tem))
return Qnil;
- /* There are plenty of other symbols which will screw up the Emacs
- session if we unintern them, as well as even more ways to use
- `setq' or `fset' or whatnot to make the Emacs session
- unusable. Let's not go down this silly road. --Stef */
- /* if (EQ (tem, Qnil) || EQ (tem, Qt))
- error ("Attempt to unintern t or nil"); */
-
XSYMBOL (tem)->interned = SYMBOL_UNINTERNED;
hash = oblookup_last_bucket_number;
@@ -4101,11 +4095,12 @@ OBARRAY defaults to the value of `obarray'. */)
#define OBARRAY_SIZE 1511
+static void init_obarray_once_for_pdumper (void);
+
void
-init_obarray (void)
+init_obarray_once (void)
{
Lisp_Object oblength;
- ptrdiff_t size = 100 + MAX_MULTIBYTE_LENGTH;
XSETFASTINT (oblength, OBARRAY_SIZE);
@@ -4128,15 +4123,25 @@ init_obarray (void)
XSYMBOL (Qt)->constant = 1;
XSYMBOL (Qt)->declared_special = true;
- /* Qt is correct even if CANNOT_DUMP. loadup.el will set to nil at end. */
+ /* Qt is correct even if not dumping. loadup.el will set to nil at end. */
Vpurify_flag = Qt;
DEFSYM (Qvariable_documentation, "variable-documentation");
+ pdumper_do_now_and_after_load (init_obarray_once_for_pdumper);
+}
+
+static void
+init_obarray_once_for_pdumper (void)
+{
+ ptrdiff_t size = 100 + MAX_MULTIBYTE_LENGTH;
read_buffer = xmalloc (size);
read_buffer_size = size;
}
+
\f
+int ndefsubr;
+
void
defsubr (struct Lisp_Subr *sname)
{
@@ -4145,6 +4150,7 @@ defsubr (struct Lisp_Subr *sname)
XSETPVECTYPE (sname, PVEC_SUBR);
XSETSUBR (tem, sname);
set_symbol_function (sym, tem);
+ ++ndefsubr;
}
#ifdef NOTDEF /* Use fset in subr.el now! */
@@ -4262,9 +4268,9 @@ load_path_check (Lisp_Object lpath)
are running uninstalled.
Uses the following logic:
- If CANNOT_DUMP: Use PATH_LOADSEARCH.
- The remainder is what happens when dumping works:
- If purify-flag (ie dumping) just use PATH_DUMPLOADSEARCH.
+ If !will_dump: Use PATH_LOADSEARCH.
+ The remainder is what happens when dumping is about to happen:
+ If dumping, just use PATH_DUMPLOADSEARCH.
Otherwise use PATH_LOADSEARCH.
If !initialized, then just return PATH_DUMPLOADSEARCH.
@@ -4287,129 +4293,109 @@ load_path_check (Lisp_Object lpath)
static Lisp_Object
load_path_default (void)
{
+ if (will_dump)
+ /* PATH_DUMPLOADSEARCH is the lisp dir in the source directory.
+ We used to add ../lisp (ie the lisp dir in the build
+ directory) at the front here, but that should not be
+ necessary, since in out of tree builds lisp/ is empty, save
+ for Makefile. */
+ return decode_env_path (0, PATH_DUMPLOADSEARCH, 0);
+
Lisp_Object lpath = Qnil;
- const char *normal;
+ const char *normal = PATH_LOADSEARCH;
+ const char *loadpath = NULL;
-#ifdef CANNOT_DUMP
#ifdef HAVE_NS
- const char *loadpath = ns_load_path ();
+ loadpath = ns_load_path ();
#endif
- normal = PATH_LOADSEARCH;
-#ifdef HAVE_NS
lpath = decode_env_path (0, loadpath ? loadpath : normal, 0);
-#else
- lpath = decode_env_path (0, normal, 0);
-#endif
-#else /* !CANNOT_DUMP */
-
- normal = NILP (Vpurify_flag) ? PATH_LOADSEARCH : PATH_DUMPLOADSEARCH;
-
- if (initialized)
+ if (!NILP (Vinstallation_directory))
{
-#ifdef HAVE_NS
- const char *loadpath = ns_load_path ();
- lpath = decode_env_path (0, loadpath ? loadpath : normal, 0);
-#else
- lpath = decode_env_path (0, normal, 0);
-#endif
- if (!NILP (Vinstallation_directory))
+ Lisp_Object tem, tem1;
+
+ /* Add to the path the lisp subdir of the installation
+ dir, if it is accessible. Note: in out-of-tree builds,
+ this directory is empty save for Makefile. */
+ tem = Fexpand_file_name (build_string ("lisp"),
+ Vinstallation_directory);
+ tem1 = Ffile_accessible_directory_p (tem);
+ if (!NILP (tem1))
{
- Lisp_Object tem, tem1;
+ if (NILP (Fmember (tem, lpath)))
+ {
+ /* We are running uninstalled. The default load-path
+ points to the eventual installed lisp directories.
+ We should not use those now, even if they exist,
+ so start over from a clean slate. */
+ lpath = list1 (tem);
+ }
+ }
+ else
+ /* That dir doesn't exist, so add the build-time
+ Lisp dirs instead. */
+ {
+ Lisp_Object dump_path =
+ decode_env_path (0, PATH_DUMPLOADSEARCH, 0);
+ lpath = nconc2 (lpath, dump_path);
+ }
- /* Add to the path the lisp subdir of the installation
- dir, if it is accessible. Note: in out-of-tree builds,
- this directory is empty save for Makefile. */
- tem = Fexpand_file_name (build_string ("lisp"),
+ /* Add site-lisp under the installation dir, if it exists. */
+ if (!no_site_lisp)
+ {
+ tem = Fexpand_file_name (build_string ("site-lisp"),
Vinstallation_directory);
tem1 = Ffile_accessible_directory_p (tem);
if (!NILP (tem1))
{
if (NILP (Fmember (tem, lpath)))
- {
- /* We are running uninstalled. The default load-path
- points to the eventual installed lisp directories.
- We should not use those now, even if they exist,
- so start over from a clean slate. */
- lpath = list1 (tem);
- }
- }
- else
- /* That dir doesn't exist, so add the build-time
- Lisp dirs instead. */
- {
- Lisp_Object dump_path =
- decode_env_path (0, PATH_DUMPLOADSEARCH, 0);
- lpath = nconc2 (lpath, dump_path);
+ lpath = Fcons (tem, lpath);
}
+ }
- /* Add site-lisp under the installation dir, if it exists. */
- if (!no_site_lisp)
- {
- tem = Fexpand_file_name (build_string ("site-lisp"),
- Vinstallation_directory);
- tem1 = Ffile_accessible_directory_p (tem);
- if (!NILP (tem1))
- {
- if (NILP (Fmember (tem, lpath)))
- lpath = Fcons (tem, lpath);
- }
- }
+ /* If Emacs was not built in the source directory,
+ and it is run from where it was built, add to load-path
+ the lisp and site-lisp dirs under that directory. */
+
+ if (NILP (Fequal (Vinstallation_directory, Vsource_directory)))
+ {
+ Lisp_Object tem2;
- /* If Emacs was not built in the source directory,
- and it is run from where it was built, add to load-path
- the lisp and site-lisp dirs under that directory. */
+ tem = Fexpand_file_name (build_string ("src/Makefile"),
+ Vinstallation_directory);
+ tem1 = Ffile_exists_p (tem);
- if (NILP (Fequal (Vinstallation_directory, Vsource_directory)))
+ /* Don't be fooled if they moved the entire source tree
+ AFTER dumping Emacs. If the build directory is indeed
+ different from the source dir, src/Makefile.in and
+ src/Makefile will not be found together. */
+ tem = Fexpand_file_name (build_string ("src/Makefile.in"),
+ Vinstallation_directory);
+ tem2 = Ffile_exists_p (tem);
+ if (!NILP (tem1) && NILP (tem2))
{
- Lisp_Object tem2;
-
- tem = Fexpand_file_name (build_string ("src/Makefile"),
- Vinstallation_directory);
- tem1 = Ffile_exists_p (tem);
-
- /* Don't be fooled if they moved the entire source tree
- AFTER dumping Emacs. If the build directory is indeed
- different from the source dir, src/Makefile.in and
- src/Makefile will not be found together. */
- tem = Fexpand_file_name (build_string ("src/Makefile.in"),
- Vinstallation_directory);
- tem2 = Ffile_exists_p (tem);
- if (!NILP (tem1) && NILP (tem2))
- {
- tem = Fexpand_file_name (build_string ("lisp"),
- Vsource_directory);
+ tem = Fexpand_file_name (build_string ("lisp"),
+ Vsource_directory);
- if (NILP (Fmember (tem, lpath)))
- lpath = Fcons (tem, lpath);
+ if (NILP (Fmember (tem, lpath)))
+ lpath = Fcons (tem, lpath);
- if (!no_site_lisp)
+ if (!no_site_lisp)
+ {
+ tem = Fexpand_file_name (build_string ("site-lisp"),
+ Vsource_directory);
+ tem1 = Ffile_accessible_directory_p (tem);
+ if (!NILP (tem1))
{
- tem = Fexpand_file_name (build_string ("site-lisp"),
- Vsource_directory);
- tem1 = Ffile_accessible_directory_p (tem);
- if (!NILP (tem1))
- {
- if (NILP (Fmember (tem, lpath)))
- lpath = Fcons (tem, lpath);
- }
+ if (NILP (Fmember (tem, lpath)))
+ lpath = Fcons (tem, lpath);
}
}
- } /* Vinstallation_directory != Vsource_directory */
+ }
+ } /* Vinstallation_directory != Vsource_directory */
- } /* if Vinstallation_directory */
- }
- else /* !initialized */
- {
- /* NORMAL refers to PATH_DUMPLOADSEARCH, ie the lisp dir in the
- source directory. We used to add ../lisp (ie the lisp dir in
- the build directory) at the front here, but that should not
- be necessary, since in out of tree builds lisp/ is empty, save
- for Makefile. */
- lpath = decode_env_path (0, normal, 0);
- }
-#endif /* !CANNOT_DUMP */
+ } /* if Vinstallation_directory */
return lpath;
}
@@ -4420,11 +4406,7 @@ init_lread (void)
/* First, set Vload_path. */
/* Ignore EMACSLOADPATH when dumping. */
-#ifdef CANNOT_DUMP
- bool use_loadpath = true;
-#else
- bool use_loadpath = NILP (Vpurify_flag);
-#endif
+ bool use_loadpath = !will_dump;
if (use_loadpath && egetenv ("EMACSLOADPATH"))
{
@@ -4475,7 +4457,7 @@ init_lread (void)
load_path_check (Vload_path);
/* Add the site-lisp directories at the front. */
- if (initialized && !no_site_lisp && PATH_SITELOADSEARCH[0] != '\0')
+ if (!will_dump && !no_site_lisp && PATH_SITELOADSEARCH[0] != '\0')
{
Lisp_Object sitelisp;
sitelisp = decode_env_path (0, PATH_SITELOADSEARCH, 0);
diff --git a/src/macfont.m b/src/macfont.m
index 3af9edc..233fa47 100644
--- a/src/macfont.m
+++ b/src/macfont.m
@@ -35,6 +35,7 @@
#include "nsterm.h"
#include "macfont.h"
#include "macuvs.h"
+#include "pdumper.h"
#include <libkern/OSByteOrder.h>
@@ -1032,12 +1033,12 @@ sorted in the canonical order (CTFontManagerCompareFontFamilyNames on
static void
macfont_init_font_change_handler (void)
{
- static bool initialized = false;
+ static bool xinitialized = false;
- if (initialized)
+ if (xinitialized)
return;
- initialized = true;
+ xinitialized = true;
CFNotificationCenterAddObserver
(CFNotificationCenterGetLocalCenter (), NULL,
macfont_handle_font_change_notification,
@@ -4054,13 +4055,12 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
}
\f
+
void
syms_of_macfont (void)
{
/* Core Text, for macOS. */
DEFSYM (Qmac_ct, "mac-ct");
- macfont_driver.type = Qmac_ct;
- register_font_driver (&macfont_driver, NULL);
/* The font property key specifying the font design destination. The
value is an unsigned integer code: 0 for WYSIWYG, and 1 for Video
@@ -4075,4 +4075,18 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
macfont_family_cache = Qnil;
staticpro (&macfont_family_cache);
+
+ pdumper_do_now_and_after_load (syms_of_macfont_for_pdumper);
+}
+
+static void
+syms_of_macfont_for_pdumper (void)
+{
+ if (RESTORING_FROM_DUMP)
+ macfont_family_cache = Qnil;
+ else
+ eassert (NILP (macfont_family_cache));
+
+ macfont_driver.type = Qmac_ct;
+ register_font_driver (&macfont_driver, NULL);
}
diff --git a/src/menu.c b/src/menu.c
index 638810b..d530689 100644
--- a/src/menu.c
+++ b/src/menu.c
@@ -1574,9 +1574,10 @@ for instance using the window manager, then this produces a quit and
void
syms_of_menu (void)
{
- staticpro (&menu_items);
menu_items = Qnil;
+ staticpro (&menu_items);
menu_items_inuse = Qnil;
+ staticpro (&menu_items_inuse);
defsubr (&Sx_popup_menu);
defsubr (&Sx_popup_dialog);
diff --git a/src/minibuf.c b/src/minibuf.c
index 57eea05..ffd1d9c 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -32,6 +32,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "window.h"
#include "keymap.h"
#include "systty.h"
+#include "pdumper.h"
/* List of buffers for use as minibuffers.
The first element of the list is used for the outermost minibuffer
@@ -1888,21 +1889,36 @@ If no minibuffer is active, return nil. */)
}
\f
+
+static void init_minibuf_once_for_pdumper (void);
+
void
init_minibuf_once (void)
{
- Vminibuffer_list = Qnil;
staticpro (&Vminibuffer_list);
+ pdumper_do_now_and_after_load (init_minibuf_once_for_pdumper);
}
-void
-syms_of_minibuf (void)
+static void
+init_minibuf_once_for_pdumper (void)
{
+ PDUMPER_IGNORE (minibuf_level);
+ PDUMPER_IGNORE (minibuf_prompt_width);
+
+ /* We run this function on first initialization and whenever we
+ restore from a pdumper image. pdumper doesn't try to preserve
+ frames, windows, and so on, so reset everything related here. */
+ Vminibuffer_list = Qnil;
minibuf_level = 0;
minibuf_prompt = Qnil;
- staticpro (&minibuf_prompt);
-
minibuf_save_list = Qnil;
+ last_minibuf_string = Qnil;
+}
+
+void
+syms_of_minibuf (void)
+{
+ staticpro (&minibuf_prompt);
staticpro (&minibuf_save_list);
DEFSYM (Qcompletion_ignore_case, "completion-ignore-case");
@@ -1912,7 +1928,6 @@ syms_of_minibuf (void)
DEFSYM (Qminibuffer_completion_table, "minibuffer-completion-table");
staticpro (&last_minibuf_string);
- last_minibuf_string = Qnil;
DEFSYM (Qminibuffer_history, "minibuffer-history");
DEFSYM (Qbuffer_name_history, "buffer-name-history");
diff --git a/src/nsfns.m b/src/nsfns.m
index cfaaf53..340fc9c 100644
--- a/src/nsfns.m
+++ b/src/nsfns.m
@@ -49,7 +49,6 @@ Updated by Christian Limpach (chris@nice.ch)
#include "macfont.h"
#endif
-
#ifdef HAVE_NS
static EmacsTooltip *ns_tooltip = nil;
@@ -3068,6 +3067,11 @@ - (NSString *)panel: (id)sender userEnteredFilename: (NSString *)filename
========================================================================== */
+static void
+syms_of_nsfns_1 (void)
+{
+
+}
void
syms_of_nsfns (void)
@@ -3147,5 +3151,6 @@ - (NSString *)panel: (id)sender userEnteredFilename: (NSString *)filename
as_status = 0;
as_script = Qnil;
+ staticpro (&as_script);
as_result = 0;
}
diff --git a/src/nsfont.m b/src/nsfont.m
index 389d0ed..236bda2 100644
--- a/src/nsfont.m
+++ b/src/nsfont.m
@@ -36,6 +36,7 @@
#include "character.h"
#include "font.h"
#include "termchar.h"
+#include "pdumper.h"
/* TODO: Drop once we can assume gnustep-gui 0.17.1. */
#ifdef NS_IMPL_GNUSTEP
@@ -1520,12 +1521,11 @@ - (void)setIntAttribute: (NSInteger)attributeTag value: (NSInteger)val
fprintf (stderr, "\n");
}
+static void syms_of_nsfont_for_pdumper (void);
void
syms_of_nsfont (void)
{
- nsfont_driver.type = Qns;
- register_font_driver (&nsfont_driver, NULL);
DEFSYM (Qcondensed, "condensed");
DEFSYM (Qexpanded, "expanded");
DEFSYM (Qapple, "apple");
@@ -1533,5 +1533,13 @@ - (void)setIntAttribute: (NSInteger)attributeTag value: (NSInteger)val
DEFVAR_LISP ("ns-reg-to-script", Vns_reg_to_script,
doc: /* Internal use: maps font registry to Unicode script. */);
+ pdumper_do_now_and_after_load (syms_of_nsfont_for_pdumper);
+}
+
+static void
+syms_of_nsfont_for_pdumper (void)
+{
+ nsfont_driver.type = Qns;
+ register_font_driver (&nsfont_driver, NULL);
ascii_printable = NULL;
}
diff --git a/src/nsmenu.m b/src/nsmenu.m
index 3e9887a..ed2e91e 100644
--- a/src/nsmenu.m
+++ b/src/nsmenu.m
@@ -37,6 +37,7 @@
#include "termhooks.h"
#include "keyboard.h"
#include "menu.h"
+#include "pdumper.h"
#define NSMENUPROFILE 0
@@ -1883,6 +1884,7 @@ - (Lisp_Object)runDialogAt: (NSPoint)p
/* Don't know how to keep track of this in Next/Open/GNUstep. Always
update menus there. */
trackingMenu = 1;
+ PDUMPER_REMEMBER_SCALAR (trackingMenu);
#endif
defsubr (&Sns_reset_menu);
defsubr (&Smenu_or_popup_active_p);
diff --git a/src/nsterm.m b/src/nsterm.m
index 7e6ec85..a7a62cb 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -59,6 +59,7 @@ Updated by Christian Limpach (chris@nice.ch)
#include "keyboard.h"
#include "buffer.h"
#include "font.h"
+#include "pdumper.h"
#ifdef NS_IMPL_GNUSTEP
#include "process.h"
@@ -8620,6 +8621,7 @@ Convert an X font name (XLFD) to an NS font name.
NSTRACE ("syms_of_nsterm");
ns_antialias_threshold = 10.0;
+ PDUMPER_REMEMBER_SCALAR (ns_antialias_threshold);
/* from 23+ we need to tell emacs what modifiers there are.. */
DEFSYM (Qmodifier_value, "modifier-value");
diff --git a/src/pdumper.c b/src/pdumper.c
new file mode 100644
index 0000000..2efbad9
--- /dev/null
+++ b/src/pdumper.c
@@ -0,0 +1,3197 @@
+#include <config.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/param.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "blockinput.h"
+#include "buffer.h"
+#include "charset.h"
+#include "coding.h"
+#include "frame.h"
+#include "getpagesize.h"
+#include "intervals.h"
+#include "lisp.h"
+#include "pdumper.h"
+#include "window.h"
+#include "fingerprint.h"
+
+/* We require an architecture in which all pointers are the same size
+ and have the same layout. */
+verify (sizeof (ptrdiff_t) == sizeof (void*));
+verify (sizeof (void (*)(void)) == sizeof (void*));
+verify (sizeof (ptrdiff_t) <= sizeof (Lisp_Object));
+verify (sizeof (ptrdiff_t) <= sizeof (EMACS_INT));
+
+bool pdumper_loading_dump;
+
+static const char dump_magic[16] = {
+ 'D', 'U', 'M', 'P', 'E', 'D',
+ 'G', 'N', 'U',
+ 'E', 'M', 'A', 'C', 'S'
+};
+
+static pdumper_hook dump_hooks[24];
+static int nr_dump_hooks = 0;
+
+static struct
+{
+ void *mem;
+ int sz;
+} remembered_data[32];
+static int nr_remembered_data = 0;
+
+/* Maximum number of cons cells in a list to print in a contiguous
+ chunk before going back to the normal dumping strategy. */
+static const ptrdiff_t max_cons_chain_depth = 256;
+
+typedef int32_t dump_off_t;
+
+enum dump_reloc_type
+ {
+ /* dump_ptr = dump_ptr + emacs_basis() */
+ RELOC_DUMP_TO_EMACS_RAW_PTR,
+ /* dump_ptr = dump_ptr + dump_base */
+ RELOC_DUMP_TO_DUMP_RAW_PTR,
+ /* dump_lv = make_lisp_ptr (
+ dump_lv + dump_base,
+ type - RELOC_DUMP_TO_DUMP_LV)
+ (Special case for symbols: make_lisp_symbol)
+ Must be second-last. */
+ RELOC_DUMP_TO_DUMP_LV,
+ /* dump_lv = make_lisp_ptr (
+ dump_lv + emacs_basis(),
+ type - RELOC_DUMP_TO_DUMP_LV)
+ (Special case for symbols: make_lisp_symbol.)
+ Must be last. */
+ RELOC_DUMP_TO_EMACS_LV = RELOC_DUMP_TO_DUMP_LV + 8,
+ };
+
+enum emacs_reloc_type
+ {
+ /* Copy raw bytes from the dump into Emacs. */
+ RELOC_EMACS_COPY_FROM_DUMP,
+ /* Set a piece of memory in Emacs to a value we store directly in
+ this relocation. The length field contains the number of bytes
+ we actually copy into Emacs. */
+ RELOC_EMACS_IMMEDIATE,
+ /* Set an aligned pointer-sized object in Emacs to a dump offset. */
+ RELOC_EMACS_DUMP_PTR_RAW,
+ /* Set an aligned pointer-sized object in Emacs to point to
+ something also in Emacs. */
+ RELOC_EMACS_EMACS_PTR_RAW,
+ /* Set an aligned Lisp_Object in Emacs to point to a value in the
+ dump. Must be last. */
+ RELOC_EMACS_DUMP_LV,
+ };
+
+#define EMACS_RELOC_TYPE_BITS 3
+#define EMACS_RELOC_LENGTH_BITS \
+ (sizeof (dump_off_t) * 8 - EMACS_RELOC_TYPE_BITS)
+
+struct emacs_reloc
+{
+ ENUM_BF (emacs_reloc_type) type : EMACS_RELOC_TYPE_BITS;
+ dump_off_t length : EMACS_RELOC_LENGTH_BITS;
+ dump_off_t emacs_offset;
+ union
+ {
+ dump_off_t dump_offset;
+ dump_off_t emacs_offset2;
+ intmax_t immediate;
+ int8_t immediate_i8;
+ int16_t immediate_i16;
+ int32_t immediate_i32;
+ } u;
+};
+
+struct dump_table_locator
+{
+ dump_off_t offset;
+ dump_off_t nr_entries;
+};
+
+struct dump_reloc
+{
+ // XXX: We have a ton of these. Combine type and offset into one
+ // 32-bit word. Force alignment.
+ enum dump_reloc_type type;
+ dump_off_t offset;
+};
+
+/* Format of an Emacs portable dump file. All offsets are relative to
+ the beginning of the file. An Emacs portable dump file is coupled
+ to exactly the Emacs binary that produced it, so details of
+ alignment and endianness are unimportant.
+
+ An Emacs dump file contains the contents of the Lisp heap.
+ On startup, Emacs can start faster by mapping a dump file into
+ memory and using the objects contained inside it instead of
+ performing initialization from scratch.
+
+ The dump file can be loaded at arbitrary locations in memory, so it
+ includes a table of relocations that let Emacs adjust the pointers
+ embedded in the dump file to account for the location where it was
+ actually loaded.
+
+ Dump files can contain pointers to other objects in the dump file
+ or to parts of the Emacs binary. */
+struct dump_header
+{
+ /* File type magic. */
+ char magic[sizeof (dump_magic)];
+
+ /* Associated Emacs binary. */
+ uint8_t fingerprint[32];
+
+ /* Relocation table for the dump file; each entry is a
+ struct dump_reloc. */
+ struct dump_table_locator dump_relocs;
+
+ /* "Relocation" table we abuse to hold information about the
+ location and type of each lisp object in the dump. We need for
+ pdumper_object_type and ultimately for conservative GC. */
+ struct dump_table_locator object_starts;
+
+ /* Relocation table for Emacs; each entry is a struct
+ emacs_reloc. */
+ struct dump_table_locator emacs_relocs;
+
+ /* Start of sub-region of hot region that we can discard after load
+ completes. The discardable region ends at hot_end. */
+ dump_off_t hot_discardable_start;
+
+ /* End of the region that we expect to have many relocations. */
+ dump_off_t hot_end;
+
+};
+
+struct dump_tailq
+{
+ Lisp_Object head;
+ Lisp_Object tail;
+};
+
+enum cold_op
+ {
+ COLD_OP_OBJECT,
+ COLD_OP_STRING,
+ COLD_OP_CHARSET,
+ COLD_OP_BUFFER,
+ };
+
+/* Information we use while we dump. Note that we're not the garbage
+ collector and can operate under looser constraints: specifically,
+ we allocate memory during the dumping process. */
+struct dump_context
+{
+ /* Header we'll write to the dump file when done. */
+ struct dump_header header;
+
+ Lisp_Object old_purify_flag;
+ Lisp_Object old_post_gc_hook;
+
+ /* File descriptor for dumpfile; < 0 if closed. */
+ int fd;
+ /* Name of dump file --- used for error reporting. */
+ Lisp_Object dump_filename;
+ /* Current offset in dump file. */
+ ptrdiff_t offset;
+
+ /* Starting offset of current object. */
+ ptrdiff_t obj_offset;
+ /* Flags for writing the current object. */
+ int flags;
+ /* Depth of cons-chain dumping. */
+ ptrdiff_t cons_chain_depth;
+
+ ptrdiff_t end_heap;
+
+ /* Hash mapping objects we've already dumped to their offsets. */
+ Lisp_Object objects_dumped;
+
+ /* Hash mapping objects to where we got them. Used for debugging. */
+ Lisp_Object referrers;
+ Lisp_Object current_referrer;
+
+#ifdef ENABLE_CHECKING
+ bool have_current_referrer;
+#endif
+
+ /* Queue of objects to dump. */
+ struct dump_tailq dump_queue;
+ /* Fixups in the dump file. */
+ Lisp_Object fixups;
+ /* Queue of copied objects for special treatment. */
+ Lisp_Object copied_queue;
+ /* Queue of cold objects to dump. */
+ Lisp_Object cold_queue;
+
+ /* Relocations in the dump. */
+ Lisp_Object dump_relocs;
+ /* Object starts. */
+ Lisp_Object object_starts;
+ /* Relocations in Emacs. */
+ Lisp_Object emacs_relocs;
+};
+
+\f
+
+#define DUMP_OBJECT_INTERN (1<<0)
+#define DUMP_OBJECT_RECORD_START (1<<1)
+#define DUMP_OBJECT_DRY_RUN (1<<2)
+#define DUMP_OBJECT_FORCE_WORD_ALIGNMENT (1<<3)
+#define DUMP_OBJECT_PROHIBIT_ENQUEUE (1<<4)
+
+static ptrdiff_t dump_object (struct dump_context *ctx, Lisp_Object object);
+static ptrdiff_t dump_object_1 (struct dump_context *ctx,
+ Lisp_Object object,
+ int flags);
+
+static void
+dump_push (Lisp_Object *where, Lisp_Object newelt)
+{
+ *where = Fcons (newelt, *where);
+}
+
+static Lisp_Object
+dump_pop (Lisp_Object *where)
+{
+ Lisp_Object ret = XCAR (*where);
+ *where = XCDR (*where);
+ return ret;
+}
+
+static bool
+dump_tracking_referrers_p (struct dump_context *ctx)
+{
+ return !NILP (ctx->referrers);
+}
+
+static void
+dump_set_have_current_referrer (struct dump_context *ctx, bool have)
+{
+#ifdef ENABLE_CHECKING
+ ctx->have_current_referrer = have;
+#endif
+}
+
+/* Define as a macro so we can avoid evaluating OBJECT
+ if we dont want referrer tracking. */
+#define DUMP_SET_REFERRER(ctx, object) \
+ do { \
+ struct dump_context *_ctx = (ctx); \
+ eassert (!_ctx->have_current_referrer); \
+ dump_set_have_current_referrer (_ctx, true); \
+ if (dump_tracking_referrers_p (_ctx)) \
+ ctx->current_referrer = (object); \
+ } while (0);
+
+static void
+dump_clear_referrer (struct dump_context *ctx)
+{
+ eassert (ctx->have_current_referrer);
+ dump_set_have_current_referrer (ctx, false);
+ if (dump_tracking_referrers_p (ctx))
+ ctx->current_referrer = Qnil;
+}
+
+static Lisp_Object
+dump_ptr_referrer (const char *label, void *address)
+{
+ char buf[128];
+ buf[0] = '\0';
+ sprintf (buf, "%s @ %p", label, address);
+ return build_string (buf);
+}
+
+static void
+print_paths_to_root (struct dump_context *ctx, Lisp_Object object);
+
+static void dump_remember_cold_op (struct dump_context *ctx,
+ enum cold_op op,
+ Lisp_Object arg);
+
+_Noreturn
+static void
+error_unsupported_dump_object (struct dump_context *ctx,
+ Lisp_Object object,
+ const char* msg)
+{
+ if (dump_tracking_referrers_p (ctx))
+ print_paths_to_root (ctx, object);
+ error ("unsupported object type in dump: %s", msg);
+}
+
+static ptrdiff_t
+emacs_basis (void)
+{
+ return (ptrdiff_t) &Vpurify_flag;
+}
+
+static ptrdiff_t
+emacs_offset (const void *emacs_ptr)
+{
+ /* TODO: assert that emacs_ptr is actually in emacs */
+ eassert (emacs_ptr != NULL);
+ ptrdiff_t emacs_ptr_value = (ptrdiff_t) emacs_ptr;
+ ptrdiff_t emacs_ptr_relative = emacs_ptr_value - emacs_basis ();
+ return emacs_ptr_relative;
+}
+
+/* Return whether OBJECT is a symbol the storage of which is built
+ into Emacs (and so is invariant across ASLR). */
+static bool
+dump_builtin_symbol_p (Lisp_Object object)
+{
+ if (!SYMBOLP (object))
+ return false;
+ char* bp = (char*) lispsym;
+ struct Lisp_Symbol *s = XSYMBOL (object);
+ char* sp = (char*) s;
+ return bp <= sp && sp < bp + sizeof (lispsym);
+}
+
+/* Return whether OBJECT has the same bit pattern in all Emacs
+ invocations --- i.e., is invariant across a dump. */
+static bool
+dump_object_self_representing_p (Lisp_Object object)
+{
+ return INTEGERP (object) || dump_builtin_symbol_p (object);
+}
+
+#define DEFINE_FROMLISP_FUNC(fn, type) \
+ static type \
+ fn (Lisp_Object value) \
+ { \
+ type result; \
+ CONS_TO_INTEGER (value, type, result); \
+ return result; \
+ }
+
+DEFINE_FROMLISP_FUNC (intmax_t_from_lisp, intmax_t);
+DEFINE_FROMLISP_FUNC (ptrdiff_t_from_lisp, ptrdiff_t);
+DEFINE_FROMLISP_FUNC (dump_off_from_lisp, dump_off_t);
+
+static void
+dump_tailq_init (struct dump_tailq *tailq)
+{
+ tailq->head = tailq->tail = Qnil;
+}
+
+static void
+dump_tailq_append (struct dump_tailq *tailq, Lisp_Object value)
+{
+ Lisp_Object link = Fcons (value, Qnil);
+ if (NILP (tailq->head))
+ {
+ eassert (NILP (tailq->tail));
+ tailq->head = tailq->tail = link;
+ }
+ else
+ {
+ eassert (!NILP (tailq->tail));
+ XSETCDR (tailq->tail, link);
+ tailq->tail = link;
+ }
+}
+
+static bool
+dump_tailq_empty_p (struct dump_tailq *tailq)
+{
+ return NILP (tailq->head);
+}
+
+static Lisp_Object
+dump_tailq_pop (struct dump_tailq *tailq)
+{
+ eassert (!dump_tailq_empty_p (tailq));
+ Lisp_Object value = XCAR (tailq->head);
+ tailq->head = XCDR (tailq->head);
+ if (NILP (tailq->head))
+ tailq->tail = Qnil;
+ return value;
+}
+
+static void
+dump_write (struct dump_context *ctx, const void *buf, ptrdiff_t nbyte)
+{
+ eassert (nbyte == 0 || buf != NULL);
+ eassert (ctx->obj_offset == 0);
+ eassert ((ctx->flags & DUMP_OBJECT_DRY_RUN) == 0);
+ if (emacs_write (ctx->fd, buf, nbyte) < nbyte)
+ report_file_error ("Could not write to dump file", ctx->dump_filename);
+ ctx->offset += nbyte;
+}
+
+static void
+dump_seek (struct dump_context *ctx, ptrdiff_t offset)
+{
+ eassert (ctx->obj_offset == 0);
+ if (lseek (ctx->fd, offset, SEEK_SET) < 0)
+ report_file_error ("Setting file position",
+ ctx->dump_filename);
+ ctx->offset = offset;
+}
+
+static void
+dump_write_zero (struct dump_context *ctx, ptrdiff_t nbytes)
+{
+ while (nbytes > 0)
+ {
+ ptrdiff_t zero = 0;
+ ptrdiff_t to_write = sizeof (zero);
+ if (to_write > nbytes)
+ to_write = nbytes;
+ dump_write (ctx, &zero, to_write);
+ nbytes -= to_write;
+ }
+}
+
+static void
+dump_align_output (struct dump_context *ctx, ptrdiff_t alignment)
+{
+ if (ctx->offset % alignment != 0)
+ dump_write_zero (ctx, alignment - (ctx->offset % alignment));
+}
+
+static ptrdiff_t
+dump_object_start (struct dump_context *ctx,
+ int alignment,
+ void *out,
+ ptrdiff_t outsz)
+{
+ eassert (ctx->obj_offset == 0);
+ if ((ctx->flags & DUMP_OBJECT_FORCE_WORD_ALIGNMENT) &&
+ alignment > sizeof (void*))
+ alignment = sizeof (void*);
+ if ((ctx->flags & DUMP_OBJECT_DRY_RUN) == 0)
+ dump_align_output (ctx, alignment);
+ ctx->obj_offset = ctx->offset;
+ memset (out, 0, outsz);
+ return ctx->offset;
+}
+
+static ptrdiff_t
+dump_object_finish (struct dump_context *ctx,
+ const void *out,
+ ptrdiff_t sz)
+{
+ ptrdiff_t offset = ctx->obj_offset;
+ eassert (offset > 0);
+ eassert (offset == ctx->offset); /* No intervening writes. */
+ ctx->obj_offset = 0;
+ if ((ctx->flags & DUMP_OBJECT_DRY_RUN) == 0)
+ dump_write (ctx, out, sz);
+ return offset;
+}
+
+/* Return offset at which OBJECT has been dumped, or 0 if OBJECT has
+ not been dumped. */
+static ptrdiff_t
+dump_recall_object (struct dump_context *ctx, Lisp_Object object)
+{
+ Lisp_Object dumped = ctx->objects_dumped;
+ return ptrdiff_t_from_lisp (Fgethash (object, dumped, make_number (0)));
+}
+
+static void
+dump_remember_object (struct dump_context *ctx,
+ Lisp_Object object,
+ ptrdiff_t offset)
+{
+ Fputhash (object, INTEGER_TO_CONS (offset), ctx->objects_dumped);
+}
+
+static void
+dump_note_reachable (struct dump_context *ctx, Lisp_Object object)
+{
+ eassert (ctx->have_current_referrer);
+ if (!dump_tracking_referrers_p (ctx))
+ return;
+ Lisp_Object referrer = ctx->current_referrer;
+ Lisp_Object obj_referrers = Fgethash (object, ctx->referrers, Qnil);
+ if (NILP (Fmemq (referrer, obj_referrers)))
+ Fputhash (object, Fcons (referrer, obj_referrers), ctx->referrers);
+}
+
+/* If this object lives in the Emacs image and not on the heap, return
+ a pointer to the object data. Otherwise, return NULL. */
+static void*
+dump_object_emacs_ptr (Lisp_Object lv)
+{
+ if (SUBRP (lv))
+ return XSUBR (lv);
+ if (dump_builtin_symbol_p (lv))
+ return XSYMBOL (lv);
+ return NULL;
+}
+
+static void
+dump_enqueue_object (struct dump_context *ctx, Lisp_Object object)
+{
+ if ((!dump_object_self_representing_p (object) ||
+ dump_object_emacs_ptr (object)) &&
+ dump_recall_object (ctx, object) == 0)
+ {
+ eassert ((ctx->flags & DUMP_OBJECT_PROHIBIT_ENQUEUE) == 0);
+
+ dump_remember_object (ctx, object, -1);
+ if (BOOL_VECTOR_P (object) || FLOATP (object))
+ dump_remember_cold_op (ctx, COLD_OP_OBJECT, object);
+ else
+ dump_tailq_append (&ctx->dump_queue, object);
+ }
+ dump_note_reachable (ctx, object);
+}
+
+static void
+print_paths_to_root_1 (struct dump_context *ctx,
+ Lisp_Object object,
+ int level)
+{
+ Lisp_Object referrers = Fgethash (object, ctx->referrers, Qnil);
+ while (!NILP (referrers))
+ {
+ Lisp_Object referrer = XCAR (referrers);
+ referrers = XCDR (referrers);
+ Lisp_Object repr = Fprin1_to_string (referrer, Qnil);
+ for (int i = 0; i < level; ++i)
+ fputc (' ', stderr);
+ fprintf (stderr, "%s\n", SDATA (repr));
+ print_paths_to_root_1 (ctx, referrer, level + 1);
+ }
+}
+
+static void
+print_paths_to_root (struct dump_context *ctx, Lisp_Object object)
+{
+ print_paths_to_root_1 (ctx, object, 0);
+}
+
+static void
+dump_remember_cold_op (struct dump_context *ctx,
+ enum cold_op op,
+ Lisp_Object arg)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+ dump_push (&ctx->cold_queue, Fcons (make_number (op), arg));
+}
+
+/* Add a dump relocation that points into Emacs.
+
+ Add a relocation that updates the pointer stored at DUMP_OFFSET to
+ point into the Emacs binary upon dump load. The pointer-sized
+ value at DUMP_OFFSET in the dump file should contain a number
+ relative to emacs_basis(). */
+static void
+dump_reloc_dump_to_emacs_raw_ptr (struct dump_context *ctx,
+ ptrdiff_t dump_offset)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+ dump_push (&ctx->dump_relocs,
+ list2 (INTEGER_TO_CONS (RELOC_DUMP_TO_EMACS_RAW_PTR),
+ INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add a dump relocation that points a Lisp_Object back at the dump.
+
+ Add a relocation that updates the Lisp_Object at DUMP_OFFSET in the
+ dump to point to another object in the dump. The Lisp_Object-sized
+ value at DUMP_OFFSET in the dump file should contain the offset of
+ the target object relative to the start of the dump. */
+static void
+dump_reloc_dump_to_dump_lv (struct dump_context *ctx,
+ ptrdiff_t dump_offset,
+ enum Lisp_Type type)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ ptrdiff_t reloc_type;
+ switch (type)
+ {
+ case Lisp_Symbol:
+ case Lisp_Misc:
+ case Lisp_String:
+ case Lisp_Vectorlike:
+ case Lisp_Cons:
+ case Lisp_Float:
+ reloc_type = RELOC_DUMP_TO_DUMP_LV + type;
+ break;
+ default:
+ emacs_abort ();
+ }
+
+ dump_push (&ctx->dump_relocs,
+ list2 (INTEGER_TO_CONS (reloc_type),
+ INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add a dump relocation that points a raw pointer back at the dump.
+
+ Add a relocation that updates the raw pointer at DUMP_OFFSET in the
+ dump to point to another object in the dump. The pointer-sized
+ value at DUMP_OFFSET in the dump file should contain the offset of
+ the target object relative to the start of the dump. */
+static void
+dump_reloc_dump_to_dump_raw_ptr (struct dump_context *ctx,
+ ptrdiff_t dump_offset)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ dump_push (&ctx->dump_relocs,
+ list2 (INTEGER_TO_CONS (RELOC_DUMP_TO_DUMP_RAW_PTR),
+ INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add a dump relocation that points to a Lisp object in Emacs.
+
+ Add a relocation that updates the Lisp_Object at DUMP_OFFSET in the
+ dump to point to a lisp object in Emacs. The Lisp_Object-sized
+ value at DUMP_OFFSET in the dump file should contain the offset of
+ the target object relative to emacs_basis(). TYPE is the type of
+ Lisp value. */
+static void
+dump_reloc_dump_to_emacs_lv (struct dump_context *ctx,
+ ptrdiff_t dump_offset,
+ enum Lisp_Type type)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ ptrdiff_t reloc_type;
+ switch (type)
+ {
+ case Lisp_Misc:
+ case Lisp_String:
+ case Lisp_Vectorlike:
+ case Lisp_Cons:
+ case Lisp_Float:
+ reloc_type = RELOC_DUMP_TO_EMACS_LV + type;
+ break;
+ default:
+ emacs_abort ();
+ }
+
+ dump_push (&ctx->dump_relocs,
+ list2 (INTEGER_TO_CONS (reloc_type),
+ INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add an Emacs relocation that copies arbitrary bytes from the dump.
+
+ When the dump is loaded, Emacs copies SIZE bytes from OFFSET in
+ dump to LOCATION in the Emacs data section. This copying happens
+ after other relocations, so it's all right to, say, copy a
+ Lisp_Value (since by the time we copy the Lisp_Value, it'll have
+ been adjusted to account for the location of the running Emacs and
+ dump file). */
+static void
+dump_emacs_reloc_copy_from_dump (struct dump_context *ctx,
+ ptrdiff_t dump_offset,
+ void* emacs_ptr,
+ ptrdiff_t size)
+{
+ eassert (size >= 0);
+ eassert (size < (1 << EMACS_RELOC_LENGTH_BITS));
+
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ if (size == 0)
+ return;
+
+ dump_push (&ctx->emacs_relocs,
+ list4 (make_number (RELOC_EMACS_COPY_FROM_DUMP),
+ INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+ INTEGER_TO_CONS (dump_offset),
+ INTEGER_TO_CONS (size)));
+}
+
+/* Add an Emacs relocation that sets values to arbitrary bytes.
+
+ When the dump is loaded, Emacs copies SIZE bytes from the
+ relocation itself to the adjusted location inside Emacs EMACS_PTR.
+ SIZE is the number of bytes to copy. See struct emacs_reloc for
+ the maximum size that this mechanism can support. The value comes
+ from VALUE_PTR.
+ */
+static void
+dump_emacs_reloc_immediate (struct dump_context *ctx,
+ const void *emacs_ptr,
+ const void *value_ptr,
+ ptrdiff_t size)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ intmax_t value = 0;
+ eassert (size <= sizeof (value));
+ memcpy (&value, value_ptr, size);
+ dump_push (&ctx->emacs_relocs,
+ list4 (make_number (RELOC_EMACS_IMMEDIATE),
+ INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+ INTEGER_TO_CONS (value),
+ INTEGER_TO_CONS (size)));
+}
+
+#define DEFINE_EMACS_IMMEDIATE_FN(fnname, type) \
+ static void \
+ fnname (struct dump_context *ctx, \
+ const type *emacs_ptr, \
+ type value) \
+ { \
+ dump_emacs_reloc_immediate ( \
+ ctx, emacs_ptr, &value, sizeof (value)); \
+ }
+
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_lv, Lisp_Object);
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_ptrdiff_t, ptrdiff_t);
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_emacs_int, EMACS_INT);
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_int, int);
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_bool, bool);
+
+/* Add an emacs relocation that makes a raw pointer in Emacs point
+ into the dump. */
+static void
+dump_emacs_reloc_to_dump_ptr_raw (struct dump_context *ctx,
+ const void* emacs_ptr,
+ ptrdiff_t dump_offset)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ dump_push (&ctx->emacs_relocs,
+ list3 (make_number (RELOC_EMACS_DUMP_PTR_RAW),
+ INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+ INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add an emacs relocation that points into the dump.
+
+ When the dump is loaded, the Lisp_Object at EMACS_ROOT in Emacs to
+ point to VALUE. VALUE can be any Lisp value; this function
+ automatically queues the value for dumping if necessary. */
+static void
+dump_emacs_reloc_to_dump_lv (struct dump_context *ctx,
+ Lisp_Object *emacs_ptr,
+ Lisp_Object value)
+{
+ if (dump_object_self_representing_p (value))
+ dump_emacs_reloc_immediate_lv (ctx, emacs_ptr, value);
+ else
+ {
+ if ((ctx->flags & DUMP_OBJECT_DRY_RUN) == 0)
+ dump_push (
+ &ctx->emacs_relocs,
+ list3 (INTEGER_TO_CONS (RELOC_EMACS_DUMP_LV + XTYPE (value)),
+ INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+ value));
+ dump_enqueue_object (ctx, value);
+ }
+}
+
+/* Add an emacs relocation that makes a raw pointer in Emacs point
+ back into the Emacs image. */
+static void
+dump_emacs_reloc_to_emacs_ptr_raw (struct dump_context *ctx,
+ void* emacs_ptr,
+ void *target_emacs_ptr)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ dump_push (&ctx->emacs_relocs,
+ list3 (make_number (RELOC_EMACS_EMACS_PTR_RAW),
+ INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+ INTEGER_TO_CONS (emacs_offset (target_emacs_ptr))));
+}
+
+/* Add an Emacs relocation that makes a raw pointer in Emacs point to
+ a different part of Emacs. */
+
+enum dump_fixup_type
+ {
+ DUMP_FIXUP_LISP_OBJECT,
+ DUMP_FIXUP_LISP_OBJECT_RAW,
+ DUMP_FIXUP_PTR_DUMP_RAW,
+ };
+
+enum dump_lv_fixup_type
+ {
+ LV_FIXUP_LISP_OBJECT,
+ LV_FIXUP_RAW_POINTER,
+ };
+
+/* Make something in the dump point to a lisp object.
+
+ CTX is a dump context. DUMP_OFFSET is the location in the dump to
+ fix. VALUE is the object to which the location in the dump
+ should point.
+
+ If FIXUP_SUBTYPE is LV_FIXUP_LISP_OBJECT, we expect a Lisp_Object
+ at DUMP_OFFSET. If it's LV_FIXUP_RAW_POINTER, we expect a pointer.
+ */
+static void
+dump_remember_fixup_lv (struct dump_context *ctx,
+ ptrdiff_t dump_offset,
+ Lisp_Object value,
+ enum dump_lv_fixup_type fixup_subtype)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ dump_push (&ctx->fixups,
+ list3 (
+ make_number (fixup_subtype == LV_FIXUP_LISP_OBJECT
+ ? DUMP_FIXUP_LISP_OBJECT
+ : DUMP_FIXUP_LISP_OBJECT_RAW),
+ INTEGER_TO_CONS (dump_offset),
+ value));
+}
+
+/* Remember to fix up the dump file such that the pointer-sized value
+ at DUMP_OFFSET points to NEW_DUMP_OFFSET in the dump file and to
+ its absolute address at runtime. */
+static void
+dump_remember_fixup_ptr_raw (struct dump_context *ctx,
+ ptrdiff_t dump_offset,
+ ptrdiff_t new_dump_offset)
+{
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ dump_push (&ctx->fixups,
+ list3 (
+ make_number (DUMP_FIXUP_PTR_DUMP_RAW),
+ INTEGER_TO_CONS (dump_offset),
+ INTEGER_TO_CONS (new_dump_offset)));
+}
+
+static void
+dump_root_visitor (Lisp_Object *root_ptr, enum gc_root_type type, void *data)
+{
+ struct dump_context *ctx = data;
+ Lisp_Object value = *root_ptr;
+ if (type == GC_ROOT_C_SYMBOL)
+ {
+ eassert (dump_builtin_symbol_p (value));
+ /* Remember to dump the object itself later along with all the
+ rest of the copied-to-Emacs objects. */
+ DUMP_SET_REFERRER (ctx, build_string ("built-in symbol list"));
+ dump_enqueue_object (ctx, value);
+ dump_clear_referrer (ctx);
+ }
+ else
+ {
+ DUMP_SET_REFERRER (ctx, dump_ptr_referrer ("emacs root", root_ptr));
+ dump_emacs_reloc_to_dump_lv (ctx, root_ptr, *root_ptr);
+ dump_clear_referrer (ctx);
+ }
+}
+
+/* Kick off the dump process by queuing up the static GC roots. */
+static void
+dump_roots (struct dump_context *ctx)
+{
+ struct gc_root_visitor visitor;
+ memset (&visitor, 0, sizeof (visitor));
+ visitor.visit = dump_root_visitor;
+ visitor.data = ctx;
+ visit_static_gc_roots (visitor);
+}
+
+static ptrdiff_t
+field_relpos (const void *in_start, const void *in_field)
+{
+ ptrdiff_t in_start_val = (ptrdiff_t) in_start;
+ ptrdiff_t in_field_val = (ptrdiff_t) in_field;
+ eassert (in_start_val <= in_field_val);
+ ptrdiff_t relpos = in_field_val - in_start_val;
+ eassert (relpos < 1024); /* Sanity check. */
+ return relpos;
+}
+
+static void
+cpyptr (void *out, const void *in)
+{
+ memcpy (out, in, sizeof (void *));
+}
+
+/* Convenience macro for regular assignment. */
+#define DUMP_FIELD_COPY(out, in, name) \
+ do { \
+ (out)->name = (in)->name; \
+ } while (0)
+
+static void
+dump_field_lv_or_rawptr (struct dump_context *ctx,
+ void *out,
+ const void *in_start,
+ const void *in_field,
+ /* opt */ const enum Lisp_Type *raw_ptr_type)
+{
+ eassert (ctx->obj_offset > 0);
+
+ Lisp_Object value;
+ ptrdiff_t relpos = field_relpos (in_start, in_field);
+ void *out_field = (char *) out + relpos;
+ if (raw_ptr_type == NULL)
+ {
+ memcpy (&value, in_field, sizeof (value));
+ if (dump_object_self_representing_p (value))
+ {
+ memcpy (out_field, &value, sizeof (value));
+ return;
+ }
+ }
+ else
+ {
+ void *ptrval;
+ cpyptr (&ptrval, in_field);
+ if (ptrval == NULL)
+ return; /* Nothing to do. */
+ switch (*raw_ptr_type)
+ {
+ case Lisp_Symbol:
+ value = make_lisp_symbol (ptrval);
+ break;
+ case Lisp_Misc:
+ case Lisp_String:
+ case Lisp_Vectorlike:
+ case Lisp_Cons:
+ case Lisp_Float:
+ value = make_lisp_ptr (ptrval, *raw_ptr_type);
+ break;
+ default:
+ emacs_abort ();
+ }
+ }
+
+ /* Now value is the Lisp_Object to which we want to point whether or
+ not the field is a raw pointer (in which case we just synthesized
+ the Lisp_Object outselves) or a Lisp_Object (in which case we
+ just copied the thing). Add a fixup or relocation. */
+
+ ptrdiff_t out_value;
+ ptrdiff_t out_field_offset = ctx->obj_offset + relpos;
+ ptrdiff_t target_offset = dump_recall_object (ctx, value);
+ if (target_offset > 0)
+ {
+ /* We've already dumped the referenced object, so we can emit
+ the value and a relocation directly instead of indirecting
+ through a fixup. */
+ out_value = target_offset;
+ if (raw_ptr_type)
+ dump_reloc_dump_to_dump_raw_ptr (ctx, out_field_offset);
+ else
+ dump_reloc_dump_to_dump_lv (ctx, out_field_offset, XTYPE (value));
+ }
+ else
+ {
+ /* We don't know about the target object yet, so add a fixup.
+ When we process the fixup, we'll have dumped the target
+ object. */
+ out_value = (ptrdiff_t) 0xDEADF00D;
+ dump_remember_fixup_lv (ctx,
+ out_field_offset,
+ value,
+ ( raw_ptr_type
+ ? LV_FIXUP_RAW_POINTER
+ : LV_FIXUP_LISP_OBJECT ));
+ if (target_offset == 0)
+ dump_enqueue_object (ctx, value);
+ }
+
+ memcpy (out_field, &out_value, sizeof (out_value));
+}
+
+/* Set a pointer field on an output object during dump.
+
+ CTX is the dump context. OFFSET is the offset at which the current
+ object starts. OUT is a pointer to the dump output object.
+ IN_START is the start of the current Emacs object. IN_FIELD is a
+ pointer to the field in that object. TYPE is the type of pointer
+ to which IN_FIELD points.
+ */
+static void
+dump_field_lv_rawptr (struct dump_context *ctx,
+ void *out,
+ const void *in_start,
+ const void *in_field,
+ enum Lisp_Type type)
+{
+ dump_field_lv_or_rawptr (ctx, out, in_start, in_field, &type);
+}
+
+/* Set a Lisp_Object field on an output object during dump.
+
+ CTX is a dump context. OFFSET is the offset at which the current
+ object starts. OUT is a pointer to the dump output object.
+ IN_START is the start of the current Emacs object. IN_FIELD is a
+ pointer to a Lisp_Object field in that object.
+
+ Arrange for the dump to contain fixups and relocations such that,
+ at load time, the given field of the output object contains a valid
+ Lisp_Object pointing to the same notional object that *IN_FIELD
+ contains now.
+
+ See idomatic usage below. */
+static void
+dump_field_lv (struct dump_context *ctx,
+ void *out,
+ const void *in_start,
+ const Lisp_Object *in_field)
+{
+ dump_field_lv_or_rawptr (ctx, out, in_start, in_field, NULL);
+}
+
+/* Note that we're going to add a manual fixup for the given field
+ later. */
+static void
+dump_field_fixup_later (struct dump_context *ctx,
+ void *out,
+ const void *in_start,
+ const void *in_field)
+{
+ // TODO: more error checking
+ (void) field_relpos (in_start, in_field);
+}
+
+/* Mark an output object field, which is as wide as a poiner, as being
+ fixed up to point to a specific offset in the dump. */
+static void
+dump_field_ptr_to_dump_offset (struct dump_context *ctx,
+ void *out,
+ const void *in_start,
+ const void *in_field,
+ ptrdiff_t target_dump_offset)
+{
+ eassert (ctx->obj_offset > 0);
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ ptrdiff_t relpos = field_relpos (in_start, in_field);
+ dump_reloc_dump_to_dump_raw_ptr (ctx, ctx->obj_offset + relpos);
+ ptrdiff_t outval = target_dump_offset;
+ memcpy ((char*) out + relpos, &outval, sizeof (outval));
+}
+
+/* Mark a field as pointing to a place inside Emacs.
+
+ CTX is the dump context. OUT points to the out-object for the
+ current dump function. IN_START points to the start of the object
+ being dumped. IN_FIELD points to the field inside the object being
+ dumped that we're dumping. The contents of this field (which
+ should be as wide as a pointer) are the Emacs pointer to dump.
+
+ */
+static void
+dump_field_emacs_ptr (struct dump_context *ctx,
+ void *out,
+ const void *in_start,
+ const void *in_field)
+{
+ eassert (ctx->obj_offset > 0);
+ if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+ return;
+
+ ptrdiff_t abs_emacs_ptr;
+ cpyptr (&abs_emacs_ptr, in_field);
+ ptrdiff_t rel_emacs_ptr = abs_emacs_ptr - emacs_basis ();
+ ptrdiff_t relpos = field_relpos (in_start, in_field);
+ cpyptr ((char*) out + relpos, &rel_emacs_ptr);
+ dump_reloc_dump_to_emacs_raw_ptr (ctx, ctx->obj_offset + relpos);
+}
+
+static ptrdiff_t
+dump_cons (struct dump_context *ctx, const struct Lisp_Cons *cons)
+{
+ struct Lisp_Cons out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ dump_field_lv (ctx, &out, cons, &cons->car);
+ dump_field_lv (ctx, &out, cons, &cons->u.cdr);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_interval_tree (struct dump_context *ctx,
+ INTERVAL tree,
+ ptrdiff_t parent_offset)
+{
+ // TODO: output tree breadth-first?
+ struct interval out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, tree, total_length);
+ DUMP_FIELD_COPY (&out, tree, position);
+ if (tree->left)
+ dump_field_fixup_later (ctx, &out, tree, &tree->left);
+ if (tree->right)
+ dump_field_fixup_later (ctx, &out, tree, &tree->right);
+ if (!tree->up_obj)
+ {
+ eassert (parent_offset != 0);
+ dump_field_ptr_to_dump_offset (
+ ctx, &out,
+ tree, &tree->up.interval,
+ parent_offset);
+ }
+ else
+ dump_field_lv (ctx, &out, tree, &tree->up.obj);
+ DUMP_FIELD_COPY (&out, tree, up_obj);
+ eassert (tree->gcmarkbit == 0);
+ DUMP_FIELD_COPY (&out, tree, write_protect);
+ DUMP_FIELD_COPY (&out, tree, visible);
+ DUMP_FIELD_COPY (&out, tree, front_sticky);
+ DUMP_FIELD_COPY (&out, tree, rear_sticky);
+ dump_field_lv (ctx, &out, tree, &tree->plist);
+ ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+ if (tree->left)
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ offset + offsetof (struct interval, left),
+ dump_interval_tree (ctx, tree->left, offset));
+ if (tree->right)
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ offset + offsetof (struct interval, right),
+ dump_interval_tree (ctx, tree->right, offset));
+ return offset;
+}
+
+static ptrdiff_t
+dump_string (struct dump_context *ctx, const struct Lisp_String *string)
+{
+ /* If we have text properties, write them _after_ the string so that
+ at runtime, the prefetcher and cache will DTRT. (We access the
+ string before its properties.).
+
+ There's special code to dump string data contiguously later on.
+ we seldom write to string data and never relocate it, so lumping
+ it together at the end of the dump saves on COW faults.
+
+ If, however, the string's size_byte field is -1, the string data
+ is actually a pointer to Emacs data segment, so we can do even
+ better by emitting a relocation instead of bothering to copy the
+ string data. */
+ struct Lisp_String out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, string, size);
+ DUMP_FIELD_COPY (&out, string, size_byte);
+ if (string->intervals)
+ dump_field_fixup_later (ctx, &out, string, &string->intervals);
+
+ if (string->size_byte == -2)
+ /* String literal in Emacs rodata. */
+ dump_field_emacs_ptr (ctx, &out, string, &string->data);
+ else
+ {
+ dump_field_fixup_later (ctx, &out, string, &string->data);
+ dump_remember_cold_op (ctx,
+ COLD_OP_STRING,
+ make_lisp_ptr ((void*) string, Lisp_String));
+ }
+
+ ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+ if (string->intervals)
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ offset + offsetof (struct Lisp_String, intervals),
+ dump_interval_tree (ctx, string->intervals, 0));
+
+ return offset;
+}
+
+static ptrdiff_t
+dump_marker (struct dump_context *ctx, const struct Lisp_Marker *marker)
+{
+ struct Lisp_Marker out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, marker, type);
+ eassert (marker->gcmarkbit == 0);
+ (void) marker->spacer; /* Do not write padding. */
+ DUMP_FIELD_COPY (&out, marker, need_adjustment);
+ DUMP_FIELD_COPY (&out, marker, insertion_type);
+ if (marker->buffer)
+ {
+ dump_field_lv_rawptr (
+ ctx, &out,
+ marker, &marker->buffer,
+ Lisp_Vectorlike);
+ dump_field_lv_rawptr (
+ ctx, &out,
+ marker, &marker->next,
+ Lisp_Misc);
+ DUMP_FIELD_COPY (&out, marker, charpos);
+ DUMP_FIELD_COPY (&out, marker, bytepos);
+ }
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_overlay (struct dump_context *ctx, const struct Lisp_Overlay *overlay)
+{
+ struct Lisp_Overlay out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, overlay, type);
+ eassert (overlay->gcmarkbit == 0);
+ (void) overlay->spacer; /* Do not write padding. */
+ dump_field_lv_rawptr (ctx, &out, overlay, &overlay->next, Lisp_Misc);
+ dump_field_lv (ctx, &out, overlay, &overlay->start);
+ dump_field_lv (ctx, &out, overlay, &overlay->end);
+ dump_field_lv (ctx, &out, overlay, &overlay->plist);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_save_value (struct dump_context *ctx,
+ const struct Lisp_Save_Value *ptr)
+{
+ struct Lisp_Save_Value out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, ptr, type);
+ eassert(ptr->gcmarkbit == 0);
+ (void) ptr->spacer; /* Do not write padding. */
+ DUMP_FIELD_COPY (&out, ptr, save_type);
+ for (int i = 0; i < SAVE_VALUE_SLOTS; i++)
+ {
+ switch (save_type (&out, i))
+ {
+ case SAVE_UNUSED:
+ break;
+ case SAVE_INTEGER:
+ DUMP_FIELD_COPY (&out, ptr, data[i].integer);
+ break;
+ case SAVE_FUNCPOINTER:
+ dump_field_emacs_ptr (ctx, &out, ptr, &ptr->data[i].funcpointer);
+ break;
+ case SAVE_OBJECT:
+ dump_field_lv (ctx, &out, ptr, &ptr->data[i].object);
+ break;
+ case SAVE_POINTER:
+ error_unsupported_dump_object(
+ ctx, make_lisp_ptr ((void *) ptr, Lisp_Misc), "SAVE_POINTER");
+ default:
+ emacs_abort ();
+ }
+ }
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static void
+dump_field_finalizer_ref (struct dump_context *ctx,
+ void *out,
+ const struct Lisp_Finalizer *finalizer,
+ struct Lisp_Finalizer *const *field)
+{
+ if (*field == &finalizers || *field == &doomed_finalizers)
+ dump_field_emacs_ptr (ctx, out, finalizer, field);
+ else
+ dump_field_lv_rawptr (ctx, out, finalizer, field, Lisp_Misc);
+}
+
+static ptrdiff_t
+dump_finalizer (struct dump_context *ctx,
+ const struct Lisp_Finalizer *finalizer)
+{
+ struct Lisp_Finalizer out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, finalizer, base.type);
+ eassert (finalizer->base.gcmarkbit == 0);
+ (void) finalizer->base.spacer; /* Do not write padding. */
+ dump_field_finalizer_ref (ctx, &out, finalizer, &finalizer->prev);
+ dump_field_finalizer_ref (ctx, &out, finalizer, &finalizer->next);
+ dump_field_lv (ctx, &out, finalizer, &finalizer->function);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_misc_any (struct dump_context *ctx, struct Lisp_Misc_Any *misc_any)
+{
+ ptrdiff_t result;
+
+ switch (misc_any->type)
+ {
+ case Lisp_Misc_Marker:
+ result = dump_marker (ctx, (struct Lisp_Marker *) misc_any);
+ break;
+
+ case Lisp_Misc_Overlay:
+ result = dump_overlay (ctx, (struct Lisp_Overlay *) misc_any);
+ break;
+
+ case Lisp_Misc_Save_Value:
+ result = dump_save_value (ctx, (struct Lisp_Save_Value *) misc_any);
+ break;
+
+ case Lisp_Misc_Finalizer:
+ result = dump_finalizer (ctx, (struct Lisp_Finalizer *) misc_any);
+ break;
+
+#ifdef HAVE_MODULES
+ case Lisp_Misc_User_Ptr:
+ error_unsupported_dump_object(
+ ctx,
+ make_lisp_ptr (misc_any, Lisp_Misc),
+ "module user ptr");
+ break;
+#endif
+
+ default:
+ case Lisp_Misc_Float: /* Not used */
+ emacs_abort ();
+ }
+
+ return result;
+}
+
+static ptrdiff_t
+dump_float (struct dump_context *ctx, const struct Lisp_Float *lfloat)
+{
+ struct Lisp_Float out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, lfloat, u.data);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_int (struct dump_context *ctx, const struct Lisp_Intfwd *intfwd)
+{
+ dump_emacs_reloc_immediate_emacs_int (ctx, intfwd->intvar, *intfwd->intvar);
+ struct Lisp_Intfwd out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, intfwd, type);
+ dump_field_emacs_ptr (ctx, &out, intfwd, &intfwd->intvar);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_bool (struct dump_context *ctx, const struct Lisp_Boolfwd *boolfwd)
+{
+ dump_emacs_reloc_immediate_bool (ctx, boolfwd->boolvar, *boolfwd->boolvar);
+ struct Lisp_Boolfwd out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, boolfwd, type);
+ dump_field_emacs_ptr (ctx, &out, boolfwd, &boolfwd->boolvar);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_obj (struct dump_context *ctx, const struct Lisp_Objfwd *objfwd)
+{
+ dump_emacs_reloc_to_dump_lv (ctx, objfwd->objvar, *objfwd->objvar);
+ struct Lisp_Objfwd out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, objfwd, type);
+ dump_field_emacs_ptr (ctx, &out, objfwd, &objfwd->objvar);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_buffer_obj (struct dump_context *ctx,
+ const struct Lisp_Buffer_Objfwd *buffer_objfwd)
+{
+ struct Lisp_Buffer_Objfwd out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, buffer_objfwd, type);
+ DUMP_FIELD_COPY (&out, buffer_objfwd, offset);
+ dump_field_lv (ctx, &out, buffer_objfwd, &buffer_objfwd->predicate);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_kboard_obj (struct dump_context *ctx,
+ const struct Lisp_Kboard_Objfwd *kboard_objfwd)
+{
+ struct Lisp_Kboard_Objfwd out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, kboard_objfwd, type);
+ DUMP_FIELD_COPY (&out, kboard_objfwd, offset);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd (struct dump_context *ctx, union Lisp_Fwd *fwd)
+{
+ ptrdiff_t offset;
+
+ switch (XFWDTYPE (fwd))
+ {
+ case Lisp_Fwd_Int:
+ offset = dump_fwd_int (ctx, &fwd->u_intfwd);
+ break;
+ case Lisp_Fwd_Bool:
+ offset = dump_fwd_bool (ctx, &fwd->u_boolfwd);
+ break;
+ case Lisp_Fwd_Obj:
+ offset = dump_fwd_obj (ctx, &fwd->u_objfwd);
+ break;
+ case Lisp_Fwd_Buffer_Obj:
+ offset = dump_fwd_buffer_obj (ctx, &fwd->u_buffer_objfwd);
+ break;
+ case Lisp_Fwd_Kboard_Obj:
+ offset = dump_fwd_kboard_obj (ctx, &fwd->u_kboard_objfwd);
+ break;
+ default:
+ emacs_abort ();
+ }
+
+ return offset;
+}
+
+static ptrdiff_t
+dump_blv (struct dump_context *ctx,
+ const struct Lisp_Buffer_Local_Value *blv)
+{
+ struct Lisp_Buffer_Local_Value out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, blv, local_if_set);
+ DUMP_FIELD_COPY (&out, blv, frame_local);
+ DUMP_FIELD_COPY (&out, blv, found);
+ if (blv->fwd)
+ dump_field_fixup_later (ctx, &out, blv, &blv->fwd);
+ dump_field_lv (ctx, &out, blv, &blv->where);
+ dump_field_lv (ctx, &out, blv, &blv->defcell);
+ dump_field_lv (ctx, &out, blv, &blv->valcell);
+ ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+ if (blv->fwd)
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ offset + offsetof (struct Lisp_Buffer_Local_Value, fwd),
+ dump_fwd (ctx, blv->fwd));
+ return offset;
+}
+
+static ptrdiff_t
+dump_symbol (struct dump_context *ctx, const struct Lisp_Symbol *symbol)
+{
+ struct Lisp_Symbol out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ eassert (symbol->gcmarkbit == 0);
+ DUMP_FIELD_COPY (&out, symbol, redirect);
+ DUMP_FIELD_COPY (&out, symbol, constant);
+ DUMP_FIELD_COPY (&out, symbol, interned);
+ DUMP_FIELD_COPY (&out, symbol, declared_special);
+ DUMP_FIELD_COPY (&out, symbol, pinned);
+ dump_field_lv (ctx, &out, symbol, &symbol->name);
+ switch (symbol->redirect)
+ {
+ case SYMBOL_PLAINVAL:
+ dump_field_lv (ctx, &out, symbol, &symbol->val.value);
+ break;
+ case SYMBOL_VARALIAS:
+ dump_field_lv_rawptr (ctx, &out, symbol,
+ &symbol->val.alias, Lisp_Symbol);
+ break;
+ case SYMBOL_LOCALIZED:
+ dump_field_fixup_later (ctx, &out, symbol, &symbol->val.blv);
+ break;
+ case SYMBOL_FORWARDED:
+ dump_field_fixup_later (ctx, &out, symbol, &symbol->val.fwd);
+ break;
+ default:
+ emacs_abort ();
+ }
+ dump_field_lv (ctx, &out, symbol, &symbol->function);
+ dump_field_lv (ctx, &out, symbol, &symbol->plist);
+ dump_field_lv_rawptr (ctx, &out, symbol, &symbol->next, Lisp_Symbol);
+
+ // XXX: linearize symbol chains
+
+ ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+
+ switch (symbol->redirect)
+ {
+ case SYMBOL_LOCALIZED:
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ offset + offsetof (struct Lisp_Symbol, val.blv),
+ dump_blv (ctx, symbol->val.blv));
+ break;
+ case SYMBOL_FORWARDED:
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ offset + offsetof (struct Lisp_Symbol, val.fwd),
+ dump_fwd (ctx, symbol->val.fwd));
+ break;
+ default:
+ break;
+ }
+ return offset;
+}
+
+static ptrdiff_t
+dump_vectorlike_generic (
+ struct dump_context *ctx,
+ const struct vectorlike_header *header)
+{
+ const struct Lisp_Vector *v = (const struct Lisp_Vector *) header;
+ ptrdiff_t size = header->size;
+ enum pvec_type pvectype = PSEUDOVECTOR_TYPE (header);
+ ptrdiff_t offset;
+
+ if (size & PSEUDOVECTOR_FLAG)
+ {
+ /* Assert that the pseudovector contains only Lisp values ---
+ but see the PVEC_SUB_CHAR_TABLE special case below. */
+ eassert (((size & PSEUDOVECTOR_REST_MASK)
+ >> PSEUDOVECTOR_REST_BITS) == 0);
+ size &= PSEUDOVECTOR_SIZE_MASK;
+ }
+
+ dump_align_output (ctx, GCALIGNMENT);
+ ptrdiff_t prefix_start_offset = ctx->offset;
+
+ ptrdiff_t skip;
+ if (pvectype == PVEC_SUB_CHAR_TABLE)
+ {
+ /* PVEC_SUB_CHAR_TABLE has a special case because it's a
+ variable-length vector (unlike other pseudovectors) and has
+ its non-Lisp data _before_ the variable-length Lisp part. */
+ const struct Lisp_Sub_Char_Table *sct =
+ (const struct Lisp_Sub_Char_Table *) header;
+ struct Lisp_Sub_Char_Table out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, sct, header.size);
+ DUMP_FIELD_COPY (&out, sct, depth);
+ DUMP_FIELD_COPY (&out, sct, min_char);
+ offset = dump_object_finish (ctx, &out, sizeof (out));
+ skip = SUB_CHAR_TABLE_OFFSET;
+ }
+ else
+ {
+ struct vectorlike_header out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, header, size);
+ offset = dump_object_finish (ctx, &out, sizeof (out));
+ skip = 0;
+ }
+
+ ptrdiff_t prefix_size = ctx->offset - prefix_start_offset;
+ eassert (prefix_size > 0);
+ ptrdiff_t skip_start = (char*) &v->contents[skip] - (char*) v;
+ eassert (skip_start >= prefix_size);
+ dump_write_zero (ctx, skip_start - prefix_size);
+ for (ptrdiff_t i = skip; i < size; ++i)
+ {
+ Lisp_Object out;
+ const Lisp_Object *vslot = &v->contents[i];
+ eassert (ctx->offset % sizeof (out) == 0);
+ dump_object_start (ctx, 1, &out, sizeof (out));
+ dump_field_lv (ctx, &out, vslot, vslot);
+ dump_object_finish (ctx, &out, sizeof (out));
+ }
+
+ return offset;
+}
+
+static void
+dump_object_start_pseudovector (
+ struct dump_context *ctx,
+ struct vectorlike_header *out_hdr,
+ ptrdiff_t out_size,
+ const struct vectorlike_header *in_hdr)
+{
+ const struct Lisp_Vector *in = (const struct Lisp_Vector *) in_hdr;
+ struct Lisp_Vector *out = (struct Lisp_Vector *) out_hdr;
+
+ eassert (vector_nbytes ((struct Lisp_Vector *) in) == out_size);
+
+ dump_object_start (ctx, GCALIGNMENT, out, out_size);
+ DUMP_FIELD_COPY (out, in, header);
+ ptrdiff_t size = in->header.size;
+ eassert (size & PSEUDOVECTOR_FLAG);
+ size &= PSEUDOVECTOR_SIZE_MASK;
+ for (ptrdiff_t i = 0; i < size; ++i)
+ dump_field_lv (ctx, out, in, &in->contents[i]);
+}
+
+/* Determine whether the hash table's hash order is stable
+ across dump and load. If it is, we don't have to trigger
+ a rehash on access. */
+static bool
+dump_hash_table_stable_p (struct Lisp_Hash_Table *hash)
+{
+ bool is_eql = hash->test.hashfn == hashfn_eql;
+ bool is_equal = hash->test.hashfn == hashfn_equal;
+ ptrdiff_t size = HASH_TABLE_SIZE (hash);
+ for (ptrdiff_t i = 0; i < size; ++i)
+ if (!NILP (HASH_HASH (hash, i)))
+ {
+ Lisp_Object key = HASH_KEY (hash, i);
+ if (!(dump_builtin_symbol_p (key) ||
+ INTEGERP (key) ||
+ (is_equal && STRINGP (key)) ||
+ ((is_equal || is_eql) && FLOATP (key))))
+ return false;
+ }
+
+ return true;
+}
+
+static ptrdiff_t
+dump_hash_table (struct dump_context *ctx,
+ const struct Lisp_Hash_Table *hash_in)
+{
+ struct Lisp_Hash_Table hash_munged = *hash_in;
+ struct Lisp_Hash_Table *hash = &hash_munged;
+
+ /* Remember to rehash this hash table on first access. After a
+ dump reload, the hash table values will have changed, so we'll
+ need to rebuild the index.
+
+ TODO: for EQ and EQL hash tables, it should be possible to rehash
+ here using the preferred load address of the dump, eliminating
+ the need to rehash-on-access if we can load the dump where we
+ want. */
+ if (hash->count > 0 && !dump_hash_table_stable_p (hash))
+ hash->count = -hash->count;
+
+ struct Lisp_Hash_Table out;
+ dump_object_start_pseudovector (
+ ctx, &out.header, sizeof (out), &hash->header);
+ DUMP_FIELD_COPY (&out, hash, count);
+ dump_field_lv (ctx, &out, hash, &hash->key_and_value);
+ dump_field_lv (ctx, &out, hash, &hash->test.name);
+ dump_field_lv (ctx, &out, hash, &hash->test.user_hash_function);
+ dump_field_lv (ctx, &out, hash, &hash->test.user_cmp_function);
+ dump_field_emacs_ptr (ctx, &out, hash, &hash->test.cmpfn);
+ dump_field_emacs_ptr (ctx, &out, hash, &hash->test.hashfn);
+ dump_field_lv_rawptr (ctx, &out, hash, &hash->next_weak, Lisp_Vectorlike);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_buffer (struct dump_context *ctx, const struct buffer *in_buffer)
+{
+ struct buffer munged_buffer = *in_buffer;
+ struct buffer *buffer = &munged_buffer;
+
+ /* Clear some buffer state for correctness upon load. */
+ if (buffer->base_buffer == NULL)
+ buffer->window_count = 0;
+ else
+ eassert (buffer->window_count == -1);
+ buffer->last_selected_window_ = Qnil;
+ buffer->display_count_ = make_number (0);
+ buffer->clip_changed = 0;
+ buffer->last_window_start = -1;
+ buffer->point_before_scroll_ = Qnil;
+
+ ptrdiff_t base_offset = 0;
+ if (buffer->base_buffer)
+ {
+ eassert (buffer->base_buffer->base_buffer == NULL);
+ base_offset = dump_object (
+ ctx,
+ make_lisp_ptr (buffer->base_buffer, Lisp_Vectorlike));
+ }
+
+ eassert ((base_offset == 0 && buffer->text == &in_buffer->own_text) ||
+ (base_offset > 0 && buffer->text != &in_buffer->own_text));
+
+ struct buffer out;
+ dump_object_start_pseudovector (
+ ctx, &out.header, sizeof (out), &buffer->header);
+ if (base_offset == 0)
+ base_offset = ctx->obj_offset;
+ eassert (base_offset > 0);
+ if (buffer->base_buffer == NULL)
+ {
+ eassert (base_offset == ctx->obj_offset);
+
+ if (BUFFER_LIVE_P (buffer))
+ {
+ dump_field_fixup_later (ctx, &out, buffer, &buffer->own_text.beg);
+ dump_remember_cold_op (
+ ctx,
+ COLD_OP_BUFFER,
+ make_lisp_ptr ((void*) in_buffer, Lisp_Vectorlike));
+ }
+ else
+ eassert (buffer->own_text.beg == NULL);
+
+ DUMP_FIELD_COPY (&out, buffer, own_text.gpt);
+ DUMP_FIELD_COPY (&out, buffer, own_text.z);
+ DUMP_FIELD_COPY (&out, buffer, own_text.gpt_byte);
+ DUMP_FIELD_COPY (&out, buffer, own_text.z_byte);
+ DUMP_FIELD_COPY (&out, buffer, own_text.gap_size);
+ DUMP_FIELD_COPY (&out, buffer, own_text.modiff);
+ DUMP_FIELD_COPY (&out, buffer, own_text.chars_modiff);
+ DUMP_FIELD_COPY (&out, buffer, own_text.save_modiff);
+ DUMP_FIELD_COPY (&out, buffer, own_text.overlay_modiff);
+ DUMP_FIELD_COPY (&out, buffer, own_text.compact);
+ DUMP_FIELD_COPY (&out, buffer, own_text.beg_unchanged);
+ DUMP_FIELD_COPY (&out, buffer, own_text.end_unchanged);
+ DUMP_FIELD_COPY (&out, buffer, own_text.unchanged_modified);
+ DUMP_FIELD_COPY (&out, buffer, own_text.overlay_unchanged_modified);
+ if (buffer->own_text.intervals)
+ dump_field_fixup_later (ctx, &out, buffer, &buffer->own_text.intervals);
+ dump_field_lv_rawptr (ctx, &out, buffer, &buffer->own_text.markers,
+ Lisp_Misc);
+ DUMP_FIELD_COPY (&out, buffer, own_text.inhibit_shrinking);
+ DUMP_FIELD_COPY (&out, buffer, own_text.redisplay);
+ }
+
+ eassert (ctx->obj_offset > 0);
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ ctx->obj_offset + offsetof (struct buffer, text),
+ base_offset + offsetof (struct buffer, own_text));
+
+ dump_field_lv_rawptr (ctx, &out, buffer, &buffer->next, Lisp_Vectorlike);
+ DUMP_FIELD_COPY (&out, buffer, pt);
+ DUMP_FIELD_COPY (&out, buffer, pt_byte);
+ DUMP_FIELD_COPY (&out, buffer, begv);
+ DUMP_FIELD_COPY (&out, buffer, begv_byte);
+ DUMP_FIELD_COPY (&out, buffer, zv);
+ DUMP_FIELD_COPY (&out, buffer, zv_byte);
+
+ if (buffer->base_buffer)
+ {
+ eassert (ctx->obj_offset != base_offset);
+ dump_field_ptr_to_dump_offset (
+ ctx, &out, buffer, &buffer->base_buffer,
+ base_offset);
+ }
+
+ DUMP_FIELD_COPY (&out, buffer, indirections);
+ DUMP_FIELD_COPY (&out, buffer, window_count);
+
+ memcpy (&out.local_flags,
+ &buffer->local_flags,
+ sizeof (out.local_flags));
+ DUMP_FIELD_COPY (&out, buffer, modtime);
+ DUMP_FIELD_COPY (&out, buffer, modtime_size);
+ DUMP_FIELD_COPY (&out, buffer, auto_save_modified);
+ DUMP_FIELD_COPY (&out, buffer, display_error_modiff);
+ DUMP_FIELD_COPY (&out, buffer, auto_save_failure_time);
+ DUMP_FIELD_COPY (&out, buffer, last_window_start);
+
+ /* Not worth serializing these caches. TODO: really? */
+ out.newline_cache = NULL;
+ out.width_run_cache = NULL;
+ out.bidi_paragraph_cache = NULL;
+
+ DUMP_FIELD_COPY (&out, buffer, prevent_redisplay_optimizations_p);
+ DUMP_FIELD_COPY (&out, buffer, clip_changed);
+
+ dump_field_lv_rawptr (ctx, &out, buffer, &buffer->overlays_before,
+ Lisp_Misc);
+
+ dump_field_lv_rawptr (ctx, &out, buffer, &buffer->overlays_after,
+ Lisp_Misc);
+
+ DUMP_FIELD_COPY (&out, buffer, overlay_center);
+ dump_field_lv (ctx, &out, buffer, &buffer->undo_list_);
+ ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+ if (!buffer->base_buffer && buffer->own_text.intervals)
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ offset + offsetof (struct buffer, own_text.intervals),
+ dump_interval_tree (ctx, buffer->own_text.intervals, 0));
+
+ return offset;
+}
+
+static ptrdiff_t
+dump_bool_vector (struct dump_context *ctx, const struct Lisp_Vector *v)
+{
+ /* No relocation needed, so we don't need dump_object_start. */
+ dump_align_output (ctx, GCALIGNMENT);
+ eassert (ctx->offset >= ctx->header.hot_end);
+ ptrdiff_t offset = ctx->offset;
+ ptrdiff_t nbytes = vector_nbytes ((struct Lisp_Vector *) v);
+ dump_write (ctx, v, nbytes);
+ return offset;
+}
+
+static ptrdiff_t
+dump_subr (struct dump_context *ctx, const struct Lisp_Subr *subr)
+{
+ struct Lisp_Subr out;
+ dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, subr, header.size);
+ dump_field_emacs_ptr (ctx, &out, subr, &subr->function.a0);
+ DUMP_FIELD_COPY (&out, subr, min_args);
+ DUMP_FIELD_COPY (&out, subr, max_args);
+ dump_field_emacs_ptr (ctx, &out, subr, &subr->symbol_name);
+ dump_field_emacs_ptr (ctx, &out, subr, &subr->intspec);
+ DUMP_FIELD_COPY (&out, subr, doc);
+ return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static void
+fill_pseudovec (struct vectorlike_header *header, Lisp_Object item)
+{
+ struct Lisp_Vector *v = (struct Lisp_Vector *) header;
+ eassert (v->header.size & PSEUDOVECTOR_FLAG);
+ ptrdiff_t size = v->header.size & PSEUDOVECTOR_SIZE_MASK;
+ for (ptrdiff_t idx = 0; idx < size; idx++)
+ v->contents[idx] = item;
+}
+
+static ptrdiff_t
+dump_nilled_pseudovec (struct dump_context *ctx,
+ const struct vectorlike_header *in)
+{
+ ptrdiff_t nbytes = vector_nbytes ((struct Lisp_Vector *) in);
+ struct vectorlike_header *in_nilled = alloca (nbytes);
+ memset (in_nilled, 0, nbytes);
+ in_nilled->size = in->size;
+ fill_pseudovec (in_nilled, Qnil);
+ struct vectorlike_header *out = alloca (nbytes);
+ memset (out, 0, nbytes);
+ dump_object_start_pseudovector (ctx, out, nbytes, in_nilled);
+ return dump_object_finish (ctx, out, nbytes);
+}
+
+static ptrdiff_t
+dump_vectorlike (struct dump_context *ctx, const struct Lisp_Vector *v)
+{
+ ptrdiff_t offset;
+ Lisp_Object lv = make_lisp_ptr ((void *) v, Lisp_Vectorlike);
+ switch (PSEUDOVECTOR_TYPE (&v->header))
+ {
+ case PVEC_FONT:
+ /* There are three kinds of font objects that all use PVEC_FONT,
+ distinguished by their size. Font specs and entities are
+ harmless data carriers that we can dump like other Lisp
+ objects. Fonts themselves are window-system-specific and
+ need to be recreated on each startup. */
+ if ((v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_SPEC_MAX &&
+ (v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_ENTITY_MAX)
+ error_unsupported_dump_object(ctx, lv, "font");
+ /* Fall through */
+ case PVEC_NORMAL_VECTOR:
+ case PVEC_COMPILED:
+ case PVEC_CHAR_TABLE:
+ case PVEC_SUB_CHAR_TABLE:
+ offset = dump_vectorlike_generic (ctx, &v->header);
+ break;
+ case PVEC_BOOL_VECTOR:
+ offset = dump_bool_vector(ctx, v);
+ break;
+ case PVEC_HASH_TABLE:
+ offset = dump_hash_table (ctx, (struct Lisp_Hash_Table *) v);
+ break;
+ case PVEC_BUFFER:
+ offset = dump_buffer (ctx, (struct buffer *) v);
+ break;
+ case PVEC_SUBR:
+ offset = dump_subr(ctx, (const struct Lisp_Subr *) v);
+ break;
+ case PVEC_FRAME:
+ case PVEC_WINDOW:
+ case PVEC_PROCESS:
+ case PVEC_TERMINAL:
+ offset = dump_nilled_pseudovec (ctx, &v->header);
+ break;
+ case PVEC_WINDOW_CONFIGURATION:
+ error_unsupported_dump_object(ctx, lv, "window configuration");
+ case PVEC_OTHER:
+ error_unsupported_dump_object(ctx, lv, "other?!");
+ case PVEC_XWIDGET:
+ error_unsupported_dump_object(ctx, lv, "xwidget");
+ case PVEC_XWIDGET_VIEW:
+ error_unsupported_dump_object(ctx, lv, "xwidget view");
+ default:
+ error_unsupported_dump_object(ctx, lv, "weird pseudovector");
+ }
+
+ return offset;
+}
+
+/* Add an object to the dump. */
+static ptrdiff_t
+dump_object_1 (struct dump_context *ctx, Lisp_Object object, int flags)
+{
+#ifdef ENABLE_CHECKING
+ eassert (!EQ (object, Vdead));
+#endif
+
+ if (flags & DUMP_OBJECT_DRY_RUN)
+ flags &= ~(DUMP_OBJECT_INTERN | DUMP_OBJECT_RECORD_START);
+
+ int saved_flags = ctx->flags;
+ flags |= ctx->flags;
+ ctx->flags = flags;
+
+ ptrdiff_t offset = dump_recall_object (ctx, object);
+ if (flags & DUMP_OBJECT_INTERN)
+ eassert (!dump_object_self_representing_p (object));
+
+ if (offset <= 0)
+ {
+ DUMP_SET_REFERRER (ctx, object);
+ switch (XTYPE (object))
+ {
+ case Lisp_String:
+ offset = dump_string (ctx, XSTRING (object));
+ break;
+ case Lisp_Vectorlike:
+ offset = dump_vectorlike (ctx, XVECTOR (object));
+ break;
+ case Lisp_Symbol:
+ offset = dump_symbol (ctx, XSYMBOL (object));
+ break;
+ case Lisp_Misc:
+ offset = dump_misc_any (ctx, XMISCANY (object));
+ break;
+ case Lisp_Cons:
+ offset = dump_cons (ctx, XCONS (object));
+ break;
+ case Lisp_Float:
+ offset = dump_float (ctx, XFLOAT (object));
+ break;
+ case_Lisp_Int:
+ eassert (!"should not be dumping int: is self-representing");
+ default:
+ emacs_abort ();
+ }
+ eassert (offset > 0);
+ if (flags & DUMP_OBJECT_INTERN)
+ dump_remember_object (ctx, object, offset);
+ if (flags & DUMP_OBJECT_RECORD_START)
+ dump_push (&ctx->object_starts,
+ list2 (INTEGER_TO_CONS (XTYPE (object)),
+ INTEGER_TO_CONS (offset)));
+
+ dump_clear_referrer (ctx);
+
+ /* If we dumped a cons cell, we put its car and cdr on the dump
+ queue; we'll eventually get around to dumping them. That's
+ fine from a correctness perspective, but but Lisp has lots of
+ lists, and code likes to traverse lists. Make sure the cons
+ cells for reasonable-sized lists are dumped next to each
+ other. */
+ if (CONSP (object) &&
+ CONSP (XCDR (object)) &&
+ flags == (DUMP_OBJECT_INTERN | DUMP_OBJECT_RECORD_START) &&
+ ctx->cons_chain_depth < max_cons_chain_depth)
+ {
+ ctx->cons_chain_depth += 1;
+ dump_object (ctx, XCDR (object));
+ ctx->cons_chain_depth -= 1;
+ }
+ }
+
+ ctx->flags = saved_flags;
+ return offset;
+}
+
+static ptrdiff_t
+dump_object (struct dump_context *ctx, Lisp_Object object)
+{
+ ptrdiff_t result;
+ if (dump_object_emacs_ptr (object) != NULL)
+ {
+ result = dump_recall_object (ctx, object);
+ eassert (result < 0);
+ if (result > -2)
+ {
+ dump_object_1 (ctx, object, DUMP_OBJECT_DRY_RUN);
+ dump_push (&ctx->copied_queue, object);
+ result = -2;
+ dump_remember_object (ctx, object, result);
+ }
+ }
+ else
+ result = dump_object_1 (
+ ctx,
+ object,
+ DUMP_OBJECT_INTERN | DUMP_OBJECT_RECORD_START);
+ return result;
+}
+
+static ptrdiff_t
+dump_charset (struct dump_context *ctx, int cs_i)
+{
+ const struct charset *cs = charset_table + cs_i;
+ struct charset out;
+ dump_object_start (ctx, sizeof (int), &out, sizeof (out));
+ DUMP_FIELD_COPY (&out, cs, id);
+ DUMP_FIELD_COPY (&out, cs, hash_index);
+ DUMP_FIELD_COPY (&out, cs, dimension);
+ memcpy (out.code_space, &cs->code_space, sizeof (cs->code_space));
+ if (cs->code_space_mask)
+ dump_field_fixup_later (ctx, &out, cs, &cs->code_space_mask);
+ DUMP_FIELD_COPY (&out, cs, code_linear_p);
+ DUMP_FIELD_COPY (&out, cs, iso_chars_96);
+ DUMP_FIELD_COPY (&out, cs, ascii_compatible_p);
+ DUMP_FIELD_COPY (&out, cs, supplementary_p);
+ DUMP_FIELD_COPY (&out, cs, compact_codes_p);
+ DUMP_FIELD_COPY (&out, cs, unified_p);
+ DUMP_FIELD_COPY (&out, cs, iso_final);
+ DUMP_FIELD_COPY (&out, cs, iso_revision);
+ DUMP_FIELD_COPY (&out, cs, emacs_mule_id);
+ DUMP_FIELD_COPY (&out, cs, method);
+ DUMP_FIELD_COPY (&out, cs, min_code);
+ DUMP_FIELD_COPY (&out, cs, max_code);
+ DUMP_FIELD_COPY (&out, cs, char_index_offset);
+ DUMP_FIELD_COPY (&out, cs, min_char);
+ DUMP_FIELD_COPY (&out, cs, max_char);
+ DUMP_FIELD_COPY (&out, cs, invalid_code);
+ memcpy (out.fast_map, &cs->fast_map, sizeof (cs->fast_map));
+ DUMP_FIELD_COPY (&out, cs, code_offset);
+ ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+ if (cs->code_space_mask)
+ dump_remember_cold_op (ctx, COLD_OP_CHARSET,
+ Fcons (INTEGER_TO_CONS (cs_i),
+ INTEGER_TO_CONS (offset)));
+ return offset;
+}
+
+static ptrdiff_t
+dump_charset_table (struct dump_context *ctx)
+{
+ dump_align_output (ctx, GCALIGNMENT);
+ ptrdiff_t offset = ctx->offset;
+ for (int i = 0; i < charset_table_used; ++i)
+ dump_charset (ctx, i);
+ dump_emacs_reloc_to_dump_ptr_raw (ctx, &charset_table, offset);
+ dump_emacs_reloc_immediate_int (
+ ctx, &charset_table_used, charset_table_used);
+ dump_emacs_reloc_immediate_emacs_int (
+ ctx, &charset_table_size, charset_table_used);
+ return offset;
+}
+
+static void
+dump_finalizer_list_head_ptr (struct dump_context *ctx,
+ struct Lisp_Finalizer **ptr)
+{
+ struct Lisp_Finalizer *value = *ptr;
+ if (value != &finalizers && value != &doomed_finalizers)
+ dump_emacs_reloc_to_dump_ptr_raw (
+ ctx, ptr,
+ dump_object (ctx, make_lisp_ptr (value, Lisp_Misc)));
+}
+
+static void
+dump_metadata_for_pdumper (struct dump_context *ctx)
+{
+ for (int i = 0; i < nr_dump_hooks; ++i)
+ dump_emacs_reloc_to_emacs_ptr_raw (ctx, &dump_hooks[i], dump_hooks[i]);
+ dump_emacs_reloc_immediate_int (ctx, &nr_dump_hooks, nr_dump_hooks);
+
+ for (int i = 0; i < nr_remembered_data; ++i)
+ {
+ dump_emacs_reloc_to_emacs_ptr_raw (
+ ctx,
+ &remembered_data[i].mem,
+ remembered_data[i].mem);
+ dump_emacs_reloc_immediate_int (
+ ctx,
+ &remembered_data[i].sz,
+ remembered_data[i].sz);
+ }
+ dump_emacs_reloc_immediate_int (
+ ctx,
+ &nr_remembered_data,
+ nr_remembered_data);
+}
+
+static void
+dump_copied_objects (struct dump_context *ctx)
+{
+ /* Sort the objects into the order in which they'll appear in Emacs. */
+ Lisp_Object copied_queue =
+ Fsort (Fnreverse (ctx->copied_queue),
+ Qdump_emacs_portable__sort_predicate_copied);
+ ctx->copied_queue = Qnil;
+
+ /* Dump the objects and generate a copy relocation for each. We'll
+ merge adjacent copy relocations upon output. */
+ while (!NILP (copied_queue))
+ {
+ Lisp_Object copied = dump_pop (&copied_queue);
+ void *optr = dump_object_emacs_ptr (copied);
+ eassert (optr != NULL);
+ ptrdiff_t offset = dump_object_1 (
+ ctx,
+ copied,
+ ( DUMP_OBJECT_FORCE_WORD_ALIGNMENT
+ | DUMP_OBJECT_PROHIBIT_ENQUEUE));
+ ptrdiff_t size = ctx->offset - offset;
+ dump_emacs_reloc_copy_from_dump (ctx, offset, optr, size);
+ }
+}
+
+static void
+dump_cold_string (struct dump_context *ctx, Lisp_Object string)
+{
+ /* Dump string contents. */
+ ptrdiff_t string_offset = dump_recall_object (ctx, string);
+ eassert (string_offset > 0);
+ ptrdiff_t total_size = SBYTES (string) + 1;
+ eassert (total_size > 0);
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ string_offset + offsetof (struct Lisp_String, data),
+ ctx->offset);
+ dump_write (ctx, XSTRING (string)->data, total_size);
+}
+
+static void
+dump_cold_charset (struct dump_context *ctx, Lisp_Object data)
+{
+ /* Dump charset lookup tables. */
+ int cs_i = XFASTINT (XCAR (data));
+ ptrdiff_t cs_dump_offset = ptrdiff_t_from_lisp (XCDR (data));
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ cs_dump_offset + offsetof (struct charset, code_space_mask),
+ ctx->offset);
+ struct charset *cs = charset_table + cs_i;
+ dump_write (ctx, cs->code_space_mask, 256);
+}
+
+static void
+dump_cold_buffer (struct dump_context *ctx, Lisp_Object data)
+{
+ /* Dump buffer text. */
+ ptrdiff_t buffer_offset = dump_recall_object (ctx, data);
+ eassert (buffer_offset > 0);
+ struct buffer *b = XBUFFER (data);
+ eassert (b->text == &b->own_text);
+ /* Zero the gap so we don't dump uninitialized bytes. */
+ memset (BUF_GPT_ADDR (b), 0, BUF_GAP_SIZE (b));
+ /* See buffer.c for this calculation. */
+ ptrdiff_t nbytes = BUF_Z_BYTE (b) - BUF_BEG_BYTE (b) + BUF_GAP_SIZE (b) + 1;
+ dump_remember_fixup_ptr_raw (
+ ctx,
+ buffer_offset + offsetof (struct buffer, own_text.beg),
+ ctx->offset);
+ dump_write (ctx, b->own_text.beg, nbytes);
+}
+
+static void
+dump_cold_data (struct dump_context *ctx)
+{
+ if (!NILP (ctx->cold_queue))
+ {
+ Lisp_Object cold_queue = Fnreverse (ctx->cold_queue);
+ ctx->cold_queue = Qnil;
+ while (!NILP (cold_queue))
+ {
+ Lisp_Object item = dump_pop (&cold_queue);
+ enum cold_op op = XFASTINT (XCAR (item));
+ Lisp_Object data = XCDR (item);
+ switch (op)
+ {
+ case COLD_OP_STRING:
+ dump_cold_string (ctx, data);
+ break;
+ case COLD_OP_CHARSET:
+ dump_cold_charset (ctx, data);
+ break;
+ case COLD_OP_BUFFER:
+ dump_cold_buffer (ctx, data);
+ break;
+ case COLD_OP_OBJECT:
+ /* Objects that we can put in the cold section
+ must not refer to other objects. */
+ eassert (dump_tailq_empty_p (&ctx->dump_queue));
+ dump_object (ctx, data);
+ eassert (dump_tailq_empty_p (&ctx->dump_queue));
+ break;
+ default:
+ emacs_abort ();
+ }
+ }
+ }
+}
+
+static void
+read_raw_ptr_and_lv (const void *mem,
+ enum Lisp_Type type,
+ void **out_ptr,
+ Lisp_Object *out_lv)
+{
+ memcpy (out_ptr, mem, sizeof (*out_ptr));
+ if (*out_ptr != NULL)
+ {
+ switch (type)
+ {
+ case Lisp_Symbol:
+ *out_lv = make_lisp_symbol (*out_ptr);
+ break;
+ case Lisp_Misc:
+ case Lisp_String:
+ case Lisp_Vectorlike:
+ case Lisp_Cons:
+ case Lisp_Float:
+ *out_lv = make_lisp_ptr (*out_ptr, type);
+ break;
+ default:
+ emacs_abort ();
+ }
+ }
+}
+
+/* Enqueue for dumping objects referenced by static non-Lisp_Object
+ pointers inside Emacs. */
+static void
+dump_user_remembered_data_hot (struct dump_context *ctx)
+{
+ for (int i = 0; i < nr_remembered_data; ++i)
+ {
+ void *mem = remembered_data[i].mem;
+ int sz = remembered_data[i].sz;
+ if (sz <= 0)
+ {
+ enum Lisp_Type type = -sz;
+ void *value;
+ Lisp_Object lv;
+ read_raw_ptr_and_lv (mem, type, &value, &lv);
+ if (value != NULL)
+ {
+ DUMP_SET_REFERRER (ctx, dump_ptr_referrer ("user data", mem));
+ dump_enqueue_object (ctx, lv);
+ dump_clear_referrer (ctx);
+ }
+ }
+ }
+}
+
+/* Dump user-specified non-relocated data. */
+static void
+dump_user_remembered_data_cold (struct dump_context *ctx)
+{
+ for (int i = 0; i < nr_remembered_data; ++i)
+ {
+ void *mem = remembered_data[i].mem;
+ int sz = remembered_data[i].sz;
+ if (sz > 0)
+ {
+ /* Scalar: try to inline the value into the relocation if
+ it's small enough; if it's bigger than we can fit in a
+ relocation, we have to copy the data into the dump proper
+ and issue a copy relocation. */
+ if (sz <= sizeof (intmax_t))
+ dump_emacs_reloc_immediate (ctx, mem, mem, sz);
+ else
+ {
+ dump_emacs_reloc_copy_from_dump (ctx, ctx->offset, mem, sz);
+ dump_write (ctx, mem, sz);
+ }
+ }
+ else
+ {
+ /* *mem is a raw pointer to a Lisp object of some sort.
+ The object to which it points should have already been
+ dumped by dump_user_remembered_data_hot. */
+ void *value;
+ Lisp_Object lv;
+ enum Lisp_Type type = -sz;
+ read_raw_ptr_and_lv (mem, type, &value, &lv);
+ if (value == NULL)
+ /* We can't just ignore NULL: the variable might have
+ transitioned from non-NULL to NULL, and we want to
+ record this fact. */
+ dump_emacs_reloc_immediate_ptrdiff_t (ctx, mem, 0);
+ else
+ {
+ if (dump_object_emacs_ptr (lv) != NULL)
+ {
+ /* We have situation like this:
+
+ static Lisp_Symbol *foo;
+ ...
+ foo = XSYMBOL(Qt);
+ ...
+ pdumper_remember_lv_raw_ptr (&foo, Lisp_Symbol);
+
+ Built-in symbols like Qt aren't in the dump!
+ They're actually in Emacs proper. We need a
+ special case to point this value back at Emacs
+ instead of to something in the dump that
+ isn't there.
+
+ An analogous situation applies to subrs, since
+ Lisp_Subr structures always live in Emacs, not
+ the dump.
+
+ */
+ dump_emacs_reloc_to_emacs_ptr_raw (
+ ctx, mem, dump_object_emacs_ptr (lv));
+ }
+ else
+ {
+ eassert (!dump_object_self_representing_p (lv));
+ ptrdiff_t dump_offset = dump_recall_object (ctx, lv);
+ if (dump_offset <= 0)
+ error ("raw-pointer object not dumped?!");
+ dump_emacs_reloc_to_dump_ptr_raw (ctx, mem, dump_offset);
+ }
+ }
+ }
+ }
+}
+
+static void
+dump_unwind_cleanup (void *data)
+{
+ // XXX: omit relocations that duplicate BSS?
+ // XXX: prevent ralloc moving
+ // XXX: dumb mode for GC
+ struct dump_context *ctx = data;
+ if (ctx->fd >= 0)
+ emacs_close (ctx->fd);
+ Vpurify_flag = ctx->old_purify_flag;
+ unblock_input ();
+}
+
+static Lisp_Object
+make_eq_hash_table (void)
+{
+ return CALLN (Fmake_hash_table, QCtest, Qeq);
+}
+
+static void
+dump_do_fixup (struct dump_context *ctx, Lisp_Object fixup)
+{
+ enum dump_fixup_type type = XFASTINT (XCAR (fixup));
+ fixup = XCDR (fixup);
+ ptrdiff_t dump_fixup_offset = ptrdiff_t_from_lisp (XCAR (fixup));
+ fixup = XCDR (fixup);
+ Lisp_Object arg = XCAR (fixup);
+ eassert (NILP (XCDR (fixup)));
+ dump_seek (ctx, dump_fixup_offset);
+ ptrdiff_t target_offset;
+ bool do_write = true;
+ switch (type)
+ {
+ case DUMP_FIXUP_LISP_OBJECT:
+ case DUMP_FIXUP_LISP_OBJECT_RAW:
+ /* Dump wants a pointer to a Lisp object.
+ If DUMP_FIXUP_LISP_OBJECT_RAW, we should stick a C pointer in
+ the dump; otherwise, a Lisp_Object. */
+ if (SUBRP (arg))
+ {
+ target_offset = emacs_offset (XSUBR (arg));
+ if (type == DUMP_FIXUP_LISP_OBJECT)
+ dump_reloc_dump_to_emacs_lv (ctx, ctx->offset, XTYPE (arg));
+ else
+ dump_reloc_dump_to_emacs_raw_ptr (ctx, ctx->offset);
+ }
+ else if (dump_builtin_symbol_p (arg))
+ {
+ eassert (dump_object_self_representing_p (arg));
+ /* These symbols are part of Emacs, so point there. If we
+ want a Lisp_Object, we're set. If we want a raw pointer,
+ we need to emit a relocation. */
+ if (type == DUMP_FIXUP_LISP_OBJECT)
+ {
+ do_write = false;
+ dump_write (ctx, &arg, sizeof (arg));
+ }
+ else
+ {
+ target_offset = emacs_offset (XSYMBOL (arg));
+ dump_reloc_dump_to_emacs_raw_ptr (ctx, ctx->offset);
+ }
+ }
+ else
+ {
+ eassert (dump_object_emacs_ptr (arg) == NULL);
+ target_offset = dump_recall_object (ctx, arg);
+ if (target_offset <= 0)
+ error ("fixup object not dumped");
+ if (type == DUMP_FIXUP_LISP_OBJECT)
+ dump_reloc_dump_to_dump_lv (ctx, ctx->offset, XTYPE (arg));
+ else
+ dump_reloc_dump_to_dump_raw_ptr (ctx, ctx->offset);
+ }
+ break;
+ case DUMP_FIXUP_PTR_DUMP_RAW:
+ /* Dump wants a raw pointer to something that's not a lisp
+ object. It knows the exact location it wants, so just
+ believe it. */
+ target_offset = ptrdiff_t_from_lisp (arg);
+ dump_reloc_dump_to_dump_raw_ptr (ctx, ctx->offset);
+ break;
+ default:
+ emacs_abort ();
+ }
+ if (do_write)
+ dump_write (ctx, &target_offset, sizeof (target_offset));
+}
+
+static ptrdiff_t
+dump_check_dump_off (struct dump_context *ctx, ptrdiff_t dump_offset)
+{
+ eassert (dump_offset > 0);
+ if (ctx)
+ eassert (dump_offset < ctx->end_heap);
+ return dump_offset;
+}
+
+static void
+dump_check_emacs_off (ptrdiff_t emacs_off)
+{
+ eassert (labs (emacs_off) <= 30*1024*1024);
+}
+
+static void
+dump_emit_dump_reloc (struct dump_context *ctx, Lisp_Object lreloc)
+{
+ struct dump_reloc reloc;
+ dump_object_start (ctx, 1, &reloc, sizeof (reloc));
+ reloc.type = XFASTINT (dump_pop (&lreloc));
+ eassert (reloc.type <= RELOC_DUMP_TO_EMACS_LV + Lisp_Float);
+ reloc.offset = dump_off_from_lisp (dump_pop (&lreloc));
+ dump_check_dump_off (ctx, reloc.offset);
+ eassert (reloc.offset % 4 == 0); // Alignment
+ eassert (NILP (lreloc));
+ dump_object_finish (ctx, &reloc, sizeof (reloc));
+}
+
+static struct emacs_reloc
+decode_emacs_reloc (struct dump_context *ctx, Lisp_Object lreloc)
+{
+ struct emacs_reloc reloc;
+ memset (&reloc, 0, sizeof (reloc));
+ int type = XFASTINT (dump_pop (&lreloc));
+ reloc.emacs_offset = dump_off_from_lisp (dump_pop (&lreloc));
+ dump_check_emacs_off (reloc.emacs_offset);
+ switch (type)
+ {
+ case RELOC_EMACS_COPY_FROM_DUMP:
+ {
+ reloc.type = type;
+ eassert (reloc.type == type);
+ reloc.u.dump_offset = dump_off_from_lisp (dump_pop (&lreloc));
+ dump_check_dump_off (ctx, reloc.u.dump_offset);
+ dump_off_t length = dump_off_from_lisp (dump_pop (&lreloc));
+ reloc.length = length;
+ if (reloc.length != length)
+ error ("relocation copy length too large");
+ }
+ break;
+ case RELOC_EMACS_IMMEDIATE:
+ {
+ reloc.type = type;
+ eassert (reloc.type == type);
+ intmax_t value = intmax_t_from_lisp (dump_pop (&lreloc));
+ ptrdiff_t size = ptrdiff_t_from_lisp (dump_pop (&lreloc));
+ reloc.u.immediate = value;
+ reloc.length = size;
+ eassert (reloc.length == size);
+ }
+ break;
+ default:
+ {
+ eassert (RELOC_EMACS_DUMP_LV <= type);
+ eassert (type <= RELOC_EMACS_DUMP_LV + Lisp_Float);
+ reloc.type = RELOC_EMACS_DUMP_LV;
+ eassert (reloc.type == RELOC_EMACS_DUMP_LV);
+ reloc.length = type - RELOC_EMACS_DUMP_LV;
+ eassert (reloc.length == type - RELOC_EMACS_DUMP_LV);
+ Lisp_Object target_value = dump_pop (&lreloc);
+ /* If the object is self-representing,
+ dump_emacs_reloc_to_dump_lv didn't do its job.
+ dump_emacs_reloc_to_dump_lv should have added a
+ RELOC_EMACS_IMMEDIATE relocation instead. */
+ eassert (!dump_object_self_representing_p (target_value));
+ reloc.u.dump_offset = dump_recall_object (ctx, target_value);
+ if (reloc.u.dump_offset <= 0)
+ {
+ Lisp_Object repr = Fprin1_to_string (target_value, Qnil);
+ error ("relocation target was not dumped: %s", SDATA (repr));
+ }
+ dump_check_dump_off (ctx, reloc.u.dump_offset);
+ }
+ break;
+ case RELOC_EMACS_EMACS_PTR_RAW:
+ reloc.type = type;
+ eassert (reloc.type == type);
+ reloc.u.emacs_offset2 = dump_off_from_lisp (dump_pop (&lreloc));
+ dump_check_emacs_off (reloc.u.emacs_offset2);
+ break;
+ case RELOC_EMACS_DUMP_PTR_RAW:
+ reloc.type = type;
+ eassert (reloc.type == type);
+ reloc.u.dump_offset = dump_off_from_lisp (dump_pop (&lreloc));
+ dump_check_dump_off (ctx, reloc.u.dump_offset);
+ break;
+ }
+
+ eassert (NILP (lreloc));
+ return reloc;
+}
+
+static void
+dump_emit_emacs_reloc (struct dump_context *ctx, Lisp_Object lreloc)
+{
+ struct emacs_reloc reloc;
+ dump_object_start (ctx, 1, &reloc, sizeof (reloc));
+ reloc = decode_emacs_reloc (ctx, lreloc);
+ dump_object_finish (ctx, &reloc, sizeof (reloc));
+}
+
+static Lisp_Object
+dump_merge_emacs_relocs (Lisp_Object lreloc_a, Lisp_Object lreloc_b)
+{
+ /* Combine copy relocations together if they're copying from
+ adjacent chunks to adjacent chunks. */
+
+ if (XFASTINT (XCAR (lreloc_a)) != RELOC_EMACS_COPY_FROM_DUMP ||
+ XFASTINT (XCAR (lreloc_b)) != RELOC_EMACS_COPY_FROM_DUMP)
+ return Qnil;
+
+ struct emacs_reloc reloc_a = decode_emacs_reloc (NULL, lreloc_a);
+ struct emacs_reloc reloc_b = decode_emacs_reloc (NULL, lreloc_b);
+
+ eassert (reloc_a.type == RELOC_EMACS_COPY_FROM_DUMP);
+ eassert (reloc_b.type == RELOC_EMACS_COPY_FROM_DUMP);
+
+ if (reloc_a.emacs_offset + reloc_a.length != reloc_b.emacs_offset)
+ return Qnil;
+
+ if (reloc_a.u.dump_offset + reloc_a.length != reloc_b.u.dump_offset)
+ return Qnil;
+
+ ptrdiff_t new_length = reloc_a.length + reloc_b.length;
+ reloc_a.length = new_length;
+ if (reloc_a.length != new_length)
+ return Qnil; /* Overflow */
+
+ return list4 (make_number (RELOC_EMACS_COPY_FROM_DUMP),
+ INTEGER_TO_CONS (reloc_a.emacs_offset),
+ INTEGER_TO_CONS (reloc_a.u.dump_offset),
+ INTEGER_TO_CONS (reloc_a.length));
+}
+
+static void
+drain_reloc_list (struct dump_context *ctx,
+ void (*handler)(struct dump_context *, Lisp_Object),
+ Lisp_Object (*merger)(Lisp_Object a, Lisp_Object b),
+ Lisp_Object *reloc_list,
+ struct dump_table_locator *out_locator)
+{
+ Lisp_Object relocs = Fsort (Fnreverse (*reloc_list),
+ Qdump_emacs_portable__sort_predicate);
+ *reloc_list = Qnil;
+ dump_align_output (ctx, sizeof (dump_off_t));
+ struct dump_table_locator locator;
+ memset (&locator, 0, sizeof (locator));
+ locator.offset = ctx->offset;
+ for (; !NILP (relocs); locator.nr_entries += 1)
+ {
+ Lisp_Object reloc = dump_pop (&relocs);
+ Lisp_Object merged;
+ while (merger != NULL &&
+ !NILP (relocs) &&
+ ((merged = merger (reloc, XCAR (relocs))), !NILP (merged)))
+ {
+ reloc = merged;
+ relocs = XCDR (relocs);
+ }
+ handler (ctx, reloc);
+ }
+ *out_locator = locator;
+}
+
+static void
+dump_do_fixups (struct dump_context *ctx)
+{
+ ptrdiff_t saved_offset = ctx->offset;
+ Lisp_Object fixups = Fsort (Fnreverse (ctx->fixups),
+ Qdump_emacs_portable__sort_predicate);
+ ctx->fixups = Qnil;
+ while (!NILP (fixups))
+ dump_do_fixup (ctx, dump_pop (&fixups));
+ dump_seek (ctx, saved_offset);
+}
+
+DEFUN ("dump-emacs-portable",
+ Fdump_emacs_portable, Sdump_emacs_portable,
+ 1, 2, 0,
+ doc: /* Dump current state of Emacs into dump file FILENAME.
+If TRACK-REFERRERS is non-nil, keep additional debugging information
+that can help track down the provenance of unsupported object
+types. */)
+ (Lisp_Object filename, Lisp_Object track_referrers)
+{
+ eassert (initialized);
+
+ if (will_dump_with_unexec)
+ error ("This Emacs instance was started under the assumption "
+ "that it would be dumped with unexec, not the portable "
+ "dumper. Dumping with the portable dumper may produce "
+ "unexpected results.");
+
+ ptrdiff_t count = SPECPDL_INDEX ();
+
+ /* Bind `command-line-processed' to nil before dumping,
+ so that the dumped Emacs will process its command line
+ and set up to work with X windows if appropriate. */
+ Lisp_Object symbol = intern ("command-line-processed");
+ specbind (symbol, Qnil);
+
+ CHECK_STRING (filename);
+ filename = Fexpand_file_name (filename, Qnil);
+ filename = ENCODE_FILE (filename);
+
+ struct dump_context ctx_buf;
+ struct dump_context *ctx = &ctx_buf;
+ memset (ctx, 0, sizeof (*ctx));
+ ctx->fd = -1;
+
+ ctx->objects_dumped = make_eq_hash_table ();
+ dump_tailq_init (&ctx->dump_queue);
+ ctx->fixups = Qnil;
+ ctx->copied_queue = Qnil;
+ ctx->cold_queue = Qnil;
+ ctx->dump_relocs = Qnil;
+ ctx->object_starts = Qnil;
+ ctx->emacs_relocs = Qnil;
+
+ ctx->current_referrer = Qnil;
+ if (!NILP (track_referrers))
+ ctx->referrers = make_eq_hash_table ();
+
+ ctx->dump_filename = filename;
+
+ record_unwind_protect_ptr (dump_unwind_cleanup, ctx);
+ block_input ();
+
+ ctx->old_purify_flag = Vpurify_flag;
+ Vpurify_flag = Qnil;
+
+ /* Make sure various weird things are less likely to happen. */
+ ctx->old_post_gc_hook = Vpost_gc_hook;
+ Vpost_gc_hook = Qnil;
+
+ ctx->fd = emacs_open (SSDATA (filename), O_RDWR | O_TRUNC | O_CREAT, 0666);
+ if (ctx->fd < 0)
+ report_file_error ("Opening dump output", filename);
+ verify (sizeof (ctx->header.magic) == sizeof (dump_magic));
+ memcpy (&ctx->header.magic, dump_magic, sizeof (dump_magic));
+ ctx->header.magic[0] = '!'; /* Note that dump is incomplete. */
+
+ verify (sizeof (fingerprint) == sizeof (ctx->header.fingerprint));
+ memcpy (ctx->header.fingerprint, fingerprint, sizeof (fingerprint));
+
+ dump_write (ctx, &ctx->header, sizeof (ctx->header));
+
+ /* Start the dump process by processing the static roots and
+ queuing up the objects to which they refer. */
+ dump_roots (ctx);
+
+ dump_charset_table (ctx);
+ dump_finalizer_list_head_ptr (ctx, &finalizers.prev);
+ dump_finalizer_list_head_ptr (ctx, &finalizers.next);
+ dump_finalizer_list_head_ptr (ctx, &doomed_finalizers.prev);
+ dump_finalizer_list_head_ptr (ctx, &doomed_finalizers.next);
+ dump_user_remembered_data_hot (ctx);
+
+ /* We've already remembered all of the GC roots themselves, but we
+ have to manually save the list of GC roots. */
+ dump_metadata_for_pdumper (ctx);
+ for (int i = 0; i < staticidx; ++i)
+ dump_emacs_reloc_to_emacs_ptr_raw (ctx, &staticvec[i], staticvec[i]);
+ dump_emacs_reloc_immediate_int (ctx, &staticidx, staticidx);
+
+ /* Dump until while we keep finding objects to dump. */
+ while (!dump_tailq_empty_p (&ctx->dump_queue))
+ dump_object (ctx, dump_tailq_pop (&ctx->dump_queue));
+
+ eassert (dump_tailq_empty_p (&ctx->dump_queue));
+ ctx->header.hot_discardable_start = ctx->offset;
+
+ dump_copied_objects (ctx);
+ eassert (dump_tailq_empty_p (&ctx->dump_queue));
+ eassert (NILP (ctx->copied_queue));
+
+ dump_align_output (ctx, getpagesize ());
+ ctx->header.hot_end = ctx->offset;
+ dump_cold_data (ctx);
+ /* dump_user_remembered_data_cold needs to be after dump_cold_data
+ in case dump_cold_data dumps a lisp object to which C code
+ points. dump_user_remembered_data_cold assumes that all lisp
+ objects have been dumped. */
+ dump_user_remembered_data_cold (ctx);
+ ctx->end_heap = ctx->offset;
+ dump_do_fixups (ctx);
+ drain_reloc_list (
+ ctx, dump_emit_dump_reloc, NULL,
+ &ctx->dump_relocs,
+ &ctx->header.dump_relocs);
+ drain_reloc_list (
+ ctx, dump_emit_dump_reloc, NULL,
+ &ctx->object_starts,
+ &ctx->header.object_starts);
+ drain_reloc_list (
+ ctx, dump_emit_emacs_reloc, dump_merge_emacs_relocs,
+ &ctx->emacs_relocs,
+ &ctx->header.emacs_relocs);
+
+ eassert (dump_tailq_empty_p (&ctx->dump_queue));
+ eassert (NILP (ctx->fixups));
+ eassert (NILP (ctx->dump_relocs));
+ eassert (NILP (ctx->emacs_relocs));
+
+ ctx->header.magic[0] = dump_magic[0]; /* Note dump is complete. */
+ dump_seek (ctx, 0);
+ dump_write (ctx, &ctx->header, sizeof (ctx->header));
+
+ return unbind_to (count, Qnil);
+
+ // XXX: consider getting rid of hooks and just rely
+ // on explicit calls?
+
+ // XXX: nullify frame_and_buffer_state
+
+ // XXX: inline stuff in pdumper.h
+
+ // XXX: preferred base address
+
+ // XXX: make offset math non-fwrapv-safe
+
+ // XXX: output symbol chains consecutively
+}
+
+DEFUN ("dump-emacs-portable--sort-predicate",
+ Fdump_emacs_portable__sort_predicate,
+ Sdump_emacs_portable__sort_predicate,
+ 2, 2, 0,
+ doc: /* Internal relocation sorting function. */)
+ (Lisp_Object a, Lisp_Object b)
+{
+ ptrdiff_t a_offset = ptrdiff_t_from_lisp (XCAR (XCDR (a)));
+ ptrdiff_t b_offset = ptrdiff_t_from_lisp (XCAR (XCDR (b)));
+ return a_offset < b_offset ? Qt : Qnil;
+}
+
+DEFUN ("dump-emacs-portable--sort-predicate-copied",
+ Fdump_emacs_portable__sort_predicate_copied,
+ Sdump_emacs_portable__sort_predicate_copied,
+ 2, 2, 0,
+ doc: /* Internal relocation sorting function. */)
+ (Lisp_Object a, Lisp_Object b)
+{
+ eassert (dump_object_emacs_ptr (a));
+ eassert (dump_object_emacs_ptr (b));
+ return dump_object_emacs_ptr (a) < dump_object_emacs_ptr (b) ? Qt : Qnil;
+}
+
+void
+pdumper_do_now_and_after_load (pdumper_hook hook)
+{
+ if (nr_dump_hooks == ARRAYELTS (dump_hooks))
+ fatal ("out of dump hooks: make dump_hooks[] bigger");
+ dump_hooks[nr_dump_hooks++] = hook;
+ hook ();
+}
+
+static void
+pdumper_remember_user_data_1 (void *mem, int nbytes)
+{
+ if (nr_remembered_data == ARRAYELTS (remembered_data))
+ fatal ("out of remembered data slots: make remembered_data[] bigger");
+ remembered_data[nr_remembered_data].mem = mem;
+ remembered_data[nr_remembered_data].sz = nbytes;
+ nr_remembered_data += 1;
+}
+
+void
+pdumper_remember_scalar (void *mem, ptrdiff_t nbytes)
+{
+ eassert (0 <= nbytes && nbytes <= INT_MAX);
+ if (nbytes > 0)
+ pdumper_remember_user_data_1 (mem, nbytes);
+}
+
+void
+pdumper_remember_lv_raw_ptr (void* ptr, enum Lisp_Type type)
+{
+ pdumper_remember_user_data_1 (ptr, -type);
+}
+
+\f
+
+struct loaded_dump
+{
+ char *start;
+ char *end;
+ struct dump_header header;
+ unsigned *mark_bits;
+};
+
+struct loaded_dump loaded_dump;
+
+/* Search for a relocation given a relocation target.
+
+ DUMP is the dump metadata structure. TABLE is the relocation table
+ to search. KEY is the dump offset to find. Return the greatest
+ relocation RELOC such that RELOC.offset <= KEY or NULL if no such
+ relocation exists. */
+static const struct dump_reloc *
+dump_find_relocation (struct loaded_dump *dump,
+ const struct dump_table_locator *table,
+ dump_off_t key)
+{
+ const struct dump_reloc *left = (void *)(dump->start + table->offset);
+ const struct dump_reloc *right = left + table->nr_entries;
+ const struct dump_reloc *found = NULL;
+
+ while (left < right)
+ {
+ const struct dump_reloc *mid = left + (right - left) / 2;
+ if (mid->offset <= key)
+ {
+ found = mid;
+ left = mid + 1;
+ if (left >= right || left->offset > key)
+ break;
+ }
+ else
+ right = mid;
+ }
+
+ return found;
+}
+
+static bool
+dump_loaded_p (void)
+{
+ return loaded_dump.start != NULL;
+}
+
+/* Return whether the OBJ points somewhere into the loaded dump image.
+ Works even when we have no dump loaded --- in this case, it just
+ returns false. */
+bool
+pdumper_object_p (const void *obj)
+{
+ const char *p = obj;
+ return loaded_dump.start <= p && p < loaded_dump.end;
+}
+
+/* Return whether OBJ points exactly to the start of some object in
+ the loaded dump image. It is a programming error to call this
+ routine for an OBJ for which pdumper_object_p would return
+ false. */
+bool
+pdumper_object_p_precise (const void *obj)
+{
+ return pdumper_find_object_type (obj) != PDUMPER_NO_OBJECT;
+}
+
+/* Return the type of the dumped object that starts at OBJ. It is a
+ programming error to call this routine for an OBJ for which
+ pdumper_object_p would return false. */
+enum Lisp_Type
+pdumper_find_object_type (const void *obj)
+{
+ eassert (pdumper_object_p (obj));
+ ptrdiff_t offset = (char *) obj - (char *)loaded_dump.start;
+ if (offset % GCALIGNMENT != 0)
+ return PDUMPER_NO_OBJECT;
+ const struct dump_reloc *reloc =
+ dump_find_relocation (&loaded_dump,
+ &loaded_dump.header.object_starts,
+ offset);
+ return (reloc != NULL && reloc->offset == offset)
+ ? reloc->type
+ : PDUMPER_NO_OBJECT;
+}
+
+static ptrdiff_t
+dump_mark_bits_nbytes (ptrdiff_t max_offset)
+{
+ ptrdiff_t bits_needed = (max_offset + GCALIGNMENT - 1) / GCALIGNMENT;
+ ptrdiff_t bytes_needed = (bits_needed + CHAR_BIT - 1) / CHAR_BIT;
+ return ROUNDUP (bytes_needed, sizeof (unsigned));
+}
+
+bool
+pdumper_marked_p (const void *obj)
+{
+ eassert (pdumper_object_p (obj));
+ ptrdiff_t offset = (char *) obj - loaded_dump.start;
+ eassert (offset % GCALIGNMENT == 0);
+ eassert (offset < loaded_dump.header.hot_discardable_start);
+ ptrdiff_t bitno = offset / GCALIGNMENT;
+ ptrdiff_t slotno = bitno / (CHAR_BIT * sizeof (unsigned));
+ unsigned *slot = &loaded_dump.mark_bits[slotno];
+ return *slot & (1U << (bitno % (CHAR_BIT * sizeof (unsigned))));
+}
+
+void
+pdumper_set_marked (const void *obj)
+{
+ eassert (pdumper_object_p (obj));
+ ptrdiff_t offset = (char *) obj - loaded_dump.start;
+ eassert (offset % GCALIGNMENT == 0);
+ eassert (offset < loaded_dump.header.hot_discardable_start);
+ ptrdiff_t bitno = offset / GCALIGNMENT;
+ ptrdiff_t slotno = bitno / (CHAR_BIT * sizeof (unsigned));
+ unsigned *slot = &loaded_dump.mark_bits[slotno];
+ *slot |= (1U << (bitno % (CHAR_BIT * sizeof (unsigned))));
+}
+
+void
+pdumper_clear_marks (void)
+{
+ memset (loaded_dump.mark_bits, 0,
+ dump_mark_bits_nbytes (loaded_dump.header.hot_discardable_start));
+}
+
+static ssize_t
+pdumper_read (int fd, void *buf, size_t bytes_to_read)
+{
+ eassert (bytes_to_read <= SSIZE_MAX);
+ size_t bytes_read = 0;
+ while (bytes_read < bytes_to_read)
+ {
+ ssize_t chunk =
+ read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
+ if (chunk < 0)
+ return chunk;
+ if (chunk == 0)
+ break;
+ bytes_read += chunk;
+ }
+
+ return bytes_read;
+}
+
+static void *
+dump_ptr (struct loaded_dump *dump, ptrdiff_t offset)
+{
+ eassert (dump->start + offset < dump->end);
+ return dump->start + offset;
+}
+
+static void *
+emacs_ptr (ptrdiff_t offset)
+{
+ // TODO: assert somehow that offset is actually inside Emacs
+ return (void *) (emacs_basis () + offset);
+}
+
+static void
+dump_do_dump_relocation (struct loaded_dump *dump,
+ struct dump_reloc reloc)
+{
+ ptrdiff_t *dump_ptr_ptr = dump_ptr (dump, reloc.offset);
+ ptrdiff_t dump_ptr = *dump_ptr_ptr;
+ ptrdiff_t dump_base = (ptrdiff_t) dump->start;
+
+ /* For -O0 debugging: optimizer realizes this variable is dead and
+ optimizes it away. */
+ ptrdiff_t orig_dump_ptr = dump_ptr;
+ (void) orig_dump_ptr;
+
+ switch (reloc.type)
+ {
+ case RELOC_DUMP_TO_EMACS_RAW_PTR:
+ dump_ptr = dump_ptr + emacs_basis ();
+ *dump_ptr_ptr = dump_ptr;
+ break;
+ case RELOC_DUMP_TO_DUMP_RAW_PTR:
+ dump_ptr = dump_ptr + dump_base;
+ *dump_ptr_ptr = dump_ptr;
+ break;
+ default:
+ {
+ enum Lisp_Type lisp_type;
+ if (RELOC_DUMP_TO_DUMP_LV <= reloc.type &&
+ reloc.type < RELOC_DUMP_TO_EMACS_LV)
+ {
+ lisp_type = reloc.type - RELOC_DUMP_TO_DUMP_LV;
+ dump_ptr = dump_ptr + dump_base;
+ }
+ else
+ {
+ eassert (RELOC_DUMP_TO_EMACS_LV <= reloc.type);
+ eassert (reloc.type < RELOC_DUMP_TO_EMACS_LV + 8);
+ lisp_type = reloc.type - RELOC_DUMP_TO_EMACS_LV;
+ dump_ptr = dump_ptr + emacs_basis ();
+ }
+
+ Lisp_Object lv;
+ if (lisp_type == Lisp_Symbol)
+ lv = make_lisp_symbol ((void *) dump_ptr);
+ else
+ lv = make_lisp_ptr ((void *) dump_ptr, lisp_type);
+
+ * (Lisp_Object *) dump_ptr_ptr = lv;
+ break;
+ }
+ }
+
+ // XXX: raw_ptr or ptr_raw. Pick one.
+}
+
+static void
+dump_do_dump_relocations (struct loaded_dump *dump)
+{
+ struct dump_header *header = &dump->header;
+ struct dump_reloc *r = dump_ptr (dump, header->dump_relocs.offset);
+ dump_off_t nr_entries = header->dump_relocs.nr_entries;
+ for (dump_off_t i = 0; i < nr_entries; ++i)
+ dump_do_dump_relocation (dump, r[i]);
+}
+
+static void
+dump_do_emacs_relocation (struct loaded_dump *dump,
+ struct emacs_reloc reloc)
+{
+ ptrdiff_t dump_base = (ptrdiff_t) dump->start;
+ ptrdiff_t pval;
+ Lisp_Object lv;
+
+ switch (reloc.type)
+ {
+ case RELOC_EMACS_COPY_FROM_DUMP:
+ eassume (reloc.length > 0);
+ memcpy (emacs_ptr (reloc.emacs_offset),
+ dump_ptr (dump, reloc.u.dump_offset),
+ reloc.length);
+ break;
+ case RELOC_EMACS_IMMEDIATE:
+ eassume (reloc.length > 0);
+ eassume (reloc.length <= sizeof (reloc.u.immediate));
+ memcpy (emacs_ptr (reloc.emacs_offset),
+ &reloc.u.immediate,
+ reloc.length);
+ break;
+ case RELOC_EMACS_DUMP_PTR_RAW:
+ pval = reloc.u.dump_offset + dump_base;
+ memcpy (emacs_ptr (reloc.emacs_offset), &pval, sizeof (pval));
+ break;
+ case RELOC_EMACS_EMACS_PTR_RAW:
+ pval = reloc.u.emacs_offset2 + emacs_basis ();
+ memcpy (emacs_ptr (reloc.emacs_offset), &pval, sizeof (pval));
+ break;
+ case RELOC_EMACS_DUMP_LV:
+ eassume (reloc.length <= Lisp_Float);
+ if (reloc.length == Lisp_Symbol)
+ lv = make_lisp_symbol (dump_ptr (dump, reloc.u.dump_offset));
+ else
+ lv = make_lisp_ptr (dump_ptr (dump, reloc.u.dump_offset),
+ reloc.length);
+ memcpy (emacs_ptr (reloc.emacs_offset), &lv, sizeof (lv));
+ break;
+ default:
+ fatal ("unrecognied relocation type %d", (int) reloc.type);
+ }
+}
+
+static void
+dump_do_emacs_relocations (struct loaded_dump *dump)
+{
+ struct dump_header *header = &dump->header;
+ struct emacs_reloc *r = dump_ptr (dump, header->emacs_relocs.offset);
+ dump_off_t nr_entries = header->emacs_relocs.nr_entries;
+ for (dump_off_t i = 0; i < nr_entries; ++i)
+ dump_do_emacs_relocation (dump, r[i]);
+}
+
+/* Load a dump from DUMP_FILENAME. We run very early in
+ initialization, so we can't use lisp, unwinding, xmalloc, and so
+ on. */
+enum pdumper_load_result
+pdumper_load (const char *dump_filename)
+{
+ int fd = -1;
+ enum pdumper_load_result err = PDUMPER_LOAD_ERROR;
+ struct loaded_dump ndump;
+ struct stat stat;
+ struct dump_header *header = &ndump.header;
+ ptrdiff_t mark_nbytes;
+
+ memset (&ndump, 0, sizeof (ndump));
+ eassert (!initialized);
+ eassert (!dump_loaded_p ());
+
+ err = PDUMPER_LOAD_FILE_NOT_FOUND;
+ fd = emacs_open (dump_filename, O_RDONLY, 0);
+ if (fd < 0)
+ goto out;
+
+ if (fstat (fd, &stat) < 0)
+ goto out;
+
+ err = PDUMPER_LOAD_BAD_FILE_TYPE;
+ if (stat.st_size < sizeof (*header))
+ goto out;
+
+ err = PDUMPER_LOAD_OOM;
+ ndump.start = malloc (stat.st_size);
+ if (ndump.start == NULL)
+ goto out;
+ eassert ((ptrdiff_t) ndump.start % GCALIGNMENT == 0);
+ ndump.end = ndump.start + stat.st_size;
+
+ err = PDUMPER_LOAD_BAD_FILE_TYPE;
+ if (pdumper_read (fd, ndump.start, stat.st_size) < stat.st_size)
+ goto out;
+
+ memcpy (header, ndump.start, sizeof (*header));
+ if (memcmp (header->magic, dump_magic, sizeof (dump_magic) != 0))
+ goto out;
+
+ err = PDUMPER_LOAD_VERSION_MISMATCH;
+ verify (sizeof (header->fingerprint) == sizeof (fingerprint));
+ if (memcmp (header->fingerprint, fingerprint, sizeof (fingerprint)))
+ goto out;
+
+ err = PDUMPER_LOAD_ERROR;
+ mark_nbytes = dump_mark_bits_nbytes (ndump.header.hot_discardable_start);
+ ndump.mark_bits = calloc (1, mark_nbytes);
+ if (ndump.mark_bits == NULL)
+ goto out;
+
+ /* Point of no return. */
+
+ gflags.dumped_with_pdumper_ = true;
+ loaded_dump = ndump;
+ memset (&ndump, 0, sizeof (ndump));
+ err = PDUMPER_LOAD_SUCCESS;
+
+ dump_do_dump_relocations (&loaded_dump);
+ dump_do_emacs_relocations (&loaded_dump);
+
+ /* Run the functions Emacs registered for doing post-dump-load
+ initialization. */
+ for (int i = 0; i < nr_dump_hooks; ++i)
+ dump_hooks[i] ();
+ gflags.initialized_ = true;
+
+ out:
+ free (ndump.mark_bits);
+ free (ndump.start);
+ if (0 <= fd)
+ emacs_close (fd);
+ return err;
+}
+
+\f
+
+void
+syms_of_pdumper (void)
+{
+ defsubr (&Sdump_emacs_portable);
+ defsubr (&Sdump_emacs_portable__sort_predicate);
+ defsubr (&Sdump_emacs_portable__sort_predicate_copied);
+ DEFSYM (Qdump_emacs_portable__sort_predicate,
+ "dump-emacs-portable--sort-predicate");
+ DEFSYM (Qdump_emacs_portable__sort_predicate_copied,
+ "dump-emacs-portable--sort-predicate-copied");
+}
diff --git a/src/pdumper.h b/src/pdumper.h
new file mode 100644
index 0000000..d6922b7
--- /dev/null
+++ b/src/pdumper.h
@@ -0,0 +1,115 @@
+/* Header file for the portable dumper.
+
+Copyright (C) 2016 Free Software Foundation,
+Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
+
+#ifndef EMACS_PDUMPER_H
+#define EMACS_PDUMPER_H
+
+#include "lisp.h"
+
+/* The portable dumper automatically preserves the Lisp heap and any C
+ variables to which the Lisp heap points. It doesn't know anything
+ about other C variables. The functions below allow code from other
+ parts of Emacs to tell the portable dumper about other bits of
+ information to preserve in dumped images.
+
+ These memory-records are themselves preserved in the dump, so call
+ the functions below only on the !initialized init path, just
+ like staticpro.
+
+ There are no special functions to preserve a global Lisp_Object.
+ You should just staticpro these. */
+
+/* Indicate in source code that we're deliberately relying on pdumper
+ not preserving the given value. Compiles to nothing --- for humans
+ only. */
+#define PDUMPER_IGNORE(thing) ((void) &(thing))
+
+/* Remember the value of THING in dumped images. THING must not
+ contain any pointers or Lisp_Object variables: these values are not
+ valid across dump and load. */
+#define PDUMPER_REMEMBER_SCALAR(thing) \
+ pdumper_remember_scalar (&(thing), sizeof (thing))
+void pdumper_remember_scalar (void *data, ptrdiff_t nbytes);
+
+/* Remember the pointer at *PTR. *PTR must be null or point to a Lisp
+ object. TYPE is the rough type of Lisp object to which *PTR
+ points. */
+void pdumper_remember_lv_raw_ptr (void* ptr, enum Lisp_Type type);
+
+/* Remember the pointer at *PTR. *PTR must be null or point to
+ something in the Emacs process image (e.g., a function). */
+void pdumper_remember_emacs_ptr (void *ptr);
+
+typedef void (*pdumper_hook)(void);
+void pdumper_do_now_and_after_load (pdumper_hook);
+
+/* Macros useful in pdumper callback functions. Assign a value if
+ we're loading a dump and the value needs to be reset to its
+ original value, and if we're initializing for the first time,
+ assert that the value has the expected original value. */
+
+#define PDUMPER_RESET(variable, value) \
+ do { \
+ if (dumped_with_pdumper) \
+ (variable) = (value); \
+ else \
+ eassert ((variable) == (value)); \
+ } while (0)
+
+#define PDUMPER_RESET_LV(variable, value) \
+ do { \
+ if (dumped_with_pdumper) \
+ (variable) = (value); \
+ else \
+ eassert (EQ ((variable), (value))); \
+ } while (0)
+
+/* Actually load a dump. */
+
+enum pdumper_load_result
+ {
+ PDUMPER_LOAD_SUCCESS,
+ PDUMPER_NOT_LOADED /* Not returned: useful for callers */,
+ PDUMPER_LOAD_FILE_NOT_FOUND,
+ PDUMPER_LOAD_BAD_FILE_TYPE,
+ PDUMPER_LOAD_OOM,
+ PDUMPER_LOAD_VERSION_MISMATCH,
+ PDUMPER_LOAD_ERROR,
+ };
+
+enum pdumper_load_result pdumper_load (const char *dump_filename);
+
+_GL_ATTRIBUTE_CONST
+bool pdumper_object_p (const void *obj);
+#define PDUMPER_NO_OBJECT ((enum Lisp_Type) -1)
+_GL_ATTRIBUTE_CONST
+enum Lisp_Type pdumper_find_object_type (const void *obj);
+_GL_ATTRIBUTE_CONST
+bool pdumper_object_p_precise (const void *obj);
+
+bool pdumper_marked_p (const void *obj);
+void pdumper_set_marked (const void *obj);
+void pdumper_clear_marks (void);
+
+void syms_of_pdumper (void);
+
+
+
+#endif
diff --git a/src/process.c b/src/process.c
index 49340b1..6bb2c02 100644
--- a/src/process.c
+++ b/src/process.c
@@ -7797,9 +7797,7 @@ init_process_emacs (int sockfd)
inhibit_sentinels = 0;
-#ifndef CANNOT_DUMP
- if (! noninteractive || initialized)
-#endif
+ if (!will_dump_with_unexec)
{
#if defined HAVE_GLIB && !defined WINDOWSNT
/* Tickle glib's child-handling code. Ask glib to wait for Emacs itself;
diff --git a/src/profiler.c b/src/profiler.c
index 07e21ae..8036fe6 100644
--- a/src/profiler.c
+++ b/src/profiler.c
@@ -21,6 +21,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "lisp.h"
#include "syssignal.h"
#include "systime.h"
+#include "pdumper.h"
/* Return A + B, but return the maximum fixnum if the result would overflow.
Assume A and B are nonnegative and in fixnum range. */
@@ -572,6 +573,8 @@ hashfn_profiler (struct hash_table_test *ht, Lisp_Object bt)
return XHASH (bt);
}
+static void syms_of_profiler_for_pdumper (void);
+
void
syms_of_profiler (void)
{
@@ -610,4 +613,22 @@ to make room for new entries. */);
defsubr (&Sprofiler_memory_stop);
defsubr (&Sprofiler_memory_running_p);
defsubr (&Sprofiler_memory_log);
+
+ pdumper_do_now_and_after_load (syms_of_profiler_for_pdumper);
+}
+
+static void
+syms_of_profiler_for_pdumper (void)
+{
+ if (dumped_with_pdumper)
+ {
+ cpu_log = Qnil;
+ memory_log = Qnil;
+ }
+ else
+ {
+ eassert (NILP (cpu_log));
+ eassert (NILP (memory_log));
+ }
+
}
diff --git a/src/search.c b/src/search.c
index e597c33..68b788b 100644
--- a/src/search.c
+++ b/src/search.c
@@ -29,6 +29,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "region-cache.h"
#include "blockinput.h"
#include "intervals.h"
+#include "pdumper.h"
#include <sys/types.h>
#include "regex.h"
@@ -3390,25 +3391,18 @@ the buffer. If the buffer doesn't have a cache, the value is nil. */)
return val;
}
\f
+
+static void syms_of_search_for_pdumper (void);
+
void
syms_of_search (void)
{
- register int i;
-
- for (i = 0; i < REGEXP_CACHE_SIZE; ++i)
+ for (int i = 0; i < REGEXP_CACHE_SIZE; ++i)
{
- searchbufs[i].buf.allocated = 100;
- searchbufs[i].buf.buffer = xmalloc (100);
- searchbufs[i].buf.fastmap = searchbufs[i].fastmap;
- searchbufs[i].regexp = Qnil;
- searchbufs[i].whitespace_regexp = Qnil;
- searchbufs[i].syntax_table = Qnil;
staticpro (&searchbufs[i].regexp);
staticpro (&searchbufs[i].whitespace_regexp);
staticpro (&searchbufs[i].syntax_table);
- searchbufs[i].next = (i == REGEXP_CACHE_SIZE-1 ? 0 : &searchbufs[i+1]);
}
- searchbuf_head = &searchbufs[0];
/* Error condition used for failing searches. */
DEFSYM (Qsearch_failed, "search-failed");
@@ -3466,4 +3460,22 @@ is to bind it with `let' around a small expression. */);
defsubr (&Sset_match_data);
defsubr (&Sregexp_quote);
defsubr (&Snewline_cache_check);
+
+ pdumper_do_now_and_after_load (syms_of_search_for_pdumper);
+}
+
+static void
+syms_of_search_for_pdumper (void)
+{
+ for (int i = 0; i < REGEXP_CACHE_SIZE; ++i)
+ {
+ searchbufs[i].buf.allocated = 100;
+ searchbufs[i].buf.buffer = xmalloc (100);
+ searchbufs[i].buf.fastmap = searchbufs[i].fastmap;
+ searchbufs[i].regexp = Qnil;
+ searchbufs[i].whitespace_regexp = Qnil;
+ searchbufs[i].syntax_table = Qnil;
+ searchbufs[i].next = (i == REGEXP_CACHE_SIZE-1 ? 0 : &searchbufs[i+1]);
+ }
+ searchbuf_head = &searchbufs[0];
}
diff --git a/src/sysdep.c b/src/sysdep.c
index 892e976..00f128c 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -1850,7 +1850,7 @@ maybe_fatal_sig (int sig)
}
void
-init_signals (bool dumping)
+init_signals (void)
{
struct sigaction thread_fatal_action;
struct sigaction action;
@@ -2001,7 +2001,7 @@ init_signals (bool dumping)
/* Don't alter signal handlers if dumping. On some machines,
changing signal handlers sets static data that would make signals
fail to work right when the dumped Emacs is run. */
- if (dumping)
+ if (will_dump)
return;
sigfillset (&process_fatal_action.sa_mask);
diff --git a/src/syssignal.h b/src/syssignal.h
index 3de83c7..5f9a0da 100644
--- a/src/syssignal.h
+++ b/src/syssignal.h
@@ -22,7 +22,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include <signal.h>
-extern void init_signals (bool);
+extern void init_signals (void);
extern void block_child_signal (sigset_t *);
extern void unblock_child_signal (sigset_t const *);
extern void block_tty_out_signal (sigset_t *);
diff --git a/src/textprop.c b/src/textprop.c
index 7af8c69..fdbb761 100644
--- a/src/textprop.c
+++ b/src/textprop.c
@@ -2340,11 +2340,10 @@ inherits it if NONSTICKINESS is nil. The `front-sticky' and
Vtext_property_default_nonsticky
= list2 (Fcons (Qsyntax_table, Qt), Fcons (Qdisplay, Qt));
- staticpro (&interval_insert_behind_hooks);
- staticpro (&interval_insert_in_front_hooks);
interval_insert_behind_hooks = Qnil;
interval_insert_in_front_hooks = Qnil;
-
+ staticpro (&interval_insert_behind_hooks);
+ staticpro (&interval_insert_in_front_hooks);
/* Common attributes one might give text. */
diff --git a/src/w32fns.c b/src/w32fns.c
index 8c8272b..4c8b651 100644
--- a/src/w32fns.c
+++ b/src/w32fns.c
@@ -56,6 +56,8 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "w32.h"
#endif
+#include "pdumper.h"
+
#include <basetyps.h>
#include <unknwn.h>
#include <commctrl.h>
@@ -9767,6 +9769,7 @@ syms_of_w32fns (void)
track_mouse_window = NULL;
w32_visible_system_caret_hwnd = NULL;
+ PDUMPER_IGNORE (w32_visible_system_caret_hwnd);
DEFSYM (Qundefined_color, "undefined-color");
DEFSYM (Qcancel_timer, "cancel-timer");
diff --git a/src/w32font.c b/src/w32font.c
index 4d15cff..69c0763 100644
--- a/src/w32font.c
+++ b/src/w32font.c
@@ -32,6 +32,8 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "w32.h"
#endif
+#include "pdumper.h"
+
/* Cleartype available on Windows XP, cleartype_natural from XP SP1.
The latter does not try to fit cleartype smoothed fonts into the
same bounding box as the non-antialiased version of the font.
@@ -2567,6 +2569,9 @@ struct font_driver w32font_driver =
/* Initialize state that does not change between invocations. This is only
called when Emacs is dumped. */
+
+static void syms_of_w32font_for_pdumper (void);
+
void
syms_of_w32font (void)
{
@@ -2746,6 +2751,12 @@ versions of Windows) characters. */);
defsubr (&Sx_select_font);
+ pdumper_do_now_and_after_load (syms_of_w32font_for_pdumper);
+}
+
+static void
+syms_of_w32font_for_pdumper (void)
+{
w32font_driver.type = Qgdi;
register_font_driver (&w32font_driver, NULL);
}
diff --git a/src/w32menu.c b/src/w32menu.c
index 7c66360..bc187e2 100644
--- a/src/w32menu.c
+++ b/src/w32menu.c
@@ -30,6 +30,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "buffer.h"
#include "coding.h" /* for ENCODE_SYSTEM */
#include "menu.h"
+#include "pdumper.h"
/* This may include sys/types.h, and that somehow loses
if this is not done before the other system files. */
@@ -1585,6 +1586,7 @@ syms_of_w32menu (void)
globals_of_w32menu ();
current_popup_menu = NULL;
+ PDUMPER_IGNORE (current_popup_menu);
DEFSYM (Qdebug_on_next_call, "debug-on-next-call");
DEFSYM (Qunsupported__w32_dialog, "unsupported--w32-dialog");
diff --git a/src/w32uniscribe.c b/src/w32uniscribe.c
index 5f91b50..000adcb 100644
--- a/src/w32uniscribe.c
+++ b/src/w32uniscribe.c
@@ -36,6 +36,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "composite.h"
#include "font.h"
#include "w32font.h"
+#include "pdumper.h"
struct uniscribe_font_info
{
@@ -1168,9 +1169,17 @@ struct font_driver uniscribe_font_driver =
as it needs to test for the existence of the Uniscribe library. */
void syms_of_w32uniscribe (void);
+static void syms_of_w32uniscribe_for_pdumper (void);
+
void
syms_of_w32uniscribe (void)
{
+ pdumper_do_now_and_after_load (syms_of_w32uniscribe_for_pdumper);
+}
+
+static void
+syms_of_w32uniscribe_for_pdumper (void)
+{
HMODULE uniscribe;
/* Don't init uniscribe when dumping */
diff --git a/src/window.c b/src/window.c
index acbefcd..aa87283 100644
--- a/src/window.c
+++ b/src/window.c
@@ -42,6 +42,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#ifdef MSDOS
#include "msdos.h"
#endif
+#include "pdumper.h"
static ptrdiff_t count_windows (struct window *);
static ptrdiff_t get_leaf_windows (struct window *, struct window **,
@@ -7274,10 +7275,56 @@ and scrolling positions. */)
return Qnil;
}
\f
+
+static void init_window_once_for_pdumper (void);
+
void
init_window_once (void)
{
+ minibuf_window = Qnil;
+ staticpro (&minibuf_window);
+
+ selected_window = Qnil;
+ staticpro (&selected_window);
+
+ Vwindow_list = Qnil;
+ staticpro (&Vwindow_list);
+
+ minibuf_selected_window = Qnil;
+ staticpro (&minibuf_selected_window);
+
+ pdumper_do_now_and_after_load (init_window_once_for_pdumper);
+}
+
+static void init_window_once_for_pdumper (void)
+{
+ window_scroll_pixel_based_preserve_x = -1;
+ window_scroll_pixel_based_preserve_y = -1;
+ window_scroll_preserve_hpos = -1;
+ window_scroll_preserve_vpos = -1;
+ PDUMPER_IGNORE (sequence_number);
+
+ PDUMPER_RESET_LV (minibuf_window, Qnil);
+ PDUMPER_RESET_LV (selected_window, Qnil);
+ PDUMPER_RESET_LV (Vwindow_list, Qnil);
+ PDUMPER_RESET_LV (minibuf_selected_window, Qnil);
+
+ /* Hack: if mode_line_in_non_selected_windows is true (which it may
+ be, if we're restoring from a dump) the guts of
+ make_initial_frame will try to access selected_window, which is
+ invalid at this point, and lose. For the purposes of creating
+ the initial frame and window, this variable must be false. */
+ bool old_mode_line_in_non_selected_windows;
+ if (dumped_with_pdumper)
+ {
+ old_mode_line_in_non_selected_windows
+ = mode_line_in_non_selected_windows;
+ mode_line_in_non_selected_windows = false;
+ }
struct frame *f = make_initial_frame ();
+ if (dumped_with_pdumper)
+ mode_line_in_non_selected_windows =
+ old_mode_line_in_non_selected_windows;
XSETFRAME (selected_frame, f);
Vterminal_frame = selected_frame;
minibuf_window = f->minibuffer_window;
@@ -7324,16 +7371,6 @@ syms_of_window (void)
DEFSYM (Qfloor, "floor");
DEFSYM (Qceiling, "ceiling");
- staticpro (&Vwindow_list);
-
- minibuf_selected_window = Qnil;
- staticpro (&minibuf_selected_window);
-
- window_scroll_pixel_based_preserve_x = -1;
- window_scroll_pixel_based_preserve_y = -1;
- window_scroll_preserve_hpos = -1;
- window_scroll_preserve_vpos = -1;
-
DEFVAR_LISP ("temp-buffer-show-function", Vtemp_buffer_show_function,
doc: /* Non-nil means call as function to display a help buffer.
The function is called with one argument, the buffer to be displayed.
diff --git a/src/xfont.c b/src/xfont.c
index c2b7317..53d3632 100644
--- a/src/xfont.c
+++ b/src/xfont.c
@@ -31,6 +31,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "character.h"
#include "charset.h"
#include "font.h"
+#include "pdumper.h"
\f
/* X core font driver. */
@@ -1113,6 +1114,9 @@ xfont_check (struct frame *f, struct font *font)
}
\f
+
+static void syms_of_xfont_for_pdumper (void);
+
void
syms_of_xfont (void)
{
@@ -1120,6 +1124,12 @@ syms_of_xfont (void)
xfont_scripts_cache = CALLN (Fmake_hash_table, QCtest, Qequal);
staticpro (&xfont_scratch_props);
xfont_scratch_props = Fmake_vector (make_number (8), Qnil);
+ pdumper_do_now_and_after_load (syms_of_xfont_for_pdumper);
+}
+
+static void
+syms_of_xfont_for_pdumper (void)
+{
xfont_driver.type = Qx;
register_font_driver (&xfont_driver, NULL);
}
diff --git a/src/xftfont.c b/src/xftfont.c
index 861ad80..c9efb6d 100644
--- a/src/xftfont.c
+++ b/src/xftfont.c
@@ -32,6 +32,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "composite.h"
#include "font.h"
#include "ftfont.h"
+#include "pdumper.h"
/* Xft font driver. */
@@ -751,6 +752,8 @@ xftfont_cached_font_ok (struct frame *f, Lisp_Object font_object,
return ok;
}
+static void syms_of_xftfont_for_pdumper (void);
+
void
syms_of_xftfont (void)
{
@@ -768,8 +771,12 @@ syms_of_xftfont (void)
This is needed with some fonts to correct vertical overlap of glyphs. */);
xft_font_ascent_descent_override = 0;
- ascii_printable[0] = 0;
+ pdumper_do_now_and_after_load (syms_of_xftfont_for_pdumper);
+}
+static void
+syms_of_xftfont_for_pdumper (void)
+{
xftfont_driver = ftfont_driver;
xftfont_driver.type = Qxft;
xftfont_driver.get_cache = xfont_driver.get_cache;
@@ -789,8 +796,8 @@ This is needed with some fonts to correct vertical overlap of glyphs. */);
xftfont_driver.shape = xftfont_shape;
#endif
/* When using X double buffering, the XftDraw structure we build
- seems to be useless once a frame is resized, so recreate it on
- ConfigureNotify and in some other cases. */
+ seems to be useless once a frame is resized, so recreate it on
+ ConfigureNotify and in some other cases. */
xftfont_driver.drop_xrender_surfaces = xftfont_drop_xrender_surfaces;
register_font_driver (&xftfont_driver, NULL);
diff --git a/src/xmenu.c b/src/xmenu.c
index 9ab7bdf..f080ffb 100644
--- a/src/xmenu.c
+++ b/src/xmenu.c
@@ -44,6 +44,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "buffer.h"
#include "coding.h"
#include "sysselect.h"
+#include "pdumper.h"
#ifdef MSDOS
#include "msdos.h"
@@ -2337,15 +2338,12 @@ DEFUN ("menu-or-popup-active-p", Fmenu_or_popup_active_p, Smenu_or_popup_active_
return (popup_activated ()) ? Qt : Qnil;
}
\f
+
+static void syms_of_xmenu_for_pdumper (void);
+
void
syms_of_xmenu (void)
{
-#ifdef USE_X_TOOLKIT
- enum { WIDGET_ID_TICK_START = 1 << 16 };
- widget_id_tick = WIDGET_ID_TICK_START;
- next_menubar_widget_id = 1;
-#endif
-
DEFSYM (Qdebug_on_next_call, "debug-on-next-call");
defsubr (&Smenu_or_popup_active_p);
@@ -2354,4 +2352,16 @@ syms_of_xmenu (void)
Ffset (intern_c_string ("accelerate-menu"),
intern_c_string (Sx_menu_bar_open_internal.symbol_name));
#endif
+
+ pdumper_do_now_and_after_load (syms_of_xmenu_for_pdumper);
+}
+
+static void
+syms_of_xmenu_for_pdumper (void)
+{
+#ifdef USE_X_TOOLKIT
+ enum { WIDGET_ID_TICK_START = 1 << 16 };
+ widget_id_tick = WIDGET_ID_TICK_START;
+ next_menubar_widget_id = 1;
+#endif
}
diff --git a/src/xselect.c b/src/xselect.c
index b997cc8..bfa0dfb 100644
--- a/src/xselect.c
+++ b/src/xselect.c
@@ -35,6 +35,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "blockinput.h"
#include "termhooks.h"
#include "keyboard.h"
+#include "pdumper.h"
#include <X11/Xproto.h>
@@ -2620,6 +2621,9 @@ x_send_client_event (Lisp_Object display, Lisp_Object dest, Lisp_Object from,
}
\f
+
+static void syms_of_xselect_for_pdumper (void);
+
void
syms_of_xselect (void)
{
@@ -2635,17 +2639,9 @@ syms_of_xselect (void)
reading_selection_reply = Fcons (Qnil, Qnil);
staticpro (&reading_selection_reply);
- reading_selection_window = 0;
- reading_which_selection = 0;
- property_change_wait_list = 0;
- prop_location_identifier = 0;
- property_change_reply = Fcons (Qnil, Qnil);
staticpro (&property_change_reply);
- converted_selections = NULL;
- conversion_fail_tag = None;
-
/* FIXME: Duplicate definition in nsselect.c. */
DEFVAR_LISP ("selection-converter-alist", Vselection_converter_alist,
doc: /* An alist associating X Windows selection-types with functions.
@@ -2724,4 +2720,18 @@ A value of 0 means wait as long as necessary. This is initialized from the
DEFSYM (Qforeign_selection, "foreign-selection");
DEFSYM (Qx_lost_selection_functions, "x-lost-selection-functions");
DEFSYM (Qx_sent_selection_functions, "x-sent-selection-functions");
+
+ pdumper_do_now_and_after_load (syms_of_xselect_for_pdumper);
+}
+
+static void
+syms_of_xselect_for_pdumper (void)
+{
+ reading_selection_window = 0;
+ reading_which_selection = 0;
+ property_change_wait_list = 0;
+ prop_location_identifier = 0;
+ property_change_reply = Fcons (Qnil, Qnil);
+ converted_selections = NULL;
+ conversion_fail_tag = None;
}
diff --git a/src/xsettings.c b/src/xsettings.c
index d7af68f..259cc57 100644
--- a/src/xsettings.c
+++ b/src/xsettings.c
@@ -32,6 +32,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "keyboard.h"
#include "blockinput.h"
#include "termhooks.h"
+#include "pdumper.h"
#include <X11/Xproto.h>
@@ -1008,13 +1009,18 @@ void
syms_of_xsettings (void)
{
current_mono_font = NULL;
+ PDUMPER_IGNORE (current_mono_font);
current_font = NULL;
+ PDUMPER_IGNORE (current_font);
first_dpyinfo = NULL;
+ PDUMPER_IGNORE (first_dpyinfo);
#ifdef HAVE_GSETTINGS
gsettings_client = NULL;
+ PDUMPER_IGNORE (gsettings_client);
#endif
#ifdef HAVE_GCONF
gconf_client = NULL;
+ PDUMPER_IGNORE (gconf_client);
#endif
DEFSYM (Qmonospace_font_name, "monospace-font-name");
diff --git a/src/xterm.c b/src/xterm.c
index bdc21e6..ea21e8f 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -81,6 +81,7 @@ along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>. */
#include "xsettings.h"
#include "sysselect.h"
#include "menu.h"
+#include "pdumper.h"
#ifdef USE_X_TOOLKIT
#include <X11/Shell.h>
@@ -12809,6 +12810,7 @@ void
syms_of_xterm (void)
{
x_error_message = NULL;
+ PDUMPER_IGNORE (x_error_message);
DEFSYM (Qvendor_specific_keysyms, "vendor-specific-keysyms");
DEFSYM (Qlatin_1, "latin-1");
^ permalink raw reply related [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 19:50 Preview: portable dumper Daniel Colascione
@ 2016-11-28 19:58 ` Burton Samograd
2016-11-28 20:11 ` Daniel Colascione
2016-11-28 20:12 ` Eli Zaretskii
` (2 subsequent siblings)
3 siblings, 1 reply; 299+ messages in thread
From: Burton Samograd @ 2016-11-28 19:58 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Emacs developers
Excellent idea and good work. Will this also allow for the dumping of Emacs at arbitrary time after it’s been run, or will there still be issues with that?
Burton Samograd
> On Nov 28, 2016, at 12:50 PM, Daniel Colascione <dancol@dancol.org> wrote:
>
> I've been working on a portable dumper for GNU Emacs. The attached patch is an early version of this work. I'll write up all the usual NEWS entries, changelogs, and (in this case necessary) dedicated documentation before I land it. I want to start getting review comments now that the code has roughly its final shape.
>
> The point of this gargantuan patch is that we can rip out our unexec implementations and replace them with loading a data file that contains an Emacs heap image. There are no dependencies on executable rewriting, disabling ASLR, or saving and restoring internal malloc state. This system works with fully position-independent executables and with any malloc implementation.
>
> Basically, there's a new dump-emacs-portable function that walks the Emacs heap and writes the data, along with necessary relocation, to a file called emacs.pdmp. On startup, early in main, we find emacs.pdmp and load it. Once we've loaded the file, we walk the list of relocations contained in the dump and adjust it to account for the runtime locations of Emacs and the dump data (neither of which we know in advance in a PIE world.)
>
> There are a few subtleties: I've carefully designed the file format to be mmap-able and to minimize the number of on-demand copies the system makes while accessing this file. For example, we stick bool-vectors and string data at the end of the dump in a contiguous block. We follow this block with the relocations, which we can throw away as soon as we've used them.
>
> An additional optimization follows, although this part isn't implemented yet: we can define a "preferred load address" for the dump and write relocation information such that if the dump and Emacs end up being where we expect them to be, we don't have to perform any relocations at all.
>
> The system gracefully degrades though. If we can't use mmap or whatever on a given platform, it's possible to just slurp the whole file into a malloced region of memory and access it from there. This approach would benefit from compression, which will reduce IO loads: LZ4 reduces the dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away
>
> The code isn't even close to optimized yet --- I've only tested it at -O0, defined GC_CHECK_MARKED_OBJECTS, and not yet inlined frequently-called functions pdumper.h --- but even so, it's within 100ms or so of an unexeced Emacs.
>
> It's also possible to dump an already-dumped Emacs, so it should be possible for users to have their own dump files.
>
> If we want to preserve the current model of a single "emacs" executable that contains itself, we can embed emacs.pdmp inside the emacs executable data section pretty easily. There's no behavior change involved.
>
> If you want to try this code, build CANNOT_DUMP=1, run ./temacs -l loadup pdump, then ./emacs (or if that doesn't work, ./emacs --dump-file=emacs.pdmp).
> <pdumper.diff>
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 19:58 ` Burton Samograd
@ 2016-11-28 20:11 ` Daniel Colascione
0 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 20:11 UTC (permalink / raw)
To: Burton Samograd; +Cc: Emacs developers
On 11/28/2016 11:58 AM, Burton Samograd wrote:
> Excellent idea and good work. Will this also allow for the dumping of Emacs at arbitrary time after it’s been run, or will there still be issues with that?
>
In theory. In practice, I want to roll out this work gradually. Step one
is to land it and keep it off by default --- we don't have to stop using
unexec just because we want to experiment with a portable dumper.
Later, we'll enable the portable dumper by default, but as an unexec
replacement, and keep unexec around. Once this arrangement works for a
few releases, we can delete unexec and talk about dumping at later
points in Emacs execution history.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 19:50 Preview: portable dumper Daniel Colascione
2016-11-28 19:58 ` Burton Samograd
@ 2016-11-28 20:12 ` Eli Zaretskii
2016-11-28 20:14 ` Daniel Colascione
` (3 more replies)
2016-11-28 21:14 ` Paul Eggert
2016-11-28 23:01 ` Stefan Monnier
3 siblings, 4 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-28 20:12 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 28 Nov 2016 11:50:31 -0800
>
> I've been working on a portable dumper for GNU Emacs. The attached patch
> is an early version of this work. I'll write up all the usual NEWS
> entries, changelogs, and (in this case necessary) dedicated
> documentation before I land it. I want to start getting review comments
> now that the code has roughly its final shape.
Thanks for your efforts, but I think this is a wrong direction, that
will eventually get us into the same problem we have with unexec: the
need to know and depend on intimate details of relocations, memory
allocation, etc. On top of that, adding Lisp objects will now require
writing its dumper back-end, so this will be a constant maintenance
burden of the kind that only a few of us can bear.
Making the initial load of preloaded Lisp files (most probably, a
single large file) fast enough to allow us to dump the dumping phase
altogether is a much more perspective direction.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:12 ` Eli Zaretskii
@ 2016-11-28 20:14 ` Daniel Colascione
2016-11-28 20:16 ` Daniel Colascione
2016-11-28 20:29 ` Eli Zaretskii
2016-11-28 20:20 ` John Wiegley
` (2 subsequent siblings)
3 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 20:14 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
On 11/28/2016 12:12 PM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 28 Nov 2016 11:50:31 -0800
>>
>> I've been working on a portable dumper for GNU Emacs. The attached patch
>> is an early version of this work. I'll write up all the usual NEWS
>> entries, changelogs, and (in this case necessary) dedicated
>> documentation before I land it. I want to start getting review comments
>> now that the code has roughly its final shape.
>
> Thanks for your efforts, but I think this is a wrong direction, that
> will eventually get us into the same problem we have with unexec: the
> need to know and depend on intimate details of relocations, memory
> allocation, etc. On top of that, adding Lisp objects will now require
> writing its dumper back-end, so this will be a constant maintenance
> burden of the kind that only a few of us can bear.
Only details of Emacs, not the system. I insist that this direction is
the one that will yield adequate performance.
> Making the initial load of preloaded Lisp files (most probably, a
> single large file) fast enough to allow us to dump the dumping phase
> altogether is a much more perspective direction.
That's what this work does. If we disagree, that's a shame. I feel
strongly enough about this issue to fork over it.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:14 ` Daniel Colascione
@ 2016-11-28 20:16 ` Daniel Colascione
2016-11-28 20:29 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 20:16 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
On 11/28/2016 12:14 PM, Daniel Colascione wrote:
> On 11/28/2016 12:12 PM, Eli Zaretskii wrote:
>>> From: Daniel Colascione <dancol@dancol.org>
>>> Date: Mon, 28 Nov 2016 11:50:31 -0800
>>>
>>> I've been working on a portable dumper for GNU Emacs. The attached patch
>>> is an early version of this work. I'll write up all the usual NEWS
>>> entries, changelogs, and (in this case necessary) dedicated
>>> documentation before I land it. I want to start getting review comments
>>> now that the code has roughly its final shape.
>>
>> Thanks for your efforts, but I think this is a wrong direction, that
>> will eventually get us into the same problem we have with unexec: the
>> need to know and depend on intimate details of relocations, memory
>> allocation, etc. On top of that, adding Lisp objects will now require
>> writing its dumper back-end, so this will be a constant maintenance
>> burden of the kind that only a few of us can bear.
>
> Only details of Emacs, not the system. I insist that this direction is
> the one that will yield adequate performance.
>
>> Making the initial load of preloaded Lisp files (most probably, a
>> single large file) fast enough to allow us to dump the dumping phase
>> altogether is a much more perspective direction.
>
> That's what this work does. If we disagree, that's a shame. I feel
> strongly enough about this issue to fork over it.
Also, this work *exists*. I don't see anyone signing up to implement the
giant elc approach, which *I* believe is fundamentally the wrong direction.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:12 ` Eli Zaretskii
2016-11-28 20:14 ` Daniel Colascione
@ 2016-11-28 20:20 ` John Wiegley
2016-11-28 20:22 ` Daniel Colascione
2016-11-28 20:31 ` Eli Zaretskii
2016-11-28 20:21 ` Paul Eggert
2016-11-29 16:55 ` Richard Stallman
3 siblings, 2 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-28 20:20 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Daniel Colascione, emacs-devel
>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
EZ> Thanks for your efforts, but I think this is a wrong direction, that will
EZ> eventually get us into the same problem we have with unexec: the need to
EZ> know and depend on intimate details of relocations, memory allocation,
EZ> etc. On top of that, adding Lisp objects will now require writing its
EZ> dumper back-end, so this will be a constant maintenance burden of the kind
EZ> that only a few of us can bear.
Is there a way to get away from such a requirement, Eli? If unexec becomes
untenable in the future, is there an alternative that doesn't place the burden
upon us to encode the right amount of information in the dumped file?
If I understand Daniel's contribution, he's proposing the equivalent of a
program loader for Emacs Lisp byte-code, stored within what amounts to a
custom object file format.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:12 ` Eli Zaretskii
2016-11-28 20:14 ` Daniel Colascione
2016-11-28 20:20 ` John Wiegley
@ 2016-11-28 20:21 ` Paul Eggert
2016-11-28 20:34 ` Eli Zaretskii
2016-11-29 16:55 ` Richard Stallman
3 siblings, 1 reply; 299+ messages in thread
From: Paul Eggert @ 2016-11-28 20:21 UTC (permalink / raw)
To: Eli Zaretskii, Daniel Colascione; +Cc: emacs-devel
On 11/28/2016 12:12 PM, Eli Zaretskii wrote:
> I think this is a wrong direction
If the proposed dumper works reliably and quickly and portably, Emacs
should be in a significantly better place than it is now. So in that
sense this is not the wrong direction.
Perhaps you're right and making lread faster would work better. However,
nobody has implemented that. If someone does implement it we can switch
to it later, regardless of whether we use the proposed dumper in the
meantime.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:20 ` John Wiegley
@ 2016-11-28 20:22 ` Daniel Colascione
2016-11-28 20:26 ` John Wiegley
2016-11-28 20:31 ` Eli Zaretskii
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 20:22 UTC (permalink / raw)
To: Eli Zaretskii, emacs-devel
On 11/28/2016 12:20 PM, John Wiegley wrote:
>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
> EZ> Thanks for your efforts, but I think this is a wrong direction, that will
> EZ> eventually get us into the same problem we have with unexec: the need to
> EZ> know and depend on intimate details of relocations, memory allocation,
> EZ> etc. On top of that, adding Lisp objects will now require writing its
> EZ> dumper back-end, so this will be a constant maintenance burden of the kind
> EZ> that only a few of us can bear.
>
> Is there a way to get away from such a requirement, Eli? If unexec becomes
> untenable in the future, is there an alternative that doesn't place the burden
> upon us to encode the right amount of information in the dumped file?
>
> If I understand Daniel's contribution, he's proposing the equivalent of a
> program loader for Emacs Lisp byte-code, stored within what amounts to a
> custom object file format.
Yes --- one that *we* control, which frees us from changes to underlying
systems. Yes, the thing has intimate knowledge of Lisp_Object internals,
but whoever changes those internals (and these changes are rare) can
change the dumper too.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:22 ` Daniel Colascione
@ 2016-11-28 20:26 ` John Wiegley
2016-11-28 20:31 ` Daniel Colascione
2016-11-28 20:34 ` Burton Samograd
0 siblings, 2 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-28 20:26 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel
>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
DC> Yes --- one that *we* control, which frees us from changes to underlying
DC> systems. Yes, the thing has intimate knowledge of Lisp_Object internals,
DC> but whoever changes those internals (and these changes are rare) can
DC> change the dumper too.
Then the difference between .elc and this new -- .eld? -- would be exactly
what? Relocation information to speed up load time?
I'm just wondering: if it's strictly a speed gain, then is there any other way
to achieve it; if it's a functionality gain, what are the advantages over just
speeding up .elc loading.
Thanks for doing this work, btw!! The value of your exploration is not lost on
me, since you've now made a concrete alternative for us to consider. If the
differences in approach are small enough, there's no reason we can't change
our minds yet again in the future. This is a purely internal matter, after
all.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:14 ` Daniel Colascione
2016-11-28 20:16 ` Daniel Colascione
@ 2016-11-28 20:29 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-28 20:29 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 28 Nov 2016 12:14:34 -0800
> Cc: emacs-devel@gnu.org
>
> I feel strongly enough about this issue to fork over it.
Don't be ridiculous. Emacs is not my pet project. You just need a
new maintainer, that's all. Because if Emacs is going in a direction
to which I object, I definitely cannot be it.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:20 ` John Wiegley
2016-11-28 20:22 ` Daniel Colascione
@ 2016-11-28 20:31 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-28 20:31 UTC (permalink / raw)
To: John Wiegley; +Cc: dancol, emacs-devel
> From: John Wiegley <jwiegley@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org
> Date: Mon, 28 Nov 2016 12:20:41 -0800
>
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
> EZ> Thanks for your efforts, but I think this is a wrong direction, that will
> EZ> eventually get us into the same problem we have with unexec: the need to
> EZ> know and depend on intimate details of relocations, memory allocation,
> EZ> etc. On top of that, adding Lisp objects will now require writing its
> EZ> dumper back-end, so this will be a constant maintenance burden of the kind
> EZ> that only a few of us can bear.
>
> Is there a way to get away from such a requirement, Eli?
Yes, by loading Lisp code directly.
> If I understand Daniel's contribution, he's proposing the equivalent of a
> program loader for Emacs Lisp byte-code, stored within what amounts to a
> custom object file format.
No, it's not only the byte code, its the entire state of Emacs after
loading that byte code and initializing variables and subroutines not
in Lisp. IOW, all of the Lisp objects in the entire heap, no matter
where they came from.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:26 ` John Wiegley
@ 2016-11-28 20:31 ` Daniel Colascione
2016-11-28 20:37 ` Burton Samograd
2016-11-28 20:39 ` John Wiegley
2016-11-28 20:34 ` Burton Samograd
1 sibling, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 20:31 UTC (permalink / raw)
To: Eli Zaretskii, emacs-devel
On 11/28/2016 12:26 PM, John Wiegley wrote:
>>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
>
> DC> Yes --- one that *we* control, which frees us from changes to underlying
> DC> systems. Yes, the thing has intimate knowledge of Lisp_Object internals,
> DC> but whoever changes those internals (and these changes are rare) can
> DC> change the dumper too.
>
> Then the difference between .elc and this new -- .eld? -- would be exactly
> what? Relocation information to speed up load time?
>
> I'm just wondering: if it's strictly a speed gain, then is there any other way
> to achieve it; if it's a functionality gain, what are the advantages over just
> speeding up .elc loading.
The dump can be used as-is without preparation, particularly if it loads
at its desired base address. Reading an ELC requires lots of parsing,
interning, and allocation of various sorts. It also prohibits any kind
of data sharing between differences instances of Emacs on the same
system, while I expect about a third to half of these dumps to be shared.
It's also possible to dump arbitrarily already-loaded Emacs processes,
which you don't get with a merged ELC, so that's a functionality difference.
If Lisp layout changes in the future, it's very easy to adjust pdumper
to match. It's really not a big maintenance burden; read the code.
> Thanks for doing this work, btw!! The value of your exploration is not lost on
> me, since you've now made a concrete alternative for us to consider. If the
> differences in approach are small enough, there's no reason we can't change
> our minds yet again in the future. This is a purely internal matter, after
> all.
Are you rejecting the patch?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:21 ` Paul Eggert
@ 2016-11-28 20:34 ` Eli Zaretskii
2016-11-28 20:47 ` John Wiegley
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-28 20:34 UTC (permalink / raw)
To: Paul Eggert; +Cc: dancol, emacs-devel
> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 28 Nov 2016 12:21:13 -0800
>
> On 11/28/2016 12:12 PM, Eli Zaretskii wrote:
> > I think this is a wrong direction
>
> If the proposed dumper works reliably and quickly and portably, Emacs
> should be in a significantly better place than it is now. So in that
> sense this is not the wrong direction.
>
> Perhaps you're right and making lread faster would work better. However,
> nobody has implemented that. If someone does implement it we can switch
> to it later, regardless of whether we use the proposed dumper in the
> meantime.
I'm not going to argue. If enough people think I'm mistaken and want
to go the portable dumper way, I will resign right here and now. It
is very easy to convince me to step down, because I hesitated to take
this job to begin with.
Your call, crowd.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:26 ` John Wiegley
2016-11-28 20:31 ` Daniel Colascione
@ 2016-11-28 20:34 ` Burton Samograd
1 sibling, 0 replies; 299+ messages in thread
From: Burton Samograd @ 2016-11-28 20:34 UTC (permalink / raw)
To: John Wiegley; +Cc: Eli Zaretskii, Daniel Colascione, emacs-devel@gnu.org
> I'm just wondering: if it's strictly a speed gain, then is there any other way
> to achieve it; if it's a functionality gain, what are the advantages over just
> speeding up .elc loading.
It sounds like it disconnects the internal unexec format from the underlying system, which could be a good portability gain. It also allows the emacs executable to load different core runtimes and would make the initial binary quite a bit smaller, if I understand correctly.
This type of patch seems like a good move to me and pulls it closer to almost every CL implementation out there.
—
Burton Samograd
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:31 ` Daniel Colascione
@ 2016-11-28 20:37 ` Burton Samograd
2016-11-28 20:44 ` Daniel Colascione
2016-11-28 20:39 ` John Wiegley
1 sibling, 1 reply; 299+ messages in thread
From: Burton Samograd @ 2016-11-28 20:37 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel@gnu.org
> It's also possible to dump arbitrarily already-loaded Emacs processes, which you don't get with a merged ELC, so that's a functionality difference.
You can dump a running process and have it come back exactly, GUI and everything? That would be a godsend, but given the dependency on lower level libraries written in C, I’m having a hard time seeing how this would work.
—
Burton Samograd
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:31 ` Daniel Colascione
2016-11-28 20:37 ` Burton Samograd
@ 2016-11-28 20:39 ` John Wiegley
1 sibling, 0 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-28 20:39 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel
>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
DC> Are you rejecting the patch?
No, not at all. I'll follow up more in my response to Eli.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:37 ` Burton Samograd
@ 2016-11-28 20:44 ` Daniel Colascione
2016-11-29 16:02 ` Ted Zlatanov
2016-11-29 16:48 ` Richard Stallman
0 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 20:44 UTC (permalink / raw)
To: Burton Samograd; +Cc: Eli Zaretskii, emacs-devel@gnu.org
On November 28, 2016 12:37:52 PM PST, Burton Samograd <burton.samograd@autodesk.com> wrote:
>
>> It's also possible to dump arbitrarily already-loaded Emacs
>processes, which you don't get with a merged ELC, so that's a
>functionality difference.
>
>You can dump a running process and have it come back exactly, GUI and
>everything? That would be a godsend, but given the dependency on lower
>level libraries written in C, I’m having a hard time seeing how this
>would work.
Frame and window objects come back dead after being restored from the dump. Buffers are intact though. If you want to preserve window configuration, you can layer saving and restoring of layout on top, just like desktop.el does.
>
>—
>Burton Samograd
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:34 ` Eli Zaretskii
@ 2016-11-28 20:47 ` John Wiegley
2016-11-28 21:14 ` Eli Zaretskii
0 siblings, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-11-28 20:47 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Paul Eggert, dancol, emacs-devel
>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
EZ> I'm not going to argue. If enough people think I'm mistaken and want to go
EZ> the portable dumper way, I will resign right here and now. It is very easy
EZ> to convince me to step down, because I hesitated to take this job to begin
EZ> with.
EZ> Your call, crowd.
OK, please let's take a few steps back and not make any rash decisions
today. :)
If Daniel's proposing an internal optimization and is willing to maintain it
-- and if it doesn't make work for other contributors -- I suggest we accept
it as long as he, or someone else, is willing to support it. At the very
least, it frees us from dependence on a glibc feature that is doomed to
disappear.
If later the work becomes difficult to support, or if Daniel disappears, won't
we just be returning to the same position as now? If that happeans, we can
take what we've learned and try another approach, such as speeding up .elc
loading. Or, if someone comes up with a better alternative in the meantime, we
can just switch to it.
Accepting this patch doesn't mean we don't try the .elc speedup idea. The only
thing we have to make sure is that it doesn't unduly increase the difficulty
of adding new Lisp objects.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:47 ` John Wiegley
@ 2016-11-28 21:14 ` Eli Zaretskii
2016-11-28 21:55 ` Daniel Colascione
2016-11-28 22:18 ` John Wiegley
0 siblings, 2 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-28 21:14 UTC (permalink / raw)
To: John Wiegley; +Cc: eggert, dancol, emacs-devel
> From: John Wiegley <jwiegley@gmail.com>
> Cc: Paul Eggert <eggert@cs.ucla.edu>, dancol@dancol.org, emacs-devel@gnu.org
> Date: Mon, 28 Nov 2016 12:47:10 -0800
>
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
> EZ> I'm not going to argue. If enough people think I'm mistaken and want to go
> EZ> the portable dumper way, I will resign right here and now. It is very easy
> EZ> to convince me to step down, because I hesitated to take this job to begin
> EZ> with.
>
> EZ> Your call, crowd.
>
> OK, please let's take a few steps back and not make any rash decisions
> today. :)
It's no longer my decision.
> If Daniel's proposing an internal optimization and is willing to maintain it
> -- and if it doesn't make work for other contributors -- I suggest we accept
> it as long as he, or someone else, is willing to support it.
Emacs future cannot depend on a single person. We had more than once
a situation where key developer(s) left leaving a feature or a whole
area unmaintained. If some development isn't a one-time job -- and
this one surely isn't -- then experience shows that any arcane code in
C that loses its experts will become unmaintained and unmaintainable.
The only hope for us to escape this fate is to do it in a way that
most contemporary contributors understand and can easily learn to
hack, which means do it mostly in Lisp, reusing the existing C
machinery that is proven by time and doesn't need maintenance --
'load' and its cousins. That was what I proposed. The advantage of
that is that Emacs then becomes a normal program, one that doesn't
dump itself in any way, shape, or form, but instead reads in a
byte-compiled file, using the normal memory allocation routines. No
more need to save any internal state, worry about relocations, ASLR,
etc. It amazes me that the advantages of that are not immediately
clear to everyone, but I guess I'm biased.
> At the very least, it frees us from dependence on a glibc feature
> that is doomed to disappear.
No one is suggesting to stay with unexec. The disagreement is about
where to go to get rid of it.
> If later the work becomes difficult to support, or if Daniel disappears, won't
> we just be returning to the same position as now? If that happeans, we can
> take what we've learned and try another approach, such as speeding up .elc
> loading. Or, if someone comes up with a better alternative in the meantime, we
> can just switch to it.
I don't think this is practical. We cannot go back, certainly not
when unexec requires the expertise it does. No one will agree to go
back. Like with unexec, the only way the portable dumper will be
thrown away is if/when it turns out it is an obstacle to Emacs
development, by which time it will be too late.
> Accepting this patch doesn't mean we don't try the .elc speedup idea. The only
> thing we have to make sure is that it doesn't unduly increase the difficulty
> of adding new Lisp objects.
Accepting this patch means we believe in this direction, consider it a
perspective one, enough so that we are willing to invest a non-trivial
amount of energy into making it happen, fix the fallout, debug the
results, etc. We are talking about at least one major release, maybe
more, and definitely several minor releases. IOW, we are talking
several years. I don't think we should gamble with such high stakes,
not unless we sure we want to go there, because this is the best
alternative of ditching unexec.
Anyway, I already wrote all this; there was a long discussion about
these issues, it's in the archives. I have nothing else to add to
what I wrote back then. This project needs to decide where it wants
to go, and the rest will be history.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 19:50 Preview: portable dumper Daniel Colascione
2016-11-28 19:58 ` Burton Samograd
2016-11-28 20:12 ` Eli Zaretskii
@ 2016-11-28 21:14 ` Paul Eggert
2016-11-28 23:01 ` Stefan Monnier
3 siblings, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2016-11-28 21:14 UTC (permalink / raw)
To: Daniel Colascione, Emacs developers
Thanks for working on this. Although it's not the approach I would have
taken, there is a real advantage to having running code. A few points:
* The portable dumper doesn't work with modules. Although this should be
OK for Emacs builds since they don't use modules either, what is your
thought about this? Will the module API need to change to support dumping?
* I am working on getting Emacs to work with gcc -fcheck-pointer-bounds,
and will boost the priority of that task and CC: you when I have
something ready. The portable dumper should work in such an environment.
I don't see any fundamental problems with this; just giving you a heads-up.
* Functions like do_dump_relocation should use intptr_t or uintptr_t,
not ptrdiff_t, to access pointers as integers (whether relocated or
not). In theory this would support architectures where intptr_t and
ptrdiff_t have different widths. Although Emacs doesn't run on such
architectures now, conceivably it could and in the meantime the code
might as well be clear about the distinction between pointers and offsets.
* pdumper_load should check that S_ISREG (stat.st_mode) (I suppose this
won't be needed if we load from the emacs executable....).
* I suggest using the type bits_word (defined in lisp.h) rather than
unsigned. This should work better on 64-bit machines (plus, on weird
platforms).
* For pdumper_read a more-portable limit is MAX_RW_COUNT (see sysdep.c)
instead of SSIZE_MAX. Maybe that value needs to move into a .h file.
* A nit: pdumper_marked_p relies on implicit conversion to bool, a style
I find confusing for bool values. In a bool function it's clearer to
write 'return E != 0;' instead of 'return E;' when E might have values
other than 0 or 1.
* I haven't thought through all the implications of this and may have
more comments later.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 21:14 ` Eli Zaretskii
@ 2016-11-28 21:55 ` Daniel Colascione
2016-11-28 22:18 ` John Wiegley
1 sibling, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 21:55 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: John Wiegley, eggert, emacs-devel
On Mon, Nov 28 2016, Eli Zaretskii wrote:
>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: Paul Eggert <eggert@cs.ucla.edu>, dancol@dancol.org, emacs-devel@gnu.org
>> Date: Mon, 28 Nov 2016 12:47:10 -0800
>>
>> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>>
>> EZ> I'm not going to argue. If enough people think I'm mistaken and want to go
>> EZ> the portable dumper way, I will resign right here and now. It is very easy
>> EZ> to convince me to step down, because I hesitated to take this job to begin
>> EZ> with.
>>
>> EZ> Your call, crowd.
>>
>> OK, please let's take a few steps back and not make any rash decisions
>> today. :)
>
> It's no longer my decision.
>
>> If Daniel's proposing an internal optimization and is willing to maintain it
>> -- and if it doesn't make work for other contributors -- I suggest we accept
>> it as long as he, or someone else, is willing to support it.
>
> Emacs future cannot depend on a single person.
We are a long way away from the day when only one person left on earth
can simultaneously hold in his mind the concepts of addition, modulus,
and binary search in C. The operating principle of this work is very
simple. If we need internals documentation, I'll write documentation ---
but there's really not much to it.
> We had more than once
> a situation where key developer(s) left leaving a feature or a whole
> area unmaintained. If some development isn't a one-time job -- and
> this one surely isn't -- then experience shows that any arcane code in
> C that loses its experts will become unmaintained and unmaintainable.
> The only hope for us to escape this fate is to do it in a way that
> most contemporary contributors understand and can easily learn to
> hack, which means do it mostly in Lisp, reusing the existing C
> machinery that is proven by time and doesn't need maintenance --
> 'load' and its cousins.
They don't need maintenance --- except that they're at least an order of
magnitude too slow right now, and nobody's stepped up to do the
maintenance work necessary to make them useful in a new use case.
> That was what I proposed. The advantage of
> that is that Emacs then becomes a normal program, one that doesn't
> dump itself in any way, shape, or form, but instead reads in a
> byte-compiled file, using the normal memory allocation routines. No
> more need to save any internal state, worry about relocations, ASLR,
> etc. It amazes me that the advantages of that are not immediately
> clear to everyone, but I guess I'm biased.
The whole point of the portable dumper is that you don't need to worry
about ASLR, ASAN, TSAN, or whatever libc people dream up next. Emacs
*is* a normal program --- it does only the things normal programs are
allowed to do and that platforms support, things like pointer assignment
and reading from a file. A "relocation" is literally just addition ---
it's not some horrible dynamic programming monstrosity like redisplay,
which I understand used to literally have a skull and crossbones in
ASCII art over the main body of the code.
Using your arguments, we should switch to Pango for text layout, FUSE
for remote access, clang-format for indentation, and --- hell, let's
just make Emacs a webapp. If you want maximum technological
democratization, you want JavaScript. See https://atom.io/.
>> At the very least, it frees us from dependence on a glibc feature
>> that is doomed to disappear.
>
> No one is suggesting to stay with unexec. The disagreement is about
> where to go to get rid of it.
>
>> If later the work becomes difficult to support, or if Daniel disappears, won't
>> we just be returning to the same position as now? If that happeans, we can
>> take what we've learned and try another approach, such as speeding up .elc
>> loading. Or, if someone comes up with a better alternative in the meantime, we
>> can just switch to it.
>
> I don't think this is practical. We cannot go back, certainly not
> when unexec requires the expertise it does. No one will agree to go
> back. Like with unexec, the only way the portable dumper will be
> thrown away is if/when it turns out it is an obstacle to Emacs
> development, by which time it will be too late.
Unexec will keep working as well as it does today.
>> Accepting this patch doesn't mean we don't try the .elc speedup idea. The only
>> thing we have to make sure is that it doesn't unduly increase the difficulty
>> of adding new Lisp objects.
>
> Accepting this patch means we believe in this direction, consider it a
> perspective one, enough so that we are willing to invest a non-trivial
> amount of energy into making it happen, fix the fallout, debug the
> results, etc. We are talking about at least one major release, maybe
> more, and definitely several minor releases. IOW, we are talking
> several years. I don't think we should gamble with such high stakes,
> not unless we sure we want to go there, because this is the best
> alternative of ditching unexec.
The stakes aren't high. As johnw wrote, unexec will keep working as well
as it does today, and nobody is stopping anyone else making
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 21:14 ` Eli Zaretskii
2016-11-28 21:55 ` Daniel Colascione
@ 2016-11-28 22:18 ` John Wiegley
2016-11-29 18:40 ` Eli Zaretskii
1 sibling, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-11-28 22:18 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: eggert, dancol, emacs-devel
>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> Accepting this patch means we believe in this direction, consider it a
> perspective one, enough so that we are willing to invest a non-trivial
> amount of energy into making it happen, fix the fallout, debug the results,
> etc. We are talking about at least one major release, maybe more, and
> definitely several minor releases. IOW, we are talking several years. I
> don't think we should gamble with such high stakes, not unless we sure we
> want to go there, because this is the best alternative of [...]
If I didn't know you were talking about pdumper vs. unexec here, I'd swear
this above paragraph was expressing my own concerns about merging Tromey's
concurrency branch. My issues then were exactly the same: very few people who
understand the C implementation, and a multi-year commitment down a road that
may not be the right solution. Yet you agreed that merging it in was fine,
since we could gain experience and at least there's someone who actually did
the work.
Why is pdumper not the same scenario? We have commitment from someone willing
to work on it, he's even willing to write documentation, and it will give us
experience in an area we're free to change at any time, because it doesn't
expose APIs to the end user.
I say we try it, while keeping unexec in place, until it becomes more obvious
that we shouldn't. If we wait too long for the "best alternative", it might
the same as what happened to concurrency: it never happens, because there's
always a concern about the unexpected-down-the-road.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 19:50 Preview: portable dumper Daniel Colascione
` (2 preceding siblings ...)
2016-11-28 21:14 ` Paul Eggert
@ 2016-11-28 23:01 ` Stefan Monnier
2016-11-28 23:17 ` Daniel Colascione
3 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-11-28 23:01 UTC (permalink / raw)
To: emacs-devel
> I've been working on a portable dumper for GNU Emacs. The attached patch is
> an early version of this work.
Cool!
> An additional optimization follows, although this part isn't implemented
> yet: we can define a "preferred load address" for the dump and write
> relocation information such that if the dump and Emacs end up being where we
> expect them to be, we don't have to perform any relocations at all.
Oh, so the in-file format is also a valid in-memory format? Great.
AFAIK, that's the crucial difference between a "fast .elc loader" and
your approach. This should be key to get really good performance
(e.g. good enough that it doesn't slow down bootstrap noticeably).
> would benefit from compression, which will reduce IO loads: LZ4 reduces the
> dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away
Hmm... interesting the "dump one big .elc" approach I played with (which
kinda works right now, tho with the known performance issues) leads to
a dumped.elc file of a size a bit below 4MB (which gzip compresses down
to ~1MB).
I assume your system is using 64bit Lisp_Object, right? So on a 32bit
Lisp_Object your dump size is probably closer to 6-7MB.
> functions pdumper.h --- but even so, it's within 100ms or so of an
> unexeced Emacs.
0.1s will still lead to a noticeable slow down of bootstrap, but at
least it should be good enough for the usual end-user who doesn't run
Emacs in batch mode.
> It's also possible to dump an already-dumped Emacs, so it should be possible
> for users to have their own dump files.
[ FWIW in theory my dumped.elc approach could also work on an
already-dumped Emacs, for use by end-users (it basically just prints the
contents of the obarray), tho it would indeed need extra work to weed out
problems printing objects like markers and such. ]
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 23:01 ` Stefan Monnier
@ 2016-11-28 23:17 ` Daniel Colascione
2016-11-29 13:06 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-28 23:17 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
On Mon, Nov 28 2016, Stefan Monnier wrote:
>> I've been working on a portable dumper for GNU Emacs. The attached patch is
>> an early version of this work.
>
> Cool!
>
>> An additional optimization follows, although this part isn't implemented
>> yet: we can define a "preferred load address" for the dump and write
>> relocation information such that if the dump and Emacs end up being where we
>> expect them to be, we don't have to perform any relocations at all.
>
> Oh, so the in-file format is also a valid in-memory format? Great.
> AFAIK, that's the crucial difference between a "fast .elc loader" and
> your approach. This should be key to get really good performance
> (e.g. good enough that it doesn't slow down bootstrap noticeably).
Yes. The file is full of valid memory images of Lisp objects --- that's
why I'm guessing it might be a bit more verbose than ELC,
pre-compression. If you guess correctly at dump time what addresses the
Emacs and dump images will have, you can direct mmap it directly and
run. With a non-PIC executable, you can do a pretty good job of that
and get performance a bit better than unexec, in theory.
If you load a dump and find that you've guessed wrong on the base
addresses, you need to relocate the image, but that can be done cheaply
and gradually, on access.
>> would benefit from compression, which will reduce IO loads: LZ4 reduces the
>> dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away
>
> Hmm... interesting the "dump one big .elc" approach I played with (which
> kinda works right now, tho with the known performance issues) leads to
> a dumped.elc file of a size a bit below 4MB (which gzip compresses down
> to ~1MB).
>
> I assume your system is using 64bit Lisp_Object, right? So on a 32bit
> Lisp_Object your dump size is probably closer to 6-7MB.
Right --- I've tested on 64-bit only. I imagine testing 32-bit and
wide-int might reveal some latent bugs.
>
>> functions pdumper.h --- but even so, it's within 100ms or so of an
>> unexeced Emacs.
>
> 0.1s will still lead to a noticeable slow down of bootstrap, but at
> least it should be good enough for the usual end-user who doesn't run
> Emacs in batch mode.
Oh, of course. But that's without any attention paid to
optimization. I'm sure it can go much faster.
>> It's also possible to dump an already-dumped Emacs, so it should be possible
>> for users to have their own dump files.
>
> [ FWIW in theory my dumped.elc approach could also work on an
> already-dumped Emacs, for use by end-users (it basically just prints the
> contents of the obarray), tho it would indeed need extra work to weed out
> problems printing objects like markers and such. ]
Yeah --- then both approaches get into the problem of how to dump
arbitrary object graphs, which in practice ends up being "dump the whole
heap except for processes, windows, and frames."
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 23:17 ` Daniel Colascione
@ 2016-11-29 13:06 ` Stefan Monnier
2016-11-29 21:19 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-11-29 13:06 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
> run. With a non-PIC executable, you can do a pretty good job of that
> and get performance a bit better than unexec, in theory.
Got me curious: how/why could it be even better than unexec?
> Right --- I've tested on 64-bit only. I imagine testing 32-bit and
> wide-int might reveal some latent bugs.
I'm not worried about that.
>>> functions pdumper.h --- but even so, it's within 100ms or so of an
>>> unexeced Emacs.
>> 0.1s will still lead to a noticeable slow down of bootstrap, but at
>> least it should be good enough for the usual end-user who doesn't run
>> Emacs in batch mode.
> Oh, of course. But that's without any attention paid to
> optimization. I'm sure it can go much faster.
Sure, but it's already fast enough to be usable.
[ Contrary to my dumped.elc, which is currently usable only for
non-short-lived interactive sessions where you're willing to spend an
extra second of startup. ]
> Yeah --- then both approaches get into the problem of how to dump
> arbitrary object graphs, which in practice ends up being "dump the whole
> heap except for processes, windows, and frames."
But the dumped.elc approach has it much more difficult here because it
has to work within the scope of a predefined existing format.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:44 ` Daniel Colascione
@ 2016-11-29 16:02 ` Ted Zlatanov
2016-11-29 17:58 ` Daniel Colascione
2016-11-29 16:48 ` Richard Stallman
1 sibling, 1 reply; 299+ messages in thread
From: Ted Zlatanov @ 2016-11-29 16:02 UTC (permalink / raw)
To: emacs-devel
On Mon, 28 Nov 2016 12:44:50 -0800 Daniel Colascione <dancol@dancol.org> wrote:
DC> Frame and window objects come back dead after being restored from the dump.
DC> Buffers are intact though. If you want to preserve window configuration, you can
DC> layer saving and restoring of layout on top, just like desktop.el does.
Since this will also be available to users, is there a way to exclude
some variables from dumping explicitly when they are declared? For
instance cache data, especially cached passwords?
Thanks
Ted
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:44 ` Daniel Colascione
2016-11-29 16:02 ` Ted Zlatanov
@ 2016-11-29 16:48 ` Richard Stallman
2016-11-29 17:32 ` Daniel Colascione
2016-11-29 17:43 ` Eli Zaretskii
1 sibling, 2 replies; 299+ messages in thread
From: Richard Stallman @ 2016-11-29 16:48 UTC (permalink / raw)
To: Daniel Colascione; +Cc: eliz, burton.samograd, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> Frame and window objects come back dead after being restored from the dump.
Can that be fixed by creating new equivalent windows and frames?
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 20:12 ` Eli Zaretskii
` (2 preceding siblings ...)
2016-11-28 20:21 ` Paul Eggert
@ 2016-11-29 16:55 ` Richard Stallman
2016-11-29 18:39 ` Eli Zaretskii
3 siblings, 1 reply; 299+ messages in thread
From: Richard Stallman @ 2016-11-29 16:55 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: dancol, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> Thanks for your efforts, but I think this is a wrong direction, that
> will eventually get us into the same problem we have with unexec: the
> need to know and depend on intimate details of relocations, memory
> allocation, etc.
Maintaining the portable dumper will be far simpler than maintaining
unexec. Unexec has to write a file in whatever format the system uses
for executables. If that is ELF format, it is very complicated,
and it wasn't designed for this purpose. It is necessary to copy
the material from the temacs executable, and modify it along the way.
All those problems will be absent for the portable dumper, which
will make its maintenance much easier.
On top of that, adding Lisp objects will now require
> writing its dumper back-end, so this will be a constant maintenance
> burden of the kind that only a few of us can bear.
Yes, but how often do we make such changes? Once every few years,
I think. And it won't be a big difficulty to do so, if the code of
the dumper is clean. I am sure it will be a lot cleaner than the code
of unexec.
> Making the initial load of preloaded Lisp files (most probably, a
> single large file) fast enough to allow us to dump the dumping phase
> altogether is a much more perspective direction.
I think this has its own difficulty -- the need to represent everything
in Lisp syntax.
I don't know a priori which one will be cleaner. It would be good to
implement this alternative so we can compare them for real rather than
just in our imaginations.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 16:48 ` Richard Stallman
@ 2016-11-29 17:32 ` Daniel Colascione
2016-11-29 19:55 ` Philippe Vaucher
2016-11-29 17:43 ` Eli Zaretskii
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 17:32 UTC (permalink / raw)
To: Richard Stallman; +Cc: eliz, burton.samograd, emacs-devel
On Tue, Nov 29 2016, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider ]]]
> [[[ whether defending the US Constitution against all enemies, ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> > Frame and window objects come back dead after being restored from the dump.
>
> Can that be fixed by creating new equivalent windows and frames?
Yes: you'd want to reuse desktop's code to save window and frame
configuration. I plan to provide a both-dump-hook and after-dump-hook
mechanism that Lisp code can use for tasks like this.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 16:48 ` Richard Stallman
2016-11-29 17:32 ` Daniel Colascione
@ 2016-11-29 17:43 ` Eli Zaretskii
2016-11-29 17:49 ` Daniel Colascione
2016-11-29 18:03 ` John Wiegley
1 sibling, 2 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 17:43 UTC (permalink / raw)
To: rms; +Cc: dancol, emacs-devel, burton.samograd
> From: Richard Stallman <rms@gnu.org>
> CC: burton.samograd@autodesk.com, eliz@gnu.org, emacs-devel@gnu.org
> Date: Tue, 29 Nov 2016 11:48:40 -0500
>
> > Frame and window objects come back dead after being restored from the dump.
>
> Can that be fixed by creating new equivalent windows and frames?
I'm actually wondering why frames and windows (and buffers, for that
matter) need to be dumped at all. The fact we do that now is just an
artifact of how unexec works: it dumps the entire data section. We
even have special code that runs at startup to undo some of the
peculiarities this causes.
However, a dumper that we control by manually telling it what to dum
and what not doesn't have to do that. IMO, that's just something that
doesn't need to exist in the brave new unexec-free world. (Note that
the "one huge .elc file" method has this built-in.)
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 17:43 ` Eli Zaretskii
@ 2016-11-29 17:49 ` Daniel Colascione
2016-11-29 18:17 ` Eli Zaretskii
2016-11-29 18:03 ` John Wiegley
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 17:49 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: burton.samograd, rms, emacs-devel
On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: Richard Stallman <rms@gnu.org>
>> CC: burton.samograd@autodesk.com, eliz@gnu.org, emacs-devel@gnu.org
>> Date: Tue, 29 Nov 2016 11:48:40 -0500
>>
>> > Frame and window objects come back dead after being restored from the dump.
>>
>> Can that be fixed by creating new equivalent windows and frames?
>
> I'm actually wondering why frames and windows (and buffers, for that
> matter) need to be dumped at all. The fact we do that now is just an
> artifact of how unexec works: it dumps the entire data section. We
> even have special code that runs at startup to undo some of the
> peculiarities this causes.
>
> However, a dumper that we control by manually telling it what to dum
> and what not doesn't have to do that. IMO, that's just something that
> doesn't need to exist in the brave new unexec-free world. (Note that
> the "one huge .elc file" method has this built-in.)
Frames and windows aren't dumped. That's why they come back as
dead objects. Lisp objects have to point to *something*, and a dead
object of the right type is a good placeholder.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 16:02 ` Ted Zlatanov
@ 2016-11-29 17:58 ` Daniel Colascione
0 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 17:58 UTC (permalink / raw)
To: emacs-devel
On Tue, Nov 29 2016, Ted Zlatanov wrote:
> On Mon, 28 Nov 2016 12:44:50 -0800 Daniel Colascione <dancol@dancol.org> wrote:
>
> DC> Frame and window objects come back dead after being restored from the dump.
> DC> Buffers are intact though. If you want to preserve window
> DC> configuration, you can
> DC> layer saving and restoring of layout on top, just like desktop.el does.
>
> Since this will also be available to users, is there a way to exclude
> some variables from dumping explicitly when they are declared? For
> instance cache data, especially cached passwords?
Just let-bind any variables you want with the values you'd prefer them
to have in the dump. We can come up with something fancier if needed.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 17:43 ` Eli Zaretskii
2016-11-29 17:49 ` Daniel Colascione
@ 2016-11-29 18:03 ` John Wiegley
2016-11-29 18:23 ` Eli Zaretskii
1 sibling, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-11-29 18:03 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: burton.samograd, dancol, rms, emacs-devel
>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
EZ> However, a dumper that we control by manually telling it what to dum and
EZ> what not doesn't have to do that. IMO, that's just something that doesn't
EZ> need to exist in the brave new unexec-free world. (Note that the "one huge
EZ> .elc file" method has this built-in.)
I'd still like to see research into the "one huge .elc file" method, for the
obvious benefits you listed. As far as I understand, the only difference
between the portable dumper, and one-huge-.elc, is efficiency. Given infinite
resources, the .elc method is much to be preferred, since everything we need
"falls out" from existing design.
I wonder what it would take to bridge that efficiency gap, or am I missing
some new feature provided by the portable dumper?
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 17:49 ` Daniel Colascione
@ 2016-11-29 18:17 ` Eli Zaretskii
0 siblings, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 18:17 UTC (permalink / raw)
To: Daniel Colascione; +Cc: burton.samograd, rms, emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Cc: rms@gnu.org, emacs-devel@gnu.org, burton.samograd@autodesk.com
> Date: Tue, 29 Nov 2016 09:49:59 -0800
>
> > I'm actually wondering why frames and windows (and buffers, for that
> > matter) need to be dumped at all. The fact we do that now is just an
> > artifact of how unexec works: it dumps the entire data section. We
> > even have special code that runs at startup to undo some of the
> > peculiarities this causes.
> >
> > However, a dumper that we control by manually telling it what to dum
> > and what not doesn't have to do that. IMO, that's just something that
> > doesn't need to exist in the brave new unexec-free world. (Note that
> > the "one huge .elc file" method has this built-in.)
>
> Frames and windows aren't dumped. That's why they come back as
> dead objects. Lisp objects have to point to *something*, and a dead
> object of the right type is a good placeholder.
Which Lisp objects need to point to frames and windows?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:03 ` John Wiegley
@ 2016-11-29 18:23 ` Eli Zaretskii
2016-11-29 18:49 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 18:23 UTC (permalink / raw)
To: John Wiegley; +Cc: burton.samograd, dancol, rms, emacs-devel
> From: John Wiegley <jwiegley@gmail.com>
> Cc: rms@gnu.org, dancol@dancol.org, emacs-devel@gnu.org, burton.samograd@autodesk.com
> Date: Tue, 29 Nov 2016 10:03:01 -0800
>
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
> EZ> However, a dumper that we control by manually telling it what to dum and
> EZ> what not doesn't have to do that. IMO, that's just something that doesn't
> EZ> need to exist in the brave new unexec-free world. (Note that the "one huge
> EZ> .elc file" method has this built-in.)
>
> I'd still like to see research into the "one huge .elc file" method, for the
> obvious benefits you listed.
Stefan and Ken made some good progress. IMO, we should actively
continue pursuing that goal.
> As far as I understand, the only difference between the portable
> dumper, and one-huge-.elc, is efficiency. Given infinite resources,
> the .elc method is much to be preferred, since everything we need
> "falls out" from existing design.
Any memory dumper will always be more efficient than loading .elc.
The challenge is to make the difference smaller than the annoyance
level of the users. Since initial testing already produces load times
below 0.5 sec, I don't see why further optimizations won't deliver
enough speedup to make it fast enough.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 16:55 ` Richard Stallman
@ 2016-11-29 18:39 ` Eli Zaretskii
2016-11-29 19:03 ` Daniel Colascione
` (2 more replies)
0 siblings, 3 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 18:39 UTC (permalink / raw)
To: rms; +Cc: dancol, emacs-devel
> From: Richard Stallman <rms@gnu.org>
> CC: dancol@dancol.org, emacs-devel@gnu.org
> Date: Tue, 29 Nov 2016 11:55:16 -0500
>
> Maintaining the portable dumper will be far simpler than maintaining
> unexec. Unexec has to write a file in whatever format the system uses
> for executables. If that is ELF format, it is very complicated,
> and it wasn't designed for this purpose. It is necessary to copy
> the material from the temacs executable, and modify it along the way.
>
> All those problems will be absent for the portable dumper, which
> will make its maintenance much easier.
Easier: yes. Easy enough: no, not IMO. Not for most of the Emacs
contributors we have and will have in the near future.
The number of people aboard who can matter-of-factly hack the Emacs
internals on the C level is consistently going down, and is already so
small they can be counted on one hand. We must make Emacs depend less
on people from this small and diminishing group, if we want the
development pace increased or even kept at its current level. To me,
that means keep as many features out of low-level C, and limit futzing
with C-level internals of Lisp objects and the Lisp interpreter to the
absolute minimum. IOW, any feature that can have an alternative
implementation in Lisp should not be done in C, even if the Lisp
implementation will be slightly slower.
> On top of that, adding Lisp objects will now require
> > writing its dumper back-end, so this will be a constant maintenance
> > burden of the kind that only a few of us can bear.
>
> Yes, but how often do we make such changes? Once every few years,
> I think.
More like once a year, maybe more.
> And it won't be a big difficulty to do so, if the code of the dumper
> is clean.
For you and me, and for a few others, sure. But Emacs cannot be
developed by a handful of people anymore. We already fall behind with
frequency of releases, with the speed of delivering new features, with
analyzing and fixing bugs reported to the tracker, etc. If we don't
attract more core developers, we will never get out of this vicious
cycle. And bringing C-level core developers on board is a very slow,
painful, and unreliable process, because C programmers are a dying
breed, and Emacs internals are not for the faint at heart.
> I am sure it will be a lot cleaner than the code of unexec.
I'm sure, too, but that's not the issue.
> > Making the initial load of preloaded Lisp files (most probably, a
> > single large file) fast enough to allow us to dump the dumping phase
> > altogether is a much more perspective direction.
>
> I think this has its own difficulty -- the need to represent everything
> in Lisp syntax.
I don't think I follow: what we preload in loadup already has Lisp
syntax.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-28 22:18 ` John Wiegley
@ 2016-11-29 18:40 ` Eli Zaretskii
2016-11-29 19:11 ` John Wiegley
2016-11-29 19:12 ` Daniel Colascione
0 siblings, 2 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 18:40 UTC (permalink / raw)
To: John Wiegley; +Cc: eggert, dancol, emacs-devel
> From: John Wiegley <jwiegley@gmail.com>
> Cc: eggert@cs.ucla.edu, dancol@dancol.org, emacs-devel@gnu.org
> Date: Mon, 28 Nov 2016 14:18:58 -0800
>
> > Accepting this patch means we believe in this direction, consider it a
> > perspective one, enough so that we are willing to invest a non-trivial
> > amount of energy into making it happen, fix the fallout, debug the results,
> > etc. We are talking about at least one major release, maybe more, and
> > definitely several minor releases. IOW, we are talking several years. I
> > don't think we should gamble with such high stakes, not unless we sure we
> > want to go there, because this is the best alternative of [...]
>
> If I didn't know you were talking about pdumper vs. unexec here, I'd swear
> this above paragraph was expressing my own concerns about merging Tromey's
> concurrency branch.
Superficially, perhaps. But as soon as you look at the details, there
are two important differences.
First, the only piece of code in the concurrency patch that is not
entirely trivial is a 60-line function (30% of whose line count is
comments and ifdef's). That's all there is to understand there. The
rest is extremely simple, almost trivial: functions that directly map
into the corresponding system threads APIs, or 2-liners for creation,
checking, and otherwise simple management of the few new object types.
By contrast, the proposed patch, even in its initial version, weighs
in at 3200 lines in its core part, and almost none of it is trivial or
simple.
What's more, it comes with the need to make important and non-trivial
decisions wrt which objects and data structures need to be dumped, and
how, something that requires intimate knowledge of the internals and
their use patterns, both at dump time and in interactive sessions.
These decisions will need to be revisited when the related objects are
added or modified, which is the part that averts me the most, because
it means extending the Lisp interpreter, which is already not easy,
will have become even harder.
The other important difference is that threads are an opt-in feature:
if you don't start any threads, Emacs with threads still works as it
did before, and you can just ignore anything related to threads. IOW,
threads are a feature we offer to Lisp programs that want to use it.
By contrast, dumping code is a central and mandatory part of building
Emacs and of its startup process. It cannot be ignored or made
optional.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:23 ` Eli Zaretskii
@ 2016-11-29 18:49 ` Daniel Colascione
2016-11-29 19:02 ` Eli Zaretskii
2016-12-01 9:18 ` Richard Stallman
0 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 18:49 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: John Wiegley, burton.samograd, rms, emacs-devel
On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: rms@gnu.org, dancol@dancol.org, emacs-devel@gnu.org,
>> burton.samograd@autodesk.com
>> Date: Tue, 29 Nov 2016 10:03:01 -0800
>>
>> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>>
>> EZ> However, a dumper that we control by manually telling it what to dum and
>> EZ> what not doesn't have to do that. IMO, that's just something that doesn't
>> EZ> need to exist in the brave new unexec-free world. (Note that the "one huge
>> EZ> .elc file" method has this built-in.)
>>
>> I'd still like to see research into the "one huge .elc file" method, for the
>> obvious benefits you listed.
>
> Stefan and Ken made some good progress. IMO, we should actively
> continue pursuing that goal.
>
>> As far as I understand, the only difference between the portable
>> dumper, and one-huge-.elc, is efficiency. Given infinite resources,
>> the .elc method is much to be preferred, since everything we need
>> "falls out" from existing design.
>
> Any memory dumper will always be more efficient than loading .elc.
> The challenge is to make the difference smaller than the annoyance
> level of the users. Since initial testing already produces load times
> below 0.5 sec, I don't see why further optimizations won't deliver
> enough speedup to make it fast enough.
Because you can't possible get away, in the ELC approach, from:
1) reading every damn byte of the file, and
2) allocating memory for every object you read.
Efficiency is always going to be at least linear in the size of the
"elc dump" --- super-linear complexity if you want read-circle.
A memory dump is linear only in the amount of data you actually
access. Plus, there's no serialization or deserialization overhead. A
memory dump is also inherently gentler on the underlying system, since
we can use many of the pages unmodified (which makes them shareable and
trivially swappable), while every damn byte of the Emacs heap after
loading ELC files will be process-private dirty storage.
I do not think the portable dumper is something that's so complex as to
not be worth the performance, especially since any lread implementation
that's even close to being acceptably fast is going to be very complex
in its own right.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:49 ` Daniel Colascione
@ 2016-11-29 19:02 ` Eli Zaretskii
2016-12-01 9:18 ` Richard Stallman
1 sibling, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 19:02 UTC (permalink / raw)
To: Daniel Colascione; +Cc: jwiegley, burton.samograd, rms, emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Cc: John Wiegley <jwiegley@gmail.com>, rms@gnu.org, emacs-devel@gnu.org, burton.samograd@autodesk.com
> Date: Tue, 29 Nov 2016 10:49:10 -0800
>
> > Any memory dumper will always be more efficient than loading .elc.
> [...]
> Because you can't possible get away, in the ELC approach, from:
>
> 1) reading every damn byte of the file, and
> 2) allocating memory for every object you read.
>
> Efficiency is always going to be at least linear in the size of the
> "elc dump" --- super-linear complexity if you want read-circle.
>
> A memory dump is linear only in the amount of data you actually
> access. Plus, there's no serialization or deserialization overhead. A
> memory dump is also inherently gentler on the underlying system, since
> we can use many of the pages unmodified (which makes them shareable and
> trivially swappable), while every damn byte of the Emacs heap after
> loading ELC files will be process-private dirty storage.
Like I said: any memory dumper will always be faster. But having less
functionality coded in non-trivial C is IMO an important goal, one we
have been pursuing for quite some time, and for a good reason.
Although it should be clear that any adequate C implementation will
always be faster than an equivalent Lisp implementation.
> I do not think the portable dumper is something that's so complex as to
> not be worth the performance
It's complex. I have read the patch. I do know my way in the Emacs
internals.
> especially since any lread implementation that's even close to being
> acceptably fast is going to be very complex in its own right.
Even if we accept your opinion on the necessity for changes in lread
(and I don't think I agree), that is a one-off job.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:39 ` Eli Zaretskii
@ 2016-11-29 19:03 ` Daniel Colascione
2016-11-29 19:59 ` Eli Zaretskii
2016-11-29 19:13 ` Paul Eggert
2016-12-01 9:18 ` Richard Stallman
2 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 19:03 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: rms, emacs-devel
On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: Richard Stallman <rms@gnu.org>
>> CC: dancol@dancol.org, emacs-devel@gnu.org
>> Date: Tue, 29 Nov 2016 11:55:16 -0500
>>
>> Maintaining the portable dumper will be far simpler than maintaining
>> unexec. Unexec has to write a file in whatever format the system uses
>> for executables. If that is ELF format, it is very complicated,
>> and it wasn't designed for this purpose. It is necessary to copy
>> the material from the temacs executable, and modify it along the way.
>>
>> All those problems will be absent for the portable dumper, which
>> will make its maintenance much easier.
>
> Easier: yes. Easy enough: no, not IMO. Not for most of the Emacs
> contributors we have and will have in the near future.
Who do you imagine maintaining Emacs? CS 101 students? We have to assume
a certain baseline level of skill on the part of our future
colleagues. I don't want to dumb down our implementation for the sake of
people we don't know and who, for all we know, may never
actually appear.
> The number of people aboard who can matter-of-factly hack the Emacs
> internals on the C level is consistently going down, and is already so
> small they can be counted on one hand. We must make Emacs depend less
> on people from this small and diminishing group, if we want the
> development pace increased or even kept at its current level.
I've discussed before why I think this argument is ill-founded. We
already have examples over the past few years of people who've
familiarized themselves with Emacs internals in order to add a feature
they want. If there's a need, people will familiarize themselves with
the editor.
Please, present some evidence that the reason we've seen a decline in
core contributions due to the complexity of the current codebase. I
don't agree that we'd see more contributors appear if we simplified our
algorithms. On the contrary, making Emacs slower will drive users away.
The solution to the problem you're raising is better internals
documentation, not oversimplification.
> To me,
> that means keep as many features out of low-level C, and limit futzing
> with C-level internals of Lisp objects and the Lisp interpreter to the
> absolute minimum. IOW, any feature that can have an alternative
> implementation in Lisp should not be done in C, even if the Lisp
> implementation will be slightly slower.
I agree that what *can* be done in Lisp should be done in Lisp. We're
talking about a facility that doesn't fall under this general
guideline. It's the job of the C core to provide to Lisp facilities
necessary to implement higher-level features. Fast startup is one of
these platform facilities, one very common in other Lisp systems.
>> On top of that, adding Lisp objects will now require
>> > writing its dumper back-end, so this will be a constant maintenance
>> > burden of the kind that only a few of us can bear.
>>
>> Yes, but how often do we make such changes? Once every few years,
>> I think.
>
> More like once a year, maybe more.
Like what? The last two new types in core have been finalizers and the
module thing, both of which have been trivial Lisp_Misc types.
By the way: it's very easy to add support for new object types
to pdumper. Have you read the code?
There are certain types of object that are especially easy to support:
Do you have a pseudovector with no non-Lisp allocations? Adding support
to pdumper for this object type is one line of code.
Do you have a pseudovector that you want to come back from dumps "dead",
like frames? Adding support to pdumper for this object type is one line
of code.
How many new objects types are we going to add that don't fall under one
of these two categories? You're optimizing for hypotheticals at the
expense of the concrete present.
>
>> And it won't be a big difficulty to do so, if the code of the dumper
>> is clean.
>
> For you and me, and for a few others, sure. But Emacs cannot be
> developed by a handful of people anymore. We already fall behind with
> frequency of releases, with the speed of delivering new features, with
> analyzing and fixing bugs reported to the tracker, etc. If we don't
> attract more core developers, we will never get out of this vicious
> cycle. And bringing C-level core developers on board is a very slow,
> painful, and unreliable process, because C programmers are a dying
> breed, and Emacs internals are not for the faint at heart.
So move more functionality to Lisp. We've done a good job of doing that
lately. We're talking about a special case. You can't move eval to
Lisp either.
If you want to pursue this path, how about moving redisplay to Lisp? You
know more about redisplay than anyone else around, I think, and
redisplay is a much hairier blob of C code than the portable dumper is.
>> I am sure it will be a lot cleaner than the code of unexec.
>
> I'm sure, too, but that's not the issue.
>
>> > Making the initial load of preloaded Lisp files (most probably, a
>> > single large file) fast enough to allow us to dump the dumping phase
>> > altogether is a much more perspective direction.
>>
>> I think this has its own difficulty -- the need to represent everything
>> in Lisp syntax.
>
> I don't think I follow: what we preload in loadup already has Lisp
> syntax.
Yes, but the data structures that we create via loadup don't all have
easy-to-digest Lisp representations, and creating them is on the same
level of complexity as the portable dumper.
*This* code exists. Where is the hypothetical sufficiently-fast lread?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:40 ` Eli Zaretskii
@ 2016-11-29 19:11 ` John Wiegley
2016-11-29 20:07 ` Eli Zaretskii
2016-11-29 19:12 ` Daniel Colascione
1 sibling, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-11-29 19:11 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: eggert, dancol, emacs-devel
>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> By contrast, dumping code is a central and mandatory part of building Emacs
> and of its startup process. It cannot be ignored or made optional.
Could we ask Daniel to include his code under an #ifdef, so until it's proven
we can always disable it, or back out the changes easily if need be? I'd like
to entertain the experiment, if we can.
I spent a good chunk of my career working on linkers and loaders, so I'm aware
of some of the complexities we're inviting in. My biggest fear is that things
like this always seem simple initially -- but the devil is in the details.
However, if we maintain a way to trivially back out, I don't see how it can
hurt to give Daniel his room to play. Maybe it could turn out better than we
think?
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:40 ` Eli Zaretskii
2016-11-29 19:11 ` John Wiegley
@ 2016-11-29 19:12 ` Daniel Colascione
1 sibling, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 19:12 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: John Wiegley, eggert, emacs-devel
On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: eggert@cs.ucla.edu, dancol@dancol.org, emacs-devel@gnu.org
>> Date: Mon, 28 Nov 2016 14:18:58 -0800
>>
>> > Accepting this patch means we believe in this direction, consider it a
>> > perspective one, enough so that we are willing to invest a non-trivial
>> > amount of energy into making it happen, fix the fallout, debug the results,
>> > etc. We are talking about at least one major release, maybe more, and
>> > definitely several minor releases. IOW, we are talking several years. I
>> > don't think we should gamble with such high stakes, not unless we sure we
>> > want to go there, because this is the best alternative of [...]
>>
>> If I didn't know you were talking about pdumper vs. unexec here, I'd swear
>> this above paragraph was expressing my own concerns about merging Tromey's
>> concurrency branch.
>
> Superficially, perhaps. But as soon as you look at the details, there
> are two important differences.
>
> First, the only piece of code in the concurrency patch that is not
> entirely trivial is a 60-line function (30% of whose line count is
> comments and ifdef's).
You're the only person I've ever seen use words like "entirely trivial"
to describe the addition of shared-memory concurrency to a legacy
single-threaded system. Even if the code for the threads system looks
simple, the concepts involved are anything but.
By contrast, the portable dumper is about marshalling, which I think is
comparatively simpler.
> That's all there is to understand there. The
> rest is extremely simple, almost trivial: functions that directly map
> into the corresponding system threads APIs, or 2-liners for creation,
> checking, and otherwise simple management of the few new object types.
> By contrast, the proposed patch, even in its initial version, weighs
> in at 3200 lines in its core part, and almost none of it is trivial or
> simple.
>
> What's more, it comes with the need to make important and non-trivial
> decisions wrt which objects and data structures need to be dumped, and
> how, something that requires intimate knowledge of the internals and
> their use patterns, both at dump time and in interactive sessions.
> These decisions will need to be revisited when the related objects are
> added or modified, which is the part that averts me the most, because
> it means extending the Lisp interpreter, which is already not easy,
> will have become even harder.
I don't see it as a problem that the internals of Emacs need to have
knowledge of the internals of Emacs. What matters is whether adding new
object types, which is already a very rare exercise, is made
prohibitively hard. Keep in mind that any programmer who adds a new
object type needs to *also* update the garbage collector, and GC and the
portable dumper require exactly the same kind of object description.
(I looked into actually unifying the GC and pdumper, but I don't think
it's possible without introducing significant new abstractions. It's not
worth it for something that happens so rarely as modification to core
Emacs data type representations.)
If you want to follow this reasoning to its logical conclusion, we
should switch to Boehm GC.
> The other important difference is that threads are an opt-in feature:
> if you don't start any threads, Emacs with threads still works as it
> did before, and you can just ignore anything related to threads. IOW,
> threads are a feature we offer to Lisp programs that want to use it.
> By contrast, dumping code is a central and mandatory part of building
> Emacs and of its startup process. It cannot be ignored or made
> optional.
The portable dumper is also optional: you can always use the legacy
unexec code, which isn't going away, or just rely on loadup (and the
hypothetical ultra-fast lread we've discussed).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:39 ` Eli Zaretskii
2016-11-29 19:03 ` Daniel Colascione
@ 2016-11-29 19:13 ` Paul Eggert
2016-11-29 19:35 ` Eli Zaretskii
2016-12-01 9:18 ` Richard Stallman
2 siblings, 1 reply; 299+ messages in thread
From: Paul Eggert @ 2016-11-29 19:13 UTC (permalink / raw)
To: Eli Zaretskii, rms; +Cc: dancol, emacs-devel
On 11/29/2016 10:39 AM, Eli Zaretskii wrote:
> If we don't attract more core developers, we will never get out of this vicious cycle.
Although we certainly could use more core developers, it's quite a
stretch to say that switching from unexec to a portable undumper will
scare recruits away. On the contrary, the portable approach is simpler
and easier for non-experts than what we have now.
> C programmers are a dying breed
I don't agree, as C is an essential component of GNU/Linux and is
robustly represented in the practical and research programming
communities. (For what it's worth, C's popularity is ranked #7 in PyPL
and #2 in TIOBE.) Although it'd be fine to replacing the core Emacs
interpreter with something better if we can scare up developer resources
to do that, in the meantime it's OK to assume C expertise for people
modifying the Emacs interpreter.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 19:13 ` Paul Eggert
@ 2016-11-29 19:35 ` Eli Zaretskii
2016-11-29 20:54 ` Paul Eggert
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 19:35 UTC (permalink / raw)
To: Paul Eggert; +Cc: dancol, rms, emacs-devel
> Cc: dancol@dancol.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 29 Nov 2016 11:13:37 -0800
>
> On 11/29/2016 10:39 AM, Eli Zaretskii wrote:
> > If we don't attract more core developers, we will never get out of this vicious cycle.
>
> Although we certainly could use more core developers, it's quite a
> stretch to say that switching from unexec to a portable undumper will
> scare recruits away.
Which is why I didn't say that.
> On the contrary, the portable approach is simpler and easier for
> non-experts than what we have now.
I didn't compare the proposed portable dumper with what we have now,
so this is a red herring.
> > C programmers are a dying breed
> I don't agree, as C is an essential component of GNU/Linux and is
> robustly represented in the practical and research programming
> communities. (For what it's worth, C's popularity is ranked #7 in PyPL
> and #2 in TIOBE.)
When did you last try to hire a C or C++ programmer?
As another data point, just look in the logs at the names of people
who make C-level changes, and count them.
I didn't say I like this, but leading a project to be viable in the
future needs to look at the reality, not at our wishful thinking.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 17:32 ` Daniel Colascione
@ 2016-11-29 19:55 ` Philippe Vaucher
0 siblings, 0 replies; 299+ messages in thread
From: Philippe Vaucher @ 2016-11-29 19:55 UTC (permalink / raw)
To: Daniel Colascione
Cc: Eli Zaretskii, burton.samograd, Richard Stallman,
Emacs developers
[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]
>
> > Can that be fixed by creating new equivalent windows and frames?
>
> Yes: you'd want to reuse desktop's code to save window and frame
> configuration. I plan to provide a both-dump-hook and after-dump-hook
> mechanism that Lisp code can use for tasks like this.
>
This comment remembered me about a proposal I was waiting to be polished
about the lisp objects representation, but since it's related it kinda
forces me to talk about it earlier :-)
I wrote an utility that lets you see your keyboards macros as emacs lisp (
https://github.com/Silex/elmacro).
In this utility, I have to deal with all kind of lisp objects that needed
to be serialized, and I found myself writing utilities that dumped windows,
frames & buffers differently than was "print" does.
Basically, instead of the traditional "#<window 3 on bla.el>" that does not
eval well, I dump it as "(elmacro-get-window 3)". This function just loops
over "(window-list)" and returns the correct window, that way I can eval
that code and it works (of course it does not work of the window 3 gets
destroyed before the code evaluation, but that's ok). A good example are
mouse clicks: https://github.com/Silex/elmacro#mouse-events-support
During the implementation I often thought "wouldn't it be nice if 'print'
(and prin1/prin1-to-string/etc) dumped ALL lisp objects as something
evalable?".
So, here is my proposal. Make all emacs lisp objects always have an "read"
friendly text representation (or at least as much as possible).
Maybe by doing so it also simplifies the dumper.
Kind regards,
Philippe
[-- Attachment #2: Type: text/html, Size: 2240 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 19:03 ` Daniel Colascione
@ 2016-11-29 19:59 ` Eli Zaretskii
2016-11-29 20:28 ` John Wiegley
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 19:59 UTC (permalink / raw)
To: Daniel Colascione; +Cc: rms, emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Date: Tue, 29 Nov 2016 11:03:35 -0800
>
> > Easier: yes. Easy enough: no, not IMO. Not for most of the Emacs
> > contributors we have and will have in the near future.
>
> Who do you imagine maintaining Emacs? CS 101 students?
Just look around, they are all here.
> We have to assume a certain baseline level of skill on the part of
> our future colleagues. I don't want to dumb down our implementation
> for the sake of people we don't know and who, for all we know, may
> never actually appear.
You are insulting most of the people on this list, some of them
long-time and fruitful contributors. They don't deserve that.
> > The number of people aboard who can matter-of-factly hack the Emacs
> > internals on the C level is consistently going down, and is already so
> > small they can be counted on one hand. We must make Emacs depend less
> > on people from this small and diminishing group, if we want the
> > development pace increased or even kept at its current level.
>
> I've discussed before why I think this argument is ill-founded. We
> already have examples over the past few years of people who've
> familiarized themselves with Emacs internals in order to add a feature
> they want. If there's a need, people will familiarize themselves with
> the editor.
I don't think this is accurate, but even if it is, it is better not to
have this bar.
> Please, present some evidence that the reason we've seen a decline in
> core contributions due to the complexity of the current codebase. I
> don't agree that we'd see more contributors appear if we simplified our
> algorithms.
You are missing the point: not complexity and algorithms are the
problem, the implementation language is the problem.
> I agree that what *can* be done in Lisp should be done in Lisp. We're
> talking about a facility that doesn't fall under this general
> guideline.
??? loadup.el is Lisp code.
> >> On top of that, adding Lisp objects will now require
> >> > writing its dumper back-end, so this will be a constant maintenance
> >> > burden of the kind that only a few of us can bear.
> >>
> >> Yes, but how often do we make such changes? Once every few years,
> >> I think.
> >
> > More like once a year, maybe more.
>
> Like what? The last two new types in core have been finalizers and the
> module thing, both of which have been trivial Lisp_Misc types.
Trivial to some, much less trivial to others.
> By the way: it's very easy to add support for new object types
> to pdumper. Have you read the code?
Yes, I have read the code. It's easy for you to add new object types,
because you wrote that code, debugged it, and didn't yet have time to
forget how it works.
It is also easy for me to tweak bidi.c to add features required by
Unicode 6.3 and 8.0. Does that mean anyone else can "easily" do that?
> If you want to pursue this path, how about moving redisplay to Lisp? You
> know more about redisplay than anyone else around, I think, and
> redisplay is a much hairier blob of C code than the portable dumper is.
If you stop arguing with me so much about each and every issue, then
perhaps I will have time to explore this. As things are, sorry:
arguing with you takes up all of my free time -- or what's left of it
after triaging and fixing bugs, helping people to implement stuff in
C, something you think is "easy" for everyone past CS101, adding and
fixing documentation, including that one Daniel Colascione failed to
write, and whatnot else that is evidently part of the job.
> >> I think this has its own difficulty -- the need to represent everything
> >> in Lisp syntax.
> >
> > I don't think I follow: what we preload in loadup already has Lisp
> > syntax.
>
> Yes, but the data structures that we create via loadup don't all have
> easy-to-digest Lisp representations, and creating them is on the same
> level of complexity as the portable dumper.
But it already works.
> *This* code exists. Where is the hypothetical sufficiently-fast lread?
In the works, I hope. It is my job as a maintainer to encourage that
development.
> > First, the only piece of code in the concurrency patch that is not
> > entirely trivial is a 60-line function (30% of whose line count is
> > comments and ifdef's).
>
> You're the only person I've ever seen use words like "entirely trivial"
> to describe the addition of shared-memory concurrency to a legacy
> single-threaded system. Even if the code for the threads system looks
> simple, the concepts involved are anything but.
Concurrency in Emacs is for people who already know what that is and
are familiar with the concepts. _After_ you are familiar with the
concepts, the code is surprisingly simple and straightforward.
> By contrast, the portable dumper is about marshalling, which I think is
> comparatively simpler.
The concepts are simpler, but the implementation is not so simple.
And some stuff is done manually, with the PDUMPER_* macros rules for
whose use are only clear to someone who knows a lot about the involved
objects and their use patterns.
> The portable dumper is also optional: you can always use the legacy
> unexec code, which isn't going away, or just rely on loadup (and the
> hypothetical ultra-fast lread we've discussed).
If that is the direction we want to move, then the efforts invested in
the development of the portable dumper are a complete waste of our
resources.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 19:11 ` John Wiegley
@ 2016-11-29 20:07 ` Eli Zaretskii
2016-11-29 20:29 ` John Wiegley
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-29 20:07 UTC (permalink / raw)
To: John Wiegley; +Cc: eggert, dancol, emacs-devel
> From: John Wiegley <jwiegley@gmail.com>
> Cc: eggert@cs.ucla.edu, dancol@dancol.org, emacs-devel@gnu.org
> Date: Tue, 29 Nov 2016 11:11:15 -0800
>
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
> > By contrast, dumping code is a central and mandatory part of building Emacs
> > and of its startup process. It cannot be ignored or made optional.
>
> Could we ask Daniel to include his code under an #ifdef, so until it's proven
> we can always disable it, or back out the changes easily if need be?
What would that solve? Don't we still need to review the patches,
compile the code once in a while, debug it as needed, etc.?
> I'd like to entertain the experiment, if we can.
Fine, but why do you need me for that? What would be my role in that
experiment?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 19:59 ` Eli Zaretskii
@ 2016-11-29 20:28 ` John Wiegley
0 siblings, 0 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-29 20:28 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Daniel Colascione, rms, emacs-devel
>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>> We have to assume a certain baseline level of skill on the part of our
>> future colleagues. I don't want to dumb down our implementation for the
>> sake of people we don't know and who, for all we know, may never actually
>> appear.
EZ> You are insulting most of the people on this list, some of them long-time
EZ> and fruitful contributors. They don't deserve that.
Eli, I don't read any insult here, just Daniel questioning the fear that we'd
code ourselves into oblivion if core allowed any more complexity. I think we
all want more Lisp and less C, I've seen you both say as much. The real point
being debated is whether some additional C wouldn't help us in the interim.
Note that I've seen both statements become true: Complex core that repulsed
new contributors, and new contributors willing to learn anything to help a
project.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 20:07 ` Eli Zaretskii
@ 2016-11-29 20:29 ` John Wiegley
2016-11-29 20:36 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-11-29 20:29 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: eggert, dancol, emacs-devel
>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>> Could we ask Daniel to include his code under an #ifdef, so until it's proven
>> we can always disable it, or back out the changes easily if need be?
> What would that solve? Don't we still need to review the patches, compile
> the code once in a while, debug it as needed, etc.?
It keeps the portable dumper fenced off. We'd compile it in by default, but it
helps future readers, for a while, by clearly demarcating when special
handling is needed for the sake of that dumper.
>> I'd like to entertain the experiment, if we can.
> Fine, but why do you need me for that? What would be my role in that
> experiment?
I guess just say OK at this point, and help with the review if you're
inclined? Daniel will handle the work of integration and documentation.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 20:29 ` John Wiegley
@ 2016-11-29 20:36 ` Daniel Colascione
2016-11-29 21:30 ` John Wiegley
2016-11-30 8:26 ` Philippe Vaucher
0 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 20:36 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: eggert, emacs-devel
On Tue, Nov 29 2016, John Wiegley wrote:
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Could we ask Daniel to include his code under an #ifdef, so until it's proven
>>> we can always disable it, or back out the changes easily if need be?
>
>> What would that solve? Don't we still need to review the patches, compile
>> the code once in a while, debug it as needed, etc.?
>
> It keeps the portable dumper fenced off. We'd compile it in by default, but it
> helps future readers, for a while, by clearly demarcating when special
> handling is needed for the sake of that dumper.
FWIW, as a logistical matter, it's not really practical to literally
#ifdef out the pdumper integration into the rest of Emacs. We can,
however, make the pdumper constructs compile down to nothing if you opt
not to enable pdumper in the build configuration. I specifically
designed the pdumper.h API to make this arrangement easy and natural.
As a matter of general taste, I _much_ prefer this style:
#ifdef HAVE_FOO
void foo_thing (void);
#else
# define foo_thing() ((void)0)
#endif
void
function1()
{
...
foo_thing();
...
}
void function2()
{
...
foo_thing();
...
}
void function3()
{
...
foo_thing();
...
}
to this style:
void
function1()
{
...
#ifdef HAVE_FOO
foo_thing();
#endif
...
}
void function2()
{
...
#ifdef HAVE_FOO
foo_thing();
#endif
...
}
void function3()
{
...
#ifdef HAVE_FOO
foo_thing();
#endif
...
}
>
>>> I'd like to entertain the experiment, if we can.
>
>> Fine, but why do you need me for that? What would be my role in that
>> experiment?
>
> I guess just say OK at this point, and help with the review if you're
> inclined? Daniel will handle the work of integration and documentation.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 19:35 ` Eli Zaretskii
@ 2016-11-29 20:54 ` Paul Eggert
2016-11-30 16:38 ` Eli Zaretskii
0 siblings, 1 reply; 299+ messages in thread
From: Paul Eggert @ 2016-11-29 20:54 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: dancol, rms, emacs-devel
On 11/29/2016 11:35 AM, Eli Zaretskii wrote:
> I didn't compare the proposed portable dumper with what we have now,
That's our fundamental disagreement, then. I am willing to settle for
real improvements to Emacs even if they are not ideal. And I think that
this sort of willingness is more likely to attract contributions from
developers.
> leading a project to be viable in the
> future needs to look at the reality, not at our wishful thinking
Daniel's proposed change consists of code that works and that runs fast.
Isn't that reality? And in contrast, isn't the lread-based approach
wishful thinking?
> When did you last try to hire a C or C++ programmer?
I don't *hire* them, I *create* them. I currently have 450 undergraduate
students, and C++ and C are the only programming languages that they
have all written and debugged code in. (This is not just my decision; it
comes from our faculty's curriculum.) According to Philip Guo's 2014
survey, the most popular languages used in teaching computer science in
top U.S. universities are Python (25-30%), Java (20-25%), C/C++
(10-15%), and MATLAB (5-10%). So from what I can see, the C pipeline is
by no means empty.
Anyway, although C has probably passed its peak and will surely die out
eventually, it's unlikely for this to be anytime soon. The prospect of
its demise shouldn't materially affect our choice of Emacs dumping
technology in the meantime.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 13:06 ` Stefan Monnier
@ 2016-11-29 21:19 ` Daniel Colascione
2016-11-29 21:35 ` Paul Eggert
0 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 21:19 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
On Tue, Nov 29 2016, Stefan Monnier wrote:
>> run. With a non-PIC executable, you can do a pretty good job of that
>> and get performance a bit better than unexec, in theory.
>
> Got me curious: how/why could it be even better than unexec?
With unexec, we're stuck with whatever heap layout malloc gives us, and
whenever we modify a byte within this heap image, the operating system
has to make a copy of that byte's host page and map that copy back into
our address space. (That is, we take a copy-on-write fault.) Pure
allocation partially, but not completely, addresses this overhead.
With the portable dumper, *we* choose the layout and we can do things
like put strings and other unlikely-to-change bits in part of the file
that probably won't get COW faults. We also avoid the need to store
malloc data structures. At runtime, we store the mark bits for objects
in the pdumper image in a contiguous dedicated memory area instead of
mutating the image, further reducing the number of COW faults we need to
incur. With pdumper and a non-PIC Emacs, you'll only pay for COW faults
on the lisp objects you actually modify.
Even *with* a PIC Emacs (which I hope is the default, because ASLR
greatly improves security), we can get these savings if
mmap(BASE_ADDRESS_STORED_IN_DUMP, ...) succeeds and we can map the dump
where we want. If we can't map the dump where we want, we'll just
relocate it.
Alternatively, if we're not memory-mapping, we can compress the dump
file --- that's much harder to do with unexec.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 20:36 ` Daniel Colascione
@ 2016-11-29 21:30 ` John Wiegley
2016-11-30 8:26 ` Philippe Vaucher
1 sibling, 0 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-29 21:30 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, eggert, emacs-devel
>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
DC> As a matter of general taste, I _much_ prefer this style:
Yes, I agree.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 21:19 ` Daniel Colascione
@ 2016-11-29 21:35 ` Paul Eggert
2016-11-29 21:50 ` Daniel Colascione
` (2 more replies)
0 siblings, 3 replies; 299+ messages in thread
From: Paul Eggert @ 2016-11-29 21:35 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
On 11/29/2016 01:19 PM, Daniel Colascione wrote:
> Even*with* a PIC Emacs (which I hope is the default, because ASLR
> greatly improves security), we can get these savings if
> mmap(BASE_ADDRESS_STORED_IN_DUMP, ...) succeeds and we can map the dump
> where we want. If we can't map the dump where we want, we'll just
> relocate it.
Although your other advantages sound good, this one sounds worrisome. If
Emacs maps the dump to the same place every time, we are giving up ASLR
for the dump itself, and won't that pose a potential security risk? If
so, perhaps it would be better to not mmap to the base address stored in
the dump (unless we can determine that ASLR is not in use, I suppose).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 21:35 ` Paul Eggert
@ 2016-11-29 21:50 ` Daniel Colascione
2016-11-29 22:01 ` Paul Eggert
2016-11-29 22:01 ` Stefan Monnier
2016-11-29 22:22 ` Philipp Stephani
2 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-29 21:50 UTC (permalink / raw)
To: Paul Eggert; +Cc: emacs-devel
On Tue, Nov 29 2016, Paul Eggert wrote:
> On 11/29/2016 01:19 PM, Daniel Colascione wrote:
>> Even*with* a PIC Emacs (which I hope is the default, because ASLR
>> greatly improves security), we can get these savings if
>> mmap(BASE_ADDRESS_STORED_IN_DUMP, ...) succeeds and we can map the dump
>> where we want. If we can't map the dump where we want, we'll just
>> relocate it.
>
> Although your other advantages sound good, this one sounds
> worrisome. If Emacs maps the dump to the same place every time, we are
> giving up ASLR for the dump itself, and won't that pose a potential
> security risk? If so, perhaps it would be better to not mmap to the
> base address stored in the dump (unless we can determine that ASLR is
> not in use, I suppose).
The Emacs dump isn't PROT_EXEC though. It's pure* data, and this
difference diminishes the risk considerably, I think. I'd at least like
to make this behavior a user option.
* We do store function pointers in the dump, and an attacker could
theoretically overwrite one of these to point where she wanted --- but
with all PROT_EXEC code in the process being randomized, where would
she point the function pointer that's under her control?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 21:35 ` Paul Eggert
2016-11-29 21:50 ` Daniel Colascione
@ 2016-11-29 22:01 ` Stefan Monnier
2016-11-29 22:22 ` Philipp Stephani
2 siblings, 0 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-11-29 22:01 UTC (permalink / raw)
To: emacs-devel
> Although your other advantages sound good, this one sounds worrisome. If
> Emacs maps the dump to the same place every time, we are giving up ASLR for
> the dump itself, and won't that pose a potential security risk? If so,
> perhaps it would be better to not mmap to the base address stored in the
> dump (unless we can determine that ASLR is not in use, I suppose).
I live very happily without ASLR for Emacs right now. So I'm not
worried about limiting ASLR to a subset of Emacs's heap.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 21:50 ` Daniel Colascione
@ 2016-11-29 22:01 ` Paul Eggert
2016-11-30 0:37 ` Daniel Colascione
2016-11-30 20:07 ` Richard Stallman
0 siblings, 2 replies; 299+ messages in thread
From: Paul Eggert @ 2016-11-29 22:01 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
On 11/29/2016 01:50 PM, Daniel Colascione wrote:
> * We do store function pointers in the dump, and an attacker could
> theoretically overwrite one of these to point where she wanted --- but
> with all PROT_EXEC code in the process being randomized, where would
> she point the function pointer that's under her control?
I'm more worried about the next level up. Although the dump is pure data
to the machine, it's not pure data to Elisp. Since the dump would
contain bytecodes, if attackers can alter the bytecodes then they can
execute whatever Elisp code they want.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 21:35 ` Paul Eggert
2016-11-29 21:50 ` Daniel Colascione
2016-11-29 22:01 ` Stefan Monnier
@ 2016-11-29 22:22 ` Philipp Stephani
2016-11-29 22:34 ` Paul Eggert
2 siblings, 1 reply; 299+ messages in thread
From: Philipp Stephani @ 2016-11-29 22:22 UTC (permalink / raw)
To: Paul Eggert, Daniel Colascione; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 707 bytes --]
Paul Eggert <eggert@cs.ucla.edu> schrieb am Di., 29. Nov. 2016 um 22:35 Uhr:
> On 11/29/2016 01:19 PM, Daniel Colascione wrote:
> > Even*with* a PIC Emacs (which I hope is the default, because ASLR
> > greatly improves security), we can get these savings if
> > mmap(BASE_ADDRESS_STORED_IN_DUMP, ...) succeeds and we can map the dump
> > where we want. If we can't map the dump where we want, we'll just
> > relocate it.
>
> Although your other advantages sound good, this one sounds worrisome. If
> Emacs maps the dump to the same place every time, we are giving up ASLR
> for the dump itself
Is that actually the case? Looking at the code, I see nothing that would
prevent dumping with ASLR enabled.
[-- Attachment #2: Type: text/html, Size: 1191 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 22:22 ` Philipp Stephani
@ 2016-11-29 22:34 ` Paul Eggert
0 siblings, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2016-11-29 22:34 UTC (permalink / raw)
To: Philipp Stephani; +Cc: emacs-devel
On 11/29/2016 02:22 PM, Philipp Stephani wrote:
> I see nothing that would prevent dumping with ASLR enabled.
Yes, that's correct. It's a performance thing, not a correctness thing.
If Emacs undumps into the same address that it dumped from then it can
avoid relocating some pointers, which should save some work and avoid
making some copies (assuming copy-on-write).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 22:01 ` Paul Eggert
@ 2016-11-30 0:37 ` Daniel Colascione
2016-11-30 7:35 ` Paul Eggert
2016-11-30 20:07 ` Richard Stallman
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-30 0:37 UTC (permalink / raw)
To: Paul Eggert; +Cc: emacs-devel
On Tue, Nov 29 2016, Paul Eggert wrote:
> On 11/29/2016 01:50 PM, Daniel Colascione wrote:
>> * We do store function pointers in the dump, and an attacker could
>> theoretically overwrite one of these to point where she wanted --- but
>> with all PROT_EXEC code in the process being randomized, where would
>> she point the function pointer that's under her control?
>
> I'm more worried about the next level up. Although the dump is pure
> data to the machine, it's not pure data to Elisp. Since the dump would
> contain bytecodes, if attackers can alter the bytecodes then they can
> execute whatever Elisp code they want.
Good point. How about this? We'll let the PIC-ness of the Emacs
executable (which we know at build time) control whether we try to map
the dump at its preferred load address. If a user is running non-PIC,
he's already vulnerable.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 0:37 ` Daniel Colascione
@ 2016-11-30 7:35 ` Paul Eggert
2016-11-30 13:33 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Paul Eggert @ 2016-11-30 7:35 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
Daniel Colascione wrote:
> How about this? We'll let the PIC-ness of the Emacs
> executable (which we know at build time) control whether we try to map
> the dump at its preferred load address. If a user is running non-PIC,
> he's already vulnerable.
Yes, that sounds fine, thanks.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 20:36 ` Daniel Colascione
2016-11-29 21:30 ` John Wiegley
@ 2016-11-30 8:26 ` Philippe Vaucher
1 sibling, 0 replies; 299+ messages in thread
From: Philippe Vaucher @ 2016-11-30 8:26 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, Paul Eggert, Emacs developers
[-- Attachment #1: Type: text/plain, Size: 724 bytes --]
>
> As a matter of general taste, I _much_ prefer this style:
>
> #ifdef HAVE_FOO
> void foo_thing (void);
> #else
> # define foo_thing() ((void)0)
> #endif
>
> void
> function1()
> {
> ...
> foo_thing();
> ...
> }
>
> to this style:
>
> void
> function1()
> {
> ...
> #ifdef HAVE_FOO
> foo_thing();
> #endif
> ...
> }
>
> void function2()
> {
> ...
> #ifdef HAVE_FOO
> foo_thing();
> #endif
> ...
> }
>
+1
ifdefs should be as much out of the way of the code as possible in my
honest opinion.
Philippe
[-- Attachment #2: Type: text/html, Size: 1226 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 7:35 ` Paul Eggert
@ 2016-11-30 13:33 ` Stefan Monnier
0 siblings, 0 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-11-30 13:33 UTC (permalink / raw)
To: emacs-devel
>> How about this? We'll let the PIC-ness of the Emacs
>> executable (which we know at build time) control whether we try to map
>> the dump at its preferred load address. If a user is running non-PIC,
>> he's already vulnerable.
> Yes, that sounds fine, thanks.
I for one would prefer "measurably faster" over "hypothetically ever so
slightly less insecure".
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 20:54 ` Paul Eggert
@ 2016-11-30 16:38 ` Eli Zaretskii
2016-11-30 18:57 ` John Wiegley
2016-11-30 21:06 ` Paul Eggert
0 siblings, 2 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-11-30 16:38 UTC (permalink / raw)
To: Paul Eggert; +Cc: dancol, rms, emacs-devel
> Cc: rms@gnu.org, dancol@dancol.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 29 Nov 2016 12:54:16 -0800
>
> Daniel's proposed change consists of code that works and that runs fast.
I don't think "works and is fast" are the only criteria for accepting
patches, or even the most important ones.
> Isn't that reality? And in contrast, isn't the lread-based approach
> wishful thinking?
Maybe, maybe not. I don't understand why we need to grasp the first
straw we see around to get rid of unexec. What's the rush?
And let me remind you that you didn't like this same idea very much
when I proposed it 2 months ago, see the discussion that starts at
http://lists.gnu.org/archive/html/bug-gnu-emacs/2016-09/msg00227.html
> from what I can see, the C pipeline is by no means empty.
Well, it somehow becomes all but empty on the other end. Believe me.
And even if we cannot agree on that, the facts of what we see here are
very clear.
> The prospect of [C]s demise shouldn't materially affect our choice
> of Emacs dumping technology in the meantime.
I think it would be a grave mistake not to be affected by that. We
would be burying our heads in the sand if we don't.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 16:38 ` Eli Zaretskii
@ 2016-11-30 18:57 ` John Wiegley
2016-11-30 19:14 ` Daniel Colascione
2016-11-30 19:29 ` Philippe Vaucher
2016-11-30 21:06 ` Paul Eggert
1 sibling, 2 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-30 18:57 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Paul Eggert, dancol, rms, emacs-devel
>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
EZ> Maybe, maybe not. I don't understand why we need to grasp the first straw
EZ> we see around to get rid of unexec. What's the rush?
Eli has a good point here, there really is no rush. Daniel, would you be
willing to continue to incubate these changes in a branch, while we also
consider and explore alternatives?
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 18:57 ` John Wiegley
@ 2016-11-30 19:14 ` Daniel Colascione
2016-11-30 21:03 ` John Wiegley
2016-11-30 19:29 ` Philippe Vaucher
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-30 19:14 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Paul Eggert, rms, emacs-devel
On Wed, Nov 30 2016, John Wiegley wrote:
>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
> EZ> Maybe, maybe not. I don't understand why we need to grasp the first straw
> EZ> we see around to get rid of unexec. What's the rush?
>
> Eli has a good point here, there really is no rush. Daniel, would you be
> willing to continue to incubate these changes in a branch, while we also
> consider and explore alternatives?
No. This code isn't disruptive to the Emacs core if it's used, so
there's no reason to keep it out of master. Experience with the
concurrency branch tells me that interesting work in some feature branch
ghetto never actually gets merged.
I maintain that Eli's concerns about maintainability are unfounded.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 18:57 ` John Wiegley
2016-11-30 19:14 ` Daniel Colascione
@ 2016-11-30 19:29 ` Philippe Vaucher
2016-11-30 19:45 ` Daniel Colascione
1 sibling, 1 reply; 299+ messages in thread
From: Philippe Vaucher @ 2016-11-30 19:29 UTC (permalink / raw)
To: Eli Zaretskii, Paul Eggert, Daniel Colascione, Richard Stallman,
Emacs developers
[-- Attachment #1: Type: text/plain, Size: 855 bytes --]
>
> EZ> Maybe, maybe not. I don't understand why we need to grasp the first
> straw
> EZ> we see around to get rid of unexec. What's the rush?
>
> Eli has a good point here, there really is no rush. Daniel, would you be
> willing to continue to incubate these changes in a branch, while we also
> consider and explore alternatives?
Isn't unexec what prevents emacs from building inside docker containers
(because of the personality syscall)?
As far as I'm concerned, that's a quite annoying emacs limitation. See bug
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23529.
That said, I understand I'm probably a minory opinion, and that because
this is a sensitive topic we should not rush it... but it's not something
that should be put in a wish list (e.g some untested branch), because it's
something that needs to be fixed sooner or later.
Philippe
[-- Attachment #2: Type: text/html, Size: 1312 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 19:29 ` Philippe Vaucher
@ 2016-11-30 19:45 ` Daniel Colascione
0 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-30 19:45 UTC (permalink / raw)
To: Philippe Vaucher
Cc: Eli Zaretskii, Paul Eggert, Richard Stallman, Emacs developers
On Wed, Nov 30 2016, Philippe Vaucher wrote:
> EZ> Maybe, maybe not. I don't understand why we need to grasp the first straw
> EZ> we see around to get rid of unexec. What's the rush?
>
> Eli has a good point here, there really is no rush. Daniel, would you be
> willing to continue to incubate these changes in a branch, while we also
> consider and explore alternatives?
>
> Isn't unexec what prevents emacs from building inside docker containers (because of the personality
> syscall)?
>
> As far as I'm concerned, that's a quite annoying emacs limitation. See bug
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23529.
With the portable dumper, we'll no longer need any personality-changing
calls. Emacs will be a normal, boring program.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 22:01 ` Paul Eggert
2016-11-30 0:37 ` Daniel Colascione
@ 2016-11-30 20:07 ` Richard Stallman
2016-11-30 20:18 ` Daniel Colascione
1 sibling, 1 reply; 299+ messages in thread
From: Richard Stallman @ 2016-11-30 20:07 UTC (permalink / raw)
To: Paul Eggert; +Cc: dancol, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> I'm more worried about the next level up. Although the dump is pure data
> to the machine, it's not pure data to Elisp. Since the dump would
> contain bytecodes, if attackers can alter the bytecodes then they can
> execute whatever Elisp code they want.
If they can change installed system files, they are already in control
of the machine, so why worry about this?
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 20:07 ` Richard Stallman
@ 2016-11-30 20:18 ` Daniel Colascione
2016-12-03 21:32 ` Richard Stallman
0 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-30 20:18 UTC (permalink / raw)
To: Richard Stallman; +Cc: Paul Eggert, emacs-devel
On Wed, Nov 30 2016, Richard Stallman wrote:
> > I'm more worried about the next level up. Although the dump is pure data
> > to the machine, it's not pure data to Elisp. Since the dump would
> > contain bytecodes, if attackers can alter the bytecodes then they can
> > execute whatever Elisp code they want.
>
> If they can change installed system files, they are already in control
> of the machine, so why worry about this?
We're not talking about changing system files on disk. We're talking
about changing the image of a system file in memory.
Here's the scenario: suppose I can convince your Emacs to parse a
carefully crafted network packet that triggers a bug in Emacs and lets
me overwrite arbitrary memory in your Emacs process. Today, I win, in
the sense that I gain complete control over your Emacs process and can
do anything Emacs can do. Address space layout randomization defeats
this attack by randomizing the memory layout of the process: if the
memory layout is randomized, my arbitrary-memory-overwrite vulnerability
doesn't let me do anything interesting, because I don't know *where* and
*what* to write using this vulnerability.
Now, if I know that your Emacs process is randomized *except* for the
location of the dump, I still win, because I can just write to the dump
image in order to make Emacs do what I want it to do. Writing the dump
works because we'll have mapped it copy-on-write, which is needed in
order to make the dump usable as an Emacs heap image. Writes work
whether or not the underlying file is itself writable. Paul is
absolutely right that we shouldn't defeat ALSR this way.
Maybe we can randomize the dump base address *per* *machine*? That way,
remote attackers would still be frustrated. We'd still be making Emacs
users vulnerable to hostile users running on the same machine.
(Or we could just randomize per-user and dump Emacs the first time it
runs for a particular user? If we do that after loading ~/.emacs, we
also improve people's startup time. Invalidating and regenerating the
dump when configuration changes would be a challenge though.)
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 19:14 ` Daniel Colascione
@ 2016-11-30 21:03 ` John Wiegley
2016-11-30 21:06 ` Paul Eggert
2016-11-30 21:35 ` Daniel Colascione
0 siblings, 2 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-30 21:03 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, Paul Eggert, rms, emacs-devel
>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
DC> No. This code isn't disruptive to the Emacs core if it's used, so there's
DC> no reason to keep it out of master. Experience with the concurrency branch
DC> tells me that interesting work in some feature branch ghetto never
DC> actually gets merged.
You've only tested this on one platform so far, yes? That alone is reason not
to merge it immediately into master. Also, I'd like to see documentation
before it goes in, because that's essential to its maintainability.
Further, I don't like hearing ultimatums, on either side, over what amounts to
an efficiency issue. This matter has already shown that it deserves a little
cooling off and further thought. Call it a ghetto if you will, but you've not
convinced me that incubating code in branches is a bad idea.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:03 ` John Wiegley
@ 2016-11-30 21:06 ` Paul Eggert
2016-11-30 21:44 ` John Wiegley
2016-12-01 3:32 ` Eli Zaretskii
2016-11-30 21:35 ` Daniel Colascione
1 sibling, 2 replies; 299+ messages in thread
From: Paul Eggert @ 2016-11-30 21:06 UTC (permalink / raw)
To: Daniel Colascione, Eli Zaretskii, rms, emacs-devel
On 11/30/2016 01:03 PM, John Wiegley wrote:
> I don't like hearing ultimatums, on either side, over what amounts to
> an efficiency issue.
Yes, I've heard too many ultimatums too.
However, this is not simply an efficiency issue. Emacs currently does
not run on FreeBSD arm64 because of the unexec problem (Bug#24892), and
it's quite likely that the set of platforms where Emacs does not run
will grow as platforms change memory management techniques. We should
not reject a working fix merely because we hypothesize that there might
be a better improvement that someone might implement later.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 16:38 ` Eli Zaretskii
2016-11-30 18:57 ` John Wiegley
@ 2016-11-30 21:06 ` Paul Eggert
1 sibling, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2016-11-30 21:06 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: dancol, rms, emacs-devel
On 11/30/2016 08:38 AM, Eli Zaretskii wrote:
> you didn't like this same idea very much
Yes and I still don't like it much, and if someone implements a better
solution I will be happy to advocate replacing the currently-proposed
approach with a better one. The Emacs dumping scheme has always been and
will remain private and Emacs-version-specific, so we will have a lot of
flexibility here.
In the meantime, if the currently-proposed approach works and is fast, I
see no reason to reject the contribution, or to delay it indefinitely by
putting it into some branch that nobody looks at. If the patch
adequately fixes a real and growing problem and we have no other working
fix, we should install it into the master branch.
>> from what I can see, the C pipeline is by no means empty.
> Well, it somehow becomes all but empty on the other end.
Sure, maybe 40 years from now C programmers will be history! That is no
reason for us to reject a patch that we need today.
If we want to switch the Emacs core from C to some other programming
environment, fine. We can think about doing that and we will have plenty
of time to think. In the meantime, the core can and should assume C.
Every solution to this more-urgent problem will involve some C hacking,
and that's OK.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:03 ` John Wiegley
2016-11-30 21:06 ` Paul Eggert
@ 2016-11-30 21:35 ` Daniel Colascione
2016-11-30 21:44 ` John Wiegley
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-11-30 21:35 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Paul Eggert, rms, emacs-devel
On Wed, Nov 30 2016, John Wiegley wrote:
>>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
>
> DC> No. This code isn't disruptive to the Emacs core if it's used, so there's
> DC> no reason to keep it out of master. Experience with the concurrency branch
> DC> tells me that interesting work in some feature branch ghetto never
> DC> actually gets merged.
>
> You've only tested this on one platform so far, yes? That alone is reason not
> to merge it immediately into master. Also, I'd like to see documentation
> before it goes in, because that's essential to its maintainability.
>
> Further, I don't like hearing ultimatums, on either side, over what amounts to
> an efficiency issue. This matter has already shown that it deserves a little
> cooling off and further thought. Call it a ghetto if you will, but you've not
> convinced me that incubating code in branches is a bad idea.
It's not an ultimatum: I'm merely being up-front about the that work I
am willing to do and the work that I am not willing to do. I am not
willing to work on this code in a branch, because if I do, it is very
likely that this code will never be merged --- just like the concurrency
branch. If we put this thing on a branch, it will be ignored. At best,
every few years, someone will bring it the portable dumper --- just like
the concurrency branch --- and ask why we don't use it. At those times,
we'll repeat the same argument we're having now and and never move
forward with merging the increasingly bitrotted code.
What will change in the future, when we talk about merging the
hypothetical branch? Nobody has expressed concern about the technical
quality of this feature. Yes, of course the final version will have more
documentation --- but that's something we can do right now, pre-commit,
not something we need to incubate on a branch. It's not as if we think
this thing might destabilize Emacs, especially if (at first) it's not
actually compiled into Emacs by default.
Putting this code on a branch is burying it. Do you want to reject code
that *exists* and that solves real problems on the basis of one
developer's philosophical objections and preference for code that does
not exist, that nobody has signed up to write, and that I insist will
never perform adequately? How disappointed do you want to make people
who've written working code that solves real problems?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:35 ` Daniel Colascione
@ 2016-11-30 21:44 ` John Wiegley
2016-11-30 21:50 ` Daniel Colascione
2016-12-01 3:41 ` Eli Zaretskii
0 siblings, 2 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-30 21:44 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, Paul Eggert, rms, emacs-devel
>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
DC> It's not an ultimatum: I'm merely being up-front about the that work I am
DC> willing to do and the work that I am not willing to do. I am not willing
DC> to work on this code in a branch, because if I do, it is very likely that
DC> this code will never be merged --- just like the concurrency branch.
BTW, the concurrency branch has the green light. If anyone is willing to work
to resolve its remaining issues, they can merge it to master once ready.
DC> Putting this code on a branch is burying it. Do you want to reject code
DC> that *exists* and that solves real problems on the basis of one
DC> developer's philosophical objections and preference for code that does not
DC> exist, that nobody has signed up to write, and that I insist will never
DC> perform adequately? How disappointed do you want to make people who've
DC> written working code that solves real problems?
Until you're code is tested on more platforms, it needs to live someplace
other than master. That's what branches are for.
I agree with Eli that it doesn't need to be merged *today*. However, I also
agree with RMS, and you, that at least it's a working solution (to some
extent), so if we can't think of something better during the time you're
ironing the wrinkles out, it ought to go in until we do.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:06 ` Paul Eggert
@ 2016-11-30 21:44 ` John Wiegley
2016-12-01 3:32 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-30 21:44 UTC (permalink / raw)
To: Paul Eggert; +Cc: Eli Zaretskii, Daniel Colascione, rms, emacs-devel
>>>>> "PE" == Paul Eggert <eggert@cs.ucla.edu> writes:
PE> However, this is not simply an efficiency issue. Emacs currently does not
PE> run on FreeBSD arm64 because of the unexec problem (Bug#24892), and it's
PE> quite likely that the set of platforms where Emacs does not run will grow
PE> as platforms change memory management techniques. We should not reject a
PE> working fix merely because we hypothesize that there might be a better
PE> improvement that someone might implement later.
That's a good point, Paul, I hadn't thought of that. And the Docker issue,
apparently.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:44 ` John Wiegley
@ 2016-11-30 21:50 ` Daniel Colascione
2016-11-30 22:20 ` John Wiegley
` (2 more replies)
2016-12-01 3:41 ` Eli Zaretskii
1 sibling, 3 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-11-30 21:50 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Paul Eggert, rms, emacs-devel
On Wed, Nov 30 2016, John Wiegley wrote:
>>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
>
> DC> It's not an ultimatum: I'm merely being up-front about the that work I am
> DC> willing to do and the work that I am not willing to do. I am not willing
> DC> to work on this code in a branch, because if I do, it is very likely that
> DC> this code will never be merged --- just like the concurrency branch.
>
> BTW, the concurrency branch has the green light. If anyone is willing to work
> to resolve its remaining issues, they can merge it to master once ready.
>
> DC> Putting this code on a branch is burying it. Do you want to reject code
> DC> that *exists* and that solves real problems on the basis of one
> DC> developer's philosophical objections and preference for code that does not
> DC> exist, that nobody has signed up to write, and that I insist will never
> DC> perform adequately? How disappointed do you want to make people who've
> DC> written working code that solves real problems?
>
> Until you're code is tested on more platforms, it needs to live someplace
> other than master. That's what branches are for.
I'll test it on the systems I can before I commit. We can enable
compilation once on individual systems once we do more thorough testing.
A branch is definitely not the right approach here, because you haven't
established any firm acceptance criteria, and what criteria you have
mentioned (write some documentation, test on more configurations) are
things I can do before the patch lands at all.
The Cairo code doesn't work at all and *that's* in mainline.
That's okay, because it's not enabled by default. Branches are for
experimental features that can't coexist with regular Emacs use
and development. The portable dumper is not such a feature.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:50 ` Daniel Colascione
@ 2016-11-30 22:20 ` John Wiegley
2016-12-01 1:37 ` Paul Eggert
2016-12-01 3:47 ` Eli Zaretskii
2 siblings, 0 replies; 299+ messages in thread
From: John Wiegley @ 2016-11-30 22:20 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, Paul Eggert, rms, emacs-devel
>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
DC> The Cairo code doesn't work at all and *that's* in mainline. That's okay,
DC> because it's not enabled by default. Branches are for experimental
DC> features that can't coexist with regular Emacs use and development. The
DC> portable dumper is not such a feature.
You do have a point there. I'll give Eli some more time to think on this, and
then we'll get back to you. Thanks for the patience.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:50 ` Daniel Colascione
2016-11-30 22:20 ` John Wiegley
@ 2016-12-01 1:37 ` Paul Eggert
2016-12-01 1:45 ` Daniel Colascione
2016-12-01 3:47 ` Eli Zaretskii
2 siblings, 1 reply; 299+ messages in thread
From: Paul Eggert @ 2016-12-01 1:37 UTC (permalink / raw)
To: Daniel Colascione, Eli Zaretskii; +Cc: rms, emacs-devel
On 11/30/2016 01:50 PM, Daniel Colascione wrote:
> I'll test it on the systems I can before I commit.
Once you get it working I can volunteer to test it on a few unusual
systems (e.g., Solaris 10 sparc32, AIX 7 POWER64); this shouldn't take
long if it's not a holiday. We should also test a 32-bit platform built
--with-wide-int.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 1:37 ` Paul Eggert
@ 2016-12-01 1:45 ` Daniel Colascione
0 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-01 1:45 UTC (permalink / raw)
To: Paul Eggert, Eli Zaretskii; +Cc: rms, emacs-devel
[-- Attachment #1: Type: text/plain, Size: 497 bytes --]
I can test the wide int, and help on exotic systems will be much appreciated
On November 30, 2016 5:37:46 PM PST, Paul Eggert <eggert@cs.ucla.edu> wrote:
>On 11/30/2016 01:50 PM, Daniel Colascione wrote:
>> I'll test it on the systems I can before I commit.
>
>Once you get it working I can volunteer to test it on a few unusual
>systems (e.g., Solaris 10 sparc32, AIX 7 POWER64); this shouldn't take
>long if it's not a holiday. We should also test a 32-bit platform built
>
>--with-wide-int.
[-- Attachment #2: Type: text/html, Size: 916 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:06 ` Paul Eggert
2016-11-30 21:44 ` John Wiegley
@ 2016-12-01 3:32 ` Eli Zaretskii
2016-12-01 9:16 ` Paul Eggert
1 sibling, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-01 3:32 UTC (permalink / raw)
To: Paul Eggert; +Cc: dancol, rms, emacs-devel
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 30 Nov 2016 13:06:11 -0800
>
> However, this is not simply an efficiency issue. Emacs currently does
> not run on FreeBSD arm64 because of the unexec problem (Bug#24892)
Let's get our facts straight, shall we? First, Emacs never worked on
that platform, and second, the current reason is that they removed a
certain library function, which happens to be called by unexec.
Removal of a library function can break any part of Emacs.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:44 ` John Wiegley
2016-11-30 21:50 ` Daniel Colascione
@ 2016-12-01 3:41 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-01 3:41 UTC (permalink / raw)
To: John Wiegley; +Cc: dancol, emacs-devel, rms, eggert
> From: John Wiegley <jwiegley@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>, rms@gnu.org, emacs-devel@gnu.org
> Date: Wed, 30 Nov 2016 13:44:14 -0800
>
> BTW, the concurrency branch has the green light. If anyone is willing to work
> to resolve its remaining issues, they can merge it to master once ready.
I will try to give it a go this weekend.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 21:50 ` Daniel Colascione
2016-11-30 22:20 ` John Wiegley
2016-12-01 1:37 ` Paul Eggert
@ 2016-12-01 3:47 ` Eli Zaretskii
2016-12-01 4:10 ` John Wiegley
2016-12-01 4:10 ` Daniel Colascione
2 siblings, 2 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-01 3:47 UTC (permalink / raw)
To: Daniel Colascione; +Cc: eggert, rms, emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>, rms@gnu.org, emacs-devel@gnu.org
> Date: Wed, 30 Nov 2016 13:50:33 -0800
>
> > Until you're code is tested on more platforms, it needs to live someplace
> > other than master. That's what branches are for.
>
> I'll test it on the systems I can before I commit. We can enable
> compilation once on individual systems once we do more thorough testing.
> A branch is definitely not the right approach here, because you haven't
> established any firm acceptance criteria, and what criteria you have
> mentioned (write some documentation, test on more configurations) are
> things I can do before the patch lands at all.
Having new features on a branch is our standard development practice.
It allows the interested people to try the feature in situations no
single person can possibly create, even if that person has access to
several platforms, because usage patterns differ a lot.
We can establish pass criteria, if you like, but the only criteria
until now were "if enough time passed and no one complained, it's
ready".
In this case, we will probably also consider alternative approaches if
they become mature.
> The Cairo code doesn't work at all and *that's* in mainline.
> That's okay, because it's not enabled by default.
The Cairo code works, it just has some redisplay bugs that no one was
able to fix. It was disabled post-factum, when we have learned about
those problems, unfortunately after its developer has departed.
> Branches are for experimental features that can't coexist with
> regular Emacs use and development.
This is simply not true.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 3:47 ` Eli Zaretskii
@ 2016-12-01 4:10 ` John Wiegley
2016-12-01 4:12 ` Daniel Colascione
2016-12-01 4:10 ` Daniel Colascione
1 sibling, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-12-01 4:10 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Daniel Colascione, emacs-devel, rms, eggert
>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
EZ> Having new features on a branch is our standard development practice. It
EZ> allows the interested people to try the feature in situations no single
EZ> person can possibly create, even if that person has access to several
EZ> platforms, because usage patterns differ a lot.
EZ> We can establish pass criteria, if you like, but the only criteria until
EZ> now were "if enough time passed and no one complained, it's ready".
EZ> In this case, we will probably also consider alternative approaches if
EZ> they become mature.
I agree with Eli here, this is what branches are for. Daniel, if you think
they have cultural baggage that makes them distasteful, that's a different
problem. Branches are a perfectly acceptable tool, and they give us an easy
way to assess progress as the work develops.
Please submit your code to a branch, and we'll ask everyone who's capable to
begin testing on as many platforms as we can find. Once enough time has passed
and no one has complained, it can move to master.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 3:47 ` Eli Zaretskii
2016-12-01 4:10 ` John Wiegley
@ 2016-12-01 4:10 ` Daniel Colascione
1 sibling, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-01 4:10 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: eggert, rms, emacs-devel
On November 30, 2016 7:47:21 PM PST, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: Paul Eggert <eggert@cs.ucla.edu>, rms@gnu.org,
>emacs-devel@gnu.org
>> Date: Wed, 30 Nov 2016 13:50:33 -0800
>>
>> > Until you're code is tested on more platforms, it needs to live
>someplace
>> > other than master. That's what branches are for.
>>
>> I'll test it on the systems I can before I commit. We can enable
>> compilation once on individual systems once we do more thorough
>testing.
>> A branch is definitely not the right approach here, because you
>haven't
>> established any firm acceptance criteria, and what criteria you have
>> mentioned (write some documentation, test on more configurations) are
>> things I can do before the patch lands at all.
>
>Having new features on a branch is our standard development practice.
No it isn't.
>It allows the interested people to try the feature in situations no
>single person can possibly create, even if that person has access to
>several platforms, because usage patterns differ a lot.
No, that's what configuration options do. Branches are seldom touched, because there are much bigger barriers to trying them.
>We can establish pass criteria, if you like, but the only criteria
>until now were "if enough time passed and no one complained, it's
>ready".
You have wanted to kill this feature at any cost, contrary to the wishes of every other Emacs contributor. What kind of fair pass criteria can I expect?
>In this case, we will probably also consider alternative approaches if
>they become mature.
Got it. So we won't consider merging the code until some lread improvement comes along somehow (which might take years) and we'll declare the lread improvement the winner no matter what the performance difference.
>> The Cairo code doesn't work at all and *that's* in mainline.
>> That's okay, because it's not enabled by default.
>
>The Cairo code works, it just has some redisplay bugs that no one was
>able to fix.
So it doesn't work.
> It was disabled post-factum, when we have learned about
>those problems, unfortunately after its developer has departed.
It's obvious within five minutes of trying the feature that it doesn't work. That's okay, because it's opt-in.
>> Branches are for experimental features that can't coexist with
>> regular Emacs use and development.
>
>This is simply not true.
It is true if you look at the history of things for which we've actually used branches.
Putting this feature on a branch is tantamount to killing it. There is no legitimate reason to keep it out of mainline, because it is harmless unless enabled, and we can keep it disabled for the moment.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 4:10 ` John Wiegley
@ 2016-12-01 4:12 ` Daniel Colascione
2016-12-01 4:49 ` John Wiegley
0 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-12-01 4:12 UTC (permalink / raw)
To: John Wiegley, Eli Zaretskii; +Cc: eggert, rms, emacs-devel
On November 30, 2016 8:10:21 PM PST, John Wiegley <jwiegley@gmail.com> wrote:
>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
>EZ> Having new features on a branch is our standard development
>practice. It
>EZ> allows the interested people to try the feature in situations no
>single
>EZ> person can possibly create, even if that person has access to
>several
>EZ> platforms, because usage patterns differ a lot.
>
>EZ> We can establish pass criteria, if you like, but the only criteria
>until
>EZ> now were "if enough time passed and no one complained, it's ready".
>
>EZ> In this case, we will probably also consider alternative approaches
>if
>EZ> they become mature.
>
>I agree with Eli here, this is what branches are for. Daniel, if you
>think
>they have cultural baggage that makes them distasteful, that's a
>different
>problem. Branches are a perfectly acceptable tool, and they give us an
>easy
>way to assess progress as the work develops.
>
>Please submit your code to a branch, and we'll ask everyone who's
>capable to
>begin testing on as many platforms as we can find. Once enough time has
>passed
>and no one has complained, it can move to master.
If that's your decision, I'm going to fork.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 4:12 ` Daniel Colascione
@ 2016-12-01 4:49 ` John Wiegley
2016-12-01 5:12 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-12-01 4:49 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, eggert, rms, emacs-devel
>>>>> Daniel Colascione <dancol@dancol.org> writes:
> If that's your decision, I'm going to fork.
I'm quite sorry to hear it, but if branches are unacceptable, then such is
your right as a free software developer.
Just so you know, I'd hope to see your patches on master within six months, if
we can find some willing testers (and one has already stepped forward). The
only way your changes *wouldn't* land is if a better alternative comes along
in the meantime -- which, of course, would mean we all win.
I don't suppose I can encourage you to maintain your fork within the Emacs.git
repository? There this thing, rhymes with "blanch", where you could work on
your version in tandem, even merge changes from us every once in a while... :)
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 4:49 ` John Wiegley
@ 2016-12-01 5:12 ` Daniel Colascione
2016-12-01 9:03 ` Matt Armstrong
2016-12-01 9:18 ` Phillip Lord
0 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-01 5:12 UTC (permalink / raw)
To: Eli Zaretskii, eggert, rms, emacs-devel
On 11/30/2016 08:49 PM, John Wiegley wrote:
>>>>>> Daniel Colascione <dancol@dancol.org> writes:
>
>> If that's your decision, I'm going to fork.
>
> I'm quite sorry to hear it, but if branches are unacceptable, then such is
> your right as a free software developer.
Branches in general are fine. Relegating this specific feature to a
branch feels political.
I would very much prefer to continue improving GNU Emacs. I'm a longtime
fan of the GNU Project. But I feel like the current gatekeepers are
standing in the way of progress. We can't even make the conservative GC
safe in the face of foreseeable compiler optimizations. This isn't the
first time that I've gotten a flat "no" in response to a good idea. If I
do fork, I can incorporate many more kinds of experimental work. One of
the first things I'd consider would be an LLVM-based tracing JIT.
Let me try one last time to explain my position: this dumping code will
make no Emacs no worse. It cannot possible destabilize Emacs when it is
compiled out of Emacs at configure-time. I specifically designed this
feature to be optional. I am perfectly happy merging my code in such a
way that it is not compiled into Emacs by default, and I am perfectly
happy waiting until we've gotten sufficient testing before enabling the
portable dumper.
In the miraculous case that someone contributes an lread optimization
that allows that the big-ELC approach to perform as well as a memory
image, I will happily back out of my code.
But because this code is harmless when compiled away and because we can
back it out without problems, I see the requirement that we put this
code into a branch as a way to kill the feature. If this code were more
fundamentally destabilizing, I would feel different --- but as it is,
since this code is *not* fundamentally destabilizing, I don't see a
technical case for a feature branch.
I am perfectly happy with a six month or longer timetable for enabling
this code at ./configure time. I expected as much. The XEmacs experience
with their portable dumper (which I have not read) was similar. I
specifically designed this code to be compatible with the current
implementation and to allow for a gradual transition.
Is anyone else signing up to make Emacs work better on modern systems?
> Just so you know, I'd hope to see your patches on master within six months,
What specific conditions need to be met before merging? Why can't I meet
these requirements before landing the diff? This code will never meet
the "nobody has complained" requirement. It's going to have to land over
some objections, because some contributors object to the very essence of
the change. If there are technical requirements, why can't I meet these
requirements before landing this work in master? If there are political
requirements, what makes you think that a branch would help me meet them?
> if
> we can find some willing testers (and one has already stepped forward). The
> only way your changes *wouldn't* land is if a better alternative comes along
> in the meantime -- which, of course, would mean we all win.
What is the fundamental difference between merging this code into
mainline in a disabled-by-default configuration and merging it into a
branch, except that it's much more convenient to try the feature in the
former case?
>
> I don't suppose I can encourage you to maintain your fork within the Emacs.git
> repository? There this thing, rhymes with "blanch", where you could work on
> your version in tandem, even merge changes from us every once in a while... :)
>
There have been cries to adopt a more modern development style, one
focused on GitHub and pull requests. I can certainly accommodate.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 5:12 ` Daniel Colascione
@ 2016-12-01 9:03 ` Matt Armstrong
2016-12-02 8:10 ` John Wiegley
2016-12-01 9:18 ` Phillip Lord
1 sibling, 1 reply; 299+ messages in thread
From: Matt Armstrong @ 2016-12-01 9:03 UTC (permalink / raw)
To: Daniel Colascione, Eli Zaretskii, eggert, rms, emacs-devel
Daniel Colascione <dancol@dancol.org> writes:
> On 11/30/2016 08:49 PM, John Wiegley wrote:
>>>>>>> Daniel Colascione <dancol@dancol.org> writes:
>>
>>> If that's your decision, I'm going to fork.
>>
>> I'm quite sorry to hear it, but if branches are unacceptable, then such is
>> your right as a free software developer.
>
> Branches in general are fine. Relegating this specific feature to a
> branch feels political.
>
> I would very much prefer to continue improving GNU Emacs. I'm a longtime
> fan of the GNU Project. But I feel like the current gatekeepers are
> standing in the way of progress.
Hey Daniel, as a mostly-irrelevant observer, I hope you do. I don't
have time to devote to Emacs the way you do. I think the best use of my
time is to try to convince you to continue doing so, by being as honest
as I can. My advice:
- take a breath, cool down
- give it time, build trust
- build on areas of agreement and common interest
I'm not an Emacs maintainer, and can't speak as one. But, I play a
similar role at work, and can tell you that the technical issues at play
in these conflicts you have with Emacs maintainers are secondary. At
their core, these discussions are all about trust. You don't fully
trust the maintainers, and they don't fully trust you (hopefully "yet"
on both accounts).
In my "day job" I have the equivalent of "commit rights" to some widely
used software critical to the company's function.
From what I see, the role is similar to that of an Emacs maintainer. It
is actually more important that the software remain functional and high
quality, and healthy for the long term, than it is to optimize
development practices for velocity of change.
Like Eli, John, and other maintainers, I'm often put in the position of
saying "no" or "slow down, I'm not sure yet" to a change. Quite often
those asking for the change don't have the same perspective I do. They
might see their change as minor. Or critical. Or urgent. Or, if what
they introduce incurs some ongoing maintenance cost, they might
underestimate that cost, or truly believe in their commitment to bearing
that cost for the long term.
What they often don't appreciate is that I've seen *all* the failures.
I've seen the "simiple risk free idea" break horribly, or later became
something we regret. I've seen authors vanish with half finished ideas,
leaving the *actual* maintainers with a mess. I've seen "critically
urgent" changes become totally unnecessary as easier, more viable
alternatives come to light (often only after careful analysis, done by
me or others, sometimes to the impatient response of those proposing
change).
In this kind of system, where there is a hierarchy of decision makers,
and quality is often more important than speed, trust is earned over
time.
The quickest way to build trust is slowly, with a demonstrated
willingness and capability to collaborate and be sane over a period of
time. Technical merit and an ability to make good logical arguments are
also important, but secondary.
True technical "arguments" are always boring. They're pure pro/con
assessments, dispassionately weighing the knowns, unknowns, risks, etc.
Things get "spicy" when it is issues of trust, power, authority,
morality, etc. Yet, I think it is human nature to try to "win" such
discussions by continuing to appeal to technical issues alone. Quite
often, it boils down to "I don't trust you yet" and "Your lack of trust
frustrates me". There is no getting around that. A decision being made
for "political reasons" is not actually bad, it is the expected norm.
As a mater of political strategy, I'd encourage you to view "please
commit to a branch" as a victory. Do it and watch trust grow in both
directions. It is the foot in the door that gets the project closer to
the place you'd like to see it go. It is a strategic move!
> Let me try one last time to explain my position: this dumping code
> will make no Emacs no worse. It cannot possible destabilize Emacs when
> it is compiled out of Emacs at configure-time. I specifically designed
> this feature to be optional. I am perfectly happy merging my code in
> such a way that it is not compiled into Emacs by default, and I am
> perfectly happy waiting until we've gotten sufficient testing before
> enabling the portable dumper.
First, let me say these your arguments don't strike me as being wildly
off base. But, I'm not even an Emacs maintainer and can't help but view
the with some suspicion born from the school of hard knocks (see
scenarios outlined above). Viewed from my experience, if I pretend to
be an Emacs maintainer, this is actually what I end up thinking:
- There will be more #ifdefs in Emacs code.
- You are not likely to admit it, but you may disappear. If you do, it
may be my job to remove the #ifdefs.
- You are particularly forceful and invested in getting your way. If I
end up saying "no" in the end, you may walk away, leaving me with a
mess to clean up.
- You are less experienced than the maintainers with the code base, but
claim a change is safe/easy/cheap/harmless. That is nice, but I've
seen these claims play out differently before.
- So, risk is non zero. I'd like to entertain ways to move this
forward with less risk.
> But because this code is harmless when compiled away and because we
> can back it out without problems, I see the requirement that we put
> this code into a branch as a way to kill the feature. If this code
> were more fundamentally destabilizing, I would feel different --- but
> as it is, since this code is *not* fundamentally destabilizing, I
> don't see a technical case for a feature branch.
The purpose of a branch is to isolate risk. Sure, any branch may never
merge back, but the primary reason to commit to a branch is to
*facilitate* future code sharing between the branch and mainline, not to
kill features.
>> Just so you know, I'd hope to see your patches on master within six
>> months,
>
> What specific conditions need to be met before merging?
I think asking for a contract is not unfair, but may be unrealistic.
Have you seen the green light given to the concurrency branch? That
branch began without a contract.
Again, from a maintainer's point of view, this kind of contractual
question is sometimes unanswerable. The maintainer may simply feel
reluctance and not know why.
Simply being willing to "play ball" over a period of time, combined with
the absence of any better options coming along, can often tip the scale
in the "okay to merge" direction. Sometimes people need to actually
"see" a diff on their own screens, run the binary themselves, etc., and
hear a few positive reports from others they trust, before they're
willing to green light something. Emailing a pre-commited patch around
is one way to do that, but git was invented to support a better workflow
directly.
Reluctance on the part of project owners is not inherently bad.
I have actually been in the position of apologizing to people after I
held off their proposed changes for a long time, only to realize that
"they were right all along" and was "obviously the right thing to do" in
hindsight. I usually thank them for being patient with my reluctance.
In reality, my reluctance is usually good judgment, and has saved more
work than it has caused harm. Often, I cannot even initially articulate
why an idea gives me pause. Sometimes it comes to me some time after
the fact.
>> if we can find some willing testers (and one has already stepped
>> forward). The only way your changes *wouldn't* land is if a better
>> alternative comes along in the meantime -- which, of course, would
>> mean we all win.
The above is John saying "yes". I can't tell you how many times I've
said yes to 95% of a proposal and people walk away because of a minor
difference of opinion on a non-essential issue.
> What is the fundamental difference between merging this code into
> mainline in a disabled-by-default configuration and merging it into a
> branch
1) the feature owner (you) absorbs the entire cost of maintaining the
branch. The maintenance cost at mainline is zero. There is zero chance
of ChangeLog, documentation, NEWS, or any mention of the feature
appearing in future snapshots, pretests, releases. As far as I know,
there is no #ifdef for Emacs documentation. If the feature is not
eventually accepted, the cost of rejecting it is zero. There is no
"backing the change out" step.
2) it is easy to review the state of the diff in its entirety against
the master at any point. Once merged you're stuck with more primitive
tools like grep and git blame.
> , except that it's much more convenient to try the feature in the
> former case?
It is probably worth enumerating the concrete differences.
1) git makes it trivial to check out a branch and build. Playing around
with the concurrency branch was pretty easy the time I did it. For the
"I want to try this for a day and report back" use case, it is fine.
2) using a feature branch long term *is* harder. Only one can be used
at a time (easily), and the feature branch owner must merge from master
regularly to keep it relevant.
3) using a branch *combined* with another branch is a pain. If I want
to test "concurrency AND portable dumper" I have to merge the two
branches myself.
There is no correct choice, they are just simple tradeoffs. I see both
options as reasonable.
If the goal is to collect a small-ish number of "I tried it on platform
X and it works" reports, and let the ideas percolate/bake for a time,
then a branch makes a lot of sense.
If the goal is to get the feature used by as many different people as
possible as fast as possible in as many different scenarios as possible,
then a branch makes less sense.
I'll point out that your reluctance to commit to a branch is about trust
and not about the technical issues above: you don't seem to believe John
when he says the intent is to facilitate a future merge.
>> I don't suppose I can encourage you to maintain your fork within the
>> Emacs.git repository? There this thing, rhymes with "blanch", where
>> you could work on your version in tandem, even merge changes from us
>> every once in a while... :)
>
> There have been cries to adopt a more modern development style, one
> focused on GitHub and pull requests. I can certainly accommodate.
I do agree that submitting patches to projects on "non github" systems
is beginning to feel archaic. I hope the era of emailing patches around
disappears.
If this is a detractor for Emacs it is also a detractor for every
project hosted on savannah.gnu.org. Rather than fork Emacs to github to
get the "modern" development style, the greater impact, in terms of
benefit to GNU, is adding the desired features to savannah.
I don't know if anybody is working on this or not.
That said, I think you're too quick to suggest forking Emacs. You'll
get more done working with Emacs folks than in spite of them, and the
stuff you work on will have more impact. Even if it feels to you like
you could go 10x faster alone, building a real community is pretty hard,
and you'd be at high risk of burn out doing it alone, and the final
product may not result in more than another non-standard Emacs that gets
little use.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 3:32 ` Eli Zaretskii
@ 2016-12-01 9:16 ` Paul Eggert
2016-12-01 17:26 ` Eli Zaretskii
0 siblings, 1 reply; 299+ messages in thread
From: Paul Eggert @ 2016-12-01 9:16 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: dancol, rms, emacs-devel
Eli Zaretskii wrote:
> Emacs never worked on that platform
Sure it did. Emacs worked until FreeBSD 11, when the arm64 port modernized its
memory-management implementation. If Glibc does something similar, it would be a
mistake to say "that's OK, Emacs never worked on that platform", as this might
give non-experts the misimpression that Emacs never worked on GNU/Linux.
> they removed a certain library function, which happens to be called by unexec.
This was no accident. They removed it because it was producing bogus values, and
it didn't reflect how FreeBSD actually allocates memory. Whether the function
worked for Emacs at all was apparently a matter of luck.
I would rather not tell FreeBSD/arm64 users that they can't run Emacs any more
because we can't be bothered to install a patch that's been contributed, not
even a patch that's compiled by default only on FreeBSD/arm64.
I sympathize with Daniel about the politics of this. This is not the sort of
change that we'd ordinarily create a new branch for, and the way this is being
handled will likely discourage further contributions.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:39 ` Eli Zaretskii
2016-11-29 19:03 ` Daniel Colascione
2016-11-29 19:13 ` Paul Eggert
@ 2016-12-01 9:18 ` Richard Stallman
2016-12-01 18:09 ` Eli Zaretskii
2 siblings, 1 reply; 299+ messages in thread
From: Richard Stallman @ 2016-12-01 9:18 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: dancol, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
I sympathize with your general point, but I don't think it is
conceivable to make Emacs's C code much simpler or easier to maintain
than it is now.
Also, I don't think the portable dumper would increase the conplexity,
not if it is written cleanly.
To increase the _number_ of modules that look at Lisp objects with a
given level of complexity does not increase the complexity. When
changing the data structures, it is no harder to update 4 places than
3 places. It adds a little more work, but it isn't _harder_ work.
I conclude that the portable dumper won't make it any harder to
find people to work on the C code.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-29 18:49 ` Daniel Colascione
2016-11-29 19:02 ` Eli Zaretskii
@ 2016-12-01 9:18 ` Richard Stallman
2016-12-01 18:11 ` Eli Zaretskii
1 sibling, 1 reply; 299+ messages in thread
From: Richard Stallman @ 2016-12-01 9:18 UTC (permalink / raw)
To: Daniel Colascione; +Cc: jwiegley, eliz, burton.samograd, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> I do not think the portable dumper is something that's so complex as to
> not be worth the performance, especially since any lread implementation
> that's even close to being acceptably fast is going to be very complex
> in its own right.
Rather than arguing theoretically about speculations,
let's see people do their best to improve the big-elc-file
approach, and then compare facts with facts.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 5:12 ` Daniel Colascione
2016-12-01 9:03 ` Matt Armstrong
@ 2016-12-01 9:18 ` Phillip Lord
1 sibling, 0 replies; 299+ messages in thread
From: Phillip Lord @ 2016-12-01 9:18 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, eggert, rms, emacs-devel
On Thu, December 1, 2016 5:12 am, Daniel Colascione wrote:
> On 11/30/2016 08:49 PM, John Wiegley wrote:
>> if we can find some willing testers (and one has already stepped
>> forward). The only way your changes *wouldn't* land is if a better
>> alternative comes along in the meantime -- which, of course, would mean
>> we all win.
>
> What is the fundamental difference between merging this code into
> mainline in a disabled-by-default configuration and merging it into a
> branch, except that it's much more convenient to try the feature in the
> former case?
I wasn't going to comment on this thread, but alas here I go!
You launched the thread with "preview" in the title, that is you don't
think that your patch is finished yes. So, you are planning on doing more
work on it, before pushing to master? In which case, you already have a
branch. Why not push that up to the main emacs git? This would make is
very easy for people to test straight off the bat. I'll build and test it
on windows for instance.
I agree with your concerns that it will just moulder on a branch, and
that's true. There is a portable dumper out there for Emacs (version 21),
which is perhaps an indicator. But then disabled-by-default work runs the
same risk. No one tests it that much.
There's enough interest in your work that having it on a branch might even
contribute to encouraging more use of feature branches as part of
development rather than long running forks. That would be good, from my
perspective.
Phil
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 9:16 ` Paul Eggert
@ 2016-12-01 17:26 ` Eli Zaretskii
2016-12-01 17:35 ` Daniel Colascione
2016-12-01 17:58 ` Paul Eggert
0 siblings, 2 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-01 17:26 UTC (permalink / raw)
To: Paul Eggert; +Cc: dancol, rms, emacs-devel
> Cc: dancol@dancol.org, rms@gnu.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 1 Dec 2016 01:16:17 -0800
>
> Emacs never worked on that platform
>
> Sure it did. Emacs worked until FreeBSD 11, when the arm64 port modernized its memory-management implementation.
From https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24892#104
Ed Maste wrote:
> arm64 support was first available in a release in FreeBSD 11.0, without sbrk, and sbrk never existed on the stable/11 branch.
Thanks, I didn't know that. So Emacs has never worked in this environment.
Doesn't that mean Emacs on arm64 was never available before FreeBSD
11, and sbrk never existed on FreeBSD/arm64?
> they removed a certain library function, which happens to be called by unexec.
>
> This was no accident. They removed it because it was producing bogus values, and it didn't reflect how FreeBSD actually allocates memory.
And they are thinking on bringing it back, see
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24892#92
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 17:26 ` Eli Zaretskii
@ 2016-12-01 17:35 ` Daniel Colascione
2016-12-01 17:58 ` Paul Eggert
1 sibling, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-01 17:35 UTC (permalink / raw)
To: Eli Zaretskii, Paul Eggert; +Cc: rms, emacs-devel
On 12/01/2016 09:26 AM, Eli Zaretskii wrote:
>> Cc: dancol@dancol.org, rms@gnu.org, emacs-devel@gnu.org
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Thu, 1 Dec 2016 01:16:17 -0800
>>
>> Emacs never worked on that platform
>>
>> Sure it did. Emacs worked until FreeBSD 11, when the arm64 port modernized its memory-management implementation.
>
> From https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24892#104
>
> Ed Maste wrote:
> > arm64 support was first available in a release in FreeBSD 11.0, without sbrk, and sbrk never existed on the stable/11 branch.
>
> Thanks, I didn't know that. So Emacs has never worked in this environment.
>
> Doesn't that mean Emacs on arm64 was never available before FreeBSD
> 11, and sbrk never existed on FreeBSD/arm64?
>
>> they removed a certain library function, which happens to be called by unexec.
>>
>> This was no accident. They removed it because it was producing bogus values, and it didn't reflect how FreeBSD actually allocates memory.
>
> And they are thinking on bringing it back, see
They shouldn't bring it back. Reserving a single region in a process as
"the heap" and doling out parts of it via mmap is an inflexible
allocation policy. mmap ought to be the only way to get memory from the
system.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 17:26 ` Eli Zaretskii
2016-12-01 17:35 ` Daniel Colascione
@ 2016-12-01 17:58 ` Paul Eggert
1 sibling, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2016-12-01 17:58 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: dancol, rms, emacs-devel
On 12/01/2016 09:26 AM, Eli Zaretskii wrote:
>> >This was no accident. They removed it because it was producing bogus values, and it didn't reflect how FreeBSD actually allocates memory.
> And they are thinking on bringing it back, see
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24892#92
That thread discusses their implementing sbrk as always returning 0 or
something trivial like that, purely to pacify broken applications that
use sbrk (0) to estimate memory usage and fail to build if sbrk does not
exist. Such a workaround obviously won't get Emacs running. In contrast,
Daniel's proposed patch should work.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 9:18 ` Richard Stallman
@ 2016-12-01 18:09 ` Eli Zaretskii
2016-12-02 2:18 ` Stefan Monnier
2016-12-02 14:27 ` Richard Stallman
0 siblings, 2 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-01 18:09 UTC (permalink / raw)
To: rms; +Cc: dancol, emacs-devel
> From: Richard Stallman <rms@gnu.org>
> CC: dancol@dancol.org, emacs-devel@gnu.org
> Date: Thu, 01 Dec 2016 04:18:03 -0500
>
> I sympathize with your general point, but I don't think it is
> conceivable to make Emacs's C code much simpler or easier to maintain
> than it is now.
I agree.
However, this is not about making existing code simpler. It's about
_adding_ more C code, where an alternative implementation might allow
doing that without such additions.
> Also, I don't think the portable dumper would increase the conplexity,
> not if it is written cleanly.
It's a new and significant mechanism, which is central to building and
starting Emacs. It needs to be studied, understood, and maintained.
Therefore, it does increase the complexity. You might argue that the
increase in complexity is not significant enough to worry about, but
it's definitely there.
> To increase the _number_ of modules that look at Lisp objects with a
> given level of complexity does not increase the complexity.
It's not just the number of modules, it's what they do with the
objects. The dumper does qualitatively different things, unlike any
other module in Emacs. It's not just "more of the same".
> When changing the data structures, it is no harder to update 4
> places than 3 places. It adds a little more work, but it isn't
> _harder_ work.
Well, if I need to do more work, the job is harder.
> I conclude that the portable dumper won't make it any harder to
> find people to work on the C code.
I agree with that conclusion. However, my problem is that, since
finding such people is already hard, making more Emacs modules depend
on that might expose us to more future maintenance problems than if we
don't. Especially if those modules are so central to Emacs as the one
being discussed here.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 9:18 ` Richard Stallman
@ 2016-12-01 18:11 ` Eli Zaretskii
2016-12-02 4:28 ` Ken Raeburn
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-01 18:11 UTC (permalink / raw)
To: rms, Ken Raeburn; +Cc: jwiegley, dancol, emacs-devel
> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, jwiegley@gmail.com, burton.samograd@autodesk.com,
> emacs-devel@gnu.org
> Date: Thu, 01 Dec 2016 04:18:04 -0500
>
> Rather than arguing theoretically about speculations,
> let's see people do their best to improve the big-elc-file
> approach, and then compare facts with facts.
Agreed.
Ken, will you be able to continue working on your branch in this
direction? Are all the changes suggested/tried by Stefan already on
that branch? If not, would it be possible for Stefan or yourself to
update the branch?
Thanks in advance.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 18:09 ` Eli Zaretskii
@ 2016-12-02 2:18 ` Stefan Monnier
2016-12-02 7:54 ` Eli Zaretskii
2016-12-02 14:27 ` Richard Stallman
1 sibling, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-02 2:18 UTC (permalink / raw)
To: emacs-devel
> It's a new and significant mechanism, which is central to building and
> starting Emacs. It needs to be studied, understood, and maintained.
FWIW, I looked at Daniel's code and found it understandable, whereas
I never managed to make any sense of the unexec code, which still just looks
like magical voodoo to me, despite my trying twice to understand it.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 18:11 ` Eli Zaretskii
@ 2016-12-02 4:28 ` Ken Raeburn
2016-12-02 4:41 ` Daniel Colascione
2016-12-02 8:03 ` Eli Zaretskii
0 siblings, 2 replies; 299+ messages in thread
From: Ken Raeburn @ 2016-12-02 4:28 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: jwiegley, dancol, rms, emacs-devel
> On Dec 1, 2016, at 13:11, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> From: Richard Stallman <rms@gnu.org>
>> CC: eliz@gnu.org, jwiegley@gmail.com, burton.samograd@autodesk.com,
>> emacs-devel@gnu.org
>> Date: Thu, 01 Dec 2016 04:18:04 -0500
>>
>> Rather than arguing theoretically about speculations,
>> let's see people do their best to improve the big-elc-file
>> approach, and then compare facts with facts.
>
> Agreed.
>
> Ken, will you be able to continue working on your branch in this
> direction? Are all the changes suggested/tried by Stefan already on
> that branch? If not, would it be possible for Stefan or yourself to
> update the branch?
I have an update I’m still working on, to refine the hash table handling I did to generate less garbage while reading normal .el or .elc files. I hope to get that uploaded soon. And I think my previous push of the branch was before Stefan’s latest patch (Oct 31?), so I’ll pull that in too, unless he’s done some more work since.
My focus has been on the dumped.elc load performance; the content generation has been pretty much all his work.
Stefan’s work on reducing “#n#” placeholder substitutions tackled the particular case of cons cells, and he suggested extending it to other types, but based on some stats I got looking over the big .elc file, I think cons cells and strings are the majority of the cases. Stefan’s code addresses the cons cells, and for property-less strings (and certain other types) the substitution pass isn’t needed, so I think we’ve addressed the biggest gains to be had in that area.
The patch I made to use a pair of hash tables for tracking previously read objects seems to have improved performance as well, but I’ve started to refine it a bit to not create a new pair of hash tables for every top-level read done, when the majority of the calls (over 95% during a loadup.el pass I instrumented) don’t actually wind up needing the hash tables.
Regardless of the outcome of this current disagreement and whether big-elc gets used, I think the lread changes may soon be ready to merge, even if the performance benefits are less drastic for normal Lisp code.
As for non-performance issues, last time I tried the big-elc code, it didn’t work for me in batch mode because of a null pointer dereference in face processing; I haven’t had a chance to look into it further but maybe in the next few days I can.
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 4:28 ` Ken Raeburn
@ 2016-12-02 4:41 ` Daniel Colascione
2016-12-02 8:08 ` Eli Zaretskii
2016-12-02 8:03 ` Eli Zaretskii
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-12-02 4:41 UTC (permalink / raw)
To: Ken Raeburn, Eli Zaretskii; +Cc: jwiegley, rms, emacs-devel
On 12/01/2016 08:28 PM, Ken Raeburn wrote:
>
>> On Dec 1, 2016, at 13:11, Eli Zaretskii <eliz@gnu.org> wrote:
>>
>>> From: Richard Stallman <rms@gnu.org>
>>> CC: eliz@gnu.org, jwiegley@gmail.com, burton.samograd@autodesk.com,
>>> emacs-devel@gnu.org
>>> Date: Thu, 01 Dec 2016 04:18:04 -0500
>>>
>>> Rather than arguing theoretically about speculations,
>>> let's see people do their best to improve the big-elc-file
>>> approach, and then compare facts with facts.
>>
>> Agreed.
>>
>> Ken, will you be able to continue working on your branch in this
>> direction? Are all the changes suggested/tried by Stefan already on
>> that branch? If not, would it be possible for Stefan or yourself to
>> update the branch?
>
> I have an update I’m still working on, to refine the hash table handling I did to generate less garbage while reading normal .el or .elc files. I hope to get that uploaded soon. And I think my previous push of the branch was before Stefan’s latest patch (Oct 31?), so I’ll pull that in too, unless he’s done some more work since.
>
> My focus has been on the dumped.elc load performance; the content generation has been pretty much all his work.
>
> Stefan’s work on reducing “#n#” placeholder substitutions tackled the particular case of cons cells, and he suggested extending it to other types, but based on some stats I got looking over the big .elc file, I think cons cells and strings are the majority of the cases. Stefan’s code addresses the cons cells, and for property-less strings (and certain other types) the substitution pass isn’t needed, so I think we’ve addressed the biggest gains to be had in that area.
>
> The patch I made to use a pair of hash tables for tracking previously read objects seems to have improved performance as well, but I’ve started to refine it a bit to not create a new pair of hash tables for every top-level read done, when the majority of the calls (over 95% during a loadup.el pass I instrumented) don’t actually wind up needing the hash tables.
>
Of course all of this work isn't in C and doesn't require specialized
skills, right?
> Regardless of the outcome of this current disagreement and whether
big-elc gets used, I think the lread changes may soon be ready to merge,
even if the performance benefits are less drastic for normal Lisp code.
Yes, elc loading performance is welcome regardless.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 2:18 ` Stefan Monnier
@ 2016-12-02 7:54 ` Eli Zaretskii
2016-12-02 8:08 ` John Wiegley
` (2 more replies)
0 siblings, 3 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-02 7:54 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 01 Dec 2016 21:18:04 -0500
>
> > It's a new and significant mechanism, which is central to building and
> > starting Emacs. It needs to be studied, understood, and maintained.
>
> FWIW, I looked at Daniel's code and found it understandable, whereas
> I never managed to make any sense of the unexec code, which still just looks
> like magical voodoo to me, despite my trying twice to understand it.
I understand both (though, admittedly, that excludes unexelf.c, which
I never needed to look at for more than 5 sec). But this is not about
you or me. Do you really believe that I'm afraid of the dumper code,
having tweaked the display engine to speak RTL? However, this is
about the next generation of Emacs maintainers who will inherit this
code and will have to maintain it.
Granted, the proposed dumper is not very complicated. But it isn't
trivial either. So if we can achieve a similar effect by using the
"normal" loadup code, which is much simpler and doesn't really require
understanding anything new, I think it's more beneficial for the
project's future.
The key issue is performance, as was stated many times. If the
performance of loadup after all the speedups is not good enough, then
we will have to conclude that this idea cannot fly, at least not yet.
(That conclusion might be revisited later, if/when disks, filesystems,
and machines become faster.) It will also not fly if no one will pick
up the gauntlet and work on that idea to get it to completion.
But IMO we should at least try implementing that idea, because its
benefits are so clear. We shouldn't give up without trying, and we
have no special reason to decide the idea isn't viable until we've
tried.
Once again, if those ideas seem strange, let alone incorrect, to
you-all out there, just say the word, and I will step down. Then I
can stop worrying about the portable dumper, and you can stop worrying
about my strange ideas. Emacs is not my private project; I'm only
entitled to promote my ideas if the others either support them or
trust me to DTRT. Please decide which one is it, and let's move on.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 4:28 ` Ken Raeburn
2016-12-02 4:41 ` Daniel Colascione
@ 2016-12-02 8:03 ` Eli Zaretskii
2016-12-02 17:24 ` Ken Raeburn
1 sibling, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-02 8:03 UTC (permalink / raw)
To: Ken Raeburn; +Cc: jwiegley, dancol, rms, emacs-devel
> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 1 Dec 2016 23:28:27 -0500
> Cc: rms@gnu.org,
> dancol@dancol.org,
> jwiegley@gmail.com,
> emacs-devel@gnu.org
>
> > Ken, will you be able to continue working on your branch in this
> > direction? Are all the changes suggested/tried by Stefan already on
> > that branch? If not, would it be possible for Stefan or yourself to
> > update the branch?
>
> I have an update I’m still working on, to refine the hash table handling I did to generate less garbage while reading normal .el or .elc files. I hope to get that uploaded soon. And I think my previous push of the branch was before Stefan’s latest patch (Oct 31?), so I’ll pull that in too, unless he’s done some more work since.
Thanks.
> Regardless of the outcome of this current disagreement and whether big-elc gets used, I think the lread changes may soon be ready to merge, even if the performance benefits are less drastic for normal Lisp code.
I agree in general, but let's discuss this again when you think the
branch is ready. I'd like to have the branch reviewed by those who
are interested (I will certainly do that myself) before we merge.
Does the branch include any benchmarks, to compare performance with
the current code, both for dumped.elc and for normal Lisp code? If
not, can they be included/added, perhaps under test/?
> As for non-performance issues, last time I tried the big-elc code, it didn’t work for me in batch mode because of a null pointer dereference in face processing; I haven’t had a chance to look into it further but maybe in the next few days I can.
Let me know if you need help in that department.
Thanks again for working on this.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 4:41 ` Daniel Colascione
@ 2016-12-02 8:08 ` Eli Zaretskii
0 siblings, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-02 8:08 UTC (permalink / raw)
To: Daniel Colascione; +Cc: jwiegley, raeburn, rms, emacs-devel
> Cc: rms@gnu.org, jwiegley@gmail.com, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 1 Dec 2016 20:41:47 -0800
>
> Of course all of this work isn't in C and doesn't require specialized
> skills, right?
It does. We also make additions in C in other areas.
The differences between this and other C changes, and your proposed
dumper, as was already written several times, are:
. there's no Lisp-level alternative for them
. these changes are mostly a one-time job, and the probability of
them to be actively maintained is very low, if not zero
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 7:54 ` Eli Zaretskii
@ 2016-12-02 8:08 ` John Wiegley
2016-12-02 8:59 ` Eli Zaretskii
2016-12-02 9:00 ` Philippe Vaucher
2016-12-02 13:04 ` Stefan Monnier
2 siblings, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-12-02 8:08 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Stefan Monnier, emacs-devel
[-- Attachment #1: Type: text/plain, Size: 2170 bytes --]
>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
EZ> The key issue is performance, as was stated many times. If the performance
EZ> of loadup after all the speedups is not good enough, then we will have to
EZ> conclude that this idea cannot fly, at least not yet. (That conclusion
EZ> might be revisited later, if/when disks, filesystems, and machines become
EZ> faster.) It will also not fly if no one will pick up the gauntlet and work
EZ> on that idea to get it to completion.
I think Daniel has helped to stimulate us by offering a close-to-working
solution to an ancient problem. The code it replaces is worse; while the
as-yet-unproven idea of a fast .elc loader is better. It's now upon us to
prove that we can beat the portable dumper, by making good on a fast .elc
loader.
That said, I don't want to let Daniel's idea wither on the vine while we prove
or disprove a fast .elc loader. Sure, we can wait a couple of months to give
that idea a chance while we review the dumper, but if we don't get an .elc
loader in that time, we should give Daniel's idea a go, if for no other reason
than that the work is almost done, and we could gain some valuable experience
-- and a way to do some real performance comparisons.
If the .elc idea pans out (and I really hope it can), I think even Daniel will
be relieved at not having to maintain his dumper, and he can help us to
extricate his helpful addition.
EZ> Once again, if those ideas seem strange, let alone incorrect, to you-all
EZ> out there, just say the word, and I will step down. Then I can stop
EZ> worrying about the portable dumper, and you can stop worrying about my
EZ> strange ideas. Emacs is not my private project; I'm only entitled to
EZ> promote my ideas if the others either support them or trust me to DTRT.
EZ> Please decide which one is it, and let's move on.
Please do not step down, Eli. Please don't. It sickens my heart every time I
hear you suggest it, so just stop doing that for my sake. :)
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 9:03 ` Matt Armstrong
@ 2016-12-02 8:10 ` John Wiegley
0 siblings, 0 replies; 299+ messages in thread
From: John Wiegley @ 2016-12-02 8:10 UTC (permalink / raw)
To: Matt Armstrong; +Cc: Eli Zaretskii, Daniel Colascione, emacs-devel, rms, eggert
>>>>> "MA" == Matt Armstrong <marmstrong@google.com> writes:
MA> I'm not an Emacs maintainer, and can't speak as one. But, I play a similar
MA> role at work, and can tell you that the technical issues at play in these
MA> conflicts you have with Emacs maintainers are secondary. At their core,
MA> these discussions are all about trust. You don't fully trust the
MA> maintainers, and they don't fully trust you (hopefully "yet" on both
MA> accounts).
This was a really wonderful post, Matt. I hope it's OK if I share it with some
friends who also maintain large projects.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 8:08 ` John Wiegley
@ 2016-12-02 8:59 ` Eli Zaretskii
2016-12-02 19:39 ` John Wiegley
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-02 8:59 UTC (permalink / raw)
To: John Wiegley; +Cc: monnier, emacs-devel
> From: John Wiegley <jwiegley@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> Date: Fri, 02 Dec 2016 00:08:55 -0800
>
> EZ> Once again, if those ideas seem strange, let alone incorrect, to you-all
> EZ> out there, just say the word, and I will step down. Then I can stop
> EZ> worrying about the portable dumper, and you can stop worrying about my
> EZ> strange ideas. Emacs is not my private project; I'm only entitled to
> EZ> promote my ideas if the others either support them or trust me to DTRT.
> EZ> Please decide which one is it, and let's move on.
>
> Please do not step down, Eli. Please don't. It sickens my heart every time I
> hear you suggest it, so just stop doing that for my sake. :)
I hear you. But with all due respect, you cannot have it both ways:
beg me to stay in charge, and then push me to give up one of the few
firm ideas I have about where Emacs should go and how. It isn't fair.
I arrived at those ideas by carefully watching Emacs development for
many years. What kind of maintainer will I be if I cannot base my
judgment and decisions on a few principal ideas I have about Emacs
development? What could replace those ideas to be the base for
decisions I need to make almost every day?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 7:54 ` Eli Zaretskii
2016-12-02 8:08 ` John Wiegley
@ 2016-12-02 9:00 ` Philippe Vaucher
2016-12-02 10:56 ` Eli Zaretskii
2016-12-02 13:04 ` Stefan Monnier
2 siblings, 1 reply; 299+ messages in thread
From: Philippe Vaucher @ 2016-12-02 9:00 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Stefan Monnier, Emacs developers
[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]
>
> Once again, if those ideas seem strange, let alone incorrect, to
> you-all out there, just say the word, and I will step down. Then I
> can stop worrying about the portable dumper, and you can stop worrying
> about my strange ideas. Emacs is not my private project; I'm only
> entitled to promote my ideas if the others either support them or
> trust me to DTRT. Please decide which one is it, and let's move on.
>
Eli, I think you are doing a fantastic job.
The only problem that seems to happen from time to time is that the
discussions starts getting personal and ends up with people saying threats
like "if you put my code in a branch I will fork Emacs" or "if you don't
agree with my ideas I will step down", which is just a way of saying "my
way or the highway". Using these rethorics is not constructive and doesn't
do anyone any good.
To me it looks like the only real problem here is communication, we should
all argue less and try to put ourselves in the other's schoes.
To Daniel, it's important that his work does not bitrot somewhere like the
concurrency branch did. It's also important that he feels there's a strong
chance of his code getting merged before he works more on it.
To Eli, it's important to have Daniel's code more "production ready"
(documentation, etc), and also to give the simpler implementation ideas a
try to avoid having to switch/maintain different systems twice in a short
period of time.
Both needs are understandable and valid. Nobody is wrong here.
To me it's clear that given the requirements, we can plan something that
would satisfy both... for example.
<Eli> Daniel, we'll give other people one month to come up with a sketch of
the alternative idea to see if it's feasible or not. If nothing comes up
after one month, we'll start working on merging your branch.
My 0.02$
Philippe
[-- Attachment #2: Type: text/html, Size: 2399 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 9:00 ` Philippe Vaucher
@ 2016-12-02 10:56 ` Eli Zaretskii
2017-05-26 19:48 ` Thien-Thi Nguyen
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-02 10:56 UTC (permalink / raw)
To: Philippe Vaucher; +Cc: monnier, emacs-devel
> From: Philippe Vaucher <philippe.vaucher@gmail.com>
> Date: Fri, 2 Dec 2016 10:00:28 +0100
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Emacs developers <emacs-devel@gnu.org>
>
> Once again, if those ideas seem strange, let alone incorrect, to
> you-all out there, just say the word, and I will step down. Then I
> can stop worrying about the portable dumper, and you can stop worrying
> about my strange ideas. Emacs is not my private project; I'm only
> entitled to promote my ideas if the others either support them or
> trust me to DTRT. Please decide which one is it, and let's move on.
>
> Eli, I think you are doing a fantastic job.
>
> The only problem that seems to happen from time to time is that the discussions starts getting personal and
> ends up with people saying threats like "if you put my code in a branch I will fork Emacs" or "if you don't agree
> with my ideas I will step down", which is just a way of saying "my way or the highway". Using these rethorics
> is not constructive and doesn't do anyone any good.
What do you expect me to do when I see code which takes Emacs in a
direction I think is wrong? There are a very few of such directions I
feel strongly we shouldn't take. What am I supposed to do when I see
one of them proposed for inclusion?
More generally, is the maintainer allowed to reject some code or
course of actions? If yes, what are the criteria for that?
> To Daniel, it's important that his work does not bitrot somewhere like the concurrency branch did. It's also
> important that he feels there's a strong chance of his code getting merged before he works more on it.
> To Eli, it's important to have Daniel's code more "production ready" (documentation, etc), and also to give the
> simpler implementation ideas a try to avoid having to switch/maintain different systems twice in a short period
> of time.
No, you've misunderstood the nature of our disagreement. There's no
disagreement about the code being production-ready: Daniel himself
said it isn't yet.
> <Eli> Daniel, we'll give other people one month to come up with a sketch of the alternative idea to see if it's
> feasible or not. If nothing comes up after one month, we'll start working on merging your branch.
That's what I said, with the exception of the "month" part.
If we are talking about better communications, then I can suggest
discussing an idea before it is coded. As a matter of fact, I
proposed a very similar idea 2 months ago, and it was voted down by
Paul, so I didn't pursue it, also because that discussion brought up
the "one large .elc file" idea, which Paul liked better (and so did
I), the only issue with it being the performance that we could get.
If Daniel have described his idea before coding it, I think at least
some of the aggravation (mine included) could have been spared.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 7:54 ` Eli Zaretskii
2016-12-02 8:08 ` John Wiegley
2016-12-02 9:00 ` Philippe Vaucher
@ 2016-12-02 13:04 ` Stefan Monnier
2016-12-02 14:45 ` Eli Zaretskii
2 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-02 13:04 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
> Granted, the proposed dumper is not very complicated. But it isn't
> trivial either. So if we can achieve a similar effect by using the
> "normal" loadup code, which is much simpler and doesn't really require
> understanding anything new, I think it's more beneficial for the
> project's future.
Ken worked on speeding up the lread.c code, and it got to be
significantly faster, but not fast enough. AFAIK it's got to the point
where it's not clear exactly how to speed it up further. Not that it
can't be done, but that it's not obvious how, so it's likely going to
require some serious rethinking and maybe restructuring/rewrite of
the code.
Is it going to happen if we don't merge the pdumper? I'm not so sure.
The main impetus behind speeding up lread.c is to replace unexec.c, so
I agree with you that merging the pdumper might mean that speeding up
lread.c will simply never happen. But I think there's also a very
serious risk that even without the pdumper, speeding up lread.c will
still never happen: I have no intention on working at speeding up
lread.c, AFAICT Ken also gave up on it, anyone else?
Personally, I think that maybe we should move in the other direction:
keep lread.c for "source code" and generalize the pdumper code so it can
also be used for the .elc files.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-01 18:09 ` Eli Zaretskii
2016-12-02 2:18 ` Stefan Monnier
@ 2016-12-02 14:27 ` Richard Stallman
1 sibling, 0 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-02 14:27 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: dancol, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> However, this is not about making existing code simpler. It's about
> _adding_ more C code, where an alternative implementation might allow
> doing that without such additions.
Yes, of course -- we all know that.
> It's a new and significant mechanism, which is central to building and
> starting Emacs. It needs to be studied, understood, and maintained.
> Therefore, it does increase the complexity.
It certainly increases the amount of code that understands Lisp object
formats. However, whether it increases the degree of complexity is
another question. I don't think it is a problem to have one more place
with the same kind and depth of complexity. That means _more_ work
on some future rare occasions, but it does not mean _harder_ work.
As long as we have people to maintain allocation and GC, I don't
think we would have trouble finding people to maintain the portable
dumper along with them.
But it's true that I have not seen the code. Someone should
look at the code and judge whether this is really true.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 13:04 ` Stefan Monnier
@ 2016-12-02 14:45 ` Eli Zaretskii
2016-12-02 14:51 ` Stefan Monnier
` (4 more replies)
0 siblings, 5 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-02 14:45 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 02 Dec 2016 08:04:46 -0500
>
> > Granted, the proposed dumper is not very complicated. But it isn't
> > trivial either. So if we can achieve a similar effect by using the
> > "normal" loadup code, which is much simpler and doesn't really require
> > understanding anything new, I think it's more beneficial for the
> > project's future.
>
> Ken worked on speeding up the lread.c code, and it got to be
> significantly faster, but not fast enough. AFAIK it's got to the point
> where it's not clear exactly how to speed it up further. Not that it
> can't be done, but that it's not obvious how, so it's likely going to
> require some serious rethinking and maybe restructuring/rewrite of
> the code.
>
> Is it going to happen if we don't merge the pdumper? I'm not so sure.
I'm willing to give that a chance. I don't see any reason to make the
decision today.
> The main impetus behind speeding up lread.c is to replace unexec.c, so
> I agree with you that merging the pdumper might mean that speeding up
> lread.c will simply never happen. But I think there's also a very
> serious risk that even without the pdumper, speeding up lread.c will
> still never happen: I have no intention on working at speeding up
> lread.c, AFAICT Ken also gave up on it, anyone else?
Judging by Ken's response, he didn't give up yet.
In any case, it should be clear to anyone that code which isn't
written cannot be used. So a reality check will get us straight.
> Personally, I think that maybe we should move in the other direction:
> keep lread.c for "source code" and generalize the pdumper code so it can
> also be used for the .elc files.
You mean, move the byte compiler to C?
If not, I don't understand the idea: pdump wants a file in a very
specific format, because it doesn't really understand the entities it
reads.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 14:45 ` Eli Zaretskii
@ 2016-12-02 14:51 ` Stefan Monnier
2016-12-02 22:24 ` Richard Stallman
2016-12-02 23:42 ` Paul Eggert
2016-12-02 15:38 ` Daniel Colascione
` (3 subsequent siblings)
4 siblings, 2 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-02 14:51 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
> You mean, move the byte compiler to C?
No. I mean that bytecomp.el would generate a file in a format which
pdumper can use (which has various consequences, such as the fact that
.elc files aren't portable any more).
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 14:45 ` Eli Zaretskii
2016-12-02 14:51 ` Stefan Monnier
@ 2016-12-02 15:38 ` Daniel Colascione
2016-12-02 17:26 ` Ken Raeburn
` (2 subsequent siblings)
4 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-02 15:38 UTC (permalink / raw)
To: Eli Zaretskii, Stefan Monnier; +Cc: emacs-devel
On 12/02/2016 06:45 AM, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 02 Dec 2016 08:04:46 -0500
>>
>>> Granted, the proposed dumper is not very complicated. But it isn't
>>> trivial either. So if we can achieve a similar effect by using the
>>> "normal" loadup code, which is much simpler and doesn't really require
>>> understanding anything new, I think it's more beneficial for the
>>> project's future.
>>
>> Ken worked on speeding up the lread.c code, and it got to be
>> significantly faster, but not fast enough. AFAIK it's got to the point
>> where it's not clear exactly how to speed it up further. Not that it
>> can't be done, but that it's not obvious how, so it's likely going to
>> require some serious rethinking and maybe restructuring/rewrite of
>> the code.
>>
>> Is it going to happen if we don't merge the pdumper? I'm not so sure.
>
> I'm willing to give that a chance. I don't see any reason to make the
> decision today.
>
>> The main impetus behind speeding up lread.c is to replace unexec.c, so
>> I agree with you that merging the pdumper might mean that speeding up
>> lread.c will simply never happen. But I think there's also a very
>> serious risk that even without the pdumper, speeding up lread.c will
>> still never happen: I have no intention on working at speeding up
>> lread.c, AFAICT Ken also gave up on it, anyone else?
>
> Judging by Ken's response, he didn't give up yet.
>
> In any case, it should be clear to anyone that code which isn't
> written cannot be used. So a reality check will get us straight.
>
>> Personally, I think that maybe we should move in the other direction:
>> keep lread.c for "source code" and generalize the pdumper code so it can
>> also be used for the .elc files.
>
> You mean, move the byte compiler to C?
Ideally, yes: one way, we'll compile with libgccjit or something. But
that's a long way away.
For now, nobody's talking about the byte compiler. Stefan's talking
about the code that *reads* the byte compiler's output. Right now, this
code is the normal lisp reader, but it doesn't have to be.
> If not, I don't understand the idea: pdump wants a file in a very
> specific format, because it doesn't really understand the entities it
> reads.
pdumper's loader is stupidly simple. It doesn't understand anything
about what it's loading. That's what makes it fast. Stefan's idea is to
make the elc format closer to the pdump format. Advantage: it'll be much
faster that way. Disadvantage: elc files will be closely coupled to the
precise Emacs builds that generated them.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 8:03 ` Eli Zaretskii
@ 2016-12-02 17:24 ` Ken Raeburn
0 siblings, 0 replies; 299+ messages in thread
From: Ken Raeburn @ 2016-12-02 17:24 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: jwiegley, dancol, Richard Stallman, emacs-devel
On Dec 2, 2016, at 03:03, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Ken Raeburn <raeburn@raeburn.org>
>>
>> Regardless of the outcome of this current disagreement and whether big-elc gets used, I think the lread changes may soon be ready to merge, even if the performance benefits are less drastic for normal Lisp code.
>
> I agree in general, but let's discuss this again when you think the
> branch is ready. I'd like to have the branch reviewed by those who
> are interested (I will certainly do that myself) before we merge.
Certainly. Some of that code is a bit subtle, and I’d appreciate the review.
> Does the branch include any benchmarks, to compare performance with
> the current code, both for dumped.elc and for normal Lisp code? If
> not, can they be included/added, perhaps under test/?
Not currently. My benchmark has usually been along the lines of:
time ./emacs -nw —eval '(progn (sit-for 0.01) (kill-emacs))’
to initialize everything (including tty display and input checking) and then quit, in a consistent fashion from run to run. After running it several times and hopefully seeing fairly consistent numbers after the first invocation or two, I estimate the average by eye. So, yeah, it could be more precise.
Since we’re timing Emacs startup itself, we can’t test it with Lisp helper functions invoked once Emacs has started like we would with many other things. And it’ll be dependent on system issues, like whether emacs and dumped.elc are in the kernel’s page cache or not. For simplicity I’ve been testing with the cache warmed up, because it eliminates the storage system hardware and OS read-ahead algorithms and such while I focus on the lread.c code, but in the real world those will sometimes be factors too, and should be considered.
I’ll think about scripting this in some way that may be more robust, and maybe adding hooks for OS-specific cache flushing and such.
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 14:45 ` Eli Zaretskii
2016-12-02 14:51 ` Stefan Monnier
2016-12-02 15:38 ` Daniel Colascione
@ 2016-12-02 17:26 ` Ken Raeburn
2016-12-02 17:47 ` Paul Eggert
[not found] ` <<jwvlgvyv10x.fsf-monnier+Inbox@gnu.org>
[not found] ` <<<jwvlgvyv10x.fsf-monnier+Inbox@gnu.org>
4 siblings, 1 reply; 299+ messages in thread
From: Ken Raeburn @ 2016-12-02 17:26 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Stefan Monnier, emacs-devel
On Dec 2, 2016, at 09:45, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 02 Dec 2016 08:04:46 -0500
>>
>>> Granted, the proposed dumper is not very complicated. But it isn't
>>> trivial either. So if we can achieve a similar effect by using the
>>> "normal" loadup code, which is much simpler and doesn't really require
>>> understanding anything new, I think it's more beneficial for the
>>> project's future.
>>
>> Ken worked on speeding up the lread.c code, and it got to be
>> significantly faster, but not fast enough. AFAIK it's got to the point
>> where it's not clear exactly how to speed it up further. Not that it
>> can't be done, but that it's not obvious how, so it's likely going to
>> require some serious rethinking and maybe restructuring/rewrite of
>> the code.
>>
>> Is it going to happen if we don't merge the pdumper? I'm not so sure.
>
> I'm willing to give that a chance. I don't see any reason to make the
> decision today.
>
>> The main impetus behind speeding up lread.c is to replace unexec.c, so
>> I agree with you that merging the pdumper might mean that speeding up
>> lread.c will simply never happen. But I think there's also a very
>> serious risk that even without the pdumper, speeding up lread.c will
>> still never happen: I have no intention on working at speeding up
>> lread.c, AFAICT Ken also gave up on it, anyone else?
>
> Judging by Ken's response, he didn't give up yet.
No, I just got busy with other things for a bit.
Stefan’s right too… since the current time sinks are the “read” functions themselves rather than in support code being called, it’s less immediately obvious where work is needed or how to proceed. And it’s certainly possible changes will be needed that aren’t as nicely localized as getc() or recursive placeholder substitution. But, I haven’t tried, yet… :-)
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 17:26 ` Ken Raeburn
@ 2016-12-02 17:47 ` Paul Eggert
0 siblings, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2016-12-02 17:47 UTC (permalink / raw)
To: Ken Raeburn, Eli Zaretskii; +Cc: Stefan Monnier, emacs-devel
Ken Raeburn wrote:
> Stefan’s right too… since the current time sinks are the “read” functions themselves rather than in support code being called, it’s less immediately obvious where work is needed or how to proceed.
Stefan is right so often that it must embarrass him. :-)
FWIW I recently tuned a private copy of lread.c to use syscalls rather than
stdio. It didn't help significantly, so I gave that up.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 8:59 ` Eli Zaretskii
@ 2016-12-02 19:39 ` John Wiegley
2016-12-02 20:11 ` Karl Fogel
0 siblings, 1 reply; 299+ messages in thread
From: John Wiegley @ 2016-12-02 19:39 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: monnier, emacs-devel
[-- Attachment #1: Type: text/plain, Size: 3434 bytes --]
>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> But with all due respect, you cannot have it both ways: beg me to stay in
> charge, and then push me to give up one of the few firm ideas I have about
> where Emacs should go and how.
I think a part of this job is always political: Balancing the twin goals of
promoting the best ideas, while also keeping everyone involved encouraged and
motivated. When we say "yes" to an idea, we might be hurting Emacs; but when
we say "no", we might be hurting that contributor, and losing their future
involvement. Sometimes that's fine -- after all, everyone else in the world
wants us to care about Emacs more than a limited-time set of contributors --
but we also cannot thrive without active contributors, so "watering the
garden" is part of our task.
We all know the portable dumper is not an ideal solution; we even have some
notion of what the ideal solution looks like. It's indeed a bit frustrating to
knowingly support a technology path that may end up incurring a lot of work,
if a much better solution might be just around the corner.
However, I ask you to consider the merits of Daniel's proposal from several
angles -- angles a regular developer may not care about at all:
- If we accept the work, we show to others that *code wins*. That is, if a
problem exists in Emacs, and someone comes to us with actual code, this has
tremendous weight. If this is our position, it encourages others to
experiment, since they'll fear less that after so much work, we might just
say "no" because it's not as good a solution as we'd like.
- If we accept the work, it encourages Daniel to play a larger role, to learn
more about the C internals, and likely to contribute even more.
- If we accept the work, and a better solution comes along later, we can
revert the change without undue labor. That is, our end cost is not so
terribly high that we're panting ourselves into a corner.
- If we accept the work, despite our disagreement, *precisely because most of
the other core developers do agree*, we're saying that their opinion holds
a lot of weight with us, and this encourages frank and open discussion. If
decisions like this are made by just you or me, against all other
objections, it diminishes the value of this mailing list in others' eyes.
It then seems like we're just making the Emacs we want, and not the Emacs
all of us want.
I think there is a "greater good" here, and it is not unduly damaged by
letting a short-term solution into the code until a longer-term solution
appears. There is much goodwill to be gained, and really not so much harm. On
the other hand, rejecting it -- despite the general sense of agreement is has
gained -- would cause a much greater social damage, which is far more
difficult to undo than one day asking Daniel to revert his work.
All that said, if you feel strongly that this proposal hurts Emacs itself, and
cannot abide it, I understand. I was hoping we could all win here, but I know
that sometimes this isn't realistic. I just hope we can all recognize that
this portable dumper idea is not so terribly important: not to fork over, nor
to resign from the amazing support you provide.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 19:39 ` John Wiegley
@ 2016-12-02 20:11 ` Karl Fogel
2016-12-02 21:22 ` Daniel Colascione
2016-12-02 22:06 ` Eli Zaretskii
0 siblings, 2 replies; 299+ messages in thread
From: Karl Fogel @ 2016-12-02 20:11 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 5682 bytes --]
Following up to what John said, with a perspective that may be useful to Eli:
Eli, I have a great deal of respect for your technical judgement and the amount of work you put into Emacs. But I have a slightly different take on what maintainership means.
When I was in a position similar to yours (one of a small group of core co-maintainers, in the Subversion project), there were a few occasions when a major technical decision went in a direction I didn't agree with. My disagreement was not of the "this will destroy the project" sort, but I did feel the decisions were poor technical choices that could be a long-term drag on the project -- creating extra maintenance work for existing developers, and creating barriers to entry for new developers. In other words, objections very similar to yours now.
In each case, discussions eventually made it clear to me that my opinion was not going to prevail. There were just too many developers who felt differently. This didn't tempt me to step down from maintainership, though I will admit that it *did* tempt me to keep score a bit, so that later on I might be able to say "Hey folks, remember when I turned out to be right about X, and Y, and Z? So please listen to me this time." I think I may even have used that privilege once or twice later :-), though I don't remember clearly if I did. What I do remember clearly is that I turned out to be wrong in at least one of the cases (at least, I established to my own satisfaction that my warnings had been unwarranted).
Your value as a maintainer is not lessened if a particular decision doesn't go the way you recommended. (Of course, if maintainership becomes unenjoyable to you because of the decision, that's a different question, and only you can make that call.) I think there is even a good argument that your continued maintainership actually becomes slightly *more* important to the project: of all the people who could spot whether the negative consequences of the decision are coming true, you would perhaps be the person most likely to spot it, and most likely to point it out to the other developers.
You must make your own decision, of course. But I hope this perspective is a useful response to your question:
"...What kind of maintainer will I be if I cannot base my judgment and
decisions on a few principal ideas I have about Emacs development?
What could replace those ideas to be the base for decisions I need to
make almost every day?"
(I pretty much agree with everything John said in his email below, too, for what it's worth.)
Best regards,
-Karl
John Wiegley <jwiegley@gmail.com> writes:
>I think a part of this job is always political: Balancing the twin goals of
>promoting the best ideas, while also keeping everyone involved encouraged and
>motivated. When we say "yes" to an idea, we might be hurting Emacs; but when
>we say "no", we might be hurting that contributor, and losing their future
>involvement. Sometimes that's fine -- after all, everyone else in the world
>wants us to care about Emacs more than a limited-time set of contributors --
>but we also cannot thrive without active contributors, so "watering the
>garden" is part of our task.
>
>We all know the portable dumper is not an ideal solution; we even have some
>notion of what the ideal solution looks like. It's indeed a bit frustrating to
>knowingly support a technology path that may end up incurring a lot of work,
>if a much better solution might be just around the corner.
>
>However, I ask you to consider the merits of Daniel's proposal from several
>angles -- angles a regular developer may not care about at all:
>
> - If we accept the work, we show to others that *code wins*. That is, if a
> problem exists in Emacs, and someone comes to us with actual code, this has
> tremendous weight. If this is our position, it encourages others to
> experiment, since they'll fear less that after so much work, we might just
> say "no" because it's not as good a solution as we'd like.
>
> - If we accept the work, it encourages Daniel to play a larger role, to learn
> more about the C internals, and likely to contribute even more.
>
> - If we accept the work, and a better solution comes along later, we can
> revert the change without undue labor. That is, our end cost is not so
> terribly high that we're panting ourselves into a corner.
>
> - If we accept the work, despite our disagreement, *precisely because most of
> the other core developers do agree*, we're saying that their opinion holds
> a lot of weight with us, and this encourages frank and open discussion. If
> decisions like this are made by just you or me, against all other
> objections, it diminishes the value of this mailing list in others' eyes.
> It then seems like we're just making the Emacs we want, and not the Emacs
> all of us want.
>
>I think there is a "greater good" here, and it is not unduly damaged by
>letting a short-term solution into the code until a longer-term solution
>appears. There is much goodwill to be gained, and really not so much harm. On
>the other hand, rejecting it -- despite the general sense of agreement is has
>gained -- would cause a much greater social damage, which is far more
>difficult to undo than one day asking Daniel to revert his work.
>
>All that said, if you feel strongly that this proposal hurts Emacs itself, and
>cannot abide it, I understand. I was hoping we could all win here, but I know
>that sometimes this isn't realistic. I just hope we can all recognize that
>this portable dumper idea is not so terribly important: not to fork over, nor
>to resign from the amazing support you provide.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 20:11 ` Karl Fogel
@ 2016-12-02 21:22 ` Daniel Colascione
2016-12-02 22:06 ` Eli Zaretskii
` (2 more replies)
2016-12-02 22:06 ` Eli Zaretskii
1 sibling, 3 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-02 21:22 UTC (permalink / raw)
To: Karl Fogel; +Cc: emacs-devel
FWIW, I'll put my patch on a branch this weekend.
On Fri, Dec 02 2016, Karl Fogel wrote:
> Following up to what John said, with a perspective that may be useful to Eli:
>
> Eli, I have a great deal of respect for your technical judgement and
> the amount of work you put into Emacs. But I have a slightly
> different take on what maintainership means.
>
> When I was in a position similar to yours (one of a small group of
> core co-maintainers, in the Subversion project), there were a few
> occasions when a major technical decision went in a direction I didn't
> agree with. My disagreement was not of the "this will destroy the
> project" sort, but I did feel the decisions were poor technical
> choices that could be a long-term drag on the project -- creating
> extra maintenance work for existing developers, and creating barriers
> to entry for new developers. In other words, objections very similar
> to yours now.
>
> In each case, discussions eventually made it clear to me that my
> opinion was not going to prevail. There were just too many developers
> who felt differently. This didn't tempt me to step down from
> maintainership, though I will admit that it *did* tempt me to keep
> score a bit, so that later on I might be able to say "Hey folks,
> remember when I turned out to be right about X, and Y, and Z?
> So please listen to me this time." I think I may even have used that
> privilege once or twice later :-), though I don't remember clearly if
> I did. What I do remember clearly is that I turned out to be wrong in
> at least one of the cases (at least, I established to my own
> satisfaction that my warnings had been unwarranted).
>
> Your value as a maintainer is not lessened if a particular decision
> doesn't go the way you recommended. (Of course, if maintainership
> becomes unenjoyable to you because of the decision, that's a different
> question, and only you can make that call.) I think there is even a
> good argument that your continued maintainership actually becomes
> slightly *more* important to the project: of all the people who could
> spot whether the negative consequences of the decision are coming
> true, you would perhaps be the person most likely to spot it, and most
> likely to point it out to the other developers.
>
> You must make your own decision, of course. But I hope this
> perspective is a useful response to your question:
>
> "...What kind of maintainer will I be if I cannot base my judgment and
> decisions on a few principal ideas I have about Emacs development?
> What could replace those ideas to be the base for decisions I need to
> make almost every day?"
>
> (I pretty much agree with everything John said in his email below,
> too, for what it's worth.)
>
> Best regards,
> -Karl
>
> John Wiegley <jwiegley@gmail.com> writes:
>>I think a part of this job is always political: Balancing the twin goals of
>>promoting the best ideas, while also keeping everyone involved encouraged and
>>motivated. When we say "yes" to an idea, we might be hurting Emacs; but when
>>we say "no", we might be hurting that contributor, and losing their future
>>involvement. Sometimes that's fine -- after all, everyone else in the world
>>wants us to care about Emacs more than a limited-time set of contributors --
>>but we also cannot thrive without active contributors, so "watering the
>>garden" is part of our task.
>>
>>We all know the portable dumper is not an ideal solution; we even have some
>>notion of what the ideal solution looks like. It's indeed a bit frustrating to
>>knowingly support a technology path that may end up incurring a lot of work,
>>if a much better solution might be just around the corner.
>>
>>However, I ask you to consider the merits of Daniel's proposal from several
>>angles -- angles a regular developer may not care about at all:
>>
>> - If we accept the work, we show to others that *code wins*. That is, if a
>> problem exists in Emacs, and someone comes to us with actual code, this has
>> tremendous weight. If this is our position, it encourages others to
>> experiment, since they'll fear less that after so much work, we might just
>> say "no" because it's not as good a solution as we'd like.
>>
>> - If we accept the work, it encourages Daniel to play a larger role, to learn
>> more about the C internals, and likely to contribute even more.
>>
>> - If we accept the work, and a better solution comes along later, we can
>> revert the change without undue labor. That is, our end cost is not so
>> terribly high that we're panting ourselves into a corner.
>>
>> - If we accept the work, despite our disagreement, *precisely because most of
>> the other core developers do agree*, we're saying that their opinion holds
>> a lot of weight with us, and this encourages frank and open discussion. If
>> decisions like this are made by just you or me, against all other
>> objections, it diminishes the value of this mailing list in others' eyes.
>> It then seems like we're just making the Emacs we want, and not the Emacs
>> all of us want.
>>
>>I think there is a "greater good" here, and it is not unduly damaged by
>>letting a short-term solution into the code until a longer-term solution
>>appears. There is much goodwill to be gained, and really not so much harm. On
>>the other hand, rejecting it -- despite the general sense of agreement is has
>>gained -- would cause a much greater social damage, which is far more
>>difficult to undo than one day asking Daniel to revert his work.
>>
>>All that said, if you feel strongly that this proposal hurts Emacs itself, and
>>cannot abide it, I understand. I was hoping we could all win here, but I know
>>that sometimes this isn't realistic. I just hope we can all recognize that
>>this portable dumper idea is not so terribly important: not to fork over, nor
>>to resign from the amazing support you provide.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 20:11 ` Karl Fogel
2016-12-02 21:22 ` Daniel Colascione
@ 2016-12-02 22:06 ` Eli Zaretskii
2016-12-02 22:28 ` Daniel Colascione
` (2 more replies)
1 sibling, 3 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-02 22:06 UTC (permalink / raw)
To: Karl Fogel; +Cc: emacs-devel
> From: Karl Fogel <kfogel@red-bean.com>
> Date: Fri, 02 Dec 2016 14:11:23 -0600
>
> Following up to what John said, with a perspective that may be useful to Eli:
Thanks, but it isn't useful. Why? because, like John, you don't think
that my opinion about the preferred solution to getting rid of unexec
is right.
What would be useful is a perspective from someone who understands the
issues and agrees with me about the preferred solution. Because only
such a person can see the dilemma through my eyes, and only that
person can find the right words.
> When I was in a position similar to yours (one of a small group of core co-maintainers, in the Subversion project), there were a few occasions when a major technical decision went in a direction I didn't agree with. My disagreement was not of the "this will destroy the project" sort, but I did feel the decisions were poor technical choices that could be a long-term drag on the project -- creating extra maintenance work for existing developers, and creating barriers to entry for new developers. In other words, objections very similar to yours now.
No, they aren't similar. In cases like the one you describe, I
invariably seek a compromise or give in. It's what you call "destroy
the project" thing that makes the difference.
> Your value as a maintainer is not lessened if a particular decision doesn't go the way you recommended. (Of course, if maintainership becomes unenjoyable to you because of the decision, that's a different question, and only you can make that call.)
"Unenjoyable"? It's intolerable. It would mean that on my watch
something happened that I believe is detrimental to a project which I
took upon myself to improve, and I saw it happening and didn't prevent
it. Can you understand my plight?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 21:22 ` Daniel Colascione
@ 2016-12-02 22:06 ` Eli Zaretskii
2016-12-02 23:15 ` Karl Fogel
2016-12-15 14:28 ` Philippe Vaucher
2 siblings, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-02 22:06 UTC (permalink / raw)
To: Daniel Colascione; +Cc: kfogel, emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 02 Dec 2016 13:22:56 -0800
> Cc: emacs-devel@gnu.org
>
> FWIW, I'll put my patch on a branch this weekend.
Thank you.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 14:51 ` Stefan Monnier
@ 2016-12-02 22:24 ` Richard Stallman
2016-12-02 23:32 ` Stefan Monnier
2016-12-03 8:28 ` Eli Zaretskii
2016-12-02 23:42 ` Paul Eggert
1 sibling, 2 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-02 22:24 UTC (permalink / raw)
To: Stefan Monnier; +Cc: eliz, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> No. I mean that bytecomp.el would generate a file in a format which
> pdumper can use (which has various consequences, such as the fact that
> .elc files aren't portable any more).
That seems like a big disadvantage to me. This could be sufficient
reason just by itself to stick with the current format for elc files.
People expect to install one elc file and have it work in multple
Emacs versions. Only rarely have we made changes that would prevent
this.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 22:06 ` Eli Zaretskii
@ 2016-12-02 22:28 ` Daniel Colascione
2016-12-03 8:48 ` Eli Zaretskii
2016-12-02 22:29 ` John Wiegley
2016-12-03 21:28 ` Richard Stallman
2 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-12-02 22:28 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Karl Fogel, emacs-devel
On Sat, Dec 03 2016, Eli Zaretskii wrote:
>> From: Karl Fogel <kfogel@red-bean.com>
>> Date: Fri, 02 Dec 2016 14:11:23 -0600
>>
>> Following up to what John said, with a perspective that may be useful to Eli:
>
> Thanks, but it isn't useful. Why? because, like John, you don't think
> that my opinion about the preferred solution to getting rid of unexec
> is right.
>
> What would be useful is a perspective from someone who understands the
> issues and agrees with me about the preferred solution. Because only
> such a person can see the dilemma through my eyes, and only that
> person can find the right words.
We've all been contributing to Emacs for many years. I think we have a
pretty good grasp of the issues and potential problems. I understand
that you think there's an existential risk that goes into this change,
but I think you're mistaken, and a lot of other people agree. We all
have access to the same information; it's unlikely (but not impossible)
that you're the only one who can see certain potential problems. Please
give this change the benefit of the doubt.
Even if your worst-case scenario comes to pass, the worst outcome
outcome will just be more fiddly maintenance. I think the pdumper code
is pretty clean; I'm open to suggestions for making it moreso. The code
doesn't depend on arcane details of underlying operating systems --- it
depends only on details of Emacs itself. We don't need ELF experts.
If there's no one left who can understand pdumper, there's no one left
who can understand the GC either. In that scenario, we're already in
Mad Max territory anyway.
You won't have to touch this code.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 22:06 ` Eli Zaretskii
2016-12-02 22:28 ` Daniel Colascione
@ 2016-12-02 22:29 ` John Wiegley
2016-12-03 21:28 ` Richard Stallman
2 siblings, 0 replies; 299+ messages in thread
From: John Wiegley @ 2016-12-02 22:29 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Karl Fogel, emacs-devel
>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
EZ> Thanks, but it isn't useful. Why? because, like John, you don't think that
EZ> my opinion about the preferred solution to getting rid of unexec is right.
Maybe I'm not understanding fully: I thought we all agreed that your preferred
solution is the best one, provided it can be implemented. In the meantime, we
have an interim measure available that is better than unexec. Since I haven't
heard anyone suggest that we should stick with the portable dumper forever, I
don't fully understand the strength of your objection.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 21:22 ` Daniel Colascione
2016-12-02 22:06 ` Eli Zaretskii
@ 2016-12-02 23:15 ` Karl Fogel
2016-12-15 14:28 ` Philippe Vaucher
2 siblings, 0 replies; 299+ messages in thread
From: Karl Fogel @ 2016-12-02 23:15 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
Daniel Colascione <dancol@dancol.org> writes:
>FWIW, I'll put my patch on a branch this weekend.
Thanks, Daniel. I will build it and try it out.
Best regards,
-Karl
>On Fri, Dec 02 2016, Karl Fogel wrote:
>> Following up to what John said, with a perspective that may be useful to Eli:
>>
>> Eli, I have a great deal of respect for your technical judgement and
>> the amount of work you put into Emacs. But I have a slightly
>> different take on what maintainership means.
>>
>> When I was in a position similar to yours (one of a small group of
>> core co-maintainers, in the Subversion project), there were a few
>> occasions when a major technical decision went in a direction I didn't
>> agree with. My disagreement was not of the "this will destroy the
>> project" sort, but I did feel the decisions were poor technical
>> choices that could be a long-term drag on the project -- creating
>> extra maintenance work for existing developers, and creating barriers
>> to entry for new developers. In other words, objections very similar
>> to yours now.
>>
>> In each case, discussions eventually made it clear to me that my
>> opinion was not going to prevail. There were just too many developers
>> who felt differently. This didn't tempt me to step down from
>> maintainership, though I will admit that it *did* tempt me to keep
>> score a bit, so that later on I might be able to say "Hey folks,
>> remember when I turned out to be right about X, and Y, and Z?
>> So please listen to me this time." I think I may even have used that
>> privilege once or twice later :-), though I don't remember clearly if
>> I did. What I do remember clearly is that I turned out to be wrong in
>> at least one of the cases (at least, I established to my own
>> satisfaction that my warnings had been unwarranted).
>>
>> Your value as a maintainer is not lessened if a particular decision
>> doesn't go the way you recommended. (Of course, if maintainership
>> becomes unenjoyable to you because of the decision, that's a different
>> question, and only you can make that call.) I think there is even a
>> good argument that your continued maintainership actually becomes
>> slightly *more* important to the project: of all the people who could
>> spot whether the negative consequences of the decision are coming
>> true, you would perhaps be the person most likely to spot it, and most
>> likely to point it out to the other developers.
>>
>> You must make your own decision, of course. But I hope this
>> perspective is a useful response to your question:
>>
>> "...What kind of maintainer will I be if I cannot base my judgment and
>> decisions on a few principal ideas I have about Emacs development?
>> What could replace those ideas to be the base for decisions I need to
>> make almost every day?"
>>
>> (I pretty much agree with everything John said in his email below,
>> too, for what it's worth.)
>>
>> Best regards,
>> -Karl
>>
>> John Wiegley <jwiegley@gmail.com> writes:
>>>I think a part of this job is always political: Balancing the twin goals of
>>>promoting the best ideas, while also keeping everyone involved encouraged and
>>>motivated. When we say "yes" to an idea, we might be hurting Emacs; but when
>>>we say "no", we might be hurting that contributor, and losing their future
>>>involvement. Sometimes that's fine -- after all, everyone else in the world
>>>wants us to care about Emacs more than a limited-time set of contributors --
>>>but we also cannot thrive without active contributors, so "watering the
>>>garden" is part of our task.
>>>
>>>We all know the portable dumper is not an ideal solution; we even have some
>>>notion of what the ideal solution looks like. It's indeed a bit frustrating to
>>>knowingly support a technology path that may end up incurring a lot of work,
>>>if a much better solution might be just around the corner.
>>>
>>>However, I ask you to consider the merits of Daniel's proposal from several
>>>angles -- angles a regular developer may not care about at all:
>>>
>>> - If we accept the work, we show to others that *code wins*. That is, if a
>>> problem exists in Emacs, and someone comes to us with actual code, this has
>>> tremendous weight. If this is our position, it encourages others to
>>> experiment, since they'll fear less that after so much work, we might just
>>> say "no" because it's not as good a solution as we'd like.
>>>
>>> - If we accept the work, it encourages Daniel to play a larger role, to learn
>>> more about the C internals, and likely to contribute even more.
>>>
>>> - If we accept the work, and a better solution comes along later, we can
>>> revert the change without undue labor. That is, our end cost is not so
>>> terribly high that we're panting ourselves into a corner.
>>>
>>> - If we accept the work, despite our disagreement, *precisely because most of
>>> the other core developers do agree*, we're saying that their opinion holds
>>> a lot of weight with us, and this encourages frank and open discussion. If
>>> decisions like this are made by just you or me, against all other
>>> objections, it diminishes the value of this mailing list in others' eyes.
>>> It then seems like we're just making the Emacs we want, and not the Emacs
>>> all of us want.
>>>
>>>I think there is a "greater good" here, and it is not unduly damaged by
>>>letting a short-term solution into the code until a longer-term solution
>>>appears. There is much goodwill to be gained, and really not so much harm. On
>>>the other hand, rejecting it -- despite the general sense of agreement is has
>>>gained -- would cause a much greater social damage, which is far more
>>>difficult to undo than one day asking Daniel to revert his work.
>>>
>>>All that said, if you feel strongly that this proposal hurts Emacs itself, and
>>>cannot abide it, I understand. I was hoping we could all win here, but I know
>>>that sometimes this isn't realistic. I just hope we can all recognize that
>>>this portable dumper idea is not so terribly important: not to fork over, nor
>>>to resign from the amazing support you provide.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 22:24 ` Richard Stallman
@ 2016-12-02 23:32 ` Stefan Monnier
2016-12-03 8:28 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-02 23:32 UTC (permalink / raw)
To: Richard Stallman; +Cc: eliz, emacs-devel
>> No. I mean that bytecomp.el would generate a file in a format which
>> pdumper can use (which has various consequences, such as the fact that
>> .elc files aren't portable any more).
> That seems like a big disadvantage to me. This could be sufficient
> reason just by itself to stick with the current format for elc files.
> People expect to install one elc file and have it work in multple
> Emacs versions. Only rarely have we made changes that would prevent
> this.
Indeed, such a change would mean that "the .elc file" would probably be
auto-generated behind the scene on-demand.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 14:51 ` Stefan Monnier
2016-12-02 22:24 ` Richard Stallman
@ 2016-12-02 23:42 ` Paul Eggert
1 sibling, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2016-12-02 23:42 UTC (permalink / raw)
To: Stefan Monnier, Eli Zaretskii; +Cc: emacs-devel
On 12/02/2016 06:51 AM, Stefan Monnier wrote:
> I mean that bytecomp.el would generate a file in a format which
> pdumper can use (which has various consequences, such as the fact that
> .elc files aren't portable any more).
The format could be easy for pdumper to use but still be portable. For
example, the pdumper format could use 64-bit little-endian
representation for Lisp_Object values, even on platforms with 32-bit or
big-endian Lisp_Objects. Although this would complicate the pdumper and
slow it down on platforms where the native format doesn't match the
pdumper representation, the pdumper would still be significantly faster
than lread.c.
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: Preview: portable dumper
[not found] ` <<E1cCwGF-0002PT-Kq@fencepost.gnu.org>
@ 2016-12-03 0:07 ` Drew Adams
2016-12-03 8:25 ` Eli Zaretskii
2016-12-03 21:30 ` Richard Stallman
0 siblings, 2 replies; 299+ messages in thread
From: Drew Adams @ 2016-12-03 0:07 UTC (permalink / raw)
To: rms, Stefan Monnier; +Cc: eliz, emacs-devel
> People expect to install one elc file and have it work in
> multple Emacs versions.
Would that it still were so.
> Only rarely have we made changes that would prevent this.
That used to be the case. Now, nearly every major release is
incompatible, AFAICT. As far as I can see, long gone are the
days when you could byte-compile in an older version and use
the byte-compiled code in a newer version.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 0:07 ` Drew Adams
@ 2016-12-03 8:25 ` Eli Zaretskii
2016-12-03 20:40 ` Joost Kremers
2016-12-03 21:30 ` Richard Stallman
1 sibling, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-03 8:25 UTC (permalink / raw)
To: Drew Adams; +Cc: emacs-devel, rms, monnier
> Date: Fri, 2 Dec 2016 16:07:24 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
>
> > Only rarely have we made changes that would prevent this.
>
> That used to be the case. Now, nearly every major release is
> incompatible, AFAICT. As far as I can see, long gone are the
> days when you could byte-compile in an older version and use
> the byte-compiled code in a newer version.
I'm not aware of such problems. AFAIK, we only made an incompatible
change once, in Emacs 23. The byte-compiled files compiled by later
versions should all be compatible. If you see a problem with that,
please report it as a bug.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 22:24 ` Richard Stallman
2016-12-02 23:32 ` Stefan Monnier
@ 2016-12-03 8:28 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-03 8:28 UTC (permalink / raw)
To: rms; +Cc: monnier, emacs-devel
> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, emacs-devel@gnu.org
> Date: Fri, 02 Dec 2016 17:24:55 -0500
>
> > No. I mean that bytecomp.el would generate a file in a format which
> > pdumper can use (which has various consequences, such as the fact that
> > .elc files aren't portable any more).
>
> That seems like a big disadvantage to me. This could be sufficient
> reason just by itself to stick with the current format for elc files.
> People expect to install one elc file and have it work in multple
> Emacs versions.
That's true, but note that Guile produces non-portable byte-compiled
files since long ago. The Guile developers did that on purpose and
they think it's TRT. I was at least initially surprised when I
learned about that.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 22:28 ` Daniel Colascione
@ 2016-12-03 8:48 ` Eli Zaretskii
2016-12-03 9:34 ` Daniel Colascione
` (2 more replies)
0 siblings, 3 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-03 8:48 UTC (permalink / raw)
To: Daniel Colascione; +Cc: kfogel, emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Cc: Karl Fogel <kfogel@red-bean.com>, emacs-devel@gnu.org
> Date: Fri, 02 Dec 2016 14:28:25 -0800
>
> We've all been contributing to Emacs for many years. I think we have a
> pretty good grasp of the issues and potential problems. I understand
> that you think there's an existential risk that goes into this change,
> but I think you're mistaken, and a lot of other people agree. We all
> have access to the same information; it's unlikely (but not impossible)
> that you're the only one who can see certain potential problems.
Assessment of trends and their impact on the future, with all the
uncertainties involved, requires more than just processing the
immediately available information.
> Please give this change the benefit of the doubt.
I thought I was already doing that. Isn't that why the code will be
on a branch soon?
> Even if your worst-case scenario comes to pass, the worst outcome
> outcome will just be more fiddly maintenance.
We already have such "fiddly maintenance" situation in several areas.
Font selection, font back-ends, character composition, complex script
shaping, and (to some extent) the interaction with X and its window
managers -- all of these are areas where bugs are left for months and
years without being fixed, and no progress is made to adapt Emacs to
the new technologies out there.
IOW, "the worst outcome" is already here. I'm desperately trying not
to make it worse.
> I think the pdumper code is pretty clean; I'm open to suggestions
> for making it moreso.
I will definitely try to keep this in mind when reviewing the code.
However, code cleanness is not the main problem, based on the above
examples. There's nothing unclean in the code in any of the areas I
mentioned. The problems are elsewhere.
> If there's no one left who can understand pdumper, there's no one
> left who can understand the GC either.
GC is stable for many years, and "just works". Bugs only happen there
when people make changes in GC code, for benefits that are not
essential. If there's no one on board who understands GC, we can
simply disallow any changes in that area.
By contrast, the portable dumper is new code. No matter how simple
and clean, it will take some time until it becomes mature enough to
raise to the current status of GC. Until that happens, we cannot
afford being in the same situation as with GC.
> You won't have to touch this code.
I most probably will, and I have no reason to be afraid of that. But
this isn't about me.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 8:48 ` Eli Zaretskii
@ 2016-12-03 9:34 ` Daniel Colascione
2016-12-03 12:47 ` Eli Zaretskii
2016-12-03 17:24 ` Paul Eggert
2016-12-03 15:56 ` Stefan Monnier
2016-12-04 23:05 ` Richard Stallman
2 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-03 9:34 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kfogel, emacs-devel
On Sat, Dec 03 2016, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: Karl Fogel <kfogel@red-bean.com>, emacs-devel@gnu.org
>> Date: Fri, 02 Dec 2016 14:28:25 -0800
>>
>> We've all been contributing to Emacs for many years. I think we have a
>> pretty good grasp of the issues and potential problems. I understand
>> that you think there's an existential risk that goes into this change,
>> but I think you're mistaken, and a lot of other people agree. We all
>> have access to the same information; it's unlikely (but not impossible)
>> that you're the only one who can see certain potential problems.
>
> Assessment of trends and their impact on the future, with all the
> uncertainties involved, requires more than just processing the
> immediately available information.
What information then?
We are both rational creatures. If we disagree, it's either because one
of us made a leap of logic that other has not or because we're starting
with different postulates. Can we please get at the root of
our disagreement?
What evidence would convince you that you were incorrect? On my part, I
might be swayed by examples of other free software projects that have
failed due to the inclusion of C code. I cannot think of any.
It is very frustrating to be asked to believe radical assertions made
without any evidence; that's what happened on the thread-safe malloc
thread, and it was very unproductive.
>> Please give this change the benefit of the doubt.
>
> I thought I was already doing that. Isn't that why the code will be
> on a branch soon?
The branch will be short-lived. I will want to merge this code
soon. It's at that time that I will most appreciate being given the
benefit of the doubt.
>> Even if your worst-case scenario comes to pass, the worst outcome
>> outcome will just be more fiddly maintenance.
>
> We already have such "fiddly maintenance" situation in several areas.
> Font selection, font back-ends, character composition, complex script
> shaping, and (to some extent) the interaction with X and its window
> managers -- all of these are areas where bugs are left for months and
> years without being fixed, and no progress is made to adapt Emacs to
> the new technologies out there.
The reason that these bugs aren't getting fixed is that they're not
bothering people enough to provoke fixes. I know that I'm much more
inclined to fix bugs when they bother me personally. It's misleading to
cite unfixed minor bugs as evidence that nobody knows how to fix the
bugs.
Placing restrictions on the people who do know how to fix these bugs is
a good way to ensure that we'll either leave them unfixed (thinking it
not worth the trouble to justify the addition of C code) or just keep
the fixes local (which I confess to doing, for this reason).
You know why I haven't worked on the GTK+-only backend I proposed? I
don't *need* to yet. The X11 one works fine, with fixes here and
there. One day, it won't work fine anymore, and either I or someone else
will make it work fine. This situation is not evidence that we're
forgetting how to program GUIs. It means that we did an adequate job.
As for "adapting Emacs to the new technologies out there": that is
precisely what the portable dumper does! I sent a patch that adapts
Emacs to modern technologies and you're complaining that nobody is doing
that. What do you expect me to think?
> IOW, "the worst outcome" is already here. I'm desperately trying not
> to make it worse.
Emacs runs fine and has many users. It's influential in the community. I
have no idea how many users we have, but I find Emacs users in every
group of developers I visit. emacs-devel is a high traffic mailing
list. We make the news on every major release. If this situation is "the
worst outcome", the worst outcome is pretty comfortable.
Bugs in the sections of code you mention get fixed when you allow them
to get fixed --- Stefan had to try very hard recently to get the
before-change-functions bug fixed, and that was a simple fix for an
egregious problem.
>> I think the pdumper code is pretty clean; I'm open to suggestions
>> for making it moreso.
>
> I will definitely try to keep this in mind when reviewing the code.
> However, code cleanness is not the main problem, based on the above
> examples. There's nothing unclean in the code in any of the areas I
> mentioned. The problems are elsewhere.
Yes. I don't think anyone is concerned about code quality. There is
literally nothing I know how to do to this code to make you feel better
about merging it. I feel like the code's very existence is what bothers
you, and I can do nothing about that except give up, which I won't
do. If my assessment is incorrect, and there is something I can do to
the code to make you feel better about it, please let me know.
>> If there's no one left who can understand pdumper, there's no one
>> left who can understand the GC either.
>
> GC is stable for many years, and "just works". Bugs only happen there
> when people make changes in GC code, for benefits that are not
> essential. If there's no one on board who understands GC, we can
> simply disallow any changes in that area.
The surest way to ensure that nobody understands a piece of code in a
free software project is to put up a big sign that says "OFF LIMITS! DO
NOT TOUCH!". People learn by doing. You need to let people experiment.
Moving fast (and breaking things occasionally) beats OWNERS. Emacs is
not pacemaker firmware. It's not flying an airplane. A bug introduced is
not the end of the world. We can bisect. You can't simultaneously lament
that the flow of new core developers is a trickle and keep the spigot
turned off.
Your position on this matter has never made any sense to me.
> By contrast, the portable dumper is new code. No matter how simple
> and clean, it will take some time until it becomes mature enough to
> raise to the current status of GC. Until that happens, we cannot
> afford being in the same situation as with GC.
Half the people on emacs-devel understand the GC. The other half could
understand it with a few days of study. The last time we had a major GC
bug (the purespace symbol reference thing), we quickly had two
independent patches for fixing it. (Mine lost, but that's history.)
Both of those path authors are on this thread. This outcome doesn't
suggest that GC is in danger of becoming abandonware. If pdumper ends up
in the same position GC is in today, that's fine.
Speaking of future-proofing Emacs: I'd appreciate fixing the
conservative GC problem I highlighted a few days ago before it becomes a
real bug. Maybe I misread the messages, but my sense was that you were
firmly against that change as well, and I never figured out why.
>> You won't have to touch this code.
>
> I most probably will, and I have no reason to be afraid of that. But
> this isn't about me.
You are the only one objecting. Perhaps, generally speaking, your
opinion carries greater weight, but it should not carry infinite
weight. Please defer to the judgment of the rest of the community. This
change is not an existential threat. We are not the enemy.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 9:34 ` Daniel Colascione
@ 2016-12-03 12:47 ` Eli Zaretskii
2016-12-03 14:36 ` Alan Mackenzie
` (2 more replies)
2016-12-03 17:24 ` Paul Eggert
1 sibling, 3 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-03 12:47 UTC (permalink / raw)
To: Daniel Colascione; +Cc: kfogel, emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Cc: kfogel@red-bean.com, emacs-devel@gnu.org
> Date: Sat, 03 Dec 2016 01:34:42 -0800
>
> > Assessment of trends and their impact on the future, with all the
> > uncertainties involved, requires more than just processing the
> > immediately available information.
>
> What information then?
>
> We are both rational creatures. If we disagree, it's either because one
> of us made a leap of logic that other has not or because we're starting
> with different postulates. Can we please get at the root of
> our disagreement?
Didn't you ever participate in a meeting discussing a complex issue,
and saw different people suggest different assessments and different
courses of actions, although they all have the same information about
the issue at their disposal? I'm sure you did. Why does it happen?
For any number of reasons: different age and experience, different
weight assigned to the same factors involved, different intuition,
different abilities to analyze a complex problem and discern the
important from the unimportant, etc. etc.
It's not about being rational. Complex problems like that cannot be
solved by deterministic algorithms, and the input data is almost
always incomplete.
> What evidence would convince you that you were incorrect?
That's easy: a significant increase in the number of active developers
working on the C level.
> On my part, I might be swayed by examples of other free software
> projects that have failed due to the inclusion of C code.
There's nothing wrong with including C code in Emacs. I'm not against
it. I'm doing that myself all the time, and others do it as well,
with my full blessing. This isn't about adding C code, as I already
said many times. This is about adding significant new components
implemented in C that solve problems which could be solved in Lisp
instead.
> It is very frustrating to be asked to believe radical assertions made
> without any evidence
I gave evidence, you just don't agree with my conclusions from that
evidence.
> >> Please give this change the benefit of the doubt.
> >
> > I thought I was already doing that. Isn't that why the code will be
> > on a branch soon?
>
> The branch will be short-lived. I will want to merge this code
> soon. It's at that time that I will most appreciate being given the
> benefit of the doubt.
Ah, so you still think the suggestion to put the code on a branch was
a political trick, rather than a step towards its potential admission?
I guess _I_ am the one who is not getting any benefit of the doubt
here.
> > We already have such "fiddly maintenance" situation in several areas.
> > Font selection, font back-ends, character composition, complex script
> > shaping, and (to some extent) the interaction with X and its window
> > managers -- all of these are areas where bugs are left for months and
> > years without being fixed, and no progress is made to adapt Emacs to
> > the new technologies out there.
>
> The reason that these bugs aren't getting fixed is that they're not
> bothering people enough to provoke fixes. I know that I'm much more
> inclined to fix bugs when they bother me personally.
So you think that bugs should only be fixed by those who are
personally bothered by them in their routine Emacs usage? That would
leave most of the bugs unfixed. It would also mean that bugs reported
by users who are not programmers, or who cannot afford investing
enough time to learn the relevant parts of the Emacs code, will never
be fixed; if such users are bothered enough by these bugs, their only
choice is simply to stop using Emacs.
As things are, and as they should be, most bugs are fixed by the core
developers, because we want to fix as many bugs as we can, even if we
personally aren't bothered at all by them. The problem that bothers
me happens when the core developers lack knowledge for fixing a bug
and cannot afford investing the time to study the relevant code and
the underlying issues/features, to acquire that knowledge.
> It's misleading to cite unfixed minor bugs as evidence that nobody
> knows how to fix the bugs.
"Minor bugs"? I'm talking about problems that slow down display of
certain fonts to a crawl, or produce badly formatted display, or cause
crashes in GC or in malloc. These bugs are definitely not minor.
> Placing restrictions on the people who do know how to fix these bugs is
> a good way to ensure that we'll either leave them unfixed (thinking it
> not worth the trouble to justify the addition of C code) or just keep
> the fixes local (which I confess to doing, for this reason).
Code review is not just about obviously wrong code. It is also about
other less obvious factors, such as how well the submission fits with
the overall design and implementation. You will find these issues
brought up in any project, and some submissions rejected on those
grounds; Emacs is no exception. Nor did I invent that here.
> There is literally nothing I know how to do to this code to make you
> feel better about merging it. I feel like the code's very existence
> is what bothers you
That's profoundly false, not to say unfair. I've stated exactly what
bothers me: I would like to see this problem fixed differently, if
possible and feasible. I also explained what would be my preferred
solution. After all this, what you say above is a strawman at best.
> If my assessment is incorrect, and there is something I can do to
> the code to make you feel better about it, please let me know.
It's not the code that bothers me, it's the pressure to accept it
without exploring the possibility of solving the same problem by the
preferred method. What's the rush? You said you are going to be
around for maintaining the code, so why can't you wait for us to try
the lread speedup first? xwidgets was on a branch for months after it
was declared ready; why this one is different?
> >> If there's no one left who can understand pdumper, there's no one
> >> left who can understand the GC either.
> >
> > GC is stable for many years, and "just works". Bugs only happen there
> > when people make changes in GC code, for benefits that are not
> > essential. If there's no one on board who understands GC, we can
> > simply disallow any changes in that area.
>
> The surest way to ensure that nobody understands a piece of code in a
> free software project is to put up a big sign that says "OFF LIMITS! DO
> NOT TOUCH!". People learn by doing.
People learn by doing, but many don't stay long enough to learn
enough. They disappear with their partial knowledge, and we are left
to deal with the bugs. If those people were to stay here longer, or
if we had someone on board who'd be able to fix the fallout quickly,
then sure, we could afford to be nonchalant about changes in those
core areas. As things are, we need to be more cautious. It's a fine
balance, for sure, which is why changes in those areas are indeed
accepted, as you can clearly see from the logs. We didn't mark those
areas OFF LIMITS. You were describing a desperate situation, which
didn't happen yet, so I said that if push comes to shove, we can apply
desperate measures. We are not there yet, and I hope we never will
be.
> You need to let people experiment. Emacs is not pacemaker
> firmware. It's not flying an airplane.
It isn't a student project, either. It is a veteran project expected
to be stable, even on the master branch. People do important things
with it, and their edits are sometimes precious. There are
distributions out there providing snapshots of master, and people use
those snapshots in their routine work. We cannot afford having master
broken except for very short times.
> A bug introduced is not the end of the world. We can bisect.
This only works in simple cases. It doesn't work when the offending
commit was fixing a problem, which we don't want to get back. It also
doesn't work when removal of the offending commit invalidates a large
series of changes which we find unjustified to back out.
And then there are bugs whose reproduction is itself a problem, for
which bisection is infeasible.
> You can't simultaneously lament that the flow of new core developers
> is a trickle and keep the spigot turned off.
The spigot is not off. There are a few cases, hopefully rare, where
changes are rejected for these reasons. But they are a minority, and
your presentation of them as a majority is a significant exaggeration,
probably to make a point.
The truth is that this is a fine balance, and I think we are generally
doing well with walking that thin line.
> Your position on this matter has never made any sense to me.
And yet people want me to continue doing this job. Which makes
absolutely no sense to me. Why would they want that if they don't
agree with how I do my job?
Would someone please try answering this question?
> Half the people on emacs-devel understand the GC. The other half could
> understand it with a few days of study.
Really? Then why did the last 2 serious bugs there wait until yours
truly debugged them and proposed a fix?
> The last time we had a major GC bug (the purespace symbol reference
> thing), we quickly had two independent patches for fixing it. (Mine
> lost, but that's history.)
No, the last two major bugs that were detected by GC were bug#24478
and bug#24640. Their root causes were not in GC, but debugging them
required familiarity with GC and related functionality. May I suggest
that you familiarize yourself with how those bugs were analyzed and
how the fix for them was found? Because this kind of problems, and
our ability to fix them in the future, is what bothers me.
> Speaking of future-proofing Emacs: I'd appreciate fixing the
> conservative GC problem I highlighted a few days ago before it becomes a
> real bug. Maybe I misread the messages, but my sense was that you were
> firmly against that change as well, and I never figured out why.
You never presented any code, so the discussion was purely
theoretical. Who knows, perhaps I misunderstood what you were
suggesting. It happened to me as well: 2 months ago I described the
idea very similar to your portable dumper, and Paul Eggert rejected it
in very firm terms. Now he supports your implementation of the same
ideas. Go figure.
> >> You won't have to touch this code.
> >
> > I most probably will, and I have no reason to be afraid of that. But
> > this isn't about me.
>
> You are the only one objecting.
Whose job is it to object?
> Perhaps, generally speaking, your opinion carries greater weight,
> but it should not carry infinite weight.
How much weight does it have? How do you determine whether that
weight is enough to override the opinions of N others?
> Please defer to the judgment of the rest of the community.
Isn't that why I agreed to put the code on a branch?
> This change is not an existential threat.
Not it alone, no. But it's in that direction which I'd like to avoid
as much as possible. Add enough such changes, and they might be such
a serious threat.
> We are not the enemy.
Neither am I.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 12:47 ` Eli Zaretskii
@ 2016-12-03 14:36 ` Alan Mackenzie
2016-12-03 15:11 ` Eli Zaretskii
` (2 more replies)
2016-12-03 17:41 ` Paul Eggert
2016-12-03 21:30 ` Richard Stallman
2 siblings, 3 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-03 14:36 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kfogel, Daniel Colascione, emacs-devel
Hello, Eli.
I haven't really been following this thread, but one tangential thing in
it jumped out at me:
On Sat, Dec 03, 2016 at 02:47:07PM +0200, Eli Zaretskii wrote:
> > From: Daniel Colascione <dancol@dancol.org>
> > Cc: kfogel@red-bean.com, emacs-devel@gnu.org
> > Date: Sat, 03 Dec 2016 01:34:42 -0800
[ .... ]
> > What evidence would convince you that you were incorrect?
> That's easy: a significant increase in the number of active developers
> working on the C level.
I have made quite a lot of changes at the C level, but I'm not really a
"core C level" developer.
However, it feels that an unusually high proportion of C level changes I
have hacked or proposed have been rejected. ("Unusual" when compared to
lisp level changes.) Examples include:
(i) Changing the method of syntax.c scanning backwards over comments. My
changes found their way into branch comment-cache in 2016-03. Despite
this change having been extensively discussed in emacs-devel, and sort of
"approved", the final patch was never considered on its merits. The
ostensible reason was that it used a cache which wasn't the syntax-ppss
cache.
(ii) Around 2015-11-17, I proposed a patch to fix bug #21869 and bug
#21333, with top line of the commit message being "Invoke
window-size-change-functions after changing echo area height.". The
problem here was that window-size-change-functions was sometimes being
called twice. You rejected my patch because you were "not keen" on
changing the order of calls in the display engine because we "didn't
fully understand what was going on". Again, I don't think this proposed
patch was really considered on its merits.
(iii) Earlier this year, we were having problems because some primitives
were not calling before-change-functions and after-change-functions the
way we might wish. My offer to analyse the code and amend it so that all
primitives would call b-c-f and a-c-f predictably was declined, the
proviso being (if I remember correctly) "unless somebody writes a solid
suite of unit tests". At the time of this rejection, I'd already
invested some time on the analysis.
It would be unfair not to mention that a lot of changes I've proposed at
the C level, and hacked, have been accepted. Yet, at the same time, when
I now consider tackling problems which need fixing at the C level, I feel
a doubt in my mind that any proposed fix would be properly considered,
and I weigh up this possibility before committing myself to spending any
time on it.
In short, I feel discouraged from working at the C level because of the
above. I might not be the only developer who feels this.
[ .... ]
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 14:36 ` Alan Mackenzie
@ 2016-12-03 15:11 ` Eli Zaretskii
2016-12-04 12:20 ` Alan Mackenzie
2016-12-03 17:36 ` Daniel Colascione
2016-12-03 21:31 ` Preview: portable dumper Richard Stallman
2 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-03 15:11 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: kfogel, dancol, emacs-devel
> Date: Sat, 3 Dec 2016 14:36:04 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: kfogel@red-bean.com, Daniel Colascione <dancol@dancol.org>,
> emacs-devel@gnu.org
>
> However, it feels that an unusually high proportion of C level changes I
> have hacked or proposed have been rejected. ("Unusual" when compared to
> lisp level changes.)
Can you estimate that proportion? You cited 3 examples, and then said
that "a lot of changes have been accepted". 3 out of many is not a
high proportion.
> (i) Changing the method of syntax.c scanning backwards over comments. My
> changes found their way into branch comment-cache in 2016-03. Despite
> this change having been extensively discussed in emacs-devel, and sort of
> "approved", the final patch was never considered on its merits. The
> ostensible reason was that it used a cache which wasn't the syntax-ppss
> cache.
I don't think I participated in that discussion or reviewed that
patch, so I cannot say anything about that.
> (ii) Around 2015-11-17, I proposed a patch to fix bug #21869 and bug
> #21333, with top line of the commit message being "Invoke
> window-size-change-functions after changing echo area height.". The
> problem here was that window-size-change-functions was sometimes being
> called twice. You rejected my patch because you were "not keen" on
> changing the order of calls in the display engine because we "didn't
> fully understand what was going on". Again, I don't think this proposed
> patch was really considered on its merits.
Please see
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#23
It indicates that we already had a patch for those problems, which was
already tested quite a lot by that time. I think it's only natural to
prefer a well tested and discussed patch to a new one trying to fix
the same problem. Reading this message further down:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#37
I see that you agreed that the problem was fixed by the patch we had,
which allowed me to close the bug.
> (iii) Earlier this year, we were having problems because some primitives
> were not calling before-change-functions and after-change-functions the
> way we might wish. My offer to analyse the code and amend it so that all
> primitives would call b-c-f and a-c-f predictably was declined, the
> proviso being (if I remember correctly) "unless somebody writes a solid
> suite of unit tests". At the time of this rejection, I'd already
> invested some time on the analysis.
That's not exactly my recollection. The analysis was not rejected, I
simply already did it when you proposed that, so it wasn't necessary.
> In short, I feel discouraged from working at the C level because of the
> above.
Please don't be discouraged. There's no policy of preventing changes
on the C level. However, for some changes which affect important
functions, I think it's prudent to require a reasonable coverage by
tests, so that we know we didn't break anything.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 8:48 ` Eli Zaretskii
2016-12-03 9:34 ` Daniel Colascione
@ 2016-12-03 15:56 ` Stefan Monnier
2016-12-03 21:31 ` Richard Stallman
2016-12-04 23:05 ` Richard Stallman
2 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-03 15:56 UTC (permalink / raw)
To: emacs-devel
> We already have such "fiddly maintenance" situation in several areas.
> Font selection, font back-ends, character composition, complex script
> shaping, and (to some extent) the interaction with X and its window
> managers -- all of these are areas where bugs are left for months and
> years without being fixed, and no progress is made to adapt Emacs to
> the new technologies out there.
From where I stand I have the impression that we have a respectable
"marketshare" among those programmers interested in programming
languages (for example, in my research area (functional programming and
type systems), most new experimental languages come with support for
Emacs first and support for other editors only show up later).
On the contrary I get the impression that our "markethare" among
programmers interested in "GUI stuff" is very much smaller.
So I don't think the lack of people willing to help us with X and font
stuff implies that there's a lack of people able to hack on things like
the GC (and if you're able to hack on the GC, then that probably carries
over to the pdumper since it's closely related).
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: Preview: portable dumper
[not found] ` <<8337i5mnb5.fsf@gnu.org>
@ 2016-12-03 16:14 ` Drew Adams
2016-12-03 16:42 ` Eli Zaretskii
0 siblings, 1 reply; 299+ messages in thread
From: Drew Adams @ 2016-12-03 16:14 UTC (permalink / raw)
To: Eli Zaretskii, Drew Adams; +Cc: emacs-devel, rms, monnier
> > > Only rarely have we made changes that would prevent this.
> >
> > That used to be the case. Now, nearly every major release is
> > incompatible, AFAICT. As far as I can see, long gone are the
> > days when you could byte-compile in an older version and use
> > the byte-compiled code in a newer version.
>
> I'm not aware of such problems. AFAIK, we only made an incompatible
> change once, in Emacs 23. The byte-compiled files compiled by later
^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^
> versions should all be compatible. If you see a problem with that,
> please report it as a bug.
A change _in_ 23? Or a change _after_ 23? If the change
occurred in 23 then files compiled in 23 should be compatible
in later releases. It should not be the case only that files
compiled in later releases are compatible with still later releases.
I cannot (always) byte-compile in 23 and use the result in
later releases, at least. (It depends on the file.)
It may or may not represent a hard incompatibility. It might
just be a change in a calling sequence or existence of some
macros. I don't know, and I don't have the time to dig into it.
I do know that I do not hesitate to have conditional code that
(normally) is all that's needed, to deal with such changes,
and that that no longer suffices.
Are you suggesting that if someone can report a true
incompatibility introduced after Emacs 23 you will actually
change the byte compiler to remove it, and that you will
backport that fix to where it was introduced and later
releases? (Otherwise, the fix doesn't really help.)
You can believe me or not. I am used to conditionalizing
code to make it work with multiple releases. And I have
always tried, for my own use, to byte-compile in the oldest
release possible, so that the result works for all of the
releases the particular library supports. I have had to
stop compiling (except for testing, to pick up warnings)
some libraries because of what I've found to be
byte-compilation incompatibility.
Yes, my claim remains vague. And yes, I'm anyway glad
to hear that there is some profession of still wanting
to keep compatibility for future releases. That is a
large consolation, for which I am grateful.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 16:14 ` Drew Adams
@ 2016-12-03 16:42 ` Eli Zaretskii
0 siblings, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-03 16:42 UTC (permalink / raw)
To: Drew Adams; +Cc: emacs-devel, rms, drew.adams, monnier
> Date: Sat, 3 Dec 2016 08:14:35 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: rms@gnu.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>
> > > > Only rarely have we made changes that would prevent this.
> > >
> > > That used to be the case. Now, nearly every major release is
> > > incompatible, AFAICT. As far as I can see, long gone are the
> > > days when you could byte-compile in an older version and use
> > > the byte-compiled code in a newer version.
> >
> > I'm not aware of such problems. AFAIK, we only made an incompatible
> > change once, in Emacs 23. The byte-compiled files compiled by later
> ^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^
> > versions should all be compatible. If you see a problem with that,
> > please report it as a bug.
>
> A change _in_ 23? Or a change _after_ 23?
The former, AFAIK.
> Are you suggesting that if someone can report a true
> incompatibility introduced after Emacs 23 you will actually
> change the byte compiler to remove it, and that you will
> backport that fix to where it was introduced and later
> releases? (Otherwise, the fix doesn't really help.)
I don't know, but at least we will know where we stand.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 9:34 ` Daniel Colascione
2016-12-03 12:47 ` Eli Zaretskii
@ 2016-12-03 17:24 ` Paul Eggert
1 sibling, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2016-12-03 17:24 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
Daniel Colascione wrote:
> I'd appreciate fixing the
> conservative GC problem I highlighted a few days ago before it becomes a
> real bug.
Although I'm not as worried about that prospect as you are, I would also
appreciate a fix for the problem, so long as it doesn't hurt performance in the
common case where the fix isn't needed. As I understand it, you want the the
conservative object scanner to count any pointer to somewhere in (or just past
the end of) an object, as opposed to counting only pointers to the start of the
object. That would be a reasonable thing to add, perhaps conditionally compiled
so that it doesn't hurt performance on typical architectures today.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 14:36 ` Alan Mackenzie
2016-12-03 15:11 ` Eli Zaretskii
@ 2016-12-03 17:36 ` Daniel Colascione
2016-12-03 17:40 ` Dmitry Gutov
2016-12-03 21:31 ` Preview: portable dumper Richard Stallman
2 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-12-03 17:36 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: kfogel, Eli Zaretskii, emacs-devel
On Sat, Dec 03 2016, Alan Mackenzie wrote:
> Hello, Eli.
>
> I haven't really been following this thread, but one tangential thing in
> it jumped out at me:
>
> On Sat, Dec 03, 2016 at 02:47:07PM +0200, Eli Zaretskii wrote:
>> > From: Daniel Colascione <dancol@dancol.org>
>> > Cc: kfogel@red-bean.com, emacs-devel@gnu.org
>> > Date: Sat, 03 Dec 2016 01:34:42 -0800
>
> [ .... ]
>
>> > What evidence would convince you that you were incorrect?
>
>> That's easy: a significant increase in the number of active developers
>> working on the C level.
>
> I have made quite a lot of changes at the C level, but I'm not really a
> "core C level" developer.
>
> However, it feels that an unusually high proportion of C level changes I
> have hacked or proposed have been rejected. ("Unusual" when compared to
> lisp level changes.) Examples include:
>
> (i) Changing the method of syntax.c scanning backwards over comments. My
> changes found their way into branch comment-cache in 2016-03. Despite
> this change having been extensively discussed in emacs-devel, and sort of
> "approved", the final patch was never considered on its merits. The
> ostensible reason was that it used a cache which wasn't the syntax-ppss
> cache.
I'd forgotten about that patch. It's further evidence of the trouble
lurking here. How about landing that code now?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 17:36 ` Daniel Colascione
@ 2016-12-03 17:40 ` Dmitry Gutov
2016-12-03 21:09 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-03 17:40 UTC (permalink / raw)
To: Daniel Colascione, Alan Mackenzie; +Cc: kfogel, Eli Zaretskii, emacs-devel
On 03.12.2016 19:36, Daniel Colascione wrote:
> I'd forgotten about that patch. It's further evidence of the trouble
> lurking here. How about landing that code now?
How about not being rash and reading up on the conversations that ended
in that patch not being applied?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 12:47 ` Eli Zaretskii
2016-12-03 14:36 ` Alan Mackenzie
@ 2016-12-03 17:41 ` Paul Eggert
2016-12-03 19:49 ` Eli Zaretskii
2016-12-03 21:30 ` Richard Stallman
2 siblings, 1 reply; 299+ messages in thread
From: Paul Eggert @ 2016-12-03 17:41 UTC (permalink / raw)
To: Eli Zaretskii, Daniel Colascione; +Cc: kfogel, emacs-devel
Eli Zaretskii wrote:
> the last two major bugs that were detected by GC were bug#24478 and bug#24640.
I don't see how this is counting. I don't see the relationship between bug#24478
and GC. Bug#24640 was a crash but the bug report was not marked as serious.
If we're merely counting GC-related things that broke Emacs, a
more-recently-fixed bug related to GC was not reported to bug-gnu-emacs at all;
it's something I fixed two days ago in the release branch, which fixed a broken
build with the Oracle Developer Studio C compiler.
On the other hand, if we're talking about truly major bugs detected by GC, then
Emacs currently doesn't work on FreeBSD 11 arm64 due to GC-related problems that
Daniel's patches would fix. This is Bug#24892, marked as serious, and it pretty
clearly outranks all the other bugs mentioned in this thread.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 17:41 ` Paul Eggert
@ 2016-12-03 19:49 ` Eli Zaretskii
0 siblings, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-03 19:49 UTC (permalink / raw)
To: Paul Eggert; +Cc: kfogel, dancol, emacs-devel
> Cc: kfogel@red-bean.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 3 Dec 2016 09:41:43 -0800
>
> Eli Zaretskii wrote:
> > the last two major bugs that were detected by GC were bug#24478 and bug#24640.
>
> I don't see how this is counting. I don't see the relationship between bug#24478
> and GC. Bug#24640 was a crash but the bug report was not marked as serious.
Crashes in GC are always serious, because you can never know what
caused them. In that case, it turned out that reading objects with
the #n=object form was broken, quite a serious problem.
> If we're merely counting GC-related things that broke Emacs, a
> more-recently-fixed bug related to GC was not reported to bug-gnu-emacs at all;
> it's something I fixed two days ago in the release branch, which fixed a broken
> build with the Oracle Developer Studio C compiler.
Indeed, problems that are revealed in GC are not very rare.
> On the other hand, if we're talking about truly major bugs detected by GC, then
> Emacs currently doesn't work on FreeBSD 11 arm64 due to GC-related problems that
> Daniel's patches would fix. This is Bug#24892, marked as serious, and it pretty
> clearly outranks all the other bugs mentioned in this thread.
There's more than one problem, and no single solution can fix them
all.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 8:25 ` Eli Zaretskii
@ 2016-12-03 20:40 ` Joost Kremers
0 siblings, 0 replies; 299+ messages in thread
From: Joost Kremers @ 2016-12-03 20:40 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Drew Adams, emacs-devel
On Sat, Dec 03 2016, Eli Zaretskii wrote:
>> Date: Fri, 2 Dec 2016 16:07:24 -0800 (PST)
>> From: Drew Adams <drew.adams@oracle.com>
>> Cc: eliz@gnu.org, emacs-devel@gnu.org
>>
>> > Only rarely have we made changes that would prevent this.
>>
>> That used to be the case. Now, nearly every major release is
>> incompatible, AFAICT. As far as I can see, long gone are the
>> days when you could byte-compile in an older version and use
>> the byte-compiled code in a newer version.
>
> I'm not aware of such problems. AFAIK, we only made an
> incompatible
> change once, in Emacs 23. The byte-compiled files compiled by
> later
> versions should all be compatible. If you see a problem with
> that,
> please report it as a bug.
Well, I may have run into a problem of this kind: a package
(`expand-region') that started behaving weird when I moved from
Emacs 24.5 to a 25 pretest. I did not recompile any packages when
I upgraded Emacs, so when I ran into the problem, I was running
Emacs 24 byte code in Emacs 25. When I recompiled `expand-region',
the problem I was experiencing went away.
I didn't investigate further (I know zilch about Emacs' byte code
and my problem was solved), so I can't be 100% sure it was due to
a byte code incompatibility, it just seemed likely. (I also didn't
consider filing a bug with Emacs, because it made sense to me
*not* to expect byte-code compatibility across major versions.)
Anyway, the details are here (no technical details, just a
description of the problem):
https://github.com/magnars/expand-region.el/issues/195
--
Joost Kremers
Life has its moments
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 17:40 ` Dmitry Gutov
@ 2016-12-03 21:09 ` Stefan Monnier
2016-12-03 21:31 ` Daniel Colascione
2016-12-04 12:34 ` Alan Mackenzie
0 siblings, 2 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-03 21:09 UTC (permalink / raw)
To: emacs-devel
>> I'd forgotten about that patch. It's further evidence of the trouble
>> lurking here. How about landing that code now?
> How about not being rash and reading up on the conversations that ended in
> that patch not being applied?
Or applying the patch I suggested instead (much simpler and leverages
the syntax-ppss cache)...
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 22:06 ` Eli Zaretskii
2016-12-02 22:28 ` Daniel Colascione
2016-12-02 22:29 ` John Wiegley
@ 2016-12-03 21:28 ` Richard Stallman
2016-12-04 15:57 ` Eli Zaretskii
2 siblings, 1 reply; 299+ messages in thread
From: Richard Stallman @ 2016-12-03 21:28 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kfogel, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> "Unenjoyable"? It's intolerable. It would mean that on my watch
> something happened that I believe is detrimental to a project which I
> took upon myself to improve, and I saw it happening and didn't prevent
> it. Can you understand my plight?
You have explained a possible drawback of the portable dumper:
it could make maintenance more difficult some years from now.
No one can doubt that this is a possibility. People do disagree about
how much of a problem it is likely to be. Many of us don't expect it
will make a big difference. You forecast that it could.
But you've got to admit that it is uncertain. We're all making
guesstimates about probabilities.
So if we install the portable dumper, please don't treat it as a
certain disaster. It would increase some problems quantitatively, and
reduce other problems quantitatively.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 0:07 ` Drew Adams
2016-12-03 8:25 ` Eli Zaretskii
@ 2016-12-03 21:30 ` Richard Stallman
1 sibling, 0 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-03 21:30 UTC (permalink / raw)
To: Drew Adams; +Cc: eliz, monnier, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> That used to be the case. Now, nearly every major release is
> incompatible, AFAICT.
Do others agree that this is the case?
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 12:47 ` Eli Zaretskii
2016-12-03 14:36 ` Alan Mackenzie
2016-12-03 17:41 ` Paul Eggert
@ 2016-12-03 21:30 ` Richard Stallman
2016-12-04 3:31 ` Eli Zaretskii
2 siblings, 1 reply; 299+ messages in thread
From: Richard Stallman @ 2016-12-03 21:30 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kfogel, dancol, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> So you think that bugs should only be fixed by those who are
> personally bothered by them in their routine Emacs usage? That would
> leave most of the bugs unfixed. It would also mean that bugs reported
> by users who are not programmers, or who cannot afford investing
> enough time to learn the relevant parts of the Emacs code, will never
> be fixed; if such users are bothered enough by these bugs, their only
> choice is simply to stop using Emacs.
The measure of real commitment to development of Emacs (or any free program)
is willingness to fix bugs _because that's important_ and not just
because they bother you.
Daniel, after the portable dumper is done, would you like to learn
some area of the display code where bugs are waiting to be fixed?
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 15:56 ` Stefan Monnier
@ 2016-12-03 21:31 ` Richard Stallman
0 siblings, 0 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-03 21:31 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> most new experimental languages come with support for
> Emacs first and support for other editors only show up later).
> On the contrary I get the impression that our "markethare" among
> programmers interested in "GUI stuff" is very much smaller.
This means we have strong support in one area of development,
and insufficient support in another. However, we need to maintain
Emacs in both areas.
Would anyone like to help us at in the redisplay and graphics C code?
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 14:36 ` Alan Mackenzie
2016-12-03 15:11 ` Eli Zaretskii
2016-12-03 17:36 ` Daniel Colascione
@ 2016-12-03 21:31 ` Richard Stallman
2016-12-04 12:41 ` Alan Mackenzie
2 siblings, 1 reply; 299+ messages in thread
From: Richard Stallman @ 2016-12-03 21:31 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: kfogel, eliz, dancol, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
Alan, would you like to become an expert on some areas of the display
code? If you become the main person working on some area, I am sure
you will get your changes in.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:09 ` Stefan Monnier
@ 2016-12-03 21:31 ` Daniel Colascione
2016-12-04 4:25 ` Stefan Monnier
2016-12-04 12:34 ` Alan Mackenzie
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-12-03 21:31 UTC (permalink / raw)
To: Stefan Monnier, emacs-devel
On 12/03/2016 01:09 PM, Stefan Monnier wrote:
>>> I'd forgotten about that patch. It's further evidence of the trouble
>>> lurking here. How about landing that code now?
>> How about not being rash and reading up on the conversations that ended in
>> that patch not being applied?
>
> Or applying the patch I suggested instead (much simpler and leverages
> the syntax-ppss cache)...
I'll have to go back and reread the thread. I do recall Alan having a
good point about syntax-ppss being incorrect for certain kinds of
two-character comment starters.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-11-30 20:18 ` Daniel Colascione
@ 2016-12-03 21:32 ` Richard Stallman
2016-12-03 21:37 ` Daniel Colascione
2016-12-03 21:54 ` Paul Eggert
0 siblings, 2 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-03 21:32 UTC (permalink / raw)
To: Daniel Colascione; +Cc: eggert, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> Here's the scenario: suppose I can convince your Emacs to parse a
> carefully crafted network packet that triggers a bug in Emacs and lets
> me overwrite arbitrary memory in your Emacs process. Today, I win, in
> the sense that I gain complete control over your Emacs process and can
> do anything Emacs can do.
That reasoning is logically valid -- but is it really a plausible
scenario that Emacs's parsing of a packet would have a bug that
clobbers other unrelated memory?
What Emacs does with the contents of an incoming packet is mainly
to turn it into Lisp objects and make that available at Lisp level.
That means not much opportunity for such a bug to occur.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:32 ` Richard Stallman
@ 2016-12-03 21:37 ` Daniel Colascione
2016-12-04 23:03 ` Richard Stallman
2016-12-03 21:54 ` Paul Eggert
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2016-12-03 21:37 UTC (permalink / raw)
To: rms; +Cc: eggert, emacs-devel
On 12/03/2016 01:32 PM, Richard Stallman wrote:
> > Here's the scenario: suppose I can convince your Emacs to parse a
> > carefully crafted network packet that triggers a bug in Emacs and lets
> > me overwrite arbitrary memory in your Emacs process. Today, I win, in
> > the sense that I gain complete control over your Emacs process and can
> > do anything Emacs can do.
>
> That reasoning is logically valid -- but is it really a plausible
> scenario that Emacs's parsing of a packet would have a bug that
> clobbers other unrelated memory?
Bitter experience with other software has shown the answer to be "yes".
The bug doesn't even have to be in Emacs --- it can be in a library we
use. For example, we link against libpng when available, and libpng has
had repeated security problems over the years:
https://www.cvedetails.com/vulnerability-list/vendor_id-7294/Libpng.html
Gnus uses libpng to display images in news and email. All of this means
that the scenario I've described can come to pass merely by browsing
emacs-devel. I'd prefer to protect us against it.
(I don't want to single out libpng --- other libraries have bugs too.
libpng is just the first one that came to mind.)
> What Emacs does with the contents of an incoming packet is mainly
> to turn it into Lisp objects and make that available at Lisp level.
> That means not much opportunity for such a bug to occur.
If you think so and are willing to take on this risk yourself, you can
build a non-position-independent Emacs and get security and performance
equivalent to what you have today. There are lots of hardening measures
available to modern software, all optional, and we should allow users to
use them.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:32 ` Richard Stallman
2016-12-03 21:37 ` Daniel Colascione
@ 2016-12-03 21:54 ` Paul Eggert
1 sibling, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2016-12-03 21:54 UTC (permalink / raw)
To: rms, Daniel Colascione; +Cc: emacs-devel
Richard Stallman wrote:
> What Emacs does with the contents of an incoming packet is mainly
> to turn it into Lisp objects and make that available at Lisp level.
> That means not much opportunity for such a bug to occur.
Sure, but incoming packets are not random and attackers can design packets to
exploit Emacs bugs. So any opportunity whatsoever in Lisp conversion is more of
concern than it would be for more-typical pointer bugs. And as Daniel mentioned,
attackers will likely focus on the occasions when Emacs converts packet contents
to non-Lisp objects.
I have been looking into improving checking in this area, by using the
-fcheck-pointer-bounds option introduced in GCC 5 and supported by some newer
CPUs. Although -fcheck-pointer-bounds should help, it won't be perfect and it
will be disabled by default due to its performance cost.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:30 ` Richard Stallman
@ 2016-12-04 3:31 ` Eli Zaretskii
2016-12-04 23:03 ` Richard Stallman
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-04 3:31 UTC (permalink / raw)
To: rms; +Cc: kfogel, dancol, emacs-devel
> From: Richard Stallman <rms@gnu.org>
> CC: dancol@dancol.org, kfogel@red-bean.com, emacs-devel@gnu.org
> Date: Sat, 03 Dec 2016 16:30:37 -0500
>
> Daniel, after the portable dumper is done, would you like to learn
> some area of the display code where bugs are waiting to be fixed?
"Display code" might be misleading. There are no bugs I know of in
display code proper (xdisp.c, dispnew.c) that I know of and are not
fixed.
I'm guessing you meant the problems I mentioned in the area of font
selection, font back-ends, and character composition. Those are the
problematic areas.
Volunteers are more than welcome to work in these areas, whether by
studying the code via bugs that need to be solved, or by any other
means.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:31 ` Daniel Colascione
@ 2016-12-04 4:25 ` Stefan Monnier
0 siblings, 0 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-04 4:25 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
> I'll have to go back and reread the thread. I do recall Alan having a good
> point about syntax-ppss being incorrect for certain kinds of two-character
> comment starters.
That was a problem with parse-partial-sexp, and AFAIK Alan fixed
it recently.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 15:11 ` Eli Zaretskii
@ 2016-12-04 12:20 ` Alan Mackenzie
2016-12-04 12:48 ` Dmitry Gutov
2016-12-04 15:53 ` Eli Zaretskii
0 siblings, 2 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-04 12:20 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kfogel, dancol, emacs-devel
Hello, Eli.
On Sat, Dec 03, 2016 at 05:11:33PM +0200, Eli Zaretskii wrote:
> > Date: Sat, 3 Dec 2016 14:36:04 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: kfogel@red-bean.com, Daniel Colascione <dancol@dancol.org>,
> > emacs-devel@gnu.org
> > However, it feels that an unusually high proportion of C level changes I
> > have hacked or proposed have been rejected. ("Unusual" when compared to
> > lisp level changes.)
> Can you estimate that proportion? You cited 3 examples, and then said
> that "a lot of changes have been accepted". 3 out of many is not a
> high proportion.
I can't really estimate it accurately (or it would take far too much
time to do so). It could be as high as 1 in 3. It could be as low as 1
in 10.
> > (i) Changing the method of syntax.c scanning backwards over comments. My
> > changes found their way into branch comment-cache in 2016-03. Despite
> > this change having been extensively discussed in emacs-devel, and sort of
> > "approved", the final patch was never considered on its merits. The
> > ostensible reason was that it used a cache which wasn't the syntax-ppss
> > cache.
> I don't think I participated in that discussion or reviewed that
> patch, so I cannot say anything about that.
I've found that thread, and I misremembered it. The preceding
discussion was not extensive. It was in the thread with Subject:
"bug#22884: 25.0.92; C/l mode editing takes waaaayy too long" and
essentially consisted of just three emails:
(i) Alan -> Eli: Thu, 3 Mar 2016 23:18:23 +0000
(ii) Eli -> Alan: Fri, 04 Mar 2016 10:32:56 +0200
(iii) Alan -> Eli: Fri, 4 Mar 2016 09:37:07 +0000
. In (i), I proposed using text properties to cache the string/comment
state of positions, and asked you whether that might overwhelm the text
property mechanism. In (ii) you replied that it shouldn't. In (iii) I
announced my intention to hack it, which I then did.
I don't remember you reviewing the patch, or expressing any further
views on it afterwards, so we agree on that. The discussion after I
submitted my patch happened in the threads Subject: "Re: [Emacs-diffs]
comment-cache 223d16f 2/3: Apply `comment-depth' text properties when
calling `back_comment'." and Subject: "Permission requested to merge
branch comment-cache into master.". This discussion was long and (to
me) unexpectedly aggressive, and didn't concern itself with the
correctness of the patch. It seemed intended to prevent the adoption of
my patch by spreading FUD. If this was indeed the intention, it was
entirely successful.
> > (ii) Around 2015-11-17, I proposed a patch to fix bug #21869 and bug
> > #21333, with top line of the commit message being "Invoke
> > window-size-change-functions after changing echo area height.". The
> > problem here was that window-size-change-functions was sometimes being
> > called twice. You rejected my patch because you were "not keen" on
> > changing the order of calls in the display engine because we "didn't
> > fully understand what was going on". Again, I don't think this proposed
> > patch was really considered on its merits.
> Please see
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#23
> It indicates that we already had a patch for those problems, which was
> already tested quite a lot by that time. I think it's only natural to
> prefer a well tested and discussed patch to a new one trying to fix
> the same problem. Reading this message further down:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#37
> I see that you agreed that the problem was fixed by the patch we had,
> which allowed me to close the bug.
I've read through that debbugs bug archive. It was a long time ago. I
remember feeling discouraged that my patch was not considered. I think
I felt that my patch fixed the fundamental problem in xdisp.c, whereas
the patch actually applied was more of a workaround. Or something like
that. But I didn't push the point at the time.
> > (iii) Earlier this year, we were having problems because some primitives
> > were not calling before-change-functions and after-change-functions the
> > way we might wish. My offer to analyse the code and amend it so that all
> > primitives would call b-c-f and a-c-f predictably was declined, the
> > proviso being (if I remember correctly) "unless somebody writes a solid
> > suite of unit tests". At the time of this rejection, I'd already
> > invested some time on the analysis.
> That's not exactly my recollection. The analysis was not rejected, I
> simply already did it when you proposed that, so it wasn't necessary.
OK. But the notion of acutally fixing the primitives so that they would
each call b-c-f and a-c-f was strongly discouraged, if I remember
correctly.
> > In short, I feel discouraged from working at the C level because of the
> > above.
> Please don't be discouraged. There's no policy of preventing changes
> on the C level. However, for some changes which affect important
> functions, I think it's prudent to require a reasonable coverage by
> tests, so that we know we didn't break anything.
Perhaps it is that last point (that C functions are more likely to be
critical to Emacs's working than lisp functions, and therefore the
testing is more important) that is the thing.
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:09 ` Stefan Monnier
2016-12-03 21:31 ` Daniel Colascione
@ 2016-12-04 12:34 ` Alan Mackenzie
2016-12-04 12:51 ` Dmitry Gutov
2016-12-04 14:08 ` Stefan Monnier
1 sibling, 2 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-04 12:34 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
On Sat, Dec 03, 2016 at 04:09:24PM -0500, Stefan Monnier wrote:
> >> I'd forgotten about that patch. It's further evidence of the trouble
> >> lurking here. How about landing that code now?
> > How about not being rash and reading up on the conversations that ended in
> > that patch not being applied?
> Or applying the patch I suggested instead (much simpler and leverages
> the syntax-ppss cache)...
Your patch was indeed simpler. Indeed, it was so simple as actually not
to work in general, as well as being ~2 orders of magnitude slower than
the current (forward-comment -1). syntax-ppss is broken (bug #22983)
and has been for many years, so to base a critical primitive on it seems
unwise. And one or two other things, too.
Other than that, yes we could apply that patch.
BTW, I've been looking at bug #22983 the last few days, since it seems
nobody else is going to bother fixing it.
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:31 ` Preview: portable dumper Richard Stallman
@ 2016-12-04 12:41 ` Alan Mackenzie
0 siblings, 0 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-04 12:41 UTC (permalink / raw)
To: Richard Stallman; +Cc: kfogel, eliz, dancol, emacs-devel
Hello, Richard.
On Sat, Dec 03, 2016 at 04:31:35PM -0500, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider ]]]
> [[[ whether defending the US Constitution against all enemies, ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> Alan, would you like to become an expert on some areas of the display
> code? If you become the main person working on some area, I am sure
> you will get your changes in.
I'd like to become an expert on the display code, but I don't think I'm
up to it. That code is just too difficult for me, unless I gave up
doing other things to be able to spend the time on it. It's highly
optimised (it has to be), and is thus very long. I don't think it's
badly written - it's just got a very difficult job to do.
> --
> Dr Richard Stallman
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 12:20 ` Alan Mackenzie
@ 2016-12-04 12:48 ` Dmitry Gutov
2016-12-04 15:53 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-04 12:48 UTC (permalink / raw)
To: Alan Mackenzie, Eli Zaretskii; +Cc: kfogel, dancol, emacs-devel
On 04.12.2016 14:20, Alan Mackenzie wrote:
> This discussion was long and (to
> me) unexpectedly aggressive, and didn't concern itself with the
> correctness of the patch.
It really did. When a patch disregards a whole class of current use
cases, and is likely to make them worse, that's very much a correctness
issue.
> It seemed intended to prevent the adoption of
> my patch by spreading FUD. If this was indeed the intention, it was
> entirely successful.
Thanks for the insult, Alan.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 12:34 ` Alan Mackenzie
@ 2016-12-04 12:51 ` Dmitry Gutov
2016-12-04 14:08 ` Stefan Monnier
1 sibling, 0 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-04 12:51 UTC (permalink / raw)
To: Alan Mackenzie, Stefan Monnier; +Cc: emacs-devel
On 04.12.2016 14:34, Alan Mackenzie wrote:
>> Or applying the patch I suggested instead (much simpler and leverages
>> the syntax-ppss cache)...
>
> Your patch was indeed simpler. Indeed, it was so simple as actually not
> to work in general, as well as being ~2 orders of magnitude slower than
> the current (forward-comment -1).
Which has never been demonstrated to be a problem in practice.
> BTW, I've been looking at bug #22983 the last few days, since it seems
> nobody else is going to bother fixing it.
The patch is there: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22983#47
You're welcome to install it.
I'm still waiting for someone to assist Vitalie with the hard-widen effort.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 12:34 ` Alan Mackenzie
2016-12-04 12:51 ` Dmitry Gutov
@ 2016-12-04 14:08 ` Stefan Monnier
2016-12-04 15:22 ` Alan Mackenzie
2016-12-04 22:24 ` forward-comment and syntax-ppss (was: Preview: portable dumper) Stefan Monnier
1 sibling, 2 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-04 14:08 UTC (permalink / raw)
To: emacs-devel
> Your patch was indeed simpler. Indeed, it was so simple as actually not
> to work in general, as well as being ~2 orders of magnitude slower than
> the current (forward-comment -1).
You make it sound like it barely ever works and it's always 100
times slower. I think this qualifies as "gross misrepresentation".
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 14:08 ` Stefan Monnier
@ 2016-12-04 15:22 ` Alan Mackenzie
2016-12-04 22:24 ` forward-comment and syntax-ppss (was: Preview: portable dumper) Stefan Monnier
1 sibling, 0 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-04 15:22 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
Hello, Stefan.
On Sun, Dec 04, 2016 at 09:08:26AM -0500, Stefan Monnier wrote:
> > Your patch was indeed simpler. Indeed, it was so simple as actually not
> > to work in general, as well as being ~2 orders of magnitude slower than
> > the current (forward-comment -1).
> You make it sound like it barely ever works and it's always 100
> times slower. I think this qualifies as "gross misrepresentation".
I don't think it does.
Your patch won't work, for example, if a lisp program sets syntax-table
text properties (which is _always_ done when
before/after-change-functions are disabled). This is a very important
scenario.
Consider (forward-comment -1) backwards over a 100 character comment.
The current code will do this in 100n units of time, for some value of
n. In your patch, we need to calculate the parse-partial-sexp state at
the comment, and this is going to involve a parse-partial-sexp moving
forward over, on average, 10,000 characters. This is going to take
10,000m units of time, for some m. 10,000m is two orders of magnitude
larger than 100n.
Now, I'm not saying your patch couldn't be improved to handle these and
other things. Firstly, it then wouldn't be a "simple" patch any longer.
And the next thing, why bother, when the comment-cache code already
exists and works?
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 12:20 ` Alan Mackenzie
2016-12-04 12:48 ` Dmitry Gutov
@ 2016-12-04 15:53 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-04 15:53 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: kfogel, dancol, emacs-devel
> Date: Sun, 4 Dec 2016 12:20:33 +0000
> Cc: kfogel@red-bean.com, dancol@dancol.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
>
> I can't really estimate it accurately (or it would take far too much
> time to do so). It could be as high as 1 in 3. It could be as low as 1
> in 10.
I hope it's the latter.
> > > (i) Changing the method of syntax.c scanning backwards over comments. My
> > > changes found their way into branch comment-cache in 2016-03. Despite
> > > this change having been extensively discussed in emacs-devel, and sort of
> > > "approved", the final patch was never considered on its merits. The
> > > ostensible reason was that it used a cache which wasn't the syntax-ppss
> > > cache.
>
> > I don't think I participated in that discussion or reviewed that
> > patch, so I cannot say anything about that.
>
> I've found that thread, and I misremembered it. The preceding
> discussion was not extensive. It was in the thread with Subject:
> "bug#22884: 25.0.92; C/l mode editing takes waaaayy too long" and
> essentially consisted of just three emails:
> (i) Alan -> Eli: Thu, 3 Mar 2016 23:18:23 +0000
> (ii) Eli -> Alan: Fri, 04 Mar 2016 10:32:56 +0200
> (iii) Alan -> Eli: Fri, 4 Mar 2016 09:37:07 +0000
> . In (i), I proposed using text properties to cache the string/comment
> state of positions, and asked you whether that might overwhelm the text
> property mechanism. In (ii) you replied that it shouldn't. In (iii) I
> announced my intention to hack it, which I then did.
>
> I don't remember you reviewing the patch, or expressing any further
> views on it afterwards, so we agree on that. The discussion after I
> submitted my patch happened in the threads Subject: "Re: [Emacs-diffs]
> comment-cache 223d16f 2/3: Apply `comment-depth' text properties when
> calling `back_comment'." and Subject: "Permission requested to merge
> branch comment-cache into master.". This discussion was long and (to
> me) unexpectedly aggressive, and didn't concern itself with the
> correctness of the patch.
I've re-read those threads and found my participation in them to be
almost non-existent. In the first one it was limited to presenting
benchmarks of the performance that was discussed. In the second one I
barely said anything at all, and when I did, it was to encourage you
to install your patch on the release branch rather than master.
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#23
>
> > It indicates that we already had a patch for those problems, which was
> > already tested quite a lot by that time. I think it's only natural to
> > prefer a well tested and discussed patch to a new one trying to fix
> > the same problem. Reading this message further down:
>
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#37
>
> > I see that you agreed that the problem was fixed by the patch we had,
> > which allowed me to close the bug.
>
> I've read through that debbugs bug archive. It was a long time ago. I
> remember feeling discouraged that my patch was not considered. I think
> I felt that my patch fixed the fundamental problem in xdisp.c, whereas
> the patch actually applied was more of a workaround. Or something like
> that. But I didn't push the point at the time.
Once again, a patch that was discussed at length and tested by several
people was committed that fixed the problem. I can understand your
frustration, but we really cannot install more than a single patch for
each problem.
> > > (iii) Earlier this year, we were having problems because some primitives
> > > were not calling before-change-functions and after-change-functions the
> > > way we might wish. My offer to analyse the code and amend it so that all
> > > primitives would call b-c-f and a-c-f predictably was declined, the
> > > proviso being (if I remember correctly) "unless somebody writes a solid
> > > suite of unit tests". At the time of this rejection, I'd already
> > > invested some time on the analysis.
>
> > That's not exactly my recollection. The analysis was not rejected, I
> > simply already did it when you proposed that, so it wasn't necessary.
>
> OK. But the notion of acutally fixing the primitives so that they would
> each call b-c-f and a-c-f was strongly discouraged, if I remember
> correctly.
Yes, because those touched very fundamental functionalities in
manipulating buffer text, which I didn't want to destabilize without
having a test suite with good coverage.
So in sum, I feel that these incidents don't have a lot in common with
the issues discussed here.
> > > In short, I feel discouraged from working at the C level because of the
> > > above.
>
> > Please don't be discouraged. There's no policy of preventing changes
> > on the C level. However, for some changes which affect important
> > functions, I think it's prudent to require a reasonable coverage by
> > tests, so that we know we didn't break anything.
>
> Perhaps it is that last point (that C functions are more likely to be
> critical to Emacs's working than lisp functions, and therefore the
> testing is more important) that is the thing.
I think treating code in C as more delicate is prudent: changes there
will almost always have wider implications than changes in Lisp, and
the code is frequently harder to understand, so better testing is IMO
justified.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:28 ` Richard Stallman
@ 2016-12-04 15:57 ` Eli Zaretskii
2016-12-04 17:12 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-04 15:57 UTC (permalink / raw)
To: rms; +Cc: kfogel, emacs-devel
> From: Richard Stallman <rms@gnu.org>
> CC: kfogel@red-bean.com, emacs-devel@gnu.org
> Date: Sat, 03 Dec 2016 16:28:58 -0500
>
> You have explained a possible drawback of the portable dumper:
> it could make maintenance more difficult some years from now.
>
> No one can doubt that this is a possibility. People do disagree about
> how much of a problem it is likely to be. Many of us don't expect it
> will make a big difference. You forecast that it could.
Thank you for acknowledging that this is a possibility. I was under
the impression that my opponents didn't agree.
> But you've got to admit that it is uncertain. We're all making
> guesstimates about probabilities.
IMO, dangerous outcomes should be taken seriously even if their
probability is low, as long as it's non-zero. "Taken seriously" means
we should do something to try to avoid the danger. That is what I'm
trying to do: find a solution for the problem that will avoid the
danger.
> So if we install the portable dumper, please don't treat it as a
> certain disaster.
The "disaster" is not the installation of the dumper. The disaster I
alluded to would be not to try to find a solution that minimizes the
above danger. If we have tried our best and failed, then it's not a
disaster, it's life: after all, we can only make the best use of the
cards we've been dealt.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 15:57 ` Eli Zaretskii
@ 2016-12-04 17:12 ` Daniel Colascione
2016-12-04 23:07 ` Richard Stallman
2016-12-06 10:38 ` Philippe Vaucher
0 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-04 17:12 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kfogel, rms, emacs-devel
On Sun, Dec 04 2016, Eli Zaretskii wrote:
>> From: Richard Stallman <rms@gnu.org>
>> CC: kfogel@red-bean.com, emacs-devel@gnu.org
>> Date: Sat, 03 Dec 2016 16:28:58 -0500
>>
>> You have explained a possible drawback of the portable dumper:
>> it could make maintenance more difficult some years from now.
>>
>> No one can doubt that this is a possibility. People do disagree about
>> how much of a problem it is likely to be. Many of us don't expect it
>> will make a big difference. You forecast that it could.
>
> Thank you for acknowledging that this is a possibility. I was under
> the impression that my opponents didn't agree.
You're misinterpreting this "agreement". Nobody else agrees that a a
significant maintenance burden is likely. This is how FUD starts.
>> But you've got to admit that it is uncertain. We're all making
>> guesstimates about probabilities.
>
> IMO, dangerous outcomes should be taken seriously even if their
> probability is low, as long as it's non-zero.
The people who moved humanity forward were not the ones huddling in
caves afraid of the mammoths outside.
We're talking about software. Anything we do, we can undo. (And please,
stop suggesting that we have some kind of duty to keep master absolutely
stable because someone, somewhere might make a master snapshot tarball
available for something. Why do we even have releases?)
> "Taken seriously" means
> we should do something to try to avoid the danger. That is what I'm
> trying to do: find a solution for the problem that will avoid the
> danger.
You keep saying that you'd prefer a "solution" that "avoids" the
"danger". There *is* no solution that avoids the "danger" you've
highlighted, which is the mere addition of C code. (No other free
software project views C code as a "danger" to be avoided at all costs.)
Your preferred approach, a magically faster lread, *will* *also*
*involve* *C* *code*. It will *also* require knowledge of Lisp layout,
since what lread does is transform text into Lisp objects. It will
*also* need touchups when we add new lisp objects.
The idea that your preferred fast lread (which, again, does not actually
exist) would be a fine, but that the portable dumper is a "danger" ---
it makes no sense. Your arguments against the portable dumper apply
also to lread.
Please don't tell me that lread is qualitatively different because it's
existing functionality or something: you're talking about complex,
disruptive additions to lread code, and that these additions would go
into a C file that already exists instead of one that doesn't
is immaterial.
>> So if we install the portable dumper, please don't treat it as a
>> certain disaster.
>
> The "disaster" is not the installation of the dumper. The disaster I
> alluded to would be not to try to find a solution that minimizes the
> above danger. If we have tried our best and failed, then it's not a
> disaster, it's life: after all, we can only make the best use of the
> cards we've been dealt.
We've known about unexec problems for years and nobody's come up with a
workable alternative. Please stop obstructing progress because you wish
it weren't so. The biggest danger here is that you get your way, block
progress, and that we're stuck with unexec and fragile hacks for many
more years.
^ permalink raw reply [flat|nested] 299+ messages in thread
* forward-comment and syntax-ppss (was: Preview: portable dumper)
2016-12-04 14:08 ` Stefan Monnier
2016-12-04 15:22 ` Alan Mackenzie
@ 2016-12-04 22:24 ` Stefan Monnier
2016-12-06 19:55 ` Alan Mackenzie
1 sibling, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-04 22:24 UTC (permalink / raw)
To: emacs-devel
>> Your patch was indeed simpler. Indeed, it was so simple as actually not
>> to work in general, as well as being ~2 orders of magnitude slower than
>> the current (forward-comment -1).
> You make it sound like it barely ever works and it's always 100
> times slower. I think this qualifies as "gross misrepresentation".
As a counter-point, I've been using the patch below since this
discussion, without noticing any adverse effects. I don't doubt that it
will occasionally mis-behave in cc-mode buffers, but I'm not sure it's
an argument against this patch. After all, there are mostly 3 entities
at play here:
- this patch (i.e. the linkage between forward-commment and syntax-ppss).
- syntax-ppss itself.
- cc-mode.
I think most of the problems that will show up with this patch can be
attributed to the current syntax-ppss implementation (e.g. its inability
to deal reliably with narrowing or with changes of the syntax-table).
Maybe those problems will best be solved not just by changing
syntax-ppss but by providing more info to syntax-ppss (e.g. requiring
major modes who narrow the buffer or change the syntax-table to
indicate to syntax-ppss how to handle it (since there are various
possible choices and syntax-ppss usually can't know which is best or
even which is right)).
So I see this patch as an opportunity to improve syntax-ppss.
Stefan
diff --git a/src/syntax.c b/src/syntax.c
index d463f7e..b56e808 100644
--- a/src/syntax.c
+++ b/src/syntax.c
@@ -297,6 +297,7 @@ SETUP_SYNTAX_TABLE_FOR_OBJECT (Lisp_Object object,
gl_state.b_property = 0;
gl_state.e_property = PTRDIFF_MAX;
gl_state.offset = 0;
+ return; /* There are no syntax-table properties here. */
}
else
{
@@ -607,6 +634,26 @@ find_defun_start (ptrdiff_t pos, ptrdiff_t pos_byte)
&& MODIFF == find_start_modiff)
return find_start_value;
+ if (!NILP (Vcomment_use_syntax_ppss))
+ {
+ EMACS_INT modiffs = CHARS_MODIFF;
+ Lisp_Object ppss = call1 (Qsyntax_ppss, make_number (pos));
+ if (modiffs != CHARS_MODIFF)
+ error ("syntax-ppss modified the buffer!");
+ TEMP_SET_PT_BOTH (opoint, opoint_byte);
+ Lisp_Object boc = Fnth (make_number (8), ppss);
+ if (NUMBERP (boc))
+ {
+ find_start_value = XINT (boc);
+ find_start_value_byte = CHAR_TO_BYTE (find_start_value);
+ }
+ else
+ {
+ find_start_value = pos;
+ find_start_value_byte = pos_byte;
+ }
+ goto found;
+ }
if (!open_paren_in_column_0_is_defun_start)
{
find_start_value = BEGV;
@@ -874,6 +921,7 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
case Sopen:
/* Assume a defun-start point is outside of strings. */
if (open_paren_in_column_0_is_defun_start
+ && NILP (Vcomment_use_syntax_ppss)
&& (from == stop
|| (temp_byte = dec_bytepos (from_byte),
FETCH_CHAR (temp_byte) == '\n')))
@@ -3453,10 +3493,7 @@ internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
else
{
tem = Fcar (external);
- if (!NILP (tem))
- state->depth = XINT (tem);
- else
- state->depth = 0;
+ state->depth = INTEGERP (tem) ? XINT (tem) : 0;
external = Fcdr (external);
external = Fcdr (external);
@@ -3680,6 +3717,11 @@ void
syms_of_syntax (void)
{
DEFSYM (Qsyntax_table_p, "syntax-table-p");
+ DEFSYM (Qsyntax_ppss, "syntax-ppss");
+ DEFVAR_LISP ("comment-use-syntax-ppss",
+ Vcomment_use_syntax_ppss,
+ doc: /* Non-nil means `forward-comment' can use `syntax-ppss' internally. */);
+ Vcomment_use_syntax_ppss = Qt;
staticpro (&Vsyntax_code_object);
^ permalink raw reply related [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 21:37 ` Daniel Colascione
@ 2016-12-04 23:03 ` Richard Stallman
0 siblings, 0 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-04 23:03 UTC (permalink / raw)
To: Daniel Colascione; +Cc: eggert, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
We seem to be talking at cross purposes. You raised the issue of
parsing network packets:
> > > Here's the scenario: suppose I can convince your Emacs to parse a
> > > carefully crafted network packet that triggers a bug in Emacs and lets
> > > me overwrite arbitrary memory in your Emacs process. Today, I win, in
> > > the sense that I gain complete control over your Emacs process and can
> > > do anything Emacs can do.
so I responded about Emacs's very simple parsing of network packets
> > That reasoning is logically valid -- but is it really a plausible
> > scenario that Emacs's parsing of a packet would have a bug that
> > clobbers other unrelated memory?
but now you've changed the subject to libpng.
> Bitter experience with other software has shown the answer to be "yes".
> The bug doesn't even have to be in Emacs --- it can be in a library we
> use. For example, we link against libpng when available,
You might be right about libpng, but if so, that has nothing
particularly to do with parsing network packets in Emacs.
There are various ways the PNG file could get into your Emacs.
Regarding png files and security, there are many ways to view a png
file in various different programs. It is useful to defend Emacs
better against such bugs, but maybe the better defense is a validator
library, separate from libpng, to carefully validate the format of a
PNG file and detect any inconsistency. Then programs could call that
validator before showing the PNG file to libpng.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 3:31 ` Eli Zaretskii
@ 2016-12-04 23:03 ` Richard Stallman
0 siblings, 0 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-04 23:03 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kfogel, dancol, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> "Display code" might be misleading. There are no bugs I know of in
> display code proper (xdisp.c, dispnew.c) that I know of and are not
> fixed.
> I'm guessing you meant the problems I mentioned in the area of font
> selection, font back-ends, and character composition. Those are the
> problematic areas.
I think of all code for putting text on the screen
as the display code.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-03 8:48 ` Eli Zaretskii
2016-12-03 9:34 ` Daniel Colascione
2016-12-03 15:56 ` Stefan Monnier
@ 2016-12-04 23:05 ` Richard Stallman
2 siblings, 0 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-04 23:05 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: kfogel, dancol, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> GC is stable for many years, and "just works". Bugs only happen there
> when people make changes in GC code, for benefits that are not
> essential. If there's no one on board who understands GC, we can
> simply disallow any changes in that area.
The issue you raised -- and it is a real consideration -- is that
future changes in Lisp object formats would likely require changes in
the portable dumper (if we use that), just as they would likely
require changes in GC.
If someday nobody can change GC, that would mean nobody can change
Lisp object formats any more.
It won't take very long before the portable dumper becomes as stable
as GC. Then it would be the same issue for both. I don't think that
having Lisp object format changes blocked by GC and the portable
dumper is much worse than having them blocked by GC alone.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 17:12 ` Daniel Colascione
@ 2016-12-04 23:07 ` Richard Stallman
2016-12-05 0:24 ` Daniel Colascione
2016-12-06 10:38 ` Philippe Vaucher
1 sibling, 1 reply; 299+ messages in thread
From: Richard Stallman @ 2016-12-04 23:07 UTC (permalink / raw)
To: Daniel Colascione; +Cc: kfogel, eliz, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> You're misinterpreting this "agreement". Nobody else agrees that a a
> significant maintenance burden is likely. This is how FUD starts.
Please don't say it with such a belligerent tone. We need to work together,
so we need to disagree without hostility.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 23:07 ` Richard Stallman
@ 2016-12-05 0:24 ` Daniel Colascione
0 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2016-12-05 0:24 UTC (permalink / raw)
To: rms; +Cc: kfogel, eliz, emacs-devel
On 12/04/2016 03:07 PM, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider ]]]
> [[[ whether defending the US Constitution against all enemies, ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> > You're misinterpreting this "agreement". Nobody else agrees that a a
> > significant maintenance burden is likely. This is how FUD starts.
>
> Please don't say it with such a belligerent tone. We need to work together,
> so we need to disagree without hostility.
Sorry. I should have not been as harsh there.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-04 17:12 ` Daniel Colascione
2016-12-04 23:07 ` Richard Stallman
@ 2016-12-06 10:38 ` Philippe Vaucher
1 sibling, 0 replies; 299+ messages in thread
From: Philippe Vaucher @ 2016-12-06 10:38 UTC (permalink / raw)
To: Daniel Colascione
Cc: kfogel, Eli Zaretskii, Richard Stallman, Emacs developers
[-- Attachment #1: Type: text/plain, Size: 1817 bytes --]
On Sun, Dec 4, 2016 at 6:12 PM, Daniel Colascione <dancol@dancol.org> wrote:
>
> We're talking about software. Anything we do, we can undo. (And please,
> stop suggesting that we have some kind of duty to keep master absolutely
> stable because someone, somewhere might make a master snapshot tarball
> available for something. Why do we even have releases?)
>
Keeping master stable depends on the workflow wanted for Emacs... but in
general, having a stable "master" is an accepted practice for any project
of a reasonable size.
The unstability can go in branches, that's what they are for. A stable
master avoids all the unecesserary noise when "master breaks" because that
affects everyone, and it allows for continuous integration as a bonus.
There is a lot of different worflows available and I think it is our job to
adapt to the project's workflow, or maybe propose a different workflow.
Here are different typical used workflows:
- only one master branch (small projects)
- the git-flow workflow: one long-lived develop branch meant for
development (unstable) from which features/releases branches might get
branched from. The branch master is on the receiving end on the merges and
anything merged to it should be stable (http://nvie.com/posts/a-
successful-git-branching-model). Somewhat "long" release cycles.
- the github workflow: master always "production ready", every
features/bugfixes gets in a branch, and is merged when ready. Emphasis on
short release cycles.
- the git/kernel release model: many topical branches like "next", "pu"
(proposed updates), "maint" (maintenance), etc.
I think in general having a stable master is desired, now I agree it's not
the end of the world if it gets unstable from time to time as long as it's
fixed quickly.
Philippe
[-- Attachment #2: Type: text/html, Size: 2466 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss (was: Preview: portable dumper)
2016-12-04 22:24 ` forward-comment and syntax-ppss (was: Preview: portable dumper) Stefan Monnier
@ 2016-12-06 19:55 ` Alan Mackenzie
2016-12-06 22:56 ` forward-comment and syntax-ppss Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-06 19:55 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
Hello, Stefan.
On Sun, Dec 04, 2016 at 05:24:45PM -0500, Stefan Monnier wrote:
> >> Your patch was indeed simpler. Indeed, it was so simple as actually not
> >> to work in general, as well as being ~2 orders of magnitude slower than
> >> the current (forward-comment -1).
> > You make it sound like it barely ever works and it's always 100
> > times slower. I think this qualifies as "gross misrepresentation".
> As a counter-point, I've been using the patch below since this
> discussion, without noticing any adverse effects. I don't doubt that it
> will occasionally mis-behave in cc-mode buffers, but I'm not sure it's
> an argument against this patch.
I'm not quite sure what you're saying here. Normally, a piece of
software not working properly is an extremely strong argument against it
(though it's an equally strong argument for fixing it).
> After all, there are mostly 3 entities at play here:
> - this patch (i.e. the linkage between forward-commment and syntax-ppss).
> - syntax-ppss itself.
> - cc-mode.
> I think most of the problems that will show up with this patch ....
What, exactly, is this patch intended to do? That's not obvious.
> .... can be attributed to the current syntax-ppss implementation (e.g.
> its inability to deal reliably with narrowing or with changes of the
> syntax-table). Maybe those problems will best be solved not just by
> changing syntax-ppss but by providing more info to syntax-ppss (e.g.
> requiring major modes who narrow the buffer or change the syntax-table
> to indicate to syntax-ppss how to handle it (since there are various
> possible choices and syntax-ppss usually can't know which is best or
> even which is right)).
I think this would be a last resort, when nothing else can work. The
syntactic primitives should not have high level lisp code poking about
in their innards. When it comes to narrowing a buffer, I think that the
syntactic context of a buffer position should not depend on whether the
buffer has been narrowed. If there are use cases this inconveniences, I
think that those use cases' use of narrowing is mistaken, and some other
tool should be used in its place.
> So I see this patch as an opportunity to improve syntax-ppss.
I don't think anybody will object to improving syntax-ppss. But, at the
same time, calling syntax-ppss from the C syntax routines should be
avoided, as far as possible. Such calls are likely to transform
side-effect free functions (such as forward-comment) into side-effect
encumbered functions, which is likely to make them less useful, and to
make Emacs more buggy, less flexible and more difficult to use.
And, just to emphasise, I think it's clear that syntax-ppss has no
business in forward-comment. syntax-ppss is slow (compared to the
current implementation of forward-comment), and there's no need for it,
given the existence of the comment-cache branch.
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-06 19:55 ` Alan Mackenzie
@ 2016-12-06 22:56 ` Stefan Monnier
2016-12-07 6:55 ` Andreas Röhler
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-06 22:56 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: emacs-devel
> I'm not quite sure what you're saying here. Normally, a piece of
> software not working properly is an extremely strong argument against it
> (though it's an equally strong argument for fixing it).
It's probably a strong argument against installing it willy nilly, yes.
> What, exactly, is this patch intended to do? That's not obvious.
It gets rid of some pathological cases where backward scanning currently
may re-scan the whole buffer over and over again, each time from
point-min, resulting in excruciatingly slow backward movement (these
cases are fairly rare, luckily).
> I think this would be a last resort, when nothing else can work. The
> syntactic primitives should not have high level lisp code poking about
> in their innards.
I don't much care for such philosophy, because I consider syntax-ppss to
be low-level Lisp code.
> When it comes to narrowing a buffer, I think that the syntactic
> context of a buffer position should not depend on whether the buffer
> has been narrowed.
As you know by now this is just one opinion, and other users of
narrow-to-region beg to differ, and hence we will need to provide a way
for users of narrow-to-region to communicate their intentions, since
without that extra information, there can be no reliable caching of
parsing.
> If there are use cases this inconveniences, I think that those use
> cases' use of narrowing is mistaken, and some other tool should be
> used in its place.
Fine by me.
>> So I see this patch as an opportunity to improve syntax-ppss.
> I don't think anybody will object to improving syntax-ppss. But, at the
> same time, calling syntax-ppss from the C syntax routines should be
> avoided, as far as possible. Such calls are likely to transform
> side-effect free functions (such as forward-comment) into side-effect
> encumbered functions, which is likely to make them less useful, and to
> make Emacs more buggy, less flexible and more difficult to use.
We've happily crossed that bridge already (tho not by calling
syntax-ppss but by calling syntax-propertize).
> And, just to emphasise, I think it's clear that syntax-ppss has no
> business in forward-comment. syntax-ppss is slow (compared to the
> current implementation of forward-comment),
The patch I showed only uses syntax-ppss when we would otherwise
re-parse the whole buffer, so it shouldn't slow things down too much.
BTW, when writing syntax-ppss I did not really intend to "speed things
up", but to get rid of pathologically slow behavior. In many cases
calling parse-partial-sexp from point-min will be faster than calling
syntax-ppss (mostly because syntax-ppss is all written in Elisp, so just
the time to consult its cache can be longer than the time to compute the
answer from scratch), but its behavior should stay "good enough" in
a longish buffers (where calling parse-partial-sexp from point-min
becomes excessive).
IOW, syntax-ppss will be slower than parse-partial-sexp when
parse-partial-sexp is fast, but it will be faster when
parse-partial-sexp is slow.
> and there's no need for it, given the existence of the
> comment-cache branch.
This branch has fundamentally the same caching problems as syntax-ppss
(i.e. in case of narrowing it may sometimes return what the user
intended and sometimes not), unsurprisingly. And it also suffers from
"Such calls are likely to transform side-effect free functions (such as
forward-comment) into side-effect encumbered functions".
And of course, it's a much larger change, with a redundant cache.
So, from where I sit I'd put it the other way around:
"there's no need for the comment-cache branch, given the existence of
this syntax-ppss-based simple patch".
Of course, you could argue that your patch works with cc-mode whereas
mine might not. It's no surprise, since you actually changed cc-mode
to accommodate your comment-cache, like you could do to accommodate
this patch.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-06 22:56 ` forward-comment and syntax-ppss Stefan Monnier
@ 2016-12-07 6:55 ` Andreas Röhler
2016-12-07 14:01 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Andreas Röhler @ 2016-12-07 6:55 UTC (permalink / raw)
To: emacs-devel
On 06.12.2016 23:56, Stefan Monnier wrote:
>> I'm not quite sure what you're saying here. Normally, a piece of
>> software not working properly is an extremely strong argument against it
>> (though it's an equally strong argument for fixing it).
> It's probably a strong argument against installing it willy nilly, yes.
>
>> What, exactly, is this patch intended to do? That's not obvious.
> It gets rid of some pathological cases where backward scanning currently
> may re-scan the whole buffer over and over again, each time from
> point-min, resulting in excruciatingly slow backward movement (these
> cases are fairly rare, luckily).
>
>> I think this would be a last resort, when nothing else can work. The
>> syntactic primitives should not have high level lisp code poking about
>> in their innards.
> I don't much care for such philosophy, because I consider syntax-ppss to
> be low-level Lisp code.
>
>> When it comes to narrowing a buffer, I think that the syntactic
>> context of a buffer position should not depend on whether the buffer
>> has been narrowed.
> As you know by now this is just one opinion, and other users of
> narrow-to-region beg to differ, and hence we will need to provide a way
> for users of narrow-to-region to communicate their intentions, since
> without that extra information, there can be no reliable caching of
> parsing.
>
>> If there are use cases this inconveniences, I think that those use
>> cases' use of narrowing is mistaken, and some other tool should be
>> used in its place.
> Fine by me.
>
>>> So I see this patch as an opportunity to improve syntax-ppss.
>> I don't think anybody will object to improving syntax-ppss. But, at the
>> same time, calling syntax-ppss from the C syntax routines should be
>> avoided, as far as possible. Such calls are likely to transform
>> side-effect free functions (such as forward-comment) into side-effect
>> encumbered functions, which is likely to make them less useful, and to
>> make Emacs more buggy, less flexible and more difficult to use.
> We've happily crossed that bridge already (tho not by calling
> syntax-ppss but by calling syntax-propertize).
>
>> And, just to emphasise, I think it's clear that syntax-ppss has no
>> business in forward-comment. syntax-ppss is slow (compared to the
>> current implementation of forward-comment),
> The patch I showed only uses syntax-ppss when we would otherwise
> re-parse the whole buffer, so it shouldn't slow things down too much.
>
> BTW, when writing syntax-ppss I did not really intend to "speed things
> up", but to get rid of pathologically slow behavior. In many cases
> calling parse-partial-sexp from point-min will be faster than calling
> syntax-ppss (mostly because syntax-ppss is all written in Elisp, so just
> the time to consult its cache can be longer than the time to compute the
> answer from scratch), but its behavior should stay "good enough" in
> a longish buffers (where calling parse-partial-sexp from point-min
> becomes excessive).
>
> IOW, syntax-ppss will be slower than parse-partial-sexp when
> parse-partial-sexp is fast, but it will be faster when
> parse-partial-sexp is slow.
>
>> and there's no need for it, given the existence of the
>> comment-cache branch.
> This branch has fundamentally the same caching problems as syntax-ppss
> (i.e. in case of narrowing it may sometimes return what the user
> intended and sometimes not), unsurprisingly. And it also suffers from
> "Such calls are likely to transform side-effect free functions (such as
> forward-comment) into side-effect encumbered functions".
> And of course, it's a much larger change, with a redundant cache.
>
> So, from where I sit I'd put it the other way around:
> "there's no need for the comment-cache branch, given the existence of
> this syntax-ppss-based simple patch".
>
> Of course, you could argue that your patch works with cc-mode whereas
> mine might not. It's no surprise, since you actually changed cc-mode
> to accommodate your comment-cache, like you could do to accommodate
> this patch.
>
>
> Stefan
>
syntax-ppss has major design issues. Just look at this comment:
;; Use the `syntax-begin-function' if available.
;; We could try using that function earlier, but:
;; - The result might not be 100% reliable, so it's better to use
;; the cache if available.
;; - The function might be slow.
;; - If this function almost always finds a safe nearby spot,
;; the cache won't be populated, so consulting it is cheap.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-07 6:55 ` Andreas Röhler
@ 2016-12-07 14:01 ` Stefan Monnier
2016-12-07 14:49 ` Andreas Röhler
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-07 14:01 UTC (permalink / raw)
To: emacs-devel
> syntax-ppss has major design issues. Just look at this comment:
> ;; Use the `syntax-begin-function' if available.
% grep syntax-begin-function **/*.el
[...]
lisp/emacs-lisp/syntax.el:(make-obsolete-variable 'syntax-begin-function nil "25.1")
[...]
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-07 14:01 ` Stefan Monnier
@ 2016-12-07 14:49 ` Andreas Röhler
2016-12-07 14:53 ` Clément Pit--Claudel
0 siblings, 1 reply; 299+ messages in thread
From: Andreas Röhler @ 2016-12-07 14:49 UTC (permalink / raw)
To: emacs-devel
On 07.12.2016 15:01, Stefan Monnier wrote:
>> syntax-ppss has major design issues. Just look at this comment:
>> ;; Use the `syntax-begin-function' if available.
> % grep syntax-begin-function **/*.el
> [...]
> lisp/emacs-lisp/syntax.el:(make-obsolete-variable 'syntax-begin-function nil "25.1")
> [...]
>
>
> Stefan
>
>
That was just an example. However, with this in mind, the file was never
ready for production. To be constructive: the major flaw in this context
are circular dependencies. Which was pointed at already. As long as
these circularities aren't dissolved, debugging might be an ordeal.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-07 14:49 ` Andreas Röhler
@ 2016-12-07 14:53 ` Clément Pit--Claudel
2016-12-07 22:04 ` Alan Mackenzie
0 siblings, 1 reply; 299+ messages in thread
From: Clément Pit--Claudel @ 2016-12-07 14:53 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 903 bytes --]
On 2016-12-07 09:49, Andreas Röhler wrote:
> That was just an example. However, with this in mind, the file was
> never ready for production. To be constructive: the major flaw in
> this context are circular dependencies. Which was pointed at already.
> As long as these circularities aren't dissolved, debugging might be
> an ordeal.
Can you give a concise description of what concretely is wrong with syntax-ppss, and how it could be made better? "it might be a bit tricky to debug" and "there's a scary comment somewhere in the source" are very weak reasons to call it "never ready for production", no?
Also: Isn't something that works fine on millions in of Emacs instances and has been in production for over 15 years "ready for production" by definition? I haven't seen many issues reported about syntax-ppss since I started following the bugs list a year ago…
Thanks!
Clément.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-07 14:53 ` Clément Pit--Claudel
@ 2016-12-07 22:04 ` Alan Mackenzie
2016-12-07 22:23 ` Clément Pit--Claudel
2016-12-08 2:32 ` Stefan Monnier
0 siblings, 2 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-07 22:04 UTC (permalink / raw)
To: Clément Pit--Claudel; +Cc: emacs-devel
Hello, Clément.
On Wed, Dec 07, 2016 at 09:53:26AM -0500, Clément Pit--Claudel wrote:
[ .... ]
> Also: Isn't something that works fine on millions in of Emacs
> instances and has been in production for over 15 years "ready for
> production" by definition? I haven't seen many issues reported about
> syntax-ppss since I started following the bugs list a year ago…
You haven't seen "many" issues about syntax-ppss? One is enough, if
it's bad enough.
Read up on bug #22983. Basically, syntax-ppss is fundamentally broken.
It won't work with narrowing of buffers, and delivers inconsistent
results which aren't in accordance with what the function's doc says it
does. This breakage isn't anything that can be fixed. The only way to
deal with it seems to be to remove syntax-ppss and replace it with a new
function.
> Thanks!
> Clément.
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-07 22:04 ` Alan Mackenzie
@ 2016-12-07 22:23 ` Clément Pit--Claudel
2016-12-08 19:31 ` Alan Mackenzie
2016-12-08 2:32 ` Stefan Monnier
1 sibling, 1 reply; 299+ messages in thread
From: Clément Pit--Claudel @ 2016-12-07 22:23 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 976 bytes --]
On 2016-12-07 17:04, Alan Mackenzie wrote:
> You haven't seen "many" issues about syntax-ppss? One is enough, if
> it's bad enough. Read up on bug #22983.
Thanks, Alan. I remember reading this long thread a while ago, though some of the stuff I had trouble following.
Stefan and you are in a better position than I am to assess the significance of that issue. The only data point that I can share is that I've found multiple packages to be broken by narrowing, to the point that I don't use narrowing any more. Instead, I use a less invasive solution with two invisible overlays covering 1..(beginning of narrowed region) and (end of narrowed region)..(buffer-size).
Some packages use (point-min) when they really mean 1 (or, if one prefers, (save-restriction (widen) (point-min))), and things start breaking in the presence of narrowing; so basically, I don't really expect things to work smoothly when I use narrowing, and so I don't use it.
Clément.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-07 22:04 ` Alan Mackenzie
2016-12-07 22:23 ` Clément Pit--Claudel
@ 2016-12-08 2:32 ` Stefan Monnier
2016-12-08 20:15 ` Alan Mackenzie
1 sibling, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-08 2:32 UTC (permalink / raw)
To: emacs-devel
> Read up on bug #22983. Basically, syntax-ppss is fundamentally broken.
> It won't work with narrowing of buffers, and delivers inconsistent
> results which aren't in accordance with what the function's doc says it
> does. This breakage isn't anything that can be fixed.
> The only way to deal with it seems to be to remove syntax-ppss and
> replace it with a new function.
And how would the new function not face the same breakage that can't be
fixed? As you should know, your comment-cache code suffers from the
exact same problem. It's a fundamental problem in caching parsing data
in the face of narrowing, given the fact that our narrowing primitive
don't say what is the intention behind it.
If you think your comment-cache works well (i.e. you consider its
failures to be caused by inappropriate use of narrow-to-region), then it's
trivial to "fix" syntax-ppss in the same way: just put a `widen` in it
and voilà: "fundamentally broken" solved.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-07 22:23 ` Clément Pit--Claudel
@ 2016-12-08 19:31 ` Alan Mackenzie
2016-12-08 19:54 ` Clément Pit--Claudel
0 siblings, 1 reply; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-08 19:31 UTC (permalink / raw)
To: Clément Pit--Claudel; +Cc: emacs-devel
Hello, Clément.
On Wed, Dec 07, 2016 at 05:23:42PM -0500, Clément Pit--Claudel wrote:
> On 2016-12-07 17:04, Alan Mackenzie wrote:
> > You haven't seen "many" issues about syntax-ppss? One is enough, if
> > it's bad enough. Read up on bug #22983.
> Thanks, Alan. I remember reading this long thread a while ago, though
> some of the stuff I had trouble following.
> Stefan and you are in a better position than I am to assess the
> significance of that issue. The only data point that I can share is
> that I've found multiple packages to be broken by narrowing, to the
> point that I don't use narrowing any more.
That's tragic. You clearly wanted to use narrowing, yet some bug, quite
likely to be #22983, made narrowing such a bad experience that you
stopped using it. Did you raise any bug reports for these failures?
> Instead, I use a less invasive solution with two invisible overlays
> covering 1..(beginning of narrowed region) and (end of narrowed
> region)..(buffer-size).
Narrowing isn't "invasive", whatever that means. It's such a natural and
useful feature that it's surprising other editors haven't got it too.
It's the pertinent bug (likely to be #22983) which has screwed up code
when narrowing is used which is at fault, not narrowing itself.
> Some packages use (point-min) when they really mean 1 (or, if one
> prefers, (save-restriction (widen) (point-min))), ....
I prefer a simple 1 myself.
> .... and things start breaking in the presence of narrowing; so
> basically, I don't really expect things to work smoothly when I use
> narrowing, and so I don't use it.
If things stopped working for me with narrowing, I'd be over the top with
rage. I use narrowing all the time. I think it's time to fix some bugs,
and hopefully persuade you that narrowing _can_ be used usefully.
> Clément.
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 19:31 ` Alan Mackenzie
@ 2016-12-08 19:54 ` Clément Pit--Claudel
2016-12-09 8:37 ` Eli Zaretskii
0 siblings, 1 reply; 299+ messages in thread
From: Clément Pit--Claudel @ 2016-12-08 19:54 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 2093 bytes --]
On 2016-12-08 14:31, Alan Mackenzie wrote:
>> Stefan and you are in a better position than I am to assess the
>> significance of that issue. The only data point that I can share is
>> that I've found multiple packages to be broken by narrowing, to the
>> point that I don't use narrowing any more.
>
> That's tragic. You clearly wanted to use narrowing, yet some bug, quite
> likely to be #22983, made narrowing such a bad experience that you
> stopped using it. Did you raise any bug reports for these failures?
These are not Emacs bugs. They are bugs in packages. The most prominent one, which affects my day to day work, is in Proof General (which I have since then become a co-maintainer of).
The problem is simple. Proof General locks portions of the source code that have been sent to an underlying interpreter/compiler. This locking is achieved by placing read-only overlays spanning from (point-min) to the current point, every time a new region is sent.
There are many such places; unfortunately, the use of (point-min) means that the overlays do not have the right starting point: they start from the beginning of the narrowed region.
It's hard to fix this issue, because uses of point-min don't carry sufficient semantic information: did the original author mean (point-min), or 1? There's no general way to tell but to read the surrounding code.
> Narrowing isn't "invasive", whatever that means. It's such a natural and
> useful feature that it's surprising other editors haven't got it too.
> It's the pertinent bug (likely to be #22983) which has screwed up code
> when narrowing is used which is at fault, not narrowing itself.
Sorry to have been unclear. Restricting the visible and editable portion of a buffer feels natural; but that's not what narrowing does: it makes it so that naively written code doesn't realize that there is anything outside of the narrowed region. That's what I meant by "invasive": as a package writer, I must always be aware that (point-min)…(point-max) may only cover a subset of the buffer.
Clément.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 2:32 ` Stefan Monnier
@ 2016-12-08 20:15 ` Alan Mackenzie
2016-12-08 20:34 ` Clément Pit--Claudel
` (3 more replies)
0 siblings, 4 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-08 20:15 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
Hello, Stefan.
On Wed, Dec 07, 2016 at 09:32:50PM -0500, Stefan Monnier wrote:
> > Read up on bug #22983. Basically, syntax-ppss is fundamentally broken.
> > It won't work with narrowing of buffers, and delivers inconsistent
> > results which aren't in accordance with what the function's doc says it
> > does. This breakage isn't anything that can be fixed.
> > The only way to deal with it seems to be to remove syntax-ppss and
> > replace it with a new function.
> And how would the new function not face the same breakage that can't be
> fixed?
It would have a sensible definition. "Returns the equivalent of
(parse-partial-sexp (point-min) POS)" is a silly definition which cannot
be coded up in any reasonable fashion. I think it should be replaced by
"returns the equivalent of (parse-partial-sexp 1 POS)", which can be
readily implemented. I propose, additionally, a new buffer local
variable which, if non-nil, will contain a buffer position to use in
place of the "1".
> As you should know, your comment-cache code suffers from the exact same
> problem.
Rubbish. Modulo any remaining bugs, it does what it says it does.
> It's a fundamental problem in caching parsing data in the face of
> narrowing, given the fact that our narrowing primitive don't say what
> is the intention behind it.
Nor should it. Our primitives `car', `cdr', `list', etc. also don't say
what the intention behind them is; they just work. Narrowing is what it
is. Simple and austere. Complexifying narrowing by introducing a
notion of "intention" would surely make it more difficult to use and
possibly foul things up massively.
> If you think your comment-cache works well (i.e. you consider its
> failures ....
What failures? If you've found any, please tell me specifically what, so
that I can fix it/them. Please stop spreading FUD like this.
> .... to be caused by inappropriate use of narrow-to-region), then it's
> trivial to "fix" syntax-ppss in the same way: just put a `widen` in it
> and voilà: "fundamentally broken" solved.
Rubbish again. "Returns the equivalent of (parse-partial-sexp
(point-min) POS)" cannot be implemented in any reasonable fashion, and
just putting a `widen' into the current code won't change that one bit.
As I said above, I think syntax-ppss should be retired and replaced by a
function which does parse-partial-sexp starting at 1 rather than
(point-min).
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 20:15 ` Alan Mackenzie
@ 2016-12-08 20:34 ` Clément Pit--Claudel
2016-12-08 21:25 ` Dmitry Gutov
2016-12-08 21:35 ` Stefan Monnier
2016-12-08 21:13 ` Stefan Monnier
` (2 subsequent siblings)
3 siblings, 2 replies; 299+ messages in thread
From: Clément Pit--Claudel @ 2016-12-08 20:34 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 333 bytes --]
On 2016-12-08 15:15, Alan Mackenzie wrote:
> As I said above, I think syntax-ppss should be retired and replaced by a
> function which does parse-partial-sexp starting at 1 rather than
> (point-min).
Doesn't the following do that?
(save-restriction (widen) (syntax-ppss))
(Or does this break the cache in some way?)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 20:15 ` Alan Mackenzie
2016-12-08 20:34 ` Clément Pit--Claudel
@ 2016-12-08 21:13 ` Stefan Monnier
2016-12-09 18:00 ` Alan Mackenzie
2016-12-08 21:24 ` Dmitry Gutov
2016-12-09 8:35 ` Eli Zaretskii
3 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-08 21:13 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: emacs-devel
>> And how would the new function not face the same breakage that can't be
>> fixed?
> It would have a sensible definition. "Returns the equivalent of
> (parse-partial-sexp (point-min) POS)" is a silly definition which cannot
> be coded up in any reasonable fashion.
Oh, it can be coded alright if we want it (it's not terribly hard,
actually). The choice was largely arbitrary and indeed the behavior is
currently an unreliable mix of point-min and 1.
Changing syntax-ppss to reliably return (parse-partial-sexp (point-min) POS)
or (parse-partial-sexp 1 POS), is not terribly hard. The issue is
mostly one of choosing between the two, where some cases want one and
others want the other, which is why so far I haven't actually decided
and left it simply unreliable when used in a narrowed buffer.
>> As you should know, your comment-cache code suffers from the exact same
>> problem.
> Rubbish. Modulo any remaining bugs, it does what it says it does.
Without your cache, the behavior of forward-comment is consistent with
a semantics like (parse-partial-sexp (point-min) POS), whereas with your
cache, forward-comment suddenly gets a new semantics similar to
(parse-partial-sexp 1 POS).
I don't claim the new behavior is worse, but it will introduce breakage
in some existing code (just like my syntax-ppss patch, of course, for
the same reasons). So does it "[do] what it says it does"?
> Nor should it. Our primitives `car', `cdr', `list', etc. also don't say
> what the intention behind them is; they just work. Narrowing is what it
> is. Simple and austere. Complexifying narrowing by introducing a
> notion of "intention" would surely make it more difficult to use and
> possibly foul things up massively.
Yet, you want to use (parse-partial-sexp 1 POS), which only makes sense
if you treat narrowing as having a particular intention.
> Rubbish again. "Returns the equivalent of (parse-partial-sexp
> (point-min) POS)" cannot be implemented in any reasonable fashion,
Of course it can. Here are two easy options:
- flush the cache is point-min is different from last time.
- index the cache with point-min.
I'm sure you too can come up with other solutions.
This is a non-issue. The real issue is to decide which behavior to get when.
> and just putting a `widen' into the current code won't change that
> one bit.
It would change the behavior to (parse-partial-sexp 1 POS) so of course,
the doc would be changed accordingly. Currently any code which relies
on syntax-ppss returning (parse-partial-sexp (point-min) POS) when
narrowed is broken anyway, so we can easily make such a change.
> As I said above, I think syntax-ppss should be retired and replaced by a
> function which does parse-partial-sexp starting at 1 rather than
> (point-min).
We're talking about changing 3 lines of doc and 2 lines of code.
Why would you describe such a trivial change as "retire and replace"?
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 20:15 ` Alan Mackenzie
2016-12-08 20:34 ` Clément Pit--Claudel
2016-12-08 21:13 ` Stefan Monnier
@ 2016-12-08 21:24 ` Dmitry Gutov
2016-12-08 21:47 ` Stefan Monnier
2016-12-09 19:07 ` Alan Mackenzie
2016-12-09 8:35 ` Eli Zaretskii
3 siblings, 2 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-08 21:24 UTC (permalink / raw)
To: Alan Mackenzie, Stefan Monnier; +Cc: emacs-devel
On 08.12.2016 22:15, Alan Mackenzie wrote:
> I propose, additionally, a new buffer local
> variable which, if non-nil, will contain a buffer position to use in
> place of the "1".
That's functionally equivalent to my proposed patch with
syntax-ppss-dont-widen, except for the case where that buffer local
value turns out to be higher than POS.
Can parse-partial-sexp parse backwards?
> Our primitives `car', `cdr', `list', etc. also don't say
> what the intention behind them is; they just work. Narrowing is what it
> is. Simple and austere.
All the primitives you've enumerated above work on simple values and
don't change the buffer state.
Narrowing is anything but austere.
> Complexifying narrowing by introducing a
> notion of "intention" would surely make it more difficult to use and
> possibly foul things up massively.
The reasons for it have been described plenty of times. And no, it will
be pretty trivial conceptually.
> As I said above, I think syntax-ppss should be retired and replaced by a
> function which does parse-partial-sexp starting at 1 rather than
> (point-min).
You keep repeating that word (replace), as though that solves anything.
If we introduce the new function and all of Emacs core switches to it,
and that function doesn't have an "escape hatch", the multi-mode
packages will be hung out to dry.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 20:34 ` Clément Pit--Claudel
@ 2016-12-08 21:25 ` Dmitry Gutov
2016-12-08 21:35 ` Stefan Monnier
1 sibling, 0 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-08 21:25 UTC (permalink / raw)
To: Clément Pit--Claudel, emacs-devel
On 08.12.2016 22:34, Clément Pit--Claudel wrote:
> Doesn't the following do that?
>
> (save-restriction (widen) (syntax-ppss))
>
> (Or does this break the cache in some way?)
Depending on what we consider a "broken cache", it could have been
broken before that call, by some other syntax-ppss caller that didn't
call `widen' beforehand.
Of course, if the cache isn't indexed with point-min, like Stefan
suggested in the other email.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 20:34 ` Clément Pit--Claudel
2016-12-08 21:25 ` Dmitry Gutov
@ 2016-12-08 21:35 ` Stefan Monnier
1 sibling, 0 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-08 21:35 UTC (permalink / raw)
To: emacs-devel
> Doesn't the following do that?
> (save-restriction (widen) (syntax-ppss))
Yes.
> (Or does this break the cache in some way?)
No, on the contrary, the cache currently works better if you widen
before calling syntax-ppss.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 21:24 ` Dmitry Gutov
@ 2016-12-08 21:47 ` Stefan Monnier
2016-12-09 7:39 ` Andreas Röhler
2016-12-09 19:07 ` Alan Mackenzie
1 sibling, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-08 21:47 UTC (permalink / raw)
To: emacs-devel
> Can parse-partial-sexp parse backwards?
No. Two reasons: one is that we don't have code that (attempts to)
do(es) that, the other is that it's fundamentally difficult/impossible
to do it reliably.
More specifically, the patch "forward-comment and syntax-ppss" is
specifically handling a case where we really don't know how to parse
backward: when moving backward, forward-comment generally tries to find
the beginning of a comment by "parsing backward", but in some cases it
really can't figure it out and (currently) resorts to calling
(parse-partial-sexp (point-min) POS). Since it may potentially do that
for every "comment-end" marker it finds (e.g. every \n in most major
modes), there are pathological cases where the performance in large
buffers can become really poor, hence the need for a cache.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 21:47 ` Stefan Monnier
@ 2016-12-09 7:39 ` Andreas Röhler
0 siblings, 0 replies; 299+ messages in thread
From: Andreas Röhler @ 2016-12-09 7:39 UTC (permalink / raw)
To: emacs-devel
On 08.12.2016 22:47, Stefan Monnier wrote:
>> Can parse-partial-sexp parse backwards?
> No. Two reasons: one is that we don't have code that (attempts to)
> do(es) that, the other is that it's fundamentally difficult/impossible
> to do it reliably.
>
> More specifically, the patch "forward-comment and syntax-ppss" is
> specifically handling a case where we really don't know how to parse
> backward: when moving backward, forward-comment generally tries to find
> the beginning of a comment by "parsing backward", but in some cases it
> really can't figure it out and (currently) resorts to calling
> (parse-partial-sexp (point-min) POS).
Another argument why syntax-ppss shouldn' t be used at all, thanks.
> Since it may potentially do that
> for every "comment-end" marker it finds (e.g. every \n in most major
> modes), there are pathological cases where the performance in large
> buffers can become really poor, hence the need for a cache.
>
>
Performance of parse-partial-sexp was very fast in its original design.
May our masters of circular dependencies have managed to slow it down?
At any case: the question is, if you need the info from point-min. If
yes, there is not other way than calling it from there. If not, don't
ask for it, give a nearer boundary.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 20:15 ` Alan Mackenzie
` (2 preceding siblings ...)
2016-12-08 21:24 ` Dmitry Gutov
@ 2016-12-09 8:35 ` Eli Zaretskii
2016-12-09 17:08 ` Alan Mackenzie
3 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-09 8:35 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: monnier, emacs-devel
> Date: Thu, 8 Dec 2016 20:15:17 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
>
> Rubbish. [...]
> [...]
> Please stop spreading FUD like this.
> [...]
> Rubbish again. [...]
Let's try to avoid such offensive language, okay?
TIA
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 19:54 ` Clément Pit--Claudel
@ 2016-12-09 8:37 ` Eli Zaretskii
0 siblings, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-09 8:37 UTC (permalink / raw)
To: Clément Pit--Claudel; +Cc: acm, emacs-devel
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Thu, 8 Dec 2016 14:54:34 -0500
> Cc: emacs-devel@gnu.org
>
> The problem is simple. Proof General locks portions of the source code that have been sent to an underlying interpreter/compiler. This locking is achieved by placing read-only overlays spanning from (point-min) to the current point, every time a new region is sent.
>
> There are many such places; unfortunately, the use of (point-min) means that the overlays do not have the right starting point: they start from the beginning of the narrowed region.
>
> It's hard to fix this issue, because uses of point-min don't carry sufficient semantic information: did the original author mean (point-min), or 1? There's no general way to tell but to read the surrounding code.
If the package doesn't use narrowing, it generally means 1, IMO.
That's a simple enough test that doesn't require reading the whole
code.
But I agree that this issue is not simple in general.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 8:35 ` Eli Zaretskii
@ 2016-12-09 17:08 ` Alan Mackenzie
2016-12-09 17:22 ` Eli Zaretskii
0 siblings, 1 reply; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-09 17:08 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: monnier, emacs-devel
Hello, Eli.
On Fri, Dec 09, 2016 at 10:35:41AM +0200, Eli Zaretskii wrote:
> > Date: Thu, 8 Dec 2016 20:15:17 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: emacs-devel@gnu.org
> >
> > Rubbish. [...]
> > [...]
> > Please stop spreading FUD like this.
> > [...]
> > Rubbish again. [...]
> Let's try to avoid such offensive language, okay?
Could you possibly suggest some alternatives which would be less
offensive, yet still express what was to be expressed?
> TIA
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 17:08 ` Alan Mackenzie
@ 2016-12-09 17:22 ` Eli Zaretskii
2016-12-09 21:55 ` Richard Stallman
0 siblings, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-09 17:22 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: monnier, emacs-devel
> Date: Fri, 9 Dec 2016 17:08:48 +0000
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
>
> > > Rubbish. [...]
> > > [...]
> > > Please stop spreading FUD like this.
> > > [...]
> > > Rubbish again. [...]
>
> > Let's try to avoid such offensive language, okay?
>
> Could you possibly suggest some alternatives which would be less
> offensive, yet still express what was to be expressed?
"This is incorrect" instead of "rubbish", "please don't exaggerate"
instead of "FUD", etc. I'm sure your command of English is much
better than mine ;-)
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 21:13 ` Stefan Monnier
@ 2016-12-09 18:00 ` Alan Mackenzie
2016-12-09 18:47 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-09 18:00 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
Hello, Stefan.
On Thu, Dec 08, 2016 at 04:13:04PM -0500, Stefan Monnier wrote:
> >> And how would the new function not face the same breakage that can't be
> >> fixed?
> > It would have a sensible definition. "Returns the equivalent of
> > (parse-partial-sexp (point-min) POS)" is a silly definition which cannot
> > be coded up in any reasonable fashion.
> Oh, it can be coded alright if we want it (it's not terribly hard,
> actually). The choice was largely arbitrary and indeed the behavior is
> currently an unreliable mix of point-min and 1.
Indeed.
> Changing syntax-ppss to reliably return (parse-partial-sexp (point-min) POS)
> or (parse-partial-sexp 1 POS), is not terribly hard. The issue is
> mostly one of choosing between the two, where some cases want one and
> others want the other, which is why so far I haven't actually decided
> and left it simply unreliable when used in a narrowed buffer.
Thanks for the explanation for leaving it unfixed, which I understand
and accept. Why don't you chose both (i.e. two distinct functions)? :-)
> >> As you should know, your comment-cache code suffers from the exact same
> >> problem.
> > Rubbish. Modulo any remaining bugs, it does what it says it does.
> Without your cache, the behavior of forward-comment is consistent with
> a semantics like (parse-partial-sexp (point-min) POS), whereas with your
> cache, forward-comment suddenly gets a new semantics similar to
> (parse-partial-sexp 1 POS).
OK, maybe. I'm struggling a bit with your analogies. But I think I see
what you're trying to say.
> I don't claim the new behavior is worse, but it will introduce breakage
> in some existing code (just like my syntax-ppss patch, of course, for
> the same reasons). So does it "[do] what it says it does"?
Yes. It certainly doesn't fail in anything like the fashion that
syntax-ppss does. Modulo any remaining bugs.
> > Nor should it. Our primitives `car', `cdr', `list', etc. also don't say
> > what the intention behind them is; they just work. Narrowing is what it
> > is. Simple and austere. Complexifying narrowing by introducing a
> > notion of "intention" would surely make it more difficult to use and
> > possibly foul things up massively.
> Yet, you want to use (parse-partial-sexp 1 POS), which only makes sense
> if you treat narrowing as having a particular intention.
No. It makes sense, full stop. If it doesn't, then the notion of
narrowing doesn't make sense either. Plainly it does. The definition
of a primitive cannot be dependent upon the psychology of the hacker
using it.
There is nothing in the notion of narrowing which suggests that it
should change the syntactic context of any text - if a certain buffer
position is inside a string, it is inside the string; it doesn't
suddenly become outside the string by virtue of where the buffer is
narrowed. Were that not the case, the notion "inside a string" would
become meaningless.
> > Rubbish again. "Returns the equivalent of (parse-partial-sexp
> > (point-min) POS)" cannot be implemented in any reasonable fashion,
> Of course it can. Here are two easy options:
> - flush the cache if point-min is different from last time.
Likely, the cache would be flushed so much it wouldn't really serve much
purpose.
> - index the cache with point-min.
I don't understand what you mean by this.
> I'm sure you too can come up with other solutions.
Maybe I could.
> This is a non-issue. The real issue is to decide which behavior to get when.
> > and just putting a `widen' into the current code won't change that
> > one bit.
> It would change the behavior to (parse-partial-sexp 1 POS) so of course,
> the doc would be changed accordingly.
You would then have a different function. Something I've been
suggesting for a long while.
> Currently any code which relies on syntax-ppss returning
> (parse-partial-sexp (point-min) POS) when narrowed is broken anyway,
> so we can easily make such a change.
No, that code is not broken. It is syntax-ppss that is broken. If you
replace syntax-ppss with a different function (as I have been
advocating), that is going to break some code which uses syntax-ppss.
> > As I said above, I think syntax-ppss should be retired and replaced by a
> > function which does parse-partial-sexp starting at 1 rather than
> > (point-min).
> We're talking about changing 3 lines of doc and 2 lines of code.
> Why would you describe such a trivial change as "retire and replace"?
Because we're talking about a different function which does a job
different from what syntax-ppss is specified to do. That is not a minor
change, regardless of how few lines of code and documentation need to be
changed.
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 18:00 ` Alan Mackenzie
@ 2016-12-09 18:47 ` Stefan Monnier
2016-12-09 22:33 ` Clément Pit--Claudel
2016-12-10 18:39 ` Andreas Röhler
0 siblings, 2 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-09 18:47 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: emacs-devel
>> > Nor should it. Our primitives `car', `cdr', `list', etc. also don't say
>> > what the intention behind them is; they just work. Narrowing is what it
>> > is. Simple and austere. Complexifying narrowing by introducing a
>> > notion of "intention" would surely make it more difficult to use and
>> > possibly foul things up massively.
>> Yet, you want to use (parse-partial-sexp 1 POS), which only makes sense
>> if you treat narrowing as having a particular intention.
> No. It makes sense, full stop. If it doesn't, then the notion of
> narrowing doesn't make sense either.
What is your understanding of "narrowing"? In my book, what it means is
"make the text outside of the region inaccessible". So why should
syntax-ppss be granted an exception (note: I don't disagree that it can
make sense to grant an exception for it, but that granting it an
exception only makes sense based on some expected usage context of
narrowing, i.e. based on the intention behind its use).
> There is nothing in the notion of narrowing which suggests that it
> should change the syntactic context of any text - if a certain buffer
> position is inside a string, it is inside the string; it doesn't
> suddenly become outside the string by virtue of where the buffer is
> narrowed. Were that not the case, the notion "inside a string" would
> become meaningless.
Let's take sm-c-mode as an example: it marks CPP preprocessor elements
as being comments. Yet, it also wants to know if there are strings
inside that "comment", so it narrows to the "comment" and then
parses the result as a chunk of C code.
[ Whether this approach is good or bad is beside the point: this
is just an example of narrowing, where the context should not affect
what is considered "inside a string" or not. ]
Other examples come from the context of
multiple-major-modes-in-a-buffer.
Yet others come from the use of narrowing in things like Info-mode and
Rmail (where a large buffer contains many "pages" and that's an
implementation detail: the user only cares about the pages and not
whether they're kept in a single buffer/file or not).
>> Of course it can. Here are two easy options:
>> - flush the cache if point-min is different from last time.
> Likely, the cache would be flushed so much it wouldn't really serve much
> purpose.
Could be. Then again, maybe not. After all, you're saying that the
only "meaningful" use is when the scan starts at 1, so the overwhelming
majority of uses should be in a widened buffer (or are errors that need
to be fixed).
Only experimentation could say that. But FWIW, my gut feeling agrees
with yours.
>> - index the cache with point-min.
> I don't understand what you mean by this.
The cache is currently (conceptually) a function which takes a buffer
position and returns to you an earlier position, along with the state at
that position.
Change this so that the function takes additionally `point-min` as
argument, so it can keep different states for the same positions,
depending on the `point-min` that was used to compute that state.
Or more concretely, replace `syntax-ppss-cache` with (assq (point-min)
syntax-ppss-cache-alist).
>> Currently any code which relies on syntax-ppss returning
>> (parse-partial-sexp (point-min) POS) when narrowed is broken anyway,
>> so we can easily make such a change.
> No, that code is not broken.
Together with the existing implementation of syntax-ppss, it's very
unlikely to work reliably.
> It is syntax-ppss that is broken. If you replace syntax-ppss with
> a different function (as I have been advocating), that is going to
> break some code which uses syntax-ppss.
I think that any such breakage could be reproduced with the current
implementation of syntax-ppss, which would mean that the problem is not
a new problem introduced by the "different" function.
>> We're talking about changing 3 lines of doc and 2 lines of code.
>> Why would you describe such a trivial change as "retire and replace"?
> Because we're talking about a different function which does a job
> different from what syntax-ppss is specified to do. That is not a minor
> change, regardless of how few lines of code and documentation need to be
> changed.
I consider the behavior of syntax-ppss when point-min!=1 to be
currently undefined (and indeed, it's unreliable).
So what we're discussing is how to refine/augment/improve the behavior,
rather than introducing a brand new behavior.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-08 21:24 ` Dmitry Gutov
2016-12-08 21:47 ` Stefan Monnier
@ 2016-12-09 19:07 ` Alan Mackenzie
2016-12-09 23:30 ` Dmitry Gutov
1 sibling, 1 reply; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-09 19:07 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Stefan Monnier, emacs-devel
Hello, Dmitry.
On Thu, Dec 08, 2016 at 11:24:12PM +0200, Dmitry Gutov wrote:
> On 08.12.2016 22:15, Alan Mackenzie wrote:
> > I propose, additionally, a new buffer local variable which, if
> > non-nil, will contain a buffer position to use in place of the "1".
> That's functionally equivalent to my proposed patch with
> syntax-ppss-dont-widen, except for the case where that buffer local
> value turns out to be higher than POS.
That could be. But a variable called wyntax-ppss-dont-widen has the
caller explicitly messing around with the flowchart of syntax-ppss,
which is not a good thing. I think my proposed new local variable is
better for this reason.
> Can parse-partial-sexp parse backwards?
No, not at all, and it's unlikely ever to be able to do so.
> > Our primitives `car', `cdr', `list', etc. also don't say
> > what the intention behind them is; they just work. Narrowing is what it
> > is. Simple and austere.
> All the primitives you've enumerated above work on simple values and
> don't change the buffer state.
Narrowing doesn't change the buffer state either, beyond what it
explicitly says it does. In particular, it doesn't change things like
whether a particular buffer position is inside a string.
> Narrowing is anything but austere.
I disagree. I think there are attempts afoot to make it non-austere,
and that these attempts are mistaken in principle.
> > Complexifying narrowing by introducing a notion of "intention" would
> > surely make it more difficult to use and possibly foul things up
> > massively.
> The reasons for it have been described plenty of times. And no, it will
> be pretty trivial conceptually.
It tends to be trivial to add stuff on. Much easier than getting rid of
stuff. And that extra stuff might well be fat rather than muscle.
> > As I said above, I think syntax-ppss should be retired and replaced by a
> > function which does parse-partial-sexp starting at 1 rather than
> > (point-min).
> You keep repeating that word (replace), as though that solves anything.
It would solve bug #22983.
> If we introduce the new function and all of Emacs core switches to it,
> and that function doesn't have an "escape hatch", the multi-mode
> packages will be hung out to dry.
I wouldn't want that to happen. Does the suggested new buffer local
variable (to specify the lower bound in the notional partial-parse-sexp)
not provide this escape hatch?
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 17:22 ` Eli Zaretskii
@ 2016-12-09 21:55 ` Richard Stallman
0 siblings, 0 replies; 299+ messages in thread
From: Richard Stallman @ 2016-12-09 21:55 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: acm, monnier, emacs-devel
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> "This is incorrect" instead of "rubbish", "please don't exaggerate"
> instead of "FUD", etc.
Those are good suggestions for replacements.
We need to work together, so it's important to make an effort
to disagree without personal harshness.
--
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 18:47 ` Stefan Monnier
@ 2016-12-09 22:33 ` Clément Pit--Claudel
2016-12-09 23:09 ` Alan Mackenzie
2016-12-10 6:56 ` Eli Zaretskii
2016-12-10 18:39 ` Andreas Röhler
1 sibling, 2 replies; 299+ messages in thread
From: Clément Pit--Claudel @ 2016-12-09 22:33 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 2448 bytes --]
On 2016-12-09 13:47, Stefan Monnier wrote:
>> There is nothing in the notion of narrowing which suggests that it
>> should change the syntactic context of any text - if a certain buffer
>> position is inside a string, it is inside the string; it doesn't
>> suddenly become outside the string by virtue of where the buffer is
>> narrowed. Were that not the case, the notion "inside a string" would
>> become meaningless.
> Let's take sm-c-mode as an example: it marks CPP preprocessor elements
> as being comments. Yet, it also wants to know if there are strings
> inside that "comment", so it narrows to the "comment" and then
> parses the result as a chunk of C code.
>
> Other examples come from the context of multiple-major-modes-in-a-buffer.
I'm not directly involved in this discussion, but thanks to both of you for these explanations. I didn't clearly understand the distinction at hand up to this point.
Alan, is it valid to say that you view narrowing as a convenient editing feature that lets you restrict operations to a subsection of the buffer? Something like a "window" over the buffer (or maybe "stencil" would be a better term?), which doesn't change any of semantic meaning, but just conveniently restricts motion?
In that sense, your vision of narrowing is similar to my applying two overlays to make the beginning of the buffer invisible up to a point, and the end of the buffer invisible starting from a point, right?
Stefan, am I correct in thinking that the narrowing that you mention above is of a different nature, in the sense that if I narrow a buffer to the range a…b it's just as if I had copied that portion of the buffer to a totally separate, disconnected buffer (possibly running a different mode), and the text would get copied back into the original buffer when I widen?
Maybe the solution is to just give these two things a different name? It seems that in the first case we're narrowing to a region without forgetting about the context — let's call this "context-aware narrowing", or "focusing". In the second case we're narrowing to a region and forgetting about the context entirely — let's call this "context-oblivious narrowing", or "sub-buffer editing".
Context-aware narrowing allows one to work on just one function easily; Context-oblivious narrowing makes things like http://demonastery.org/2013/04/emacs-narrow-to-region-indirect/ possible.
Cheers,
Clément.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 22:33 ` Clément Pit--Claudel
@ 2016-12-09 23:09 ` Alan Mackenzie
2016-12-10 6:56 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-09 23:09 UTC (permalink / raw)
To: Clément Pit--Claudel; +Cc: emacs-devel
Hello, Clément.
On Fri, Dec 09, 2016 at 05:33:09PM -0500, Clément Pit--Claudel wrote:
> On 2016-12-09 13:47, Stefan Monnier wrote:
> >> There is nothing in the notion of narrowing which suggests that it
> >> should change the syntactic context of any text - if a certain
> >> buffer position is inside a string, it is inside the string; it
> >> doesn't suddenly become outside the string by virtue of where the
> >> buffer is narrowed. Were that not the case, the notion "inside a
> >> string" would become meaningless.
> > Let's take sm-c-mode as an example: it marks CPP preprocessor
> > elements as being comments. Yet, it also wants to know if there are
> > strings inside that "comment", so it narrows to the "comment" and
> > then parses the result as a chunk of C code.
> > Other examples come from the context of
> > multiple-major-modes-in-a-buffer.
> I'm not directly involved in this discussion, but thanks to both of you
> for these explanations. I didn't clearly understand the distinction at
> hand up to this point.
> Alan, is it valid to say that you view narrowing as a convenient
> editing feature that lets you restrict operations to a subsection of
> the buffer? Something like a "window" over the buffer (or maybe
> "stencil" would be a better term?), which doesn't change any of
> semantic meaning, but just conveniently restricts motion? In that
> sense, your vision of narrowing is similar to my applying two overlays
> to make the beginning of the buffer invisible up to a point, and the
> end of the buffer invisible starting from a point, right?
Yes, I think that's a very good summary.
> Stefan, am I correct in thinking that the narrowing that you mention
> above is of a different nature, in the sense that if I narrow a buffer
> to the range a…b it's just as if I had copied that portion of the
> buffer to a totally separate, disconnected buffer (possibly running a
> different mode), and the text would get copied back into the original
> buffer when I widen?
I'm not Stefan, but that's pretty much my understanding of his
understanding.
> Maybe the solution is to just give these two things a different name?
> It seems that in the first case we're narrowing to a region without
> forgetting about the context — let's call this "context-aware
> narrowing", or "focusing". In the second case we're narrowing to a
> region and forgetting about the context entirely — let's call this
> "context-oblivious narrowing", or "sub-buffer editing".
No, I don't think this is right. The narrowing itself is simply
narrowing. It's what you do on the narrowed buffer that makes the
difference. In @dfn{ACM-narrowed-processing), the parse state of a
position would be determined by (parse-partial-sexp 1 POS). In
@dfn{SM-narrowed-processing} it would be determined by
(parse-partial-sexp (point-min) POS).
I think the main conceptual mistake people are making at the moment is
attributing to the act of narrowing, differences in attitude to the
processing done on narrowed buffers.
> Context-aware narrowing allows one to work on just one function easily;
> Context-oblivious narrowing makes things like
> http://demonastery.org/2013/04/emacs-narrow-to-region-indirect/
> possible.
> Cheers,
> Clément.
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 19:07 ` Alan Mackenzie
@ 2016-12-09 23:30 ` Dmitry Gutov
2016-12-11 0:24 ` Stefan Monnier
2016-12-11 10:17 ` Alan Mackenzie
0 siblings, 2 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-09 23:30 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
On 09.12.2016 21:07, Alan Mackenzie wrote:
> That could be. But a variable called wyntax-ppss-dont-widen has the
> caller explicitly messing around with the flowchart of syntax-ppss,
> which is not a good thing. I think my proposed new local variable is
> better for this reason.
It's better in one sense and worse in another. Since parse-partial-sexp
can't parse backwards, you're leaving a whole range of values of POS on
which syntax-ppss will have to just raise an error.
> Narrowing doesn't change the buffer state either, beyond what it
> explicitly says it does.
It changes state and, as such, behavior of a huge number of functions
(both core and third-party ones).
> In particular, it doesn't change things like
> whether a particular buffer position is inside a string.
Not every buffer position is unequivocally "inside a strong" or not.
Take this example.
// Here's a string: "abc".
Is abc inside a string? That depends on what exactly we mean by that,
for a given use case.
> It tends to be trivial to add stuff on. Much easier than getting rid of
> stuff. And that extra stuff might well be fat rather than muscle.
That's just rhetoric.
>> You keep repeating that word (replace), as though that solves anything.
>
> It would solve bug #22983.
No, it won't. For instance, because the bug is complaining about the
behavior of syntax-ppss (see, I can play these word games too).
More importantly, any change you would make to the "new" function you
could make to syntax-ppss instead.
>> and that function doesn't have an "escape hatch", the multi-mode
>> packages will be hung out to dry.
>
> I wouldn't want that to happen. Does the suggested new buffer local
> variable (to specify the lower bound in the notional partial-parse-sexp)
> not provide this escape hatch?
It would (if we're talking about changing syntax-ppss, and not anything
else). It does come with its own weird semantics, however, mentioned at
the top of this message.
Further, the good part of this proposal is more or less equivalent to
"indexing syntax-ppss cache by point-min" as proposed by Stefan.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 22:33 ` Clément Pit--Claudel
2016-12-09 23:09 ` Alan Mackenzie
@ 2016-12-10 6:56 ` Eli Zaretskii
1 sibling, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-10 6:56 UTC (permalink / raw)
To: Clément Pit--Claudel; +Cc: emacs-devel
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Fri, 9 Dec 2016 17:33:09 -0500
>
> Alan, is it valid to say that you view narrowing as a convenient editing feature that lets you restrict operations to a subsection of the buffer? Something like a "window" over the buffer (or maybe "stencil" would be a better term?), which doesn't change any of semantic meaning, but just conveniently restricts motion?
> In that sense, your vision of narrowing is similar to my applying two overlays to make the beginning of the buffer invisible up to a point, and the end of the buffer invisible starting from a point, right?
It is similar, but not equivalent, because neither search commands nor
the display engine will stop where the invisible text starts.
> Stefan, am I correct in thinking that the narrowing that you mention above is of a different nature, in the sense that if I narrow a buffer to the range a…b it's just as if I had copied that portion of the buffer to a totally separate, disconnected buffer (possibly running a different mode), and the text would get copied back into the original buffer when I widen?
This is the only meaning of narrowing that exists in Emacs as of now.
> Maybe the solution is to just give these two things a different name?
The other thing would have to be implemented first. And if it is
indistinguishable from invisible text, then we already have it, just
under a different name ;-)
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 18:47 ` Stefan Monnier
2016-12-09 22:33 ` Clément Pit--Claudel
@ 2016-12-10 18:39 ` Andreas Röhler
1 sibling, 0 replies; 299+ messages in thread
From: Andreas Röhler @ 2016-12-10 18:39 UTC (permalink / raw)
To: emacs-devel
On 09.12.2016 19:47, Stefan Monnier wrote:
>
> Let's take sm-c-mode as an example: it marks CPP preprocessor elements
> as being comments. Yet, it also wants to know if there are strings
> inside that "comment", so it narrows to the "comment" and then
> parses the result as a chunk of C code.
> [ Whether this approach is good or bad is beside the point: this
> is just an example of narrowing, where the context should not affect
> what is considered "inside a string" or not. ]
>
IMHO that the kind of proceeding to avoid, unless complexity and trouble
grow likewise.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 23:30 ` Dmitry Gutov
@ 2016-12-11 0:24 ` Stefan Monnier
2016-12-11 10:17 ` Alan Mackenzie
1 sibling, 0 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-11 0:24 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Alan Mackenzie, emacs-devel
> It's better in one sense and worse in another. Since parse-partial-sexp
> can't parse backwards, you're leaving a whole range of values of POS on
> which syntax-ppss will have to just raise an error.
I don't think it's a big deal: such a situation would necessarily be an
error elsewhere, so signaling an error would be the right thing to do anyway.
Admittedly, an alternative would be to provide syntax-ppss with a bunch
of "forced data", i.e. an alist of (POS . STATE) where syntax-ppss
should start parsing from the nearest earlier POS and use the
corresponding STATE as starting state. But it's probably not worth
the trouble (and might even be slightly problematic for (syntax-ppss
POS) since some callers would want it to return STATE while others might
like to know "the state computed from the earlier POS&STATE").
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-09 23:30 ` Dmitry Gutov
2016-12-11 0:24 ` Stefan Monnier
@ 2016-12-11 10:17 ` Alan Mackenzie
2016-12-11 10:40 ` Andreas Röhler
2016-12-16 1:33 ` Dmitry Gutov
1 sibling, 2 replies; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-11 10:17 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Stefan Monnier, emacs-devel
Hello, Dmitry.
On Sat, Dec 10, 2016 at 01:30:24AM +0200, Dmitry Gutov wrote:
> On 09.12.2016 21:07, Alan Mackenzie wrote:
> > That could be. But a variable called syntax-ppss-dont-widen has the
> > caller explicitly messing around with the flowchart of syntax-ppss,
> > which is not a good thing. I think my proposed new local variable is
> > better for this reason.
> It's better in one sense and worse in another. Since parse-partial-sexp
> can't parse backwards, you're leaving a whole range of values of POS on
> which syntax-ppss will have to just raise an error.
I don't think this is an issue. (syntax-ppss-1 POS), when
new-local-variable is non-nil simulates (parse-partial-sexp
new-local-variable POS), and the latter would throw an error for
precisely the same values of POS as the former.
> > Narrowing doesn't change the buffer state either, beyond what it
> > explicitly says it does.
> It changes state and, as such, behavior of a huge number of functions
> (both core and third-party ones).
Maybe, but these changes are the direct desired effect of narrowing.
They're not nasty unforeseen side effects.
> > In particular, it doesn't change things like
> > whether a particular buffer position is inside a string.
> Not every buffer position is unequivocally "inside a strong" or not.
> Take this example.
> // Here's a string: "abc".
> Is abc inside a string? That depends on what exactly we mean by that,
> for a given use case.
That may be the case, but that has nothing to do with what I said ("In
particular, it doesn't change ...."). Whether abc is inside a string or
not doesn't vary by the narrowing. It's determined by other factors.
> > It tends to be trivial to add stuff on. Much easier than getting rid of
> > stuff. And that extra stuff might well be fat rather than muscle.
> That's just rhetoric.
I don't think so. In particular, I was thinking of what I think is
called "hard widening". I think it would be a mistake to add this
feature. At the moment, narrowing is a strong, direct, simple facility
- it does what it says it does and no more. Were "hard widening" to be
introduced, we could easily end up with a confused mess of `widen' and
`hard-widen' through the code, with nobody being quite sure which one to
use and why. There might be lots of depressing errors where the wrong
one is used.
> >> You keep repeating that word (replace), as though that solves anything.
> > It would solve bug #22983.
> No, it won't. For instance, because the bug is complaining about the
> behavior of syntax-ppss (see, I can play these word games too).
Oh dear, Dmitry! ;-)
> More importantly, any change you would make to the "new" function you
> could make to syntax-ppss instead.
The new function would be a different function, i.e. it would have a
different definition. Just for clarity, it would not be written from
scratch, it would be based on the current syntax-ppss.
> >> and that function doesn't have an "escape hatch", the multi-mode
> >> packages will be hung out to dry.
> > I wouldn't want that to happen. Does the suggested new buffer local
> > variable (to specify the lower bound in the notional partial-parse-sexp)
> > not provide this escape hatch?
> It would (if we're talking about changing syntax-ppss, and not anything
> else). It does come with its own weird semantics, however, mentioned at
> the top of this message.
> Further, the good part of this proposal is more or less equivalent to
> "indexing syntax-ppss cache by point-min" as proposed by Stefan.
I think I agree here. But "good part" implies the existence of a "less
good part". What, in your view, constitues this "less good part"?
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-11 10:17 ` Alan Mackenzie
@ 2016-12-11 10:40 ` Andreas Röhler
2016-12-16 1:33 ` Dmitry Gutov
1 sibling, 0 replies; 299+ messages in thread
From: Andreas Röhler @ 2016-12-11 10:40 UTC (permalink / raw)
To: emacs-devel; +Cc: Alan Mackenzie
On 11.12.2016 11:17, Alan Mackenzie wrote:
> Hello, Dmitry.
>
>> That's just rhetoric.
> I don't think so. In particular, I was thinking of what I think is
> called "hard widening". I think it would be a mistake to add this
> feature.
+1
> At the moment, narrowing is a strong, direct, simple facility
> - it does what it says it does and no more. Were "hard widening" to be
> introduced, we could easily end up with a confused mess of `widen' and
> `hard-widen'
+999 :)
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 21:22 ` Daniel Colascione
2016-12-02 22:06 ` Eli Zaretskii
2016-12-02 23:15 ` Karl Fogel
@ 2016-12-15 14:28 ` Philippe Vaucher
2017-10-18 23:36 ` Kaushal Modi
2 siblings, 1 reply; 299+ messages in thread
From: Philippe Vaucher @ 2016-12-15 14:28 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Karl Fogel, Emacs developers
[-- Attachment #1: Type: text/plain, Size: 292 bytes --]
On Fri, Dec 2, 2016 at 10:22 PM, Daniel Colascione <dancol@dancol.org>
wrote:
> FWIW, I'll put my patch on a branch this weekend.
>
Sorry if I missed it, but was this done? I can't find a relevant branch by
looking at https://github.com/emacs-mirror/emacs/branches/active
Thanks,
Philippe
[-- Attachment #2: Type: text/html, Size: 735 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-11 10:17 ` Alan Mackenzie
2016-12-11 10:40 ` Andreas Röhler
@ 2016-12-16 1:33 ` Dmitry Gutov
2016-12-16 2:06 ` Drew Adams
2016-12-16 20:06 ` Alan Mackenzie
1 sibling, 2 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-16 1:33 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
On 11.12.2016 12:17, Alan Mackenzie wrote:
> I don't think this is an issue. (syntax-ppss-1 POS), when
> new-local-variable is non-nil simulates (parse-partial-sexp
> new-local-variable POS), and the latter would throw an error for
> precisely the same values of POS as the former.
Indeed, but this way the first argument will be implicit, and, unlike
point-min, often non-obvious to the code calling syntax-ppss. It's easy
to produce a buffer position lower than the arbitrary value of the new
variable. It's harder to produce a buffer position lower than point-min.
Anyway, it's a minor issue. A good error message should be sufficient.
> Maybe, but these changes are the direct desired effect of narrowing.
> They're not nasty unforeseen side effects.
"nasty" or "unforeseen" is beside the point. Like described previously,
some code doesn't really know how to behave in the face of narrowing if
the user's intent that resulted in narrowing, isn't known.
>>> In particular, it doesn't change things like
>>> whether a particular buffer position is inside a string.
>
>> Not every buffer position is unequivocally "inside a strong" or not.
>
>> Take this example.
>
>> // Here's a string: "abc".
>
>> Is abc inside a string? That depends on what exactly we mean by that,
>> for a given use case.
>
> That may be the case, but that has nothing to do with what I said ("In
> particular, it doesn't change ...."). Whether abc is inside a string or
> not doesn't vary by the narrowing. It's determined by other factors.
It can be determined by the narrowing (i.e. a function named
`inside-a-string?' could confidently know whether to rely on the current
narrowing or ignore it, to produce its result, if narrowing were
annotated with intent).
> In particular, I was thinking of what I think is
> called "hard widening". I think it would be a mistake to add this
> feature. At the moment, narrowing is a strong, direct, simple facility
> - it does what it says it does and no more.
From where I'm standing, the things you use it for (hide buffer
contents) can be implemented in a much better fashion using overlays
with invisibility (in fact, one such implementation exists already),
which don't introduce the difficulties of changing point-min and
point-max. Thus, without making things harder for many other facilities.
> Were "hard widening" to be
> introduced, we could easily end up with a confused mess of `widen' and
> `hard-widen' through the code, with nobody being quite sure which one to
> use and why. There might be lots of depressing errors where the wrong
> one is used.
These are issues of design and documentation. The previous discussions
of hard-widen didn't come up with any real conceptual problems. Just
with low-level implementation ones.
>> More importantly, any change you would make to the "new" function you
>> could make to syntax-ppss instead.
>
> The new function would be a different function, i.e. it would have a
> different definition. Just for clarity, it would not be written from
> scratch, it would be based on the current syntax-ppss.
And then it would be utterly useless for third-party code, for years to
come. Further, if it uses a separate variable for its cache, and the new
function is used in the code, while the old function continues to be
used by third-party code (and also stable packages like Gnus and
CC-Mode) for compatibility reasons, we'll have two syntax caches that
will interact in fascinating, difficult to predict ways.
Am I interpreting your intention correctly?
>> Further, the good part of this proposal is more or less equivalent to
>> "indexing syntax-ppss cache by point-min" as proposed by Stefan.
>
> I think I agree here. But "good part" implies the existence of a "less
> good part". What, in your view, constitues this "less good part"?
Well... both your and Stefan's proposals will end up actually having to
maintain the index of syntax-ppss caches, and having to invalidate those
values somehow. I'm sure it's possible to arrive at a satisfactory
algorithm (like only keeping around two values, for point-min equal to 1
and the most recent other value), but there must be use cases which some
other invalidation approach would be better.
With syntax-ppss-dont-widen, the caller code can take the responsibility
of maintaining the values of syntax-ppss-cache and syntax-ppss-last. And
it turns out, it's rather easy to bind them to nil around the same
pieces of code that starts with (save-restriction (narrow-to-region
...)). Which currently seems to give the same result as the idea in the
previous paragraph, with much less code.
In short, letting the caller code control cache invalidation has its
benefits.
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-16 1:33 ` Dmitry Gutov
@ 2016-12-16 2:06 ` Drew Adams
2016-12-16 8:12 ` Eli Zaretskii
` (3 more replies)
2016-12-16 20:06 ` Alan Mackenzie
1 sibling, 4 replies; 299+ messages in thread
From: Drew Adams @ 2016-12-16 2:06 UTC (permalink / raw)
To: Dmitry Gutov, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
> From where I'm standing, the things you use it for (hide
> buffer contents)
No. Narrowing is not used to "hide buffer contents". That's not
the point of narrowing, even if it is one effect. It's used to
make some buffer contents inaccessible to certain (many, typical,
ordinary) operations. It is not just you who does not see parts
of the buffer; it is also Emacs features that do not perceive
them.
> can be implemented in a much better fashion using overlays
> with invisibility (in fact, one such implementation exists
> already),
No. It's not about invisibility. Overlays do not offer
anything that comes close to replacing or simulating the
effect of narrowing.
> which don't introduce the difficulties of changing point-min and
> point-max.
What difficulties? What's hard about changing those?
That's the essential design of narrowing - the entire point
of it: change `point-min' and `point-max'. If you're not
doing that then you are not in any way giving users (and
code) any kind of a replacement for narrowing. And if you
are doing that then, well, you're narrowing the buffer.
At the base, narrowing is nothing more than binding the
meanings/behavior of `point-min' and `point-max'.
> Thus, without making things harder for many other facilities.
No one has given an example of how narrowing makes things
hard "for many other facilities" - or even for one facility.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 2:06 ` Drew Adams
@ 2016-12-16 8:12 ` Eli Zaretskii
2016-12-16 12:32 ` Dmitry Gutov
` (2 subsequent siblings)
3 siblings, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-16 8:12 UTC (permalink / raw)
To: Drew Adams; +Cc: acm, emacs-devel, monnier, dgutov
> Date: Thu, 15 Dec 2016 18:06:09 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
>
> > From where I'm standing, the things you use it for (hide
> > buffer contents)
>
> No. Narrowing is not used to "hide buffer contents". That's not
> the point of narrowing, even if it is one effect. It's used to
> make some buffer contents inaccessible to certain (many, typical,
> ordinary) operations. It is not just you who does not see parts
> of the buffer; it is also Emacs features that do not perceive
> them.
Narrowing is used for both of these effects. One clear example of the
former is Info mode, where the buffer is always narrowed to the
current node.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 2:06 ` Drew Adams
2016-12-16 8:12 ` Eli Zaretskii
@ 2016-12-16 12:32 ` Dmitry Gutov
2016-12-16 16:22 ` Drew Adams
2016-12-16 19:25 ` Alan Mackenzie
2016-12-16 14:24 ` Clément Pit--Claudel
[not found] ` <<83oa0c8f7r.fsf@gnu.org>
3 siblings, 2 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-16 12:32 UTC (permalink / raw)
To: Drew Adams, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
On 16.12.2016 04:06, Drew Adams wrote:
>> From where I'm standing, the things you use it for (hide
>> buffer contents)
>
> No. Narrowing is not used to "hide buffer contents".
Very well. :)
> That's not
> the point of narrowing, even if it is one effect. It's used to
> make some buffer contents inaccessible to certain (many, typical,
> ordinary) operations. It is not just you who does not see parts
> of the buffer; it is also Emacs features that do not perceive
> them.
Why should some facilities ignore narrowing, then? E.g. font-lock and
indentation, which have traditionally done so.
> At the base, narrowing is nothing more than binding the
> meanings/behavior of `point-min' and `point-max'.
Yes. That's literally its definition.
> No one has given an example of how narrowing makes things
> hard "for many other facilities" - or even for one facility.
You haven't been paying attention.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 2:06 ` Drew Adams
2016-12-16 8:12 ` Eli Zaretskii
2016-12-16 12:32 ` Dmitry Gutov
@ 2016-12-16 14:24 ` Clément Pit--Claudel
2016-12-16 16:22 ` Drew Adams
[not found] ` <<83oa0c8f7r.fsf@gnu.org>
3 siblings, 1 reply; 299+ messages in thread
From: Clément Pit--Claudel @ 2016-12-16 14:24 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 444 bytes --]
On 2016-12-15 21:06, Drew Adams wrote:
>> Thus, without making things harder for many other facilities.
> No one has given an example of how narrowing makes things
> hard "for many other facilities" - or even for one facility.
I think I gave one example of how narrowing breaks proof-general; in fact, I explained I developed my own alternative to narrowing with overlays because other packages broke in the face of regular narrowing.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-16 14:24 ` Clément Pit--Claudel
@ 2016-12-16 16:22 ` Drew Adams
2016-12-16 19:50 ` Clément Pit--Claudel
0 siblings, 1 reply; 299+ messages in thread
From: Drew Adams @ 2016-12-16 16:22 UTC (permalink / raw)
To: Clément Pit--Claudel, emacs-devel
> >> Thus, without making things harder for many other facilities.
> > No one has given an example of how narrowing makes things
> > hard "for many other facilities" - or even for one facility.
>
> I think I gave one example of how narrowing breaks proof-general; in fact, I
> explained I developed my own alternative to narrowing with overlays because
> other packages broke in the face of regular narrowing.
"Narrowing breaks proof-general". Really? Narrowing did that?
This is all I saw that you said about it:
There are many such places; unfortunately, the use of (point-min)
means that the overlays do not have the right starting point:
they start from the beginning of the narrowed region.
It's hard to fix this issue, because uses of point-min don't
carry sufficient semantic information: did the original author
mean (point-min), or 1? There's no general way to tell but to
read the surrounding code.
IIUC, the problem is that p-g makes use of some code that uses
`point-min', and p-g does not know whether that code intended
for `point-min' to mean the start of a buffer restriction or
position 1 of the buffer.
My answer to that is that `point-min' ALWAYS means the start
of the buffer restriction (narrowing), if it is restricted.
If p-g cannot count on that then it cannot count on using
code that doesn't understand that. It should know what code
that it reuses does; if not, all bets are off.
It sounds like it is not narrowing that broke p-g, but its
use of some broken code that uses `point-min' without
understanding that it refers to the beginning of a buffer
restriction. Or perhaps p-g just does not know what the
code it reuses does.
For p-g to decide whether it should `widen', it should be
enough for it to know whether _it_ wants to work with the
full buffer. If it tries to reuse some other code that is
flakey or unsure in that regard, then I'd say that p-g is
breaking itself.
Code that uses `point-min' should always expect that the
buffer is (because it could be) narrowed. It should never
assume that `point-min' is necessarily position 1.
Am I missing something?
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-16 12:32 ` Dmitry Gutov
@ 2016-12-16 16:22 ` Drew Adams
2016-12-16 19:14 ` Dmitry Gutov
2016-12-16 19:25 ` Alan Mackenzie
1 sibling, 1 reply; 299+ messages in thread
From: Drew Adams @ 2016-12-16 16:22 UTC (permalink / raw)
To: Dmitry Gutov, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
> > That's not the point of narrowing, even if it is one effect.
> > It's used to make some buffer contents inaccessible to certain
> > (many, typical, ordinary) operations. It is not just you who
> > does not see parts of the buffer; it is also Emacs features
> > that do not perceive them.
>
> Why should some facilities ignore narrowing, then? E.g.
> font-lock and indentation, which have traditionally done so.
I said "certain...operations". You said "some facilities".
I did not say that _every_ operation/facility needs or
takes advantage of such text inaccessibility. Those that
instead want to ignore narrowing do so. No problem.
That narrowing restricts the buffer limits is its effect.
Code can use that effect to any purpose it wants.
If some code deals with a narrowed buffer and it doesn't
really want to - if it instead really wants to pretend it
is not restricted, it's free to do that. Not hard to do.
I think you're referring to contexts where some code
narrows the buffer (presumably for a good reason), and
some other code has to deal with that narrowed buffer
even though it does not want to.
That's a question of designing the two bits of code. The
latter bit is working with what the former bit dishes out.
It has choices for how to deal with it, one of which is to
`save-restriction', then `widen' and do its
ignore-the-restriction thing.
See my reply to Clément for the case of reusing code that
is broken because it does not expect `point-min' to refer
to a buffer-restriction limit.
> > No one has given an example of how narrowing makes
> > things hard "for many other facilities" - or even
> > for one facility.
>
> You haven't been paying attention.
Prove it. See above. I don't see a problem with
narrowing. I do see that some code might mistakenly
think that the current buffer limits are the full
buffer limits, i.e., that it might mistakenly use
`point-min' when it should use 1. That's not the
fault of narrowing or a problem introduced by narrowing.
It's a misunderstanding of what `point-min' is about.
There is also the problem of someone mistakenly
reusing code that _rightfully_ uses `point-min'
(because it wants the restriction limit, not 1).
That's not the fault of the reused code or of narrowing.
That's the fault of the reuser.
Some code is wrong. People make mistakes when coding,
and not thinking that a buffer might be narrowed is
one possible mistake. And not realizing that some
code one reuses intentionally acts differently when
the buffer is narrowed is another possible mistake.
Buffer restriction is the _general_ case. An
unrestricted buffer is a special "restriction" case,
even if it is the more usual case.
Maybe we need to beef up the doc of `point-min', to
emphasize that it is the lower _buffer-restriction_
limit. And maybe some users need to better document
their functions, to make it clear when they respect a
narrowed buffer (as the Isearch doc makes clear, for
instance).
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
[not found] ` <<83oa0c8f7r.fsf@gnu.org>
@ 2016-12-16 16:22 ` Drew Adams
2016-12-16 21:00 ` Eli Zaretskii
[not found] ` <<8337hn8u81.fsf@gnu.org>
0 siblings, 2 replies; 299+ messages in thread
From: Drew Adams @ 2016-12-16 16:22 UTC (permalink / raw)
To: Eli Zaretskii, Drew Adams; +Cc: acm, emacs-devel, monnier, dgutov
> > > From where I'm standing, the things you use it for (hide
> > > buffer contents)
> >
> > No. Narrowing is not used to "hide buffer contents". That's not
> > the point of narrowing, even if it is one effect. It's used to
> > make some buffer contents inaccessible to certain (many, typical,
> > ordinary) operations. It is not just you who does not see parts
^^^^^^^^
> > of the buffer; it is also Emacs features that do not perceive them.
>
> Narrowing is used for both of these effects.
Both what effects? The only effect of narrowing is to, in
effect, change what `point-min' and `point-max' amount to.
It just restricts the effective buffer limits; nothing more.
When I said it is not used to "hide buffer contents" I meant
that it is not used ONLY to hide buffer contents from a user's
eyes. The rest of that paragraph makes that clear. It was a
response to the claim that narrowing is ONLY for hiding buffer
contents.
Certainly, hiding text from a user's eyes is one effect, and so
one use, of narrowing. But it is not the point of narrowing;
its full effect. There are other ways to hide text from a
user's eyes, but they do not substitute for what narrowing does.
> One clear example of the former is Info mode, where the buffer
> is always narrowed to the current node.
Nope. That's an example of just what I said: Narrowing does
more than just hide text from a user's eyes - and the use of
narrowing by Info relies on that "more".
A displayed node is a narrowed portion of the buffer, and
much about "nodeness" and operations on a node are related
to that narrowing. Try widening and then just making the
text outside the node invisible, and see how far you get
with Info.
Could Info have been implemented differently, so as not to
use narrowing? Sure. Could it have used text invisibility
to get the job done? Maybe, but only if it were taught to
not only hide invisible text but to also ignore it for
other purposes.
Text property `invisible' does not do what narrowing does.
You can add it to text, to hide it, but then you need to
teach code to ignore it for certain operations. Which
operations? To replace narrowing, all operations that
work off of the buffer limits.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 16:22 ` Drew Adams
@ 2016-12-16 19:14 ` Dmitry Gutov
2016-12-16 23:16 ` Drew Adams
0 siblings, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-16 19:14 UTC (permalink / raw)
To: Drew Adams, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
On 16.12.2016 18:22, Drew Adams wrote:
> I said "certain...operations". You said "some facilities".
> I did not say that _every_ operation/facility needs or
> takes advantage of such text inaccessibility. Those that
> instead want to ignore narrowing do so. No problem.
The facilities do not "want", they can't make these choices for us. The
difficulty is in deciding which Lisp code should ignore narrowing, and
which shouldn't.
And also, for each arbitrary piece of code, for the programmer to
remember to make that choice.
> That narrowing restricts the buffer limits is its effect.
> Code can use that effect to any purpose it wants.
Except if the other code, which the current code wants to call inside a
narrowing (and respect that narrowing!), always widens.
> That's a question of designing the two bits of code.
Because that's never difficult.
And if only all code was written by one person.
> The
> latter bit is working with what the former bit dishes out.
> It has choices for how to deal with it, one of which is to
> `save-restriction', then `widen' and do its
> ignore-the-restriction thing.
That only helps to retain the narrowing bounds for the caller code.
> See my reply to Clément for the case of reusing code that
> is broken because it does not expect `point-min' to refer
> to a buffer-restriction limit.
It's not enough to know what the code you're reusing does, if there's no
way to make that code respect your narrowing.
>> You haven't been paying attention.
>
> Prove it. See above. I don't see a problem with
> narrowing.
We've had numerous discussions on this subject already. If you still
don't see any problem, good luck with that.
> Maybe we need to beef up the doc of `point-min', to
> emphasize that it is the lower _buffer-restriction_
> limit. And maybe some users need to better document
> their functions, to make it clear when they respect a
> narrowed buffer (as the Isearch doc makes clear, for
> instance).
Neither of that helps in the actual problem cases.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 12:32 ` Dmitry Gutov
2016-12-16 16:22 ` Drew Adams
@ 2016-12-16 19:25 ` Alan Mackenzie
2016-12-16 20:06 ` Dmitry Gutov
1 sibling, 1 reply; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-16 19:25 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Stefan Monnier, Drew Adams, emacs-devel
Hello, Dmitry.
On Fri, Dec 16, 2016 at 02:32:26PM +0200, Dmitry Gutov wrote:
> Why should some facilities ignore narrowing, then? E.g. font-lock and
> indentation, which have traditionally done so.
They don't. No text outside of the visible region is indented or
fontified. But, if necessary, text outside the visible region is
inspected to get the needed context for the operation within it.
> > At the base, narrowing is nothing more than binding the
> > meanings/behavior of `point-min' and `point-max'.
> Yes. That's literally its definition.
> > No one has given an example of how narrowing makes things
> > hard "for many other facilities" - or even for one facility.
> You haven't been paying attention.
I keep hearing about how complicaated narrowing supposedly is, but I
can't help feeling that's due to misunderstandings about what narrowing
actually is, or somebody attributing blame to narrowing when it really
belongs elsewhere. I think it's quite a long time ago since anybody
gave any examples of narrowing supposedly making things difficult. A
reminder of what these things are would be most welcome now.
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 16:22 ` Drew Adams
@ 2016-12-16 19:50 ` Clément Pit--Claudel
2016-12-16 23:16 ` Drew Adams
0 siblings, 1 reply; 299+ messages in thread
From: Clément Pit--Claudel @ 2016-12-16 19:50 UTC (permalink / raw)
To: Drew Adams, emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 727 bytes --]
On 2016-12-16 11:22, Drew Adams wrote:
>>>> Thus, without making things harder for many other facilities.
>>> No one has given an example of how narrowing makes things
>>> hard "for many other facilities" - or even for one facility.
>> I think I gave one example of how narrowing breaks proof-general; in fact, I
>> explained I developed my own alternative to narrowing with overlays because
>> other packages broke in the face of regular narrowing.
> "Narrowing breaks proof-general". Really? Narrowing did that?
I mean that narrowing a buffer causes Proof General to misbehave. That's obviously a Proof General bug; I only intended it as an example of a facility for which narrowing made things harder.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 19:25 ` Alan Mackenzie
@ 2016-12-16 20:06 ` Dmitry Gutov
2016-12-16 23:16 ` Drew Adams
0 siblings, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-16 20:06 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: Stefan Monnier, Drew Adams, emacs-devel
On 16.12.2016 21:25, Alan Mackenzie wrote:
>> Why should some facilities ignore narrowing, then? E.g. font-lock and
>> indentation, which have traditionally done so.
>
> They don't. No text outside of the visible region is indented
Why would the text outside be indented? indent-for-tab-command indents
the current line. IOW, it ignores all the effect that narrowing could
have (in some modes; others don't bother to widen).
> or
> fontified.
Yes, it is. Maybe not entirely, because of jit-lock, but the first thing
font-lock-default-fontify-buffer does is call widen.
> But, if necessary, text outside the visible region is
> inspected to get the needed context for the operation within it.
Which, for many purposes, is as bad as ignoring the narrowing.
> I think it's quite a long time ago since anybody
> gave any examples of narrowing supposedly making things difficult. A
> reminder of what these things are would be most welcome now.
Should that happen every couple of months, or so? We have search in the
mailing list archives, you know.
1. Classical example of mixing narrowing for two different purposes, and
how that backfires:
M-: (info "(elisp) Variables")
Select some region, and to focus on reading it better (or some other
reason you use narrowing interactively for), narrow to it: C-x n n.
Finish reading it, try to get back to the node with `C-x n w'.
Instead, you see the whole elisp.info.
2. Multi-mode usage where the framework delegates to each subregion's
major mode for fontification and indentation.
I'm not going to recap it all again now, but, suffice to say, font-lock
rules, indentation functions and syntax-ppss ignoring the narrowing
imposed on them by the calling code is highly inconvenient.
We don't want the JS indentation or highlighting rules to look at HTML
code, and vice versa.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 1:33 ` Dmitry Gutov
2016-12-16 2:06 ` Drew Adams
@ 2016-12-16 20:06 ` Alan Mackenzie
2016-12-16 22:08 ` Dmitry Gutov
1 sibling, 1 reply; 299+ messages in thread
From: Alan Mackenzie @ 2016-12-16 20:06 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Stefan Monnier, emacs-devel
Hello, Dmitry.
On Fri, Dec 16, 2016 at 03:33:45AM +0200, Dmitry Gutov wrote:
> On 11.12.2016 12:17, Alan Mackenzie wrote:
> > I don't think this is an issue. (syntax-ppss-1 POS), when
> > new-local-variable is non-nil simulates (parse-partial-sexp
> > new-local-variable POS), and the latter would throw an error for
> > precisely the same values of POS as the former.
> Indeed, but this way the first argument will be implicit, and, unlike
> point-min, often non-obvious to the code calling syntax-ppss.
I thought the main idea of the facility was to provide a base point for
a sub-buffer in multi-major-mode modes. In this case, the code calling
syntax-ppss isn't going to be using positions less than
new-local-variable anyway, it's going to be working entirely inside the
pertinent sub-buffer.
> It's easy to produce a buffer position lower than the arbitrary value
> of the new variable. It's harder to produce a buffer position lower
> than point-min.
The new variable wouldn't have an "arbitrary" (i.e. random) value. It
would be carefully set by code which knows what it's doing. I can't see
how it would be easier (accidentally) to produce an invalid position for
syntax-ppss.
> Anyway, it's a minor issue. A good error message should be sufficient.
> > Maybe, but these changes are the direct desired effect of narrowing.
> > They're not nasty unforeseen side effects.
> "nasty" or "unforeseen" is beside the point. Like described previously,
> some code doesn't really know how to behave in the face of narrowing if
> the user's intent that resulted in narrowing, isn't known.
You keep saying this, but I can't remember any examples of what you mean
by "the user's intent ... not being known". Can you give such an
example of how code would do things differently on the basis of
"knowing" what the "user meant"? And I don't mean here the misuse of
(point-min) to mean "Position 1". That is just a bug.
Narrowing is just narrowing, and is fundamentally clear and unambiguous.
That's why it has become such a powerful and well-used tool, both
directly by users, and by Lisp code.
[ .... ]
> > That may be the case, but that has nothing to do with what I said ("In
> > particular, it doesn't change ...."). Whether abc is inside a string or
> > not doesn't vary by the narrowing. It's determined by other factors.
> It can be determined by the narrowing (i.e. a function named
> `inside-a-string?' could confidently know whether to rely on the current
> narrowing or ignore it, to produce its result, if narrowing were
> annotated with intent).
Why would anybody need such a facility? Could you give a concrete use
case, please?
> > In particular, I was thinking of what I think is
> > called "hard widening". I think it would be a mistake to add this
> > feature. At the moment, narrowing is a strong, direct, simple facility
> > - it does what it says it does and no more.
> From where I'm standing, the things you use it for (hide buffer
> contents) can be implemented in a much better fashion using overlays
> with invisibility (in fact, one such implementation exists already),
> which don't introduce the difficulties of changing point-min and
> point-max.
What difficulties?
> Thus, without making things harder for many other facilities.
Narrowing is such a fundamental feature of Emacs that attempting to
supersede it by something else would be bound to fail, or at least be
very painful for a long time. Maybe your invisibility overlays might
cover some use cases of narrowing, but they would be bound to fail on
others.
> > Were "hard widening" to be introduced, we could easily end up with a
> > confused mess of `widen' and `hard-widen' through the code, with
> > nobody being quite sure which one to use and why. There might be
> > lots of depressing errors where the wrong one is used.
> These are issues of design and documentation. The previous discussions
> of hard-widen didn't come up with any real conceptual problems. Just
> with low-level implementation ones.
It would foul up much existing code. It would be bound to. There will
be code around which depends on widening meaning exactly "widen". If you
change the meaning of `widen' to `(narrow-to-region some-start some-end)'
things will go wrong.
> >> More importantly, any change you would make to the "new" function you
> >> could make to syntax-ppss instead.
> > The new function would be a different function, i.e. it would have a
> > different definition. Just for clarity, it would not be written from
> > scratch, it would be based on the current syntax-ppss.
> And then it would be utterly useless for third-party code, for years to
> come. Further, if it uses a separate variable for its cache, and the new
> function is used in the code, while the old function continues to be
> used by third-party code (and also stable packages like Gnus and
> CC-Mode) for compatibility reasons, we'll have two syntax caches that
> will interact in fascinating, difficult to predict ways.
I don't agree with you on any of this. Of course, the devil would be in
the details, but I do think the Emacs team has enough experience of
changes to be able to do this one in a non-destructive fashion. Don't
forget, the current syntax-ppss doesn't and can't work properly.
> Am I interpreting your intention correctly?
I don't think so. `syntax-ppss-1' would be usable in the same way as
`syntax-ppss', so a backward compatibility alias would leave things
working at least as well as they do at the moment.
> >> Further, the good part of this proposal is more or less equivalent to
> >> "indexing syntax-ppss cache by point-min" as proposed by Stefan.
> > I think I agree here. But "good part" implies the existence of a "less
> > good part". What, in your view, constitues this "less good part"?
> Well... both your and Stefan's proposals will end up actually having to
> maintain the index of syntax-ppss caches, and having to invalidate those
> values somehow. I'm sure it's possible to arrive at a satisfactory
> algorithm (like only keeping around two values, for point-min equal to 1
> and the most recent other value), but there must be use cases which some
> other invalidation approach would be better.
That was precisely the mechanism I was looking at a few days ago. I
agree it could probably be improved upon, perhaps after gaining
experience with it.
> With syntax-ppss-dont-widen, the caller code can take the responsibility
> of maintaining the values of syntax-ppss-cache and syntax-ppss-last.
The calling code shouldn't even be aware of the internal details of the
syntax-ppss cache, far less be burdened with the responsibility of
maintaining it. Such high coupling between syntax-ppss and its users
would make it pretty much impossible to change the cache mechanism at a
later date if we wanted to do that. syntax-ppss should be as black-box
like as possible.
> And it turns out, it's rather easy to bind them to nil around the same
> pieces of code that starts with (save-restriction (narrow-to-region
> ...)). Which currently seems to give the same result as the idea in the
> previous paragraph, with much less code.
> In short, letting the caller code control cache invalidation has its
> benefits.
They are by far outweighed by the disadvantages.
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 16:22 ` forward-comment and syntax-ppss Drew Adams
@ 2016-12-16 21:00 ` Eli Zaretskii
[not found] ` <<8337hn8u81.fsf@gnu.org>
1 sibling, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-16 21:00 UTC (permalink / raw)
To: Drew Adams; +Cc: acm, emacs-devel, monnier, drew.adams, dgutov
> Date: Fri, 16 Dec 2016 08:22:49 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: dgutov@yandex.ru, acm@muc.de, monnier@iro.umontreal.ca,
> emacs-devel@gnu.org
>
> > > > From where I'm standing, the things you use it for (hide
> > > > buffer contents)
> > >
> > > No. Narrowing is not used to "hide buffer contents". That's not
> > > the point of narrowing, even if it is one effect. It's used to
> > > make some buffer contents inaccessible to certain (many, typical,
> > > ordinary) operations. It is not just you who does not see parts
> ^^^^^^^^
> > > of the buffer; it is also Emacs features that do not perceive them.
> >
> > Narrowing is used for both of these effects.
>
> Both what effects?
The ones you mentioned above.
> > One clear example of the former is Info mode, where the buffer
> > is always narrowed to the current node.
>
> Nope.
You can say "nope" all you want, the fact stays.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 20:06 ` Alan Mackenzie
@ 2016-12-16 22:08 ` Dmitry Gutov
2016-12-16 23:08 ` Stefan Monnier
2016-12-16 23:16 ` Drew Adams
0 siblings, 2 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-16 22:08 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
On 16.12.2016 22:06, Alan Mackenzie wrote:
> I thought the main idea of the facility was to provide a base point for
> a sub-buffer in multi-major-mode modes. In this case, the code calling
> syntax-ppss isn't going to be using positions less than
> new-local-variable anyway, it's going to be working entirely inside the
> pertinent sub-buffer.
Subregion, you mean? (Same buffer).
But why? Because the multi-major-mode would narrow around the subregion
as well? If we're considering chiefly this use case, it would make sense
to just look at the lower narrowing bound, instead of the new variable.
That would require less code all around.
> The new variable wouldn't have an "arbitrary" (i.e. random) value. It
> would be carefully set by code which knows what it's doing. I can't see
> how it would be easier (accidentally) to produce an invalid position for
> syntax-ppss.
First, let's remember this.
Second, the code that sets the variable and the code that calls
syntax-ppss will be different. Typically, they will reside in different
packages, and they might be authored by different people.
Imagine that our framework code performed the narrowing, then set the
variable to the lower bound of the narrowing, and then called some major
mode's fontification code which (for some reason) called widen and then
syntax-ppss. In this case, it can supply a lower value to syntax-ppss
than the original narrowing's lower bound.
This situation is a typical problematic one, and a bad error message can
make it more difficult to investigate.
> I can't remember any examples of what you mean
> by "the user's intent ... not being known". Can you give such an
> example of how code would do things differently on the basis of
> "knowing" what the "user meant"? And I don't mean here the misuse of
> (point-min) to mean "Position 1". That is just a bug.
For instance, c-indent-line could ignore the narrowing when it was made
interactively by the user, as typically used by users such as yourself.
But when narrowing is made programmatically, by the multi-mode framework
code, c-indent-line would honor it.
Also see the Info example in the other email. Having two types of
narrowing could help fix that scenario.
>> It can be determined by the narrowing (i.e. a function named
>> `inside-a-string?' could confidently know whether to rely on the current
>> narrowing or ignore it, to produce its result, if narrowing were
>> annotated with intent).
>
> Why would anybody need such a facility? Could you give a concrete use
> case, please?
font-lock-syntactic-face-function normally looks at the current syntax
state (string, comment, etc).
In the multiple-mode framework, we want to support subregions inside
comments or strings, and to highlight them accordingly. E.g. code
example inside a comment, or SQL inside a heredoc literal.
> Narrowing is such a fundamental feature of Emacs that attempting to
> supersede it by something else would be bound to fail, or at least be
> very painful for a long time. Maybe your invisibility overlays might
> cover some use cases of narrowing, but they would be bound to fail on
> others.
The idea is to move the user-facing narrowing commands to some other
foundation (like the invisible and maybe intangible text properties),
freeing the narrowing facility for use by the Lisp code.
> It would foul up much existing code. It would be bound to. There will
> be code around which depends on widening meaning exactly "widen". If you
> change the meaning of `widen' to `(narrow-to-region some-start some-end)'
> things will go wrong.
So far I don't have reasons to believe that it would foul up any use
cases that worked fine until now.
> Of course, the devil would be in
> the details, but I do think the Emacs team has enough experience of
> changes to be able to do this one in a non-destructive fashion.
Yes, by not making a mountain out of a molehill and just changing the
existing function. Like the Emacs team has opined already.
> Don't
> forget, the current syntax-ppss doesn't and can't work properly.
Are you trying to hypnotize the readers?
> `syntax-ppss-1' would be usable in the same way as
> `syntax-ppss', so a backward compatibility alias would leave things
> working at least as well as they do at the moment.
Migrating all the code out there to use the new function is a big job by
itself, and it would spread over years. That's why we rarely do
something like that.
Logistics aside, either they are different, or they are the same. If
there are no use cases for syntax-ppss in which it can't be replaced
with syntax-ppss-1, there's literally zero reasons to create a new
function instead of fixing the existing one.
> That was precisely the mechanism I was looking at a few days ago. I
> agree it could probably be improved upon, perhaps after gaining
> experience with it.
I'm saying it can't be perfect without taking into account the calling
code. Leaving all cache management to it is one way to do that (but not
the only one, to be clear).
But if you'd like to gain some personal experience with it, you're
welcome to make the planned changes in your copy of Emacs, download
mmm-mode, adapt it accordingly (there are just a few places in
mmm-region.el, all calling narrow-to-region), and play around with it.
>> With syntax-ppss-dont-widen, the caller code can take the responsibility
>> of maintaining the values of syntax-ppss-cache and syntax-ppss-last.
>
> The calling code shouldn't even be aware of the internal details of the
> syntax-ppss cache, far less be burdened with the responsibility of
> maintaining it.
That's not a given. Like you said above, the caller code will have to
"knows what it's doing". A multiple-mode framework will have to be
familiar with internal structure of a number of packages anyway, at
least to some extent.
> Such high coupling between syntax-ppss and its users
> would make it pretty much impossible to change the cache mechanism at
> a later date if we wanted to do that. syntax-ppss should be as
> black-box like as possible.
If we change the cache mechanism, we can adapt these special callers to
it. Yes, normally, we want to limit the public API in order to improve
the stability of the ecosystem.
In this case, however, there are only 2-3 active packages that work in
the "multiple-mode" space. And there are unlikely to crow up many more
of them because it's a hard problem.
Even if I'm wrong, and by some miracle (or hard work of improving our
internals, of course) many new packages like that start to appear, we'll
have time to choose a stable interface and update the current multi-mode
packages. In the meantime, syntax-ppss-dont-widen's documentation will
just have to be clear that any packages using it are on their own.
>> In short, letting the caller code control cache invalidation has its
>> benefits.
>
> They are by far outweighed by the disadvantages.
While I don't agree, you've pointed out the disadvantage of the simplest
approach to caller-controlled cache invalidation. There can be others.
Maybe a macro like `with-separate-syntax-ppss-cache'.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 22:08 ` Dmitry Gutov
@ 2016-12-16 23:08 ` Stefan Monnier
2016-12-17 0:18 ` Dmitry Gutov
2016-12-17 7:42 ` Eli Zaretskii
2016-12-16 23:16 ` Drew Adams
1 sibling, 2 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-16 23:08 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Alan Mackenzie, emacs-devel
> The idea is to move the user-facing narrowing commands to some other
> foundation (like the invisible and maybe intangible text properties),
> freeing the narrowing facility for use by the Lisp code.
While it might be a good direction, it's not as simple as it sounds,
since many uses of the C-x n n command looks like "narrow to the region,
and then apply <blabla> to the whole buffer": I think it will be tricky
to make overlays that mimick this behavior sufficiently well.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-16 19:14 ` Dmitry Gutov
@ 2016-12-16 23:16 ` Drew Adams
2016-12-17 0:08 ` Dmitry Gutov
0 siblings, 1 reply; 299+ messages in thread
From: Drew Adams @ 2016-12-16 23:16 UTC (permalink / raw)
To: Dmitry Gutov, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
> > That narrowing restricts the buffer limits is its effect.
> > Code can use that effect to any purpose it wants.
>
> Except if the other code, which the current code wants to call inside a
> narrowing (and respect that narrowing!), always widens.
Therein lies your mistake - mistaken assumption, perhaps.
Why should "the current code" call other code that widens,
and yet expect that it will _not_ widen?
The fault here is not from the "other code" narrowing or
widening. It is from the "current code" calling such
"other code". Code needs to choose wisely any "other code"
that it tries to use.
IOW, this sounds a lot like a case where the doctor should
tell you, "If it hurts, stop doing that."
But if the called code is indeed defective - if it should
in fact restore the initial buffer restriction after it
widens or narrows to do its work, then that should be
fixed. That's no reason to say that it should never
widen or narrow.
> > That's a question of designing the two bits of code.
>
> Because that's never difficult.
> And if only all code was written by one person.
Why? If you call a function in your code, regardless of who
wrote that function, you need to know what it does, no?
And in particular, you need to know how it handles a buffer
restriction - at least if you are _counting on_ some particular
behavior wrt that. You need to know whether you need to wrap
its call (or some other bits of your code) in `save-restriction',
for example.
Or to otherwise set and save some restriction limits, and then
restore them at the proper time. Restriction limits are just
buffer positions. Setting and saving them is not rocket science.
This is similar to knowing whether called code moves point or
changes which buffer is current or which window is selected or...
It sounds like you are complaining that when you use function
`abcde' it doesn't behave as you would like. Maybe consider
not trying to expect it to do something that it does not do?
Or (again) if the code being called is defective, and it should
really restore the initial restriction, then fix that.
> > See my reply to Clément for the case of reusing code that
> > is broken because it does not expect `point-min' to refer
> > to a buffer-restriction limit.
>
> It's not enough to know what the code you're reusing does,
> if there's no way to make that code respect your narrowing.
What do you mean by "respect your narrowing"? If you mean
that there is no way to prevent called code from applying
a different buffer restriction (including widening completely),
then, well, duh!
Just as there is no way to prevent it from assigning a different
value to a variable that is visible to it. Or to prevent it
from changing the current buffer (and even leaving it changed),
or the selected window. Or any number of other things.
If code that you're thinking about calling from your code does
something you do not want, then, well, don't call it. Call
some other code - maybe write your own, possibly enhance the
code you were thinking about calling unwisely,...
> >> You haven't been paying attention.
> >
> > Prove it. See above. I don't see a problem with
> > narrowing.
>
> We've had numerous discussions on this subject already.
> If you still don't see any problem, good luck with that.
That's one way to evacuate the discussion... How about:
If you still see a problem, and you cannot articulate it,
good luck with that.
If you think it is important enough, perhaps speak concretely,
in terms of particular code. Show a case where something
really needs to be changed in Emacs, because there is an
inherent problem.
> > Maybe we need to beef up the doc of `point-min', to
> > emphasize that it is the lower _buffer-restriction_
> > limit. And maybe some users need to better document
> > their functions, to make it clear when they respect a
> > narrowed buffer (as the Isearch doc makes clear, for
> > instance).
>
> Neither of that helps in the actual problem cases.
What "actual problem cases"?
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-16 22:08 ` Dmitry Gutov
2016-12-16 23:08 ` Stefan Monnier
@ 2016-12-16 23:16 ` Drew Adams
1 sibling, 0 replies; 299+ messages in thread
From: Drew Adams @ 2016-12-16 23:16 UTC (permalink / raw)
To: Dmitry Gutov, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
> The idea is to move the user-facing narrowing commands to some other
> foundation (like the invisible and maybe intangible text properties),
> freeing the narrowing facility for use by the Lisp code.
I couldn't disagree more. Fix the _code_, if it is mistaken.
Code should know whether it should "respect" the current
buffer restriction. If for some reason it cannot, then it
needs to punt - raise an "I don't know what I'm doing!" error
or whatever. If the user is involved, and the user can tell
it whether the restriction should be respected, then it could
conceivably ask the user.
There is no royal road here. This is no different from
calling code blindly and somehow expecting wise behavior.
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-16 19:50 ` Clément Pit--Claudel
@ 2016-12-16 23:16 ` Drew Adams
2016-12-17 2:39 ` Clément Pit--Claudel
0 siblings, 1 reply; 299+ messages in thread
From: Drew Adams @ 2016-12-16 23:16 UTC (permalink / raw)
To: Clément Pit--Claudel, emacs-devel
> > "Narrowing breaks proof-general". Really? Narrowing did that?
>
> I mean that narrowing a buffer causes Proof General to misbehave. That's
> obviously a Proof General bug; I only intended it as an example of a
> facility for which narrowing made things harder.
OK. But that's like saying that variables break program `xyz'
because variables make things harder. Yes, things that change
state make things harder, and that includes narrowing. If this
were Haskell then things wouldn't be so hard...
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-16 20:06 ` Dmitry Gutov
@ 2016-12-16 23:16 ` Drew Adams
2016-12-17 0:42 ` Dmitry Gutov
0 siblings, 1 reply; 299+ messages in thread
From: Drew Adams @ 2016-12-16 23:16 UTC (permalink / raw)
To: Dmitry Gutov, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
> 1. Classical example of mixing narrowing for two different purposes, and
> how that backfires: M-: (info "(elisp) Variables")
> Select some region, and to focus on reading it better (or some other
> reason you use narrowing interactively for), narrow to it: C-x n n.
> Finish reading it, try to get back to the node with `C-x n w'.
>
> Instead, you see the whole elisp.info.
Not a problem for me. And doesn't have to be a problem for
vanilla Emacs users either, if the proper commands are added.
For me, narrowing (and widening) pushes the current buffer
limits to a list. And it is trivial to return to any previous
restriction (including fully wide).
In my case:
1. `C-x n n' to "focus on reading" a paragraph in the node.
2. `C-x n x' to return to the full node (not all of Info).
End of story.
or:
1. `C-x n w' to see all of the Info file (for context or whatever).
2. `C-x n x' to return to seeing just the node.
or:
`C-x n n' here and `C-x n n' there, any number of times & places,
followed by `C-x n x' with a prefix arg, to choose any of those
restrictions again. Or just cycle among them by repeating
`C-x n x'. Either within a buffer or across buffers.
----
zz-narrow-repeat is an interactive compiled Lisp function in
`zones.el'. It is bound to C-x n x.
(zz-narrow-repeat ARG)
Cycle to the next buffer restriction (narrowing).
This is a repeatable version of `zz-narrow'.
Note that if the value of `zz-izones-var' is not buffer-local
then you can use this command to cycle among regions in multiple
buffers.
----
It's a repeatable version of `zz-narrow' (prefix arg acts the same):
----
zz-narrow is an interactive compiled Lisp function in `zones.el'.
(zz-narrow ARG &optional MSGP)
Widen to a previous buffer restriction (narrowing).
The candidates are the zones in the current `zz-izones-var'.
With no prefix arg, widen to the previous narrowing.
With a plain prefix arg (`C-u'), widen completely.
With a zero prefix arg (`C-0'), widen completely and reset (empty)
the list of zones for this buffer.
With a numeric prefix arg N, widen abs(N) times (to the abs(N)th
previous narrowing). Positive and negative args work the same,
except that a negative arg also pops entries off the ring: it removes
the ring entries from the most recent back through the (-)Nth one.
----
`C-x n x' is simple, but it goes a long way.
If better control of restrictions by users is what you're after,
then the code in `zones.el' is a good start, IMO.
> 2. Multi-mode usage where the framework delegates to each
> subregion's major mode for fontification and indentation.
I can't speak to that (too vague for me). But a wild guess might
be that maybe "the framework" should _not_ blindly so delegate.
IOW, if it hurts, don't do it. Maybe a better sauce is needed
for the recipe. Or maybe the delegate itself needs to be
improved, before it can be delegated this way.
> I'm not going to recap it all again now, but, suffice to say, font-lock
> rules, indentation functions and syntax-ppss ignoring the narrowing
> imposed on them by the calling code is highly inconvenient.
Then maybe that should be fixed. Or maybe the fix should be in
the calling code. Seems like font-lock etc. code should be able
to do whatever narrowing it needs to do (modulo, perhaps,
communicating what it's done). If that is wrong for a potential
caller then the caller might need to do something different.
Anyway, font-lock, indentation, and syntax-ppss are hardly your
average bits of code. Perhaps some communication is in order,
between them and potential callers, so that they can be properly
or more easily used in more contexts.
That's a far cry from wanting to simply replace narrowing for
users with some kind of invisible-text hack.
> We don't want the JS indentation or highlighting rules to look at HTML
> code, and vice versa.
So create environments for them - quarantine them. Or fix them
so whatever they do to wrt buffer restriction is benign outside
them.
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
[not found] ` <<8337hn8u81.fsf@gnu.org>
@ 2016-12-16 23:16 ` Drew Adams
0 siblings, 0 replies; 299+ messages in thread
From: Drew Adams @ 2016-12-16 23:16 UTC (permalink / raw)
To: Eli Zaretskii, Drew Adams; +Cc: acm, emacs-devel, monnier, dgutov
> > > One clear example of the former is Info mode, where the
> > > buffer is always narrowed to the current node.
> >
> > Nope. That's an example of just what I said: Narrowing does
> > more than just hide text from a user's eyes - and the use of
> > narrowing by Info relies on that "more".
>
> You can say "nope" all you want, the fact stays.
What fact is that? How is the narrowing done by Info ONLY
hiding text from user eyes?
You know perfectly well that plenty of the behavior of Info
depends on narrowing to nodes. It is not at all simply about
hiding text from the user. Try removing the narrowing, and
see how well the Info _code_ behaves.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 23:16 ` Drew Adams
@ 2016-12-17 0:08 ` Dmitry Gutov
0 siblings, 0 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-17 0:08 UTC (permalink / raw)
To: Drew Adams, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
On 17.12.2016 01:16, Drew Adams wrote:
> The fault here is not from the "other code" narrowing or
> widening. It is from the "current code" calling such
> "other code". Code needs to choose wisely any "other code"
> that it tries to use.
We don't have a lot of choice if we want to avoid code duplication.
Simply copying it over and removing the widen call is not an option.
> But if the called code is indeed defective - if it should
> in fact restore the initial buffer restriction after it
> widens or narrows to do its work, then that should be
> fixed.
The point is that "its work" does not honor the narrowing. And we want
it to.
> And in particular, you need to know how it handles a buffer
> restriction - at least if you are _counting on_ some particular
> behavior wrt that. You need to know whether you need to wrap
> its call (or some other bits of your code) in `save-restriction',
> for example.
Knowing doesn't help either.
> Or to otherwise set and save some restriction limits, and then
> restore them at the proper time. Restriction limits are just
> buffer positions. Setting and saving them is not rocket science.
Doesn't help.
> It sounds like you are complaining that when you use function
> `abcde' it doesn't behave as you would like. Maybe consider
> not trying to expect it to do something that it does not do?
You're so eager to type out heaps of condescending text. Maybe try
reading carefully next time?
> possibly enhance the
> code you were thinking about calling unwisely,...
What do you think these discussions are all about? Shrugging and
complaining about our collective bad luck?
Of course it's about improving code, as well as the guidelines to do so
regarding the usage of narrowing.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 23:08 ` Stefan Monnier
@ 2016-12-17 0:18 ` Dmitry Gutov
2016-12-17 7:37 ` Marcin Borkowski
2016-12-17 7:42 ` Eli Zaretskii
1 sibling, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-17 0:18 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Alan Mackenzie, emacs-devel
On 17.12.2016 01:08, Stefan Monnier wrote:
> While it might be a good direction, it's not as simple as it sounds,
> since many uses of the C-x n n command looks like "narrow to the region,
> and then apply <blabla> to the whole buffer": I think it will be tricky
> to make overlays that mimick this behavior sufficiently well.
I suppose it would break that usage, yes.
mark-whole-buffer should still work, though, as well as command
sequences that use it. Or if it doesn't, we could provide a replacement
command on that same binding. And some getters, like
visible-area-beginning and visible-area-end, for functions that know
that they want to act on user-mode narrowing (*).
Someone who actually uses narrowing interactively should weigh in on how
costly that transition might be.
(*) Not sure if that leads to the similar problems as we have now,
though, like some basic code not knowing whether to use point-min or
visible-area-beginning.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 23:16 ` Drew Adams
@ 2016-12-17 0:42 ` Dmitry Gutov
2016-12-17 1:30 ` Drew Adams
0 siblings, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-17 0:42 UTC (permalink / raw)
To: Drew Adams, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
On 17.12.2016 01:16, Drew Adams wrote:
> Not a problem for me. And doesn't have to be a problem for
> vanilla Emacs users either, if the proper commands are added.
You're actually proposing a change to the current narrowing-widening UI
and API. That should indicate that the current one is insufficient, or,
has problems.
I'm not in any hurry to adopt it, though, because it doesn't solve any
multi-mode problems. Not by itself, at least.
> In my case:
> 1. `C-x n n' to "focus on reading" a paragraph in the node.
> 2. `C-x n x' to return to the full node (not all of Info).
> End of story.
What happens if you press `C-x n x' once more?
In any case, you've introduced "a different kind of narrowing" yourself.
Your `C-x n n' command does something else than just call narrow-to-region.
>> 2. Multi-mode usage where the framework delegates to each
>> subregion's major mode for fontification and indentation.
>
> I can't speak to that (too vague for me).
I've described it in more detail numerous times. Please go ahead and use
the search.
> But a wild guess might
> be that maybe "the framework" should _not_ blindly so delegate.
> IOW, if it hurts, don't do it. Maybe a better sauce is needed
> for the recipe. Or maybe the delegate itself needs to be
> improved, before it can be delegated this way.
> [...]
Not helpful.
> Seems like font-lock etc. code should be able
> to do whatever narrowing it needs to do (modulo, perhaps,
> communicating what it's done).
We want to highlight different regions using different sets of font-lock
rules.
> If that is wrong for a potential
> caller then the caller might need to do something different.
What?
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-17 0:42 ` Dmitry Gutov
@ 2016-12-17 1:30 ` Drew Adams
2016-12-17 1:40 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Drew Adams @ 2016-12-17 1:30 UTC (permalink / raw)
To: Dmitry Gutov, Alan Mackenzie; +Cc: Stefan Monnier, emacs-devel
> > Not a problem for me. And doesn't have to be a problem for
> > vanilla Emacs users either, if the proper commands are added.
>
> You're actually proposing a change to the current narrowing-widening UI
> and API. That should indicate that the current one is insufficient, or,
> has problems.
I didn't propose anything. The point is that it is not a big
deal to remember two positions and return the buffer limits
to them later.
> I'm not in any hurry to adopt it, though, because it doesn't solve any
> multi-mode problems. Not by itself, at least.
I'm not asking you to adopt it.
I'm asking you not to replace narrowing, as changing buffer
limits, to something else, based on overlays or whatever.
I'm asking you not to reserve narrowing for Lisp and give
users some ersatz for it. Users and Lisp are not two
different universes.
> > In my case:
> > 1. `C-x n n' to "focus on reading" a paragraph in the node.
> > 2. `C-x n x' to return to the full node (not all of Info).
> > End of story.
>
> What happens if you press `C-x n x' once more?
If you have other restrictions in the ring, then it picks
up the previous one. And so on, until it gets to the
beginning, and then it wraps around to the last one.
So if you have only one restriction, as in just #1 then #2,
above, then repeating `C-x n x' gets you back to #1: the
paragraph.
IOW, it's like any other ring in Emacs.
> In any case, you've introduced "a different kind of narrowing" yourself.
> Your `C-x n n' command does something else than just call narrow-to-region.
Oh, yes; a GIANT change (?) ;-)
(defadvice narrow-to-region (before zz-add-zone activate)
"Push the region limits to the current `zz-izones-var'.
You can use `C-x n x' to widen to a previous buffer restriction.
This is a destructive operation. The list structure of the variable
value can be modified."
(when (or (interactive-p) zz-add-zone-anyway-p)
(let ((start (ad-get-arg 0))
(end (ad-get-arg 1)))
(unless start (setq start (region-beginning))) ; Needed for Emacs 20.
(unless end (setq end (region-end)))
(zz-add-zone start end nil nil nil 'MSG))))
Pretty drastic, huh? It adds the restriction to a ring variable.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 1:30 ` Drew Adams
@ 2016-12-17 1:40 ` Stefan Monnier
2016-12-17 2:26 ` Drew Adams
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-17 1:40 UTC (permalink / raw)
To: Drew Adams; +Cc: Alan Mackenzie, emacs-devel, Dmitry Gutov
> Oh, yes; a GIANT change (?) ;-)
[...]
> Pretty drastic, huh? It adds the restriction to a ring variable.
The point is that now any piece of code which uses `widen` will have
to decide whether to widen like the original `widen` or like your
advised C-x n w
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-17 1:40 ` Stefan Monnier
@ 2016-12-17 2:26 ` Drew Adams
2016-12-17 14:37 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Drew Adams @ 2016-12-17 2:26 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Alan Mackenzie, emacs-devel, Dmitry Gutov
> > Oh, yes; a GIANT change (?) ;-)
> > Pretty drastic, huh? It adds the restriction to a ring variable.
>
> The point is that now any piece of code which uses `widen` will have
> to decide whether to widen like the original `widen` or like your
> advised C-x n w
No. `C-x n w' is not advised. Nor is `widen' advised. Nor
redefined in any way.
But again, I am not asking that you adopt my code or my approach.
The point is that users are not bereft of ways to return to a
previous restriction. I answered the "problem" posed by Eli for
narrowing in Info.
I'm not even asking that you provide or enhance commands in
some other way, in order to, as I showed, make it easier for
users to return to a previous restriction.
I'm asking only that narrowing continue to be usable by both
users and Lisp code, in the same way, as a change in buffer
limits, rather than, say, a fiddle with overlays.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 23:16 ` Drew Adams
@ 2016-12-17 2:39 ` Clément Pit--Claudel
2016-12-17 4:50 ` Wording of the Elisp manuals (was: forward-comment and syntax-ppss) Marcin Borkowski
0 siblings, 1 reply; 299+ messages in thread
From: Clément Pit--Claudel @ 2016-12-17 2:39 UTC (permalink / raw)
To: Drew Adams, emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 772 bytes --]
On 2016-12-16 18:16, Drew Adams wrote:
>>> "Narrowing breaks proof-general". Really? Narrowing did that?
>>
>> I mean that narrowing a buffer causes Proof General to misbehave. That's
>> obviously a Proof General bug; I only intended it as an example of a
>> facility for which narrowing made things harder.
>
> OK. But that's like saying that variables break program `xyz'
> because variables make things harder. Yes, things that change
> state make things harder, and that includes narrowing. If this
> were Haskell then things wouldn't be so hard...
Yup. I think a significant part of the issue is just that some devs forget that using point-min may require widening. I didn't know about for a while, even after I started programming in ELisp.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Wording of the Elisp manuals (was: forward-comment and syntax-ppss)
2016-12-17 2:39 ` Clément Pit--Claudel
@ 2016-12-17 4:50 ` Marcin Borkowski
2016-12-20 9:22 ` Wording of the Elisp manuals Michael Heerdegen
2016-12-20 10:52 ` Wording of the Elisp manuals (was: forward-comment and syntax-ppss) Jean-Christophe Helary
0 siblings, 2 replies; 299+ messages in thread
From: Marcin Borkowski @ 2016-12-17 4:50 UTC (permalink / raw)
To: Clément Pit--Claudel; +Cc: Drew Adams, emacs-devel
On 2016-12-17, at 03:39, Clément Pit--Claudel <clement.pit@gmail.com> wrote:
> On 2016-12-16 18:16, Drew Adams wrote:
>>>> "Narrowing breaks proof-general". Really? Narrowing did that?
>>>
>>> I mean that narrowing a buffer causes Proof General to misbehave. That's
>>> obviously a Proof General bug; I only intended it as an example of a
>>> facility for which narrowing made things harder.
>>
>> OK. But that's like saying that variables break program `xyz'
>> because variables make things harder. Yes, things that change
>> state make things harder, and that includes narrowing. If this
>> were Haskell then things wouldn't be so hard...
>
> Yup. I think a significant part of the issue is just that some devs forget that using point-min may require widening. I didn't know about for a while, even after I started programming in ELisp.
True. I wanted to say "Hey, that means that narrowing should be
mentioned in the Emacs Lisp Intro". But I checked, and it is, and
fairly early for that matter.
I wonder how many things like that are waiting for the brave developer
coming from the background of some saner language/environment. ;-) (In
case someone misses it: this is of course tongue-in-cheek, I /do/ like
Emacs and Elisp. Each language has its quirks; the point is,
E\(macs\|lisp\) have /lots/ of them because (1) history and (2) Elisp is
a text editor language, not a general purpose programming environment)
Here is, I suppose, a minor bug. The Elisp reference says:
,----
| This manual attempts to be a full description of Emacs Lisp. For
| a beginner’s introduction to Emacs Lisp, see ‘An Introduction to Emacs
| Lisp Programming’, by Bob Chassell, also published by the Free
| Software Foundation. This manual presumes considerable familiarity
| with the use of Emacs for editing; see ‘The GNU Emacs Manual’ for this
| basic information.
`----
OTOH, Elisp Intro says:
,----
| Perhaps you want to understand programming; perhaps you want to
| extend Emacs; or perhaps you want to become a programmer. This
| introduction to Emacs Lisp is designed to get you started: to guide you
| in learning the fundamentals of programming, and more importantly, to
| show you how you can teach yourself to go further.
`----
I am pretty sure that any experienced programmer who did use Emacs for
a while, after reading the above paragraph, will skip Elisp Intro
entirely ("They want to teach me conditionals and loops? I know that
already!")
Wouldn't it be a good idea to state clearly somewhere that ELisp Intro
is not only about introduction to programming, but also about
introdcution to Emacs as the Elisp environment? Narrowing is first
mentioned in Chapter 9 of the Elisp Reference and only described in
detail in Chapter 29 (!).
Just my 2 cents.
Best,
--
Marcin Borkowski
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 0:18 ` Dmitry Gutov
@ 2016-12-17 7:37 ` Marcin Borkowski
0 siblings, 0 replies; 299+ messages in thread
From: Marcin Borkowski @ 2016-12-17 7:37 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: Alan Mackenzie, Stefan Monnier, emacs-devel
On 2016-12-17, at 01:18, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 17.12.2016 01:08, Stefan Monnier wrote:
>
>> While it might be a good direction, it's not as simple as it sounds,
>> since many uses of the C-x n n command looks like "narrow to the region,
>> and then apply <blabla> to the whole buffer": I think it will be tricky
>> to make overlays that mimick this behavior sufficiently well.
>
> I suppose it would break that usage, yes.
>
> mark-whole-buffer should still work, though, as well as command
> sequences that use it. Or if it doesn't, we could provide a replacement
> command on that same binding. And some getters, like
> visible-area-beginning and visible-area-end, for functions that know
> that they want to act on user-mode narrowing (*).
>
> Someone who actually uses narrowing interactively should weigh in on how
> costly that transition might be.
>
> (*) Not sure if that leads to the similar problems as we have now,
> though, like some basic code not knowing whether to use point-min or
> visible-area-beginning.
I use narrowing interactively /a lot/. (It's almost one of the Emacs'
killer features.) Sometimes I just want to perform a query-replace
limited to some region (I know it takes the active region into accout,
I just sometimes like narrowing better - then, I can quit query-replace,
fix something and hit (C-)M-% again. (I could use recursive edit, but
I seldom remember about it.)
Another use case is plain visibility: sometimes I just don't want to see
anything but the part I'm working on.
Maybe there are other use cases. I'll try to notice when I use
narrowing and see what they are.
Best,
--
Marcin Borkowski
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-16 23:08 ` Stefan Monnier
2016-12-17 0:18 ` Dmitry Gutov
@ 2016-12-17 7:42 ` Eli Zaretskii
2016-12-17 14:41 ` Stefan Monnier
1 sibling, 1 reply; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-17 7:42 UTC (permalink / raw)
To: Stefan Monnier; +Cc: acm, emacs-devel, dgutov
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 16 Dec 2016 18:08:06 -0500
> Cc: Alan Mackenzie <acm@muc.de>, emacs-devel@gnu.org
>
> > The idea is to move the user-facing narrowing commands to some other
> > foundation (like the invisible and maybe intangible text properties),
> > freeing the narrowing facility for use by the Lisp code.
>
> While it might be a good direction, it's not as simple as it sounds,
> since many uses of the C-x n n command looks like "narrow to the region,
> and then apply <blabla> to the whole buffer": I think it will be tricky
> to make overlays that mimick this behavior sufficiently well.
Do you have an alternative proposal?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 2:26 ` Drew Adams
@ 2016-12-17 14:37 ` Stefan Monnier
0 siblings, 0 replies; 299+ messages in thread
From: Stefan Monnier @ 2016-12-17 14:37 UTC (permalink / raw)
To: Drew Adams; +Cc: Alan Mackenzie, emacs-devel, Dmitry Gutov
> But again, I am not asking that you adopt my code or my approach.
That's not what we're discussing. We're using that code to try and
illustrate the kind of difficulties that narrowing+widen introduces and
which are relevant in the context of syntax-ppss.
And which both you and Alan seem completely unable to understand for
reasons which baffle me.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 7:42 ` Eli Zaretskii
@ 2016-12-17 14:41 ` Stefan Monnier
2016-12-17 14:49 ` Eli Zaretskii
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-17 14:41 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: acm, emacs-devel, dgutov
>> > The idea is to move the user-facing narrowing commands to some other
>> > foundation (like the invisible and maybe intangible text properties),
>> > freeing the narrowing facility for use by the Lisp code.
>> While it might be a good direction, it's not as simple as it sounds,
>> since many uses of the C-x n n command looks like "narrow to the region,
>> and then apply <blabla> to the whole buffer": I think it will be tricky
>> to make overlays that mimick this behavior sufficiently well.
> Do you have an alternative proposal?
Yes, I think it's easier to keep using narrow-to-region for that, but to
supplement the narrowing with some information as to whether widening
that narrowing is permissible or not (along the lines of
font-lock-dont-widen, but not specific to font-lock and allowing
intermediate narrowing so that we can widen the user's narrowing in an
Info buffer without widening all the way to point-min==1).
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 14:41 ` Stefan Monnier
@ 2016-12-17 14:49 ` Eli Zaretskii
2016-12-17 15:09 ` Lars Ingebrigtsen
2016-12-17 15:38 ` Stefan Monnier
0 siblings, 2 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-17 14:49 UTC (permalink / raw)
To: Stefan Monnier; +Cc: acm, emacs-devel, dgutov
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru, acm@muc.de, emacs-devel@gnu.org
> Date: Sat, 17 Dec 2016 09:41:18 -0500
>
> > Do you have an alternative proposal?
>
> Yes, I think it's easier to keep using narrow-to-region for that, but to
> supplement the narrowing with some information as to whether widening
> that narrowing is permissible or not (along the lines of
> font-lock-dont-widen, but not specific to font-lock and allowing
> intermediate narrowing so that we can widen the user's narrowing in an
> Info buffer without widening all the way to point-min==1).
What about nested narrowing? Don't we want to be able to keep the
information about the outer restriction, so that we'd be able to
restore it later?
Also, just when would widening or narrowing be "not permissible"?
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 14:49 ` Eli Zaretskii
@ 2016-12-17 15:09 ` Lars Ingebrigtsen
2016-12-18 20:08 ` Dmitry Gutov
2017-02-05 23:33 ` Drew Adams
2016-12-17 15:38 ` Stefan Monnier
1 sibling, 2 replies; 299+ messages in thread
From: Lars Ingebrigtsen @ 2016-12-17 15:09 UTC (permalink / raw)
To: emacs-devel
I haven't really followed the discussion here, so this has probably been
all said before. :-) But I, too, find the narrowing functionality in
Emacs to be less than perfect.
There's three major use cases:
1) Modes that use narrowing as a required thing for the mode to work
properly. The main offender here is Info-mode, and I think it should
just be rewritten to not rely on narrowing to work. That is: No
Narrowing In Modes. There's always a better way to achieve the same
effects, I think.
2) The programmatic use in Lisp programs always looks like this:
(save-restriction
(narrow-to-region (point) (point-max))
...)
or something, which is less clear than it could be. As we did with
`(save-excursion (set-buffer...))' we could introduce a new special form
for this behaviour, like
(with-region (point) (point-max)
...)
3) Interactive narrowing should be stack-based, so that you can easily
narrow to things, do something, then narrow some more, and then pop that
narrowing to return to the previous. It's something I have wanted for a
long time.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 14:49 ` Eli Zaretskii
2016-12-17 15:09 ` Lars Ingebrigtsen
@ 2016-12-17 15:38 ` Stefan Monnier
2016-12-19 2:31 ` Dmitry Gutov
1 sibling, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-17 15:38 UTC (permalink / raw)
To: emacs-devel
> Also, just when would widening or narrowing be "not permissible"?
Good question. AFAIK this question (and other similar) is the
fundamental issue w.r.t when should syntax-ppss (or any replacement
there of) use point-min and when it should use something else.
My tentative answer is simply "when the code that installed this
narrowing said so".
This said, another option is to declare that such are misuses of
narrowing (like Lars suggests) and then use some other way to indicate
which "buffer start" position to use in syntax-ppss. As a matter of
fact that's consistent with what I advocated back in the
"indentation-context" discussion (and was then implemented and later
removed).
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 15:09 ` Lars Ingebrigtsen
@ 2016-12-18 20:08 ` Dmitry Gutov
2017-02-05 23:33 ` Drew Adams
1 sibling, 0 replies; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-18 20:08 UTC (permalink / raw)
To: Lars Ingebrigtsen, emacs-devel
On 17.12.2016 17:09, Lars Ingebrigtsen wrote:
> 1) Modes that use narrowing as a required thing for the mode to work
> properly. The main offender here is Info-mode, and I think it should
> just be rewritten to not rely on narrowing to work. That is: No
> Narrowing In Modes. There's always a better way to achieve the same
> effects, I think.
Info could be implemented differently, I agree. And maybe it's a good idea.
There's no comparably powerful alternative for the multiple-major-modes
situation, however.
> As we did with
> `(save-excursion (set-buffer...))' we could introduce a new special form
> for this behaviour, like
>
> (with-region (point) (point-max)
> ...)
Not sure how much that would solve. The example mentioned previously, as
I understood it, was about users calling narrow-to-region interactively,
and following it by another interactive function that affects all
accessible part of the buffer.
> 3) Interactive narrowing should be stack-based, so that you can easily
> narrow to things, do something, then narrow some more, and then pop that
> narrowing to return to the previous.
This probably won't require any changes in the semantics of the
narrowing primitives.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-17 15:38 ` Stefan Monnier
@ 2016-12-19 2:31 ` Dmitry Gutov
2016-12-19 13:12 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-19 2:31 UTC (permalink / raw)
To: Stefan Monnier, emacs-devel
On 17.12.2016 17:38, Stefan Monnier wrote:
> This said, another option is to declare that such are misuses of
> narrowing (like Lars suggests) and then use some other way to indicate
> which "buffer start" position to use in syntax-ppss. As a matter of
> fact that's consistent with what I advocated back in the
> "indentation-context" discussion (and was then implemented and later
> removed).
That removal's on me, at least in part. So if we don't agree on an
alternative approach, I'd post a patch to bring prog-widen (better name
wanted?) back.
That said, it won't solve everything.
1. Minor thing: if we wanted to use it for Info, it would probably need
a more neutral name, not prog- suffix.
2. Using prog-widen in syntax-ppss is by itself equivalent to using a
syntax-ppss-dont-widen variable, which Alan doesn't like, in that
point-min cache management is left up to the caller. Do you like that
approach, or prefer a more opaque cache invalidation implemented inside
syntax-ppss?
All that aside, here's an approach that I feel hasn't been given due
attention in previous discussions: why don't we push all widen calls up
the call stack?
- Prohibit font-lock-keywords from widening.
font-lock-default-fontify-region already widens when needed. CC Mode is
the only violator I know of, but we can probably ignore it either way
because it would raise errors otherwise.
- Prohibit indent-line-function's from widening. Do that instead in
indent-for-tab-command.
- Likewise for M-x imenu and any similar facilities. Keep widening to
the commands, or some high level code that is not major mode-specific.
font-lock-default-fontify-region is a counter-example of sorts, but it
works for mmm-mode because of font-lock-dont-widen.
All these are hard to verify for arbitrary code, but we'd have these
rules as a "Mixed-mode major modes" guidelines, and could make sure they
are followed in the built-in modes. Most of them, at least.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-19 2:31 ` Dmitry Gutov
@ 2016-12-19 13:12 ` Stefan Monnier
2016-12-19 13:34 ` Dmitry Gutov
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-19 13:12 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: emacs-devel
> - Likewise for M-x imenu and any similar facilities. Keep widening to the
> commands, or some high level code that is not major
> mode-specific. font-lock-default-fontify-region is a counter-example of
> sorts, but it works for mmm-mode because of font-lock-dont-widen.
But that means any command which somewhere internally uses syntax-ppss
would need to widen? So any change which introduces a new call to
syntax-ppss would imply revising all callers of this code and all
callers of those callers (etc...) to make sure they widen?
What if they don't want to widen because they're commands which *should*
limit themselves to the narrowed region?
That seems to presume that narrowing is only used for multi-mode kind of
contexts, which currently is the exception rather than the rule.
I think the core of the problem is the use of narrowing for widely
different purposes, so indeed we should decide that some of those uses
are simply inappropriate and provided something else for those uses.
But I think the case "user-level narrow for Q&R" is hard to replace
reliably, so I'd rather keep narrowing for *that* purpose, and try to
introduce something else for the other uses (e.g. for multi-mode).
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-19 13:12 ` Stefan Monnier
@ 2016-12-19 13:34 ` Dmitry Gutov
2016-12-19 14:17 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-19 13:34 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
On 19.12.2016 15:12, Stefan Monnier wrote:
> But that means any command which somewhere internally uses syntax-ppss
> would need to widen? So any change which introduces a new call to
> syntax-ppss would imply revising all callers of this code and all
> callers of those callers (etc...) to make sure they widen?
No, syntax-ppss would probably be an exception, and call `widen' itself,
as well as include a syntax-ppss-dont-widen variable, or
syntax-ppss-parse-start, like proposed by Alan.
This seems necessary to have a stable syntax cache, e.g. to have it as a
basis of comments navigation.
> What if they don't want to widen because they're commands which *should*
> limit themselves to the narrowed region?
I see no big obstacles to that, so far.
> That seems to presume that narrowing is only used for multi-mode kind of
> contexts, which currently is the exception rather than the rule.
The idea is to preserve its usability for both cases, by moving the
choice when to narrow or widen up the stack, closer to the commands that
the user invokes (if indeed the code is called interactively), because
they are likely to be more aware of the user's intent.
> I think the core of the problem is the use of narrowing for widely
> different purposes, so indeed we should decide that some of those uses
> are simply inappropriate and provided something else for those uses.
That's one option. But again, prog-widen doesn't solve everything, and
you haven't commented on syntax-ppss's cache management ideas.
> But I think the case "user-level narrow for Q&R" is hard to replace
> reliably, so I'd rather keep narrowing for *that* purpose, and try to
> introduce something else for the other uses (e.g. for multi-mode).
You're saying that python-indent-line and its callees should call
prog-widen, right?
I'm saying that's unnecessary, and that indent-for-tab-command can do
all the widening itself. Same for font-lock. And that takes care of the
two known obstacles in the mixed-major-mode space, with very little code
or changes to the API.
Even if we add "something else", upon deciding that it's really needed,
later, the current proposal shouldn't hurt.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-19 13:34 ` Dmitry Gutov
@ 2016-12-19 14:17 ` Stefan Monnier
2016-12-19 23:16 ` Dmitry Gutov
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-19 14:17 UTC (permalink / raw)
To: emacs-devel
> No, syntax-ppss would probably be an exception, and call `widen' itself, as
> well as include a syntax-ppss-dont-widen variable, or
> syntax-ppss-parse-start, like proposed by Alan.
Then I misunderstood. What is the problem that your proposal aims to fix?
> This seems necessary to have a stable syntax cache, e.g. to have it
> as a basis of comments navigation.
But if we have something like syntax-ppss-parse-start, why not use that
same info more generally when some function wants to widen (e.g. make
it also replace font-lock-dont-widen)?
>> That seems to presume that narrowing is only used for multi-mode kind of
>> contexts, which currently is the exception rather than the rule.
> The idea is to preserve its usability for both cases, by moving the choice
> when to narrow or widen up the stack, closer to the commands that the user
> invokes (if indeed the code is called interactively), because they are
> likely to be more aware of the user's intent.
But the problem is the linkage between the narrowing and the widening,
not between the user-command and the widening.
In general, the command currently executed does not know who put the
narrowing nor why, so it still doesn't know what's the intention behind
it, and hence can't decide whether to widen or not, nor or much to widen.
>> I think the core of the problem is the use of narrowing for widely
>> different purposes, so indeed we should decide that some of those uses
>> are simply inappropriate and provided something else for those uses.
> That's one option. But again, prog-widen doesn't solve everything, and you
Not sure which problem you're referring to. Do you mean that when
syntax-ppss-parse-start is changed, syntax-ppss is not warned?
> haven't commented on syntax-ppss's cache management ideas.
Not sure on which part you want my input. I can't remember all that's
been suggested, so here's just some random opinions of mine on the
subject:
- we could/should have a macro (with-new-syntax-ppss-context
... &rest BODY) which runs BODY such that calls to syntax-ppss in
there can be cached within this scope but don't use the outer
syntax-ppss cache and don't affect it. This could be used when
a command wants to temporarily use an ad-hoc syntax-table (typically
using with-syntax-table), for example. It should probably be made to
obey the current narrowing limits, since such uses often additionally
want to restrict the parsing to a subpart of the buffer.
- if we introduce a syntax-ppss-parse-start, it should not have `ppss`
in its name, it should probably try to include some of the design of
indentation-context, and it should probably be modified via functions
(rather than directly via setq), so that those functions can flush the
syntax-ppss cache when needed. This said, if needed it'd probably be
OK to have syntax-ppss auto-detect changes instead (after all,
syntax-ppss is still implemented in Elisp, so there is a lot of room
for speed up).
>> But I think the case "user-level narrow for Q&R" is hard to replace
>> reliably, so I'd rather keep narrowing for *that* purpose, and try to
>> introduce something else for the other uses (e.g. for multi-mode).
> You're saying that python-indent-line and its callees should call
> prog-widen, right?
Not necessarily, no.
> I'm saying that's unnecessary, and that indent-for-tab-command can do all
> the widening itself.
I wouldn't do it in indent-for-tab-command, but I agree that it should
be done in indent-according-to-mode rather than forcing every major
mode's indentation function to widen.
> Same for font-lock. And that takes care of the two known obstacles in
> the mixed-major-mode space, with very little code or changes to
> the API.
Sure. I think the current discussion is about "what should
prog-widen/syntax-ppss-parse-start look like" more than "who/when should
we call prog-widen".
> Even if we add "something else", upon deciding that it's really needed,
> later, the current proposal shouldn't hurt.
If by "current proposal" you mean to prog-widen in
indent-according-to-mode, then I agree it's a good idea.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-19 14:17 ` Stefan Monnier
@ 2016-12-19 23:16 ` Dmitry Gutov
2016-12-20 13:33 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2016-12-19 23:16 UTC (permalink / raw)
To: Stefan Monnier, emacs-devel
On 19.12.2016 16:17, Stefan Monnier wrote:
> Then I misunderstood.
Maybe I worded that too strongly. How about consider the major
facilities case-by-case, and see if moving widening up the stack makes
sense? Just doing that for indentation and font-lock (maybe there's even
nothing to change there) would be a major win already.
> What is the problem that your proposal aims to fix?
To solve the "widen problem" that mixed-mode frameworks have with
existing modes. Namely, when they delegate to some language-specific
code, it could call `widen' before doing its work. Examples of
language-specific code are the values of font-lock-keywords,
indent-line-function and imenu-create-index-function (though the last
one is less important).
And to solve them in a non-intrusive way that does not ask for new
abstractions, and thus does not affect the rest of the discussion (about
syntax-ppss, forward-comment, Info). That should simplify things.
>> This seems necessary to have a stable syntax cache, e.g. to have it
>> as a basis of comments navigation.
>
> But if we have something like syntax-ppss-parse-start, why not use that
> same info more generally when some function wants to widen (e.g. make
> it also replace font-lock-dont-widen)?
I'd imagine having a new variable syntax-dont-widen, that would serve
both for syntax-ppss and font-lock. Still, that does not resolve the
issue of indexing by point-min.
> But the problem is the linkage between the narrowing and the widening,
> not between the user-command and the widening.
There are different problems. I'm most focused on the issue of code
undoing narrowing willy-nilly.
> In general, the command currently executed does not know who put the
> narrowing nor why, so it still doesn't know what's the intention behind
> it, and hence can't decide whether to widen or not, nor or much to widen.
Aside from the problem of Info (which we know can be implemented in a
different way), an interactive command does know, with 98% probability,
that it's the user who instituted the narrowing.
And even in Info-mode as it works now, we can provide an Info-specific
widening command (bound to the usual key sequence) which would only
widen to the current page instead of the whole buffer.
>> That's one option. But again, prog-widen doesn't solve everything, and you
>
> Not sure which problem you're referring to. Do you mean that when
> syntax-ppss-parse-start is changed, syntax-ppss is not warned?
It can track the value of point-min, but it won't really know what to do
with the caches corresponding to the previous point-min values. Keep
them around? Discard some of them?
See the end of this message:
http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00630.html
>> haven't commented on syntax-ppss's cache management ideas.
>
> Not sure on which part you want my input.
Some help choosing would be nice. :)
> I can't remember all that's
> been suggested, so here's just some random opinions of mine on the
> subject:
> - we could/should have a macro (with-new-syntax-ppss-context
> ... &rest BODY) which runs BODY such that calls to syntax-ppss in
> there can be cached within this scope but don't use the outer
> syntax-ppss cache and don't affect it. This could be used when
> a command wants to temporarily use an ad-hoc syntax-table (typically
> using with-syntax-table), for example. It should probably be made to
> obey the current narrowing limits, since such uses often additionally
> want to restrict the parsing to a subpart of the buffer.
That seems sound. And, basically, all that's necessary for mixed-mode
packages to work. Except for the "interleaving pieces of code" case,
which I think Christopher mentioned is the case in Antlr. But that's an
advanced problem.
> - if we introduce a syntax-ppss-parse-start, it should not have `ppss`
> in its name, it should probably try to include some of the design of
> indentation-context,
Which ones? Having both lower and upper bound? It's not indentation
specific, so using the name is out of the question, I think.
> and it should probably be modified via functions
> (rather than directly via setq), so that those functions can flush the
> syntax-ppss cache when needed.
This sounds interesting. Care to add some details? Just a hook with
functions that return an integer?
If we're going this way, an ability to substitute the cache, instead of
just flushing it, sounds like a logical continuation of that idea. That
might require a getter and setter for the cache, like we have for
match-data.
We probably don't need to do that now, just keep the option open.
> This said, if needed it'd probably be
> OK to have syntax-ppss auto-detect changes instead (after all,
> syntax-ppss is still implemented in Elisp, so there is a lot of room
> for speed up).
It might not we the fastest, but I'd like it to be algorithmically sound.
> I wouldn't do it in indent-for-tab-command, but I agree that it should
> be done in indent-according-to-mode rather than forcing every major
> mode's indentation function to widen.
Shouldn't indent-for-tab-command call indent-according-to-mode, then?
>> Same for font-lock. And that takes care of the two known obstacles in
>> the mixed-major-mode space, with very little code or changes to
>> the API.
>
> Sure. I think the current discussion is about "what should
> prog-widen/syntax-ppss-parse-start look like" more than "who/when should
> we call prog-widen".
I prefer to think of this as the next step.
>> Even if we add "something else", upon deciding that it's really needed,
>> later, the current proposal shouldn't hurt.
>
> If by "current proposal" you mean to prog-widen in
> indent-according-to-mode, then I agree it's a good idea.
Yes, and also agree about a similar rule for font-lock-keywords.
I really meant widen, though. Not sure what calling prog-widen in there
would mean, semantically, because according to the previous discussion
prog-narrowing was something a mixed-mode code would institute. But it's
down the call stack from indent-according-to-mode (via a mixed-mode
aware indent-line-function), not above it.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Wording of the Elisp manuals
2016-12-17 4:50 ` Wording of the Elisp manuals (was: forward-comment and syntax-ppss) Marcin Borkowski
@ 2016-12-20 9:22 ` Michael Heerdegen
2016-12-20 10:52 ` Wording of the Elisp manuals (was: forward-comment and syntax-ppss) Jean-Christophe Helary
1 sibling, 0 replies; 299+ messages in thread
From: Michael Heerdegen @ 2016-12-20 9:22 UTC (permalink / raw)
To: Marcin Borkowski; +Cc: emacs-devel
Marcin Borkowski <mbork@mbork.pl> writes:
> Wouldn't it be a good idea to state clearly somewhere that ELisp Intro
> is not only about introduction to programming, but also about
> introdcution to Emacs as the Elisp environment?
Do we expect that an more or less experienced programmer (who probably
already has a basic understanding of Lisp) needs to read the Elisp
Intro? I would OTOH not be good if we say that reading the Intro is
mandatory, but everything is also covered by the manual.
Michael.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Wording of the Elisp manuals (was: forward-comment and syntax-ppss)
2016-12-17 4:50 ` Wording of the Elisp manuals (was: forward-comment and syntax-ppss) Marcin Borkowski
2016-12-20 9:22 ` Wording of the Elisp manuals Michael Heerdegen
@ 2016-12-20 10:52 ` Jean-Christophe Helary
2016-12-20 16:01 ` Eli Zaretskii
1 sibling, 1 reply; 299+ messages in thread
From: Jean-Christophe Helary @ 2016-12-20 10:52 UTC (permalink / raw)
To: emacs-devel
> On 2016 Dec 17, at 13:50, Marcin Borkowski <mbork@mbork.pl> wrote:
>
> Wouldn't it be a good idea to state clearly somewhere that ELisp Intro
> is not only about introduction to programming, but also about
> introdcution to Emacs as the Elisp environment? Narrowing is first
> mentioned in Chapter 9 of the Elisp Reference and only described in
> detail in Chapter 29 (!).
The problem with the Elisp Intro is that it is extremely verbose and it is not really easy to find the information you're looking for.
It also says that people who want an introduction to emacs should use the interactive tutorial so it's not really easy for people who know the basics of Emacs and of programming to find it useful before committing oneself to read a good bunch of it.
Jean-Christophe Helary
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: forward-comment and syntax-ppss
2016-12-19 23:16 ` Dmitry Gutov
@ 2016-12-20 13:33 ` Stefan Monnier
2017-11-29 23:48 ` multi-modes and widen (again), was: " Dmitry Gutov
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2016-12-20 13:33 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: emacs-devel
> It can track the value of point-min, but it won't really know what to do
> with the caches corresponding to the previous point-min values.
I'd expect they'd get flushed by syntax-ppss-flush-cache.
But I don't really see why you think there's a problem to solve there.
I assume syntax-ppss would usually "widen to syntax-start-pos" or
somesuch, so there should not be many different point-mins indexed there.
>> - we could/should have a macro (with-new-syntax-ppss-context
>> ... &rest BODY) which runs BODY such that calls to syntax-ppss in
>> there can be cached within this scope but don't use the outer
>> syntax-ppss cache and don't affect it. This could be used when
>> a command wants to temporarily use an ad-hoc syntax-table (typically
>> using with-syntax-table), for example. It should probably be made to
>> obey the current narrowing limits, since such uses often additionally
>> want to restrict the parsing to a subpart of the buffer.
> That seems sound. And, basically, all that's necessary for mixed-mode
> packages to work.
But the multi-mode case would additionally want to preserve the "inner
cache" across uses of with-new-syntax-ppss-context.
>> - if we introduce a syntax-ppss-parse-start, it should not have `ppss`
>> in its name, it should probably try to include some of the design of
>> indentation-context,
> Which ones? Having both lower and upper bound?
My memory is too fuzzy about that to have a clear opinion.
> It's not indentation specific, so using the name is out of the
> question, I think.
I wasn't thinking about re-using the name, no.
>> This said, if needed it'd probably be OK to have syntax-ppss
>> auto-detect changes instead (after all, syntax-ppss is still
>> implemented in Elisp, so there is a lot of room for speed up).
> It might not we the fastest, but I'd like it to be algorithmically sound.
Detecting changes is algorithmically cheap.
>> I wouldn't do it in indent-for-tab-command, but I agree that it should
>> be done in indent-according-to-mode rather than forcing every major
>> mode's indentation function to widen.
> Shouldn't indent-for-tab-command call indent-according-to-mode, then?
Doesn't it? Oh wait, it technically doesn't because it wants to handle
the `no-indent` return value of `indent-line-function` differently.
But conceptually, I still consider it a call to indent-according-to-mode ;-)
> Yes, and also agree about a similar rule for font-lock-keywords.
That's already the case, no?
> I really meant widen, though. Not sure what calling prog-widen in there
> would mean, semantically, because according to the previous discussion
> prog-narrowing was something a mixed-mode code would institute. But it's
> down the call stack from indent-according-to-mode (via a mixed-mode aware
> indent-line-function), not above it.
Indeed, but I meant "somewhere between a call to
indent-according-to-mode and the call to major-mode specific
indent-line-function".
The point is that the indent-line-function should not need to widen,
although maybe the buffer is not fully widened (depending on whether
the multi-mode system uses narrowing, maybe).
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Wording of the Elisp manuals (was: forward-comment and syntax-ppss)
2016-12-20 10:52 ` Wording of the Elisp manuals (was: forward-comment and syntax-ppss) Jean-Christophe Helary
@ 2016-12-20 16:01 ` Eli Zaretskii
0 siblings, 0 replies; 299+ messages in thread
From: Eli Zaretskii @ 2016-12-20 16:01 UTC (permalink / raw)
To: Jean-Christophe Helary; +Cc: emacs-devel
> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Tue, 20 Dec 2016 19:52:28 +0900
>
>
> > On 2016 Dec 17, at 13:50, Marcin Borkowski <mbork@mbork.pl> wrote:
> >
> > Wouldn't it be a good idea to state clearly somewhere that ELisp Intro
> > is not only about introduction to programming, but also about
> > introdcution to Emacs as the Elisp environment? Narrowing is first
> > mentioned in Chapter 9 of the Elisp Reference and only described in
> > detail in Chapter 29 (!).
>
> The problem with the Elisp Intro is that it is extremely verbose and it is not really easy to find the information you're looking for.
>
> It also says that people who want an introduction to emacs should use the interactive tutorial so it's not really easy for people who know the basics of Emacs and of programming to find it useful before committing oneself to read a good bunch of it.
I don't think we have a problem with the Introduction. It's a gentle
introduction to many topics, including ELisp, and is written in a
certain style that fits its purpose. The reference we have to it in
the other manuals is just a reminder that it exists and could be
useful for some. If people like it, fine. If they don't, they can
skip it without losing anything, except perhaps some pleasure from
reading a well-written document.
I see no problem in the fact that narrowing is described in Chapter 29
of the ELisp manual. No one in their right minds should read that
manual in its entirety in one go, top to bottom. Instead, one should
look up the subject they are interested in, and then read around
that. In this case, the description of point-min, reachable via index
searching commands, says this right at its beginning:
-- Function: point-min
This function returns the minimum accessible value of point in the
current buffer. This is normally 1, but if narrowing is in effect,
it is the position of the start of the region that you narrowed to.
(*Note Narrowing::.)
And that cross-reference will get you directly to Chapter 29, without
the need to read the 28 chapters preceding it.
So I don't think there's any problem here we need to worry about.
Thanks.
^ permalink raw reply [flat|nested] 299+ messages in thread
* RE: forward-comment and syntax-ppss
2016-12-17 15:09 ` Lars Ingebrigtsen
2016-12-18 20:08 ` Dmitry Gutov
@ 2017-02-05 23:33 ` Drew Adams
1 sibling, 0 replies; 299+ messages in thread
From: Drew Adams @ 2017-02-05 23:33 UTC (permalink / raw)
To: Lars Ingebrigtsen, emacs-devel
> 3) Interactive narrowing should be stack-based, so that
> you can easily narrow to things, do something, then
> narrow some more, and then pop that narrowing to return
> to the previous. It's something I have wanted for a
> long time.
Well, you have it. That's just what I showed with the
`zones.el' code I use.
See this message in the same thread:
http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00693.html
and https://www.emacswiki.org/emacs/MultipleNarrowings
for a general description.
Just as you requested, when you narrow the buffer
interactively (only), that narrowing (restriction, or
"zone") is put on a stack.
Only it's not just a stack. It's a ring. You can access
it as a stack, a list, or a ring, or directly access
(restore) any restriction by its identifier or its ring
position. And markers are used for the restriction limits,
so the zones adjust to text insertions and deletions.
And don't think that the restrictions on the ring need
to all be nested inside each other like russian dolls.
They can each have any buffer limits. They can even be
in different buffers (or not). And you can have any
number of such rings at the same time.
Though very simple, it is really quite useful, IMO.
This message shows the code that pushes the restriction
to the current ring:
http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00702.html
It just advises `narrow-to-region' (which is defined in C).
Of course, _use_ of the ring is not limited to interactive
use. It is just automatic pushing to the ring that is done
only interactively.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-02 10:56 ` Eli Zaretskii
@ 2017-05-26 19:48 ` Thien-Thi Nguyen
2017-05-26 20:26 ` Kaushal Modi
0 siblings, 1 reply; 299+ messages in thread
From: Thien-Thi Nguyen @ 2017-05-26 19:48 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 805 bytes --]
() Eli Zaretskii <eliz@gnu.org>
() Fri, 02 Dec 2016 12:56:15 +0200
What do you expect me to do when I see code which takes Emacs
in a direction I think is wrong? There are a very few of
such directions I feel strongly we shouldn't take. What am I
supposed to do when I see one of them proposed for inclusion?
Look ahead to see the fork, point it out in ways people can
understand, and help them manage it when they arrive. If luck
smiles, everyone will remember and learn; if not, only you will.
--
Thien-Thi Nguyen -----------------------------------------------
(defun responsep (query)
(pcase (context query)
(`(technical ,ml) (correctp ml))
...)) 748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2017-05-26 19:48 ` Thien-Thi Nguyen
@ 2017-05-26 20:26 ` Kaushal Modi
2017-05-27 7:27 ` Thien-Thi Nguyen
0 siblings, 1 reply; 299+ messages in thread
From: Kaushal Modi @ 2017-05-26 20:26 UTC (permalink / raw)
To: emacs-devel, ttn
[-- Attachment #1: Type: text/plain, Size: 366 bytes --]
On Fri, May 26, 2017 at 3:44 PM Thien-Thi Nguyen <ttn@gnu.org> wrote:
> Look ahead to see the fork, point it out in ways people can
> understand, and help them manage it when they arrive. If luck
> smiles, everyone will remember and learn; if not, only you will.
>
??
I don't think the fork is happening. You are replying to a very old thread.
--
Kaushal Modi
[-- Attachment #2: Type: text/html, Size: 742 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2017-05-26 20:26 ` Kaushal Modi
@ 2017-05-27 7:27 ` Thien-Thi Nguyen
0 siblings, 0 replies; 299+ messages in thread
From: Thien-Thi Nguyen @ 2017-05-27 7:27 UTC (permalink / raw)
To: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 533 bytes --]
() Kaushal Modi <kaushal.modi@gmail.com>
() Fri, 26 May 2017 20:26:25 +0000
I don't think the fork is happening.
"There is no fork." :-D
You are replying to a very old thread.
It seems we both are. Hopefully that's not a problem.
--
Thien-Thi Nguyen -----------------------------------------------
(defun responsep (query)
(pcase (context query)
(`(technical ,ml) (correctp ml))
...)) 748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2016-12-15 14:28 ` Philippe Vaucher
@ 2017-10-18 23:36 ` Kaushal Modi
2017-10-19 10:12 ` Jeremie Courreges-Anglas
0 siblings, 1 reply; 299+ messages in thread
From: Kaushal Modi @ 2017-10-18 23:36 UTC (permalink / raw)
To: Philippe Vaucher; +Cc: Karl Fogel, Daniel Colascione, Emacs developers
[-- Attachment #1: Type: text/plain, Size: 490 bytes --]
@Daniel
Just bumping the thread. Would like to know what came out of this.
On Thu, Dec 15, 2016, 9:31 AM Philippe Vaucher <philippe.vaucher@gmail.com>
wrote:
> On Fri, Dec 2, 2016 at 10:22 PM, Daniel Colascione <dancol@dancol.org>
> wrote:
>
>> FWIW, I'll put my patch on a branch this weekend.
>>
>
> Sorry if I missed it, but was this done? I can't find a relevant branch by
> looking at https://github.com/emacs-mirror/emacs/branches/active
>
> Thanks,
> Philippe
>
--
Kaushal Modi
[-- Attachment #2: Type: text/html, Size: 1375 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2017-10-18 23:36 ` Kaushal Modi
@ 2017-10-19 10:12 ` Jeremie Courreges-Anglas
2018-02-12 20:18 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: Jeremie Courreges-Anglas @ 2017-10-19 10:12 UTC (permalink / raw)
To: Kaushal Modi
Cc: Karl Fogel, Philippe Vaucher, Daniel Colascione, Emacs developers
[-- Attachment #1: Type: text/plain, Size: 934 bytes --]
On Wed, Oct 18 2017, Kaushal Modi <kaushal.modi@gmail.com> wrote:
> @Daniel
>
> Just bumping the thread. Would like to know what came out of this.
Same here. I don't really care whether the portable dumper or the "big
.elc file" approach is taken, but it would be cool to see emacs out of
the malloc and unexec business. This code is fragile and tends to break
on less popular platforms (OpenBSD in my case).
> On Thu, Dec 15, 2016, 9:31 AM Philippe Vaucher <philippe.vaucher@gmail.com>
> wrote:
>
>> On Fri, Dec 2, 2016 at 10:22 PM, Daniel Colascione <dancol@dancol.org>
>> wrote:
>>
>>> FWIW, I'll put my patch on a branch this weekend.
>>>
>>
>> Sorry if I missed it, but was this done? I can't find a relevant branch by
>> looking at https://github.com/emacs-mirror/emacs/branches/active
>>
>> Thanks,
>> Philippe
>>
--
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* multi-modes and widen (again), was: Re: forward-comment and syntax-ppss
2016-12-20 13:33 ` Stefan Monnier
@ 2017-11-29 23:48 ` Dmitry Gutov
2017-11-30 13:05 ` Stefan Monnier
0 siblings, 1 reply; 299+ messages in thread
From: Dmitry Gutov @ 2017-11-29 23:48 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
Hi Stefan,
I'm only now working on this. :-(
On 12/20/16 1:33 PM, Stefan Monnier wrote:
>> I really meant widen, though. Not sure what calling prog-widen in there
>> would mean, semantically, because according to the previous discussion
>> prog-narrowing was something a mixed-mode code would institute. But it's
>> down the call stack from indent-according-to-mode (via a mixed-mode aware
>> indent-line-function), not above it.
>
> Indeed, but I meant "somewhere between a call to
> indent-according-to-mode and the call to major-mode specific
> indent-line-function".
>
> The point is that the indent-line-function should not need to widen,
> although maybe the buffer is not fully widened (depending on whether
> the multi-mode system uses narrowing, maybe).
Yes, so, I've pushed a change like this into scratch/widen-less. It
removes prog-widen, thus leaving prog-indentation-context too complex
for the only remaining use it has left.
So I was thinking of replacing it with just a variable prog-first-column
(and an accessor function for convenience). The CHUNKS (or, now ...REST)
element will go away because nobody has found any practical use for it
by now.
prog-first-column is definitely needed, though. After thinking on it for
some time, I still haven't come up with a suitable replacement.
WDYT?
It is unfortunate that we're doing it this late in Emacs 26 development
cycle, but if we agree on this change, it will be fairly small.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: multi-modes and widen (again), was: Re: forward-comment and syntax-ppss
2017-11-29 23:48 ` multi-modes and widen (again), was: " Dmitry Gutov
@ 2017-11-30 13:05 ` Stefan Monnier
2017-12-01 23:57 ` Dmitry Gutov
0 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2017-11-30 13:05 UTC (permalink / raw)
To: Dmitry Gutov; +Cc: emacs-devel
> Yes, so, I've pushed a change like this into scratch/widen-less. It removes
> prog-widen, thus leaving prog-indentation-context too complex for the only
> remaining use it has left.
Looks good, thanks.
Stefan
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: multi-modes and widen (again), was: Re: forward-comment and syntax-ppss
2017-11-30 13:05 ` Stefan Monnier
@ 2017-12-01 23:57 ` Dmitry Gutov
0 siblings, 0 replies; 299+ messages in thread
From: Dmitry Gutov @ 2017-12-01 23:57 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
On 11/30/17 1:05 PM, Stefan Monnier wrote:
>> Yes, so, I've pushed a change like this into scratch/widen-less. It removes
>> prog-widen, thus leaving prog-indentation-context too complex for the only
>> remaining use it has left.
>
> Looks good, thanks.
Thank you. I've pushed another update, please take a look.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2017-10-19 10:12 ` Jeremie Courreges-Anglas
@ 2018-02-12 20:18 ` Daniel Colascione
2018-02-13 16:37 ` Eli Zaretskii
` (3 more replies)
0 siblings, 4 replies; 299+ messages in thread
From: Daniel Colascione @ 2018-02-12 20:18 UTC (permalink / raw)
To: Kaushal Modi, Philippe Vaucher, Karl Fogel, Emacs developers
On 10/19/2017 03:12 AM, Jeremie Courreges-Anglas wrote:
> On Wed, Oct 18 2017, Kaushal Modi <kaushal.modi@gmail.com> wrote:
>> @Daniel
>>
>> Just bumping the thread. Would like to know what came out of this.
>
> Same here. I don't really care whether the portable dumper or the "big
> .elc file" approach is taken, but it would be cool to see emacs out of
> the malloc and unexec business. This code is fragile and tends to break
> on less popular platforms (OpenBSD in my case).
I've pushed the portable dumper to the pdumper Savannah branch. It
should support a fully PIC Emacs.
I implemented demand-paging logic and backed all of it out: the demand
paging worked fine, but I wasn't able to figure out a good way to get GC
not to page in the whole region almost immediately anyway. I was
surprised to find that my Emacs GCs almost immediately after start.
Timing on the current code is nevertheless pretty good. Running emacs -Q
-nw -batch --eval '(progn (garbage-collect) (kill-emacs))' 100 times,
unexec: 22.47ms per invocation
pdumper: 28.19ms per invocation
I think this degree of startup overhead is acceptable.
I could implement the non-PIC mode I'd previously mentioned; I'm
confident it would be faster than unexec. But with PIC doing this well,
why bother? We want a PIC Emacs for security reasons anyway.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-12 20:18 ` Daniel Colascione
@ 2018-02-13 16:37 ` Eli Zaretskii
2018-02-14 21:03 ` Philipp Stephani
` (4 more replies)
2018-02-15 4:28 ` Stefan Monnier
` (2 subsequent siblings)
3 siblings, 5 replies; 299+ messages in thread
From: Eli Zaretskii @ 2018-02-13 16:37 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 12 Feb 2018 12:18:36 -0800
>
> I've pushed the portable dumper to the pdumper Savannah branch. It
> should support a fully PIC Emacs.
Thanks. I'd urge people to try this branch and report any issues they
see.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-13 16:37 ` Eli Zaretskii
@ 2018-02-14 21:03 ` Philipp Stephani
2018-02-15 0:42 ` Daniel Colascione
2018-02-15 23:31 ` Ken Brown
` (3 subsequent siblings)
4 siblings, 1 reply; 299+ messages in thread
From: Philipp Stephani @ 2018-02-14 21:03 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Daniel Colascione, emacs-devel
[-- Attachment #1.1: Type: text/plain, Size: 447 bytes --]
Eli Zaretskii <eliz@gnu.org> schrieb am Di., 13. Feb. 2018 um 17:39 Uhr:
> > From: Daniel Colascione <dancol@dancol.org>
> > Date: Mon, 12 Feb 2018 12:18:36 -0800
> >
> > I've pushed the portable dumper to the pdumper Savannah branch. It
> > should support a fully PIC Emacs.
>
> Thanks. I'd urge people to try this branch and report any issues they
> see.
>
>
Got a crash when building on Debian testing. I've attached the output of
'bt full'.
[-- Attachment #1.2: Type: text/html, Size: 837 bytes --]
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gdb.txt --]
[-- Type: text/plain; charset="Shift_JIS"; name="gdb.txt", Size: 31042 bytes --]
#0 raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:51
set = {
__val = {0, 0, 0, 0, 0, 0, 0, 1095216660480, 204351844644765126, 2450524270238223137, 42930436372038613, 47150946645313795, 434264250787087240, 36173497990776353, 140721880134016, 6082251}
}
pid = <optimized out>
tid = <optimized out>
#1 0x0000000000588efb in terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:388
No locals.
#2 0x00000000006233cc in die (msg=0x7762b0 "offset < dump_private.header.discardable_start", file=0x774860 "pdumper.c", line=4718) at alloc.c:7789
No locals.
#3 0x000000000062dfb0 in pdumper_marked_p_impl (obj=0x7fb64a872168) at pdumper.c:4718
offset = 6930792
bitno = 29767524975390623
#4 0x0000000000588d83 in pdumper_marked_p (obj=0x7fb64a872168) at pdumper.h:205
No locals.
#5 0x000000000061c21f in cons_marked_p (c=0x7fb64a872168) at alloc.c:4163
No locals.
#6 0x0000000000621b74 in mark_object (arg=XIL(0x7fb64a87216b)) at alloc.c:7077
ptr = 0x7fb64a872168
obj = XIL(0x7fb64a872168)
po = 0x7fb64a872168
m = 0x7fb64ab440f4
cdr_count = 0
#7 0x000000000061d978 in mark_maybe_object (obj=XIL(0x7fb64a87216b)) at alloc.c:5018
po = 0x7fb64a872168
m = 0x0
#8 0x000000000061ddd9 in mark_memory (start=0x7ffc5dad7590, end=0x7ffc5dad9cb8) at alloc.c:5210
pp = 0x7ffc5dad8dc8 "k!\207J\266\177"
#9 0x000000000061de0e in mark_stack (bottom=0x7ffc5dad9cb8 " \037\215\002", end=0x7ffc5dad7590 "\240u\255]\374\177") at alloc.c:5417
No locals.
#10 0x00000000006ed837 in mark_one_thread (thread=0xe28540 <main_thread>) at thread.c:617
stack_top = 0x7ffc5dad7590
#11 0x00000000006ed9cf in mark_threads_callback (ignore=0x0) at thread.c:650
thread_obj = XIL(0xe28545)
iter = 0xe28540 <main_thread>
#12 0x000000000061de55 in flush_stack_call_func (func=0x6ed93c <mark_threads_callback>, arg=0x0) at alloc.c:5444
end = 0x7ffc5dad7590
self = 0xe28540 <main_thread>
sentry = {
o = {
__max_align_ll = 0,
__max_align_ld = <invalid float value>
}
}
#13 0x00000000006eda01 in mark_threads () at thread.c:657
No locals.
#14 0x0000000000620162 in garbage_collect_1 (end=0x7ffc5dad7780) at alloc.c:6322
nextb = 0x0
stack_top_variable = 0 '\000'
message_p = false
count = 25
start = {
tv_sec = 1518641881,
tv_nsec = 37396085
}
retval = XIL(0)
tot_before = 0
visitor = {
visit = 0x61fe06 <mark_object_root_visitor>,
data = 0x0
}
total = {XIL(0), XIL(0), XIL(0x7ffc5dad7600), XIL(0x5825cb), XIL(0x7ffc5dad7630), make_number(1623929), XIL(0x7ffc5dad7630), XIL(0xe11620), XIL(0xd260), XIL(0xd260)}
#15 0x0000000000620844 in Fgarbage_collect () at alloc.c:6493
end = 0x7ffc5dad7780
sentry = {
o = {
__max_align_ll = 0,
__max_align_ld = <invalid float value>
}
}
#16 0x0000000000585c2c in maybe_gc () at lisp.h:5002
No locals.
#17 0x0000000000655136 in eval_sub (form=XIL(0x29d1aa3)) at eval.c:2195
fun = make_number(1664453)
val = XIL(0x7ffc5dad78c0)
original_fun = XIL(0x5dad7888)
original_args = XIL(0x7fb64a673420)
funcar = XIL(0x5825cb)
count = 0
argvals = {XIL(0x58399f), XIL(0xd8fc45), XIL(0x7ffc5dad7830), XIL(0x5839c4), XIL(0x58399f), XIL(0xe043c0), XIL(0), XIL(0)}
#18 0x000000000068e41d in readevalloop (readcharfun=XIL(0x6a50), infile0=0x7ffc5dad7ad0, sourcename=XIL(0x29622c4), printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
count1 = 25
c = 35
val = XIL(0x29d1aa3)
count = 21
b = 0x0
continue_reading_p = true
lex_bound = XIL(0)
whole_buffer = false
first_sexp = false
macroexpand = XIL(0)
#19 0x000000000068c6ea in Fload (file=XIL(0x7fb64a20e0d4), noerror=XIL(0), nomessage=XIL(0xc660), nosuffix=XIL(0), must_suffix=XIL(0xc660)) at lread.c:1457
input = {
stream = 0x295ebc0,
lookahead = 0 '\000',
buf = "\000\000\000"
}
stream = 0x295ebc0
fd = 5
fd_index = 13
count = 13
found = XIL(0x295e854)
efound = XIL(0x295e854)
hist_file_name = XIL(0x295e854)
newer = false
compiled = true
handler = XIL(0x2942473)
safe_p = true
fmode = 0x7864bb "r"
version = 23
is_module = false
is_elc = 1
#20 0x0000000000654c54 in Fautoload_do_load (fundef=XIL(0x7fb64a731193), funname=XIL(0x7fb64992cda0), macro_only=XIL(0)) at eval.c:2075
count = 11
kind = XIL(0)
ignore_errors = XIL(0)
#21 0x00000000006571d1 in Ffuncall (nargs=1, args=0x7ffc5dad7d00) at eval.c:2839
fun = XIL(0x7fb64a731193)
original_fun = XIL(0x7fb64992cda0)
funcar = XIL(0x2cd0)
numargs = 0
val = XIL(0)
count = 10
#22 0x00000000006ad1ca in exec_byte_code (bytestr=XIL(0x7fb64a6e98b4), vector=XIL(0x7fb64a6e8685), maxdepth=make_number(23), args_template=make_number(257), nargs=1, args=0x7ffc5dad8688) at bytecode.c:632
op = 0
type = (unknown: 1251200684)
targets = {0x6b0161 <exec_byte_code+15833>, 0x6b0189 <exec_byte_code+15873>, 0x6b018b <exec_byte_code+15875>, 0x6b018d <exec_byte_code+15877>, 0x6b018f <exec_byte_code+15879>, 0x6b018f <exec_byte_code+15879>, 0x6b01f4 <exec_byte_code+15980>, 0x6b0268 <exec_byte_code+16096>, 0x6ac8f2 <exec_byte_code+1386>, 0x6ac8f4 <exec_byte_code+1388>, 0x6ac8f6 <exec_byte_code+1390>, 0x6ac8f8 <exec_byte_code+1392>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac900 <exec_byte_code+1400>, 0x6ac8c1 <exec_byte_code+1337>, 0x6acdd5 <exec_byte_code+2637>, 0x6acdd7 <exec_byte_code+2639>, 0x6acdd9 <exec_byte_code+2641>, 0x6acddb <exec_byte_code+2643>, 0x6acddd <exec_byte_code+2645>, 0x6acddd <exec_byte_code+2645>, 0x6ace12 <exec_byte_code+2698>, 0x6acde3 <exec_byte_code+2651>, 0x6ad0e7 <exec_byte_code+3423>, 0x6ad0e9 <exec_byte_code+3425>, 0x6ad0eb <exec_byte_code+3427>, 0x6ad0ed <exec_byte_code+3429>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0a1 <exec_byte_code+3353>, 0x6ad0b8 <exec_byte_code+3376>, 0x6ad197 <exec_byte_code+3599>, 0x6ad199 <exec_byte_code+3601>, 0x6ad19b <exec_byte_code+3603>, 0x6ad19d <exec_byte_code+3605>, 0x6ad19f <exec_byte_code+3607>, 0x6ad19f <exec_byte_code+3607>, 0x6ad151 <exec_byte_code+3529>, 0x6ad168 <exec_byte_code+3552>, 0x6ad24c <exec_byte_code+3780>, 0x6ad24e <exec_byte_code+3782>, 0x6ad250 <exec_byte_code+3784>, 0x6ad252 <exec_byte_code+3786>, 0x6ad254 <exec_byte_code+3788>, 0x6ad254 <exec_byte_code+3788>, 0x6ad206 <exec_byte_code+3710>, 0x6ad21d <exec_byte_code+3733>, 0x6adac7 <exec_byte_code+5951>, 0x6ad9ad <exec_byte_code+5669>, 0x6ad9a4 <exec_byte_code+5660>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6adcf8 <exec_byte_code+6512>, 0x6ade17 <exec_byte_code+6799>, 0x6ade81 <exec_byte_code+6905>, 0x6adeec <exec_byte_code+7012>, 0x6adf58 <exec_byte_code+7120>, 0x6acc22 <exec_byte_code+2202>, 0x6accaa <exec_byte_code+2338>, 0x6adfdc <exec_byte_code+7252>, 0x6acb60 <exec_byte_code+2008>, 0x6acd12 <exec_byte_code+2442>, 0x6ae051 <exec_byte_code+7369>, 0x6ae0b9 <exec_byte_code+7473>, 0x6ae101 <exec_byte_code+7545>, 0x6ae169 <exec_byte_code+7649>, 0x6ae1bb <exec_byte_code+7731>, 0x6ae28c <exec_byte_code+7940>, 0x6ae2d4 <exec_byte_code+8012>, 0x6ae33c <exec_byte_code+8116>, 0x6ae3c1 <exec_byte_code+8249>, 0x6ae409 <exec_byte_code+8321>, 0x6ae451 <exec_byte_code+8393>, 0x6ae4b9 <exec_byte_code+8497>, 0x6ae521 <exec_byte_code+8601>, 0x6ae589 <exec_byte_code+8705>, 0x6ae60e <exec_byte_code+8838>, 0x6ae660 <exec_byte_code+8920>, 0x6ae6b2 <exec_byte_code+9002>, 0x6ae783 <exec_byte_code+9211>, 0x6ae7fd <exec_byte_code+9333>, 0x6ae877 <exec_byte_code+9455>, 0x6aea46 <exec_byte_code+9918>, 0x6aeab3 <exec_byte_code+10027>, 0x6aeb20 <exec_byte_code+10136>, 0x6aeb8d <exec_byte_code+10245>, 0x6aebfa <exec_byte_code+10354>, 0x6aec4c <exec_byte_code+10436>, 0x6aecca <exec_byte_code+10562>, 0x6aed1c <exec_byte_code+10644>, 0x6aed6e <exec_byte_code+10726>, 0x6aedc0 <exec_byte_code+10808>, 0x6aeecc <exec_byte_code+11076>, 0x6ad827 <exec_byte_code+5279>, 0x6aef2a <exec_byte_code+11170>, 0x6aef72 <exec_byte_code+11242>, 0x6af03e <exec_byte_code+11446>, 0x6af0a7 <exec_byte_code+11551>, 0x6af105 <exec_byte_code+11645>, 0x6af14d <exec_byte_code+11717>, 0x6af193 <exec_byte_code+11787>, 0x6af1d9 <exec_byte_code+11857>, 0x6af227 <exec_byte_code+11935>, 0x6b0161 <exec_byte_code+15833>, 0x6af27f <exec_byte_code+12023>, 0x6af2c5 <exec_byte_code+12093>, 0x6af30b <exec_byte_code+12163>, 0x6af351 <exec_byte_code+12233>, 0x6af397 <exec_byte_code+12303>, 0x6af3dd <exec_byte_code+12373>, 0x6ad827 <exec_byte_code+5279>, 0x6b0161 <exec_byte_code+15833>, 0x6af425 <exec_byte_code+12445>, 0x6af47a <exec_byte_code+12530>, 0x6af4c2 <exec_byte_code+12602>, 0x6af50a <exec_byte_code+12674>, 0x6af572 <exec_byte_code+12778>, 0x6af5da <exec_byte_code+12882>, 0x6af622 <exec_byte_code+12954>, 0x6af73f <exec_byte_code+13239>, 0x6af7a7 <exec_byte_code+13343>, 0x6af80f <exec_byte_code+13447>, 0x6af877 <exec_byte_code+13551>, 0x6af8bd <exec_byte_code+13621>, 0x6b0161 <exec_byte_code+15833>, 0x6ad75b <exec_byte_code+5075>, 0x6ad303 <exec_byte_code+3963>, 0x6acace <exec_byte_code+1862>, 0x6ad3b6 <exec_byte_code+4142>, 0x6ad43a <exec_byte_code+4274>, 0x6ad4bb <exec_byte_code+4403>, 0x6ad70f <exec_byte_code+4999>, 0x6ad724 <exec_byte_code+5020>, 0x6ad04e <exec_byte_code+3270>, 0x6ad7de <exec_byte_code+5206>, 0x6ad85e <exec_byte_code+5334>, 0x6ad8ec <exec_byte_code+5476>, 0x6ad935 <exec_byte_code+5549>, 0x6adb13 <exec_byte_code+6027>, 0x6adb90 <exec_byte_code+6152>, 0x6adc15 <exec_byte_code+6285>, 0x6adc75 <exec_byte_code+6381>, 0x6ad2b5 <exec_byte_code+3885>, 0x6af905 <exec_byte_code+13693>, 0x6af98a <exec_byte_code+13826>, 0x6af9d2 <exec_byte_code+13898>, 0x6afa1a <exec_byte_code+13970>, 0x6afa62 <exec_byte_code+14042>, 0x6afaaa <exec_byte_code+14114>, 0x6afb12 <exec_byte_code+14218>, 0x6afb7a <exec_byte_code+14322>, 0x6afbe2 <exec_byte_code+14426>, 0x6afc4a <exec_byte_code+14530>, 0x6afdc0 <exec_byte_code+14904>, 0x6afe28 <exec_byte_code+15008>, 0x6afe90 <exec_byte_code+15112>, 0x6afed8 <exec_byte_code+15184>, 0x6aff40 <exec_byte_code+15288>, 0x6affa8 <exec_byte_code+15392>, 0x6afff0 <exec_byte_code+15464>, 0x6b0038 <exec_byte_code+15536>, 0x6aee12 <exec_byte_code+10890>, 0x6aee64 <exec_byte_code+10972>, 0x6b008a <exec_byte_code+15618>, 0x6b00f6 <exec_byte_code+15726>, 0x6b0161 <exec_byte_code+15833>, 0x6ad53c <exec_byte_code+4532>, 0x6ad559 <exec_byte_code+4561>, 0x6ad5c8 <exec_byte_code+4672>, 0x6ad637 <exec_byte_code+4783>, 0x6ad6a3 <exec_byte_code+4891>, 0x6ae20d <exec_byte_code+7813>, 0x6ae704 <exec_byte_code+9084>, 0x6aefbf <exec_byte_code+11319>, 0x6b02fb <exec_byte_code+16243>, 0x6b0370 <exec_byte_code+16360>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0406 <exec_byte_code+16510>, 0x6b048d <exec_byte_code+16645>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b06b7 <exec_byte_code+17199> <repeats 64 times>}
const_length = 106
bytestr_length = 1176
vectorp = 0x7fb64a6e8688
quitcounter = 104 'h'
stack_items = 24
sa_avail = 15016
sa_count = 8
sa_must_free = false
alloc = 0x7ffc5dad7c90
item_bytes = 192
stack_base = 0x7ffc5dad7c90
top = 0x7ffc5dad7d00
stack_lim = 0x7ffc5dad7d50
bytestr_data = 0x7ffc5dad7d50 "\306 \210\b\203\021"
pc = 0x7ffc5dad7e88 "\210\202d\003\016A㝃m\001\332\026B\001\206O\001\n\211A\022\242\344\001!\211@\001A\211\004GU\204e\001\333\345\005\003\313O\"\210\346\002!\266\005\202d\003\016A睃\345\001\313\350\351\016C\"\003\206\203\001\n\211A\022\242\211\262\r\313\332\036D\322\003\003\003#)\266\203\203\237\001\006\n\327\313O\262\vڲ\001\352\353\006\f!!\262\v\211\203\300\001\314\016E\006\fC\"\026E\006\t\203\341\001\016E\262\n\202\341\001\006\t\203\327\001\006\t\006\v\006\vAB\241\210\006\tA\262\n\202\341\001\006\n\016EB\211\026E\262\n\210\202d\003\016A읃\r\002\353\002\206\366\001\n\211A\022\242!\352\001!\355\001!\203\003\002"...
count = 8
result = XIL(0)
#23 0x0000000000657c82 in funcall_lambda (fun=XIL(0x7fb64a6e8655), nargs=1, arg_vector=0x7ffc5dad8680) at eval.c:3023
size = 5
val = XIL(0x1200e043c0)
syms_left = make_number(257)
next = XIL(0x62ee70)
lexenv = XIL(0x7ffc5dad85b0)
count = 8
i = 0
optional = false
rest = false
previous_optional_or_rest = 127
#24 0x0000000000657086 in Ffuncall (nargs=2, args=0x7ffc5dad8678) at eval.c:2824
fun = XIL(0x7fb64a6e8655)
original_fun = XIL(0x7fb6498e4260)
funcar = make_number(1623929)
numargs = 1
val = XIL(0x7ffc5dad8660)
count = 7
#25 0x00000000006ad1ca in exec_byte_code (bytestr=XIL(0x7fb64a6ed05c), vector=XIL(0x7fb64a6e9af5), maxdepth=make_number(20), args_template=make_number(0), nargs=0, args=0x7ffc5dad9268) at bytecode.c:632
op = 1
type = CONDITION_CASE
targets = {0x6b0161 <exec_byte_code+15833>, 0x6b0189 <exec_byte_code+15873>, 0x6b018b <exec_byte_code+15875>, 0x6b018d <exec_byte_code+15877>, 0x6b018f <exec_byte_code+15879>, 0x6b018f <exec_byte_code+15879>, 0x6b01f4 <exec_byte_code+15980>, 0x6b0268 <exec_byte_code+16096>, 0x6ac8f2 <exec_byte_code+1386>, 0x6ac8f4 <exec_byte_code+1388>, 0x6ac8f6 <exec_byte_code+1390>, 0x6ac8f8 <exec_byte_code+1392>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac900 <exec_byte_code+1400>, 0x6ac8c1 <exec_byte_code+1337>, 0x6acdd5 <exec_byte_code+2637>, 0x6acdd7 <exec_byte_code+2639>, 0x6acdd9 <exec_byte_code+2641>, 0x6acddb <exec_byte_code+2643>, 0x6acddd <exec_byte_code+2645>, 0x6acddd <exec_byte_code+2645>, 0x6ace12 <exec_byte_code+2698>, 0x6acde3 <exec_byte_code+2651>, 0x6ad0e7 <exec_byte_code+3423>, 0x6ad0e9 <exec_byte_code+3425>, 0x6ad0eb <exec_byte_code+3427>, 0x6ad0ed <exec_byte_code+3429>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0a1 <exec_byte_code+3353>, 0x6ad0b8 <exec_byte_code+3376>, 0x6ad197 <exec_byte_code+3599>, 0x6ad199 <exec_byte_code+3601>, 0x6ad19b <exec_byte_code+3603>, 0x6ad19d <exec_byte_code+3605>, 0x6ad19f <exec_byte_code+3607>, 0x6ad19f <exec_byte_code+3607>, 0x6ad151 <exec_byte_code+3529>, 0x6ad168 <exec_byte_code+3552>, 0x6ad24c <exec_byte_code+3780>, 0x6ad24e <exec_byte_code+3782>, 0x6ad250 <exec_byte_code+3784>, 0x6ad252 <exec_byte_code+3786>, 0x6ad254 <exec_byte_code+3788>, 0x6ad254 <exec_byte_code+3788>, 0x6ad206 <exec_byte_code+3710>, 0x6ad21d <exec_byte_code+3733>, 0x6adac7 <exec_byte_code+5951>, 0x6ad9ad <exec_byte_code+5669>, 0x6ad9a4 <exec_byte_code+5660>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6adcf8 <exec_byte_code+6512>, 0x6ade17 <exec_byte_code+6799>, 0x6ade81 <exec_byte_code+6905>, 0x6adeec <exec_byte_code+7012>, 0x6adf58 <exec_byte_code+7120>, 0x6acc22 <exec_byte_code+2202>, 0x6accaa <exec_byte_code+2338>, 0x6adfdc <exec_byte_code+7252>, 0x6acb60 <exec_byte_code+2008>, 0x6acd12 <exec_byte_code+2442>, 0x6ae051 <exec_byte_code+7369>, 0x6ae0b9 <exec_byte_code+7473>, 0x6ae101 <exec_byte_code+7545>, 0x6ae169 <exec_byte_code+7649>, 0x6ae1bb <exec_byte_code+7731>, 0x6ae28c <exec_byte_code+7940>, 0x6ae2d4 <exec_byte_code+8012>, 0x6ae33c <exec_byte_code+8116>, 0x6ae3c1 <exec_byte_code+8249>, 0x6ae409 <exec_byte_code+8321>, 0x6ae451 <exec_byte_code+8393>, 0x6ae4b9 <exec_byte_code+8497>, 0x6ae521 <exec_byte_code+8601>, 0x6ae589 <exec_byte_code+8705>, 0x6ae60e <exec_byte_code+8838>, 0x6ae660 <exec_byte_code+8920>, 0x6ae6b2 <exec_byte_code+9002>, 0x6ae783 <exec_byte_code+9211>, 0x6ae7fd <exec_byte_code+9333>, 0x6ae877 <exec_byte_code+9455>, 0x6aea46 <exec_byte_code+9918>, 0x6aeab3 <exec_byte_code+10027>, 0x6aeb20 <exec_byte_code+10136>, 0x6aeb8d <exec_byte_code+10245>, 0x6aebfa <exec_byte_code+10354>, 0x6aec4c <exec_byte_code+10436>, 0x6aecca <exec_byte_code+10562>, 0x6aed1c <exec_byte_code+10644>, 0x6aed6e <exec_byte_code+10726>, 0x6aedc0 <exec_byte_code+10808>, 0x6aeecc <exec_byte_code+11076>, 0x6ad827 <exec_byte_code+5279>, 0x6aef2a <exec_byte_code+11170>, 0x6aef72 <exec_byte_code+11242>, 0x6af03e <exec_byte_code+11446>, 0x6af0a7 <exec_byte_code+11551>, 0x6af105 <exec_byte_code+11645>, 0x6af14d <exec_byte_code+11717>, 0x6af193 <exec_byte_code+11787>, 0x6af1d9 <exec_byte_code+11857>, 0x6af227 <exec_byte_code+11935>, 0x6b0161 <exec_byte_code+15833>, 0x6af27f <exec_byte_code+12023>, 0x6af2c5 <exec_byte_code+12093>, 0x6af30b <exec_byte_code+12163>, 0x6af351 <exec_byte_code+12233>, 0x6af397 <exec_byte_code+12303>, 0x6af3dd <exec_byte_code+12373>, 0x6ad827 <exec_byte_code+5279>, 0x6b0161 <exec_byte_code+15833>, 0x6af425 <exec_byte_code+12445>, 0x6af47a <exec_byte_code+12530>, 0x6af4c2 <exec_byte_code+12602>, 0x6af50a <exec_byte_code+12674>, 0x6af572 <exec_byte_code+12778>, 0x6af5da <exec_byte_code+12882>, 0x6af622 <exec_byte_code+12954>, 0x6af73f <exec_byte_code+13239>, 0x6af7a7 <exec_byte_code+13343>, 0x6af80f <exec_byte_code+13447>, 0x6af877 <exec_byte_code+13551>, 0x6af8bd <exec_byte_code+13621>, 0x6b0161 <exec_byte_code+15833>, 0x6ad75b <exec_byte_code+5075>, 0x6ad303 <exec_byte_code+3963>, 0x6acace <exec_byte_code+1862>, 0x6ad3b6 <exec_byte_code+4142>, 0x6ad43a <exec_byte_code+4274>, 0x6ad4bb <exec_byte_code+4403>, 0x6ad70f <exec_byte_code+4999>, 0x6ad724 <exec_byte_code+5020>, 0x6ad04e <exec_byte_code+3270>, 0x6ad7de <exec_byte_code+5206>, 0x6ad85e <exec_byte_code+5334>, 0x6ad8ec <exec_byte_code+5476>, 0x6ad935 <exec_byte_code+5549>, 0x6adb13 <exec_byte_code+6027>, 0x6adb90 <exec_byte_code+6152>, 0x6adc15 <exec_byte_code+6285>, 0x6adc75 <exec_byte_code+6381>, 0x6ad2b5 <exec_byte_code+3885>, 0x6af905 <exec_byte_code+13693>, 0x6af98a <exec_byte_code+13826>, 0x6af9d2 <exec_byte_code+13898>, 0x6afa1a <exec_byte_code+13970>, 0x6afa62 <exec_byte_code+14042>, 0x6afaaa <exec_byte_code+14114>, 0x6afb12 <exec_byte_code+14218>, 0x6afb7a <exec_byte_code+14322>, 0x6afbe2 <exec_byte_code+14426>, 0x6afc4a <exec_byte_code+14530>, 0x6afdc0 <exec_byte_code+14904>, 0x6afe28 <exec_byte_code+15008>, 0x6afe90 <exec_byte_code+15112>, 0x6afed8 <exec_byte_code+15184>, 0x6aff40 <exec_byte_code+15288>, 0x6affa8 <exec_byte_code+15392>, 0x6afff0 <exec_byte_code+15464>, 0x6b0038 <exec_byte_code+15536>, 0x6aee12 <exec_byte_code+10890>, 0x6aee64 <exec_byte_code+10972>, 0x6b008a <exec_byte_code+15618>, 0x6b00f6 <exec_byte_code+15726>, 0x6b0161 <exec_byte_code+15833>, 0x6ad53c <exec_byte_code+4532>, 0x6ad559 <exec_byte_code+4561>, 0x6ad5c8 <exec_byte_code+4672>, 0x6ad637 <exec_byte_code+4783>, 0x6ad6a3 <exec_byte_code+4891>, 0x6ae20d <exec_byte_code+7813>, 0x6ae704 <exec_byte_code+9084>, 0x6aefbf <exec_byte_code+11319>, 0x6b02fb <exec_byte_code+16243>, 0x6b0370 <exec_byte_code+16360>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0406 <exec_byte_code+16510>, 0x6b048d <exec_byte_code+16645>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b06b7 <exec_byte_code+17199> <repeats 64 times>}
const_length = 209
bytestr_length = 1715
vectorp = 0x7fb64a6e9af8
quitcounter = 10 '\n'
stack_items = 21
sa_avail = 14501
sa_count = 7
sa_must_free = false
alloc = 0x7ffc5dad8670
item_bytes = 168
stack_base = 0x7ffc5dad8670
top = 0x7ffc5dad8678
stack_lim = 0x7ffc5dad8718
bytestr_data = 0x7ffc5dad8718 "\306 \020\307\021\n\023\307\024\310\311!\211\307=\204\060"
pc = 0x7ffc5dad8d1a "\210\307\016@\211\203L\006\211@\002\204E\006\211;\203E\006\201", <incomplete sequence \310>
count = 7
result = XIL(0x7ffc5dad9030)
#26 0x0000000000657c82 in funcall_lambda (fun=XIL(0x7fb64a6e9ac5), nargs=0, arg_vector=0x7ffc5dad9268) at eval.c:3023
size = 5
val = XIL(0x1200e043c0)
syms_left = make_number(0)
next = XIL(0xd89480)
lexenv = XIL(0x4)
count = 7
i = 0
optional = false
rest = false
previous_optional_or_rest = 127
#27 0x0000000000657086 in Ffuncall (nargs=1, args=0x7ffc5dad9260) at eval.c:2824
fun = XIL(0x7fb64a6e9ac5)
original_fun = XIL(0x7fb6498e56d0)
funcar = make_number(1661816)
numargs = 0
val = XIL(0x7ffc5dad9240)
count = 6
#28 0x00000000006ad1ca in exec_byte_code (bytestr=XIL(0x7fb64a6edc9c), vector=XIL(0x7fb64a6ed21d), maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7ffc5dad98a0) at bytecode.c:632
op = 0
type = (unknown: 1571657732)
targets = {0x6b0161 <exec_byte_code+15833>, 0x6b0189 <exec_byte_code+15873>, 0x6b018b <exec_byte_code+15875>, 0x6b018d <exec_byte_code+15877>, 0x6b018f <exec_byte_code+15879>, 0x6b018f <exec_byte_code+15879>, 0x6b01f4 <exec_byte_code+15980>, 0x6b0268 <exec_byte_code+16096>, 0x6ac8f2 <exec_byte_code+1386>, 0x6ac8f4 <exec_byte_code+1388>, 0x6ac8f6 <exec_byte_code+1390>, 0x6ac8f8 <exec_byte_code+1392>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac900 <exec_byte_code+1400>, 0x6ac8c1 <exec_byte_code+1337>, 0x6acdd5 <exec_byte_code+2637>, 0x6acdd7 <exec_byte_code+2639>, 0x6acdd9 <exec_byte_code+2641>, 0x6acddb <exec_byte_code+2643>, 0x6acddd <exec_byte_code+2645>, 0x6acddd <exec_byte_code+2645>, 0x6ace12 <exec_byte_code+2698>, 0x6acde3 <exec_byte_code+2651>, 0x6ad0e7 <exec_byte_code+3423>, 0x6ad0e9 <exec_byte_code+3425>, 0x6ad0eb <exec_byte_code+3427>, 0x6ad0ed <exec_byte_code+3429>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0a1 <exec_byte_code+3353>, 0x6ad0b8 <exec_byte_code+3376>, 0x6ad197 <exec_byte_code+3599>, 0x6ad199 <exec_byte_code+3601>, 0x6ad19b <exec_byte_code+3603>, 0x6ad19d <exec_byte_code+3605>, 0x6ad19f <exec_byte_code+3607>, 0x6ad19f <exec_byte_code+3607>, 0x6ad151 <exec_byte_code+3529>, 0x6ad168 <exec_byte_code+3552>, 0x6ad24c <exec_byte_code+3780>, 0x6ad24e <exec_byte_code+3782>, 0x6ad250 <exec_byte_code+3784>, 0x6ad252 <exec_byte_code+3786>, 0x6ad254 <exec_byte_code+3788>, 0x6ad254 <exec_byte_code+3788>, 0x6ad206 <exec_byte_code+3710>, 0x6ad21d <exec_byte_code+3733>, 0x6adac7 <exec_byte_code+5951>, 0x6ad9ad <exec_byte_code+5669>, 0x6ad9a4 <exec_byte_code+5660>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6adcf8 <exec_byte_code+6512>, 0x6ade17 <exec_byte_code+6799>, 0x6ade81 <exec_byte_code+6905>, 0x6adeec <exec_byte_code+7012>, 0x6adf58 <exec_byte_code+7120>, 0x6acc22 <exec_byte_code+2202>, 0x6accaa <exec_byte_code+2338>, 0x6adfdc <exec_byte_code+7252>, 0x6acb60 <exec_byte_code+2008>, 0x6acd12 <exec_byte_code+2442>, 0x6ae051 <exec_byte_code+7369>, 0x6ae0b9 <exec_byte_code+7473>, 0x6ae101 <exec_byte_code+7545>, 0x6ae169 <exec_byte_code+7649>, 0x6ae1bb <exec_byte_code+7731>, 0x6ae28c <exec_byte_code+7940>, 0x6ae2d4 <exec_byte_code+8012>, 0x6ae33c <exec_byte_code+8116>, 0x6ae3c1 <exec_byte_code+8249>, 0x6ae409 <exec_byte_code+8321>, 0x6ae451 <exec_byte_code+8393>, 0x6ae4b9 <exec_byte_code+8497>, 0x6ae521 <exec_byte_code+8601>, 0x6ae589 <exec_byte_code+8705>, 0x6ae60e <exec_byte_code+8838>, 0x6ae660 <exec_byte_code+8920>, 0x6ae6b2 <exec_byte_code+9002>, 0x6ae783 <exec_byte_code+9211>, 0x6ae7fd <exec_byte_code+9333>, 0x6ae877 <exec_byte_code+9455>, 0x6aea46 <exec_byte_code+9918>, 0x6aeab3 <exec_byte_code+10027>, 0x6aeb20 <exec_byte_code+10136>, 0x6aeb8d <exec_byte_code+10245>, 0x6aebfa <exec_byte_code+10354>, 0x6aec4c <exec_byte_code+10436>, 0x6aecca <exec_byte_code+10562>, 0x6aed1c <exec_byte_code+10644>, 0x6aed6e <exec_byte_code+10726>, 0x6aedc0 <exec_byte_code+10808>, 0x6aeecc <exec_byte_code+11076>, 0x6ad827 <exec_byte_code+5279>, 0x6aef2a <exec_byte_code+11170>, 0x6aef72 <exec_byte_code+11242>, 0x6af03e <exec_byte_code+11446>, 0x6af0a7 <exec_byte_code+11551>, 0x6af105 <exec_byte_code+11645>, 0x6af14d <exec_byte_code+11717>, 0x6af193 <exec_byte_code+11787>, 0x6af1d9 <exec_byte_code+11857>, 0x6af227 <exec_byte_code+11935>, 0x6b0161 <exec_byte_code+15833>, 0x6af27f <exec_byte_code+12023>, 0x6af2c5 <exec_byte_code+12093>, 0x6af30b <exec_byte_code+12163>, 0x6af351 <exec_byte_code+12233>, 0x6af397 <exec_byte_code+12303>, 0x6af3dd <exec_byte_code+12373>, 0x6ad827 <exec_byte_code+5279>, 0x6b0161 <exec_byte_code+15833>, 0x6af425 <exec_byte_code+12445>, 0x6af47a <exec_byte_code+12530>, 0x6af4c2 <exec_byte_code+12602>, 0x6af50a <exec_byte_code+12674>, 0x6af572 <exec_byte_code+12778>, 0x6af5da <exec_byte_code+12882>, 0x6af622 <exec_byte_code+12954>, 0x6af73f <exec_byte_code+13239>, 0x6af7a7 <exec_byte_code+13343>, 0x6af80f <exec_byte_code+13447>, 0x6af877 <exec_byte_code+13551>, 0x6af8bd <exec_byte_code+13621>, 0x6b0161 <exec_byte_code+15833>, 0x6ad75b <exec_byte_code+5075>, 0x6ad303 <exec_byte_code+3963>, 0x6acace <exec_byte_code+1862>, 0x6ad3b6 <exec_byte_code+4142>, 0x6ad43a <exec_byte_code+4274>, 0x6ad4bb <exec_byte_code+4403>, 0x6ad70f <exec_byte_code+4999>, 0x6ad724 <exec_byte_code+5020>, 0x6ad04e <exec_byte_code+3270>, 0x6ad7de <exec_byte_code+5206>, 0x6ad85e <exec_byte_code+5334>, 0x6ad8ec <exec_byte_code+5476>, 0x6ad935 <exec_byte_code+5549>, 0x6adb13 <exec_byte_code+6027>, 0x6adb90 <exec_byte_code+6152>, 0x6adc15 <exec_byte_code+6285>, 0x6adc75 <exec_byte_code+6381>, 0x6ad2b5 <exec_byte_code+3885>, 0x6af905 <exec_byte_code+13693>, 0x6af98a <exec_byte_code+13826>, 0x6af9d2 <exec_byte_code+13898>, 0x6afa1a <exec_byte_code+13970>, 0x6afa62 <exec_byte_code+14042>, 0x6afaaa <exec_byte_code+14114>, 0x6afb12 <exec_byte_code+14218>, 0x6afb7a <exec_byte_code+14322>, 0x6afbe2 <exec_byte_code+14426>, 0x6afc4a <exec_byte_code+14530>, 0x6afdc0 <exec_byte_code+14904>, 0x6afe28 <exec_byte_code+15008>, 0x6afe90 <exec_byte_code+15112>, 0x6afed8 <exec_byte_code+15184>, 0x6aff40 <exec_byte_code+15288>, 0x6affa8 <exec_byte_code+15392>, 0x6afff0 <exec_byte_code+15464>, 0x6b0038 <exec_byte_code+15536>, 0x6aee12 <exec_byte_code+10890>, 0x6aee64 <exec_byte_code+10972>, 0x6b008a <exec_byte_code+15618>, 0x6b00f6 <exec_byte_code+15726>, 0x6b0161 <exec_byte_code+15833>, 0x6ad53c <exec_byte_code+4532>, 0x6ad559 <exec_byte_code+4561>, 0x6ad5c8 <exec_byte_code+4672>, 0x6ad637 <exec_byte_code+4783>, 0x6ad6a3 <exec_byte_code+4891>, 0x6ae20d <exec_byte_code+7813>, 0x6ae704 <exec_byte_code+9084>, 0x6aefbf <exec_byte_code+11319>, 0x6b02fb <exec_byte_code+16243>, 0x6b0370 <exec_byte_code+16360>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0406 <exec_byte_code+16510>, 0x6b048d <exec_byte_code+16645>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b06b7 <exec_byte_code+17199> <repeats 64 times>}
const_length = 77
bytestr_length = 443
vectorp = 0x7fb64a6ed220
quitcounter = 39 '\''
stack_items = 13
sa_avail = 15837
sa_count = 5
sa_must_free = false
alloc = 0x7ffc5dad9250
item_bytes = 104
stack_base = 0x7ffc5dad9250
top = 0x7ffc5dad9260
stack_lim = 0x7ffc5dad92b8
bytestr_data = 0x7ffc5dad92b8 "\b\203\b"
pc = 0x7ffc5dad942c "\210)\210\375\376\377\"\210\201H"
count = 5
result = XIL(0x5)
#29 0x0000000000657c82 in funcall_lambda (fun=XIL(0x7fb64a6ed1ed), nargs=0, arg_vector=0x7ffc5dad98a0) at eval.c:3023
size = 5
val = XIL(0)
syms_left = make_number(0)
next = XIL(0)
lexenv = XIL(0)
count = 5
i = 6483568
optional = false
rest = false
previous_optional_or_rest = 127
#30 0x00000000006578b9 in apply_lambda (fun=XIL(0x7fb64a6ed1ed), args=XIL(0), count=4) at eval.c:2959
args_left = XIL(0)
i = 0
numargs = 0
arg_vector = 0x7ffc5dad98a0
tem = make_number(1444944)
sa_avail = 16384
sa_count = 5
sa_must_free = false
#31 0x0000000000655a66 in eval_sub (form=XIL(0x7fb64a83c86b)) at eval.c:2332
fun = XIL(0x7fb64a6ed1ed)
val = XIL(0x28ec250)
original_fun = XIL(0x7fb6498e8df8)
original_args = XIL(0)
funcar = make_number(1663778)
count = 4
argvals = {XIL(0x118), XIL(0x7fb64a85ea50), XIL(0x7fb64a1e3500), XIL(0), XIL(0x7ffc5dad99c0), XIL(0xe0bf50), XIL(0x584ae0), XIL(0)}
#32 0x0000000000654dd5 in Feval (form=XIL(0x7fb64a83c86b), lexical=XIL(0)) at eval.c:2107
count = 3
#33 0x000000000058df4e in top_level_2 () at keyboard.c:1122
No locals.
#34 0x0000000000652e25 in internal_condition_case (bfun=0x58df2b <top_level_2>, handlers=XIL(0x53d0), hfun=0x58d921 <cmd_error>) at eval.c:1336
val = XIL(0x5825cb)
c = 0x291b590
#35 0x000000000058df96 in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
No locals.
#36 0x000000000065231b in internal_catch (tag=XIL(0xccc0), func=0x58df50 <top_level_1>, arg=XIL(0)) at eval.c:1101
val = XIL(0)
c = 0x291b470
#37 0x000000000058de77 in command_loop () at keyboard.c:1091
No locals.
#38 0x000000000058d408 in recursive_edit_1 () at keyboard.c:698
count = 1
val = XIL(0x7ffc5dad9bc0)
#39 0x000000000058d600 in Frecursive_edit () at keyboard.c:769
count = 0
buffer = XIL(0)
#40 0x000000000058b446 in main (argc=9, argv=0x7ffc5dad9e68) at emacs.c:1915
stack_bottom_variable = 0x28d1f20
do_initial_setlocale = true
skip_args = 2
no_loadup = false
junk = 0x0
dname_arg = 0x0
ch_to_dir = 0x0
original_pwd = 0x0
argv0_base = 0x7ffc5dadc248 "emacs"
is_temacs = false
loaded_dump = 0x28b54d0 "../src/emacs.pdmp"
dump_mode = 0x0
disable_aslr = false
rlim = {
rlim_cur = 10022912,
rlim_max = 18446744073709551615
}
sockfd = -1
module_assertions = false
You can't do that without a process to debug.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-14 21:03 ` Philipp Stephani
@ 2018-02-15 0:42 ` Daniel Colascione
0 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2018-02-15 0:42 UTC (permalink / raw)
To: Philipp Stephani, Eli Zaretskii; +Cc: emacs-devel
On 02/14/2018 01:03 PM, Philipp Stephani wrote:
>
>
> Eli Zaretskii <eliz@gnu.org <mailto:eliz@gnu.org>> schrieb am Di., 13.
> Feb. 2018 um 17:39 Uhr:
>
> > From: Daniel Colascione <dancol@dancol.org
> <mailto:dancol@dancol.org>>
> > Date: Mon, 12 Feb 2018 12:18:36 -0800
> >
> > I've pushed the portable dumper to the pdumper Savannah branch. It
> > should support a fully PIC Emacs.
>
> Thanks. I'd urge people to try this branch and report any issues they
> see.
>
>
> Got a crash when building on Debian testing. I've attached the output of
> 'bt full'.
Thanks. There were two bugs, both relating to conservative GC:
1) To make conservative GC work, we record in the dump the start
position and Lisp type of each object in the dump. The problem was that
we were recording object-start records for objects in the discardable
region. With the right stack garbage as input, conservative GC would
find one of these object-start records, use it as evidence that it was
looking at a real object, then call mark_object, which died because it
knows that objects in the discardable region shouldn't be live. Fixed by
just not emitting these discardable-region object-start records anymore.
2) When evaluating a Lisp_Object on the stack in mark_maybe_object,
occasionally, we can find a word with a valid pointer component but the
wrong type tags. Now we check that the Lisp_Object's tagged type matches
the actual type of the object in the dump.
I don't know why it was so hard to repro these on my machine. I had to
bootstrap a few times before Emacs began crashing.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-12 20:18 ` Daniel Colascione
2018-02-13 16:37 ` Eli Zaretskii
@ 2018-02-15 4:28 ` Stefan Monnier
2018-02-15 22:13 ` Daniel Colascione
2018-02-15 18:34 ` andres.ramirez
2018-02-19 22:01 ` Daniele Nicolodi
3 siblings, 1 reply; 299+ messages in thread
From: Stefan Monnier @ 2018-02-15 4:28 UTC (permalink / raw)
To: emacs-devel
> I've pushed the portable dumper to the pdumper Savannah branch. It should
> support a fully PIC Emacs.
Thank you very much Daniel. I hope this can move to master soon.
I just tried it (on Debian testing, with the latest commit at the time,
i.e. 5232a17c365f871a84603caba81712415a260042) and here's my experience:
- compilation fails with
In file included from character.h:27:0,
from buffer.h:27,
from pdumper.c:18:
pdumper.c: In function ‘intmax_t_to_lisp’:
pdumper.c:699:29: error: conversion to ‘EMACS_INT {aka int}’ from ‘intmax_t {aka long long int}’ may alter its value [-Werror=conversion]
return INTEGER_TO_CONS (value); \
^
lisp.h:3572:19: note: in definition of macro ‘INTEGER_TO_CONS’
? make_number (i) \
^
pdumper.c:703:1: note: in expansion of macro ‘DEFINE_TOLISP_FUNC’
DEFINE_TOLISP_FUNC (intmax_t_to_lisp, intmax_t);
^~~~~~~~~~~~~~~~~~
pdumper.c: In function ‘pdumper_load’:
pdumper.c:5049:14: error: conversion to ‘unsigned int’ from ‘off_t {aka long long int}’ may alter its value [-Werror=conversion]
.size = dump_size - header->cold_start,
^~~~~~~~~
pdumper.c:5072:21: error: conversion to ‘intptr_t {aka int}’ from ‘off_t {aka long long int}’ may alter its value [-Werror=conversion]
dump_public.end = dump_public.start + dump_size;
^~~~~~~~~~~
In file included from character.h:27:0,
from buffer.h:27,
from pdumper.c:18:
I fixed those with the patch below, but it seems there's some issues
with off_t vs size_t (I guess a dump file shouldn't be bigger than the
available address space, so even though the max size of files can be
much bigger, we can use smaller integers to refer to size&position
within dump files).
- after that the dump itself crashed as follows:
pdumper.c:2468: Emacs fatal error: assertion failed: vector_nbytes ((struct Lisp_Vector *) in) == out_size
Backtrace below.
Stefan
Finding pointers to doc strings...
Finding pointers to doc strings...done
Dumping under the name bootstrap-emacs.pdmp
dumping fingerprint: d003f05f220e1040ec992fccec03d65c4934d9181cbde233f17892969593f435
pdumper.c:2468: Emacs fatal error: assertion failed: vector_nbytes ((struct Lisp_Vector *) in) == out_size
Thread 1 "temacs" hit Breakpoint 1, terminate_due_to_signal (sig=6,
backtrace_limit=2147483647) at emacs.c:357
(gdb) bt
#0 0x08194a5a in terminate_due_to_signal (sig=6, backtrace_limit=2147483647)
at emacs.c:357
#1 0x08216a9e in die (msg=0x833c654 "vector_nbytes ((struct Lisp_Vector *) in) == out_size", file=0x833b6f8 "pdumper.c", line=2468) at alloc.c:7796
#2 0x08227321 in dump_object_start_pseudovector (ctx=ctx@entry=0xbfffec08, out_hdr=out_hdr@entry=0xbfffe730, out_size=out_size@entry=536, in_hdr=0xbfffe948)
at pdumper.c:2468
#3 0x082284d0 in dump_buffer (ctx=0xbfffec08, in_buffer=0x8718d20)
at pdumper.c:2632
#4 0x08228be1 in dump_vectorlike (ctx=ctx@entry=0xbfffec08, v=0x8718d20)
at pdumper.c:2825
#5 0x08228dc1 in dump_object_1 (ctx=ctx@entry=0xbfffec08, object=object@entry=XIL(0x8718d25)) at pdumper.c:2889
#6 0x08228f2e in dump_object (ctx=ctx@entry=0xbfffec08, object=XIL(0x8718d25)) at pdumper.c:2954
#7 0x0822981f in Fdump_emacs_portable (filename=XIL(0x922ce54), track_referrers=XIL(0)) at pdumper.c:3780
#8 0x08251349 in eval_sub (form=...) at eval.c:2290
#9 0x08252337 in Fif (args=XIL(0x88f13db)) at eval.c:414
#10 0x08250ff4 in eval_sub (form=...) at eval.c:2239
#11 0x082518e9 in Fprogn (body=...) at eval.c:459
#12 0x08253ef5 in Flet (args=XIL(0x88b029b)) at eval.c:973
#13 0x08250ff4 in eval_sub (form=...) at eval.c:2239
#14 0x08252337 in Fif (args=XIL(0x87f975b)) at eval.c:414
#15 0x08250ff4 in eval_sub (form=...) at eval.c:2239
#16 0x08286d3c in readevalloop (readcharfun=XIL(0x34c8), infile0=infile0@entry=0xbffff1a4, sourcename=...,
sourcename@entry=XIL(0x873c434), printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
#17 0x08287cbc in Fload (file=XIL(0x8723644), noerror=XIL(0), nomessage=XIL(0), nosuffix=XIL(0), must_suffix=XIL(0)) at lread.c:1457
#18 0x082513c7 in eval_sub (form=...) at eval.c:2301
#19 0x0825450a in Feval (form=XIL(0x87128bb), lexical=XIL(0)) at eval.c:2107
#20 0x081992fe in top_level_2 () at keyboard.c:1122
#21 0x0824c263 in internal_condition_case (bfun=0x81992c8 <top_level_2>, handlers=XIL(0x2988), hfun=0x819eb61 <cmd_error>) at eval.c:1336
#22 0x0819cbaf in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
#23 0x0824c160 in internal_catch (tag=XIL(0x64e0), func=0x819caf0 <top_level_1>, arg=XIL(0)) at eval.c:1101
#24 0x0819b8ef in command_loop () at keyboard.c:1091
#25 0x0819e233 in recursive_edit_1 () at keyboard.c:698
#26 0x0819e896 in Frecursive_edit () at keyboard.c:769
#27 0x08195aaa in main (argc=<optimized out>, argv=<optimized out>)
at emacs.c:1915
Lisp Backtrace:
"dump-emacs-portable" (0xbfffed5c)
"if" (0xbfffee4c)
"let" (0xbfffef8c)
"if" (0xbffff05c)
"load" (0xbffff28c)
(gdb)
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-12 20:18 ` Daniel Colascione
2018-02-13 16:37 ` Eli Zaretskii
2018-02-15 4:28 ` Stefan Monnier
@ 2018-02-15 18:34 ` andres.ramirez
2018-02-19 22:01 ` Daniele Nicolodi
3 siblings, 0 replies; 299+ messages in thread
From: andres.ramirez @ 2018-02-15 18:34 UTC (permalink / raw)
To: Daniel Colascione
Cc: Karl Fogel, Philippe Vaucher, Emacs developers, Kaushal Modi
Hello.
Doing make bootstrap on archlinux with cc (GCC) 7.2.1 20180116
I am getting this:
--8<---------------cut here---------------start------------->8---
CC pdumper.o
CC data.o
In file included from character.h:27:0,
from buffer.h:27,
from pdumper.c:18:
pdumper.c: In function ‘intmax_t_to_lisp’:
pdumper.c:699:29: error: conversion to ‘EMACS_INT {aka int}’ from ‘intmax_t {aka long long int}’ may alter its value [-Werror=conversion]
return INTEGER_TO_CONS (value); \
^
lisp.h:3572:19: note: in definition of macro ‘INTEGER_TO_CONS’
? make_number (i) \
^
pdumper.c:703:1: note: in expansion of macro ‘DEFINE_TOLISP_FUNC’
DEFINE_TOLISP_FUNC (intmax_t_to_lisp, intmax_t);
^~~~~~~~~~~~~~~~~~
pdumper.c: In function ‘pdumper_load’:
pdumper.c:5049:14: error: conversion to ‘unsigned int’ from ‘off_t {aka long long int}’ may alter its value [-Werror=conversion]
.size = dump_size - header->cold_start,
^~~~~~~~~
pdumper.c:5072:21: error: conversion to ‘intptr_t {aka int}’ from ‘off_t {aka long long int}’ may alter its value [-Werror=conversion]
dump_public.end = dump_public.start + dump_size;
^~~~~~~~~~~
cc1: some warnings being treated as errors
make[2]: *** [Makefile:386: pdumper.o] Error 1
make[2]: *** Waiting for unfinished jobs....
--8<---------------cut here---------------end--------------->8---
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-15 4:28 ` Stefan Monnier
@ 2018-02-15 22:13 ` Daniel Colascione
2018-02-15 22:30 ` Paul Eggert
2018-02-15 22:35 ` Paul Eggert
0 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2018-02-15 22:13 UTC (permalink / raw)
To: Stefan Monnier, emacs-devel
On 02/14/2018 08:28 PM, Stefan Monnier wrote:
>> I've pushed the portable dumper to the pdumper Savannah branch. It should
>> support a fully PIC Emacs.
>
> Thank you very much Daniel. I hope this can move to master soon.
> I just tried it (on Debian testing, with the latest commit at the time,
> i.e. 5232a17c365f871a84603caba81712415a260042) and here's my experience:
>
> - compilation fails with
>
> In file included from character.h:27:0,
> from buffer.h:27,
> from pdumper.c:18:
> pdumper.c: In function ‘intmax_t_to_lisp’:
> pdumper.c:699:29: error: conversion to ‘EMACS_INT {aka int}’ from ‘intmax_t {aka long long int}’ may alter its value [-Werror=conversion]
> return INTEGER_TO_CONS (value); \
Fixed. I'm tempted to just convert INTEGER_TO_CONS to a pair of
functions (one signed, one unsigned) that just take an {,u}intmax_t.
> pdumper.c:2468: Emacs fatal error: assertion failed: vector_nbytes ((struct Lisp_Vector *) in) == out_size
Modified the code to tolerate end-of-pseudovector padding in 32-bit builds.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-15 22:13 ` Daniel Colascione
@ 2018-02-15 22:30 ` Paul Eggert
2018-02-15 22:35 ` Daniel Colascione
2018-02-15 22:35 ` Paul Eggert
1 sibling, 1 reply; 299+ messages in thread
From: Paul Eggert @ 2018-02-15 22:30 UTC (permalink / raw)
To: Daniel Colascione, Stefan Monnier, emacs-devel
On 02/15/2018 02:13 PM, Daniel Colascione wrote:
> I'm tempted to just convert INTEGER_TO_CONS to a pair of functions
> (one signed, one unsigned) that just take an {,u}intmax_t.
Sometimes one does not know whether an integer is signed. For example,
the ino_t type is signed on some platforms, and not on others. This is
why INTEGER_CONS is written the way it is: it is supposed to do the
right thing regardless of the signedness of its argument.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-15 22:30 ` Paul Eggert
@ 2018-02-15 22:35 ` Daniel Colascione
2018-02-15 22:56 ` Paul Eggert
0 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2018-02-15 22:35 UTC (permalink / raw)
To: Paul Eggert; +Cc: Stefan Monnier, emacs-devel
[-- Attachment #1: Type: text/html, Size: 1139 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-15 22:13 ` Daniel Colascione
2018-02-15 22:30 ` Paul Eggert
@ 2018-02-15 22:35 ` Paul Eggert
1 sibling, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2018-02-15 22:35 UTC (permalink / raw)
To: Daniel Colascione, emacs-devel
[-- Attachment #1: Type: text/plain, Size: 893 bytes --]
I had the following problem on Fedora 27 when building the pdumper
branch just now:
CC emacs.o
emacs.c: In function ‘load_dump’:
emacs.c:771:5: error: jump skips variable initialization
[-Werror=jump-misses-init]
goto out;
^~~~
emacs.c:817:2: note: label ‘out’ defined here
out:
^~~
emacs.c:800:15: note: ‘path_exec’ declared here
const char *path_exec = PATH_EXEC;
^~~~~~~~~
emacs.c:788:5: error: jump skips variable initialization
[-Werror=jump-misses-init]
goto out;
^~~~
emacs.c:817:2: note: label ‘out’ defined here
out:
^~~
emacs.c:800:15: note: ‘path_exec’ declared here
const char *path_exec = PATH_EXEC;
^~~~~~~~~
cc1: all warnings being treated as errors
The attached patch fixes it.
[-- Attachment #2: 0001-src-emacs.c-load_dump-Pacify-GCC-Wjump-misses-init.patch --]
[-- Type: text/x-patch, Size: 1079 bytes --]
From 80c596f9a992c9424a5a15716a6639f5efedbed5 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 15 Feb 2018 14:34:49 -0800
Subject: [PATCH] * src/emacs.c (load_dump): Pacify GCC -Wjump-misses-init.
---
src/emacs.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/emacs.c b/src/emacs.c
index 0d7c6ed9b3..7ed4f2758e 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -748,6 +748,7 @@ load_dump (int *inout_argc,
/* Look for an explicitly-specified dump file. */
+ const char *path_exec = PATH_EXEC;
char *dump_file = find_and_remove_dump_file_argument (&argc, &argv);
if (initialized && dump_file)
/* TODO: maybe more thoroughly scrub process environment in order
@@ -797,7 +798,6 @@ load_dump (int *inout_argc,
FIXME: this doesn't work with emacs-XX.YY.ZZ.pdmp versioned files. */
argv0_base = "emacs";
- const char *path_exec = PATH_EXEC;
#ifdef WINDOWSNT
/* On MS-Windows, PATH_EXEC normally starts with a literal
"%emacs_dir%", so it will never work without some tweaking. */
--
2.14.3
^ permalink raw reply related [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-15 22:35 ` Daniel Colascione
@ 2018-02-15 22:56 ` Paul Eggert
0 siblings, 0 replies; 299+ messages in thread
From: Paul Eggert @ 2018-02-15 22:56 UTC (permalink / raw)
To: Daniel Colascione; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]
Here are some problems I ran into on Fedora 27 x86-64. Proposed patch
attached.
CC pdumper.o
In file included from character.h:27:0,
from buffer.h:27,
from pdumper.c:18:
pdumper.c: In function ‘dump_object_1’:
pdumper.c:2920:70: error: left-hand operand of comma expression has no
effect [-Werror=unused-value]
eassert (("should not be dumping int: is self-representing",
0));
^
lisp.h:195:43: note: in definition of macro ‘eassert’
# define eassert(cond) ((void) (false && (cond))) /* Check COND
compiles. */
^~~~
pdumper.c: In function ‘dump_vectorlike’:
pdumper.c:2823:10: error: this statement may fall through
[-Werror=implicit-fallthrough=]
if ((v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_SPEC_MAX &&
^
pdumper.c:2827:5: note: here
case PVEC_NORMAL_VECTOR:
^~~~
pdumper.c: At top level:
pdumper.c:119:0: error: macro "MAX_OFF_T" is not used
[-Werror=unused-macros]
# define MAX_OFF_T ((sizeof (off_t) == sizeof (int32_t)) ? \
cc1: all warnings being treated as errors
make[1]: *** [Makefile:386: pdumper.o] Error 1
make[1]: Leaving directory '/home/eggert/src/gnu/emacs/pdumper/src'
make: *** [Makefile:420: src] Error 2
[-- Attachment #2: 0001-Pacify-pdumper-for-GCC-7.3.1.patch --]
[-- Type: text/x-patch, Size: 1658 bytes --]
From 990d36af757a3d16b08f438f96431e8d07ae371e Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 15 Feb 2018 14:54:50 -0800
Subject: [PATCH] Pacify pdumper for GCC 7.3.1
* src/pdumper.c (MAX_OFF_T): Remove.
(dump_vectorlike): Add FALLTHROUGH.
(dump_object_1): Avoid unportable eassert.
---
src/pdumper.c | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
diff --git a/src/pdumper.c b/src/pdumper.c
index 499c8dc27c..9a286c0247 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -115,11 +115,6 @@ verify (sizeof (off_t) == sizeof (int32_t) ||
sizeof (off_t) == sizeof (int64_t));
verify (CHAR_BIT == 8);
-#ifndef MAX_OFF_T
-# define MAX_OFF_T ((sizeof (off_t) == sizeof (int32_t)) ? \
- INT32_MAX : INT64_MAX)
-#endif
-
#define DIVIDE_ROUND_UP(x, y) (((x) + (y) - 1) / (y))
static const char dump_magic[16] = {
@@ -2823,7 +2818,7 @@ dump_vectorlike (struct dump_context *ctx, const struct Lisp_Vector *v)
if ((v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_SPEC_MAX &&
(v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_ENTITY_MAX)
error_unsupported_dump_object(ctx, lv, "font");
- /* Fall through */
+ FALLTHROUGH;
case PVEC_NORMAL_VECTOR:
case PVEC_COMPILED:
case PVEC_CHAR_TABLE:
@@ -2917,7 +2912,7 @@ dump_object_1 (struct dump_context *ctx, Lisp_Object object)
offset = dump_float (ctx, XFLOAT (object));
break;
case_Lisp_Int:
- eassert (("should not be dumping int: is self-representing", 0));
+ emacs_abort ();
default:
emacs_abort ();
}
--
2.14.3
^ permalink raw reply related [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-13 16:37 ` Eli Zaretskii
2018-02-14 21:03 ` Philipp Stephani
@ 2018-02-15 23:31 ` Ken Brown
2018-02-15 23:36 ` Daniel Colascione
2018-02-17 1:01 ` Clément Pit-Claudel
` (2 subsequent siblings)
4 siblings, 1 reply; 299+ messages in thread
From: Ken Brown @ 2018-02-15 23:31 UTC (permalink / raw)
To: Eli Zaretskii, Daniel Colascione; +Cc: emacs-devel
On 2/13/2018 11:37 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 12 Feb 2018 12:18:36 -0800
>>
>> I've pushed the portable dumper to the pdumper Savannah branch. It
>> should support a fully PIC Emacs.
>
> Thanks. I'd urge people to try this branch and report any issues they
> see.
I just tried to build on 64-bit Cygwin, and the build fails as follows:
[...]
Dumping under the name bootstrap-emacs.pdmp
dumping fingerprint:
923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
Dump complete
Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
Reloc counts: hot=919268 discardable=5790
make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
make[2]: Entering directory '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
ELC ../../pdumper/lisp/emacs-lisp/macroexp.elc
ELC ../../pdumper/lisp/emacs-lisp/cconv.elc
ELC ../../pdumper/lisp/emacs-lisp/byte-opt.elc
ELC ../../pdumper/lisp/emacs-lisp/bytecomp.elc
emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out of memory
There's probably some obvious explanation, but I don't see it at the moment.
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-15 23:31 ` Ken Brown
@ 2018-02-15 23:36 ` Daniel Colascione
2018-02-16 1:56 ` Ken Brown
0 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2018-02-15 23:36 UTC (permalink / raw)
To: Ken Brown; +Cc: Eli Zaretskii, emacs-devel
[-- Attachment #1: Type: text/html, Size: 2308 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-15 23:36 ` Daniel Colascione
@ 2018-02-16 1:56 ` Ken Brown
2018-02-16 2:36 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: Ken Brown @ 2018-02-16 1:56 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel
On 2/15/2018 6:36 PM, Daniel Colascione wrote:
> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>
> On 2/13/2018 11:37 AM, Eli Zaretskii wrote:
> >> From: Daniel Colascione <dancol@dancol.org>
> >> Date: Mon, 12 Feb 2018 12:18:36 -0800
> >>
> >> I've pushed the portable dumper to the pdumper Savannah branch. It
> >> should support a fully PIC Emacs.
> >
> > Thanks. I'd urge people to try this branch and report any issues
> they
> > see.
>
> I just tried to build on 64-bit Cygwin, and the build fails as follows:
>
> [...]
> Dumping under the name bootstrap-emacs.pdmp
> dumping fingerprint:
> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
> Dump complete
> Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
> Reloc counts: hot=919268 discardable=5790
> make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
> make[2]: Entering directory
> '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
> ELC ../../pdumper/lisp/emacs-lisp/macroexp.elc
> ELC ../../pdumper/lisp/emacs-lisp/cconv.elc
> ELC ../../pdumper/lisp/emacs-lisp/byte-opt.elc
> ELC ../../pdumper/lisp/emacs-lisp/bytecomp.elc
> emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
> of memory
>
> There's probably some obvious explanation, but I don't see it at the
> moment.
>
>
> I'm not entirely surprised to see Cygwin fall over here. We could just
> use the Windows memory mapping functions directly, but I'd prefer to
> stick with the POSIX API if we can make it work.
I agree.
> Any idea where in the
> mmap sequence we fail?
I haven't looked at the code yet, so I don't understand the question.
If you give me some guidance, I'll try to investigate.
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-16 1:56 ` Ken Brown
@ 2018-02-16 2:36 ` Daniel Colascione
2018-02-17 23:38 ` Ken Brown
0 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2018-02-16 2:36 UTC (permalink / raw)
To: Ken Brown; +Cc: Eli Zaretskii, emacs-devel
On 02/15/2018 05:56 PM, Ken Brown wrote:
> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>
>> On 2/13/2018 11:37 AM, Eli Zaretskii wrote:
>> >> From: Daniel Colascione <dancol@dancol.org>
>> >> Date: Mon, 12 Feb 2018 12:18:36 -0800
>> >>
>> >> I've pushed the portable dumper to the pdumper Savannah
>> branch. It
>> >> should support a fully PIC Emacs.
>> >
>> > Thanks. I'd urge people to try this branch and report any issues
>> they
>> > see.
>>
>> I just tried to build on 64-bit Cygwin, and the build fails as
>> follows:
>>
>> [...]
>> Dumping under the name bootstrap-emacs.pdmp
>> dumping fingerprint:
>> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>> Dump complete
>> Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
>> Reloc counts: hot=919268 discardable=5790
>> make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>> make[2]: Entering directory
>> '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>> ELC ../../pdumper/lisp/emacs-lisp/macroexp.elc
>> ELC ../../pdumper/lisp/emacs-lisp/cconv.elc
>> ELC ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>> ELC ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>> emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
>> of memory
>>
>> There's probably some obvious explanation, but I don't see it at the
>> moment.
>>
>>
>> I'm not entirely surprised to see Cygwin fall over here. We could just
>> use the Windows memory mapping functions directly, but I'd prefer to
>> stick with the POSIX API if we can make it work.
>
> I agree.
>
>> Any idea where in the mmap sequence we fail?
>
> I haven't looked at the code yet, so I don't understand the question. If
> you give me some guidance, I'll try to investigate.
Thanks. I think the trouble must be in dump_mmap_contiguous. We report
all errors from this function as "out of memory". dump_mmap_contiguous
takes an array of mapping descriptors (think ELF segments or something)
and maps them all into a single contiguous region of virtual memory. On
POSIX systems, we reserve a chunk of address space with a big PROT_NONE
anonymous mapping, then carve it up into the separate mappings that we
really want. Windows doesn't support atomic mmap replacement, so Cygwin
has to emulate it, and I wouldn't be surprised if something's going
wrong in this emulation.
That you're able to compile a few elisp files before bootstrap starts
failing makes me wonder whether we're dealing with Cygwin's perennial
problem with address space collisions and some kind of dynamic-base DLL.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-13 16:37 ` Eli Zaretskii
2018-02-14 21:03 ` Philipp Stephani
2018-02-15 23:31 ` Ken Brown
@ 2018-02-17 1:01 ` Clément Pit-Claudel
2018-02-19 17:06 ` Daniel Colascione
2018-02-17 11:53 ` Charles A. Roelli
2018-02-20 0:54 ` Andy Moreton
4 siblings, 1 reply; 299+ messages in thread
From: Clément Pit-Claudel @ 2018-02-17 1:01 UTC (permalink / raw)
To: emacs-devel
On 2018-02-13 11:37, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 12 Feb 2018 12:18:36 -0800
>>
>> I've pushed the portable dumper to the pdumper Savannah branch. It
>> should support a fully PIC Emacs.
>
> Thanks. I'd urge people to try this branch and report any issues they
> see.
Thanks Daniel. I compiled with the flags suggested in DEBUG and ran for a few minutes. No issues to report on that test run, which includes heavy face customizations, plenty of subprocesses, and about 100 packages :) Wonderful work!
Some notes:
* After bootstrapping (pbootstrap), load_dump took 77.82ms ± 13.63ms
* After the final dump (pdump), load_dump took 33.70ms ± 10.22ms
* I tried to use dump-emacs-portable, but it crashed. I've pasted the backtrace at the bottom of this email.
Thanks again!
Clément.
$ gdb ./emacs
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./emacs...done.
SIGINT is used by the debugger.
Are you sure you want to change it? (y or n) [answered Y; input not from terminal]
DISPLAY = :0
TERM = dumb
Breakpoint 1 at 0x59b9a3: file emacs.c, line 357.
Temporary breakpoint 2 at 0x5c92db: file sysdep.c, line 1075.
(gdb) run --batch -l ~/.emacs.d/init.el --eval '(dump-emacs-portable "test.pdump")'
Starting program: /build/emacs/pdumper/src/emacs --batch -l ~/.emacs.d/init.el --eval '(dump-emacs-portable "test.pdump")'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
load_dump completed in 16.418 milliseconds: result=0
[New Thread 0x7fffe49d8700 (LWP 31691)]
dumping fingerprint: 0c14ce6dcf659b419ed01ba19f6847c88519a8d7f04f432885fd65dee2d040de
Thread 1 "emacs" hit Breakpoint 1, terminate_due_to_signal (sig=6, backtrace_limit=40) at emacs.c:357
357 {
(gdb) bt full
#0 terminate_due_to_signal (sig=6, backtrace_limit=40) at emacs.c:357
No locals.
#1 0x00000000005caeed in emacs_abort () at sysdep.c:2437
No locals.
#2 0x00000000005afb2c in unblock_input_to (level=-1) at keyboard.c:7161
No locals.
#3 0x00000000005afb43 in unblock_input () at keyboard.c:7177
No locals.
#4 0x0000000000642c24 in dump_unwind_cleanup (data=0x7fffffffb470) at pdumper.c:3395
ctx = 0x7fffffffb470
#5 0x00000000006722fb in do_one_unbind (this_binding=0x7fffffffb050, unwinding=true, bindflag=SET_INTERNAL_UNBIND)
at eval.c:3457
No locals.
#6 0x000000000067272f in unbind_to (count=3, value=XIL(0)) at eval.c:3573
this_binding = {
kind = SPECPDL_UNWIND_PTR,
unwind = {
kind = SPECPDL_UNWIND_PTR,
func = 0x642be3 <dump_unwind_cleanup>,
arg = XIL(0x7fffffffb470)
},
unwind_ptr = {
kind = SPECPDL_UNWIND_PTR,
---Type <return> to continue, or q <return> to quit---
func = 0x642be3 <dump_unwind_cleanup>,
arg = 0x7fffffffb470
},
unwind_int = {
kind = SPECPDL_UNWIND_PTR,
func = 0x642be3 <dump_unwind_cleanup>,
arg = -19344
},
unwind_void = {
kind = SPECPDL_UNWIND_PTR,
func = 0x642be3 <dump_unwind_cleanup>
},
let = {
kind = SPECPDL_UNWIND_PTR,
symbol = XIL(0x642be3),
old_value = XIL(0x7fffffffb470),
where = XIL(0x7fffe53c1505),
saved_value = XIL(0)
},
bt = {
kind = SPECPDL_UNWIND_PTR,
debug_on_exit = false,
function = XIL(0x642be3),
---Type <return> to continue, or q <return> to quit---
args = 0x7fffffffb470,
nargs = 140737039308037
}
}
quitf = XIL(0)
#7 0x000000000066b247 in unwind_to_catch (catch=0xed1690, value=XIL(0x3ac8d23)) at eval.c:1149
last_time = false
#8 0x000000000066c81b in signal_or_quit (error_symbol=XIL(0x5310), data=XIL(0x3ac8d03), keyboard_quit=false)
at eval.c:1674
unwind_data = XIL(0x3ac8d23)
conditions = XIL(0x7fffe53e3a1b)
string = XIL(0x3ac8d23)
real_error_symbol = XIL(0x5310)
clause = XIL(0xc180)
h = 0xed1690
#9 0x000000000066c429 in Fsignal (error_symbol=XIL(0x5310), data=XIL(0x3ac8d03)) at eval.c:1569
No locals.
#10 0x000000000059862e in xsignal (error_symbol=XIL(0x5310), data=XIL(0x3ac8d03)) at lisp.h:4109
No locals.
#11 0x000000000066c91c in xsignal1 (error_symbol=XIL(0x5310), arg=XIL(0x25ff544)) at eval.c:1703
No locals.
#12 0x000000000066d1f6 in verror (m=0x79ee58 "unsupported object type in dump: %s", ap=0x7fffffffb240) at eval.c:1888
No locals.
---Type <return> to continue, or q <return> to quit---
#13 0x000000000066d2a7 in error (m=0x79ee58 "unsupported object type in dump: %s") at eval.c:1900
ap = <error reading variable ap (Attempt to dereference a generic pointer.)>
#14 0x000000000063b107 in error_unsupported_dump_object (ctx=0x7fffffffb470, object=XIL(0x11082b5),
msg=0x79fad7 "window configuration") at pdumper.c:635
No locals.
#15 0x0000000000641519 in dump_vectorlike (ctx=0x7fffffffb470, v=0x11082b0) at pdumper.c:2848
offset = 32767
lv = XIL(0x11082b5)
#16 0x00000000006416dd in dump_object_1 (ctx=0x7fffffffb470, object=XIL(0x11082b5)) at pdumper.c:2900
offset = -1
#17 0x0000000000641918 in dump_object (ctx=0x7fffffffb470, object=XIL(0x11082b5)) at pdumper.c:2965
result = 0
#18 0x000000000064414a in Fdump_emacs_portable (filename=XIL(0x25ff564), track_referrers=XIL(0)) at pdumper.c:3794
count = 13
symbol = XIL(0x7fffe4a96068)
ctx_buf = {
header = {
magic = "!UMPEDGNUEMACS\000",
fingerprint = "\f\024\316m\317e\233A\236\320\033\241\237hGȅ\031\250\327\360OC(\205\375e\336\342\320", <incomplete sequence \336>,
dump_relocs = {
offset = 0,
nr_entries = 0
---Type <return> to continue, or q <return> to quit---
},
object_starts = {
offset = 0,
nr_entries = 0
},
emacs_relocs = {
offset = 0,
nr_entries = 0
},
discardable_start = 0,
cold_start = 0
},
old_purify_flag = XIL(0),
old_post_gc_hook = XIL(0),
fd = 6,
dump_filename = XIL(0x25ff564),
offset = 2411504,
obj_offset = 0,
flags = {
dump_object_contents = true,
dump_object_starts = true,
pack_objects = false,
assert_already_seen = false,
---Type <return> to continue, or q <return> to quit---
defer_hash_tables = true,
defer_symbols = false
},
end_heap = 0,
objects_dumped = XIL(0x11141c5),
referrers = XIL(0),
current_referrer = XIL(0),
have_current_referrer = true,
dump_queue = {
zero_weight_objects = {
head = XIL(0x3a792f3),
tail = XIL(0xeb5b83),
length = 1565
},
one_weight_normal_objects = {
head = XIL(0x3977d73),
tail = XIL(0x1dd1a33),
length = 388
},
one_weight_strong_objects = {
head = XIL(0x3ac9753),
tail = XIL(0x17a46c3),
length = 42646
---Type <return> to continue, or q <return> to quit---
},
fancy_weight_objects = {
head = XIL(0x3acb753),
tail = XIL(0x34fa903),
length = 164
},
link_weights = XIL(0x154aee5),
sequence_numbers = XIL(0x157ecc5),
next_sequence_number = 116616
},
deferred_hash_tables = XIL(0x3a55c03),
deferred_symbols = XIL(0),
fixups = XIL(0x3ac8c63),
symbol_aux = XIL(0),
copied_queue = XIL(0x3aab0e3),
cold_queue = XIL(0x3ac9f23),
dump_relocs = XIL(0x3ab1603),
object_starts = XIL(0x3ac8cf3),
emacs_relocs = XIL(0x3aa8fa3),
number_hot_relocations = 0,
number_discardable_relocations = 0
}
ctx = 0x7fffffffb470
---Type <return> to continue, or q <return> to quit---
header_start = 0
header_end = 80
hot_start = 80
hot_end = 0
discardable_end = 52608
number_hot_relocations = 0
number_discardable_relocations = 52608
cold_end = 0
#19 0x000000000066e7c9 in eval_sub (form=XIL(0x2783483)) at eval.c:2290
i = 2
maxargs = 2
args_left = XIL(0)
numargs = make_number(1)
fun = XIL(0xdb5045)
val = XIL(0x7fffe53c1500)
original_fun = XIL(0x7fffe46766b8)
original_args = XIL(0x2783493)
funcar = XIL(0x7fffffffb670)
count = 12
argvals = {XIL(0x2789394), XIL(0), XIL(0x7fffffffb730), XIL(0x672073), XIL(0), XIL(0xe3c470), XIL(0x7ad0),
XIL(0x7ad0)}
#20 0x000000000066dc63 in Feval (form=XIL(0x2783483), lexical=XIL(0)) at eval.c:2107
count = 11
---Type <return> to continue, or q <return> to quit---
#21 0x0000000000670641 in funcall_subr (subr=0xdb8a00 <Seval>, numargs=1, args=0x7fffffffb948) at eval.c:2899
internal_argbuf = {XIL(0x2783483), XIL(0), XIL(0x7fffffffb800), XIL(0x59662f), XIL(0xdb8a05), XIL(0xdb8a00),
XIL(0x7fffffffb820), XIL(0x596657)}
internal_args = 0x7fffffffb7d0
#22 0x0000000000670150 in Ffuncall (nargs=2, args=0x7fffffffb940) at eval.c:2822
fun = XIL(0xdb8a05)
original_fun = XIL(0x5430)
funcar = XIL(0x7fffe58c665b)
numargs = 1
val = XIL(0)
count = 10
#23 0x00000000006c3f5c in exec_byte_code (bytestr=XIL(0x7fffe58c6d5c), vector=XIL(0x7fffe58c5b2d),
maxdepth=make_number(23), args_template=make_number(257), nargs=1, args=0x7fffffffc4c8) at bytecode.c:632
op = 1
type = CATCHER
targets = {0x6c7447 <exec_byte_code+17448>, 0x6c7476 <exec_byte_code+17495>, 0x6c7478 <exec_byte_code+17497>,
0x6c747a <exec_byte_code+17499>, 0x6c747c <exec_byte_code+17501>, 0x6c747c <exec_byte_code+17501>,
0x6c74ee <exec_byte_code+17615>, 0x6c7572 <exec_byte_code+17747>, 0x6c35c0 <exec_byte_code+1441>,
0x6c35c2 <exec_byte_code+1443>, 0x6c35c4 <exec_byte_code+1445>, 0x6c35c6 <exec_byte_code+1447>,
0x6c35c8 <exec_byte_code+1449>, 0x6c35c8 <exec_byte_code+1449>, 0x6c35d1 <exec_byte_code+1458>,
0x6c357d <exec_byte_code+1374>, 0x6c3afd <exec_byte_code+2782>, 0x6c3aff <exec_byte_code+2784>,
0x6c3b01 <exec_byte_code+2786>, 0x6c3b03 <exec_byte_code+2788>, 0x6c3b05 <exec_byte_code+2790>,
0x6c3b05 <exec_byte_code+2790>, 0x6c3b4f <exec_byte_code+2864>, 0x6c3b0e <exec_byte_code+2799>,
---Type <return> to continue, or q <return> to quit---
0x6c3e3f <exec_byte_code+3616>, 0x6c3e41 <exec_byte_code+3618>, 0x6c3e43 <exec_byte_code+3620>,
0x6c3e45 <exec_byte_code+3622>, 0x6c3e47 <exec_byte_code+3624>, 0x6c3e47 <exec_byte_code+3624>,
0x6c3dde <exec_byte_code+3519>, 0x6c3dfe <exec_byte_code+3551>, 0x6c3f1a <exec_byte_code+3835>,
0x6c3f1c <exec_byte_code+3837>, 0x6c3f1e <exec_byte_code+3839>, 0x6c3f20 <exec_byte_code+3841>,
0x6c3f22 <exec_byte_code+3843>, 0x6c3f22 <exec_byte_code+3843>, 0x6c3eb9 <exec_byte_code+3738>,
0x6c3ed9 <exec_byte_code+3770>, 0x6c3ffd <exec_byte_code+4062>, 0x6c3fff <exec_byte_code+4064>,
0x6c4001 <exec_byte_code+4066>, 0x6c4003 <exec_byte_code+4068>, 0x6c4005 <exec_byte_code+4070>,
0x6c4005 <exec_byte_code+4070>, 0x6c3f9c <exec_byte_code+3965>, 0x6c3fbc <exec_byte_code+3997>,
0x6c4981 <exec_byte_code+6498>, 0x6c4854 <exec_byte_code+6197>, 0x6c4848 <exec_byte_code+6185>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c4be0 <exec_byte_code+7105>,
0x6c4d17 <exec_byte_code+7416>, 0x6c4d8b <exec_byte_code+7532>, 0x6c4e00 <exec_byte_code+7649>,
0x6c4e76 <exec_byte_code+7767>, 0x6c391b <exec_byte_code+2300>, 0x6c39b3 <exec_byte_code+2452>,
0x6c4f07 <exec_byte_code+7912>, 0x6c3847 <exec_byte_code+2088>, 0x6c3a28 <exec_byte_code+2569>,
0x6c4f86 <exec_byte_code+8039>, 0x6c4ffb <exec_byte_code+8156>, 0x6c504a <exec_byte_code+8235>,
0x6c50bf <exec_byte_code+8352>, 0x6c511b <exec_byte_code+8444>, 0x6c520f <exec_byte_code+8688>,
0x6c525e <exec_byte_code+8767>, 0x6c52d3 <exec_byte_code+8884>, 0x6c536b <exec_byte_code+9036>,
0x6c53ba <exec_byte_code+9115>, 0x6c5409 <exec_byte_code+9194>, 0x6c547e <exec_byte_code+9311>,
0x6c54f3 <exec_byte_code+9428>, 0x6c5568 <exec_byte_code+9545>, 0x6c5600 <exec_byte_code+9697>,
0x6c565c <exec_byte_code+9789>, 0x6c56b8 <exec_byte_code+9881>, 0x6c57ac <exec_byte_code+10125>,
0x6c5836 <exec_byte_code+10263>, 0x6c58c0 <exec_byte_code+10401>, 0x6c5aa2 <exec_byte_code+10883>,
0x6c5b1c <exec_byte_code+11005>, 0x6c5b96 <exec_byte_code+11127>, 0x6c5c10 <exec_byte_code+11249>,
0x6c5c8a <exec_byte_code+11371>, 0x6c5ce6 <exec_byte_code+11463>, 0x6c5d74 <exec_byte_code+11605>,
---Type <return> to continue, or q <return> to quit---
0x6c5dd0 <exec_byte_code+11697>, 0x6c5e2c <exec_byte_code+11789>, 0x6c5e88 <exec_byte_code+11881>,
0x6c5fb5 <exec_byte_code+12182>, 0x6c46c7 <exec_byte_code+5800>, 0x6c601a <exec_byte_code+12283>,
0x6c6069 <exec_byte_code+12362>, 0x6c6155 <exec_byte_code+12598>, 0x6c61c5 <exec_byte_code+12710>,
0x6c622a <exec_byte_code+12811>, 0x6c6279 <exec_byte_code+12890>, 0x6c62c6 <exec_byte_code+12967>,
0x6c6313 <exec_byte_code+13044>, 0x6c6368 <exec_byte_code+13129>, 0x6c7447 <exec_byte_code+17448>,
0x6c63c7 <exec_byte_code+13224>, 0x6c6414 <exec_byte_code+13301>, 0x6c6461 <exec_byte_code+13378>,
0x6c64ae <exec_byte_code+13455>, 0x6c64fb <exec_byte_code+13532>, 0x6c6548 <exec_byte_code+13609>,
0x6c46c7 <exec_byte_code+5800>, 0x6c7447 <exec_byte_code+17448>, 0x6c6597 <exec_byte_code+13688>,
0x6c65f1 <exec_byte_code+13778>, 0x6c6640 <exec_byte_code+13857>, 0x6c668f <exec_byte_code+13936>,
0x6c6704 <exec_byte_code+14053>, 0x6c6779 <exec_byte_code+14170>, 0x6c67c8 <exec_byte_code+14249>,
0x6c690b <exec_byte_code+14572>, 0x6c6980 <exec_byte_code+14689>, 0x6c69f5 <exec_byte_code+14806>,
0x6c6a6a <exec_byte_code+14923>, 0x6c6ab7 <exec_byte_code+15000>, 0x6c7447 <exec_byte_code+17448>,
0x6c45e3 <exec_byte_code+5572>, 0x6c40bf <exec_byte_code+4256>, 0x6c379c <exec_byte_code+1917>,
0x6c41a3 <exec_byte_code+4484>, 0x6c4240 <exec_byte_code+4641>, 0x6c42da <exec_byte_code+4795>,
0x6c4590 <exec_byte_code+5489>, 0x6c45a8 <exec_byte_code+5513>, 0x6c3d81 <exec_byte_code+3426>,
0x6c4682 <exec_byte_code+5731>, 0x6c46ff <exec_byte_code+5856>, 0x6c478c <exec_byte_code+5997>,
0x6c47d1 <exec_byte_code+6066>, 0x6c49ce <exec_byte_code+6575>, 0x6c4a4e <exec_byte_code+6703>,
0x6c4ae6 <exec_byte_code+6855>, 0x6c4b50 <exec_byte_code+6961>, 0x6c406d <exec_byte_code+4174>,
0x6c6b06 <exec_byte_code+15079>, 0x6c6b9e <exec_byte_code+15231>, 0x6c6bed <exec_byte_code+15310>,
0x6c6c3c <exec_byte_code+15389>, 0x6c6c8b <exec_byte_code+15468>, 0x6c6cda <exec_byte_code+15547>,
0x6c6d4f <exec_byte_code+15664>, 0x6c6dc4 <exec_byte_code+15781>, 0x6c6e39 <exec_byte_code+15898>,
0x6c6eae <exec_byte_code+16015>, 0x6c703f <exec_byte_code+16416>, 0x6c70b4 <exec_byte_code+16533>,
0x6c7129 <exec_byte_code+16650>, 0x6c7178 <exec_byte_code+16729>, 0x6c71ed <exec_byte_code+16846>,
---Type <return> to continue, or q <return> to quit---
0x6c7262 <exec_byte_code+16963>, 0x6c72b1 <exec_byte_code+17042>, 0x6c7300 <exec_byte_code+17121>,
0x6c5ee4 <exec_byte_code+11973>, 0x6c5f40 <exec_byte_code+12065>, 0x6c735c <exec_byte_code+17213>,
0x6c73d2 <exec_byte_code+17331>, 0x6c7447 <exec_byte_code+17448>, 0x6c4374 <exec_byte_code+4949>,
0x6c439a <exec_byte_code+4987>, 0x6c4419 <exec_byte_code+5114>, 0x6c4498 <exec_byte_code+5241>,
0x6c4514 <exec_byte_code+5365>, 0x6c5177 <exec_byte_code+8536>, 0x6c5714 <exec_byte_code+9973>,
0x6c60bd <exec_byte_code+12446>, 0x6c7621 <exec_byte_code+17922>, 0x6c76a0 <exec_byte_code+18049>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c774c <exec_byte_code+18221>,
0x6c77f2 <exec_byte_code+18387>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c79ff <exec_byte_code+18912> <repeats 64 times>}
const_length = 106
bytestr_length = 1176
vectorp = 0x7fffe58c5b30
quitcounter = 104 'h'
stack_items = 24
sa_avail = 15016
sa_count = 8
sa_must_free = false
alloc = 0x7fffffffb8b0
item_bytes = 192
stack_base = 0x7fffffffb8b0
top = 0x7fffffffb940
---Type <return> to continue, or q <return> to quit---
stack_lim = 0x7fffffffb970
bytestr_data = 0x7fffffffb970 "\306 \210\b\203\021"
pc = 0x7fffffffbad8 "\266\005\202d\003\016A睃\345\001\313\350\351\016C\"\003\206\203\001\n\211A\022\242\211\262\r\313\332\036D\322\003\003\003#)\266\203\203\237\001\006\n\327\313O\262\vڲ\001\352\353\006\f!!\262\v\211\203\300\001\314\016E\006\fC\"\026E\006\t\203\341\001\016E\262\n\202\341\001\006\t\203\327\001\006\t\006\v\006\vAB\241\210\006\tA\262\n\202\341\001\006\n\016EB\211\026E\262\n\210\202d\003\016A읃\r\002\353\002\206\366\001\n\211A\022\242!\352\001!\355\001!\203\003\002\211\262\002\356\002\313\332#\266\003\202d\003\016A-\002\353\002\206\036\002\n\211A\022\242!\352\001!\356\001\313ډ$\266\003\202d\003\016", <incomplete sequence \360>...
count = 8
result = XIL(0)
#24 0x0000000000670dc0 in funcall_lambda (fun=XIL(0x7fffe58c5afd), nargs=1, arg_vector=0x7fffffffc4c0) at eval.c:3023
size = 5
val = XIL(0x7fffe58c5afd)
syms_left = make_number(257)
next = XIL(0x1200db6c45)
lexenv = XIL(0x7fffffffc400)
count = 8
i = 77324306848
optional = false
rest = false
previous_optional_or_rest = false
#25 0x0000000000670194 in Ffuncall (nargs=2, args=0x7fffffffc4b8) at eval.c:2824
fun = XIL(0x7fffe58c5afd)
---Type <return> to continue, or q <return> to quit---
original_fun = XIL(0x7fffe4a91128)
funcar = XIL(0x7fffffffc4a0)
numargs = 1
val = XIL(0x7fffe4694d90)
count = 7
#26 0x00000000006c3f5c in exec_byte_code (bytestr=XIL(0x7fffe58ca504), vector=XIL(0x7fffe58c6f9d),
maxdepth=make_number(20), args_template=make_number(0), nargs=0, args=0x7fffffffd2c8) at bytecode.c:632
op = 1
type = CONDITION_CASE
targets = {0x6c7447 <exec_byte_code+17448>, 0x6c7476 <exec_byte_code+17495>, 0x6c7478 <exec_byte_code+17497>,
0x6c747a <exec_byte_code+17499>, 0x6c747c <exec_byte_code+17501>, 0x6c747c <exec_byte_code+17501>,
0x6c74ee <exec_byte_code+17615>, 0x6c7572 <exec_byte_code+17747>, 0x6c35c0 <exec_byte_code+1441>,
0x6c35c2 <exec_byte_code+1443>, 0x6c35c4 <exec_byte_code+1445>, 0x6c35c6 <exec_byte_code+1447>,
0x6c35c8 <exec_byte_code+1449>, 0x6c35c8 <exec_byte_code+1449>, 0x6c35d1 <exec_byte_code+1458>,
0x6c357d <exec_byte_code+1374>, 0x6c3afd <exec_byte_code+2782>, 0x6c3aff <exec_byte_code+2784>,
0x6c3b01 <exec_byte_code+2786>, 0x6c3b03 <exec_byte_code+2788>, 0x6c3b05 <exec_byte_code+2790>,
0x6c3b05 <exec_byte_code+2790>, 0x6c3b4f <exec_byte_code+2864>, 0x6c3b0e <exec_byte_code+2799>,
0x6c3e3f <exec_byte_code+3616>, 0x6c3e41 <exec_byte_code+3618>, 0x6c3e43 <exec_byte_code+3620>,
0x6c3e45 <exec_byte_code+3622>, 0x6c3e47 <exec_byte_code+3624>, 0x6c3e47 <exec_byte_code+3624>,
0x6c3dde <exec_byte_code+3519>, 0x6c3dfe <exec_byte_code+3551>, 0x6c3f1a <exec_byte_code+3835>,
0x6c3f1c <exec_byte_code+3837>, 0x6c3f1e <exec_byte_code+3839>, 0x6c3f20 <exec_byte_code+3841>,
0x6c3f22 <exec_byte_code+3843>, 0x6c3f22 <exec_byte_code+3843>, 0x6c3eb9 <exec_byte_code+3738>,
0x6c3ed9 <exec_byte_code+3770>, 0x6c3ffd <exec_byte_code+4062>, 0x6c3fff <exec_byte_code+4064>,
---Type <return> to continue, or q <return> to quit---
0x6c4001 <exec_byte_code+4066>, 0x6c4003 <exec_byte_code+4068>, 0x6c4005 <exec_byte_code+4070>,
0x6c4005 <exec_byte_code+4070>, 0x6c3f9c <exec_byte_code+3965>, 0x6c3fbc <exec_byte_code+3997>,
0x6c4981 <exec_byte_code+6498>, 0x6c4854 <exec_byte_code+6197>, 0x6c4848 <exec_byte_code+6185>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c4be0 <exec_byte_code+7105>,
0x6c4d17 <exec_byte_code+7416>, 0x6c4d8b <exec_byte_code+7532>, 0x6c4e00 <exec_byte_code+7649>,
0x6c4e76 <exec_byte_code+7767>, 0x6c391b <exec_byte_code+2300>, 0x6c39b3 <exec_byte_code+2452>,
0x6c4f07 <exec_byte_code+7912>, 0x6c3847 <exec_byte_code+2088>, 0x6c3a28 <exec_byte_code+2569>,
0x6c4f86 <exec_byte_code+8039>, 0x6c4ffb <exec_byte_code+8156>, 0x6c504a <exec_byte_code+8235>,
0x6c50bf <exec_byte_code+8352>, 0x6c511b <exec_byte_code+8444>, 0x6c520f <exec_byte_code+8688>,
0x6c525e <exec_byte_code+8767>, 0x6c52d3 <exec_byte_code+8884>, 0x6c536b <exec_byte_code+9036>,
0x6c53ba <exec_byte_code+9115>, 0x6c5409 <exec_byte_code+9194>, 0x6c547e <exec_byte_code+9311>,
0x6c54f3 <exec_byte_code+9428>, 0x6c5568 <exec_byte_code+9545>, 0x6c5600 <exec_byte_code+9697>,
0x6c565c <exec_byte_code+9789>, 0x6c56b8 <exec_byte_code+9881>, 0x6c57ac <exec_byte_code+10125>,
0x6c5836 <exec_byte_code+10263>, 0x6c58c0 <exec_byte_code+10401>, 0x6c5aa2 <exec_byte_code+10883>,
0x6c5b1c <exec_byte_code+11005>, 0x6c5b96 <exec_byte_code+11127>, 0x6c5c10 <exec_byte_code+11249>,
0x6c5c8a <exec_byte_code+11371>, 0x6c5ce6 <exec_byte_code+11463>, 0x6c5d74 <exec_byte_code+11605>,
0x6c5dd0 <exec_byte_code+11697>, 0x6c5e2c <exec_byte_code+11789>, 0x6c5e88 <exec_byte_code+11881>,
0x6c5fb5 <exec_byte_code+12182>, 0x6c46c7 <exec_byte_code+5800>, 0x6c601a <exec_byte_code+12283>,
0x6c6069 <exec_byte_code+12362>, 0x6c6155 <exec_byte_code+12598>, 0x6c61c5 <exec_byte_code+12710>,
0x6c622a <exec_byte_code+12811>, 0x6c6279 <exec_byte_code+12890>, 0x6c62c6 <exec_byte_code+12967>,
0x6c6313 <exec_byte_code+13044>, 0x6c6368 <exec_byte_code+13129>, 0x6c7447 <exec_byte_code+17448>,
0x6c63c7 <exec_byte_code+13224>, 0x6c6414 <exec_byte_code+13301>, 0x6c6461 <exec_byte_code+13378>,
---Type <return> to continue, or q <return> to quit---
0x6c64ae <exec_byte_code+13455>, 0x6c64fb <exec_byte_code+13532>, 0x6c6548 <exec_byte_code+13609>,
0x6c46c7 <exec_byte_code+5800>, 0x6c7447 <exec_byte_code+17448>, 0x6c6597 <exec_byte_code+13688>,
0x6c65f1 <exec_byte_code+13778>, 0x6c6640 <exec_byte_code+13857>, 0x6c668f <exec_byte_code+13936>,
0x6c6704 <exec_byte_code+14053>, 0x6c6779 <exec_byte_code+14170>, 0x6c67c8 <exec_byte_code+14249>,
0x6c690b <exec_byte_code+14572>, 0x6c6980 <exec_byte_code+14689>, 0x6c69f5 <exec_byte_code+14806>,
0x6c6a6a <exec_byte_code+14923>, 0x6c6ab7 <exec_byte_code+15000>, 0x6c7447 <exec_byte_code+17448>,
0x6c45e3 <exec_byte_code+5572>, 0x6c40bf <exec_byte_code+4256>, 0x6c379c <exec_byte_code+1917>,
0x6c41a3 <exec_byte_code+4484>, 0x6c4240 <exec_byte_code+4641>, 0x6c42da <exec_byte_code+4795>,
0x6c4590 <exec_byte_code+5489>, 0x6c45a8 <exec_byte_code+5513>, 0x6c3d81 <exec_byte_code+3426>,
0x6c4682 <exec_byte_code+5731>, 0x6c46ff <exec_byte_code+5856>, 0x6c478c <exec_byte_code+5997>,
0x6c47d1 <exec_byte_code+6066>, 0x6c49ce <exec_byte_code+6575>, 0x6c4a4e <exec_byte_code+6703>,
0x6c4ae6 <exec_byte_code+6855>, 0x6c4b50 <exec_byte_code+6961>, 0x6c406d <exec_byte_code+4174>,
0x6c6b06 <exec_byte_code+15079>, 0x6c6b9e <exec_byte_code+15231>, 0x6c6bed <exec_byte_code+15310>,
0x6c6c3c <exec_byte_code+15389>, 0x6c6c8b <exec_byte_code+15468>, 0x6c6cda <exec_byte_code+15547>,
0x6c6d4f <exec_byte_code+15664>, 0x6c6dc4 <exec_byte_code+15781>, 0x6c6e39 <exec_byte_code+15898>,
0x6c6eae <exec_byte_code+16015>, 0x6c703f <exec_byte_code+16416>, 0x6c70b4 <exec_byte_code+16533>,
0x6c7129 <exec_byte_code+16650>, 0x6c7178 <exec_byte_code+16729>, 0x6c71ed <exec_byte_code+16846>,
0x6c7262 <exec_byte_code+16963>, 0x6c72b1 <exec_byte_code+17042>, 0x6c7300 <exec_byte_code+17121>,
0x6c5ee4 <exec_byte_code+11973>, 0x6c5f40 <exec_byte_code+12065>, 0x6c735c <exec_byte_code+17213>,
0x6c73d2 <exec_byte_code+17331>, 0x6c7447 <exec_byte_code+17448>, 0x6c4374 <exec_byte_code+4949>,
0x6c439a <exec_byte_code+4987>, 0x6c4419 <exec_byte_code+5114>, 0x6c4498 <exec_byte_code+5241>,
0x6c4514 <exec_byte_code+5365>, 0x6c5177 <exec_byte_code+8536>, 0x6c5714 <exec_byte_code+9973>,
0x6c60bd <exec_byte_code+12446>, 0x6c7621 <exec_byte_code+17922>, 0x6c76a0 <exec_byte_code+18049>,
---Type <return> to continue, or q <return> to quit---
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c774c <exec_byte_code+18221>,
0x6c77f2 <exec_byte_code+18387>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c79ff <exec_byte_code+18912> <repeats 64 times>}
const_length = 209
bytestr_length = 1715
vectorp = 0x7fffe58c6fa0
quitcounter = 7 '\a'
stack_items = 21
sa_avail = 14501
sa_count = 7
sa_must_free = false
alloc = 0x7fffffffc4b0
item_bytes = 168
stack_base = 0x7fffffffc4b0
top = 0x7fffffffc4b8
stack_lim = 0x7fffffffc558
bytestr_data = 0x7fffffffc558 "\306 \020\307\021\n\023\307\024\310\311!\211\307=\204\060"
pc = 0x7fffffffcb5a "\210\307\016@\211\203L\006\211@\002\204E\006\211;\203E\006\201", <incomplete sequence \310>
count = 7
result = XIL(0x7fffffffd0b0)
---Type <return> to continue, or q <return> to quit---
#27 0x0000000000670dc0 in funcall_lambda (fun=XIL(0x7fffe58c6f6d), nargs=0, arg_vector=0x7fffffffd2c8) at eval.c:3023
size = 5
val = XIL(0x7fffe58c6f6d)
syms_left = make_number(0)
next = XIL(0x1200db4dc5)
lexenv = XIL(0x7fffffffd200)
count = 7
i = 77324306848
optional = false
rest = false
previous_optional_or_rest = false
#28 0x0000000000670194 in Ffuncall (nargs=1, args=0x7fffffffd2c0) at eval.c:2824
fun = XIL(0x7fffe58c6f6d)
original_fun = XIL(0x7fffe4a92598)
funcar = XIL(0xec5215)
numargs = 0
val = XIL(0xec5215)
count = 6
#29 0x00000000006c3f5c in exec_byte_code (bytestr=XIL(0x7fffe58cb144), vector=XIL(0x7fffe58ca6c5),
maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7fffffffdb00) at bytecode.c:632
op = 0
type = CATCHER
targets = {0x6c7447 <exec_byte_code+17448>, 0x6c7476 <exec_byte_code+17495>, 0x6c7478 <exec_byte_code+17497>,
---Type <return> to continue, or q <return> to quit---
0x6c747a <exec_byte_code+17499>, 0x6c747c <exec_byte_code+17501>, 0x6c747c <exec_byte_code+17501>,
0x6c74ee <exec_byte_code+17615>, 0x6c7572 <exec_byte_code+17747>, 0x6c35c0 <exec_byte_code+1441>,
0x6c35c2 <exec_byte_code+1443>, 0x6c35c4 <exec_byte_code+1445>, 0x6c35c6 <exec_byte_code+1447>,
0x6c35c8 <exec_byte_code+1449>, 0x6c35c8 <exec_byte_code+1449>, 0x6c35d1 <exec_byte_code+1458>,
0x6c357d <exec_byte_code+1374>, 0x6c3afd <exec_byte_code+2782>, 0x6c3aff <exec_byte_code+2784>,
0x6c3b01 <exec_byte_code+2786>, 0x6c3b03 <exec_byte_code+2788>, 0x6c3b05 <exec_byte_code+2790>,
0x6c3b05 <exec_byte_code+2790>, 0x6c3b4f <exec_byte_code+2864>, 0x6c3b0e <exec_byte_code+2799>,
0x6c3e3f <exec_byte_code+3616>, 0x6c3e41 <exec_byte_code+3618>, 0x6c3e43 <exec_byte_code+3620>,
0x6c3e45 <exec_byte_code+3622>, 0x6c3e47 <exec_byte_code+3624>, 0x6c3e47 <exec_byte_code+3624>,
0x6c3dde <exec_byte_code+3519>, 0x6c3dfe <exec_byte_code+3551>, 0x6c3f1a <exec_byte_code+3835>,
0x6c3f1c <exec_byte_code+3837>, 0x6c3f1e <exec_byte_code+3839>, 0x6c3f20 <exec_byte_code+3841>,
0x6c3f22 <exec_byte_code+3843>, 0x6c3f22 <exec_byte_code+3843>, 0x6c3eb9 <exec_byte_code+3738>,
0x6c3ed9 <exec_byte_code+3770>, 0x6c3ffd <exec_byte_code+4062>, 0x6c3fff <exec_byte_code+4064>,
0x6c4001 <exec_byte_code+4066>, 0x6c4003 <exec_byte_code+4068>, 0x6c4005 <exec_byte_code+4070>,
0x6c4005 <exec_byte_code+4070>, 0x6c3f9c <exec_byte_code+3965>, 0x6c3fbc <exec_byte_code+3997>,
0x6c4981 <exec_byte_code+6498>, 0x6c4854 <exec_byte_code+6197>, 0x6c4848 <exec_byte_code+6185>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c4be0 <exec_byte_code+7105>,
0x6c4d17 <exec_byte_code+7416>, 0x6c4d8b <exec_byte_code+7532>, 0x6c4e00 <exec_byte_code+7649>,
0x6c4e76 <exec_byte_code+7767>, 0x6c391b <exec_byte_code+2300>, 0x6c39b3 <exec_byte_code+2452>,
0x6c4f07 <exec_byte_code+7912>, 0x6c3847 <exec_byte_code+2088>, 0x6c3a28 <exec_byte_code+2569>,
0x6c4f86 <exec_byte_code+8039>, 0x6c4ffb <exec_byte_code+8156>, 0x6c504a <exec_byte_code+8235>,
0x6c50bf <exec_byte_code+8352>, 0x6c511b <exec_byte_code+8444>, 0x6c520f <exec_byte_code+8688>,
---Type <return> to continue, or q <return> to quit---
0x6c525e <exec_byte_code+8767>, 0x6c52d3 <exec_byte_code+8884>, 0x6c536b <exec_byte_code+9036>,
0x6c53ba <exec_byte_code+9115>, 0x6c5409 <exec_byte_code+9194>, 0x6c547e <exec_byte_code+9311>,
0x6c54f3 <exec_byte_code+9428>, 0x6c5568 <exec_byte_code+9545>, 0x6c5600 <exec_byte_code+9697>,
0x6c565c <exec_byte_code+9789>, 0x6c56b8 <exec_byte_code+9881>, 0x6c57ac <exec_byte_code+10125>,
0x6c5836 <exec_byte_code+10263>, 0x6c58c0 <exec_byte_code+10401>, 0x6c5aa2 <exec_byte_code+10883>,
0x6c5b1c <exec_byte_code+11005>, 0x6c5b96 <exec_byte_code+11127>, 0x6c5c10 <exec_byte_code+11249>,
0x6c5c8a <exec_byte_code+11371>, 0x6c5ce6 <exec_byte_code+11463>, 0x6c5d74 <exec_byte_code+11605>,
0x6c5dd0 <exec_byte_code+11697>, 0x6c5e2c <exec_byte_code+11789>, 0x6c5e88 <exec_byte_code+11881>,
0x6c5fb5 <exec_byte_code+12182>, 0x6c46c7 <exec_byte_code+5800>, 0x6c601a <exec_byte_code+12283>,
0x6c6069 <exec_byte_code+12362>, 0x6c6155 <exec_byte_code+12598>, 0x6c61c5 <exec_byte_code+12710>,
0x6c622a <exec_byte_code+12811>, 0x6c6279 <exec_byte_code+12890>, 0x6c62c6 <exec_byte_code+12967>,
0x6c6313 <exec_byte_code+13044>, 0x6c6368 <exec_byte_code+13129>, 0x6c7447 <exec_byte_code+17448>,
0x6c63c7 <exec_byte_code+13224>, 0x6c6414 <exec_byte_code+13301>, 0x6c6461 <exec_byte_code+13378>,
0x6c64ae <exec_byte_code+13455>, 0x6c64fb <exec_byte_code+13532>, 0x6c6548 <exec_byte_code+13609>,
0x6c46c7 <exec_byte_code+5800>, 0x6c7447 <exec_byte_code+17448>, 0x6c6597 <exec_byte_code+13688>,
0x6c65f1 <exec_byte_code+13778>, 0x6c6640 <exec_byte_code+13857>, 0x6c668f <exec_byte_code+13936>,
0x6c6704 <exec_byte_code+14053>, 0x6c6779 <exec_byte_code+14170>, 0x6c67c8 <exec_byte_code+14249>,
0x6c690b <exec_byte_code+14572>, 0x6c6980 <exec_byte_code+14689>, 0x6c69f5 <exec_byte_code+14806>,
0x6c6a6a <exec_byte_code+14923>, 0x6c6ab7 <exec_byte_code+15000>, 0x6c7447 <exec_byte_code+17448>,
0x6c45e3 <exec_byte_code+5572>, 0x6c40bf <exec_byte_code+4256>, 0x6c379c <exec_byte_code+1917>,
0x6c41a3 <exec_byte_code+4484>, 0x6c4240 <exec_byte_code+4641>, 0x6c42da <exec_byte_code+4795>,
0x6c4590 <exec_byte_code+5489>, 0x6c45a8 <exec_byte_code+5513>, 0x6c3d81 <exec_byte_code+3426>,
0x6c4682 <exec_byte_code+5731>, 0x6c46ff <exec_byte_code+5856>, 0x6c478c <exec_byte_code+5997>,
---Type <return> to continue, or q <return> to quit---
0x6c47d1 <exec_byte_code+6066>, 0x6c49ce <exec_byte_code+6575>, 0x6c4a4e <exec_byte_code+6703>,
0x6c4ae6 <exec_byte_code+6855>, 0x6c4b50 <exec_byte_code+6961>, 0x6c406d <exec_byte_code+4174>,
0x6c6b06 <exec_byte_code+15079>, 0x6c6b9e <exec_byte_code+15231>, 0x6c6bed <exec_byte_code+15310>,
0x6c6c3c <exec_byte_code+15389>, 0x6c6c8b <exec_byte_code+15468>, 0x6c6cda <exec_byte_code+15547>,
0x6c6d4f <exec_byte_code+15664>, 0x6c6dc4 <exec_byte_code+15781>, 0x6c6e39 <exec_byte_code+15898>,
0x6c6eae <exec_byte_code+16015>, 0x6c703f <exec_byte_code+16416>, 0x6c70b4 <exec_byte_code+16533>,
0x6c7129 <exec_byte_code+16650>, 0x6c7178 <exec_byte_code+16729>, 0x6c71ed <exec_byte_code+16846>,
0x6c7262 <exec_byte_code+16963>, 0x6c72b1 <exec_byte_code+17042>, 0x6c7300 <exec_byte_code+17121>,
0x6c5ee4 <exec_byte_code+11973>, 0x6c5f40 <exec_byte_code+12065>, 0x6c735c <exec_byte_code+17213>,
0x6c73d2 <exec_byte_code+17331>, 0x6c7447 <exec_byte_code+17448>, 0x6c4374 <exec_byte_code+4949>,
0x6c439a <exec_byte_code+4987>, 0x6c4419 <exec_byte_code+5114>, 0x6c4498 <exec_byte_code+5241>,
0x6c4514 <exec_byte_code+5365>, 0x6c5177 <exec_byte_code+8536>, 0x6c5714 <exec_byte_code+9973>,
0x6c60bd <exec_byte_code+12446>, 0x6c7621 <exec_byte_code+17922>, 0x6c76a0 <exec_byte_code+18049>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c774c <exec_byte_code+18221>,
0x6c77f2 <exec_byte_code+18387>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>,
0x6c79ff <exec_byte_code+18912> <repeats 64 times>}
const_length = 77
bytestr_length = 443
vectorp = 0x7fffe58ca6c8
quitcounter = 41 ')'
stack_items = 13
---Type <return> to continue, or q <return> to quit---
sa_avail = 15837
sa_count = 5
sa_must_free = false
alloc = 0x7fffffffd2b0
item_bytes = 104
stack_base = 0x7fffffffd2b0
top = 0x7fffffffd2c0
stack_lim = 0x7fffffffd318
bytestr_data = 0x7fffffffd318 "\b\203\b"
pc = 0x7fffffffd48c "\210)\210\375\376\377\"\210\201H"
count = 5
result = XIL(0xcff5ab3b)
#30 0x0000000000670dc0 in funcall_lambda (fun=XIL(0x7fffe58ca695), nargs=0, arg_vector=0x7fffffffdb00) at eval.c:3023
size = 5
val = XIL(0x7fffffffdb00)
syms_left = make_number(0)
next = XIL(0x7fffffffdab0)
lexenv = XIL(0)
count = 5
i = 6770506
optional = false
rest = false
previous_optional_or_rest = false
---Type <return> to continue, or q <return> to quit---
#31 0x00000000006709e8 in apply_lambda (fun=XIL(0x7fffe58ca695), args=XIL(0), count=4) at eval.c:2959
args_left = XIL(0)
i = 0
numargs = 0
arg_vector = 0x7fffffffdb00
tem = XIL(0)
sa_avail = 16384
sa_count = 5
sa_must_free = false
#32 0x000000000066e973 in eval_sub (form=XIL(0x7fffe5a1c16b)) at eval.c:2332
fun = XIL(0x7fffe58ca695)
val = XIL(0x7fffe53c1500)
original_fun = XIL(0x7fffe4a95cc0)
original_args = XIL(0)
funcar = XIL(0x7fffffffdc10)
count = 4
argvals = {XIL(0x5976c7), XIL(0xeb4920), XIL(0x7fffffffdcd0), XIL(0x672073), XIL(0), XIL(0xe3c470),
XIL(0x7ad0), XIL(0x7ad0)}
#33 0x000000000066dc63 in Feval (form=XIL(0x7fffe5a1c16b), lexical=XIL(0)) at eval.c:2107
count = 3
#34 0x00000000005a0b0e in top_level_2 () at keyboard.c:1122
No locals.
#35 0x000000000066bc8e in internal_condition_case (bfun=0x5a0af1 <top_level_2>, handlers=XIL(0x5310),
---Type <return> to continue, or q <return> to quit---
hfun=0x5a04de <cmd_error>) at eval.c:1336
val = XIL(0xed1570)
c = 0xed1690
#36 0x00000000005a0b52 in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
No locals.
#37 0x000000000066b12d in internal_catch (tag=XIL(0xc7e0), func=0x5a0b10 <top_level_1>, arg=XIL(0)) at eval.c:1101
val = XIL(0x7fffffffde00)
c = 0xed1570
#38 0x00000000005a0a43 in command_loop () at keyboard.c:1091
No locals.
#39 0x000000000059ffd8 in recursive_edit_1 () at keyboard.c:698
count = 1
val = XIL(0x7fffffffde60)
#40 0x00000000005a01c6 in Frecursive_edit () at keyboard.c:769
count = 0
buffer = XIL(0)
#41 0x000000000059df6b in main (argc=6, argv=0x7fffffffe0b8) at emacs.c:1929
stack_bottom_variable = 0x0
do_initial_setlocale = true
skip_args = 1
no_loadup = false
junk = 0x0
dname_arg = 0x0
---Type <return> to continue, or q <return> to quit---
ch_to_dir = 0x0
original_pwd = 0x0
argv0_base = 0x7fffffffe4d9 "emacs"
is_temacs = false
loaded_dump = 0x0
dump_mode = 0x0
disable_aslr = false
rlim = {
rlim_cur = 10022912,
rlim_max = 18446744073709551615
}
sockfd = -1
Lisp Backtrace:
"dump-emacs-portable" (0xffffb6d0)
"eval" (0xffffb948)
"command-line-1" (0xffffc4c0)
"command-line" (0xffffd2c8)
"normal-top-level" (0xffffdb00)
(gdb)
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-13 16:37 ` Eli Zaretskii
` (2 preceding siblings ...)
2018-02-17 1:01 ` Clément Pit-Claudel
@ 2018-02-17 11:53 ` Charles A. Roelli
2018-02-17 12:09 ` Alan Third
2018-02-20 0:54 ` Andy Moreton
4 siblings, 1 reply; 299+ messages in thread
From: Charles A. Roelli @ 2018-02-17 11:53 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: dancol, emacs-devel
> Date: Tue, 13 Feb 2018 18:37:43 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> CC: emacs-devel@gnu.org
> Reply-to: Eli Zaretskii <eliz@gnu.org>
>
> > From: Daniel Colascione <dancol@dancol.org>
> > Date: Mon, 12 Feb 2018 12:18:36 -0800
> >
> > I've pushed the portable dumper to the pdumper Savannah branch. It
> > should support a fully PIC Emacs.
>
> Thanks. I'd urge people to try this branch and report any issues they
> see.
On Apple's version of FSF GCC, I saw these kinds of errors:
CC pdumper.o
pdumper.c:63: warning: unknown option after ‘#pragma GCC diagnostic’ kind
pdumper.c: In function ‘dump_reloc_set_offset’:
pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
pdumper.c:291: error: #pragma GCC diagnostic not allowed inside functions
They're fixed locally with this patch:
diff --git a/src/pdumper.c b/src/pdumper.c
index a05b2c4..844ec25 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -54,7 +54,10 @@
#ifdef HAVE_PDUMPER
-#ifdef __GNUC__
+#if defined(__GNUC__) && !defined(__APPLE__) /* Apple's old GCC (based
+ on FSF's GCC) does not
+ support pragmas in
+ functions. */
# pragma GCC diagnostic error "-Wconversion"
# pragma GCC diagnostic error "-Wshadow"
# define ALLOW_IMPLICIT_CONVERSION \
Afterwards I had only this error remaining:
pdumper.c: In function ‘dump_anonymous_allocate_posix’:
pdumper.c:4043: error: ‘MAP_ANONYMOUS’ undeclared (first use in this function)
pdumper.c:4043: error: (Each undeclared identifier is reported only once
pdumper.c:4043: error: for each function it appears in.)
Seems like MAP_ANONYMOUS is not defined on all systems -- I don't know
where to go from there.
^ permalink raw reply related [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-17 11:53 ` Charles A. Roelli
@ 2018-02-17 12:09 ` Alan Third
2018-02-17 14:12 ` Charles A. Roelli
0 siblings, 1 reply; 299+ messages in thread
From: Alan Third @ 2018-02-17 12:09 UTC (permalink / raw)
To: Charles A. Roelli; +Cc: Eli Zaretskii, dancol, emacs-devel
On Sat, Feb 17, 2018 at 12:53:48PM +0100, Charles A. Roelli wrote:
> On Apple's version of FSF GCC, I saw these kinds of errors:
>
> CC pdumper.o
> pdumper.c:63: warning: unknown option after ‘#pragma GCC diagnostic’ kind
> pdumper.c: In function ‘dump_reloc_set_offset’:
> pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
> pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
> pdumper.c:291: error: #pragma GCC diagnostic not allowed inside functions
>
> They're fixed locally with this patch:
>
> diff --git a/src/pdumper.c b/src/pdumper.c
> index a05b2c4..844ec25 100644
> --- a/src/pdumper.c
> +++ b/src/pdumper.c
> @@ -54,7 +54,10 @@
>
> #ifdef HAVE_PDUMPER
>
> -#ifdef __GNUC__
> +#if defined(__GNUC__) && !defined(__APPLE__) /* Apple's old GCC (based
> + on FSF's GCC) does not
> + support pragmas in
> + functions. */
> # pragma GCC diagnostic error "-Wconversion"
> # pragma GCC diagnostic error "-Wshadow"
> # define ALLOW_IMPLICIT_CONVERSION \
I’m pretty sure this will disable these pragmas for more recent clang
versions on macOS, and probably any newer version of GCC people have
compiled themselves..
--
Alan Third
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-17 12:09 ` Alan Third
@ 2018-02-17 14:12 ` Charles A. Roelli
0 siblings, 0 replies; 299+ messages in thread
From: Charles A. Roelli @ 2018-02-17 14:12 UTC (permalink / raw)
To: Alan Third; +Cc: eliz, dancol, emacs-devel
> From: Alan Third <alan@idiocy.org>
>
> On Sat, Feb 17, 2018 at 12:53:48PM +0100, Charles A. Roelli wrote:
> > On Apple's version of FSF GCC, I saw these kinds of errors:
> >
> > CC pdumper.o
> > pdumper.c:63: warning: unknown option after ‘#pragma GCC diagnostic’ kind
> > pdumper.c: In function ‘dump_reloc_set_offset’:
> > pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
> > pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
> > pdumper.c:291: error: #pragma GCC diagnostic not allowed inside functions
> >
> > They're fixed locally with this patch:
> >
> > diff --git a/src/pdumper.c b/src/pdumper.c
> > index a05b2c4..844ec25 100644
> > --- a/src/pdumper.c
> > +++ b/src/pdumper.c
> > @@ -54,7 +54,10 @@
> >
> > #ifdef HAVE_PDUMPER
> >
> > -#ifdef __GNUC__
> > +#if defined(__GNUC__) && !defined(__APPLE__) /* Apple's old GCC (based
> > + on FSF's GCC) does not
> > + support pragmas in
> > + functions. */
> > # pragma GCC diagnostic error "-Wconversion"
> > # pragma GCC diagnostic error "-Wshadow"
> > # define ALLOW_IMPLICIT_CONVERSION \
>
> I’m pretty sure this will disable these pragmas for more recent clang
> versions on macOS, and probably any newer version of GCC people have
> compiled themselves..
> --
> Alan Third
Whoops, my bad. If there's a feature test for pragmas, that would of
course be better.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-16 2:36 ` Daniel Colascione
@ 2018-02-17 23:38 ` Ken Brown
2018-02-17 23:59 ` Ken Brown
2018-02-18 0:02 ` Daniel Colascione
0 siblings, 2 replies; 299+ messages in thread
From: Ken Brown @ 2018-02-17 23:38 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel
On 2/15/2018 9:36 PM, Daniel Colascione wrote:
> On 02/15/2018 05:56 PM, Ken Brown wrote:
>> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
[...]
>>> I just tried to build on 64-bit Cygwin, and the build fails as
>>> follows:
>>>
>>> [...]
>>> Dumping under the name bootstrap-emacs.pdmp
>>> dumping fingerprint:
>>> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>> Dump complete
>>> Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
>>> Reloc counts: hot=919268 discardable=5790
>>> make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>>> make[2]: Entering directory
>>> '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>> ELC ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>> ELC ../../pdumper/lisp/emacs-lisp/cconv.elc
>>> ELC ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>> ELC ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>> emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
>>> of memory
>>>
>>> There's probably some obvious explanation, but I don't see it at the
>>> moment.
>>>
>>>
>>> I'm not entirely surprised to see Cygwin fall over here. We could
>>> just use the Windows memory mapping functions directly, but I'd
>>> prefer to stick with the POSIX API if we can make it work.
>>
>> I agree.
>>
>>> Any idea where in the mmap sequence we fail?
>>
>> I haven't looked at the code yet, so I don't understand the question.
>> If you give me some guidance, I'll try to investigate.
>
> Thanks. I think the trouble must be in dump_mmap_contiguous. We report
> all errors from this function as "out of memory". dump_mmap_contiguous
> takes an array of mapping descriptors (think ELF segments or something)
> and maps them all into a single contiguous region of virtual memory. On
> POSIX systems, we reserve a chunk of address space with a big PROT_NONE
> anonymous mapping, then carve it up into the separate mappings that we
> really want. Windows doesn't support atomic mmap replacement, so Cygwin
> has to emulate it, and I wouldn't be surprised if something's going
> wrong in this emulation.
You're right that the problem is in dump_mmap_contiguous. I stepped
through it in gdb and found the following:
1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
2. We enter the loop at pdumper.c:4451 with i = 0. dump_map_file is
called, which calls dump_map_file_posix with protection =
DUMP_MEMORY_ACCESS_READWRITE.
3. This calls mmap at line 4219 with mem_prot = PROT_READ | PROT_WRITE
and mem_flags = MAP_PRIVATE | MAP_FIXED.
4. This mmap call fails.
I don't know if this is a bug in Cygwin's mmap or simply a limitation
that has to be worked around. Should I take this to the Cygwin mailing
list, or do you have other ideas as to how to proceed?
> That you're able to compile a few elisp files before bootstrap starts
> failing...
That's an illusion. I was doing a parallel build, and the output I
posted simply showed the first few files that emacs was trying to compile.
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-17 23:38 ` Ken Brown
@ 2018-02-17 23:59 ` Ken Brown
2018-02-18 0:02 ` Daniel Colascione
1 sibling, 0 replies; 299+ messages in thread
From: Ken Brown @ 2018-02-17 23:59 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel
On 2/17/2018 6:38 PM, Ken Brown wrote:
> On 2/15/2018 9:36 PM, Daniel Colascione wrote:
>> On 02/15/2018 05:56 PM, Ken Brown wrote:
>>> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
> [...]
>>>> I just tried to build on 64-bit Cygwin, and the build fails as
>>>> follows:
>>>>
>>>> [...]
>>>> Dumping under the name bootstrap-emacs.pdmp
>>>> dumping fingerprint:
>>>> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>>> Dump complete
>>>> Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
>>>> Reloc counts: hot=919268 discardable=5790
>>>> make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>>>> make[2]: Entering directory
>>>> '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>>> ELC ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>>> ELC ../../pdumper/lisp/emacs-lisp/cconv.elc
>>>> ELC ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>>> ELC ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>>> emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
>>>> of memory
>>>>
>>>> There's probably some obvious explanation, but I don't see it at
>>>> the
>>>> moment.
>>>>
>>>>
>>>> I'm not entirely surprised to see Cygwin fall over here. We could
>>>> just use the Windows memory mapping functions directly, but I'd
>>>> prefer to stick with the POSIX API if we can make it work.
>>>
>>> I agree.
>>>
>>>> Any idea where in the mmap sequence we fail?
>>>
>>> I haven't looked at the code yet, so I don't understand the question.
>>> If you give me some guidance, I'll try to investigate.
>>
>> Thanks. I think the trouble must be in dump_mmap_contiguous. We report
>> all errors from this function as "out of memory". dump_mmap_contiguous
>> takes an array of mapping descriptors (think ELF segments or
>> something) and maps them all into a single contiguous region of
>> virtual memory. On POSIX systems, we reserve a chunk of address space
>> with a big PROT_NONE anonymous mapping, then carve it up into the
>> separate mappings that we really want. Windows doesn't support atomic
>> mmap replacement, so Cygwin has to emulate it, and I wouldn't be
>> surprised if something's going wrong in this emulation.
>
> You're right that the problem is in dump_mmap_contiguous. I stepped
> through it in gdb and found the following:
>
> 1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
>
> 2. We enter the loop at pdumper.c:4451 with i = 0. dump_map_file is
> called, which calls dump_map_file_posix with protection =
> DUMP_MEMORY_ACCESS_READWRITE.
>
> 3. This calls mmap at line 4219 with mem_prot = PROT_READ | PROT_WRITE
> and mem_flags = MAP_PRIVATE | MAP_FIXED.
>
> 4. This mmap call fails.
>
> I don't know if this is a bug in Cygwin's mmap or simply a limitation
> that has to be worked around. Should I take this to the Cygwin mailing
> list, or do you have other ideas as to how to proceed?
The following looks relevant:
https://sourceware.org/ml/cygwin/2013-04/msg00372.html
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-17 23:38 ` Ken Brown
2018-02-17 23:59 ` Ken Brown
@ 2018-02-18 0:02 ` Daniel Colascione
2018-02-19 13:30 ` Ken Brown
1 sibling, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2018-02-18 0:02 UTC (permalink / raw)
To: Ken Brown; +Cc: Eli Zaretskii, emacs-devel
[-- Attachment #1: Type: text/html, Size: 5619 bytes --]
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-18 0:02 ` Daniel Colascione
@ 2018-02-19 13:30 ` Ken Brown
2018-02-19 17:03 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: Ken Brown @ 2018-02-19 13:30 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel
On 2/17/2018 7:02 PM, Daniel Colascione wrote:
> On Feb 17, 2018 3:38 PM, Ken Brown <kbrown@cornell.edu> wrote:
>
> On 2/15/2018 9:36 PM, Daniel Colascione wrote:
> > On 02/15/2018 05:56 PM, Ken Brown wrote:
> >> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
> >>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
> [...]
> >>> I just tried to build on 64-bit Cygwin, and the build fails as
> >>> follows:
> >>>
> >>> [...]
> >>> Dumping under the name bootstrap-emacs.pdmp
> >>> dumping fingerprint:
> >>>
> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
> >>> Dump complete
> >>> Byte counts: header=80 hot=13187392 discardable=119424
> cold=9086640
> >>> Reloc counts: hot=919268 discardable=5790
> >>> make -C ../lisp compile-first
> EMACS="../src/bootstrap-emacs.exe"
> >>> make[2]: Entering directory
> >>> '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
> >>> ELC ../../pdumper/lisp/emacs-lisp/macroexp.elc
> >>> ELC ../../pdumper/lisp/emacs-lisp/cconv.elc
> >>> ELC ../../pdumper/lisp/emacs-lisp/byte-opt.elc
> >>> ELC ../../pdumper/lisp/emacs-lisp/bytecomp.elc
> >>> emacs: could not load dump file
> "../src/bootstrap-emacs.pdmp": out
> >>> of memory
> >>>
> >>> There's probably some obvious explanation, but I don't see
> it at the
> >>> moment.
> >>>
> >>>
> >>> I'm not entirely surprised to see Cygwin fall over here. We could
> >>> just use the Windows memory mapping functions directly, but I'd
> >>> prefer to stick with the POSIX API if we can make it work.
> >>
> >> I agree.
> >>
> >>> Any idea where in the mmap sequence we fail?
> >>
> >> I haven't looked at the code yet, so I don't understand the
> question.
> >> If you give me some guidance, I'll try to investigate.
> >
> > Thanks. I think the trouble must be in dump_mmap_contiguous. We
> report
> > all errors from this function as "out of memory".
> dump_mmap_contiguous
> > takes an array of mapping descriptors (think ELF segments or
> something)
> > and maps them all into a single contiguous region of virtual
> memory. On
> > POSIX systems, we reserve a chunk of address space with a big
> PROT_NONE
> > anonymous mapping, then carve it up into the separate mappings
> that we
> > really want. Windows doesn't support atomic mmap replacement, so
> Cygwin
> > has to emulate it, and I wouldn't be surprised if something's going
> > wrong in this emulation.
>
> You're right that the problem is in dump_mmap_contiguous. I stepped
> through it in gdb and found the following:
>
> 1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
>
> 2. We enter the loop at pdumper.c:4451 with i = 0. dump_map_file is
> called, which calls dump_map_file_posix with protection =
> DUMP_MEMORY_ACCESS_READWRITE.
>
> 3. This calls mmap at line 4219 with mem_prot = PROT_READ | PROT_WRITE
> and mem_flags = MAP_PRIVATE | MAP_FIXED.
>
> 4. This mmap call fails.
>
> I don't know if this is a bug in Cygwin's mmap or simply a limitation
> that has to be worked around. Should I take this to the Cygwin mailing
> list, or do you have other ideas as to how to proceed?
>
> I'd definitely let the Cygwin people know. If the problem is that Cygwin
> doesn't properly support atomic map replacement, I think we can work
> around the problem by using the same unmap-before-remap approach we use
> for the native Windows code, but keep using the POSIX memory functions.
> I can come up with a patch blind, but you'd be in a better position to
> iterate quickly.
OK, here's Corinna's answer:
https://cygwin.com/ml/cygwin/2018-02/msg00202.html
It seems like she's suggesting the same unmap-before-remap approach that
you mentioned.
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-19 13:30 ` Ken Brown
@ 2018-02-19 17:03 ` Daniel Colascione
2018-02-19 22:33 ` Ken Brown
2018-02-20 1:16 ` Andy Moreton
0 siblings, 2 replies; 299+ messages in thread
From: Daniel Colascione @ 2018-02-19 17:03 UTC (permalink / raw)
To: Ken Brown; +Cc: Eli Zaretskii, emacs-devel
On 02/19/2018 05:30 AM, Ken Brown wrote:
> On 2/17/2018 7:02 PM, Daniel Colascione wrote:
>> On Feb 17, 2018 3:38 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>
>> On 2/15/2018 9:36 PM, Daniel Colascione wrote:
>> > On 02/15/2018 05:56 PM, Ken Brown wrote:
>> >> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>> >>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>> [...]
>> >>> I just tried to build on 64-bit Cygwin, and the build
>> fails as
>> >>> follows:
>> >>>
>> >>> [...]
>> >>> Dumping under the name bootstrap-emacs.pdmp
>> >>> dumping fingerprint:
>> >>>
>> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>> >>> Dump complete
>> >>> Byte counts: header=80 hot=13187392 discardable=119424
>> cold=9086640
>> >>> Reloc counts: hot=919268 discardable=5790
>> >>> make -C ../lisp compile-first
>> EMACS="../src/bootstrap-emacs.exe"
>> >>> make[2]: Entering directory
>> >>> '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>> >>> ELC ../../pdumper/lisp/emacs-lisp/macroexp.elc
>> >>> ELC ../../pdumper/lisp/emacs-lisp/cconv.elc
>> >>> ELC ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>> >>> ELC ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>> >>> emacs: could not load dump file
>> "../src/bootstrap-emacs.pdmp": out
>> >>> of memory
>> >>>
>> >>> There's probably some obvious explanation, but I don't see
>> it at the
>> >>> moment.
>> >>>
>> >>>
>> >>> I'm not entirely surprised to see Cygwin fall over here. We
>> could
>> >>> just use the Windows memory mapping functions directly, but I'd
>> >>> prefer to stick with the POSIX API if we can make it work.
>> >>
>> >> I agree.
>> >>
>> >>> Any idea where in the mmap sequence we fail?
>> >>
>> >> I haven't looked at the code yet, so I don't understand the
>> question.
>> >> If you give me some guidance, I'll try to investigate.
>> >
>> > Thanks. I think the trouble must be in dump_mmap_contiguous. We
>> report
>> > all errors from this function as "out of memory".
>> dump_mmap_contiguous
>> > takes an array of mapping descriptors (think ELF segments or
>> something)
>> > and maps them all into a single contiguous region of virtual
>> memory. On
>> > POSIX systems, we reserve a chunk of address space with a big
>> PROT_NONE
>> > anonymous mapping, then carve it up into the separate mappings
>> that we
>> > really want. Windows doesn't support atomic mmap replacement, so
>> Cygwin
>> > has to emulate it, and I wouldn't be surprised if something's
>> going
>> > wrong in this emulation.
>>
>> You're right that the problem is in dump_mmap_contiguous. I stepped
>> through it in gdb and found the following:
>>
>> 1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
>>
>> 2. We enter the loop at pdumper.c:4451 with i = 0. dump_map_file is
>> called, which calls dump_map_file_posix with protection =
>> DUMP_MEMORY_ACCESS_READWRITE.
>>
>> 3. This calls mmap at line 4219 with mem_prot = PROT_READ |
>> PROT_WRITE
>> and mem_flags = MAP_PRIVATE | MAP_FIXED.
>>
>> 4. This mmap call fails.
>>
>> I don't know if this is a bug in Cygwin's mmap or simply a limitation
>> that has to be worked around. Should I take this to the Cygwin
>> mailing
>> list, or do you have other ideas as to how to proceed?
>>
>> I'd definitely let the Cygwin people know. If the problem is that
>> Cygwin doesn't properly support atomic map replacement, I think we can
>> work around the problem by using the same unmap-before-remap approach
>> we use for the native Windows code, but keep using the POSIX memory
>> functions. I can come up with a patch blind, but you'd be in a better
>> position to iterate quickly.
>
> OK, here's Corinna's answer:
>
> https://cygwin.com/ml/cygwin/2018-02/msg00202.html
>
> It seems like she's suggesting the same unmap-before-remap approach that
> you mentioned.
Thanks. I've pushed a fix, I think. Try it now.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-17 1:01 ` Clément Pit-Claudel
@ 2018-02-19 17:06 ` Daniel Colascione
2018-02-19 22:00 ` Clément Pit-Claudel
0 siblings, 1 reply; 299+ messages in thread
From: Daniel Colascione @ 2018-02-19 17:06 UTC (permalink / raw)
To: Clément Pit-Claudel, emacs-devel
On 02/16/2018 05:01 PM, Clément Pit-Claudel wrote:
> * I tried to use dump-emacs-portable, but it crashed. I've pasted the backtrace at the bottom of this email.
Should be fixed. Note that the dump will still fail: we can't dump
window configuration objects. But at least Emacs won't crash.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-19 17:06 ` Daniel Colascione
@ 2018-02-19 22:00 ` Clément Pit-Claudel
0 siblings, 0 replies; 299+ messages in thread
From: Clément Pit-Claudel @ 2018-02-19 22:00 UTC (permalink / raw)
To: Daniel Colascione, emacs-devel
On 2018-02-19 12:06, Daniel Colascione wrote:
> On 02/16/2018 05:01 PM, Clément Pit-Claudel wrote:
>> * I tried to use dump-emacs-portable, but it crashed. I've pasted the backtrace at the bottom of this email.
>
> Should be fixed. Note that the dump will still fail: we can't dump window configuration objects. But at least Emacs won't crash.
Thanks, this seems to have partly done the trick. I now get "Invalid face: eldoc-highlight-function-argument" when trying to redump, which is indeed better than a crash. Is that the error you expected?
I tried redumping after loading a smaller init file, but I got assertion failures; here's a repro:
(gdb) run --batch --eval '(dump-emacs-portable "test.pdump")'
Starting program: /build/emacs/pdumper/src/emacs --batch --eval '(dump-emacs-portable "test.pdump")'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
load_dump completed in 14.372 milliseconds: result=0
[New Thread 0x7fffe49d7700 (LWP 9176)]
dumping fingerprint: b0a6626af263696a377e04b184acc7f4ca0786a175a63b064366fda2f150ada3
pdumper.c:2603: Emacs fatal error: assertion failed: XHASH_TABLE (table_rehashed)->count >= 0
Thread 1 "emacs" hit Breakpoint 1, terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:357
357 {
(gdb) bt full
#0 terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:357
No locals.
#1 0x000000000063a821 in die (msg=0x79f9e0 "XHASH_TABLE (table_rehashed)->count >= 0", file=0x79ef40 "pdumper.c", line=2603) at alloc.c:7782
No locals.
#2 0x00000000006400b0 in check_hash_table_rehash (table_orig=XIL(0x7fffe5429495)) at pdumper.c:2603
table_rehashed = XIL(0xec6465)
expected_contents = XIL(0x7fffffffb290)
#3 0x00000000006403ce in dump_hash_table (ctx=0x7fffffffb550, hash_in=0x7fffe5429490) at pdumper.c:2651
is_stable = false
hash_munged = {
header = {
size = 140737488335680,
gcaligned = 64 '@'
},
weak = XIL(0x681095),
hash = XIL(0x7fffffffb340),
next = XIL(0xbaf),
index = XIL(0x7fffdffb6025),
count = 15701936,
next_free = 140737488335696,
pure = 191,
rehash_threshold = 0,
rehash_size = -3.62270368e+19,
key_and_value = XIL(0x8d44),
test = {
name = XIL(0x7fffffffb370),
user_hash_function = XIL(0x5970a8),
user_cmp_function = make_number(4520),
cmpfn = 0xec65e0,
hashfn = 0x7fffffffb3d0
},
next_weak = 0x682490 <hash_lookup+228>
}
hash = 0xbac
out = {
header = {
size = 140736951181349,
gcaligned = 37 '%'
},
weak = XIL(0),
hash = XIL(0x7fffffffb3b0),
next = XIL(0x7fffffff),
index = make_number(-1),
count = -2147483648,
next_free = 140737488335824,
pure = 216,
rehash_threshold = 2.3509887e-38,
rehash_size = 2.53382788e-41,
key_and_value = XIL(0xec65e0),
test = {
name = XIL(0xffffffffffffffff),
user_hash_function = XIL(0x683ec7),
user_cmp_function = make_number(0),
cmpfn = 0x46ae3ed52235,
hashfn = 0x7fffffffb420
},
next_weak = 0xe359a0 <lispsym>
}
#4 0x0000000000641566 in dump_vectorlike (ctx=0x7fffffffb550, v=0x7fffe5429490) at pdumper.c:2919
offset = 32767
lv = XIL(0x7fffe5429495)
#5 0x000000000064179b in dump_object_1 (ctx=0x7fffffffb550, object=XIL(0x7fffe5429495)) at pdumper.c:2988
offset = -1
#6 0x00000000006419d6 in dump_object (ctx=0x7fffffffb550, object=XIL(0x7fffe5429495)) at pdumper.c:3053
result = 0
#7 0x0000000000644203 in Fdump_emacs_portable (filename=XIL(0xeca154), track_referrers=XIL(0)) at pdumper.c:3884
count = 13
symbol = XIL(0x7fffe4a94478)
ctx_buf = {
header = {
magic = "!UMPEDGNUEMACS\000",
fingerprint = "\260\246bj\362cij7~\004\261\204\254\307\364\312\a\206\241u\246;\006Cf\375\242\361P\255\243",
dump_relocs = {
offset = 0,
nr_entries = 0
},
object_starts = {
offset = 0,
nr_entries = 0
},
emacs_relocs = {
offset = 0,
nr_entries = 0
},
discardable_start = 0,
cold_start = 0
},
old_purify_flag = XIL(0),
old_post_gc_hook = XIL(0),
fd = 7,
dump_filename = XIL(0xeca154),
offset = 601000,
obj_offset = 0,
flags = {
dump_object_contents = true,
dump_object_starts = true,
pack_objects = false,
assert_already_seen = false,
defer_hash_tables = true,
defer_symbols = false
},
end_heap = 0,
objects_dumped = XIL(0xec65e5),
referrers = XIL(0),
current_referrer = XIL(0),
have_current_referrer = true,
dump_queue = {
zero_weight_objects = {
head = XIL(0x145cf53),
tail = XIL(0xeb8ec3),
length = 1529
},
one_weight_normal_objects = {
head = XIL(0x1499113),
tail = XIL(0x1499113),
length = 1
},
one_weight_strong_objects = {
head = XIL(0x1498403),
tail = XIL(0xfaa033),
length = 1713
},
fancy_weight_objects = {
head = XIL(0),
tail = XIL(0),
length = 0
},
link_weights = XIL(0xeff225),
sequence_numbers = XIL(0xef97b5),
next_sequence_number = 18098
},
deferred_hash_tables = XIL(0x145cf63),
deferred_symbols = XIL(0),
fixups = XIL(0x1498493),
symbol_aux = XIL(0),
copied_queue = XIL(0x149b163),
cold_queue = XIL(0x1499643),
dump_relocs = XIL(0x149b193),
object_starts = XIL(0x1498523),
emacs_relocs = XIL(0x149bbc3),
number_hot_relocations = 0,
number_discardable_relocations = 0
}
ctx = 0x7fffffffb550
header_start = 0
header_end = 80
hot_start = 80
hot_end = 0
discardable_end = 52608
number_hot_relocations = 0
number_discardable_relocations = 52608
cold_end = 0
#8 0x000000000066e7e1 in eval_sub (form=XIL(0xefb043)) at eval.c:2293
i = 2
maxargs = 2
args_left = XIL(0)
numargs = make_number(1)
fun = XIL(0xdb6045)
val = XIL(0x7fffe53c0500)
original_fun = XIL(0x7fffe46746b8)
original_args = XIL(0xefb053)
funcar = XIL(0x7fffffffb750)
count = 12
argvals = {XIL(0xf11c44), XIL(0), XIL(0x7fffffffb810), XIL(0x67208b), XIL(0), XIL(0xe3d470), XIL(0x7ad0), XIL(0x7ad0)}
#9 0x000000000066dc7b in Feval (form=XIL(0xefb043), lexical=XIL(0)) at eval.c:2110
count = 11
#10 0x0000000000670659 in funcall_subr (subr=0xdb9a00 <Seval>, numargs=1, args=0x7fffffffba28) at eval.c:2902
internal_argbuf = {XIL(0xefb043), XIL(0), XIL(0x7fffffffb8e0), XIL(0x5966ed), XIL(0xdb9a05), XIL(0xdb9a00), XIL(0x7fffffffb900), XIL(0x596715)}
internal_args = 0x7fffffffb8b0
#11 0x0000000000670168 in Ffuncall (nargs=2, args=0x7fffffffba20) at eval.c:2825
fun = XIL(0xdb9a05)
original_fun = XIL(0x5430)
funcar = XIL(0x7fffe58c57db)
numargs = 1
val = XIL(0)
count = 10
#12 0x00000000006c3f74 in exec_byte_code (bytestr=XIL(0x7fffe58c5edc), vector=XIL(0x7fffe58c4cad), maxdepth=make_number(23), args_template=make_number(257), nargs=1, args=0x7fffffffc5a8) at bytecode.c:632
op = 1
type = CONDITION_CASE
targets = {0x6c745f <exec_byte_code+17448>, 0x6c748e <exec_byte_code+17495>, 0x6c7490 <exec_byte_code+17497>, 0x6c7492 <exec_byte_code+17499>, 0x6c7494 <exec_byte_code+17501>, 0x6c7494 <exec_byte_code+17501>,
0x6c7506 <exec_byte_code+17615>, 0x6c758a <exec_byte_code+17747>, 0x6c35d8 <exec_byte_code+1441>, 0x6c35da <exec_byte_code+1443>, 0x6c35dc <exec_byte_code+1445>, 0x6c35de <exec_byte_code+1447>, 0x6c35e0 <exec_byte_code+1449>,
0x6c35e0 <exec_byte_code+1449>, 0x6c35e9 <exec_byte_code+1458>, 0x6c3595 <exec_byte_code+1374>, 0x6c3b15 <exec_byte_code+2782>, 0x6c3b17 <exec_byte_code+2784>, 0x6c3b19 <exec_byte_code+2786>, 0x6c3b1b <exec_byte_code+2788>,
0x6c3b1d <exec_byte_code+2790>, 0x6c3b1d <exec_byte_code+2790>, 0x6c3b67 <exec_byte_code+2864>, 0x6c3b26 <exec_byte_code+2799>, 0x6c3e57 <exec_byte_code+3616>, 0x6c3e59 <exec_byte_code+3618>, 0x6c3e5b <exec_byte_code+3620>,
0x6c3e5d <exec_byte_code+3622>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3df6 <exec_byte_code+3519>, 0x6c3e16 <exec_byte_code+3551>, 0x6c3f32 <exec_byte_code+3835>, 0x6c3f34 <exec_byte_code+3837>,
0x6c3f36 <exec_byte_code+3839>, 0x6c3f38 <exec_byte_code+3841>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3ed1 <exec_byte_code+3738>, 0x6c3ef1 <exec_byte_code+3770>, 0x6c4015 <exec_byte_code+4062>,
0x6c4017 <exec_byte_code+4064>, 0x6c4019 <exec_byte_code+4066>, 0x6c401b <exec_byte_code+4068>, 0x6c401d <exec_byte_code+4070>, 0x6c401d <exec_byte_code+4070>, 0x6c3fb4 <exec_byte_code+3965>, 0x6c3fd4 <exec_byte_code+3997>,
0x6c4999 <exec_byte_code+6498>, 0x6c486c <exec_byte_code+6197>, 0x6c4860 <exec_byte_code+6185>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c4bf8 <exec_byte_code+7105>, 0x6c4d2f <exec_byte_code+7416>, 0x6c4da3 <exec_byte_code+7532>, 0x6c4e18 <exec_byte_code+7649>, 0x6c4e8e <exec_byte_code+7767>,
0x6c3933 <exec_byte_code+2300>, 0x6c39cb <exec_byte_code+2452>, 0x6c4f1f <exec_byte_code+7912>, 0x6c385f <exec_byte_code+2088>, 0x6c3a40 <exec_byte_code+2569>, 0x6c4f9e <exec_byte_code+8039>, 0x6c5013 <exec_byte_code+8156>,
0x6c5062 <exec_byte_code+8235>, 0x6c50d7 <exec_byte_code+8352>, 0x6c5133 <exec_byte_code+8444>, 0x6c5227 <exec_byte_code+8688>, 0x6c5276 <exec_byte_code+8767>, 0x6c52eb <exec_byte_code+8884>, 0x6c5383 <exec_byte_code+9036>,
0x6c53d2 <exec_byte_code+9115>, 0x6c5421 <exec_byte_code+9194>, 0x6c5496 <exec_byte_code+9311>, 0x6c550b <exec_byte_code+9428>, 0x6c5580 <exec_byte_code+9545>, 0x6c5618 <exec_byte_code+9697>, 0x6c5674 <exec_byte_code+9789>,
0x6c56d0 <exec_byte_code+9881>, 0x6c57c4 <exec_byte_code+10125>, 0x6c584e <exec_byte_code+10263>, 0x6c58d8 <exec_byte_code+10401>, 0x6c5aba <exec_byte_code+10883>, 0x6c5b34 <exec_byte_code+11005>,
0x6c5bae <exec_byte_code+11127>, 0x6c5c28 <exec_byte_code+11249>, 0x6c5ca2 <exec_byte_code+11371>, 0x6c5cfe <exec_byte_code+11463>, 0x6c5d8c <exec_byte_code+11605>, 0x6c5de8 <exec_byte_code+11697>,
0x6c5e44 <exec_byte_code+11789>, 0x6c5ea0 <exec_byte_code+11881>, 0x6c5fcd <exec_byte_code+12182>, 0x6c46df <exec_byte_code+5800>, 0x6c6032 <exec_byte_code+12283>, 0x6c6081 <exec_byte_code+12362>,
0x6c616d <exec_byte_code+12598>, 0x6c61dd <exec_byte_code+12710>, 0x6c6242 <exec_byte_code+12811>, 0x6c6291 <exec_byte_code+12890>, 0x6c62de <exec_byte_code+12967>, 0x6c632b <exec_byte_code+13044>,
0x6c6380 <exec_byte_code+13129>, 0x6c745f <exec_byte_code+17448>, 0x6c63df <exec_byte_code+13224>, 0x6c642c <exec_byte_code+13301>, 0x6c6479 <exec_byte_code+13378>, 0x6c64c6 <exec_byte_code+13455>,
0x6c6513 <exec_byte_code+13532>, 0x6c6560 <exec_byte_code+13609>, 0x6c46df <exec_byte_code+5800>, 0x6c745f <exec_byte_code+17448>, 0x6c65af <exec_byte_code+13688>, 0x6c6609 <exec_byte_code+13778>,
0x6c6658 <exec_byte_code+13857>, 0x6c66a7 <exec_byte_code+13936>, 0x6c671c <exec_byte_code+14053>, 0x6c6791 <exec_byte_code+14170>, 0x6c67e0 <exec_byte_code+14249>, 0x6c6923 <exec_byte_code+14572>,
0x6c6998 <exec_byte_code+14689>, 0x6c6a0d <exec_byte_code+14806>, 0x6c6a82 <exec_byte_code+14923>, 0x6c6acf <exec_byte_code+15000>, 0x6c745f <exec_byte_code+17448>, 0x6c45fb <exec_byte_code+5572>,
0x6c40d7 <exec_byte_code+4256>, 0x6c37b4 <exec_byte_code+1917>, 0x6c41bb <exec_byte_code+4484>, 0x6c4258 <exec_byte_code+4641>, 0x6c42f2 <exec_byte_code+4795>, 0x6c45a8 <exec_byte_code+5489>, 0x6c45c0 <exec_byte_code+5513>,
0x6c3d99 <exec_byte_code+3426>, 0x6c469a <exec_byte_code+5731>, 0x6c4717 <exec_byte_code+5856>, 0x6c47a4 <exec_byte_code+5997>, 0x6c47e9 <exec_byte_code+6066>, 0x6c49e6 <exec_byte_code+6575>, 0x6c4a66 <exec_byte_code+6703>,
0x6c4afe <exec_byte_code+6855>, 0x6c4b68 <exec_byte_code+6961>, 0x6c4085 <exec_byte_code+4174>, 0x6c6b1e <exec_byte_code+15079>, 0x6c6bb6 <exec_byte_code+15231>, 0x6c6c05 <exec_byte_code+15310>,
0x6c6c54 <exec_byte_code+15389>, 0x6c6ca3 <exec_byte_code+15468>, 0x6c6cf2 <exec_byte_code+15547>, 0x6c6d67 <exec_byte_code+15664>, 0x6c6ddc <exec_byte_code+15781>, 0x6c6e51 <exec_byte_code+15898>,
0x6c6ec6 <exec_byte_code+16015>, 0x6c7057 <exec_byte_code+16416>, 0x6c70cc <exec_byte_code+16533>, 0x6c7141 <exec_byte_code+16650>, 0x6c7190 <exec_byte_code+16729>, 0x6c7205 <exec_byte_code+16846>,
0x6c727a <exec_byte_code+16963>, 0x6c72c9 <exec_byte_code+17042>, 0x6c7318 <exec_byte_code+17121>, 0x6c5efc <exec_byte_code+11973>, 0x6c5f58 <exec_byte_code+12065>, 0x6c7374 <exec_byte_code+17213>,
0x6c73ea <exec_byte_code+17331>, 0x6c745f <exec_byte_code+17448>, 0x6c438c <exec_byte_code+4949>, 0x6c43b2 <exec_byte_code+4987>, 0x6c4431 <exec_byte_code+5114>, 0x6c44b0 <exec_byte_code+5241>, 0x6c452c <exec_byte_code+5365>,
0x6c518f <exec_byte_code+8536>, 0x6c572c <exec_byte_code+9973>, 0x6c60d5 <exec_byte_code+12446>, 0x6c7639 <exec_byte_code+17922>, 0x6c76b8 <exec_byte_code+18049>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c7764 <exec_byte_code+18221>, 0x6c780a <exec_byte_code+18387>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c7a17 <exec_byte_code+18912> <repeats 64 times>}
const_length = 106
bytestr_length = 1176
vectorp = 0x7fffe58c4cb0
quitcounter = 103 'g'
stack_items = 24
sa_avail = 15016
sa_count = 8
sa_must_free = false
alloc = 0x7fffffffb990
item_bytes = 192
stack_base = 0x7fffffffb990
top = 0x7fffffffba20
stack_lim = 0x7fffffffba50
bytestr_data = 0x7fffffffba50 "\306 \210\b\203\021"
pc = 0x7fffffffbbb8 "\266\005\202d\003\016A睃\345\001\313\350\351\016C\"\003\206\203\001\n\211A\022\242\211\262\r\313\332\036D\322\003\003\003#)\266\203\203\237\001\006\n\327\313O\262\vڲ\001\352\353\006\f!!\262\v\211\203\300\001\314\016E\006\fC\"\026E\006\t\203\341\001\016E\262\n\202\341\001\006\t\203\327\001\006\t\006\v\006\vAB\241\210\006\tA\262\n\202\341\001\006\n\016EB\211\026E\262\n\210\202d\003\016A읃\r\002\353\002\206\366\001\n\211A\022\242!\352\001!\355\001!\203\003\002\211\262\002\356\002\313\332#\266\003\202d\003\016A-\002\353\002\206\036\002\n\211A\022\242!\352\001!\356\001\313ډ$\266\003\202d\003\016", <incomplete sequence \360>...
count = 8
result = XIL(0xffff800000003c71)
#13 0x0000000000670dd8 in funcall_lambda (fun=XIL(0x7fffe58c4c7d), nargs=1, arg_vector=0x7fffffffc5a0) at eval.c:3026
size = 5
val = XIL(0x7fffe58c4c7d)
syms_left = make_number(257)
next = XIL(0x1200efac90)
lexenv = XIL(0x7fffffffc4e0)
count = 8
i = 77324310944
optional = false
rest = false
previous_optional_or_rest = false
#14 0x00000000006701ac in Ffuncall (nargs=2, args=0x7fffffffc598) at eval.c:2827
fun = XIL(0x7fffe58c4c7d)
original_fun = XIL(0x7fffe4a8f2a8)
funcar = XIL(0x7fffffffc580)
numargs = 1
val = XIL(0x7fffe4692d90)
count = 7
#15 0x00000000006c3f74 in exec_byte_code (bytestr=XIL(0x7fffe58c9904), vector=XIL(0x7fffe58c611d), maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7fffffffd2f8) at bytecode.c:632
op = 1
type = CONDITION_CASE
targets = {0x6c745f <exec_byte_code+17448>, 0x6c748e <exec_byte_code+17495>, 0x6c7490 <exec_byte_code+17497>, 0x6c7492 <exec_byte_code+17499>, 0x6c7494 <exec_byte_code+17501>, 0x6c7494 <exec_byte_code+17501>,
0x6c7506 <exec_byte_code+17615>, 0x6c758a <exec_byte_code+17747>, 0x6c35d8 <exec_byte_code+1441>, 0x6c35da <exec_byte_code+1443>, 0x6c35dc <exec_byte_code+1445>, 0x6c35de <exec_byte_code+1447>, 0x6c35e0 <exec_byte_code+1449>,
0x6c35e0 <exec_byte_code+1449>, 0x6c35e9 <exec_byte_code+1458>, 0x6c3595 <exec_byte_code+1374>, 0x6c3b15 <exec_byte_code+2782>, 0x6c3b17 <exec_byte_code+2784>, 0x6c3b19 <exec_byte_code+2786>, 0x6c3b1b <exec_byte_code+2788>,
0x6c3b1d <exec_byte_code+2790>, 0x6c3b1d <exec_byte_code+2790>, 0x6c3b67 <exec_byte_code+2864>, 0x6c3b26 <exec_byte_code+2799>, 0x6c3e57 <exec_byte_code+3616>, 0x6c3e59 <exec_byte_code+3618>, 0x6c3e5b <exec_byte_code+3620>,
0x6c3e5d <exec_byte_code+3622>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3df6 <exec_byte_code+3519>, 0x6c3e16 <exec_byte_code+3551>, 0x6c3f32 <exec_byte_code+3835>, 0x6c3f34 <exec_byte_code+3837>,
0x6c3f36 <exec_byte_code+3839>, 0x6c3f38 <exec_byte_code+3841>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3ed1 <exec_byte_code+3738>, 0x6c3ef1 <exec_byte_code+3770>, 0x6c4015 <exec_byte_code+4062>,
0x6c4017 <exec_byte_code+4064>, 0x6c4019 <exec_byte_code+4066>, 0x6c401b <exec_byte_code+4068>, 0x6c401d <exec_byte_code+4070>, 0x6c401d <exec_byte_code+4070>, 0x6c3fb4 <exec_byte_code+3965>, 0x6c3fd4 <exec_byte_code+3997>,
0x6c4999 <exec_byte_code+6498>, 0x6c486c <exec_byte_code+6197>, 0x6c4860 <exec_byte_code+6185>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c4bf8 <exec_byte_code+7105>, 0x6c4d2f <exec_byte_code+7416>, 0x6c4da3 <exec_byte_code+7532>, 0x6c4e18 <exec_byte_code+7649>, 0x6c4e8e <exec_byte_code+7767>,
0x6c3933 <exec_byte_code+2300>, 0x6c39cb <exec_byte_code+2452>, 0x6c4f1f <exec_byte_code+7912>, 0x6c385f <exec_byte_code+2088>, 0x6c3a40 <exec_byte_code+2569>, 0x6c4f9e <exec_byte_code+8039>, 0x6c5013 <exec_byte_code+8156>,
0x6c5062 <exec_byte_code+8235>, 0x6c50d7 <exec_byte_code+8352>, 0x6c5133 <exec_byte_code+8444>, 0x6c5227 <exec_byte_code+8688>, 0x6c5276 <exec_byte_code+8767>, 0x6c52eb <exec_byte_code+8884>, 0x6c5383 <exec_byte_code+9036>,
0x6c53d2 <exec_byte_code+9115>, 0x6c5421 <exec_byte_code+9194>, 0x6c5496 <exec_byte_code+9311>, 0x6c550b <exec_byte_code+9428>, 0x6c5580 <exec_byte_code+9545>, 0x6c5618 <exec_byte_code+9697>, 0x6c5674 <exec_byte_code+9789>,
0x6c56d0 <exec_byte_code+9881>, 0x6c57c4 <exec_byte_code+10125>, 0x6c584e <exec_byte_code+10263>, 0x6c58d8 <exec_byte_code+10401>, 0x6c5aba <exec_byte_code+10883>, 0x6c5b34 <exec_byte_code+11005>,
0x6c5bae <exec_byte_code+11127>, 0x6c5c28 <exec_byte_code+11249>, 0x6c5ca2 <exec_byte_code+11371>, 0x6c5cfe <exec_byte_code+11463>, 0x6c5d8c <exec_byte_code+11605>, 0x6c5de8 <exec_byte_code+11697>,
0x6c5e44 <exec_byte_code+11789>, 0x6c5ea0 <exec_byte_code+11881>, 0x6c5fcd <exec_byte_code+12182>, 0x6c46df <exec_byte_code+5800>, 0x6c6032 <exec_byte_code+12283>, 0x6c6081 <exec_byte_code+12362>,
0x6c616d <exec_byte_code+12598>, 0x6c61dd <exec_byte_code+12710>, 0x6c6242 <exec_byte_code+12811>, 0x6c6291 <exec_byte_code+12890>, 0x6c62de <exec_byte_code+12967>, 0x6c632b <exec_byte_code+13044>,
0x6c6380 <exec_byte_code+13129>, 0x6c745f <exec_byte_code+17448>, 0x6c63df <exec_byte_code+13224>, 0x6c642c <exec_byte_code+13301>, 0x6c6479 <exec_byte_code+13378>, 0x6c64c6 <exec_byte_code+13455>,
0x6c6513 <exec_byte_code+13532>, 0x6c6560 <exec_byte_code+13609>, 0x6c46df <exec_byte_code+5800>, 0x6c745f <exec_byte_code+17448>, 0x6c65af <exec_byte_code+13688>, 0x6c6609 <exec_byte_code+13778>,
0x6c6658 <exec_byte_code+13857>, 0x6c66a7 <exec_byte_code+13936>, 0x6c671c <exec_byte_code+14053>, 0x6c6791 <exec_byte_code+14170>, 0x6c67e0 <exec_byte_code+14249>, 0x6c6923 <exec_byte_code+14572>,
0x6c6998 <exec_byte_code+14689>, 0x6c6a0d <exec_byte_code+14806>, 0x6c6a82 <exec_byte_code+14923>, 0x6c6acf <exec_byte_code+15000>, 0x6c745f <exec_byte_code+17448>, 0x6c45fb <exec_byte_code+5572>,
0x6c40d7 <exec_byte_code+4256>, 0x6c37b4 <exec_byte_code+1917>, 0x6c41bb <exec_byte_code+4484>, 0x6c4258 <exec_byte_code+4641>, 0x6c42f2 <exec_byte_code+4795>, 0x6c45a8 <exec_byte_code+5489>, 0x6c45c0 <exec_byte_code+5513>,
0x6c3d99 <exec_byte_code+3426>, 0x6c469a <exec_byte_code+5731>, 0x6c4717 <exec_byte_code+5856>, 0x6c47a4 <exec_byte_code+5997>, 0x6c47e9 <exec_byte_code+6066>, 0x6c49e6 <exec_byte_code+6575>, 0x6c4a66 <exec_byte_code+6703>,
0x6c4afe <exec_byte_code+6855>, 0x6c4b68 <exec_byte_code+6961>, 0x6c4085 <exec_byte_code+4174>, 0x6c6b1e <exec_byte_code+15079>, 0x6c6bb6 <exec_byte_code+15231>, 0x6c6c05 <exec_byte_code+15310>,
0x6c6c54 <exec_byte_code+15389>, 0x6c6ca3 <exec_byte_code+15468>, 0x6c6cf2 <exec_byte_code+15547>, 0x6c6d67 <exec_byte_code+15664>, 0x6c6ddc <exec_byte_code+15781>, 0x6c6e51 <exec_byte_code+15898>,
0x6c6ec6 <exec_byte_code+16015>, 0x6c7057 <exec_byte_code+16416>, 0x6c70cc <exec_byte_code+16533>, 0x6c7141 <exec_byte_code+16650>, 0x6c7190 <exec_byte_code+16729>, 0x6c7205 <exec_byte_code+16846>,
0x6c727a <exec_byte_code+16963>, 0x6c72c9 <exec_byte_code+17042>, 0x6c7318 <exec_byte_code+17121>, 0x6c5efc <exec_byte_code+11973>, 0x6c5f58 <exec_byte_code+12065>, 0x6c7374 <exec_byte_code+17213>,
0x6c73ea <exec_byte_code+17331>, 0x6c745f <exec_byte_code+17448>, 0x6c438c <exec_byte_code+4949>, 0x6c43b2 <exec_byte_code+4987>, 0x6c4431 <exec_byte_code+5114>, 0x6c44b0 <exec_byte_code+5241>, 0x6c452c <exec_byte_code+5365>,
0x6c518f <exec_byte_code+8536>, 0x6c572c <exec_byte_code+9973>, 0x6c60d5 <exec_byte_code+12446>, 0x6c7639 <exec_byte_code+17922>, 0x6c76b8 <exec_byte_code+18049>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c7764 <exec_byte_code+18221>, 0x6c780a <exec_byte_code+18387>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c7a17 <exec_byte_code+18912> <repeats 64 times>}
const_length = 207
bytestr_length = 1604
vectorp = 0x7fffe58c6120
quitcounter = 5 '\005'
stack_items = 13
sa_avail = 14676
sa_count = 7
sa_must_free = false
alloc = 0x7fffffffc590
item_bytes = 104
stack_base = 0x7fffffffc590
top = 0x7fffffffc598
stack_lim = 0x7fffffffc5f8
bytestr_data = 0x7fffffffc5f8 "\306 \020\307\021\n\023\307\024\310\311!\211\307=\204\060"
pc = 0x7fffffffcb8b "\210\307\016@\211\203\335\005\211@\002\204\326\005\211;\203\326\005\201", <incomplete sequence \304>
count = 7
result = XIL(0x7fffffffd0e0)
#16 0x0000000000670dd8 in funcall_lambda (fun=XIL(0x7fffe58c60ed), nargs=0, arg_vector=0x7fffffffd2f8) at eval.c:3026
size = 5
val = XIL(0x7fffe58c60ed)
syms_left = make_number(0)
next = XIL(0x1200db5dc5)
lexenv = XIL(0x7fffffffd230)
count = 7
i = 77324310944
optional = false
rest = false
previous_optional_or_rest = false
#17 0x00000000006701ac in Ffuncall (nargs=1, args=0x7fffffffd2f0) at eval.c:2827
fun = XIL(0x7fffe58c60ed)
original_fun = XIL(0x7fffe4a90718)
funcar = XIL(0xec6215)
numargs = 0
val = XIL(0xec6215)
count = 6
#18 0x00000000006c3f74 in exec_byte_code (bytestr=XIL(0x7fffe58ca554), vector=XIL(0x7fffe58c9ad5), maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7fffffffdb30) at bytecode.c:632
op = 0
type = CATCHER
targets = {0x6c745f <exec_byte_code+17448>, 0x6c748e <exec_byte_code+17495>, 0x6c7490 <exec_byte_code+17497>, 0x6c7492 <exec_byte_code+17499>, 0x6c7494 <exec_byte_code+17501>, 0x6c7494 <exec_byte_code+17501>,
0x6c7506 <exec_byte_code+17615>, 0x6c758a <exec_byte_code+17747>, 0x6c35d8 <exec_byte_code+1441>, 0x6c35da <exec_byte_code+1443>, 0x6c35dc <exec_byte_code+1445>, 0x6c35de <exec_byte_code+1447>, 0x6c35e0 <exec_byte_code+1449>,
0x6c35e0 <exec_byte_code+1449>, 0x6c35e9 <exec_byte_code+1458>, 0x6c3595 <exec_byte_code+1374>, 0x6c3b15 <exec_byte_code+2782>, 0x6c3b17 <exec_byte_code+2784>, 0x6c3b19 <exec_byte_code+2786>, 0x6c3b1b <exec_byte_code+2788>,
0x6c3b1d <exec_byte_code+2790>, 0x6c3b1d <exec_byte_code+2790>, 0x6c3b67 <exec_byte_code+2864>, 0x6c3b26 <exec_byte_code+2799>, 0x6c3e57 <exec_byte_code+3616>, 0x6c3e59 <exec_byte_code+3618>, 0x6c3e5b <exec_byte_code+3620>,
0x6c3e5d <exec_byte_code+3622>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3df6 <exec_byte_code+3519>, 0x6c3e16 <exec_byte_code+3551>, 0x6c3f32 <exec_byte_code+3835>, 0x6c3f34 <exec_byte_code+3837>,
0x6c3f36 <exec_byte_code+3839>, 0x6c3f38 <exec_byte_code+3841>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3ed1 <exec_byte_code+3738>, 0x6c3ef1 <exec_byte_code+3770>, 0x6c4015 <exec_byte_code+4062>,
0x6c4017 <exec_byte_code+4064>, 0x6c4019 <exec_byte_code+4066>, 0x6c401b <exec_byte_code+4068>, 0x6c401d <exec_byte_code+4070>, 0x6c401d <exec_byte_code+4070>, 0x6c3fb4 <exec_byte_code+3965>, 0x6c3fd4 <exec_byte_code+3997>,
0x6c4999 <exec_byte_code+6498>, 0x6c486c <exec_byte_code+6197>, 0x6c4860 <exec_byte_code+6185>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c4bf8 <exec_byte_code+7105>, 0x6c4d2f <exec_byte_code+7416>, 0x6c4da3 <exec_byte_code+7532>, 0x6c4e18 <exec_byte_code+7649>, 0x6c4e8e <exec_byte_code+7767>,
0x6c3933 <exec_byte_code+2300>, 0x6c39cb <exec_byte_code+2452>, 0x6c4f1f <exec_byte_code+7912>, 0x6c385f <exec_byte_code+2088>, 0x6c3a40 <exec_byte_code+2569>, 0x6c4f9e <exec_byte_code+8039>, 0x6c5013 <exec_byte_code+8156>,
0x6c5062 <exec_byte_code+8235>, 0x6c50d7 <exec_byte_code+8352>, 0x6c5133 <exec_byte_code+8444>, 0x6c5227 <exec_byte_code+8688>, 0x6c5276 <exec_byte_code+8767>, 0x6c52eb <exec_byte_code+8884>, 0x6c5383 <exec_byte_code+9036>,
0x6c53d2 <exec_byte_code+9115>, 0x6c5421 <exec_byte_code+9194>, 0x6c5496 <exec_byte_code+9311>, 0x6c550b <exec_byte_code+9428>, 0x6c5580 <exec_byte_code+9545>, 0x6c5618 <exec_byte_code+9697>, 0x6c5674 <exec_byte_code+9789>,
0x6c56d0 <exec_byte_code+9881>, 0x6c57c4 <exec_byte_code+10125>, 0x6c584e <exec_byte_code+10263>, 0x6c58d8 <exec_byte_code+10401>, 0x6c5aba <exec_byte_code+10883>, 0x6c5b34 <exec_byte_code+11005>,
0x6c5bae <exec_byte_code+11127>, 0x6c5c28 <exec_byte_code+11249>, 0x6c5ca2 <exec_byte_code+11371>, 0x6c5cfe <exec_byte_code+11463>, 0x6c5d8c <exec_byte_code+11605>, 0x6c5de8 <exec_byte_code+11697>,
0x6c5e44 <exec_byte_code+11789>, 0x6c5ea0 <exec_byte_code+11881>, 0x6c5fcd <exec_byte_code+12182>, 0x6c46df <exec_byte_code+5800>, 0x6c6032 <exec_byte_code+12283>, 0x6c6081 <exec_byte_code+12362>,
0x6c616d <exec_byte_code+12598>, 0x6c61dd <exec_byte_code+12710>, 0x6c6242 <exec_byte_code+12811>, 0x6c6291 <exec_byte_code+12890>, 0x6c62de <exec_byte_code+12967>, 0x6c632b <exec_byte_code+13044>,
0x6c6380 <exec_byte_code+13129>, 0x6c745f <exec_byte_code+17448>, 0x6c63df <exec_byte_code+13224>, 0x6c642c <exec_byte_code+13301>, 0x6c6479 <exec_byte_code+13378>, 0x6c64c6 <exec_byte_code+13455>,
0x6c6513 <exec_byte_code+13532>, 0x6c6560 <exec_byte_code+13609>, 0x6c46df <exec_byte_code+5800>, 0x6c745f <exec_byte_code+17448>, 0x6c65af <exec_byte_code+13688>, 0x6c6609 <exec_byte_code+13778>,
0x6c6658 <exec_byte_code+13857>, 0x6c66a7 <exec_byte_code+13936>, 0x6c671c <exec_byte_code+14053>, 0x6c6791 <exec_byte_code+14170>, 0x6c67e0 <exec_byte_code+14249>, 0x6c6923 <exec_byte_code+14572>,
0x6c6998 <exec_byte_code+14689>, 0x6c6a0d <exec_byte_code+14806>, 0x6c6a82 <exec_byte_code+14923>, 0x6c6acf <exec_byte_code+15000>, 0x6c745f <exec_byte_code+17448>, 0x6c45fb <exec_byte_code+5572>,
0x6c40d7 <exec_byte_code+4256>, 0x6c37b4 <exec_byte_code+1917>, 0x6c41bb <exec_byte_code+4484>, 0x6c4258 <exec_byte_code+4641>, 0x6c42f2 <exec_byte_code+4795>, 0x6c45a8 <exec_byte_code+5489>, 0x6c45c0 <exec_byte_code+5513>,
0x6c3d99 <exec_byte_code+3426>, 0x6c469a <exec_byte_code+5731>, 0x6c4717 <exec_byte_code+5856>, 0x6c47a4 <exec_byte_code+5997>, 0x6c47e9 <exec_byte_code+6066>, 0x6c49e6 <exec_byte_code+6575>, 0x6c4a66 <exec_byte_code+6703>,
0x6c4afe <exec_byte_code+6855>, 0x6c4b68 <exec_byte_code+6961>, 0x6c4085 <exec_byte_code+4174>, 0x6c6b1e <exec_byte_code+15079>, 0x6c6bb6 <exec_byte_code+15231>, 0x6c6c05 <exec_byte_code+15310>,
0x6c6c54 <exec_byte_code+15389>, 0x6c6ca3 <exec_byte_code+15468>, 0x6c6cf2 <exec_byte_code+15547>, 0x6c6d67 <exec_byte_code+15664>, 0x6c6ddc <exec_byte_code+15781>, 0x6c6e51 <exec_byte_code+15898>,
0x6c6ec6 <exec_byte_code+16015>, 0x6c7057 <exec_byte_code+16416>, 0x6c70cc <exec_byte_code+16533>, 0x6c7141 <exec_byte_code+16650>, 0x6c7190 <exec_byte_code+16729>, 0x6c7205 <exec_byte_code+16846>,
0x6c727a <exec_byte_code+16963>, 0x6c72c9 <exec_byte_code+17042>, 0x6c7318 <exec_byte_code+17121>, 0x6c5efc <exec_byte_code+11973>, 0x6c5f58 <exec_byte_code+12065>, 0x6c7374 <exec_byte_code+17213>,
0x6c73ea <exec_byte_code+17331>, 0x6c745f <exec_byte_code+17448>, 0x6c438c <exec_byte_code+4949>, 0x6c43b2 <exec_byte_code+4987>, 0x6c4431 <exec_byte_code+5114>, 0x6c44b0 <exec_byte_code+5241>, 0x6c452c <exec_byte_code+5365>,
0x6c518f <exec_byte_code+8536>, 0x6c572c <exec_byte_code+9973>, 0x6c60d5 <exec_byte_code+12446>, 0x6c7639 <exec_byte_code+17922>, 0x6c76b8 <exec_byte_code+18049>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c7764 <exec_byte_code+18221>, 0x6c780a <exec_byte_code+18387>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>,
0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c7a17 <exec_byte_code+18912> <repeats 64 times>}
const_length = 77
bytestr_length = 443
vectorp = 0x7fffe58c9ad8
quitcounter = 41 ')'
stack_items = 13
sa_avail = 15837
sa_count = 5
sa_must_free = false
alloc = 0x7fffffffd2e0
item_bytes = 104
stack_base = 0x7fffffffd2e0
top = 0x7fffffffd2f0
stack_lim = 0x7fffffffd348
bytestr_data = 0x7fffffffd348 "\b\203\b"
pc = 0x7fffffffd4bc "\210)\210\375\376\377\"\210\201H"
count = 5
result = XIL(0x7ffff56555e0)
#19 0x0000000000670dd8 in funcall_lambda (fun=XIL(0x7fffe58c9aa5), nargs=0, arg_vector=0x7fffffffdb30) at eval.c:3026
size = 5
val = XIL(0x7fffffffdb30)
syms_left = make_number(0)
next = XIL(0x7fffffffdae0)
lexenv = XIL(0)
count = 5
i = 6770530
optional = false
rest = false
previous_optional_or_rest = false
#20 0x0000000000670a00 in apply_lambda (fun=XIL(0x7fffe58c9aa5), args=XIL(0), count=4) at eval.c:2962
args_left = XIL(0)
i = 0
numargs = 0
arg_vector = 0x7fffffffdb30
tem = XIL(0)
sa_avail = 16384
sa_count = 5
sa_must_free = false
#21 0x000000000066e98b in eval_sub (form=XIL(0x7fffe5a1b57b)) at eval.c:2335
fun = XIL(0x7fffe58c9aa5)
val = XIL(0x7fffe53c0500)
original_fun = XIL(0x7fffe4a940d0)
original_args = XIL(0)
funcar = XIL(0x7fffffffdc40)
count = 4
argvals = {XIL(0x597785), XIL(0xeb5930), XIL(0x7fffffffdd00), XIL(0x67208b), XIL(0), XIL(0xe3d470), XIL(0x7ad0), XIL(0x7ad0)}
#22 0x000000000066dc7b in Feval (form=XIL(0x7fffe5a1b57b), lexical=XIL(0)) at eval.c:2110
count = 3
#23 0x00000000005a0bcc in top_level_2 () at keyboard.c:1122
No locals.
#24 0x000000000066bc68 in internal_condition_case (bfun=0x5a0baf <top_level_2>, handlers=XIL(0x5310), hfun=0x5a059c <cmd_error>) at eval.c:1336
val = XIL(0xed25a8)
c = 0xed2690
#25 0x00000000005a0c10 in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
No locals.
#26 0x000000000066b107 in internal_catch (tag=XIL(0xc7e0), func=0x5a0bce <top_level_1>, arg=XIL(0)) at eval.c:1101
val = XIL(0x7fffffffde30)
c = 0xed2570
#27 0x00000000005a0b01 in command_loop () at keyboard.c:1091
No locals.
#28 0x00000000005a0096 in recursive_edit_1 () at keyboard.c:698
count = 1
val = XIL(0x7fffffffde90)
#29 0x00000000005a0284 in Frecursive_edit () at keyboard.c:769
count = 0
buffer = XIL(0)
#30 0x000000000059e029 in main (argc=4, argv=0x7fffffffe0e8) at emacs.c:1929
stack_bottom_variable = 0x0
do_initial_setlocale = true
skip_args = 1
no_loadup = false
junk = 0x0
dname_arg = 0x0
ch_to_dir = 0x0
original_pwd = 0x0
argv0_base = 0x7fffffffe4fe "emacs"
is_temacs = false
loaded_dump = 0x0
dump_mode = 0x0
disable_aslr = false
rlim = {
rlim_cur = 10022912,
rlim_max = 18446744073709551615
}
sockfd = -1
Lisp Backtrace:
"dump-emacs-portable" (0xffffb7b0)
"eval" (0xffffba28)
"command-line-1" (0xffffc5a0)
"command-line" (0xffffd2f8)
"normal-top-level" (0xffffdb30)
(gdb)
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-12 20:18 ` Daniel Colascione
` (2 preceding siblings ...)
2018-02-15 18:34 ` andres.ramirez
@ 2018-02-19 22:01 ` Daniele Nicolodi
2018-02-20 0:28 ` Daniel Colascione
3 siblings, 1 reply; 299+ messages in thread
From: Daniele Nicolodi @ 2018-02-19 22:01 UTC (permalink / raw)
To: Daniel Colascione, Emacs developers
[-- Attachment #1: Type: text/plain, Size: 405 bytes --]
On 12/02/2018 13:18, Daniel Colascione wrote:
> I've pushed the portable dumper to the pdumper Savannah branch.
Hi Daniel,
the recently introduced `dmpstruct.awk` script fails with a non-GNU awk:
POSIX awk does not like a | occurring just after a grouping ( or just
before a ) in a regular expression. The attached patch fixes the syntax
preserving the meaning of the regular expressions.
Cheers,
Dan
[-- Attachment #2: dmpstruct.patch --]
[-- Type: text/plain, Size: 509 bytes --]
diff --git a/src/dmpstruct.awk b/src/dmpstruct.awk
index 608c26fcc5..0a22030405 100755
--- a/src/dmpstruct.awk
+++ b/src/dmpstruct.awk
@@ -10,10 +10,10 @@ BEGIN {
struct_name = $2
close (tmpfile)
}
-/^(enum|struct|union) [a-zA-Z0-9_]+([\t ]|\/\*.*\*\/)*$/, /^(| )};$/ {
+/^(enum|struct|union) [a-zA-Z0-9_]+([\t ]|\/\*.*\*\/)*$/, /^( )?};$/ {
print $0 > tmpfile
}
-/^(| )};$/ {
+/^( )?};$/ {
if (struct_name != "") {
fflush (tmpfile)
cmd = "../lib-src/make-fingerprint -r " tmpfile
^ permalink raw reply related [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-19 17:03 ` Daniel Colascione
@ 2018-02-19 22:33 ` Ken Brown
2018-02-20 16:32 ` Ken Brown
2018-02-20 1:16 ` Andy Moreton
1 sibling, 1 reply; 299+ messages in thread
From: Ken Brown @ 2018-02-19 22:33 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel
On 2/19/2018 12:03 PM, Daniel Colascione wrote:
> On 02/19/2018 05:30 AM, Ken Brown wrote:
>> On 2/17/2018 7:02 PM, Daniel Colascione wrote:
>>> On Feb 17, 2018 3:38 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>>
>>> On 2/15/2018 9:36 PM, Daniel Colascione wrote:
>>> > On 02/15/2018 05:56 PM, Ken Brown wrote:
>>> >> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>> >>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>> [...]
>>> >>> I just tried to build on 64-bit Cygwin, and the build
>>> fails as
>>> >>> follows:
>>> >>>
>>> >>> [...]
>>> >>> Dumping under the name bootstrap-emacs.pdmp
>>> >>> dumping fingerprint:
>>> >>>
>>> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>> >>> Dump complete
>>> >>> Byte counts: header=80 hot=13187392 discardable=119424
>>> cold=9086640
>>> >>> Reloc counts: hot=919268 discardable=5790
>>> >>> make -C ../lisp compile-first
>>> EMACS="../src/bootstrap-emacs.exe"
>>> >>> make[2]: Entering directory
>>> >>> '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>> >>> ELC ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>> >>> ELC ../../pdumper/lisp/emacs-lisp/cconv.elc
>>> >>> ELC ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>> >>> ELC ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>> >>> emacs: could not load dump file
>>> "../src/bootstrap-emacs.pdmp": out
>>> >>> of memory
>>> >>>
>>> >>> There's probably some obvious explanation, but I don't see
>>> it at the
>>> >>> moment.
>>> >>>
>>> >>>
>>> >>> I'm not entirely surprised to see Cygwin fall over here. We
>>> could
>>> >>> just use the Windows memory mapping functions directly, but I'd
>>> >>> prefer to stick with the POSIX API if we can make it work.
>>> >>
>>> >> I agree.
>>> >>
>>> >>> Any idea where in the mmap sequence we fail?
>>> >>
>>> >> I haven't looked at the code yet, so I don't understand the
>>> question.
>>> >> If you give me some guidance, I'll try to investigate.
>>> >
>>> > Thanks. I think the trouble must be in dump_mmap_contiguous. We
>>> report
>>> > all errors from this function as "out of memory".
>>> dump_mmap_contiguous
>>> > takes an array of mapping descriptors (think ELF segments or
>>> something)
>>> > and maps them all into a single contiguous region of virtual
>>> memory. On
>>> > POSIX systems, we reserve a chunk of address space with a big
>>> PROT_NONE
>>> > anonymous mapping, then carve it up into the separate mappings
>>> that we
>>> > really want. Windows doesn't support atomic mmap replacement, so
>>> Cygwin
>>> > has to emulate it, and I wouldn't be surprised if something's
>>> going
>>> > wrong in this emulation.
>>>
>>> You're right that the problem is in dump_mmap_contiguous. I stepped
>>> through it in gdb and found the following:
>>>
>>> 1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
>>>
>>> 2. We enter the loop at pdumper.c:4451 with i = 0. dump_map_file is
>>> called, which calls dump_map_file_posix with protection =
>>> DUMP_MEMORY_ACCESS_READWRITE.
>>>
>>> 3. This calls mmap at line 4219 with mem_prot = PROT_READ |
>>> PROT_WRITE
>>> and mem_flags = MAP_PRIVATE | MAP_FIXED.
>>>
>>> 4. This mmap call fails.
>>>
>>> I don't know if this is a bug in Cygwin's mmap or simply a
>>> limitation
>>> that has to be worked around. Should I take this to the Cygwin
>>> mailing
>>> list, or do you have other ideas as to how to proceed?
>>>
>>> I'd definitely let the Cygwin people know. If the problem is that
>>> Cygwin doesn't properly support atomic map replacement, I think we
>>> can work around the problem by using the same unmap-before-remap
>>> approach we use for the native Windows code, but keep using the POSIX
>>> memory functions. I can come up with a patch blind, but you'd be in a
>>> better position to iterate quickly.
>>
>> OK, here's Corinna's answer:
>>
>> https://cygwin.com/ml/cygwin/2018-02/msg00202.html
>>
>> It seems like she's suggesting the same unmap-before-remap approach
>> that you mentioned.
>
> Thanks. I've pushed a fix, I think. Try it now.
That fixes it.
Thanks.
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-19 22:01 ` Daniele Nicolodi
@ 2018-02-20 0:28 ` Daniel Colascione
0 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2018-02-20 0:28 UTC (permalink / raw)
To: Daniele Nicolodi, Emacs developers
On 02/19/2018 02:01 PM, Daniele Nicolodi wrote:
> On 12/02/2018 13:18, Daniel Colascione wrote:
>> I've pushed the portable dumper to the pdumper Savannah branch.
>
> Hi Daniel,
>
> the recently introduced `dmpstruct.awk` script fails with a non-GNU awk:
> POSIX awk does not like a | occurring just after a grouping ( or just
> before a ) in a regular expression. The attached patch fixes the syntax
> preserving the meaning of the regular expressions.
Thanks. Applied.
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-13 16:37 ` Eli Zaretskii
` (3 preceding siblings ...)
2018-02-17 11:53 ` Charles A. Roelli
@ 2018-02-20 0:54 ` Andy Moreton
4 siblings, 0 replies; 299+ messages in thread
From: Andy Moreton @ 2018-02-20 0:54 UTC (permalink / raw)
To: emacs-devel
On Tue 13 Feb 2018, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 12 Feb 2018 12:18:36 -0800
>>
>> I've pushed the portable dumper to the pdumper Savannah branch. It
>> should support a fully PIC Emacs.
>
> Thanks. I'd urge people to try this branch and report any issues they
> see.
The branch currently fails for out of tree builds (tested with mingw64
on msys2 for 64bit Windows). Something like this is needed to fix it:
diff --git a/src/Makefile.in b/src/Makefile.in
index e2950a02eb..0df59b90a8 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -411,11 +411,13 @@ base_obj =
$(W32_OBJ) $(WINDOW_SYSTEM_OBJ) $(XGSELOBJ) $(JSON_OBJ)
obj = $(base_obj) $(NS_OBJC_OBJ)
-dmpstruct_headers=lisp.h buffer.h intervals.h charset.h
+dmpstruct_headers = ${srcdir}/lisp.h ${srcdir}/buffer.h ${srcdir}/intervals.h \
+ ${srcdir}/charset.h
pdumper.o: dmpstruct.h
-dmpstruct.h: dmpstruct.awk
+dmpstruct.h: ${srcdir}/dmpstruct.awk
dmpstruct.h: $(libsrc)/make-fingerprint$(EXEEXT) $(dmpstruct_headers)
- POSIXLY_CORRECT=1 awk -f dmpstruct.awk $(dmpstruct_headers) > $@
+ POSIXLY_CORRECT=1 awk -f ${srcdir}/dmpstruct.awk \
+ $(dmpstruct_headers) > $@
## Object files used on some machine or other.
## These go in the DOC file on all machines in case they are needed.
^ permalink raw reply related [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-19 17:03 ` Daniel Colascione
2018-02-19 22:33 ` Ken Brown
@ 2018-02-20 1:16 ` Andy Moreton
1 sibling, 0 replies; 299+ messages in thread
From: Andy Moreton @ 2018-02-20 1:16 UTC (permalink / raw)
To: emacs-devel
On Mon 19 Feb 2018, Daniel Colascione wrote:
>
> Thanks. I've pushed a fix, I think. Try it now.
I'm sure Ken will soon report some more rigorous testing, but I have
successfully built a Cygwin w32 build of emacs, which runs gnus well
enough to send this. I haven't tried the X11 build.
AndyM
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-19 22:33 ` Ken Brown
@ 2018-02-20 16:32 ` Ken Brown
2018-02-20 17:23 ` Daniel Colascione
0 siblings, 1 reply; 299+ messages in thread
From: Ken Brown @ 2018-02-20 16:32 UTC (permalink / raw)
To: Daniel Colascione; +Cc: Eli Zaretskii, emacs-devel
On 2/19/2018 5:33 PM, Ken Brown wrote:
> On 2/19/2018 12:03 PM, Daniel Colascione wrote:
>> Thanks. I've pushed a fix, I think. Try it now.
>
> That fixes it.
But then the following commit broke the build:
commit fca8372020248318f1d0007378607a912b19026e
Author: Daniel Colascione <dancol@dancol.org>
Date: Mon Feb 19 16:35:09 2018 -0800
eassert -> eassume in pdumper
More precisely, it broke a "standard" build (./configure && make). The
build fails as follows:
[...]
Finding pointers to doc strings...done
Dumping under the name bootstrap-emacs.pdmp
dumping fingerprint:
d3d1723e85f76355cab6880175f65823cbad50b923958d6dcb4b5e78a766a04e
make[1]: *** [Makefile:779: bootstrap-emacs.exe] Segmentation fault
(core dumped)
If I configure with --enable-checking, or if I build without
optimization, then the build succeeds again.
This is on 64-bit Cygwin. I haven't tried any other platform.
Ken
^ permalink raw reply [flat|nested] 299+ messages in thread
* Re: Preview: portable dumper
2018-02-20 16:32 ` Ken Brown
@ 2018-02-20 17:23 ` Daniel Colascione
0 siblings, 0 replies; 299+ messages in thread
From: Daniel Colascione @ 2018-02-20 17:23 UTC (permalink / raw)
To: Ken Brown; +Cc: Eli Zaretskii, emacs-devel
On 02/20/2018 08:32 AM, Ken Brown wrote:
> On 2/19/2018 5:33 PM, Ken Brown wrote:
>> On 2/19/2018 12:03 PM, Daniel Colascione wrote:
>>> Thanks. I've pushed a fix, I think. Try it now.
>>
>> That fixes it.
>
> But then the following commit broke the build:
>
> commit fca8372020248318f1d0007378607a912b19026e
> Author: Daniel Colascione <dancol@dancol.org>
> Date: Mon Feb 19 16:35:09 2018 -0800
>
> eassert -> eassume in pdumper
>
> More precisely, it broke a "standard" build (./configure && make). The
> build fails as follows:
Reverted.
^ permalink raw reply [flat|nested] 299+ messages in thread
end of thread, other threads:[~2018-02-20 17:23 UTC | newest]
Thread overview: 299+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-28 19:50 Preview: portable dumper Daniel Colascione
2016-11-28 19:58 ` Burton Samograd
2016-11-28 20:11 ` Daniel Colascione
2016-11-28 20:12 ` Eli Zaretskii
2016-11-28 20:14 ` Daniel Colascione
2016-11-28 20:16 ` Daniel Colascione
2016-11-28 20:29 ` Eli Zaretskii
2016-11-28 20:20 ` John Wiegley
2016-11-28 20:22 ` Daniel Colascione
2016-11-28 20:26 ` John Wiegley
2016-11-28 20:31 ` Daniel Colascione
2016-11-28 20:37 ` Burton Samograd
2016-11-28 20:44 ` Daniel Colascione
2016-11-29 16:02 ` Ted Zlatanov
2016-11-29 17:58 ` Daniel Colascione
2016-11-29 16:48 ` Richard Stallman
2016-11-29 17:32 ` Daniel Colascione
2016-11-29 19:55 ` Philippe Vaucher
2016-11-29 17:43 ` Eli Zaretskii
2016-11-29 17:49 ` Daniel Colascione
2016-11-29 18:17 ` Eli Zaretskii
2016-11-29 18:03 ` John Wiegley
2016-11-29 18:23 ` Eli Zaretskii
2016-11-29 18:49 ` Daniel Colascione
2016-11-29 19:02 ` Eli Zaretskii
2016-12-01 9:18 ` Richard Stallman
2016-12-01 18:11 ` Eli Zaretskii
2016-12-02 4:28 ` Ken Raeburn
2016-12-02 4:41 ` Daniel Colascione
2016-12-02 8:08 ` Eli Zaretskii
2016-12-02 8:03 ` Eli Zaretskii
2016-12-02 17:24 ` Ken Raeburn
2016-11-28 20:39 ` John Wiegley
2016-11-28 20:34 ` Burton Samograd
2016-11-28 20:31 ` Eli Zaretskii
2016-11-28 20:21 ` Paul Eggert
2016-11-28 20:34 ` Eli Zaretskii
2016-11-28 20:47 ` John Wiegley
2016-11-28 21:14 ` Eli Zaretskii
2016-11-28 21:55 ` Daniel Colascione
2016-11-28 22:18 ` John Wiegley
2016-11-29 18:40 ` Eli Zaretskii
2016-11-29 19:11 ` John Wiegley
2016-11-29 20:07 ` Eli Zaretskii
2016-11-29 20:29 ` John Wiegley
2016-11-29 20:36 ` Daniel Colascione
2016-11-29 21:30 ` John Wiegley
2016-11-30 8:26 ` Philippe Vaucher
2016-11-29 19:12 ` Daniel Colascione
2016-11-29 16:55 ` Richard Stallman
2016-11-29 18:39 ` Eli Zaretskii
2016-11-29 19:03 ` Daniel Colascione
2016-11-29 19:59 ` Eli Zaretskii
2016-11-29 20:28 ` John Wiegley
2016-11-29 19:13 ` Paul Eggert
2016-11-29 19:35 ` Eli Zaretskii
2016-11-29 20:54 ` Paul Eggert
2016-11-30 16:38 ` Eli Zaretskii
2016-11-30 18:57 ` John Wiegley
2016-11-30 19:14 ` Daniel Colascione
2016-11-30 21:03 ` John Wiegley
2016-11-30 21:06 ` Paul Eggert
2016-11-30 21:44 ` John Wiegley
2016-12-01 3:32 ` Eli Zaretskii
2016-12-01 9:16 ` Paul Eggert
2016-12-01 17:26 ` Eli Zaretskii
2016-12-01 17:35 ` Daniel Colascione
2016-12-01 17:58 ` Paul Eggert
2016-11-30 21:35 ` Daniel Colascione
2016-11-30 21:44 ` John Wiegley
2016-11-30 21:50 ` Daniel Colascione
2016-11-30 22:20 ` John Wiegley
2016-12-01 1:37 ` Paul Eggert
2016-12-01 1:45 ` Daniel Colascione
2016-12-01 3:47 ` Eli Zaretskii
2016-12-01 4:10 ` John Wiegley
2016-12-01 4:12 ` Daniel Colascione
2016-12-01 4:49 ` John Wiegley
2016-12-01 5:12 ` Daniel Colascione
2016-12-01 9:03 ` Matt Armstrong
2016-12-02 8:10 ` John Wiegley
2016-12-01 9:18 ` Phillip Lord
2016-12-01 4:10 ` Daniel Colascione
2016-12-01 3:41 ` Eli Zaretskii
2016-11-30 19:29 ` Philippe Vaucher
2016-11-30 19:45 ` Daniel Colascione
2016-11-30 21:06 ` Paul Eggert
2016-12-01 9:18 ` Richard Stallman
2016-12-01 18:09 ` Eli Zaretskii
2016-12-02 2:18 ` Stefan Monnier
2016-12-02 7:54 ` Eli Zaretskii
2016-12-02 8:08 ` John Wiegley
2016-12-02 8:59 ` Eli Zaretskii
2016-12-02 19:39 ` John Wiegley
2016-12-02 20:11 ` Karl Fogel
2016-12-02 21:22 ` Daniel Colascione
2016-12-02 22:06 ` Eli Zaretskii
2016-12-02 23:15 ` Karl Fogel
2016-12-15 14:28 ` Philippe Vaucher
2017-10-18 23:36 ` Kaushal Modi
2017-10-19 10:12 ` Jeremie Courreges-Anglas
2018-02-12 20:18 ` Daniel Colascione
2018-02-13 16:37 ` Eli Zaretskii
2018-02-14 21:03 ` Philipp Stephani
2018-02-15 0:42 ` Daniel Colascione
2018-02-15 23:31 ` Ken Brown
2018-02-15 23:36 ` Daniel Colascione
2018-02-16 1:56 ` Ken Brown
2018-02-16 2:36 ` Daniel Colascione
2018-02-17 23:38 ` Ken Brown
2018-02-17 23:59 ` Ken Brown
2018-02-18 0:02 ` Daniel Colascione
2018-02-19 13:30 ` Ken Brown
2018-02-19 17:03 ` Daniel Colascione
2018-02-19 22:33 ` Ken Brown
2018-02-20 16:32 ` Ken Brown
2018-02-20 17:23 ` Daniel Colascione
2018-02-20 1:16 ` Andy Moreton
2018-02-17 1:01 ` Clément Pit-Claudel
2018-02-19 17:06 ` Daniel Colascione
2018-02-19 22:00 ` Clément Pit-Claudel
2018-02-17 11:53 ` Charles A. Roelli
2018-02-17 12:09 ` Alan Third
2018-02-17 14:12 ` Charles A. Roelli
2018-02-20 0:54 ` Andy Moreton
2018-02-15 4:28 ` Stefan Monnier
2018-02-15 22:13 ` Daniel Colascione
2018-02-15 22:30 ` Paul Eggert
2018-02-15 22:35 ` Daniel Colascione
2018-02-15 22:56 ` Paul Eggert
2018-02-15 22:35 ` Paul Eggert
2018-02-15 18:34 ` andres.ramirez
2018-02-19 22:01 ` Daniele Nicolodi
2018-02-20 0:28 ` Daniel Colascione
2016-12-02 22:06 ` Eli Zaretskii
2016-12-02 22:28 ` Daniel Colascione
2016-12-03 8:48 ` Eli Zaretskii
2016-12-03 9:34 ` Daniel Colascione
2016-12-03 12:47 ` Eli Zaretskii
2016-12-03 14:36 ` Alan Mackenzie
2016-12-03 15:11 ` Eli Zaretskii
2016-12-04 12:20 ` Alan Mackenzie
2016-12-04 12:48 ` Dmitry Gutov
2016-12-04 15:53 ` Eli Zaretskii
2016-12-03 17:36 ` Daniel Colascione
2016-12-03 17:40 ` Dmitry Gutov
2016-12-03 21:09 ` Stefan Monnier
2016-12-03 21:31 ` Daniel Colascione
2016-12-04 4:25 ` Stefan Monnier
2016-12-04 12:34 ` Alan Mackenzie
2016-12-04 12:51 ` Dmitry Gutov
2016-12-04 14:08 ` Stefan Monnier
2016-12-04 15:22 ` Alan Mackenzie
2016-12-04 22:24 ` forward-comment and syntax-ppss (was: Preview: portable dumper) Stefan Monnier
2016-12-06 19:55 ` Alan Mackenzie
2016-12-06 22:56 ` forward-comment and syntax-ppss Stefan Monnier
2016-12-07 6:55 ` Andreas Röhler
2016-12-07 14:01 ` Stefan Monnier
2016-12-07 14:49 ` Andreas Röhler
2016-12-07 14:53 ` Clément Pit--Claudel
2016-12-07 22:04 ` Alan Mackenzie
2016-12-07 22:23 ` Clément Pit--Claudel
2016-12-08 19:31 ` Alan Mackenzie
2016-12-08 19:54 ` Clément Pit--Claudel
2016-12-09 8:37 ` Eli Zaretskii
2016-12-08 2:32 ` Stefan Monnier
2016-12-08 20:15 ` Alan Mackenzie
2016-12-08 20:34 ` Clément Pit--Claudel
2016-12-08 21:25 ` Dmitry Gutov
2016-12-08 21:35 ` Stefan Monnier
2016-12-08 21:13 ` Stefan Monnier
2016-12-09 18:00 ` Alan Mackenzie
2016-12-09 18:47 ` Stefan Monnier
2016-12-09 22:33 ` Clément Pit--Claudel
2016-12-09 23:09 ` Alan Mackenzie
2016-12-10 6:56 ` Eli Zaretskii
2016-12-10 18:39 ` Andreas Röhler
2016-12-08 21:24 ` Dmitry Gutov
2016-12-08 21:47 ` Stefan Monnier
2016-12-09 7:39 ` Andreas Röhler
2016-12-09 19:07 ` Alan Mackenzie
2016-12-09 23:30 ` Dmitry Gutov
2016-12-11 0:24 ` Stefan Monnier
2016-12-11 10:17 ` Alan Mackenzie
2016-12-11 10:40 ` Andreas Röhler
2016-12-16 1:33 ` Dmitry Gutov
2016-12-16 2:06 ` Drew Adams
2016-12-16 8:12 ` Eli Zaretskii
2016-12-16 12:32 ` Dmitry Gutov
2016-12-16 16:22 ` Drew Adams
2016-12-16 19:14 ` Dmitry Gutov
2016-12-16 23:16 ` Drew Adams
2016-12-17 0:08 ` Dmitry Gutov
2016-12-16 19:25 ` Alan Mackenzie
2016-12-16 20:06 ` Dmitry Gutov
2016-12-16 23:16 ` Drew Adams
2016-12-17 0:42 ` Dmitry Gutov
2016-12-17 1:30 ` Drew Adams
2016-12-17 1:40 ` Stefan Monnier
2016-12-17 2:26 ` Drew Adams
2016-12-17 14:37 ` Stefan Monnier
2016-12-16 14:24 ` Clément Pit--Claudel
2016-12-16 16:22 ` Drew Adams
2016-12-16 19:50 ` Clément Pit--Claudel
2016-12-16 23:16 ` Drew Adams
2016-12-17 2:39 ` Clément Pit--Claudel
2016-12-17 4:50 ` Wording of the Elisp manuals (was: forward-comment and syntax-ppss) Marcin Borkowski
2016-12-20 9:22 ` Wording of the Elisp manuals Michael Heerdegen
2016-12-20 10:52 ` Wording of the Elisp manuals (was: forward-comment and syntax-ppss) Jean-Christophe Helary
2016-12-20 16:01 ` Eli Zaretskii
[not found] ` <<83oa0c8f7r.fsf@gnu.org>
2016-12-16 16:22 ` forward-comment and syntax-ppss Drew Adams
2016-12-16 21:00 ` Eli Zaretskii
[not found] ` <<8337hn8u81.fsf@gnu.org>
2016-12-16 23:16 ` Drew Adams
2016-12-16 20:06 ` Alan Mackenzie
2016-12-16 22:08 ` Dmitry Gutov
2016-12-16 23:08 ` Stefan Monnier
2016-12-17 0:18 ` Dmitry Gutov
2016-12-17 7:37 ` Marcin Borkowski
2016-12-17 7:42 ` Eli Zaretskii
2016-12-17 14:41 ` Stefan Monnier
2016-12-17 14:49 ` Eli Zaretskii
2016-12-17 15:09 ` Lars Ingebrigtsen
2016-12-18 20:08 ` Dmitry Gutov
2017-02-05 23:33 ` Drew Adams
2016-12-17 15:38 ` Stefan Monnier
2016-12-19 2:31 ` Dmitry Gutov
2016-12-19 13:12 ` Stefan Monnier
2016-12-19 13:34 ` Dmitry Gutov
2016-12-19 14:17 ` Stefan Monnier
2016-12-19 23:16 ` Dmitry Gutov
2016-12-20 13:33 ` Stefan Monnier
2017-11-29 23:48 ` multi-modes and widen (again), was: " Dmitry Gutov
2017-11-30 13:05 ` Stefan Monnier
2017-12-01 23:57 ` Dmitry Gutov
2016-12-16 23:16 ` Drew Adams
2016-12-09 8:35 ` Eli Zaretskii
2016-12-09 17:08 ` Alan Mackenzie
2016-12-09 17:22 ` Eli Zaretskii
2016-12-09 21:55 ` Richard Stallman
2016-12-03 21:31 ` Preview: portable dumper Richard Stallman
2016-12-04 12:41 ` Alan Mackenzie
2016-12-03 17:41 ` Paul Eggert
2016-12-03 19:49 ` Eli Zaretskii
2016-12-03 21:30 ` Richard Stallman
2016-12-04 3:31 ` Eli Zaretskii
2016-12-04 23:03 ` Richard Stallman
2016-12-03 17:24 ` Paul Eggert
2016-12-03 15:56 ` Stefan Monnier
2016-12-03 21:31 ` Richard Stallman
2016-12-04 23:05 ` Richard Stallman
2016-12-02 22:29 ` John Wiegley
2016-12-03 21:28 ` Richard Stallman
2016-12-04 15:57 ` Eli Zaretskii
2016-12-04 17:12 ` Daniel Colascione
2016-12-04 23:07 ` Richard Stallman
2016-12-05 0:24 ` Daniel Colascione
2016-12-06 10:38 ` Philippe Vaucher
2016-12-02 9:00 ` Philippe Vaucher
2016-12-02 10:56 ` Eli Zaretskii
2017-05-26 19:48 ` Thien-Thi Nguyen
2017-05-26 20:26 ` Kaushal Modi
2017-05-27 7:27 ` Thien-Thi Nguyen
2016-12-02 13:04 ` Stefan Monnier
2016-12-02 14:45 ` Eli Zaretskii
2016-12-02 14:51 ` Stefan Monnier
2016-12-02 22:24 ` Richard Stallman
2016-12-02 23:32 ` Stefan Monnier
2016-12-03 8:28 ` Eli Zaretskii
2016-12-02 23:42 ` Paul Eggert
2016-12-02 15:38 ` Daniel Colascione
2016-12-02 17:26 ` Ken Raeburn
2016-12-02 17:47 ` Paul Eggert
[not found] ` <<jwvlgvyv10x.fsf-monnier+Inbox@gnu.org>
[not found] ` <<E1cCwGF-0002PT-Kq@fencepost.gnu.org>
2016-12-03 0:07 ` Drew Adams
2016-12-03 8:25 ` Eli Zaretskii
2016-12-03 20:40 ` Joost Kremers
2016-12-03 21:30 ` Richard Stallman
[not found] ` <<<jwvlgvyv10x.fsf-monnier+Inbox@gnu.org>
[not found] ` <<<E1cCwGF-0002PT-Kq@fencepost.gnu.org>
[not found] ` <<2b63d48d-a678-49c2-a3a9-4f91d8d8bdb4@default>
[not found] ` <<8337i5mnb5.fsf@gnu.org>
2016-12-03 16:14 ` Drew Adams
2016-12-03 16:42 ` Eli Zaretskii
2016-12-02 14:27 ` Richard Stallman
2016-11-28 21:14 ` Paul Eggert
2016-11-28 23:01 ` Stefan Monnier
2016-11-28 23:17 ` Daniel Colascione
2016-11-29 13:06 ` Stefan Monnier
2016-11-29 21:19 ` Daniel Colascione
2016-11-29 21:35 ` Paul Eggert
2016-11-29 21:50 ` Daniel Colascione
2016-11-29 22:01 ` Paul Eggert
2016-11-30 0:37 ` Daniel Colascione
2016-11-30 7:35 ` Paul Eggert
2016-11-30 13:33 ` Stefan Monnier
2016-11-30 20:07 ` Richard Stallman
2016-11-30 20:18 ` Daniel Colascione
2016-12-03 21:32 ` Richard Stallman
2016-12-03 21:37 ` Daniel Colascione
2016-12-04 23:03 ` Richard Stallman
2016-12-03 21:54 ` Paul Eggert
2016-11-29 22:01 ` Stefan Monnier
2016-11-29 22:22 ` Philipp Stephani
2016-11-29 22:34 ` Paul Eggert
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).