bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

To reproduce, on a 32-bit system:

 ./configure ... --with-wide-int
 make
 cd test/automated
 make

This causes the cl-lib-tests to fail in this innocent check:

  (ert-deftest cl-lib-test-first ()
    (should (null (cl-first '()))) <<<<<<<<<<<<<<<<<

The error is "Invalid function ((nil))".

The same test runs fine in a 32-bit build with 32-bit ints.

While digging into this, I found out that if I compile cl-lib-test.el
with an Emacs executable that doesn't use wide ints, the test
succeeds.  So I compared the .elc files, and they indeed differ.
Moreover, by comparing the *.elc files in the entire Emacs tree, I
found 3 files that differ: cl-generic.elc, srt-wy.elc, and one file in
Eshell whose name I don't remember.

Focusing my attention on cl-generic, which might explain why
cl-lib-tests fails, how can I debug this?  Would presenting
disassembly of a part of the byte code where I see differences help
someone to come up with ideas?  Are there any other directions to
investigate this?

Did anyone run the test suite successfully in such a 32-bit build?  Do
all the *.elc files come out identical to a "normal" 32-bit build?

TIA


In GNU Emacs 25.0.50.189 (i686-pc-mingw32)
 of 2015-06-21 on HOME-C4E4A596F7
Repository revision: 821a3633564f36857968c7fe2b8bb6681a895905
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --prefix=/d/usr --enable-checking=yes,glyphs --with-wide-int
 'CFLAGS=-gdwarf-4 -g3 -O0''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message dired format-spec
rfc822 mml easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util help-fns mail-prsvr mail-utils time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 79448 7064)
 (symbols 56 18398 0)
 (miscs 48 36 96)
 (strings 16 12848 4918)
 (string-bytes 1 326874)
 (vectors 16 10639)
 (vector-slots 8 396234 8479)
 (floats 8 114 55)
 (intervals 40 251 67)
 (buffers 856 11))

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

Thanks for reporting that.  It appears to be a bug in the garbage collector, and 
is likely to be hard to reproduce.  I couldn't reproduce it, but I did a 'make 
bootstrap' on Fedora x86-64 (configured with --with-wide-int and compiled with 
gcc -m32 so it's really x86), and got a core dump in a completely different area 
that (of course!) went away when I compiled without optimization.

Rather than try to debug it directly, I thought about what might have caused the 
problem, and re-audited the garbage collector with the recent Qnil==0 changes in 
mind.  This did uncover a bug, and the attached patch (which we will need 
anyway) allowed me to do a "make bootstrap" successfully in the same 
configuration.  I installed this into the master as commit 
93f4f67ba93b78e8b31e498e8ce7bce4c8298b76; please give it a try in your setup 
when you have the time.

[-- Attachment #2: 0001-Fix-GC-bugs-with-wide-int-and-Qnil-0.patch --]
[-- Type: text/x-diff, Size: 7785 bytes --]

From 2bc416753ea9dbdeb27719957076ceb909886bd1 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Wed, 24 Jun 2015 20:10:03 -0700
Subject: [PATCH] Fix GC bugs --with-wide-int and Qnil == 0

Use the same alignment for the !USE_LSB_TAG case as for the
more-typical USE_LSB_TAG case.  The attempt to support arbitrary
alignments with !USE_LSB_TAG had subtle bugs in garbage collection
once we changed the representation of symbols so that Qnil == 0.
Problem reported by Eli Zaretskii (Bug#20862).
* src/alloc.c (XMALLOC_HEADER_ALIGNMENT) [XMALLOC_OVERRUN_CHECK]:
* src/alloc.c (vector_alignment, union aligned_Lisp_Symbol)
(union aligned_Lisp_Misc, maybe_lisp_pointer, pure_alloc):
Use same alignment for !USE_LSB_TAG as for USE_LSB_TAG.
* src/alloc.c (POINTERS_MIGHT_HIDE_IN_OBJECTS): Remove.
This optimization in the !USE_LSB_TAG case is no longer valid when
symbols are represented via offsets.  Change the only use to
assume that pointers might hide in objects.
* src/lisp.h (alignas) [!USE_LSB_TAG]:
Require support in this case, too.
(TAG_SYMOFFSET, XSYMBOL) [!USE_LSB_TAG]: Do not shift the offset.
This is OK, because the !USE_LSB_TAG case now applies only when
Lisp_Object is wider than void *, so there's no longer any need
to shift the offset.  Not shifting the offset means that
symbol representations have the same alignment as pointers,
which the GC assumes.
---
 src/alloc.c | 59 ++++++++++-------------------------------------------------
 src/lisp.h  | 11 ++---------
 2 files changed, 12 insertions(+), 58 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index a956e95..c9bdcc2 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -528,12 +528,8 @@ buffer_memory_full (ptrdiff_t nbytes)
    alignment that Emacs needs for C types and for USE_LSB_TAG.  */
 #define XMALLOC_BASE_ALIGNMENT alignof (max_align_t)
 
-#if USE_LSB_TAG
-# define XMALLOC_HEADER_ALIGNMENT \
-    COMMON_MULTIPLE (GCALIGNMENT, XMALLOC_BASE_ALIGNMENT)
-#else
-# define XMALLOC_HEADER_ALIGNMENT XMALLOC_BASE_ALIGNMENT
-#endif
+#define XMALLOC_HEADER_ALIGNMENT \
+   COMMON_MULTIPLE (GCALIGNMENT, XMALLOC_BASE_ALIGNMENT)
 #define XMALLOC_OVERRUN_SIZE_SIZE				\
    (((XMALLOC_OVERRUN_CHECK_SIZE + sizeof (size_t)		\
       + XMALLOC_HEADER_ALIGNMENT - 1)				\
@@ -2730,7 +2726,7 @@ enum
   {
     /* Alignment of struct Lisp_Vector objects.  */
     vector_alignment = COMMON_MULTIPLE (ALIGNOF_STRUCT_LISP_VECTOR,
-					USE_LSB_TAG ? GCALIGNMENT : 1),
+					GCALIGNMENT),
 
     /* Vector size requests are a multiple of this.  */
     roundup_size = COMMON_MULTIPLE (vector_alignment, word_size)
@@ -3299,15 +3295,13 @@ usage: (make-byte-code ARGLIST BYTE-CODE CONSTANTS DEPTH &optional DOCSTRING INT
  ***********************************************************************/
 
 /* Like struct Lisp_Symbol, but padded so that the size is a multiple
-   of the required alignment if LSB tags are used.  */
+   of the required alignment.  */
 
 union aligned_Lisp_Symbol
 {
   struct Lisp_Symbol s;
-#if USE_LSB_TAG
   unsigned char c[(sizeof (struct Lisp_Symbol) + GCALIGNMENT - 1)
 		  & -GCALIGNMENT];
-#endif
 };
 
 /* Each symbol_block is just under 1020 bytes long, since malloc
@@ -3411,15 +3405,13 @@ Its value is void, and its function definition and property list are nil.  */)
  ***********************************************************************/
 
 /* Like union Lisp_Misc, but padded so that its size is a multiple of
-   the required alignment when LSB tags are used.  */
+   the required alignment.  */
 
 union aligned_Lisp_Misc
 {
   union Lisp_Misc m;
-#if USE_LSB_TAG
   unsigned char c[(sizeof (union Lisp_Misc) + GCALIGNMENT - 1)
 		  & -GCALIGNMENT];
-#endif
 };
 
 /* Allocation of markers and other objects that share that structure.
@@ -4628,13 +4620,13 @@ mark_maybe_object (Lisp_Object obj)
 }
 
 /* Return true if P can point to Lisp data, and false otherwise.
-   USE_LSB_TAG needs Lisp data to be aligned on multiples of GCALIGNMENT.
-   Otherwise, assume that Lisp data is aligned on even addresses.  */
+   Symbols are implemented via offsets not pointers, but the offsets
+   are also multiples of GCALIGNMENT.  */
 
 static bool
 maybe_lisp_pointer (void *p)
 {
-  return !((intptr_t) p % (USE_LSB_TAG ? GCALIGNMENT : 2));
+  return (uintptr_t) p % GCALIGNMENT == 0;
 }
 
 /* If P points to Lisp data, mark that as live if it isn't already
@@ -4722,27 +4714,6 @@ mark_maybe_pointer (void *p)
    miss objects if __alignof__ were used.  */
 #define GC_POINTER_ALIGNMENT alignof (void *)
 
-/* Define POINTERS_MIGHT_HIDE_IN_OBJECTS to 1 if marking via C pointers does
-   not suffice, which is the typical case.  A host where a Lisp_Object is
-   wider than a pointer might allocate a Lisp_Object in non-adjacent halves.
-   If USE_LSB_TAG, the bottom half is not a valid pointer, but it should
-   suffice to widen it to to a Lisp_Object and check it that way.  */
-#if USE_LSB_TAG || VAL_MAX < UINTPTR_MAX
-# if !USE_LSB_TAG && VAL_MAX < UINTPTR_MAX >> GCTYPEBITS
-  /* If tag bits straddle pointer-word boundaries, neither mark_maybe_pointer
-     nor mark_maybe_object can follow the pointers.  This should not occur on
-     any practical porting target.  */
-#  error "MSB type bits straddle pointer-word boundaries"
-# endif
-  /* Marking via C pointers does not suffice, because Lisp_Objects contain
-     pointer words that hold pointers ORed with type bits.  */
-# define POINTERS_MIGHT_HIDE_IN_OBJECTS 1
-#else
-  /* Marking via C pointers suffices, because Lisp_Objects contain pointer
-     words that hold unmodified pointers.  */
-# define POINTERS_MIGHT_HIDE_IN_OBJECTS 0
-#endif
-
 /* Mark Lisp objects referenced from the address range START+OFFSET..END
    or END+OFFSET..START.  */
 
@@ -4788,8 +4759,7 @@ mark_memory (void *start, void *end)
       {
 	void *p = *(void **) ((char *) pp + i);
 	mark_maybe_pointer (p);
-	if (POINTERS_MIGHT_HIDE_IN_OBJECTS)
-	  mark_maybe_object (XIL ((intptr_t) p));
+	mark_maybe_object (XIL ((intptr_t) p));
       }
 }
 
@@ -5148,22 +5118,13 @@ static void *
 pure_alloc (size_t size, int type)
 {
   void *result;
-#if USE_LSB_TAG
-  size_t alignment = GCALIGNMENT;
-#else
-  size_t alignment = alignof (EMACS_INT);
-
-  /* Give Lisp_Floats an extra alignment.  */
-  if (type == Lisp_Float)
-    alignment = alignof (struct Lisp_Float);
-#endif
 
  again:
   if (type >= 0)
     {
       /* Allocate space for a Lisp object from the beginning of the free
 	 space with taking account of alignment.  */
-      result = ALIGN (purebeg + pure_bytes_used_lisp, alignment);
+      result = ALIGN (purebeg + pure_bytes_used_lisp, GCALIGNMENT);
       pure_bytes_used_lisp = ((char *)result - (char *)purebeg) + size;
     }
   else
diff --git a/src/lisp.h b/src/lisp.h
index 198f116..c3289c9 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -278,10 +278,7 @@ error !;
 #endif
 
 #ifndef alignas
-# define alignas(alignment) /* empty */
-# if USE_LSB_TAG
-#  error "USE_LSB_TAG requires alignas"
-# endif
+# error "alignas not defined"
 #endif
 
 #ifdef HAVE_STRUCT_ATTRIBUTE_ALIGNED
@@ -731,9 +728,7 @@ struct Lisp_Symbol
 
 /* Yield an integer that contains a symbol tag along with OFFSET.
    OFFSET should be the offset in bytes from 'lispsym' to the symbol.  */
-#define TAG_SYMOFFSET(offset)				    \
-  TAG_PTR (Lisp_Symbol,					    \
-	   ((uintptr_t) (offset) >> (USE_LSB_TAG ? 0 : GCTYPEBITS)))
+#define TAG_SYMOFFSET(offset) TAG_PTR (Lisp_Symbol, offset)
 
 /* XLI_BUILTIN_LISPSYM (iQwhatever) is equivalent to
    XLI (builtin_lisp_symbol (Qwhatever)),
@@ -899,8 +894,6 @@ INLINE struct Lisp_Symbol *
 XSYMBOL (Lisp_Object a)
 {
   uintptr_t i = (uintptr_t) XUNTAG (a, Lisp_Symbol);
-  if (! USE_LSB_TAG)
-    i <<= GCTYPEBITS;
   void *p = (char *) lispsym + i;
   return p;
 }
-- 
2.1.0

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

> Date: Wed, 24 Jun 2015 20:31:10 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: 20862@debbugs.gnu.org
> 
> Thanks for reporting that. It appears to be a bug in the garbage collector, and is likely to be hard to reproduce. I couldn't reproduce it, but I did a 'make bootstrap' on Fedora x86-64 (configured with --with-wide-int and compiled with gcc -m32 so it's really x86), and got a core dump in a completely different area that (of course!) went away when I compiled without optimization.

Do you remember which Lisp file was being compiled when you got a core
dump?

I had something similar while trying to debug this: crashes while
compiling cedet/srecode/proj-obj.el and sometimes also ibuffer.el.
And yes, it's a very elusive crash: it only happens during a full
bootstrap, and even repeating the exact same bootstrap in a different
directory makes it disappear!  But I was luckier than you, in that I
did succeed reproducing these crashes in an unoptimized build.  By
slowly and painfully tracking these crashes, I found out that they are
due to an un-interned symbol whose name is "THIS", created by this
form:

  (let ((print-gensym nil) (print-quoted t)) (format "%S" (cons 'fn (cl--make-usage-args orig-args))))

That symbol gets GC'ed (I see "DEAD" in its function cell) while it's
still alive, and then Emacs crashes trying to print that symbol's name
in the call to 'format', because GC recycles that symbol's name and
replaces it with a NULL pointer.  So your analysis:

> Rather than try to debug it directly, I thought about what might have caused the problem, and re-audited the garbage collector with the recent Qnil==0 changes in mind. This did uncover a bug, and the attached patch (which we will need anyway) allowed me to do a "make bootstrap" successfully in the same configuration.

and the changes in the patch related to symbols-as-offsets and their
alignment on the stack, make perfect sense to me, because they explain
how come stack marking didn't mark this symbol, and thus allowed it to
be GC'ed.

> I installed this into the master as commit 93f4f67ba93b78e8b31e498e8ce7bce4c8298b76; please give it a try in your setup when you have the time. 

I did, and the crashes are gone, thanks.  The cl-lib-tests also
succeed.

However, there still seems to be some subtle problem, because the
byte-compiled files don't all compare equal.  (I've seen the same
problem before your patches as well.)  I used the following command to
find the *.elc files that are different:

 diff -r -a -u -I"in Emacs version 25\.0\.50" ./lisp ../int32/lisp --exclude="*.el" --exclude="*.el~" | grep -a "^diff "

where "../int32/" is the directory where I built Emacs without
"--with-wide-int".  This reveals differences in the following files:

 cedet/semantic/texi.elc   cedet/semantic/util.elc
 cedet/srecode/srt-wy.elc  emacs-lisp/cl-generic.elc

Some of the differences are insignificant (different label numbers
used, or different file offsets due to a longer Emacs version string),
but others seem to be significant.  For example, the byte code of
cl--generic-struct-tag in cl-generic.elc has a few different bytes.
Likewise with the byte code of semantic-texi-expand-tag in texi.elc,
of semantic-something-to-tag-table in util.elc, and of
srecode-template-wy--parse-table in srt-wy.elc.

The list of *.elc files that differ appears to depend on optimization
level: the above list was obtained with -O0; compiling with -O1 leaves
only cl-generic.elc and srt-wy.elc different, and compiling with -O2
brings util.elc back, and also adds differences in
eshell/esh-proc.elc.  Or maybe the actual factor is the specific order
in which the files are compiled (I bootstrap with "make -j8"), which
determines which other Lisp files are available as *.el or *.elc,
because bootstrapping without parallel Make execution again leaves
only cl-generic.elc and srt-wy.elc?

Do you see something similar on your system?  How to go about
debugging this?

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Paul Eggert]

Eli Zaretskii wrote:

> Do you remember which Lisp file was being compiled when you got a core
> dump?

Sorry, no.  I don't think it matters -- if the garbage collector is bad anything 
can happen.

>  maybe the actual factor is the specific order
> in which the files are compiled (I bootstrap with "make -j8"),

Yes, that's plausible.

> Do you see something similar on your system?  How to go about
> debugging this?

I expect it'd require someone with expertise in byte-compiling.  It's not clear 
that it's a bug, other than a minor performance issue.  I do see something 
similar on my system.  The following files differed in one comparison I did:

cedet/semantic/grammar-wy.elc
cedet/semantic/wisent/javat-wy.elc
cedet/semantic/wisent/js-wy.elc
cedet/semantic/wisent/python-wy.elc
cedet/srecode/insert.elc
cedet/srecode/srt-wy.elc
gnus/auth-source.elc
gnus/message.elc
gnus/mm-decode.elc
org/ob-C.elc
progmodes/antlr-mode.elc
progmodes/cc-langs.elc
progmodes/cc-mode.elc
textmodes/rst.elc

In one case I looked at it appears to be gensym values, which ought to be OK.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

> Date: Fri, 26 Jun 2015 12:59:40 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: 20862@debbugs.gnu.org
> 
> In one case I looked at it appears to be gensym values, which ought to be OK.

Can you elaborate?  What are "gensym values"?

Thanks.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Paul Eggert]

Eli Zaretskii wrote:
> Can you elaborate?  What are "gensym values"?

Generated symbols of some sort.  They appear to be like the output that the 
printer uses when it's printing a recursive data structure.  E.g., one .elc file 
says ‘(#3=#:G66300 e)’ where the other says ‘(#3=#:G44900 e)’.  I doubt whether 
this difference matters, though I haven't investigated.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Glenn Morris]

Paul Eggert wrote:

> Eli Zaretskii wrote:
>> Can you elaborate?  What are "gensym values"?
>
> Generated symbols of some sort.  They appear to be like the output
> that the printer uses when it's printing a recursive data structure.
> E.g., one .elc file says '(#3=#:G66300 e)' where the other says
> '(#3=#:G44900 e)'.  I doubt whether this difference matters, though I
> haven't investigated.

I think this is because cl--gensym-counter, as used by cl-gensym, is
initialized to a random value based on the time. (This is an issue for
reproducible builds, if we care about those.)

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

> Date: Fri, 26 Jun 2015 13:45:24 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: 20862@debbugs.gnu.org
> 
> Eli Zaretskii wrote:
> > Can you elaborate?  What are "gensym values"?
> 
> Generated symbols of some sort.  They appear to be like the output that the 
> printer uses when it's printing a recursive data structure.  E.g., one .elc file 
> says ‘(#3=#:G66300 e)’ where the other says ‘(#3=#:G44900 e)’.  I doubt whether 
> this difference matters, though I haven't investigated.

Thanks.  Those not the differences that I see that worry me.  I guess
I should look at the disassembly differences next.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  20862@debbugs.gnu.org
> Date: Fri, 26 Jun 2015 17:30:26 -0400
> 
> Paul Eggert wrote:
> 
> > Eli Zaretskii wrote:
> >> Can you elaborate?  What are "gensym values"?
> >
> > Generated symbols of some sort.  They appear to be like the output
> > that the printer uses when it's printing a recursive data structure.
> > E.g., one .elc file says '(#3=#:G66300 e)' where the other says
> > '(#3=#:G44900 e)'.  I doubt whether this difference matters, though I
> > haven't investigated.
> 
> I think this is because cl--gensym-counter, as used by cl-gensym, is
> initialized to a random value based on the time. (This is an issue for
> reproducible builds, if we care about those.)

I think the *.elc files should be reproducible, yes.  Otherwise, it's
hard to know if there's some subtle problem in the byte compiler or
its support routines.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Paul Eggert]

Eli Zaretskii wrote:
> I think the *.elc files should be reproducible, yes.  Otherwise, it's
> hard to know if there's some subtle problem in the byte compiler or
> its support routines.

The simplest fix would be to initialize cl--gensym-counter to 0, as is done in 
Common Lisp, so that Emacs generates reproducible output.  However, the 
documentation for cl-gensym says this:

(Internally, the variable @code{cl--gensym-counter} holds the counter
used to generate names.  It is incremented after each use.  In Common
Lisp this is initialized with 0, but this package initializes it with
a random time-dependent value to avoid trouble when two files that
each used @code{cl-gensym} in their compilation are loaded together.
Uninterned symbols become interned when the compiler writes them out
to a file and the Emacs loader loads them, so their names have to be
treated a bit more carefully than in Common Lisp where uninterned
symbols remain uninterned after loading.)


This passage is dismaying, as it suggests that it's a matter of luck as to 
whether trouble is avoided when two files that each used cl-gensym are loaded 
together.  I guess the trouble that's avoided is the usual issue with macro 
capture.  (Maybe we should file another bug report for this....)

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

> Date: Sat, 27 Jun 2015 08:56:30 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: 20862@debbugs.gnu.org
> 
> This passage is dismaying, as it suggests that it's a matter of luck as to 
> whether trouble is avoided when two files that each used cl-gensym are loaded 
> together.

Yes, and I think it should not be too hard to come up with a number or
string that will not be random, and yet will be unique to each file.

> I guess the trouble that's avoided is the usual issue with macro 
> capture.  (Maybe we should file another bug report for this....)

Indeed, I think it's worth another bug report.

Thanks.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Andreas Schwab]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Uninterned symbols become interned when the compiler writes them out

That isn't true.  #: prefixed symbols are uninterned symbols.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

Andreas Schwab wrote:
> That isn't true.  #: prefixed symbols are uninterned symbols.

Good point, thanks.  The doc I quoted predates the #: syntax, and evidently 
wasn't updated when the #: syntax was introduced and used.  Since that doc 
wording is wrong, and there's now no reason to initialize cl--gensym-counter to 
anything other than 0, I boldly installed the attached patch.  It passes 'make 
bootstrap' and 'make check' (at least, as well as 'make check' did before -- it 
always fails for me in epg-roundtrip-2).

[-- Attachment #2: 0001-Initialize-cl-gensym-counter-to-0.patch --]
[-- Type: text/x-diff, Size: 4035 bytes --]

From c66ff5757c6eeb495eeb283f34fc0713d1fb5270 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 27 Jun 2015 10:57:02 -0700
Subject: [PATCH] Initialize cl--gensym-counter to 0

Previously it was initialized to a random value, which made it
harder to reproduce earlier Emacs runs.  The need for a random
value went away when Emacs introduced and used the #: syntax for
uninterned symbols (Bug#20862).
* doc/misc/cl.texi (Creating Symbols, Common Lisp Compatibility):
Document that cl--gensym-counter now starts with 0.
* lisp/emacs-lisp/cl-lib.el (cl--gensym-counter): Remove.
(cl--random-time): Move to near only remaining use.
* lisp/emacs-lisp/cl-macs.el (cl--gensym-counter): Initialize to 0.
---
 doc/misc/cl.texi           | 15 +++------------
 lisp/emacs-lisp/cl-lib.el  | 15 +++++----------
 lisp/emacs-lisp/cl-macs.el |  2 +-
 3 files changed, 9 insertions(+), 23 deletions(-)

diff --git a/doc/misc/cl.texi b/doc/misc/cl.texi
index d7b3f4a..1f38ca9 100644
--- a/doc/misc/cl.texi
+++ b/doc/misc/cl.texi
@@ -2877,14 +2877,8 @@ their names will not conflict with ``real'' variables in the user's
 code.
 
 (Internally, the variable @code{cl--gensym-counter} holds the counter
-used to generate names.  It is incremented after each use.  In Common
-Lisp this is initialized with 0, but this package initializes it with
-a random time-dependent value to avoid trouble when two files that
-each used @code{cl-gensym} in their compilation are loaded together.
-Uninterned symbols become interned when the compiler writes them out
-to a file and the Emacs loader loads them, so their names have to be
-treated a bit more carefully than in Common Lisp where uninterned
-symbols remain uninterned after loading.)
+used to generate names.  It is initialized with zero and incremented
+after each use.)
 @end defun
 
 @defun cl-gentemp &optional x
@@ -4543,10 +4537,7 @@ example, local @code{special} declarations, which are purely
 advisory in Emacs Lisp, do not rigorously obey the scoping rules
 set down in Steele's book.
 
-The variable @code{cl--gensym-counter} starts out with a pseudo-random
-value rather than with zero.  This is to cope with the fact that
-generated symbols become interned when they are written to and
-loaded back from a file.
+The variable @code{cl--gensym-counter} starts out with zero.
 
 The @code{cl-defstruct} facility is compatible, except that structures
 are of type @code{:type vector :named} by default rather than some
diff --git a/lisp/emacs-lisp/cl-lib.el b/lisp/emacs-lisp/cl-lib.el
index b6f3a79..2dd0519 100644
--- a/lisp/emacs-lisp/cl-lib.el
+++ b/lisp/emacs-lisp/cl-lib.el
@@ -249,16 +249,6 @@ so that they are registered at compile-time as well as run-time."
       `(progn ,@body))))           ; Avoid loading cl-macs.el for cl-eval-when.
 
 
-;;; Symbols.
-
-(defun cl--random-time ()
-  (let* ((time (copy-sequence (current-time-string))) (i (length time)) (v 0))
-    (while (>= (cl-decf i) 0) (setq v (+ (* v 3) (aref time i))))
-    v))
-
-(defvar cl--gensym-counter (* (logand (cl--random-time) 1023) 100))
-
-
 ;;; Numbers.
 
 (define-obsolete-function-alias 'cl-floatp-safe 'floatp "24.4")
@@ -298,6 +288,11 @@ If true return the decimal value of digit CHAR in RADIX."
   (let ((n (aref cl-digit-char-table char)))
     (and n (< n (or radix 10)) n)))
 
+(defun cl--random-time ()
+  (let* ((time (copy-sequence (current-time-string))) (i (length time)) (v 0))
+    (while (>= (cl-decf i) 0) (setq v (+ (* v 3) (aref time i))))
+    v))
+
 (defvar cl--random-state
   (vector 'cl--random-state-tag -1 30 (cl--random-time)))
 
diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 636c543..5bcf088 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -161,7 +161,7 @@ whether X is known at compile time, macroexpand it completely in
 
 ;;; Symbols.
 
-(defvar cl--gensym-counter)
+(defvar cl--gensym-counter 0)
 ;;;###autoload
 (defun cl-gensym (&optional prefix)
   "Generate a new uninterned symbol.
-- 
2.1.0

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

> Date: Sat, 27 Jun 2015 11:01:53 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, Glenn Morris <rgm@gnu.org>, 
>  20862@debbugs.gnu.org
> 
> I boldly installed the attached patch.  It passes 'make bootstrap'
> and 'make check'

Thanks.

Did you try "make -jN", for some reasonably large N?

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Paul Eggert]

Eli Zaretskii wrote:
> Did you try "make -jN", for some reasonably large N?

I used 'make -j5' on a 4-core host.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

> Date: Sat, 27 Jun 2015 11:40:40 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: schwab@linux-m68k.org, rgm@gnu.org, 20862@debbugs.gnu.org
> 
> Eli Zaretskii wrote:
> > Did you try "make -jN", for some reasonably large N?
> 
> I used 'make -j5' on a 4-core host.

Thanks, that's reassuring.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Eli Zaretskii]

> Date: Sat, 27 Jun 2015 22:41:57 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: schwab@linux-m68k.org, 20862@debbugs.gnu.org
> 
> > Date: Sat, 27 Jun 2015 11:40:40 -0700
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > CC: schwab@linux-m68k.org, rgm@gnu.org, 20862@debbugs.gnu.org
> > 
> > Eli Zaretskii wrote:
> > > Did you try "make -jN", for some reasonably large N?
> > 
> > I used 'make -j5' on a 4-core host.
> 
> Thanks, that's reassuring.

After updating from Savannah, I no longer see any problems related to
the original report.  So I guess we can close this bug now, right?

Thanks.

bug#20862: 25.0.50; 32-bit Emacs configured --with-wide-int miscompiles CL

[Paul Eggert]

Eli Zaretskii wrote:

> After updating from Savannah, I no longer see any problems related to
> the original report.  So I guess we can close this bug now, right?

Yes, thanks for checking.  Done.