bug#11519: "Wrong type argument: characterp" building custom-deps while boostrapping

[Eli Zaretskii <eliz@gnu.org>]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  lekktu@gmail.com,  11519@debbugs.gnu.org
> Date: Mon, 21 May 2012 09:59:50 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Stefan Monnier <monnier@iro.umontreal.ca>
> >> Cc: Eli Zaretskii <eliz@gnu.org>,  11519@debbugs.gnu.org
> >> Date: Sun, 20 May 2012 21:50:04 -0400
> >> 
> >> > The bug disappears if you revert this change:
> >> [...]
> >> >   Remove no-byte-compile setting from some lisp/language files.
> >> 
> >> My gut tells me this is just a trigger but not the cause of the bug :-(
> >
> > What could be the cause, given that this commit triggers it?
> 
> Perhaps some constant sharing that the byte compiler introduces?

Maybe.  But based on the evidence below, and also on the fact that the
problem is so elusive and behaves like a classical heisenbug, my first
suspicion is elsewhere.

After trying a few tricks, I succeeded to catch the bug in GDB.  (The
full backtrace is near the end of this message.)  Here's what I found.

First, this really happens while cus-dep.el scans ethio-util.el.  The
Lisp call which throws the error is this:

                   (re-search-forward
		     (concat "(provide[ \t\n]+\\('\\|(quote[ \t\n]\\)[ \t\n]*"
			     (regexp-quote name) "[ \t\n)]")
		     nil t))

What I see on the C level is this:

(gdb) bt
  #0  xsignal2 (error_symbol=56891898, arg1=56963794, arg2=16781132)
      at eval.c:1740
  #1  0x01024a51 in wrong_type_argument (predicate=56891898, value=16781132)
      at data.c:111
  #2  0x0102b4e8 in Faref (array=61843973, idx=16781132) at data.c:2090
  #3  0x0129e66e in char_table_translate (table=61843973, ch=4195283)
      at chartab.c:680
  #4  0x01142f6a in re_search_2 (bufp=0x1933c00,
      str1=0x10757948 "\002╘⌐\002\303¿\002\312\212\002\307╖\002\333¼\002\334ק\002\341\214\217ß\002\341\214\216ß\002\303▒\002╘╗\002\341\214\215ß\002\303╗\002\341\214\214ß\002\341\214ßכ\002\305⌐\002\307ע\002\305┐\002\307ח\002\305¿\002\305\262\002\303צ\002\306\263\002\341\214\212ß\002\341\214ßי\002\303\220",
      size1=51024, str2=0x1077fb17 <Address 0x1077fb17 out of bounds>, size2=0,
      startpos=23749, range=27240, regs=0x19351f0, stop=51024) at regex.c:4422
  #5  0x010fbd70 in search_buffer (string=272417249, pos=1, pos_byte=1,
      lim=48448, lim_byte=51025, n=1, RE=1, trt=61843973, inverse_trt=61841925,
      posix=0) at search.c:1205
  #6  0x010fb578 in search_command (string=272417249, bound=56838170,
      noerror=56838194, count=56838170, direction=1, RE=1, posix=0)
      at search.c:997
  #7  0x010fef92 in Fre_search_forward (regexp=272417249, bound=56838170,
      noerror=56838194, count=56838170) at search.c:2162
  (gdb) fr 7
  #7  0x010fef92 in Fre_search_forward (regexp=272417249, bound=56838170,
      noerror=56838194, count=56838170) at search.c:2162
  2162      return search_command (regexp, bound, noerror, count, 1, 1, 0);
  (gdb) p regexp
  $9 = 272417249
  (gdb) xstring
  $10 = (struct Lisp_String *) 0x103cc1e0
  "(provide[	 \n]+\\('\\|(quote[	 \n]\\)[	 \n]*ethio-util[	 \n)]"
  (gdb) fr 3
  #3  0x0129e66e in char_table_translate (table=61843973, ch=4195283)
      at chartab.c:680
  680       value = Faref (table, make_number (ch));
  (gdb) p table
  $11 = 61843973
  (gdb) xtype
  Lisp_Vectorlike
  PVEC_CHAR_TABLE
  (gdb) xchartable
  $12 = (struct Lisp_Char_Table *) 0x3afaa00
  Purpose: "case-table"  3 extra slots
  (gdb) p current_buffer->text->beg
  $13 = (
      unsigned char *) 0x10744948 ";;; ethio-util.el --- utilities for Ethiopic  -*- coding: utf-8-emacs; -*-\n\n;; Copyright (C) 1997-1998, 2002-2012  Free Software Foundation, Inc.\n;; Copyright (C) 1997, 1998, 1999, 2000, 2001, 2002, 20"...
  (gdb) p current_buffer->pt
  $14 = 1

So we are searching from buffer beginning, and the buffer text looks
perfectly OK.  But look what "text" is regex.c trying to search:

#4  0x01142f6a in re_search_2 (bufp=0x1933c00,
    str1=0x10757948 "\002╘⌐\002\303¿\002\312\212\002\307╖\002\333¼\002\334ק\002\341\214\217ß\002\341\214\216ß\002\303▒\002╘╗\002\341\214\215ß\002\303╗\002\341\214\214ß\002\341\214ßכ\002\305⌐\002\307ע\002\305┐\002\307ח\002\305¿\002\305\262\002\303צ\002\306\263\002\341\214\212ß\002\341\214ßי\002\303\220",
    size1=51024, str2=0x1077fb17 <Address 0x1077fb17 out of bounds>, size2=0,
    startpos=23749, range=27240, regs=0x19351f0, stop=51024) at regex.c:4422

Total garbage.

So I think that what happened is that something, probably the
translation through a char-table, caused allocation of a large chunk
of memory, which in turn relocated the text of the current buffer
behind regex.c's back, which still uses C pointers to the old location
of the buffer text.  Here's how search.c calls re_search_2:

      p1 = BEGV_ADDR;
      s1 = GPT_BYTE - BEGV_BYTE;
      p2 = GAP_END_ADDR;
      s2 = ZV_BYTE - GPT_BYTE;
      if (s1 < 0)
	{
	  p2 = p1;
	  s2 = ZV_BYTE - BEGV_BYTE;
	  s1 = 0;
	}
      if (s2 < 0)
	{
	  s1 = ZV_BYTE - BEGV_BYTE;
	  s2 = 0;
	}
      re_match_object = Qnil;

      while (n < 0)
	{
	  EMACS_INT val;
	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
			     (NILP (Vinhibit_changing_match_data)
			      ? &search_regs : &search_regs_1),
			     /* Don't allow match past current point */
			     pos_byte - BEGV_BYTE);

We pass s1 and s2, which are pointers to buffer text, so if buffer
text is relocated, we are screwed.

Does this explanation sound reasonable?  If so, any ideas how to fix
this?

Here's the full backtrace:

  #0  xsignal2 (error_symbol=56891898, arg1=56963794, arg2=16781132)
      at eval.c:1740
  #1  0x01024a51 in wrong_type_argument (predicate=56891898, value=16781132)
      at data.c:111
  #2  0x0102b4e8 in Faref (array=61843973, idx=16781132) at data.c:2090
  #3  0x0129e66e in char_table_translate (table=61843973, ch=4195283)
      at chartab.c:680
  #4  0x01142f6a in re_search_2 (bufp=0x1933c00,
      str1=0x10757948 "\002╘⌐\002\303¿\002\312\212\002\307╖\002\333¼\002\334ק\002 341\214\\217ß\002\341\214\216ß\002\303▒\002╘╗\002\341\214\215ß\002\303╗\002\341\214\214ß\002\341\214ßכ\002\305⌐\002\307ע\002\305┐\002\307ח\002\305¿\002\305\262\002\303צ\002\306\263\002\341\214\212ß\002\341\214ßי\002\303\220",
      size1=51024, str2=0x1077fb17 <Address 0x1077fb17 out of bounds>, size2=0,
      startpos=23749, range=27240, regs=0x19351f0, stop=51024) at regex.c:4422
  #5  0x010fbd70 in search_buffer (string=272417249, pos=1, pos_byte=1,
      lim=48448, lim_byte=51025, n=1, RE=1, trt=61843973, inverse_trt=61841925,
      posix=0) at search.c:1205
  #6  0x010fb578 in search_command (string=272417249, bound=56838170,
      noerror=56838194, count=56838170, direction=1, RE=1, posix=0)
      at search.c:997
  #7  0x010fef92 in Fre_search_forward (regexp=272417249, bound=56838170,
      noerror=56838194, count=56838170) at search.c:2162
  #8  0x01036e84 in Ffuncall (nargs=4, args=0x82df40) at eval.c:2946
  #9  0x011236db in exec_byte_code (bytestr=66621569, vector=62818821,
      maxdepth=32, args_template=56838170, nargs=0, args=0x0) at bytecode.c:785
  #10 0x01037c4e in funcall_lambda (fun=61992805, nargs=0, arg_vector=0x82e1b4)
      at eval.c:3166
  #11 0x0103712c in Ffuncall (nargs=1, args=0x82e1b0) at eval.c:2984
  #12 0x01034c3c in eval_sub (form=59285366) at eval.c:2255
  #13 0x01030431 in Fprogn (args=59285382) at eval.c:364
  #14 0x010302b1 in Fif (args=59285334) at eval.c:315
  #15 0x01034986 in eval_sub (form=59285310) at eval.c:2231
  #16 0x01030431 in Fprogn (args=59285390) at eval.c:364
  #17 0x01030389 in Fcond (args=59285398) at eval.c:342
  #18 0x01034986 in eval_sub (form=59284782) at eval.c:2231
  #19 0x01030431 in Fprogn (args=59309078) at eval.c:364
  #20 0x01031bb3 in FletX (args=59286038) at eval.c:983
  #21 0x01034986 in eval_sub (form=59285950) at eval.c:2231
  #22 0x01030431 in Fprogn (args=59309158) at eval.c:364
  #23 0x010320db in Fwhile (args=59285942) at eval.c:1075
  #24 0x01034986 in eval_sub (form=59285934) at eval.c:2231
  #25 0x01030431 in Fprogn (args=59309166) at eval.c:364
  #26 0x01032044 in Flet (args=59285606) at eval.c:1053
  #27 0x01034986 in eval_sub (form=59245286) at eval.c:2231
  #28 0x01030431 in Fprogn (args=62798582) at eval.c:364
  #29 0x01034986 in eval_sub (form=62798590) at eval.c:2231
  #30 0x01030291 in Fif (args=62798606) at eval.c:314
  #31 0x01034986 in eval_sub (form=62798614) at eval.c:2231
  #32 0x0103530e in eval_sub (form=59245270) at eval.c:2344
  #33 0x01030431 in Fprogn (args=59309182) at eval.c:364
  #34 0x01032044 in Flet (args=59245262) at eval.c:1053
  #35 0x01034986 in eval_sub (form=59245198) at eval.c:2231
  #36 0x01030431 in Fprogn (args=59308166) at eval.c:364
  #37 0x01037b1a in funcall_lambda (fun=59308190, nargs=1, arg_vector=0x82f1c0)
      at eval.c:3159
  #38 0x01037461 in apply_lambda (fun=59308198, args=59276566) at eval.c:3043
  #39 0x0103533d in eval_sub (form=59276590) at eval.c:2347
  #40 0x01030431 in Fprogn (args=59276558) at eval.c:364
  #41 0x01037b1a in funcall_lambda (fun=59275998, nargs=0, arg_vector=0x82f420)
      at eval.c:3159
  #42 0x01037461 in apply_lambda (fun=59275990, args=56838170) at eval.c:3043
  #43 0x0103533d in eval_sub (form=59187782) at eval.c:2347
  #44 0x0103263c in Funwind_protect (args=59187766) at eval.c:1304
  #45 0x01034986 in eval_sub (form=59187790) at eval.c:2231
  #46 0x01030431 in Fprogn (args=59181374) at eval.c:364
  #47 0x01032044 in Flet (args=59187798) at eval.c:1053
  #48 0x01034986 in eval_sub (form=59187830) at eval.c:2231
  #49 0x01030431 in Fprogn (args=59181366) at eval.c:364
  #50 0x010302b1 in Fif (args=59066374) at eval.c:315
  #51 0x01034986 in eval_sub (form=59066382) at eval.c:2231
  #52 0x01030431 in Fprogn (args=59210614) at eval.c:364
  #53 0x01037b1a in funcall_lambda (fun=59210590, nargs=0, arg_vector=0x82fb30)
      at eval.c:3159
  #54 0x01037461 in apply_lambda (fun=59210582, args=56838170) at eval.c:3043
  #55 0x0103533d in eval_sub (form=58576574) at eval.c:2347
  #56 0x0103450f in Feval (form=58576574, lexical=56838170) at eval.c:2137
  #57 0x01005123 in top_level_2 () at keyboard.c:1169
  #58 0x01032ab9 in internal_condition_case (bfun=0x1005107 <top_level_2>,
      handlers=56891826, hfun=0x1004cba <cmd_error>) at eval.c:1448
  #59 0x01005155 in top_level_1 (ignore=56838170) at keyboard.c:1177
  #60 0x0103247c in internal_catch (tag=56889874, func=0x1005125 <top_level_1>,
      arg=56838170) at eval.c:1205
  #61 0x01005090 in command_loop () at keyboard.c:1132
  #62 0x01004678 in recursive_edit_1 () at keyboard.c:759
  #63 0x0100499a in Frecursive_edit () at keyboard.c:823
  #64 0x010027c9 in main (argc=39, argv=0xa34058) at emacs.c:1711

  Lisp Backtrace:
  "re-search-forward" (0x82df44)
  "custom-make-dependencies" (0x82e1b4)
  "funcall" (0x82e1b0)
  "if" (0x82e430)
  "cond" (0x82e5c0)
  "let*" (0x82e790)
  "while" (0x82e930)
  "let" (0x82eb40)
  "progn" (0x82ec90)
  "if" (0x82ede0)
  "when" (0x82eef0)
  "let" (0x82f0f0)
  "command-line-1" (0x82f1c0)
  "command-line" (0x82f420)
  "unwind-protect" (0x82f6d0)
  "let" (0x82f8d0)
  "if" (0x82fa60)
  "normal-top-level" (0x82fb30)