Bootstrap problem.

Bootstrap problem.

[Lute Kamstra]

Consider the following situation.

Add a definition to lisp file A and make it available to other lisp
files by adding an autoload cookie.  Commit the change.  Then add some
code to lisp file B that depends on the presence of this definition to
compile.  Commit this change as well.

As I found out the hard way, this leads to problems in two cases:

1. Someone does a fresh checkout from CVS and does a make bootstrap.
   This fails because bootstrap-emacs that tries to compile file B
   uses the autoloads in ldefs-boot.el which wasn't updated and thus
   doesn't contain an autoload of the required definition in file A.

   This can, of course, be solved by regenerating ldefs-boot.el and
   committing it.  It's probably a good idea to mention this
   explicitly somewhere.

2. Someone has a CVS tree and did the last update before the change in
   file A.  Then that person does per next update after the change to
   file B and then does a make bootstrap.  bootstrap-emacs uses the
   old loaddefs.el that does not contain an autoload of the required
   definition in file A and fails to compile file B.

   A second make bootstrap would work fine as this would use the new
   loaddefs.el that was created during the first make bootstrap that
   failed.

Can't the bootstrap process be changed so that bootstrap-emacs uses
loaddefs.el with autoloads that are up to date?  This would make
ldefs-boot.el obsolete and one wouldn't have to think about
regenerating it under some circumstances.  It would also solve the
second problem.

Lute.

Re: Bootstrap problem.

[Kim F. Storm]

Lute Kamstra <Lute.Kamstra.lists@xs4all.nl> writes:

> Consider the following situation.
>
> Add a definition to lisp file A and make it available to other lisp
> files by adding an autoload cookie.  Commit the change.  Then add some
> code to lisp file B that depends on the presence of this definition to
> compile.  Commit this change as well.

I don't see why you cannot simply add

  (eval-when-compile
    (require 'A))

to B.  

Since A will be loaded anyway due to the autoload cookie, it doesn't
cost anything, and it makes it clear that B uses features of A.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Bootstrap problem.

[Lute Kamstra]

Lute Kamstra <Lute.Kamstra.lists@xs4all.nl> writes:

> Add a definition to lisp file A and make it available to other lisp
> files by adding an autoload cookie.  Commit the change.  Then add
> some code to lisp file B that depends on the presence of this
> definition to compile.  Commit this change as well.
>
> As I found out the hard way, this leads to problems in two cases:

[...]

> 2. Someone has a CVS tree and did the last update before the change in
>    file A.  Then that person does per next update after the change to
>    file B and then does a make bootstrap.  bootstrap-emacs uses the
>    old loaddefs.el that does not contain an autoload of the required
>    definition in file A and fails to compile file B.
>
>    A second make bootstrap would work fine as this would use the new
>    loaddefs.el that was created during the first make bootstrap that
>    failed.

Strange: I just looked more carefully at the bootstrap process and it
seems that this second problem should not occur.  From what I now
understand, bootstrap already does what I want: it first updates
loaddefs.el and then builds bootstrap-emacs and dumps it with the
up-to-date loaddefs.el loaded.

But now I don't know how to explain Chris Moore's recent bug report on
emacs-pretest-bug.

Lute.

Re: Bootstrap problem.

[Lute Kamstra]

Lute Kamstra <Lute.Kamstra.lists@xs4all.nl> writes:

> Lute Kamstra <Lute.Kamstra.lists@xs4all.nl> writes:
>
>> Add a definition to lisp file A and make it available to other lisp
>> files by adding an autoload cookie.  Commit the change.  Then add
>> some code to lisp file B that depends on the presence of this
>> definition to compile.  Commit this change as well.
>>
>> As I found out the hard way, this leads to problems in two cases:
>
> [...]
>
>> 2. Someone has a CVS tree and did the last update before the change in
>>    file A.  Then that person does per next update after the change to
>>    file B and then does a make bootstrap.  bootstrap-emacs uses the
>>    old loaddefs.el that does not contain an autoload of the required
>>    definition in file A and fails to compile file B.
>>
>>    A second make bootstrap would work fine as this would use the new
>>    loaddefs.el that was created during the first make bootstrap that
>>    failed.
>
> Strange: I just looked more carefully at the bootstrap process and it
> seems that this second problem should not occur.  From what I now
> understand, bootstrap already does what I want: it first updates
> loaddefs.el and then builds bootstrap-emacs and dumps it with the
> up-to-date loaddefs.el loaded.
>
> But now I don't know how to explain Chris Moore's recent bug report on
> emacs-pretest-bug.

I figured it out: a make clean deletes the emacs executable, but not
loaddefs.el.  So a subsequent make bootstrap cannot use the old emacs
to regenerate loaddefs.el and bootstrap-emacs uses the old loaddefs.el
to compile the new lisp files.

We could let make clean delete loaddefs.el as well.  Or would that
have any negative effects that I'm not aware of?

Lute.

Re: Bootstrap problem.

[Richard Stallman]

    We could let make clean delete loaddefs.el as well.  Or would that
    have any negative effects that I'm not aware of?

`make clean' should not delete loaddefs.el because that file is part
of the Emacs distribution.  However, if we want to change
bootstrapping to delete loaddefs.el at some stage, that's ok.  Just
don't put it inside of `make clean'.

bootstrap problem

[djh]

Eli, et al,

Here is the latest results attempting to make a boostrap with the latest cvs on cygwin 5.19. I attempted to run the line of code that broke (dumped the core) and repeat it.  See results below:

------
Compiling /cygdrive/c/emacs/cvs/emacs/lisp/emacs-lisp/byte-opt.el
Fatal error (6)/bin/sh: line 1:  1388 Aborted                 (core dumped) EMACSLOADPATH=/cygdrive/c/emacs/cvs/emacs/lisp ../src/bootstrap-emacs.exe -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done $el
make[2]: *** [compile] Error 1

------
Then by hand
$ cd ...cvs/emacs/src
$ export EMACSLOADPATH=/cygdrive/c/emacs/cvs/emacs/lisp
$ ./bootstrap-emacs.exe -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done byte-opt.el

Fatal error (6)zsh: abort (core dumped)  ./bootstrap-emacs.exe -batch --no-site-file --multibyte -f  byte-opt.el

Any ideas or advice to get more data to isolate the problem is appreciated.

Regards,
  Darel Henman

Re: bootstrap problem

[Eli Zaretskii]

> From: "djh" <henman@it.to-be.co.jp>
> Date: Thu, 25 May 2006 13:33:49 +0900
> cc: 
> 
> Then by hand
> $ cd ...cvs/emacs/src
> $ export EMACSLOADPATH=/cygdrive/c/emacs/cvs/emacs/lisp
> $ ./bootstrap-emacs.exe -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done byte-opt.el
> 
> Fatal error (6)zsh: abort (core dumped)  ./bootstrap-emacs.exe -batch --no-site-file --multibyte -f  byte-opt.el
> 
> Any ideas or advice to get more data to isolate the problem is appreciated.

Now that we know which command crashes, run it under GDB:

 $ cd ...cvs/emacs/src
 $ export EMACSLOADPATH=/cygdrive/c/emacs/cvs/emacs/lisp
 $ gdb ./bootstrap-emacs.exe
 $ run -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done byte-opt.el

Now when Emacs crashes, GDB will get control.  Then type these two
commands:

  (gdb) backtrace
  (gdb) xbacktrace

and tell what they print.

TIA

Re: bootstrap problem

[djh]

Eli et al,
  here is some more information about the cygwin 5.19 + x-window build status.

> Now that we know which command crashes, run it under GDB:
> 
>  $ cd ...cvs/emacs/src
>  $ export EMACSLOADPATH=/cygdrive/c/emacs/cvs/emacs/lisp
>  $ gdb ./bootstrap-emacs.exe
>  $ run -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done byte-opt.el
> 
 export EMACSLOADPATH=/cygdrive/c/emacs/cvs/emacs/lisp
prc[0]:cvs/emacs/src $ gdb ./bootstrap-emacs.exe
GNU gdb 6.3.50_2004-12-28-cvs (cygwin-special)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-cygwin"...
Environment variable "DISPLAY" not defined.
TERM = xterm
Breakpoint 1 at 0x200a1076: file emacs.c, line 464.
Breakpoint 2 at 0x200ba389: file sysdep.c, line 1395.
(gdb) run -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done byte-opt.el
Starting program: /cygdrive/c/emacs/cvs/emacs/src/bootstrap-emacs.exe -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done byte-opt.el
---Type <return> to continue, or q <return> to quit---

Program received signal SIGSEGV, Segmentation fault.
0x610ad945 in pthread_mutexattr_init () from /usr/bin/cygwin1.dll

----
> Now when Emacs crashes, GDB will get control.  Then type these two
> commands:
> 
>   (gdb) backtrace
>   (gdb) xbacktrace
> 
> and tell what they print.
> 

(gdb) backtrace
#0  0x610ad945 in pthread_mutexattr_init () from /usr/bin/cygwin1.dll
#1  0x6108dd7f in _sigfe () from /usr/bin/cygwin1.dll
#2  0x00000003 in ?? ()
#3  0x0022ee18 in ?? ()
#4  0x00000001 in ?? ()
#5  0x0022ee18 in ?? ()
#6  0x0022ee38 in ?? ()
#7  0x200a2100 in main (argc=539893664, argv=0x4) at emacs.c:1062

Lisp Backtrace:
0x0 Lisp type 0
Cannot access memory at address 0x0

xbacktrace
0x0 Lisp type 0
Cannot access memory at address 0x0

Regards,
  Darel Henman

Re: bootstrap problem

[Eli Zaretskii]

> Cc: "djh" <henman@it.to-be.co.jp>, <emacs-devel@gnu.org>
> From: "djh" <henman@it.to-be.co.jp>
> Date: Fri, 26 May 2006 13:47:58 +0900
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x610ad945 in pthread_mutexattr_init () from /usr/bin/cygwin1.dll

So Emacs does, indeed, crash.  However, your previous transcript
indicated it was a SIGABRT, not SIGSEGV:

> Fatal error (6)zsh: abort (core dumped)  ./bootstrap-emacs.exe -batch --no-site-file --multibyte -f  byte-opt.el

Signal 6 is SIGABRT, IIRC.

> (gdb) backtrace
> #0  0x610ad945 in pthread_mutexattr_init () from /usr/bin/cygwin1.dll
> #1  0x6108dd7f in _sigfe () from /usr/bin/cygwin1.dll
> #2  0x00000003 in ?? ()
> #3  0x0022ee18 in ?? ()
> #4  0x00000001 in ?? ()
> #5  0x0022ee18 in ?? ()
> #6  0x0022ee38 in ?? ()
> #7  0x200a2100 in main (argc=539893664, argv=0x4) at emacs.c:1062

Ouch!  Was your Emacs compiled with "-gdwarf-2 -g3" compiler switches?
If not, please reconfigure to use these and rebuild.  It will be very
hard to debug this without a reliable debug info.  Also, please make
sure you have the latest version of GDB available under Cygwin.  If
there's a version of cygwin1.dll with full debug info, please use that
as well in your next trials.

If none of the above helps to get a more meaningful backtrace, I'm
afraid you will need to ask on the Cygwin list for more help.  We need
some way of getting a meaningful backtrace, to know where Emacs
crashes, and I don't know enough about Cygwin internals to suggest how
to get it, except with the above advice.

> Lisp Backtrace:
> 0x0 Lisp type 0
> Cannot access memory at address 0x0
> 
> xbacktrace
> 0x0 Lisp type 0
> Cannot access memory at address 0x0

This is expected when C-level backtrace is garbled as above.

Thanks.

cygwn 5.19 with X-win bootstrap

[djh]

I don't know to be happy or not.  Where I was getting a 
> Signal 6 is SIGABRT, IIRC.
> 
> Ouch!  Was your Emacs compiled with "-gdwarf-2 -g3" compiler switches?
> If not, please reconfigure to use these and rebuild. 

(1) I did an update from cvs and then
(2) added the "-dgward-2 -g3" options as advised above like below:

export CFLAGS="-gdwarf-2 -g3"     in my install script.

ran the install script.  But, it ran succesfully!

Thanks
  Darel Henman

Re: cygwn 5.19 with X-win bootstrap

[Eli Zaretskii]

> Cc: <emacs-devel@gnu.org>
> From: "djh" <henman@it.to-be.co.jp>
> Date: Tue, 30 May 2006 16:03:19 +0900
> 
> (1) I did an update from cvs and then
> (2) added the "-dgward-2 -g3" options as advised above like below:
> 
> export CFLAGS="-gdwarf-2 -g3"     in my install script.
> 
> ran the install script.  But, it ran succesfully!

Do we have a Heisenbug?

What happens if you build with CFLAGS="-gdwarf-2 -g3 -O2"?

Re: cygwn 5.19 with X-win bootstrap

[djh]

Yes. It appears to be a Heisenbug.  But, the -O2 flushed it out again.
 
> Do we have a Heisenbug?
> 
> What happens if you build with CFLAGS="-gdwarf-2 -g3 -O2"?
> 

Again the backtrace didn't show much.  Does anyone wknow would to get a cygwin dll with debuggin information in it?

Regards
    Darel Henman

---------------- Result
Compiling /cygdrive/c/emacs/cvs/emacs/lisp/emacs-lisp/byte-opt.el
Fatal error (6)/bin/sh: line 1:  3712 Aborted       (core dumped) EMACSLOADPATH=/cygdrive/c/emacs/cvs/emacs/lisp ../src/bootstrap-emacs.exe -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done $el
make[2]: *** [compile] Error 1
make[2]: Leaving directory `/cygdrive/c/emacs/cvs/emacs/lisp'


# doing it by hand with gdb
cd $e/cvs/emacs/src
export EMACSLOADPATH=/cygdrive/c/emacs/cvs/emacs/lisp
gdb ./bootstrap-emacs.exe
Starting program: /cygdrive/c/emacs/cvs/emacs/src/bootstrap-emacs.exe -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done byte-opt.el
---Type <return> to continue, or q <return> to quit---  <RETURN>

Program received signal SIGSEGV, Segmentation fault.
0x610ad945 in pthread_mutexattr_init () from /usr/bin/cygwin1.dll
(gdb) backtrace
#0  0x610ad945 in pthread_mutexattr_init () from /usr/bin/cygwin1.dll
#1  0x6108dd7f in _sigfe () from /usr/bin/cygwin1.dll
#2  0x00000003 in ?? ()
#3  0x0022ee18 in ?? ()
#4  0x00000001 in ?? ()
#5  0x0022ee18 in ?? ()
#6  0x0022ee38 in ?? ()
#7  0x200a24d0 in main (argc=539893664, argv=0x4) at emacs.c:1062

Lisp Backtrace:
0x0 Lisp type 0
Cannot access memory at address 0x0

----------------  end of data

Re: cygwn 5.19 with X-win bootstrap

[Eli Zaretskii]

> Cc: <emacs-devel@gnu.org>
> From: "djh" <henman@it.to-be.co.jp>
> Date: Thu, 01 Jun 2006 13:56:47 +0900
> 
> Yes. It appears to be a Heisenbug.  But, the -O2 flushed it out again.

No, I think it's a bug that happens only with optimised code.

> Again the backtrace didn't show much.  Does anyone wknow would to get a cygwin dll with debuggin information in it?

What happens if you repeatedly type "step" until you find yourself in
some Emacs function?  Once you get to an Emacs function, "bt" should
work correctly, and even if it doesn't, we will know in what place
Emacs crashes.

Re: cygwn 5.19 with X-win bootstrap

[djh]

> From: Eli Zaretskii <eliz@gnu.org>
> .....
> No, I think it's a bug that happens only with optimised code.
> ....
> What happens if you repeatedly type "step" until you find yourself in
> some Emacs function?  Once you get to an Emacs function, "bt" should
> work correctly, and even if it doesn't, we will know in what place
> Emacs crashes.
> 

Here is the result:
Using gdb and stepping through: I was using the 'n' command and the <RET> to continue repeating it.

-------------------------------- start of gdb session 
cvs/emacs/lisp $   gdb ../src/bootstrap-emacs.exe
(gdb) b 855
Breakpoint 1 at 0x200a23c2: file emacs.c, line 855.
(gdb) run -batch --no-site-file --multibyte -f batch-byte-compile-if-not-done  byte-opt.el

858       if (!initialized)
(gdb) n
851       char *junk = 0;
(gdb)
858       if (!initialized)
(gdb)
883       sort_args (argc, argv);
(gdb)
884       argc = 0;
(gdb)
885       while (argv[argc]) argc++;
(gdb)
887       if (argmatch (argv, argc, "-version", "--version", 3, NULL, &skip_args)
(gdb)
1006      if (1
(gdb)
1021          newlim = re_max_failures * ratio + 200000;
(gdb)
1018          ratio += ratio / 3;
(gdb)
1021          newlim = re_max_failures * ratio + 200000;
(gdb)
1029          if (newlim > rlim.rlim_max)
(gdb)
1035          if (rlim.rlim_cur < newlim)
(gdb)
1036            rlim.rlim_cur = newlim;
(gdb)
1038          setrlimit (RLIMIT_STACK, &rlim);
(gdb)
1043      stack_bottom = &stack_bottom_variable;
(gdb)
1050      clearerr (stdin);
(gdb)
1054      memory_warnings (0, malloc_warning);
(gdb)
1050      clearerr (stdin);
(gdb)
1054      memory_warnings (0, malloc_warning);
(gdb)
1058      free (realloc (malloc (4), 4));
(gdb)
1062      uninterrupt_malloc ();
(gdb)

Program received signal SIGSEGV, Segmentation fault.
0x610ad945 in pthread_mutexattr_init () from /usr/bin/cygwin1.dll
(gdb)
Single stepping until exit from function pthread_mutexattr_init,
which has no line number information.
0x7c94eaf0 in ?? () from ntdll.dll
(gdb)
Cannot find bounds of current function

---------------------- end of gdb output  ----------

Re: cygwn 5.19 with X-win bootstrap

[Eli Zaretskii]

> Cc: <emacs-devel@gnu.org>
> From: "djh" <henman@it.to-be.co.jp>
> Date: Fri, 02 Jun 2006 18:03:27 +0900
> 
> 1062      uninterrupt_malloc ();
> (gdb)
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x610ad945 in pthread_mutexattr_init () from /usr/bin/cygwin1.dll
> (gdb)
> Single stepping until exit from function pthread_mutexattr_init,
> which has no line number information.
> 0x7c94eaf0 in ?? () from ntdll.dll
> (gdb)
> Cannot find bounds of current function

So it crashes inside pthread_mutexattr_init, which is called from
within uninterrupt_malloc.  Just to be sure, perhaps step into
uninterrupt_malloc and see whether my conclusion is correct, and which
of the two calls to pthread_mutexattr_init crashes.

If my conclusion is right, I think you need to take this up with
Cygwin maintainers.  Perhaps, if you show them the code in
uninterrupt_malloc that calls pthread functions, they will be able to
figure out the problem.

Thanks for looking into this.

Re: cygwn 5.19 with X-win bootstrap

[djh]

Eli, et al,
  sorry I was sent out of town for some testing and was unable to continue with testing cvs emasc with cygwin using x-widows.  I have taken up the matter with cygwin people and will upgrade to try using newer (not yet released versions) of cygwin and gdb
, which is reported be better.  The older versions reported SIGSEGV when it wasn't really the case....  

I report again when I find out more info.

Thanks for your support.

Darel Henman

Re:
> . ....
> So it crashes inside pthread_mutexattr_init, which is called from
> within uninterrupt_malloc.  Just to be sure, perhaps step into
> uninterrupt_malloc and see whether my conclusion is correct, and which
> of the two calls to pthread_mutexattr_init crashes.
> 
> If my conclusion is right, I think you need to take this up with
> Cygwin maintainers.  Perhaps, if you show them the code in
> uninterrupt_malloc that calls pthread functions, they will be able to
> figure out the problem.
> 
> Thanks for looking into this.
>