bootstrap not regenerating some *loaddefs

bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

[This is on Windows, but I think it happens in GNU/Linux and other
environments as well.]

After a maintaner-clean, I'm getting errors in bootstrap:

In toplevel form:
calc/calc-aent.el:30:1:Error: Cannot open load file: calc-loaddefs.el

In toplevel form:
eshell/em-alias.el:96:1:Error: Cannot open load file: esh-groups

etc.

AFAICS, the reason is that maintainer-clean deletes these files, but
bootstrap does not regenerate them it if ldefs-boot.el happens to be
up-to-date with respect to their sources. And, after the bootstrap,

  cd lisp
  make autoloads

doesn't either. The only answer is deleting loaddefs.el and doing
"make autoloads recompile" again.

Can someone think of a fix that does not involve too much trickery in
our already-tricky-enough makefiles?

     Juanma

Re: bootstrap not regenerating some *loaddefs

[Jason Rumney]

Juanma Barranquero wrote:
> doesn't either. The only answer is deleting loaddefs.el and doing
> "make autoloads recompile" again.
>   

Shouldn't make maintainer-clean delete loaddefs.el?

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Tue, Jun 17, 2008 at 17:13, Jason Rumney <jasonr@gnu.org> wrote:

> Shouldn't make maintainer-clean delete loaddefs.el?

It does. But bootstrapping copies ldefs-boot.el over it:

# Need separate version for sh and native cmd.exe
# Note that bootstrap-clean-$(SHELLTYPE) copies ldefs-boot.el to loaddefs.el,
# and thus the almost-empty loaddefs.el crafted by the $(lisp)/loaddefs.el
# target can NOT be built _after_ bootstrap-clean-$(SHELLTYPE) does its
# thing, or else an empty loaddefs.el will overwrite the full one.
bootstrap-clean: $(lisp)/loaddefs.el
        $(MAKE) $(MFLAGS) bootstrap-clean-$(SHELLTYPE)

bootstrap-clean-CMD:
#       if exist "$(EMACS)" $(MAKE) $(MFLAGS) autoloads
        cp $(lisp)/ldefs-boot.el $(lisp)/loaddefs.el
        -for %%f in (. $(WINS)) do for %%g in (%%f\*.elc) do @$(DEL) %%g

bootstrap-clean-SH:
#       if test -f "$(EMACS)"; then $(MAKE) $(MFLAGS) autoloads; fi
#       -rm -f $(lisp)/*.elc $(lisp)/*/*.elc
        cp $(lisp)/ldefs-boot.el $(lisp)/loaddefs.el
        -for dir in . $(WINS); do rm -f $$dir/*.elc; done

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

"Juanma Barranquero" wrote:

> On Tue, Jun 17, 2008 at 17:13, Jason Rumney <jasonr@gnu.org> wrote:
>
>> Shouldn't make maintainer-clean delete loaddefs.el?
>
> It does. But bootstrapping copies ldefs-boot.el over it:

Makefile.in does not do this any more. Can you make the w32 makefile
behave the same way?

Re: bootstrap not regenerating some *loaddefs

[Stefan Monnier]

> AFAICS, the reason is that maintainer-clean deletes these files, but
> bootstrap does not regenerate them it if ldefs-boot.el happens to be
> up-to-date with respect to their sources. And, after the bootstrap,

>   cd lisp
>   make autoloads

> doesn't either. The only answer is deleting loaddefs.el and doing
> "make autoloads recompile" again.

"make maintainer-clean" should remove lisp/loaddefs.el as well.
Doesn't it?  Or is it insufficient?


        Stefan

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Tue, Jun 17, 2008 at 18:38, Glenn Morris <rgm@gnu.org> wrote:

> Makefile.in does not do this any more. Can you make the w32 makefile
> behave the same way?

I suppose, but... what it is ldefs-boot.el used for, then?

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Tue, Jun 17, 2008 at 19:34, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> "make maintainer-clean" should remove lisp/loaddefs.el as well.
> Doesn't it?  Or is it insufficient?

It is currently insufficient, because during bootstrap (on Windows, at
least) there's always a loaddefs.el, which is a copy of ldefs-boot.el.
If updating that loaddefs.el does not need to update calc's autoloads,
calc/calc-loaddefs.el is not generated (AFAICS).

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

"Juanma Barranquero" wrote:

> On Tue, Jun 17, 2008 at 18:38, Glenn Morris <rgm@gnu.org> wrote:
>
>> Makefile.in does not do this any more. Can you make the w32 makefile
>> behave the same way?
>
> I suppose, but... what it is ldefs-boot.el used for, then?

See loadup.el. I think it will all work if you copy Stefan's
2008-04-11 lisp/Makefile.in change.

I think you are having a version of the problem summarized as "If it
hurts, don't do that":

http://lists.gnu.org/archive/html/emacs-devel/2008-04/msg00886.html

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Tue, Jun 17, 2008 at 19:46, Glenn Morris <rgm@gnu.org> wrote:

> See loadup.el. I think it will all work if you copy Stefan's
> 2008-04-11 lisp/Makefile.in change.

> I think you are having a version of the problem summarized as "If it
> hurts, don't do that":

Well, not exactly. I was not idly deleting calc-loaddefs.el just for
the fun of it, you know... I did "make maintainer-clean" & "make
bootstrap". There's nothing "don't do that" in that particular
sequence of actions.

Unless you're referring to copying ldefs-boot.el over loaddefs.el.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Stefan Monnier]

>>> Shouldn't make maintainer-clean delete loaddefs.el?
>> It does. But bootstrapping copies ldefs-boot.el over it:

> Makefile.in does not do this any more. Can you make the w32 makefile
> behave the same way?

Yes, please,


        Stefan

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Tue, Jun 17, 2008 at 19:46, Glenn Morris <rgm@gnu.org> wrote:

> See loadup.el. I think it will all work if you copy Stefan's
> 2008-04-11 lisp/Makefile.in change.

And BTW, I'd like to add my voice to Eli's, and ask everybody who
changes the (non-Windows) makefiles please to send to emacs-devel an
explanation of what the change does and why. That would help to keep
the Windows makefiles in synch.

http://lists.gnu.org/archive/html/emacs-devel/2008-05/msg00110.html

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Tue, Jun 17, 2008 at 19:46, Glenn Morris <rgm@gnu.org> wrote:

> I think it will all work if you copy Stefan's
> 2008-04-11 lisp/Makefile.in change.

No, it doesn't.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Stefan Monnier]

>> I think it will all work if you copy Stefan's
>> 2008-04-11 lisp/Makefile.in change.

> No, it doesn't.

Can you try and track down why it doesn't?
The way it works in non-w32 is that loadup.el loads ldefs-boot.el in
case loading loaddefs.el failed, so removing loaddefs.el should work
just as well as copying it from ldefs-boot.el.


        Stefan

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

> Can you try and track down why it doesn't?
> The way it works in non-w32 is that loadup.el loads ldefs-boot.el in
> case loading loaddefs.el failed, so removing loaddefs.el should work
> just as well as copying it from ldefs-boot.el.

The following patch works for me in the following situations:

 - bootstrapping a fresh checkout
 - bootstrapping an already existing workspace
 - "make maintaner-clean" & "make bootstrap install"

However, I'd like to hear from Eli before committing this, as it is
his code that I'm removing.

   Juanma



2008-06-18  Juanma Barranquero  <lekktu@gmail.com>

	* makefile.w32-in (bootstrap-clean-CMD, bootstrap-clean-SH):
	Don't copy ldefs-boot.el over loaddefs.el.
	(bootstrap-clean): Remove loaddefs.el and don't depend on it.
	(loaddefs.el-SH, loaddefs.el-CMD): Add coding cookie to loaddefs.el.



Index: lisp/makefile.w32-in
===================================================================
RCS file: /sources/emacs/emacs/lisp/makefile.w32-in,v
retrieving revision 1.81
diff -u -2 -r1.81 makefile.w32-in
--- lisp/makefile.w32-in	6 May 2008 08:05:36 -0000	1.81
+++ lisp/makefile.w32-in	18 Jun 2008 10:50:15 -0000
@@ -138,13 +138,15 @@
 	echo "(defvar cvs-global-menu nil)" >>$@
 	echo " " >> $@
-	echo ";;; Local Variables:" >> $@
-	echo ";;; version-control: never" >> $@
-	echo ";;; no-byte-compile: t" >> $@
-	echo ";;; no-update-autoloads: t" >> $@
-	echo ";;; End:" >> $@
+	echo ";; Local Variables:" >> $@
+	echo ";; version-control: never" >> $@
+	echo ";; no-byte-compile: t" >> $@
+	echo ";; no-update-autoloads: t" >> $@
+	echo ";; coding: utf-8" >> $@
+	echo ";; End:" >> $@
 	echo ";;; loaddefs.el ends here" >> $@

 loaddefs.el-CMD:
 	echo ;;; loaddefs.el --- automatically extracted autoloads> $@
+	echo ;;>> $@
 	echo ;;; Code:>> $@
 	echo (autoload 'define-minor-mode "easy-mmode")>> $@
@@ -156,9 +158,10 @@
 	echo (defvar cvs-global-menu nil)>> $@
 	echo. >> $@
-	echo ;;; Local Variables:>> $@
-	echo ;;; version-control: never>> $@
-	echo ;;; no-byte-compile: t>> $@
-	echo ;;; no-update-autoloads: t>> $@
-	echo ;;; End:>> $@
+	echo ;; Local Variables:>> $@
+	echo ;; version-control: never>> $@
+	echo ;; no-byte-compile: t>> $@
+	echo ;; no-update-autoloads: t>> $@
+	echo ;; coding: utf-8>> $@
+	echo ;; End:>> $@
 	echo ;;; loaddefs.el ends here>> $@

@@ -385,14 +388,10 @@

 # Need separate version for sh and native cmd.exe
-# Note that bootstrap-clean-$(SHELLTYPE) copies ldefs-boot.el to loaddefs.el,
-# and thus the almost-empty loaddefs.el crafted by the $(lisp)/loaddefs.el
-# target can NOT be built _after_ bootstrap-clean-$(SHELLTYPE) does its
-# thing, or else an empty loaddefs.el will overwrite the full one.
-bootstrap-clean: $(lisp)/loaddefs.el
+bootstrap-clean:
+	- $(DEL) $(lisp)/loaddefs.el
 	$(MAKE) $(MFLAGS) bootstrap-clean-$(SHELLTYPE)

 bootstrap-clean-CMD:
 #	if exist "$(EMACS)" $(MAKE) $(MFLAGS) autoloads
-	cp $(lisp)/ldefs-boot.el $(lisp)/loaddefs.el
 	-for %%f in (. $(WINS)) do for %%g in (%%f\*.elc) do @$(DEL) %%g

@@ -400,5 +399,4 @@
 #	if test -f "$(EMACS)"; then $(MAKE) $(MFLAGS) autoloads; fi
 #	-rm -f $(lisp)/*.elc $(lisp)/*/*.elc
-	cp $(lisp)/ldefs-boot.el $(lisp)/loaddefs.el
 	-for dir in . $(WINS); do rm -f $$dir/*.elc; done

Re: bootstrap not regenerating some *loaddefs

[Stefan Monnier]

>> Can you try and track down why it doesn't?
>> The way it works in non-w32 is that loadup.el loads ldefs-boot.el in
>> case loading loaddefs.el failed, so removing loaddefs.el should work
>> just as well as copying it from ldefs-boot.el.

> The following patch works for me in the following situations:

>  - bootstrapping a fresh checkout
>  - bootstrapping an already existing workspace
>  - "make maintaner-clean" & "make bootstrap install"

> However, I'd like to hear from Eli before committing this, as it is
> his code that I'm removing.

The patch seem to be doing the right thing, thank you,


        Stefan

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Wed, Jun 18, 2008 at 17:20, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> The patch seem to be doing the right thing

This one is better; it also removes some autoloads and defvars that
are currently added to loaddefs.el on Windows and which would be
unnecessary now.

   Juanma



2008-06-18  Juanma Barranquero  <lekktu@gmail.com>

	* makefile.w32-in (bootstrap-clean-CMD, bootstrap-clean-SH):
	Don't copy ldefs-boot.el over loaddefs.el.
	(bootstrap-clean): Remove loaddefs.el and don't depend on it.
	(loaddefs.el-SH, loaddefs.el-CMD): Don't add autoloads and defvars
	to loaddefs.el; they are not needed now.  Add coding cookie.


Index: lisp/makefile.w32-in
===================================================================
RCS file: /sources/emacs/emacs/lisp/makefile.w32-in,v
retrieving revision 1.81
diff -u -2 -r1.81 makefile.w32-in
--- lisp/makefile.w32-in	6 May 2008 08:05:36 -0000	1.81
+++ lisp/makefile.w32-in	18 Jun 2008 11:57:13 -0000
@@ -130,35 +130,24 @@
 	echo ";;; loaddefs.el --- automatically extracted autoloads" > $@
 	echo ";;" >> $@; echo ";;; Code:" >> $@
-	echo "(autoload 'define-minor-mode \"easy-mmode\")" >>$@
-	echo "(autoload 'define-ccl-program \"ccl\")" >>$@
-	echo "(autoload 'regexp-opt \"regexp-opt\")" >>$@
-	echo "(autoload 'string-to-list \"mule-util\")" >>$@
-	echo "(autoload 'define-derived-mode \"derived\")" >>$@
-	echo "(autoload 'encoded-kbd-mode \"encoded-kb\")" >>$@
-	echo "(defvar cvs-global-menu nil)" >>$@
 	echo "\f" >> $@
-	echo ";;; Local Variables:" >> $@
-	echo ";;; version-control: never" >> $@
-	echo ";;; no-byte-compile: t" >> $@
-	echo ";;; no-update-autoloads: t" >> $@
-	echo ";;; End:" >> $@
+	echo ";; Local Variables:" >> $@
+	echo ";; version-control: never" >> $@
+	echo ";; no-byte-compile: t" >> $@
+	echo ";; no-update-autoloads: t" >> $@
+	echo ";; coding: utf-8" >> $@
+	echo ";; End:" >> $@
 	echo ";;; loaddefs.el ends here" >> $@

 loaddefs.el-CMD:
 	echo ;;; loaddefs.el --- automatically extracted autoloads> $@
+	echo ;;>> $@
 	echo ;;; Code:>> $@
-	echo (autoload 'define-minor-mode "easy-mmode")>> $@
-	echo (autoload 'define-ccl-program "ccl")>> $@
-	echo (autoload 'regexp-opt "regexp-opt")>> $@
-	echo (autoload 'string-to-list "mule-util")>> $@
-	echo (autoload 'define-derived-mode "derived")>> $@
-	echo (autoload 'encoded-kbd-mode "encoded-kb")>> $@
-	echo (defvar cvs-global-menu nil)>> $@
 	echo.\f>> $@
-	echo ;;; Local Variables:>> $@
-	echo ;;; version-control: never>> $@
-	echo ;;; no-byte-compile: t>> $@
-	echo ;;; no-update-autoloads: t>> $@
-	echo ;;; End:>> $@
+	echo ;; Local Variables:>> $@
+	echo ;; version-control: never>> $@
+	echo ;; no-byte-compile: t>> $@
+	echo ;; no-update-autoloads: t>> $@
+	echo ;; coding: utf-8>> $@
+	echo ;; End:>> $@
 	echo ;;; loaddefs.el ends here>> $@

@@ -385,14 +374,10 @@

 # Need separate version for sh and native cmd.exe
-# Note that bootstrap-clean-$(SHELLTYPE) copies ldefs-boot.el to loaddefs.el,
-# and thus the almost-empty loaddefs.el crafted by the $(lisp)/loaddefs.el
-# target can NOT be built _after_ bootstrap-clean-$(SHELLTYPE) does its
-# thing, or else an empty loaddefs.el will overwrite the full one.
-bootstrap-clean: $(lisp)/loaddefs.el
+bootstrap-clean:
+	- $(DEL) $(lisp)/loaddefs.el
 	$(MAKE) $(MFLAGS) bootstrap-clean-$(SHELLTYPE)

 bootstrap-clean-CMD:
 #	if exist "$(EMACS)" $(MAKE) $(MFLAGS) autoloads
-	cp $(lisp)/ldefs-boot.el $(lisp)/loaddefs.el
 	-for %%f in (. $(WINS)) do for %%g in (%%f\*.elc) do @$(DEL) %%g

@@ -400,5 +385,4 @@
 #	if test -f "$(EMACS)"; then $(MAKE) $(MFLAGS) autoloads; fi
 #	-rm -f $(lisp)/*.elc $(lisp)/*/*.elc
-	cp $(lisp)/ldefs-boot.el $(lisp)/loaddefs.el
 	-for dir in . $(WINS); do rm -f $$dir/*.elc; done

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

"Juanma Barranquero" wrote:

> +	echo ";; Local Variables:" >> $@
> +	echo ";; version-control: never" >> $@
> +	echo ";; no-byte-compile: t" >> $@
> +	echo ";; no-update-autoloads: t" >> $@
> +	echo ";; coding: utf-8" >> $@
> +	echo ";; End:" >> $@

Do you need to create a basic loaddefs by hand for some special
reason, or can you just rely on autoload-ensure-default-file, like the
Unix build does?

Re the W32 lisp makefile, I would suggest syncing the autogen-clean
target and its use (it is referenced in INSTALL.CVS).

A quick glance shows that the w32 makefile looks a bit out of date, eg
re mh-autoloads, emacs-deps (I can't understand this one personally),
custom-deps and finder-data prereqs, cus-load target, etc. A more
general sync/update may be good. Whether you want to copy the ELCFILES
stuff that allows for parallel byte-compiling, I leave up to you...

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Wed, Jun 18, 2008 at 18:47, Glenn Morris <rgm@gnu.org> wrote:

> Do you need to create a basic loaddefs by hand for some special
> reason, or can you just rely on autoload-ensure-default-file, like the
> Unix build does?

I don't know, it's been that way for a long time (at least since
2000-09-14), and at 30 minutes each bootstrap I'm not very inclined to
try to "fix it", truth be told (other than the current breakage, of
course).

> Re the W32 lisp makefile, I would suggest syncing the autogen-clean
> target and its use (it is referenced in INSTALL.CVS).
>
> A quick glance shows that the w32 makefile looks a bit out of date, eg
> re mh-autoloads, emacs-deps (I can't understand this one personally),
> custom-deps and finder-data prereqs, cus-load target, etc. A more
> general sync/update may be good. Whether you want to copy the ELCFILES
> stuff that allows for parallel byte-compiling, I leave up to you...

I don't think changing around the Windows makefiles for no reason is a
good idea. Revamping the whole makefile stuff on Windows would be
good, perhaps (Eli has said several times that the MS-DOS method of
generating the makefiles from the generic unixy ones via sed could
work). But it is quite a task waiting for a brave soul. I personally
have headaches every time I have to fix a problem in a makefile.

    Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Wed, 18 Jun 2008 16:23:27 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: Glenn Morris <rgm@gnu.org>, Jason Rumney <jasonr@gnu.org>,
> 	emacs-devel Mailinglist <emacs-devel@gnu.org>
> 
> However, I'd like to hear from Eli before committing this, as it is
> his code that I'm removing.

I see nothing harmful in your changes (including the ones you posted
in a follow-up), and have no special sentiments for my code.  However,
please test the result under `make -j 2 XMFLAGS="-j 2" bootstrap', to
make sure they don't break parallel builds.

And thanks for working on this.

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Wed, 18 Jun 2008 18:59:49 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: Jason Rumney <jasonr@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>,
> 	emacs-devel Mailinglist <emacs-devel@gnu.org>
> 
> I don't think changing around the Windows makefiles for no reason is a
> good idea.

Yes, if it ain't broken, don't fix it.  Especially since the
Makefile's in lib-src and src are also based on an old version of Unix
Makefile's, and so direct copy is not possible without also changing
those two quite heavily.

> Revamping the whole makefile stuff on Windows would be good, perhaps
> (Eli has said several times that the MS-DOS method of generating the
> makefiles from the generic unixy ones via sed could work).

It would work, but it has a disadvantage of requiring Sed for building
Emacs.  We don't require that currently.

> But it is quite a task waiting for a brave soul.

And a rainy day ;-)

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Wed, Jun 18, 2008 at 20:12, Eli Zaretskii <eliz@gnu.org> wrote:

> and have no special sentiments for my code.

Shame on you, you're breaking its little ASCII heart by saying that...

> However,
> please test the result under `make -j 2 XMFLAGS="-j 2" bootstrap', to
> make sure they don't break parallel builds.

Aha, good idea. Will do (it'll take a while).

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Miles Bader]

"Juanma Barranquero" <lekktu@gmail.com> writes:
>> and have no special sentiments for my code.
>
> Shame on you, you're breaking its little ASCII heart by saying that...

</3 :-(

-miles

-- 
Opportunity, n. A favorable occasion for grasping a disappointment.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

>> However,
>> please test the result under `make -j 2 XMFLAGS="-j 2" bootstrap', to
>> make sure they don't break parallel builds.
>
> Aha, good idea. Will do (it'll take a while).

Well, I've tested it with parallel builds and, *apparently* it works
OK. I say "apparently" because the parallel build needs sh, and the
sh.exe I have is not in good speaking terms with the rest of the tools
I use for building Emacs. So I wouldn't rule out that it has worked
without anything particularly "parallel" going on.

Could someone who regularly does parallel builds on Windows please try
the patch?

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Thu, 19 Jun 2008 02:36:15 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> the parallel build needs sh

Why do you think so?  Unless I'm missing something, that's not true:
"make -j" should work with cmd.exe as well.

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

"Juanma Barranquero" wrote:

> I don't think changing around the Windows makefiles for no reason is a
> good idea.

a) I doubt the changes to lisp/Makefile.in were made for no reason.

b) minimizing the differences between the Unix and Windows Makefiles
is desirable in itself.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Thu, Jun 19, 2008 at 05:15, Eli Zaretskii <eliz@gnu.org> wrote:

> Why do you think so?  Unless I'm missing something, that's not true:
> "make -j" should work with cmd.exe as well.

I don't have the log (I deleted it), but I did a maintainer-clean
followed by a bootstrap and I received messages that told clearly that
-j needed sh.exe.

Could be a thing of the make I use (the one in UnxUtils).

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Thu, Jun 19, 2008 at 08:23, Glenn Morris <rgm@gnu.org> wrote:

> a) I doubt the changes to lisp/Makefile.in were made for no reason.

How does that follow, or relate, to what I said? (about changing the
*Windows makefiles*.)

> b) minimizing the differences between the Unix and Windows Makefiles
> is desirable in itself.

Please, don't hesitate to do it. If you send the patches to
emacs-devel, I'll test them with my usual build setup.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Thu, Jun 19, 2008 at 10:23, Juanma Barranquero <lekktu@gmail.com> wrote:

> Could be a thing of the make I use (the one in UnxUtils).

I've tried bootstrapping with an assortment of tools (including the
make and sh from MSys, etc.) but I cannot get the parallel build to
work, even without my patch.

So I'd be grateful if someone could try it with a parallel build and
report whether it works OK.

  Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Thu, 19 Jun 2008 10:23:20 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> I don't have the log (I deleted it), but I did a maintainer-clean
> followed by a bootstrap and I received messages that told clearly that
> -j needed sh.exe.
> 
> Could be a thing of the make I use (the one in UnxUtils).

That's it: it's probably a very old version of Make.  Make 3.81
supports -j with any shell.  It also builds out of the box with MinGW,
so you don't need to keep using the one from UnxUtils.  I can even
send you a couple of patches that fix some problems in v3.81.

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Thu, 19 Jun 2008 12:53:17 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Thu, Jun 19, 2008 at 10:23, Juanma Barranquero <lekktu@gmail.com> wrote:
> 
> > Could be a thing of the make I use (the one in UnxUtils).
> 
> I've tried bootstrapping with an assortment of tools (including the
> make and sh from MSys, etc.) but I cannot get the parallel build to
> work, even without my patch.

How does it fail?  Or is the failure that it doesn't run in parallel?
If so, simply get the official distro of GNU Make 3.81 and build it
with MinGW, it doesn't need anything except the compiler and Binutils
to be built.

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

"Juanma Barranquero" wrote:

> On Thu, Jun 19, 2008 at 08:23, Glenn Morris <rgm@gnu.org> wrote:
>
>> a) I doubt the changes to lisp/Makefile.in were made for no reason.
>
> How does that follow, or relate, to what I said? (about changing the
> *Windows makefiles*.)

Because the changes I refered to should be independent of whether
using Unix or Windows, since the emacs lisp files and their
dependencies are independent of arch. The only differences should be
the Windows-specific shell commands (or whatever). For example, from a
quick glance at makefile.w32-in:

CVSREAD=1 should be handled on Windows, if the concept is applicable
there.

It is claimed that everything that uses emacs during build must depend
on emacs-deps.

`bootstrap-clean' should run autogen-clean.

autogen-clean should exist on all archs, since it is referred to in
INSTALL.CVS.

You should avoid creating a base loaddefs by hand on windows, because
it should not be necessary, and otherwise someone will have to
remember to update the makefile rule every time the version in
autoload.el is changed.

finder-data and custom-deps should not depend on $(lisp)/loaddefs.el
because it should not be necessary now, and because if they do it can
cause a redumping of Emacs.

mh-autoloads should not be listed as a prereq of anything, because the
normal autoloads rule takes care of it.

recompile should depend on LOADDEFS, not just mh-autoloads.

People on Windows may want to take advantage of parallel
byte-compiling just as much as those on Unix.

etc.


It's basically just a case of syncing the makefile.w32-in with
Makefile.in.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Thu, Jun 19, 2008 at 21:13, Glenn Morris <rgm@gnu.org> wrote:

> Because the changes I refered to should be independent of whether
> using Unix or Windows, since the emacs lisp files and their
> dependencies are independent of arch.

That does not answer my question.

> You should avoid creating a base loaddefs by hand on windows, because
> it should not be necessary, and otherwise someone will have to
> remember to update the makefile rule every time the version in
> autoload.el is changed.

I agree that "it should not be necessary", but why was it deemed
necessary would perhaps be better answered by the developer that
implemented that on Windows in the first place.

> It's basically just a case of syncing the makefile.w32-in with
> Makefile.in.

As I said: don't let my total lack of interest in fixing the makefiles
stop you from doing it.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Stefan Monnier]

>> You should avoid creating a base loaddefs by hand on windows, because
>> it should not be necessary, and otherwise someone will have to
>> remember to update the makefile rule every time the version in
>> autoload.el is changed.

> I agree that "it should not be necessary", but why was it deemed
> necessary would perhaps be better answered by the developer that
> implemented that on Windows in the first place.

It used to be done in Unix as well.  It's just that the w32 makefile did
not stay in sync with the Makefile.in.


        Stefan

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Thu, Jun 19, 2008 at 21:02, Eli Zaretskii <eliz@gnu.org> wrote:

> How does it fail?

Well, at the moment, using recent MinGW and MSYS, a parallel build
(with or without sh.exe) literally crashes my laptop...

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

"Juanma Barranquero" wrote:

>> It's basically just a case of syncing the makefile.w32-in with
>> Makefile.in.
>
> As I said: don't let my total lack of interest in fixing the makefiles
> stop you from doing it.

I was just trying to comply with the request to "post here a message
saying that it needs to be done and explaining the reasons for the
change". I don't feel responsible for the Windows makefiles and am not
going to do any more.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 06:17, Glenn Morris <rgm@gnu.org> wrote:

> I was just trying to comply with the request to "post here a message
> saying that it needs to be done and explaining the reasons for the
> change".

No. You're laying a plan about how could the makefiles be changed,
while the request was to please explain changes that are introduced.

  "I've added a new target XXX and the existing target YYY has a new
dependency on ZZZ because that's needed with parallel builds."

vs.

  "The way to go is for the Windows people to rebuild the Windows
makefiles from the ground up so they're identical to the Unix ones."

Do you really fail to see the difference?

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Jason Rumney]

Juanma Barranquero wrote:

> I agree that "it should not be necessary", but why was it deemed
> necessary would perhaps be better answered by the developer that
> implemented that on Windows in the first place.
>   

It's a recent change to Makefile.in that has not been made to 
makefile.w32-in. It was necessary before because it was necessary on all 
platforms.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 09:54, Jason Rumney <jasonr@gnu.org> wrote:

> It's a recent change to Makefile.in that has not been made to
> makefile.w32-in. It was necessary before because it was necessary on all
> platforms.

Getting rid of that difference would be a good idea, then. One of many
changes that would've been easier at the time...

But first we should at least commit the changes I've done (or a
parallel-friendly version of them, assuming they are not, which I
don't know, because I can't try them in a parallel build), as they
solve a specific problem.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 00:44:54 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Thu, Jun 19, 2008 at 21:02, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > How does it fail?
> 
> Well, at the moment, using recent MinGW and MSYS, a parallel build
> (with or without sh.exe) literally crashes my laptop...

At which step does it crash?

Also, what exactly from MSYS do you use for the build?  If you are not
using Bash, then what else is needed from there?  (I thought nothing
was needed if cmd.exe is used as the shell.)

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Thu, 19 Jun 2008 15:13:18 -0400
> Cc: Jason Rumney <jasonr@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>,
> 	emacs-devel Mailinglist <emacs-devel@gnu.org>
> 
> "Juanma Barranquero" wrote:
> 
> > On Thu, Jun 19, 2008 at 08:23, Glenn Morris <rgm@gnu.org> wrote:
> >
> >> a) I doubt the changes to lisp/Makefile.in were made for no reason.
> >
> > How does that follow, or relate, to what I said? (about changing the
> > *Windows makefiles*.)
> 
> Because the changes I refered to should be independent of whether
> using Unix or Windows, since the emacs lisp files and their
> dependencies are independent of arch. The only differences should be
> the Windows-specific shell commands (or whatever).

Well, those ``only differences'' _are_ the main source of the
difficulty, and the main reason why people prefer not to mess with the
Windows Makefiles, as long as they are working.

> CVSREAD=1 should be handled on Windows, if the concept is applicable
> there.

Even such a seemingly easy change has subtle dark corners, and if done
haphazardly, will break the build.  Windows doesn't come with `chmod'
out of the box, so we either need to require it for the Windows build
(which is in general bad, since that makes it harder for Windows users
to build Emacs), or use Windows equivalents.  But to use Windows
equivalents, one must first to:

  . make sure the equivalent command is supported on all Windows
    version, since Windows 95 to the latest Windows Vista;

  . make sure the syntax and semantics of the equivalent command is
    the same, no matter from what Windows version it comes and what
    Windows shell (command.com or cmd.exe) is used, because as the
    Windows shells evolve, the add and sometimes remove features;

  . make sure that if a ported Unixy shell is used, the equivalent
    command will not be misinterpreted by the shell, because it is
    already some shell built-in (for example, `type' is a Windows
    command to dump file's contents to the screen, akin to `cat' on
    Unix, but is something very different to Bash).

So each such change more often than not turns out to be a small
research project, which explains our reluctance to make changes that
are not strictly required.

> finder-data and custom-deps should not depend on $(lisp)/loaddefs.el
> because it should not be necessary now, and because if they do it can
> cause a redumping of Emacs.

Why is it ``not necessary''?  Don't these targets cause Emacs to read
all the *.el files anymore?

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 10:45, Eli Zaretskii <eliz@gnu.org> wrote:

> At which step does it crash?

Difficult to say, I don't have a log. However, I've been able to do a
parallel build now on another computer (with my lisp/makefile.w32-in
patches, BTW). I think the problems might be related to erroneously
doing

make -j 2 XMFLAGS="-j 2" bootstrap install

instead of as two different steps

make -j 2 XMFLAGS="-j 2" bootstrap
make -j 2 XMFLAGS="-j 2" install

but I won't be able to retry on the laptop until monday or so.

> Also, what exactly from MSYS do you use for the build?  If you are not
> using Bash, then what else is needed from there?  (I thought nothing
> was needed if cmd.exe is used as the shell.)

I never use bash as a shell, always cmd. But if sh.exe is in the path,
the MSYS make tries to use it.

I'm currently upgrading MSYS to "Technology Preview: MSYS-1.0.11", and
will remove sh.exe from the path and see whether I can get a parallel
build without bash.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 11:27:10 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> > Also, what exactly from MSYS do you use for the build?  If you are not
> > using Bash, then what else is needed from there?  (I thought nothing
> > was needed if cmd.exe is used as the shell.)
> 
> I never use bash as a shell, always cmd. But if sh.exe is in the path,
> the MSYS make tries to use it.

That's true for the native Windows build of Make as well.  When I want
to test with cmd.exe, I rename sh.exe to something else to get it out
of the way.  (The alternative is to have "SHELL = cmd" in the
Makefile, but I don't want to change them.)

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 11:27:10 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Fri, Jun 20, 2008 at 10:45, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > At which step does it crash?
> 
> Difficult to say, I don't have a log. However, I've been able to do a
> parallel build now on another computer (with my lisp/makefile.w32-in
> patches, BTW). I think the problems might be related to erroneously
> doing
> 
> make -j 2 XMFLAGS="-j 2" bootstrap install
> 
> instead of as two different steps
> 
> make -j 2 XMFLAGS="-j 2" bootstrap
> make -j 2 XMFLAGS="-j 2" install

I don't see why the former should crash, so it would be good to see a
log.

Re: bootstrap not regenerating some *loaddefs

[Jason Rumney]

Eli Zaretskii wrote:
> That's true for the native Windows build of Make as well.  When I want
> to test with cmd.exe, I rename sh.exe to something else to get it out
> of the way.  (The alternative is to have "SHELL = cmd" in the
> Makefile, but I don't want to change them.)
>   

You can also put that on the command line:

  make SHELL=cmd

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 11:03:35 +0100
> From: Jason Rumney <jasonr@gnu.org>
> CC: Juanma Barranquero <lekktu@gmail.com>, emacs-devel@gnu.org
> 
> Eli Zaretskii wrote:
> > That's true for the native Windows build of Make as well.  When I want
> > to test with cmd.exe, I rename sh.exe to something else to get it out
> > of the way.  (The alternative is to have "SHELL = cmd" in the
> > Makefile, but I don't want to change them.)
> >   
> 
> You can also put that on the command line:
> 
>   make SHELL=cmd

Did you really tried that, with sh.exe on PATH, and if so, did it
really work?  I think this didn't work in the Windows port of the
official release of Make 3.81 (I have it patched since then), that's
why I didn't mention this in my mail.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 11:50, Eli Zaretskii <eliz@gnu.org> wrote:

> I don't see why the former should crash, so it would be good to see a
> log.

I'll try to get a log next week, though I'm dragging my feet at the
idea of willingly inflicting a crash upon my Windows XP...

  Juanma

Re: bootstrap not regenerating some *loaddefs

[Jason Rumney]

Eli Zaretskii wrote:
> Did you really tried that, with sh.exe on PATH, and if so, did it
> really work?  I think this didn't work in the Windows port of the
> official release of Make 3.81 (I have it patched since then), that's
> why I didn't mention this in my mail.
>   

It used to work for me. I don't have sh.exe in my path anymore to test 
if it still works though.

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 11:27:10 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> I'm currently upgrading MSYS to "Technology Preview: MSYS-1.0.11", and
> will remove sh.exe from the path and see whether I can get a parallel
> build without bash.

It bootstrapped with "-j 2" for me, using sh.exe.  So I think you can
commit your changes.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 12:48, Eli Zaretskii <eliz@gnu.org> wrote:

> It bootstrapped with "-j 2" for me, using sh.exe.  So I think you can
> commit your changes.

OK.

I'll wait for the bootstrap I'm currently running with the MSYS
technology preview, though.

Two interesting points:

  C:\> make --help
  Usage: make [options] [target] ...
  Options:
    [... lots of options ...]
    --unix                      Run in UNIX mode (use sh.exe subshell).
    --win32                     Run in Win32 mode (use Win32 subshell).

And, when run with --win32, I can see messages like this:

  make - --win32 - --jobserver-fds=3,4 -j -j 2 -C ../lib-src cleanall
  make[1]: Entering directory `/c/emacs/trunk/lib-src'
  make[1]: warning: -jN forced in submake: disabling jobserver mode.

so it seems that it is now trying to do the right thing wrt -j and submakes.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

> OK.

s/\.$/, thanks for testing./;

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 13:10:59 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Fri, Jun 20, 2008 at 12:48, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > It bootstrapped with "-j 2" for me, using sh.exe.  So I think you can
> > commit your changes.
> 
> OK.

It's bootstrapping now with cmd.exe, and is looking fine so far.

>   C:\> make --help
>   Usage: make [options] [target] ...
>   Options:
>     [... lots of options ...]
>     --unix                      Run in UNIX mode (use sh.exe subshell).
>     --win32                     Run in Win32 mode (use Win32 subshell).
> 
> And, when run with --win32, I can see messages like this:
> 
>   make - --win32 - --jobserver-fds=3,4 -j -j 2 -C ../lib-src cleanall
>   make[1]: Entering directory `/c/emacs/trunk/lib-src'
>   make[1]: warning: -jN forced in submake: disabling jobserver mode.
> 
> so it seems that it is now trying to do the right thing wrt -j and submakes.

Yes, but.  MSYS is actually a fork of Cygwin, and behaves very similar
to it.  Specifically, in this case it supports the jobserver method of
running parallel jobs, which is not yet implemented in the native
Windows build.  So with MSYS Make, you don't need to use the XMFLAGS
trick (although using it shouldn't hurt).

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 14:02, Eli Zaretskii <eliz@gnu.org> wrote:

> It's bootstrapping now with cmd.exe, and is looking fine so far.

Could you please explain, if you have the time, the exact
configuration you're using to build Emacs? (Which tools, versions,
etc.)

> Yes, but.  MSYS is actually a fork of Cygwin, and behaves very similar
> to it.  Specifically, in this case it supports the jobserver method of
> running parallel jobs, which is not yet implemented in the native
> Windows build.  So with MSYS Make, you don't need to use the XMFLAGS
> trick (although using it shouldn't hurt).

Well, I've been unable to bootstrap Emacs, parallel or not, using
--win32 aniway. It does work, though, using --unix.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 15:24:21 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Fri, Jun 20, 2008 at 14:02, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > It's bootstrapping now with cmd.exe, and is looking fine so far.
> 
> Could you please explain, if you have the time, the exact
> configuration you're using to build Emacs? (Which tools, versions,
> etc.)

    D:\gnu\emacs>gcc --version
    gcc (GCC) 3.4.2 (mingw-special)
    Copyright (C) 2004 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


    D:\gnu\emacs>ld --version
    GNU ld version 2.15.91 20040904
    Copyright 2002 Free Software Foundation, Inc.
    This program is free software; you may redistribute it under the terms of
    the GNU General Public License.  This program has absolutely no warranty.

    D:\usr\eli\data>sh -c "echo $ZSH_VERSION"
    3.0.5-nt-beta-0.90

    D:\gnu\emacs>"echo.exe" --version
    echo (GNU coreutils) 5.3.0
    Written by FIXME unknown.

    Copyright (C) 2005 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    D:\gnu\emacs>cp --version
    cp (GNU coreutils) 5.3.0
    Written by Torbjorn Granlund, David MacKenzie, and Jim Meyering.

    Copyright (C) 2005 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    D:\gnu\emacs>rm --version
    rm (GNU coreutils) 5.3.0
    Written by Paul Rubin, David MacKenzie, Richard Stallman, and Jim Meyering.

    Copyright (C) 2005 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    D:\gnu\emacs>make --version
    GNU Make 3.81
    Copyright (C) 2006  Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.
    There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
    PARTICULAR PURPOSE.

    This program built for Windows32

I have MinGW runtime v3.14 and W32 API v3.11.

Make is patched by me, so it's not stock 3.81 (but none of the patches
is related to building Emacs, so it shouldn't matter).  sh.exe is an
old port of zsh by Amol Deshpande, which I patched to fix a few bad
bugs.  Unfortunately, it only compiles with a Microsoft compiler, I
never had time to express its fork emulation in a way that GCC and ld
will grok.

To test a CMD build, I just rename sh.exe to something else, as I
already wrote.

I think that's it.  If you need to know something else, please ask.

> > Yes, but.  MSYS is actually a fork of Cygwin, and behaves very similar
> > to it.  Specifically, in this case it supports the jobserver method of
> > running parallel jobs, which is not yet implemented in the native
> > Windows build.  So with MSYS Make, you don't need to use the XMFLAGS
> > trick (although using it shouldn't hurt).
> 
> Well, I've been unable to bootstrap Emacs, parallel or not, using
> --win32 aniway. It does work, though, using --unix.

You shouldn't need to use --win32.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 16:29, Eli Zaretskii <eliz@gnu.org> wrote:

>    D:\gnu\emacs>gcc --version
>    gcc (GCC) 3.4.2 (mingw-special)

>    D:\gnu\emacs>ld --version
>    GNU ld version 2.15.91 20040904

> I have MinGW runtime v3.14 and W32 API v3.11.

I use a newer GCC and ld, which shouldn't be a problem, and identical
runtime and API.

> I think that's it.  If you need to know something else, please ask.

I'll try to reproduce that setup (I'd like to get the parallel builds
working, if only to be able to test it).

Do you compile the coreutils (echo, cp and rm) from sources, or are
they from some distribution (Gnuwin32, MSYS, etc.)?

> You shouldn't need to use --win32.

I wanted to know if using --win32 would make renaming sh.exe unnecessary.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 15:24:21 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Fri, Jun 20, 2008 at 14:02, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > It's bootstrapping now with cmd.exe, and is looking fine so far.

It succeeded, btw.  So I think you definitely should commit the
changes.

Thanks.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 17:07, Eli Zaretskii <eliz@gnu.org> wrote:

> It succeeded, btw.  So I think you definitely should commit the
> changes.

Committed.

Thanks for testing.

   Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Fri, 20 Jun 2008 17:06:56 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> Do you compile the coreutils (echo, cp and rm) from sources, or are
> they from some distribution (Gnuwin32, MSYS, etc.)?

All of them are binaries from Gnuwin32 distribution.  I made my own
port of `ls' (because the one from the Gnuwin32 distro is unbelievably
slow, due to a misfeature in the ported glibc they use), but `ls' is
not used for building Emacs.

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

Eli Zaretskii wrote:

>> finder-data and custom-deps should not depend on $(lisp)/loaddefs.el
>> because it should not be necessary now, and because if they do it can
>> cause a redumping of Emacs.
>
> Why is it ``not necessary''?  Don't these targets cause Emacs to read
> all the *.el files anymore?

I went to the trouble of putting detailed comments and references in
lisp/Makefile.in, please read them.

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

"Juanma Barranquero" wrote:

>   "I've added a new target XXX and the existing target YYY has a new
> dependency on ZZZ because that's needed with parallel builds."

If you won't take my word for it, and if the code, comments, and
ChangeLogs are insufficient, feel free to ask specific questions.

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: lekktu@gmail.com,  jasonr@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Fri, 20 Jun 2008 13:46:28 -0400
> 
> Eli Zaretskii wrote:
> 
> >> finder-data and custom-deps should not depend on $(lisp)/loaddefs.el
> >> because it should not be necessary now, and because if they do it can
> >> cause a redumping of Emacs.
> >
> > Why is it ``not necessary''?  Don't these targets cause Emacs to read
> > all the *.el files anymore?
> 
> I went to the trouble of putting detailed comments and references in
> lisp/Makefile.in, please read them.

Please trust me that I did so before asking the question.  I simply
don't understand what you are saying there, perhaps because most of
that comment doesn't describe the present solution, but rather
different alternatives that were not taken.

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

Eli Zaretskii wrote:

> Please trust me that I did so before asking the question.  I simply
> don't understand what you are saying there, perhaps because most of
> that comment doesn't describe the present solution, but rather
> different alternatives that were not taken.

Sorry both for doubting you and for being unclear. :)

The reason for making custom-deps and finder-data depend on loaddefs
was that these rules used to scan _all_ the *.el files. This included
loaddefs and friends. In a parallel make, loaddefs etc could be
changed while custom-deps/finder-data were trying to read them. (I
think you know this bit.)

Since there is no useful information for custom-deps/finder-data in
loaddefs etc, I made custom-make-dependencies and
finder-compile-keywords-make-dist not scan these files any more (see
finder-no-scan-regexp etc). So to answer your original question,
custom-deps/finder-data indeed no longer scan _all_ *.el files, so
should no longer care if loaddefs etc are modified while they run.

Making these rules depend on loaddefs had undesirable side effects, as
described in the comments, so it was good to remove this.

I have tested this with `make -j4 updates', the source of the original
problem.

Re: bootstrap not regenerating some *loaddefs

[Stefan Monnier]

> loaddefs and friends. In a parallel make, loaddefs etc could be
> changed while custom-deps/finder-data were trying to read them. (I

BTW, why is that a problem?  Since those files contain nothing useful
w.r.t the function that scans them, I'd expect that whichever
intermediate state they happen to see would still result in the
same nothingness.  Or was it a problem of locking?


        Stefan

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

Stefan Monnier wrote:

>> loaddefs and friends. In a parallel make, loaddefs etc could be
>> changed while custom-deps/finder-data were trying to read them.
>
> BTW, why is that a problem?  Since those files contain nothing useful
> w.r.t the function that scans them, I'd expect that whichever
> intermediate state they happen to see would still result in the
> same nothingness.

Apparently:

http://lists.gnu.org/archive/html/emacs-pretest-bug/2007-01/msg00472.html

   To read a file emacs does a stat syscall followed by an open call.
   In the parallel make case it is possible that loaddefs.el gets
   deleted in between those 2 calls...

Re: bootstrap not regenerating some *loaddefs

[dhruva]

On Fri, Jun 20, 2008 at 8:36 PM, Juanma Barranquero <lekktu@gmail.com> wrote:
> I wanted to know if using --win32 would make renaming sh.exe unnecessary.

Jumping in late as I was not following the thread. I use MinGW build
and have the MSYS sh.exe in the path. The way I overcome the usage of
sh.exe is by issuing 'make SHELL=cmd'. Initially, I would rename the
sh.exe to prevent it from being used. I never had any issues but have
not checked the parallel build.

-dhruva

-- 
Contents reflect my personal views only!

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 20 Jun 2008 16:15:35 -0400
> 
> Since there is no useful information for custom-deps/finder-data in
> loaddefs etc, I made custom-make-dependencies and
> finder-compile-keywords-make-dist not scan these files any more (see
> finder-no-scan-regexp etc).

Thanks.  It is now crystal clear that the dependency of these two
targets on loaddefs.el can be removed.

Btw, should we also add ldefs-boot.el to the list of files not to be
scanned?

Re: bootstrap not regenerating some *loaddefs

[Glenn Morris]

Eli Zaretskii wrote:

> Btw, should we also add ldefs-boot.el to the list of files not to be
> scanned?

It's not auto-generated, so scanning it would do no harm, but neither
would it do any good, so yes, I added it. Thanks.

Re: bootstrap not regenerating some *loaddefs

[Juanma Barranquero]

On Fri, Jun 20, 2008 at 16:29, Eli Zaretskii <eliz@gnu.org> wrote:

> I think that's it.  If you need to know something else, please ask.

Using the GnuWin32 coreutils and make, I've been able to do a parallel
CMD build in two computers. Thanks.

In my laptop I still get crashes, even in non-parallel bootstraps,
during compilation of .el files. I'm starting to suspect a hardware
error, most likely a bad RAM chip.

  Juanma

Re: bootstrap not regenerating some *loaddefs

[Eli Zaretskii]

> Date: Mon, 23 Jun 2008 13:01:42 +0200
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Fri, Jun 20, 2008 at 16:29, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I think that's it.  If you need to know something else, please ask.
> 
> Using the GnuWin32 coreutils and make, I've been able to do a parallel
> CMD build in two computers.

Great!

(Yes, it's not easy to find a good-enough quality toolset on Windows,
since many ports are really broken in subtle ways.  Even some GnuWin32
ports are broken, but fortunately not Coreutils.)