Bootstrap failure on MS-Windows

Bootstrap failure on MS-Windows

[Dani Moncayo]

Hello,

Today, I've tried to do a bootstrap of the current trunk, after two
weeks (since 2013-10-19), using the same method (autogen + mysconfig +
make) and the same build environment of the last time.

But "make" fails at this point:

  make[2]: Leaving directory `/usr/home/dani/emacs/build/lisp'
  if test "no" = "yes"; then \
    rm -f bootstrap-emacs.exe; \
    ln temacs.exe bootstrap-emacs.exe; \
  else \
    ./temacs --batch --load loadup bootstrap || exit 1; \
    test "X" = X ||  -zex emacs.exe; \
    mv -f emacs.exe bootstrap-emacs.exe; \
  fi
  Warning: arch-independent data dir
`%emacs_dir%/share/emacs/24.3.50/etc/': Permission denied
  Error: charsets directory not found:
 c:/msys/home/dani/emacs/build/src/%emacs_dir%/share/emacs/24.3.50/etc/charsets
  Emacs will not function correctly without the character map files.
  Please check your installation!
  Makefile:836: recipe for target `bootstrap-emacs.exe' failed
  make[1]: *** [bootstrap-emacs.exe] Error 1
  make[1]: Leaving directory `/usr/home/dani/emacs/build/src'
  Makefile:384: recipe for target `src' failed
  make: *** [src] Error 2

We've already seen this error not long ago [1], and that time, the
problem was related to the way of translating MSYS paths to Windows
native paths.  This fragment was added then to `configure.ac' to fix
the problem:

  #### When building with MinGW inside the MSYS tree, 'pwd' produces
  #### directories relative to the root of the MSYS tree,
  #### e.g. '/home/user/foo' instead of '/d/MSYS/home/user/foo'.  When
  #### such a value of srcdir is written to the top-level Makefile, it
  #### gets propagated to src/epaths.h, and that causes temacs to fail,
  #### because, being a MinGW program that knows nothing of MSYS root
  #### substitution, it cannot find the data directory.  "pwd -W"
  #### produces Windows-style 'd:/foo/bar' absolute directory names, so
  #### we use it here to countermand that lossage.
  test "$MSYSTEM" = "MINGW32" && abs_srcdir=`(cd "$abs_srcdir"; pwd -W
| sed -e 's,^\([[A-Za-z]]\):,/\1,')`

So I've done one test at this point, to try to track the problem down.
I've inserted a couple of sentences around the above one, in order to
see whether the root of my source tree was correctly stored in
`abs_srcdir':

  echo "(before) abs_srcdir=$abs_srcdir"
  test "$MSYSTEM" = "MINGW32" && abs_srcdir=`(cd "$abs_srcdir"; pwd -W
| sed -e 's,^\([A-Za-z]\):,/\1,')`
  echo "(after) abs_srcdir=$abs_srcdir"

And this is what I see:

  (before) abs_srcdir=
  (after) abs_srcdir=/C/msys/home/dani/emacs/build

Questions:

1. `abs_srcdir' seems to be empty or undefined before the above
assignment (because "echo" prints nothing).  That seems already wrong.

2. Oddly enough, the value after the assignment suggests that
`abs_srcdir' had a value, because otherwise `pwd -W' would have been
executed in my MSYS home directory, and then the assigned value would
have been "/C/msys/home/dani".

3. In any case, the value actually assigned to `abs_srcdir' is wrong,
since my source code tree is under "/C/msys/home/dani/emacs/repo"
("/C/msys/home/dani/emacs/build" is my build directory).


Help please.  TIA.


--- Footnotes ---

[1] http://lists.gnu.org/archive/html/emacs-devel/2013-09/msg00210.html


-- 
Dani Moncayo

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Dani Moncayo wrote:

> We've already seen this error not long ago [1], and that time, the
> problem was related to the way of translating MSYS paths to Windows
> native paths.  This fragment was added then to `configure.ac' to fix
> the problem:

As I said in
http://lists.gnu.org/archive/html/emacs-devel/2013-10/msg00730.html

that fragment currently has no effect. I suggest an alternative in that
post, and another one in
http://lists.gnu.org/archive/html/emacs-devel/2013-10/msg00761.html

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

BTW, if I need to suggest an actual patch for this, it would be
something like the following (obv. untested). This is the only place
left in the Makefiles that uses absolute filenames, so I don't think
anything else is needed.

*** Makefile.in	2013-11-05 07:54:03 +0000
--- Makefile.in	2013-11-05 08:20:51 +0000
***************
*** 342,349 ****
  # nt/epaths.nt as the template.
  # Use the value of ${locallisppath} supplied by `configure',
  # to support the --enable-locallisppath argument.
  epaths-force-w32: FRC
! 	@(w32srcdir=`echo "${abs_srcdir}" | ${msys_to_w32}` ; 	\
  	  prefixpattern=`echo '${prefix}' | ${msys_to_w32} | ${msys_sed_sh_escape}` ; \
  	  locallisppath=`echo '${locallisppath}' | ${msys_lisppath_to_w32} | ${msys_prefix_subst}` ; \
  	  sed < ${srcdir}/nt/epaths.nt > epaths.h.$$$$		\
--- 342,357 ----
  # nt/epaths.nt as the template.
  # Use the value of ${locallisppath} supplied by `configure',
  # to support the --enable-locallisppath argument.
+ #
+ # When building with MinGW inside the MSYS tree, 'pwd' produces directories
+ # relative to the root of the MSYS tree, e.g. '/home/user/foo' instead of
+ # '/d/MSYS/home/user/foo'.  If such a value of srcdir is written to
+ # src/epaths.h, that causes temacs to fail, because, being a MinGW
+ # program that knows nothing of MSYS root substitution, it cannot find
+ # the data directory.  "pwd -W" produces Windows-style 'd:/foo/bar'
+ # absolute directory names, so we use it here to countermand that lossage.
  epaths-force-w32: FRC
! 	@(w32srcdir=`cd "$srcdir"; pwd -W | sed -e 's,^\([[A-Za-z]]\):,/\1,' | ${msys_to_w32}` ; 	\
  	  prefixpattern=`echo '${prefix}' | ${msys_to_w32} | ${msys_sed_sh_escape}` ; \
  	  locallisppath=`echo '${locallisppath}' | ${msys_lisppath_to_w32} | ${msys_prefix_subst}` ; \
  	  sed < ${srcdir}/nt/epaths.nt > epaths.h.$$$$		\

=== modified file 'configure.ac'
*** configure.ac	2013-11-05 07:11:24 +0000
--- configure.ac	2013-11-05 08:21:10 +0000
***************
*** 419,435 ****
  		[Show Gtk+/Gdk deprecation warnings for Gtk+ >= 3.0])],
  [ac_enable_gtk_deprecation_warnings="${enableval}"],[])
  
- #### When building with MinGW inside the MSYS tree, 'pwd' produces
- #### directories relative to the root of the MSYS tree,
- #### e.g. '/home/user/foo' instead of '/d/MSYS/home/user/foo'.  When
- #### such a value of srcdir is written to the top-level Makefile, it
- #### gets propagated to src/epaths.h, and that causes temacs to fail,
- #### because, being a MinGW program that knows nothing of MSYS root
- #### substitution, it cannot find the data directory.  "pwd -W"
- #### produces Windows-style 'd:/foo/bar' absolute directory names, so
- #### we use it here to countermand that lossage.
- test "$MSYSTEM" = "MINGW32" && abs_srcdir=`(cd "$abs_srcdir"; pwd -W | sed -e 's,^\([[A-Za-z]]\):,/\1,')`
- 
  ### Canonicalize the configuration name.
  
  AC_CANONICAL_HOST
--- 419,424 ----

Re: Bootstrap failure on MS-Windows

[Dani Moncayo]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

On Tue, Nov 5, 2013 at 9:24 AM, Glenn Morris <rgm@gnu.org> wrote:
>
> BTW, if I need to suggest an actual patch for this, it would be
> something like the following (obv. untested). This is the only place
> left in the Makefiles that uses absolute filenames, so I don't think
> anything else is needed.

Thank you for the patch, Glenn.

I've just tried to bootstrap (autogen + msysconfig + make bootstrap)
with your patch applied, but it fails the same way.

I'm attaching the full output of "make bootstrap".

BTW, as Óscar already said [1], there is a second problem here: Just
after running autogen.sh (in the source tree) and msysconfig (in the
-empty- build tree), "make bootstrap" re-runs autogen.sh (see line 137
of the attached file) and the makefiles are re-generated.  This should
be unnecessary in this case, no?

---- Footnotes ---

[1] http://lists.gnu.org/archive/html/emacs-devel/2013-10/msg00746.html

-- 
Dani Moncayo

[-- Attachment #2: make.log.zip --]
[-- Type: application/zip, Size: 9129 bytes --]

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> Date: Tue, 5 Nov 2013 19:14:27 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> On Tue, Nov 5, 2013 at 9:24 AM, Glenn Morris <rgm@gnu.org> wrote:
> >
> > BTW, if I need to suggest an actual patch for this, it would be
> > something like the following (obv. untested). This is the only place
> > left in the Makefiles that uses absolute filenames, so I don't think
> > anything else is needed.
> 
> Thank you for the patch, Glenn.
> 
> I've just tried to bootstrap (autogen + msysconfig + make bootstrap)
> with your patch applied, but it fails the same way.

I think there's a mistake in the patch:

  ! 	@(w32srcdir=`cd "$srcdir"; pwd -W | sed -e 's,^\([[A-Za-z]]\):,/\1,' | ${msys_to_w32}` ; 	\
                                                         ^^^^^^^^^^

This should be "[A-Za-z]", i.e. only single pair of brackets.  (The
second one was necessary in an Autoconf file, to protect from
expansion by m4.)

> BTW, as Óscar already said [1], there is a second problem here: Just
> after running autogen.sh (in the source tree) and msysconfig (in the
> -empty- build tree), "make bootstrap" re-runs autogen.sh (see line 137
> of the attached file) and the makefiles are re-generated.  This should
> be unnecessary in this case, no?

This is on purpose, as this snippet from the top-level Makefile shows:

  # Bootstrapping does the following:
  #  * Remove files to start from a bootstrap-clean slate.
  #  * Run autogen.sh.
  #  * Rebuild Makefile, to update the build procedure itself.
  #  * Do the actual build.
  bootstrap: bootstrap-clean FRC
	  cd $(srcdir) && ./autogen.sh
	  $(MAKE) $(MFLAGS) MAKEFILE_NAME=force-Makefile force-Makefile
	  $(MAKE) $(MFLAGS) info all

Re: Bootstrap failure on MS-Windows

[Dani Moncayo]

>> I've just tried to bootstrap (autogen + msysconfig + make bootstrap)
>> with your patch applied, but it fails the same way.
>
> I think there's a mistake in the patch:
>
>   !     @(w32srcdir=`cd "$srcdir"; pwd -W | sed -e 's,^\([[A-Za-z]]\):,/\1,' | ${msys_to_w32}` ;        \
>                                                          ^^^^^^^^^^
>
> This should be "[A-Za-z]", i.e. only single pair of brackets.  (The
> second one was necessary in an Autoconf file, to protect from
> expansion by m4.)

I've tried again with that correction, but I see the same failure.

I may provide any necessary info to investigate this further (though I
think that this problem should be trivial to reproduce by anyone who
builds Emacs on MS-Windows).

>> BTW, as Óscar already said [1], there is a second problem here: Just
>> after running autogen.sh (in the source tree) and msysconfig (in the
>> -empty- build tree), "make bootstrap" re-runs autogen.sh (see line 137
>> of the attached file) and the makefiles are re-generated.  This should
>> be unnecessary in this case, no?
>
> This is on purpose, as this snippet from the top-level Makefile shows:

OK, thanks for clarifying this.


-- 
Dani Moncayo

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> Date: Tue, 5 Nov 2013 20:46:24 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: Glenn Morris <rgm@gnu.org>, Emacs development discussions <emacs-devel@gnu.org>
> 
> >> I've just tried to bootstrap (autogen + msysconfig + make bootstrap)
> >> with your patch applied, but it fails the same way.
> >
> > I think there's a mistake in the patch:
> >
> >   !     @(w32srcdir=`cd "$srcdir"; pwd -W | sed -e 's,^\([[A-Za-z]]\):,/\1,' | ${msys_to_w32}` ;        \
> >                                                          ^^^^^^^^^^
> >
> > This should be "[A-Za-z]", i.e. only single pair of brackets.  (The
> > second one was necessary in an Autoconf file, to protect from
> > expansion by m4.)
> 
> I've tried again with that correction, but I see the same failure.
> 
> I may provide any necessary info to investigate this further (though I
> think that this problem should be trivial to reproduce by anyone who
> builds Emacs on MS-Windows).

The relevant piece of information is src/epaths.h.  But really, Dani,
you are the only person who is motivated enough to dig into this, so
please do.  The issue cannot be too complicated: what happens here is
that Emacs does not find its lisp subdirectory, and therefore looks
for the one in the installation tree, which doesn't yet exist.

> >> BTW, as Óscar already said [1], there is a second problem here: Just
> >> after running autogen.sh (in the source tree) and msysconfig (in the
> >> -empty- build tree), "make bootstrap" re-runs autogen.sh (see line 137
> >> of the attached file) and the makefiles are re-generated.  This should
> >> be unnecessary in this case, no?
> >
> > This is on purpose, as this snippet from the top-level Makefile shows:
> 
> OK, thanks for clarifying this.

What you and Óscar need to get used to is not to run the configure
script before bootstrapping, but just say "make bootstrap", since the
bootstrap includes regeneration of the configure script and the
following configure step.

Re: Bootstrap failure on MS-Windows

[Dani Moncayo]

>> I've tried again with that correction, but I see the same failure.
>>
>> I may provide any necessary info to investigate this further (though I
>> think that this problem should be trivial to reproduce by anyone who
>> builds Emacs on MS-Windows).
>
> The relevant piece of information is src/epaths.h.  But really, Dani,
> you are the only person who is motivated enough to dig into this, so
> please do.  The issue cannot be too complicated: what happens here is
> that Emacs does not find its lisp subdirectory, and therefore looks
> for the one in the installation tree, which doesn't yet exist.

It seems that Glenn's patch (with Eli's fix) was almost correct.

After a bit of observation to the output of "msysconfig", I found a
problem in this line:

  @(w32srcdir=`cd "$srcdir"; pwd -W | sed -e 's,^\([A-Za-z]\):,/\1,' |
${msys_to_w32}` ; \
                   ^^^^^^^

It seems that the shell was expanding "$s" to the empty string, so
that the command finally executed was "cd rcdir; pwd -W | ..."

I've fixed this by replacing "$srcdir" with "${srcdir}".

So this is the final patch that has worked for me:

diff --git a/Makefile.in b/Makefile.in
index 461f0cb..39bdc5d 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -342,8 +342,16 @@ msys_sed_sh_escape=sed -e 's/[];$$*.^[]/\\\\&/g'
 # nt/epaths.nt as the template.
 # Use the value of ${locallisppath} supplied by `configure',
 # to support the --enable-locallisppath argument.
+#
+# When building with MinGW inside the MSYS tree, 'pwd' produces directories
+# relative to the root of the MSYS tree, e.g. '/home/user/foo' instead of
+# '/d/MSYS/home/user/foo'.  If such a value of srcdir is written to
+# src/epaths.h, that causes temacs to fail, because, being a MinGW
+# program that knows nothing of MSYS root substitution, it cannot find
+# the data directory.  "pwd -W" produces Windows-style 'd:/foo/bar'
+# absolute directory names, so we use it here to countermand that lossage.
 epaths-force-w32: FRC
- @(w32srcdir=`echo "${abs_srcdir}" | ${msys_to_w32}` ; \
+ @(w32srcdir=`cd "${srcdir}"; pwd -W | sed -e 's,^\([A-Za-z]\):,/\1,'
| ${msys_to_w32}` ; \
   prefixpattern=`echo '${prefix}' | ${msys_to_w32} | ${msys_sed_sh_escape}` ; \
   locallisppath=`echo '${locallisppath}' | ${msys_lisppath_to_w32} |
${msys_prefix_subst}` ; \
   sed < ${srcdir}/nt/epaths.nt > epaths.h.$$$$ \
diff --git a/configure.ac b/configure.ac
index cb97564..a8fb34b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -419,17 +419,6 @@ AC_ARG_ENABLE(gtk-deprecation-warnings,
  [Show Gtk+/Gdk deprecation warnings for Gtk+ >= 3.0])],
 [ac_enable_gtk_deprecation_warnings="${enableval}"],[])

-#### When building with MinGW inside the MSYS tree, 'pwd' produces
-#### directories relative to the root of the MSYS tree,
-#### e.g. '/home/user/foo' instead of '/d/MSYS/home/user/foo'.  When
-#### such a value of srcdir is written to the top-level Makefile, it
-#### gets propagated to src/epaths.h, and that causes temacs to fail,
-#### because, being a MinGW program that knows nothing of MSYS root
-#### substitution, it cannot find the data directory.  "pwd -W"
-#### produces Windows-style 'd:/foo/bar' absolute directory names, so
-#### we use it here to countermand that lossage.
-test "$MSYSTEM" = "MINGW32" && abs_srcdir=`(cd "$abs_srcdir"; pwd -W
| sed -e 's,^\([[A-Za-z]]\):,/\1,')`
-
 ### Canonicalize the configuration name.

 AC_CANONICAL_HOST


-- 
Dani Moncayo

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Duh, that was just me being dumb wrt autoconf versus Makefile syntax.
Applied.

Re: Bootstrap failure on MS-Windows

[Dani Moncayo]

> So this is the final patch that has worked for me:

I want to suggest the following simplification/improvement.  It is
tested on a real bootstrap.

Explanation:
* The ouput of "pwd -W" is in windows-native format (c:/foo/bar), and
we are converting it to MSYS-format (/c/foo/bar) and then converting
it back to windows-native format.  That double conversion is obviously
pointless.
* "${srcdir}" should expand to an existing directory.  If somehow it
expands to anything else, it is better to error out, because something
is going wrong.  So I've chained the commands with "&&" instead of
";".


diff --git a/Makefile.in b/Makefile.in
index 984dcea..3de8209 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -351,7 +351,7 @@ msys_sed_sh_escape=sed -e 's/[];$$*.^[]/\\\\&/g'
 # the data directory.  "pwd -W" produces Windows-style 'd:/foo/bar'
 # absolute directory names, so we use it here to countermand that lossage.
 epaths-force-w32: FRC
- @(w32srcdir=`cd "${srcdir}"; pwd -W | sed -e 's,^\([A-Za-z]\):,/\1,'
| ${msys_to_w32}` ; \
+ @(w32srcdir=`cd "${srcdir}" && pwd -W` ; \
   prefixpattern=`echo '${prefix}' | ${msys_to_w32} | ${msys_sed_sh_escape}` ; \
   locallisppath=`echo '${locallisppath}' | ${msys_lisppath_to_w32} |
${msys_prefix_subst}` ; \
   sed < ${srcdir}/nt/epaths.nt > epaths.h.$$$$ \



-- 
Dani Moncayo

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Dani Moncayo wrote:

> - @(w32srcdir=`cd "${srcdir}"; pwd -W | sed -e 's,^\([A-Za-z]\):,/\1,'
> | ${msys_to_w32}` ; \
> + @(w32srcdir=`cd "${srcdir}" && pwd -W` ; \

If that works, why doesn't plain old w32srcdir == abs_srcdir work?

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Wed, 06 Nov 2013 15:32:08 -0500
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> Dani Moncayo wrote:
> 
> > - @(w32srcdir=`cd "${srcdir}"; pwd -W | sed -e 's,^\([A-Za-z]\):,/\1,'
> > | ${msys_to_w32}` ; \
> > + @(w32srcdir=`cd "${srcdir}" && pwd -W` ; \
> 
> If that works, why doesn't plain old w32srcdir == abs_srcdir work?

Because "pwd -W" produces d:/foo/bar/baz from /usr/local/baz.

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Eli Zaretskii wrote:

>> > - @(w32srcdir=`cd "${srcdir}"; pwd -W | sed -e 's,^\([A-Za-z]\):,/\1,'
>> > | ${msys_to_w32}` ; \
>> > + @(w32srcdir=`cd "${srcdir}" && pwd -W` ; \
>> 
>> If that works, why doesn't plain old w32srcdir == abs_srcdir work?
>
> Because "pwd -W" produces d:/foo/bar/baz from /usr/local/baz.

Duh, I totally failed to read the -W.

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

cant-test-this-but-it-makes-sense-to-me-and-no-one-objected-so-applied

Re: Bootstrap failure on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> What you and Óscar need to get used to is not to run the configure
> script before bootstrapping, but just say "make bootstrap", since the
> bootstrap includes regeneration of the configure script and the
> following configure step.

I don't run the configure script before bootstrapping on a directory
used on a previous build. I just do `make bootstrap -j4'.

What I *think* that happens is:

 * the bootstrap target depends, directly or indirectly, on the
   configure (and, possibly, autogen) scripts.

 * when those are dirty, the script is executed before cleaning the
   build directory.

 * as the previous step deletes the files generated by `./configure',
   the script is executed again.

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Óscar Fuentes wrote:

> I don't run the configure script before bootstrapping on a directory
> used on a previous build. I just do `make bootstrap -j4'.
>
> What I *think* that happens is:
>
>  * the bootstrap target depends, directly or indirectly, on the
>    configure (and, possibly, autogen) scripts.
>
>  * when those are dirty, the script is executed before cleaning the
>    build directory.
>
>  * as the previous step deletes the files generated by `./configure',
>    the script is executed again.

FWIW I got fed up by the overzealousness of this some time ago.
Personally I use something like
make maintainer-clean
configure
make

rather than make bootstrap.

I imagine `make bootstrap' is being technically correct.

(AFAIK none of this is new or has any relevance to the initial topic of
this discussion.)

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Glenn Morris wrote:

> make maintainer-clean

Although this too triggers a re-configuration sometimes.
Again, I imagine it is technically TRTD.

Re: Bootstrap failure on MS-Windows

[Andy Moreton]

On Tue 05 Nov 2013, Eli Zaretskii wrote:
> What you and Óscar need to get used to is not to run the configure
> script before bootstrapping, but just say "make bootstrap", since the
> bootstrap includes regeneration of the configure script and the
> following configure step.

I also run autogen and msyconfig with a clean tree before running "make
bootstrap". When "make bootstrap" reruns configure, it produces
different output to configure run on its own, and it looks like it is
ignoring the site config that msysconfig provides.

    AndyM

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 06 Nov 2013 13:59:26 +0000
> 
> When "make bootstrap" reruns configure, it produces different output
> to configure run on its own, and it looks like it is ignoring the
> site config that msysconfig provides.

Not here.  If it does that on your system, that's some bug that needs
to be investigated.

Re: Bootstrap failure on MS-Windows

[Andy Moreton]

On Wed 06 Nov 2013, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Wed, 06 Nov 2013 13:59:26 +0000
>> 
>> When "make bootstrap" reruns configure, it produces different output
>> to configure run on its own, and it looks like it is ignoring the
>> site config that msysconfig provides.
>
> Not here.  If it does that on your system, that's some bug that needs
> to be investigated.

Thanks for the hint Eli - you are right as usual.

After further investigation this turned out to be a problem in my
wrapper script which was passing configure args in the environment
rather than on the configure command line.

This is explained nicely in "Defining Variables" in autoconf info.

    AndyM

Re: Bootstrap failure on MS-Windows

[Andy Moreton]

On Wed 06 Nov 2013, Andy Moreton wrote:

> On Wed 06 Nov 2013, Eli Zaretskii wrote:
>
>>> From: Andy Moreton <andrewjmoreton@gmail.com>
>>> Date: Wed, 06 Nov 2013 13:59:26 +0000
>>> 
>>> When "make bootstrap" reruns configure, it produces different output
>>> to configure run on its own, and it looks like it is ignoring the
>>> site config that msysconfig provides.
>>
>> Not here.  If it does that on your system, that's some bug that needs
>> to be investigated.
>
> Thanks for the hint Eli - you are right as usual.
>
> After further investigation this turned out to be a problem in my
> wrapper script which was passing configure args in the environment
> rather than on the configure command line.

After more testing, it turns out this is not the case.

The scenario that goes wrong is:
1) Begin with a clean bzr checkout
2) Run 'autogen.sh" and 'nt/msysconfig.sh' and 'make' as usual
3) Time passes, and configure.ac is updated after a bzr pull
4) Run 'make bootstrap', and observe that the CONFIG_SITE file
   is not used when configure is run from the make, leading to
   incorrect configure results and a failed bootstrap.

--[make bootstrap]--------------------------------------------
rm -f config.cache config.log
cd .. && ./autogen.sh
Checking whether you have the necessary tools...
(Read INSTALL.BZR for more details on building Emacs)

Checking for autoconf (need at least version 2.65)...
ok
Checking for automake (need at least version 1.11)...
ok
Your system has the required tools, running autoreconf...
You can now run `./configure'.
/usr/bin/make  MAKEFILE_NAME=force-Makefile force-Makefile
make[1]: Entering directory `/c/emacs/src/emacs/trunk/obj-mingw32'
if [ -x ./config.status ]; then \
             ./config.status --recheck; \
        else                            \
             ../configure --cache-file=/dev/null; \
        fi
running CONFIG_SHELL=/bin/sh /bin/sh ../configure --prefix=C:/emacs-trunk --with-pkg-config-prog=C:/emacs/mingw-devel/bin/pkg-config --enable-checking --without-rsvg CPPFLAGS= -IC:/emacs/mingw-devel/include -IC:/emacs/mingw-devel/src/xpm/3.5.1/libXpm-3.5.1-src/lib LDFLAGS= -LC:/emacs/mingw-devel/lib --no-create --no-recursion
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
--[make bootstrap]--------------------------------------------

The easiest way to notice the incorrect configure results is to look for
the ACL tests. In a good msys configure, this comes from the site file,
whereas in the failed case it runs the autoconf tests.

    AndyM

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 08 Nov 2013 10:32:02 +0000
> 
> The scenario that goes wrong is:
> 1) Begin with a clean bzr checkout
> 2) Run 'autogen.sh" and 'nt/msysconfig.sh' and 'make' as usual
> 3) Time passes, and configure.ac is updated after a bzr pull
> 4) Run 'make bootstrap', and observe that the CONFIG_SITE file
>    is not used when configure is run from the make, leading to
>    incorrect configure results and a failed bootstrap.

Don't do 4).  Instead, just do "make", and let it decide whether it
needs to regenerate configure.

The problem here is that "make bootstrap", when invoked outside of the
source tree, does not see the file GNUmakefile, which arranges for CFG
to have the proper value.

Re: Bootstrap failure on MS-Windows

[Andy Moreton]

On Fri 08 Nov 2013, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Fri, 08 Nov 2013 10:32:02 +0000
>> 
>> The scenario that goes wrong is:
>> 1) Begin with a clean bzr checkout
>> 2) Run 'autogen.sh" and 'nt/msysconfig.sh' and 'make' as usual
>> 3) Time passes, and configure.ac is updated after a bzr pull
>> 4) Run 'make bootstrap', and observe that the CONFIG_SITE file
>>    is not used when configure is run from the make, leading to
>>    incorrect configure results and a failed bootstrap.
>
> Don't do 4).  Instead, just do "make", and let it decide whether it
> needs to regenerate configure.
>
> The problem here is that "make bootstrap", when invoked outside of the
> source tree, does not see the file GNUmakefile, which arranges for CFG
> to have the proper value.

Thanks Eli, I hadn't noticed that.

This does rely on the value of MSYSTEM in the environment, which is only
set if the msys bash shell was invoked via "mingw/msys/1.0/msys.bat"
i.e. it is not set by msys bash itself. I'll set MSYSTEM in my wrapper
script and see if it helps.

    AndyM

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 08 Nov 2013 13:34:20 +0000
> 
> This does rely on the value of MSYSTEM in the environment, which is only
> set if the msys bash shell was invoked via "mingw/msys/1.0/msys.bat"
> i.e. it is not set by msys bash itself.

That's not true.  I don't use msys.bat, either, and I do have MSYSTEM
in the environment.  My MSYS Bash window is started with "bash --login",
if it helps.

Re: Bootstrap failure on MS-Windows

[Andy Moreton]

On Fri 08 Nov 2013, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Fri, 08 Nov 2013 13:34:20 +0000
>> 
>> This does rely on the value of MSYSTEM in the environment, which is only
>> set if the msys bash shell was invoked via "mingw/msys/1.0/msys.bat"
>> i.e. it is not set by msys bash itself.
>
> That's not true.  I don't use msys.bat, either, and I do have MSYSTEM
> in the environment.  My MSYS Bash window is started with "bash --login",
> if it helps.

I did check before posting. I was using a non-login shell though,
whereas you used a login shell which will read msys /etc/profile, and
thus set MSYSTEM.

    AndyM

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 09 Nov 2013 12:36:43 +0000
> 
> >> This does rely on the value of MSYSTEM in the environment, which is only
> >> set if the msys bash shell was invoked via "mingw/msys/1.0/msys.bat"
> >> i.e. it is not set by msys bash itself.
> >
> > That's not true.  I don't use msys.bat, either, and I do have MSYSTEM
> > in the environment.  My MSYS Bash window is started with "bash --login",
> > if it helps.
> 
> I did check before posting. I was using a non-login shell though,
> whereas you used a login shell which will read msys /etc/profile, and
> thus set MSYSTEM.

Any reason not to use --login?

Re: Bootstrap failure on MS-Windows

[Andy Moreton]

On Sat 09 Nov 2013, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 09 Nov 2013 12:36:43 +0000
>> 
>> >> This does rely on the value of MSYSTEM in the environment, which is only
>> >> set if the msys bash shell was invoked via "mingw/msys/1.0/msys.bat"
>> >> i.e. it is not set by msys bash itself.
>> >
>> > That's not true.  I don't use msys.bat, either, and I do have MSYSTEM
>> > in the environment.  My MSYS Bash window is started with "bash --login",
>> > if it helps.
>> 
>> I did check before posting. I was using a non-login shell though,
>> whereas you used a login shell which will read msys /etc/profile, and
>> thus set MSYSTEM.
>
> Any reason not to use --login?

I didn't know it was needed, and I haven't seen it mentioned in the
build instructions. I use cygwin tools for everything else, so my build
script runs there and invokes non-interactive msys shells with a minimal
environment.

    AndyM

Re: Bootstrap failure on MS-Windows

[Jarek Czekalski]

Maybe "uname" could be an option to detect mingw? On my system it 
returns "windows32" for non-login, and "MINGW32_NT-6.1" for --login 
shell. Cygwin returns CYGWIN_NT-6.1-WOW64, so it won't be mistaken as mingw.

Jarek

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> Date: Sat, 09 Nov 2013 14:24:21 +0100
> From: Jarek Czekalski <jarekczek@poczta.onet.pl>
> 
> Maybe "uname" could be an option to detect mingw? On my system it 
> returns "windows32" for non-login, and "MINGW32_NT-6.1" for --login 
> shell. Cygwin returns CYGWIN_NT-6.1-WOW64, so it won't be mistaken as mingw.

Thanks, but I believe the need for this will be eliminated, once the
change that makes configure read mingw-cfg.site directly is committed.

Re: Bootstrap failure on MS-Windows

[Dani Moncayo]

>> Maybe "uname" could be an option to detect mingw? On my system it
>> returns "windows32" for non-login, and "MINGW32_NT-6.1" for --login
>> shell. Cygwin returns CYGWIN_NT-6.1-WOW64, so it won't be mistaken as mingw.
>
> Thanks, but I believe the need for this will be eliminated, once the
> change that makes configure read mingw-cfg.site directly is committed.

If you refer to the patch I sent [1], the reading of
'nt/mingw-cfg.site' from the configure script is conditioned to the
MSYS environment.  So, with that approach, we still need a way of
checking whether the shell is running on a MSYS environment.


--- Footnotes ---

http://lists.gnu.org/archive/html/emacs-devel/2013-11/msg00295.html


-- 
Dani Moncayo

Re: Bootstrap failure on MS-Windows

[Eli Zaretskii]

> Date: Sat, 9 Nov 2013 15:33:55 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: Jarek Czekalski <jarekczek@poczta.onet.pl>, 
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> >> Maybe "uname" could be an option to detect mingw? On my system it
> >> returns "windows32" for non-login, and "MINGW32_NT-6.1" for --login
> >> shell. Cygwin returns CYGWIN_NT-6.1-WOW64, so it won't be mistaken as mingw.
> >
> > Thanks, but I believe the need for this will be eliminated, once the
> > change that makes configure read mingw-cfg.site directly is committed.
> 
> If you refer to the patch I sent [1], the reading of
> 'nt/mingw-cfg.site' from the configure script is conditioned to the
> MSYS environment.  So, with that approach, we still need a way of
> checking whether the shell is running on a MSYS environment.

'uname' is invoked by config.guess, so if the value it returns is
needed, it is already available to the configure script in the form of
'opsys' variable, albeit somewhat later than where you wanted to read
the site-config file.

Relying on a raw 'uname' is IMO wrong, because it would preclude
setting --host etc. from the configure command-line args.

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Dani Moncayo wrote:

> (though I think that this problem should be trivial to reproduce by
> anyone who builds Emacs on MS-Windows).

Apparently not; eg
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=15675#38

Re: Bootstrap failure on MS-Windows

[Dani Moncayo]

On Tue, Nov 5, 2013 at 9:29 PM, Glenn Morris <rgm@gnu.org> wrote:
> Dani Moncayo wrote:
>
>> (though I think that this problem should be trivial to reproduce by
>> anyone who builds Emacs on MS-Windows).
>
> Apparently not; eg
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=15675#38

To reproduce the problem, one must (a) have the source tree under the
MSYS tree, and (b) invoke the configure script using a relative path.
And it seems that currently I'm the only person who builds Emacs with
this setup.

-- 
Dani Moncayo

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Eli Zaretskii wrote:

> I think there's a mistake in the patch:
>
>   ! 	@(w32srcdir=`cd "$srcdir"; pwd -W | sed -e 's,^\([[A-Za-z]]\):,/\1,' | ${msys_to_w32}` ; 	\
>                                                          ^^^^^^^^^^
>
> This should be "[A-Za-z]", i.e. only single pair of brackets.  (The
> second one was necessary in an Autoconf file, to protect from
> expansion by m4.)

Sorry; you are absolutely right.

Re: Bootstrap failure on MS-Windows

[Glenn Morris]

Then I don't know, sorry.