bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Dani Moncayo]

Hello,

Sometimes, when building the trunk on MS-Windows, the make process get
stuck when this like of `src/makefile' is executed:

  cmd /c "fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"

What happens is that a new cmd.exe session is open and waiting for
input, when the expected behavior for that session is to execute the
command between double quotes and exit.  So I have to copy&paste the
command, execute it and exit the cmd session so that the build process
can continue.

This problem is known to Eli, Juanma and those who build Emacs on
Windows, and is related to MSYS and the translation it performs with
filesystem paths ("/c" is translated to "c:\" for example).

If you have MSYS installed in your system, it is easy to reproduce the problem:
1. Open a cmd.exe console.
2. Run: sh  (to start a bash session).
3. Run: cmd /c "dir".

--> A new cmd session is created, it is waiting for input and the
"dir" command has not been executed.

Fortunately, I just found a solution for this: just remove the space
between the `/c' and the double quote:
3. Run: cmd /c"dir".

--> This time the cmd session executes the "dir" command and then
ends, returning control to the bash session.


So, please, apply the following patch to the trunk for fixing this
problem (I've just tested it):

=== modified file 'src/makefile.w32-in'
--- src/makefile.w32-in 2012-11-17 23:16:24 +0000
+++ src/makefile.w32-in 2012-11-21 20:21:30 +0000
@@ -234,7 +234,7 @@
 gl-stamp: ../lib-src/$(BLD)/make-docfile.exe $(GLOBAL_SOURCES)
        - $(DEL) gl-tmp
        "$(THISDIR)/../lib-src/$(BLD)/make-docfile" -d . -g
$(SOME_MACHINE_OBJECTS) $(obj) > gl-tmp
-       cmd /c "fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"
+       cmd /c"fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"
        - $(DEL) gl-tmp
        echo timestamp > $@


Thanks.


In GNU Emacs 24.3.50.1 (i386-mingw-nt6.1.7601)
 of 2012-11-21 on MS-W7-DANI
Bzr revision: 110971 monnier@iro.umontreal.ca-20121121163435-89teaio2wo47frly
Windowing system distributor `Microsoft Corp.', version 6.1.7601
Configured using:
 `configure --with-gcc (4.7) --no-opt --enable-checking --cflags
 -Ic:/emacs/libs/libXpm-3.5.10/include -Ic:/emacs/libs/libXpm-3.5.10/src
 -Ic:/emacs/libs/libpng-1.2.37-lib/include -Ic:/emacs/libs/zlib-1.2.5
 -Ic:/emacs/libs/giflib-4.1.4-1-lib/include
 -Ic:/emacs/libs/jpeg-6b-4-lib/include
 -Ic:/emacs/libs/tiff-3.8.2-1-lib/include
 -Ic:/emacs/libs/libxml2-2.7.8-w32-bin/include/libxml2
 -Ic:/emacs/libs/gnutls-3.0.9-w32-bin/include
 -Ic:/emacs/libs/libiconv-1.9.2-1-lib/include'


-- 
Dani Moncayo

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Dani Moncayo]

> $(SOME_MACHINE_OBJECTS) $(obj) > gl-tmp
> -       cmd /c "fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"
> +       cmd /c"fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"
>         - $(DEL) gl-tmp
>         echo timestamp > $@

I've grepped the sources and there is another instance of this
problem, in that same file (src/makefile.w32-in):
   @cmd /c rem true

This sentence must not be executed in my current builds because I've
not had to touch it for my build to succeeded.  But if the sentence is
executed under some circumstances, the same problem will crop up.

So here is a new version of the patch which fixed that case too and
includes an informative commentary:

=== modified file 'src/makefile.w32-in'
--- src/makefile.w32-in 2012-11-17 23:16:24 +0000
+++ src/makefile.w32-in 2012-11-21 22:42:06 +0000
@@ -228,13 +228,17 @@
        xterm.o xfns.o xmenu.o xselect.o xrdb.o xsmfns.o dbusbind.o
 obj = $(GLOBAL_SOURCES:.c=.o)

+# WARNING: Every `cmd /c' command must have the form:
+#     cmd /c"some-command"
+# without any space between the /c and the first double quote.
+# This is needed to avoid problems when sh.exe is used as shell.
 globals.h: gl-stamp
-       @cmd /c rem true
+       @cmd /c"rem true"

 gl-stamp: ../lib-src/$(BLD)/make-docfile.exe $(GLOBAL_SOURCES)
        - $(DEL) gl-tmp
        "$(THISDIR)/../lib-src/$(BLD)/make-docfile" -d . -g
$(SOME_MACHINE_OBJECTS) $(obj) > gl-tmp
-       cmd /c "fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"
+       cmd /c"fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"
        - $(DEL) gl-tmp
        echo timestamp > $@


-- 
Dani Moncayo

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Eli Zaretskii]

> Date: Wed, 21 Nov 2012 21:49:53 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> 
> Sometimes, when building the trunk on MS-Windows, the make process get
> stuck when this like of `src/makefile' is executed:
> 
>   cmd /c "fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"
> 
> What happens is that a new cmd.exe session is open and waiting for
> input, when the expected behavior for that session is to execute the
> command between double quotes and exit.  So I have to copy&paste the
> command, execute it and exit the cmd session so that the build process
> can continue.
> 
> This problem is known to Eli, Juanma and those who build Emacs on
> Windows, and is related to MSYS and the translation it performs with
> filesystem paths ("/c" is translated to "c:\" for example).
> 
> If you have MSYS installed in your system, it is easy to reproduce the problem:
> 1. Open a cmd.exe console.
> 2. Run: sh  (to start a bash session).
> 3. Run: cmd /c "dir".
> 
> --> A new cmd session is created, it is waiting for input and the
> "dir" command has not been executed.
> 
> Fortunately, I just found a solution for this: just remove the space
> between the `/c' and the double quote:
> 3. Run: cmd /c"dir".
> 
> --> This time the cmd session executes the "dir" command and then
> ends, returning control to the bash session.

Sorry, I don't want to do this.  This might work now, but it does so
by pure luck, and might break in some future version of cmd, because
there _should_ be a space between /c and the command that follows.

The solution to this is simple: don't involve MSYS in building the
native port of Emacs.  It boils down to removing MSYS from PATH in the
shell window where you run the build scripts.

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Dani Moncayo]

> Sorry, I don't want to do this.  This might work now, but it does so
> by pure luck, and might break in some future version of cmd, because
> there _should_ be a space between /c and the command that follows.

Is that documented somewhere?  I've read the help of "cmd" and I don't see it.

> The solution to this is simple: don't involve MSYS in building the
> native port of Emacs.  It boils down to removing MSYS from PATH in the
> shell window where you run the build scripts.

That it one possible solution, but certainly not the one I'd like to
pick up, because it imposes an unnecessary restriction on the way of
building Emacs, and this particular restriction annoys me, because
Emacs can be build on Windows using the bash sell, and I like to do
so.  The only problem I've observed when doing it is the one explained
in this thread, and I'd like to fix it.

In the (unlikely) event that a future version of cmd.exe gives
problems when invoked that way, we could find a solution for it, but I
doubt it will ever happen.

-- 
Dani Moncayo

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Eli Zaretskii]

> Date: Thu, 22 Nov 2012 08:19:18 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: 12955@debbugs.gnu.org
> 
> > Sorry, I don't want to do this.  This might work now, but it does so
> > by pure luck, and might break in some future version of cmd, because
> > there _should_ be a space between /c and the command that follows.
> 
> Is that documented somewhere?  I've read the help of "cmd" and I don't see it.

I do:

  D:\usr\eli\data>cmd /?
  Starts a new instance of the Windows XP command interpreter

  CMD [/A | /U] [/Q] [/D] [/E:ON | /E:OFF] [/F:ON | /F:OFF] [/V:ON | /V:OFF]
      [[/S] [/C | /K] string]
                    ^^^
> > The solution to this is simple: don't involve MSYS in building the
> > native port of Emacs.  It boils down to removing MSYS from PATH in the
> > shell window where you run the build scripts.
> 
> That it one possible solution, but certainly not the one I'd like to
> pick up, because it imposes an unnecessary restriction on the way of
> building Emacs, and this particular restriction annoys me, because
> Emacs can be build on Windows using the bash sell, and I like to do
> so.

I don't quite understand your line of thinking.  If there is an
annoyance here (I don't see it), then it is self-imposed, because you
are deliberately using an unsupported environment -- unsupported
_precisely_ because of problems like this one.  Meanwhile, a supported
way of building Emacs is just one mouse click away -- start a normal
cmd shell window, making sure MSYS is not on PATH, and that's it.  I'm
using this exclusively without any problems (although I do have MSYS
installed).  What restriction this presents, when it uses commands
that are available out of the box on Windows (with the exception of 2
programs from a single Coreutils package)?

> The only problem I've observed when doing it is the one explained
> in this thread

You forget the previous ones.  I still remember them.

> and I'd like to fix it.

I don't mind fixing it, just not in the kludgey way you suggest.
Playing tricks with white space and quotes is a maintenance burden in
the long run: someone forgets or doesn't know about the importance of
the missing blank and commits a change that breaks things.  Treatment
of quotes in cmd is one of the trickiest issues ever, it depends on
Registry settings and the contents of the command line, so can subtly
break without notice.  We had a similar problem in configure.bat just
a few months ago.

> In the (unlikely) event that a future version of cmd.exe gives
> problems when invoked that way, we could find a solution for it, but I
> doubt it will ever happen.

I would like to find a good solution now.  Does it work for you to get
rid of the "cmd /c" part entirely and remove the quotes, i.e. use
this:

	fc.exe /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h

?  (Note the ".exe" part, it's important because "fc" is a shell
builtin in Bash.)  If "/b" causes the same trouble as "/c" in the cmd
command, we can make a Make variable, set to "//b" under MSYS and to
"/b" otherwise.  MSYS can be recognized by having MSYSTEM in the
environment (Make converts all environment variables to Make
variables, so you can use ifdef etc.).

FYI, I'd like to deprecate the Unixy shell parts of the Windows
makefiles some time soon, leaving only the cmd parts.  Supporting 2
kinds of shells with such different semantics is a PITA.  In parallel,
I'd like to make it possible to configure and build a native w32 Emacs
using MSYS and the mainline Unixy configury and Makefiles.  When that
goal is reached, the old configure.bat and makefile.w32-in's will
become a fallback solution for those who cannot or don't want to
install MSYS and for MSVC builds.  If you or someone else wants to
work on such an MSYS build, _that_ would be a worthy investment of
time and energy, and an excellent use of MSYS the way MSYS is supposed
to be used.  When used that way, MSYS really shines.

If you are willing to work on the MSYS build, I promise you all the
support I can give.  Otherwise, you really should start migrating to
cmd.

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Dani Moncayo]

> I don't quite understand your line of thinking.  If there is an
> annoyance here (I don't see it), then it is self-imposed, because you
> are deliberately using an unsupported environment -- unsupported
> _precisely_ because of problems like this one.

I thought it was supported, since the makefiles do care about
SHELLTYPE being CMD or SH, and since the build process work just fine
with SH (modulo the problem at hand).

> Meanwhile, a supported
> way of building Emacs is just one mouse click away -- start a normal
> cmd shell window, making sure MSYS is not on PATH, and that's it.  I'm
> using this exclusively without any problems (although I do have MSYS
> installed).

Agreed, that is a supported way (the one that uses CMD as shell).

>  What restriction this presents, when it uses commands
> that are available out of the box on Windows (with the exception of 2
> programs from a single Coreutils package)?

The restriction is preventing the use of the SH shell.  And those two
programs are included in the msys-base package of MinGW, which I find
very convenient and easy to install (with its package manager).

>> The only problem I've observed when doing it is the one explained
>> in this thread
>
> You forget the previous ones.  I still remember them.

The problems I remember are all the same: the one discussed here.  But
my memory is not perfect and I could be wrong.

>> and I'd like to fix it.
>
> I don't mind fixing it, just not in the kludgey way you suggest.
> Playing tricks with white space and quotes is a maintenance burden in
> the long run: someone forgets or doesn't know about the importance of
> the missing blank and commits a change that breaks things.  Treatment
> of quotes in cmd is one of the trickiest issues ever, it depends on
> Registry settings and the contents of the command line, so can subtly
> break without notice.  We had a similar problem in configure.bat just
> a few months ago.

Agreed.

>> In the (unlikely) event that a future version of cmd.exe gives
>> problems when invoked that way, we could find a solution for it, but I
>> doubt it will ever happen.
>
> I would like to find a good solution now.  Does it work for you to get
> rid of the "cmd /c" part entirely and remove the quotes, i.e. use
> this:
>
>         fc.exe /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h
>
> ?  (Note the ".exe" part, it's important because "fc" is a shell
> builtin in Bash.)  If "/b" causes the same trouble as "/c" in the cmd
> command, we can make a Make variable, set to "//b" under MSYS and to
> "/b" otherwise.  MSYS can be recognized by having MSYSTEM in the
> environment (Make converts all environment variables to Make
> variables, so you can use ifdef etc.).

Yes that seems to work for me.  I've tested both cases (SH and CMD).
This is the patch I've used:

=== modified file 'nt/gmake.defs'
--- nt/gmake.defs       2012-10-17 19:02:44 +0000
+++ nt/gmake.defs       2012-11-22 18:39:57 +0000
@@ -69,10 +69,12 @@
 ifeq "$(findstring ECHO, $(sh_output))" "ECHO"
 THE_SHELL = $(COMSPEC)$(ComSpec)
 SHELLTYPE=CMD
+FORWARD_SLASH=/
 else
 USING_SH = 1
 THE_SHELL = $(SHELL)
 SHELLTYPE=SH
+FORWARD_SLASH=//
 endif

 MAKETYPE=gmake

=== modified file 'src/makefile.w32-in'
--- src/makefile.w32-in 2012-11-17 23:16:24 +0000
+++ src/makefile.w32-in 2012-11-22 18:40:52 +0000
@@ -234,7 +234,7 @@
 gl-stamp: ../lib-src/$(BLD)/make-docfile.exe $(GLOBAL_SOURCES)
        - $(DEL) gl-tmp
        "$(THISDIR)/../lib-src/$(BLD)/make-docfile" -d . -g
$(SOME_MACHINE_OBJECTS) $(obj) > gl-tmp
-       cmd /c "fc /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h"
+       fc.exe $(FORWARD_SLASH)b gl-tmp globals.h >nul 2>&1 || $(CP)
gl-tmp globals.h
        - $(DEL) gl-tmp
        echo timestamp > $@


> FYI, I'd like to deprecate the Unixy shell parts of the Windows
> makefiles some time soon, leaving only the cmd parts.  Supporting 2
> kinds of shells with such different semantics is a PITA.  In parallel,
> I'd like to make it possible to configure and build a native w32 Emacs
> using MSYS and the mainline Unixy configury and Makefiles.  When that
> goal is reached, the old configure.bat and makefile.w32-in's will
> become a fallback solution for those who cannot or don't want to
> install MSYS and for MSVC builds.

That sounds like a good plan to me.

> If you or someone else wants to
> work on such an MSYS build, _that_ would be a worthy investment of
> time and energy, and an excellent use of MSYS the way MSYS is supposed
> to be used.  When used that way, MSYS really shines.
>
> If you are willing to work on the MSYS build, I promise you all the
> support I can give.  Otherwise, you really should start migrating to
> cmd.

I'm kind of a novice in these matters, but if you give me some
guidelines to get started and I find enough time, I'd like to learn
and try to help.

Thanks.

-- 
Dani Moncayo

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Eli Zaretskii]

> Date: Thu, 22 Nov 2012 20:16:18 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: 12955@debbugs.gnu.org
> 
> > I don't quite understand your line of thinking.  If there is an
> > annoyance here (I don't see it), then it is self-imposed, because you
> > are deliberately using an unsupported environment -- unsupported
> > _precisely_ because of problems like this one.
> 
> I thought it was supported, since the makefiles do care about
> SHELLTYPE being CMD or SH, and since the build process work just fine
> with SH (modulo the problem at hand).

The SH part is not for MSYS.  It's for Cygwin Bash.  It also worked
with the old port of zsh, which I used for some time.

> The restriction is preventing the use of the SH shell.  And those two
> programs are included in the msys-base package of MinGW, which I find
> very convenient and easy to install (with its package manager).

Installing the GnuWin32 port of Coreutils is equally easy.

> > I would like to find a good solution now.  Does it work for you to get
> > rid of the "cmd /c" part entirely and remove the quotes, i.e. use
> > this:
> >
> >         fc.exe /b gl-tmp globals.h >nul 2>&1 || $(CP) gl-tmp globals.h
> >
> > ?  (Note the ".exe" part, it's important because "fc" is a shell
> > builtin in Bash.)  If "/b" causes the same trouble as "/c" in the cmd
> > command, we can make a Make variable, set to "//b" under MSYS and to
> > "/b" otherwise.  MSYS can be recognized by having MSYSTEM in the
> > environment (Make converts all environment variables to Make
> > variables, so you can use ifdef etc.).
> 
> Yes that seems to work for me.  I've tested both cases (SH and CMD).

I committed a variant of that in trunk revision 110989.  Please test.

> > If you or someone else wants to
> > work on such an MSYS build, _that_ would be a worthy investment of
> > time and energy, and an excellent use of MSYS the way MSYS is supposed
> > to be used.  When used that way, MSYS really shines.
> >
> > If you are willing to work on the MSYS build, I promise you all the
> > support I can give.  Otherwise, you really should start migrating to
> > cmd.
> 
> I'm kind of a novice in these matters, but if you give me some
> guidelines to get started and I find enough time, I'd like to learn
> and try to help.

Thanks for the offer.

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Dani Moncayo]

>> Yes that seems to work for me.  I've tested both cases (SH and CMD).
>
> I committed a variant of that in trunk revision 110989.  Please test.

I've tried to invoke mingw32-make with and without MSYS' bash on PATH,
and it works for me in both scenarios.

Therefore, let's close this bug report.

Thank you so much.

PS: The build now goes fine for my use-case, i.e., when I run
mingw32-make from a cmd.exe shell, having the MSYS bash is on PATH.
But out of curiosity, I've just tried to invoke mingw32-make right the
MSYS bash shell, and in this second case a child cmd.exe shell is soon
started and waiting for input.  (Just for the record)

-- 
Dani Moncayo

bug#12955: 24.3.50; Build process on MS-Windows: sometimes needs "human intervention"

[Eli Zaretskii]

> Date: Fri, 23 Nov 2012 20:55:47 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: 12955-done@debbugs.gnu.org
> 
> But out of curiosity, I've just tried to invoke mingw32-make right the
> MSYS bash shell, and in this second case a child cmd.exe shell is soon
> started and waiting for input.  (Just for the record)

Maybe MSYSTEM is not in the environment in that case, because MSYS
removes it when it calls non-MSYS programs (in this case,
mingw32-make).