Supporting parameters that include the = character in configure.bat

Supporting parameters that include the = character in configure.bat

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 2502 bytes --]

Hello,

I have seen several mentions on this list of the desire to make
configure.bat support parameters that include the = character.  The --cflags
parameter was mentioned specifically.  For example Eli Zaretskii wanted to
be able to call configure.bat with --cflags
-DSITELOAD_PURESIZE_EXTRA=100000.

There is a way to do this on Windows operating systems based on Windows NT.
The following batch file demonstrates how this might be accomplished.

<batch_file>
@echo off

set sep1=
:again

if "%1" == "--cflags" goto usercflags
if "%1" == "" goto checkutils

:usercflags
shift
set usercflags=%usercflags%%sep1%%~1
set sep1= %nothing%
shift
goto again

:checkutils

echo usercflags=%usercflags%
</batch_file>

The key here is the usage of the ~ character between the % and the 1.  This
essentially causes " characters around the parameter to be removed.  Thus
when this batch file is called as follows
  test.bat --cflags "-DSITELOAD_PURESIZE_EXTRA=100000" --cflags "-DTEST=1"
the output is as follows
  usercflags=-DSITELOAD_PURESIZE_EXTRA=100000 -DTEST=1

As long as the parameter is surrounded by " characters, the = character will
not be treated as a separator.

The major drawback of this is that it will not work on Windows 9x because
command.com does not support this functionality.  I see two possible
solutions to this problem.  The first is to have two versions of
configure.bat, a main version that uses this trick and another version that
is backwards compatible with Windows 9x.  Another possible solution is to
somehow have configure.bat test to see if this functionality is supported
and use it if it is available and otherwise not use the functionality.  The
following batch file shows how this might be accomplished.

<batch_file>
@echo off

set sep1=
:again

if "%1" == "--cflags" goto usercflags
if "%1" == "" goto checkutils

:usercflags

if "%OS%" == "Windows_NT" goto ucflagnt
goto ucflag9x

:ucflagnt
shift
set usercflags=%usercflags%%sep1%%~1
set sep1= %nothing%
shift
goto again

:ucflag9x
shift
set usercflags=%usercflags%%sep1%%1
set sep1= %nothing%
shift
goto again

:checkutils

echo usercflags=%usercflags%
</batch_file>

The drawback of this approach is that it would make an already complex batch
file even more complex.  It does however make it possible to support a much
needed feature, at least on Windows operating systems based on Windows NT,
without the maintenance headache that using two batch files would introduce.

What do you all think about this approach?

[-- Attachment #2: Type: text/html, Size: 2988 bytes --]

Re: Supporting parameters that include the = character in configure.bat

[Eli Zaretskii]

> From: Ben Key <bkey76@gmail.com>
> Date: Sun, 10 Apr 2011 02:51:34 -0500
> Cc: 
> 
> The key here is the usage of the ~ character between the % and the 1.  This
> essentially causes " characters around the parameter to be removed.  Thus
> when this batch file is called as follows
>   test.bat --cflags "-DSITELOAD_PURESIZE_EXTRA=100000" --cflags "-DTEST=1"
> the output is as follows
>   usercflags=-DSITELOAD_PURESIZE_EXTRA=100000 -DTEST=1
> 
> As long as the parameter is surrounded by " characters, the = character will
> not be treated as a separator.

Thanks!

> The major drawback of this is that it will not work on Windows 9x because
> command.com does not support this functionality.

This isn't a problem, because we don't support building Emacs on
Windows 9X anyway.

So please prepare a patch along these lines.  It should include a
corresponding addition to the "usage" text in the batch file and a
similar addition to instructions in nt/INSTALL.

Thanks again for working on this.

Re: Supporting parameters that include the = character in configure.bat

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

Hello,

Eli Zaretskii wrote:
> This isn't a problem, because we don't support building Emacs on
> Windows 9X anyway.

I was not aware of this.  Configure.bat still has some comments that refer
to Windows 9x and command.com so I thought it was still a consideration.

> So please prepare a patch along these lines.  It should include a
> corresponding addition to the "usage" text in the batch file and a
> similar addition to instructions in nt/INSTALL.

Before submitting a patch I wanted to do some more research into this
technique to ensure that it will always work.  It turns out that the %~1
functionality to strip " characters only works if command extensions are
enabled.  As far as I know they are always enabled by default but they can
be disabled by a system administrator.  This means that it will still be
necessary to determine if command extensions are enabled before attempting
to use this functionality.  This can be done as follows.

<batch_file>
@echo off
set use_extensions=1
setlocal enableextensions
if errorlevel 1 set use_extensions=0

set sep1=
:again

if "%1" == "--cflags" goto usercflags
if "%1" == "" goto checkutils

:usercflags

if "%use_extensions%" == "1" goto ucflagex
goto ucflag

:ucflagex
shift
set usercflags=%usercflags%%sep1%%~1
set sep1= %nothing%
shift
goto again

:ucflag
shift
set usercflags=%usercflags%%sep1%%1
set sep1= %nothing%
shift
goto again

:checkutils

echo usercflags=%usercflags%
</batch_file>

I assume that I should use this technique when implementing my patch.  If
this is what you want me to do, just let me know.

[-- Attachment #2: Type: text/html, Size: 1924 bytes --]

Re: Supporting parameters that include the = character in configure.bat

[Eli Zaretskii]

> From: Ben Key <bkey76@gmail.com>
> Date: Sun, 10 Apr 2011 03:46:39 -0500
> Cc: Emacs-devel@gnu.org
> 
> > This isn't a problem, because we don't support building Emacs on
> > Windows 9X anyway.
> 
> I was not aware of this.  Configure.bat still has some comments that refer
> to Windows 9x and command.com

Yeah, we need to remove that at some point.

> @echo off
> set use_extensions=1
> setlocal enableextensions
> if errorlevel 1 set use_extensions=0

You cannot rely on the fact that errorlevel is zero before entering
the batch file.  See "setlocal /?", it suggests a more reliable
method.

> if "%use_extensions%" == "1" goto ucflagex
> goto ucflag

What happens if extensions are disabled?  Will arguments with "="
work?  If not, we need to display a warning.

Anyway, "cmd /?" seems to imply that even if the extensions are
disabled in the Registry, one can enable them in an inferior shell by
invoking "cmd /e:on".  If that is true, perhaps if you determine that
the extensions are disabled, you should reinvoke the batch file with
"cmd /e:on".  If this works it's better than punting.

Re: Supporting parameters that include the = character in configure.bat

[Eli Zaretskii]

> From: Ben Key <bkey76@gmail.com>
> Date: Sun, 10 Apr 2011 07:27:54 -0500
> 
> What is the oldest version of Windows supported by Emacs?

For building Emacs?  Windows NT4, I think.  Although I doubt that the
current development sources were tested with anything but W2K, so NT4
support should be regarded as purely theoretical at this point, and if
command extensions aren't supported there, I have no problems with
saying in nt/INSTALL that --cflags switches with embedded "=" aren't
supported there, either.

For running Emacs, we still want to support Windows 9X, but that's
unrelated to this discussion.

Re: Supporting parameters that include the = character in configure.bat

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

Hello,

I am working on the patch to configure.bat to support using parameters that
include the = character.  I need some input on changes I am making to the
usage message.  My changes are as follows:

The cflags and ldflags arguments support parameters that include the =
character.  However, since the = character is normally treated as a
separator character you will need to enclose any parameter that includes
the = character in quotes.  For example, to include
-DSITELOAD_PURESIZE_EXTRA=100000 as one of the cflags you would run
configure.bat as follows:
configure.bat --cflags "-DSITELOAD_PURESIZE_EXTRA=100000"

Note that this capability of processing parameters that include the =
character depend on command extensions.  This batch file attempts to enable
command extensions.  If command extensions cannot be enabled, a warning
message will be displayed.

What is your opinion of these changes?  If I should change the wording
simply send me the recommended changes.

Thanks.

[-- Attachment #2: Type: text/html, Size: 1121 bytes --]

Re: Supporting parameters that include the = character in configure.bat

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1419 bytes --]

Hello,

I am working on a patch for configure.bat to support using parameters that
include the = character.  As part of this patch I need to make changes to
the nt/INSTALL file.  The section that needs to be changed is as follows.

Because of limitations of the stock Windows command shell, certain
characters (quotes, backslashes and equal signs) can be problematic
and should not be used in arguments to configure.  That means that
forward slashes must be used in paths passed to the compiler and
linker via the --cflags and --ldflags options, and that it is
currently not possible to pass a macro like -DFOO=BAR (though -DFOO
is perfectly valid).

The change I am making to configure.bat modifies the parsing of the --cflags
and --ldflags options to add support for parameters like -DFOO=BAR as long
as the parameter is enclosed in quotes.  I need some input on how I should
rewrite this section of the document.  The part that confuses me is the
statement that quotes should not be used in arguments to configure.  My
technique for supporting parameters that include the = character requires
that the parameter be enclosed in quotes.  However, this section of
nt/INSTALL says quotes are not supported.  Perhaps if I had more information
on the history of this paragraph, and specifically what the problems with
quotes are I would know how to rewrite this paragraph.

I appreciate any input you might have to offer.

[-- Attachment #2: Type: text/html, Size: 1549 bytes --]

Re: Supporting parameters that include the = character in configure.bat

[Eli Zaretskii]

> From: Ben Key <bkey76@gmail.com>
> Date: Mon, 11 Apr 2011 19:17:59 -0500
> Cc: Emacs-devel@gnu.org
> 
> The cflags and ldflags arguments support parameters that include the =
> character.  However, since the = character is normally treated as a
> separator character you will need to enclose any parameter that includes
> the = character in quotes.  For example, to include
> -DSITELOAD_PURESIZE_EXTRA=100000 as one of the cflags you would run
> configure.bat as follows:
> configure.bat --cflags "-DSITELOAD_PURESIZE_EXTRA=100000"
> 
> Note that this capability of processing parameters that include the =
> character depend on command extensions.  This batch file attempts to enable
> command extensions.  If command extensions cannot be enabled, a warning
> message will be displayed.
> 
> What is your opinion of these changes?  If I should change the wording
> simply send me the recommended changes.

Sounds fine to me.

Re: Supporting parameters that include the = character in configure.bat

[Eli Zaretskii]

> From: Ben Key <bkey76@gmail.com>
> Date: Mon, 11 Apr 2011 21:41:50 -0500
> Cc: Emacs-devel@gnu.org
> 
> I am working on a patch for configure.bat to support using parameters that
> include the = character.  As part of this patch I need to make changes to
> the nt/INSTALL file.  The section that needs to be changed is as follows.
> 
> Because of limitations of the stock Windows command shell, certain
> characters (quotes, backslashes and equal signs) can be problematic
> and should not be used in arguments to configure.  That means that
> forward slashes must be used in paths passed to the compiler and
> linker via the --cflags and --ldflags options, and that it is
> currently not possible to pass a macro like -DFOO=BAR (though -DFOO
> is perfectly valid).
> 
> The change I am making to configure.bat modifies the parsing of the --cflags
> and --ldflags options to add support for parameters like -DFOO=BAR as long
> as the parameter is enclosed in quotes.  I need some input on how I should
> rewrite this section of the document.  The part that confuses me is the
> statement that quotes should not be used in arguments to configure.

Simply remove the part about quotes (with the caveat about command
extensions), because it is no longer true with your changes.

The reason it is there now is that using quotes will have caused the
quotes to wind up in a C macro, which will then trigger weird
compilation errors.  People actually bumped into this, which is why
that portion of the instructions was written.

The part about backslashes should stay, though: these are treated as
escape characters by the C preprocessor, which will generally ruin any
file names that use backslashes in the values of the --cflags or --ldflags.

Re: Supporting parameters that include the = character in configure.bat

[Ben Key]

[-- Attachment #1: Type: text/plain, Size: 1911 bytes --]

Hello,

The following is my first attempt at rewriting the section in nt/INSTALL
that used to warn against using quotes, backslashes and equal signs.  Be
forewarned that writing documentation is not one of my strong points.

Because of limitations of the stock Windows command shell backslash
characters can be problematic and should not be used in arguments to
configure.  That means that forward slashes must be used in paths
passed to the compiler and linker via the --cflags and --ldflags
options.  Note that it is possible to pass a macro like -DFOO=BAR via
the --cflags and --ldflags options as long as it is enclosed in
quotes.  The code that parses the --cflags and --ldflags options
removes any surrounding quotes (but only if command extensions are
available).  If command extensions are disabled, an attempt will be
made to enable them.  If command extensions are not available a
warning message will be displayed informing you that "using
parameters that include the = character by enclosing them in quotes
will not be supported."  Note that at this time only the --cflags and
--ldflags options support using the = character.  The = character
will be treated as a separator character for all other options.

This is part of the patch I am working on to provide support for --cflags
and --ldflags options like -DFOO=BAR.

Please let me know what you think of this change.

Also, I need to know if the following comments in configure.bat are still
valid.

Rem WARNING -- COMMAND.COM on some systems only looks at the first
Rem            8 characters of a label.  So do NOT be tempted to change
Rem            chkapi* into something fancier like checkw32api
Rem You HAVE been warned!

I would like to know if I can safely use labels that are longer than 8
characters now.  From what I understand, we no longer support building Emacs
on Windows 9x so this should no longer be a consideration.

Thanks.

[-- Attachment #2: Type: text/html, Size: 2160 bytes --]

Re: Supporting parameters that include the = character in configure.bat

[Eli Zaretskii]

> From: Ben Key <bkey76@gmail.com>
> Date: Wed, 13 Apr 2011 18:17:29 -0500
> Cc: Emacs-devel@gnu.org
> 
> The following is my first attempt at rewriting the section in nt/INSTALL
> that used to warn against using quotes, backslashes and equal signs.  Be
> forewarned that writing documentation is not one of my strong points.

Thanks.  I suggest a slight rewording as follows:

  Because of limitations of the stock Windows command shells, special
  care is needed to pass some characters in the arguments of the
  --cflags and --ldflags options.  Backslashes should not be used in
  file names passed to the compiler and linker via these options.  Use
  forward slashes instead.  If the arguments to these two options
  include the `=' character, like when passing a -DFOO=bar preprocessor
  option, the argument with the `=' character should be enclosed in
  quotes, like this:

    configure --cflags "-DFOO=bar"

  Support for options that include the `=' character require "command
  extensions" to be enabled.  (They are enabled by default, but your
  system administrator could have changed that.  See "cmd /?" for
  details.)  If command extensions are disabled, a warning message might
  be displayed informing you that "using parameters that include the =
  character by enclosing them in quotes will not be supported."

> Also, I need to know if the following comments in configure.bat are still
> valid.
> 
> Rem WARNING -- COMMAND.COM on some systems only looks at the first
> Rem            8 characters of a label.  So do NOT be tempted to change
> Rem            chkapi* into something fancier like checkw32api
> Rem You HAVE been warned!

You can delete that warning.  You can also delete the parts of the
batch file which probe the environment for enough space, that problem
is also specific to command.com.  But please make these two changes a
separate commit, because they are unrelated to the issue of passing
options with the `=' characters.

> I would like to know if I can safely use labels that are longer than 8
> characters now.

Yes, you can.  But don't feel compelled to do that if you don't have
to ;-)

Thanks again for working on this.

Re: Supporting parameters that include the = character in configure.bat

[Juanma Barranquero]

On Thu, Apr 14, 2011 at 07:11, Eli Zaretskii <eliz@gnu.org> wrote:

> You can also delete the parts of the
> batch file which probe the environment for enough space, that problem
> is also specific to command.com.

And if you do that, you can remove this bit from nt/INSTALL:

    In addition, using 4NT or TCC as your shell is known to fail the build
    process, at least since 4NT version 3.01.

because the last time I tried, the only thing that made 4NT/TCC fail
was this line:

  if not "%$foo$%" == "123456789_123456789_123456789_123456789_123"
goto SmallEnv

as a consecuence of %$ being an internal variable for 4NT (it means
"all the remaining arguments").

    Juanma

Re: Supporting parameters that include the = character in configure.bat

[Ben Key]

[-- Attachment #1.1: Type: text/plain, Size: 1420 bytes --]

Hello,

I have attached to this message a patch that modifies the code in
configure.bat that parses the --cflags and --ldflags options to support
parameters that include the = character.  Note that this functionality
depends on command extensions.  Configure.bat now attempts to enable command
extensions and displays a warning message if they could not be
enabled.  If configure.bat
could not enable command extensions the old parsing code is used that does
not support parameters that include the = character.

Note that the technique I used requires that any --cflags or --ldflags
parameter that includes the = character be enclosed in quotes to prevent the
= character from being interpreted as a separator.  The quotes are striped
when the parameter is processed.  Note the the parameter does not need to be
enclosed in quotes if it does not include the = character.

This patch includes changes to the nt/INSTALL file provided by Eli
Zaretskii.

I tested these changes on Windows 2000, Windows XP, and Windows 7.  I also
tested in the case in which command extensions are disabled via the registry
on Windows XP.  Note that in the case in which command extensions are
disabled the setlocal command successfully enabled them.  I do not know of a
case in which configure.bat will fall back to the old parsing code (perhaps
Windows NT, I do not know because I do not have a PC running Windows NT to
use for testing).

[-- Attachment #1.2: Type: text/html, Size: 1664 bytes --]

[-- Attachment #2: emacs-nt-configure.patch --]
[-- Type: application/octet-stream, Size: 4482 bytes --]

=== modified file 'nt/INSTALL'
--- nt/INSTALL	2011-01-26 08:36:39 +0000
+++ nt/INSTALL	2011-04-14 17:34:05 +0000
@@ -220,13 +220,23 @@
   absolutely sure the produced binaries will never need to be run under
   a debugger.
 
-  Because of limitations of the stock Windows command shell, certain
-  characters (quotes, backslashes and equal signs) can be problematic
-  and should not be used in arguments to configure.  That means that
-  forward slashes must be used in paths passed to the compiler and
-  linker via the --cflags and --ldflags options, and that it is
-  currently not possible to pass a macro like -DFOO=BAR (though -DFOO
-  is perfectly valid).
+  Because of limitations of the stock Windows command shells, special
+  care is needed to pass some characters in the arguments of the
+  --cflags and --ldflags options.  Backslashes should not be used in
+  file names passed to the compiler and linker via these options.  Use
+  forward slashes instead.  If the arguments to these two options
+  include the `=' character, like when passing a -DFOO=bar preprocessor
+  option, the argument with the `=' character should be enclosed in
+  quotes, like this:
+
+    configure --cflags "-DFOO=bar"
+
+  Support for options that include the `=' character require "command
+  extensions" to be enabled.  (They are enabled by default, but your
+  system administrator could have changed that.  See "cmd /?" for
+  details.)  If command extensions are disabled, a warning message might
+  be displayed informing you that "using parameters that include the =
+  character by enclosing them in quotes will not be supported."
 
   N.B.  It is normal to see a few error messages output while configure
   is running, when gcc support is being tested.  These cannot be

=== modified file 'nt/configure.bat'
--- nt/configure.bat	2011-01-29 12:36:11 +0000
+++ nt/configure.bat	2011-04-14 17:35:15 +0000
@@ -75,6 +75,19 @@
 
 :start
 rem ----------------------------------------------------------------------
+rem   Attempt to enable command extensions.  Set use_extensions to 1 if
+rem   they are available and 0 if they are not available.
+set use_extensions=1
+setlocal ENABLEEXTENSIONS
+if "%CMDEXTVERSION%" == "" set use_extensions=0
+if "%use_extensions%" == "1" goto afterext
+
+echo. Command extensions are not available.  Using parameters that include the =
+echo. character by enclosing them in quotes will not be supported.
+
+:afterext
+
+rem ----------------------------------------------------------------------
 rem   Default settings.
 set prefix=
 set nodebug=N
@@ -136,6 +149,20 @@
 echo.   --without-xpm           do not use XPM library even if it is installed
 echo.   --with-svg              use the RSVG library (experimental)
 echo.   --distfiles             path to files for make dist, e.g. libXpm.dll
+if "%use_extensions%" == "0" goto end
+echo.
+echo. The cflags and ldflags arguments support parameters that include the =
+echo. character.  However, since the = character is normally treated as a
+echo. separator character you will need to enclose any parameter that includes
+echo. the = character in quotes.  For example, to include
+echo. -DSITELOAD_PURESIZE_EXTRA=100000 as one of the cflags you would run
+echo. configure.bat as follows:
+echo. configure.bat --cflags "-DSITELOAD_PURESIZE_EXTRA=100000"
+echo.
+echo. Note that this capability of processing parameters that include the =
+echo. character depends on command extensions.  This batch file attempts to
+echo. enable command extensions.  If command extensions cannot be enabled, a
+echo. warning message will be displayed.
 goto end
 
 rem ----------------------------------------------------------------------
@@ -198,6 +225,17 @@
 rem ----------------------------------------------------------------------
 
 :usercflags
+if "%use_extensions%" == "1" goto ucflagex
+goto ucflagne
+
+:ucflagex
+shift
+set usercflags=%usercflags%%sep1%%~1
+set sep1= %nothing%
+shift
+goto again
+
+:ucflagne
 shift
 set usercflags=%usercflags%%sep1%%1
 set sep1= %nothing%
@@ -207,6 +245,17 @@
 rem ----------------------------------------------------------------------
 
 :userldflags
+if "%use_extensions%" == "1" goto ulflagex
+goto ulflagne
+
+:ulflagex
+shift
+set userldflags=%userldflags%%sep2%%~1
+set sep2= %nothing%
+shift
+goto again
+
+:ulflagne
 shift
 set userldflags=%userldflags%%sep2%%1
 set sep2= %nothing%

Re: Supporting parameters that include the = character in configure.bat

[Eli Zaretskii]

> From: Ben Key <bkey76@gmail.com>
> Date: Thu, 14 Apr 2011 19:56:03 -0500
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs-devel@gnu.org
> 
> I have attached to this message a patch that modifies the code in
> configure.bat that parses the --cflags and --ldflags options to support
> parameters that include the = character.

Looks fine to me.  Please go ahead and commit it (and don't forget the
ChangeLog entry ;-).

Thanks for doing this.