Making gnulib imports build on Windows

Making gnulib imports build on Windows

[Eli Zaretskii]

One problem with these imports that may mean a relatively significant
change in the build prerequisites is the fact that during the build,
getopt.in.h needs to be edited into getopt.h.  This editing is done by
Sed in lib/Makefile.in:

  getopt.h: getopt.in.h $(ARG_NONNULL_H)
	  $(AM_V_GEN)rm -f $@-t $@ && \
	  { echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
	    sed -e 's|@''HAVE_GETOPT_H''@|$(HAVE_GETOPT_H)|g' \
		-e 's|@''INCLUDE_NEXT''@|$(INCLUDE_NEXT)|g' \
		-e 's|@''PRAGMA_SYSTEM_HEADER''@|@PRAGMA_SYSTEM_HEADER@|g' \
		-e 's|@''PRAGMA_COLUMNS''@|@PRAGMA_COLUMNS@|g' \
		-e 's|@''NEXT_GETOPT_H''@|$(NEXT_GETOPT_H)|g' \
		-e '/definition of _GL_ARG_NONNULL/r $(ARG_NONNULL_H)' \
		< $(srcdir)/getopt.in.h; \
	  } > $@-t && \
	  mv -f $@-t $@

Until now, building Emacs for Windows did not require Sed.  I see the
following possible ways to deal with this:

  1) Add Sed to programs required for the build, and run it as part of
     each build.

  2) Have a maintainer-only rule to produce getopt.h for Windows (by
     using Sed).  Someone(TM) will invoke that rule whenever
     getopt.in.h changes, and commit the result to the bzr repo, to be
     used by all the other people who build Emacs on Windows.

  3) Write an auxiliary C program to perform replacements like Sed
     does (but without all the other bells and whistles in Sed), build
     it at the beginning of the build, and use it instead of Sed.

I don't like asking every end user to have Sed, especially since many
Windows ports of Sed are notoriously broken (I had to port it myself
to get a reliable tool).  And since 3) sounds like wheel reinvention,
I tend to alternative 2).

Comments and ideas are welcome.

Re: Making gnulib imports build on Windows

[Paul Eggert]

On 01/26/11 11:29, Eli Zaretskii wrote:
>   2) Have a maintainer-only rule to produce getopt.h for Windows (by
>      using Sed).

Yes, that sounds like the best approach to me as well.  I expect
this sort of thing can be useful for other tasks needed for
an MS-DOS build.  Also, surely it is OK if these maintainer-only
tasks make extra assumptions about tools, for example, if they
assume the use of GNU 'make' or GNU 'sed'.

Re: Making gnulib imports build on Windows

[Eli Zaretskii]

> Date: Wed, 26 Jan 2011 13:48:28 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> On 01/26/11 11:29, Eli Zaretskii wrote:
> >   2) Have a maintainer-only rule to produce getopt.h for Windows (by
> >      using Sed).
> 
> Yes, that sounds like the best approach to me as well.

Thanks for the feedback.

> I expect this sort of thing can be useful for other tasks needed for
> an MS-DOS build.

I wasn't talking about the MS-DOS build (which already uses Sed and
the original Makefile.in files, so there's no issues in that case).  I
was talking about the MS-Windows build, which has its own
makefile.w32-in templates.  The issue is what to put in
makefile.w32-in that does the equivalent of editing of *.in.h files.

Re: Making gnulib imports build on Windows

[Andy Moreton]

On Wed 26 Jan 2011, Eli Zaretskii wrote:

> Until now, building Emacs for Windows did not require Sed.  I see the
> following possible ways to deal with this:

[...]

> I don't like asking every end user to have Sed, especially since many
> Windows ports of Sed are notoriously broken (I had to port it myself
> to get a reliable tool).  And since 3) sounds like wheel reinvention,
> I tend to alternative 2).
>
> Comments and ideas are welcome.

Although Cygwin and MSYS are widely used for building emacs for Windows,
I agree that option 2) is the best. The maintainer-only rules
for generating the pre-canned configuration and headers should minimise
the doffernces from the standard POSIX makefile infrastructure as far as
possible.

In a perfect world, this would allow the use of mingw32 or mingw-w64
cross compilers to build Windows emacs binaries on POSIX hosts.

What can I do to help this happen ?

    AndyM

Re: Making gnulib imports build on Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 26 Jan 2011 22:40:30 +0000
> 
> Although Cygwin and MSYS are widely used for building emacs for Windows,
> I agree that option 2) is the best.

Thanks for the feedback.

> The maintainer-only rules for generating the pre-canned
> configuration and headers should minimise the doffernces from the
> standard POSIX makefile infrastructure as far as possible.
> 
> In a perfect world, this would allow the use of mingw32 or mingw-w64
> cross compilers to build Windows emacs binaries on POSIX hosts.
> 
> What can I do to help this happen ?

Sorry, I'm not sure I'm following: help what happen?  If you mean
unbreak the Windows build, then either (a) describing what exactly
needs to be done for that or (b) actually doing that is what's
needed.  I'm almost done doing (a) myself, but I can never be sure I
figured out everything until I actually make the changes and try
building.

Re: Making gnulib imports build on Windows

[Andy Moreton]

On Thu 27 Jan 2011, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Wed, 26 Jan 2011 22:40:30 +0000
>> 
>> Although Cygwin and MSYS are widely used for building emacs for Windows,
>> I agree that option 2) is the best.
>
> Thanks for the feedback.
>
>> The maintainer-only rules for generating the pre-canned
>> configuration and headers should minimise the doffernces from the
>> standard POSIX makefile infrastructure as far as possible.
>> 
>> In a perfect world, this would allow the use of mingw32 or mingw-w64
>> cross compilers to build Windows emacs binaries on POSIX hosts.
>> 
>> What can I do to help this happen ?
>
> Sorry, I'm not sure I'm following: help what happen?  If you mean
> unbreak the Windows build, then either (a) describing what exactly
> needs to be done for that or (b) actually doing that is what's
> needed.  I'm almost done doing (a) myself, but I can never be sure I
> figured out everything until I actually make the changes and try
> building.

Sorry if I was unclear. Initially the w32 build needs to be repaired,
but the longer term aim should surely be reducing the differences
between w32 build scripts ans the rest of emacs. This includes allowing
for additional toolchains (e.g. Mingw-w64, llvm).

Mingw32 and wingw-w64 cross toolchains are available in some Linux distros. If 
developers on other platforms could at least build the win32 port, it
would make it easier for those developers to check whether their changes
break the w32 build. It might also simplify building tarballs and binary
releases.

    AndyM

Re: Making gnulib imports build on Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Thu, 27 Jan 2011 14:22:35 +0000
> 
> Mingw32 and wingw-w64 cross toolchains are available in some Linux distros. If 
> developers on other platforms could at least build the win32 port, it
> would make it easier for those developers to check whether their changes
> break the w32 build. It might also simplify building tarballs and binary
> releases.

IMO, a developer who uses a cross-compilation environment on a Posix
platform should simply run the stock configure script in the top-level
directory, using the cross-compiler, cross-linker, etc.  If there are
any problems with that (e.g., the tests made by the configure script
do not cater to MinGW targets), they should be fixed in the Posix
configury stuff, I think.

Do you agree?  If so, the first step to see how well (or how badly ;-)
this works is to try.

Re: Making gnulib imports build on Windows

[Andy Moreton]

On Thu 27 Jan 2011, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Thu, 27 Jan 2011 14:22:35 +0000
>> 
>> Mingw32 and wingw-w64 cross toolchains are available in some Linux
>> distros. If developers on other platforms could at least build the
>> win32 port, it would make it easier for those developers to check
>> whether their changes break the w32 build. It might also simplify
>> building tarballs and binary releases.
>
> IMO, a developer who uses a cross-compilation environment on a Posix
> platform should simply run the stock configure script in the top-level
> directory, using the cross-compiler, cross-linker, etc.  If there are
> any problems with that (e.g., the tests made by the configure script
> do not cater to MinGW targets), they should be fixed in the Posix
> configury stuff, I think.
>
> Do you agree?  If so, the first step to see how well (or how badly ;-)
> this works is to try.

Yes. I've not tried it, but I suspect that the POSIX configury will ignore
the /nt directory and other Win32 stuff entirely.

    AndyM

Re: Making gnulib imports build on Windows

[Lennart Borgman]

On Thu, Jan 27, 2011 at 3:22 PM, Andy Moreton <andrewjmoreton@gmail.com> wrote:
>
> Sorry if I was unclear. Initially the w32 build needs to be repaired,
> but the longer term aim should surely be reducing the differences
> between w32 build scripts ans the rest of emacs. This includes allowing
> for additional toolchains (e.g. Mingw-w64, llvm).
>
> Mingw32 and wingw-w64 cross toolchains are available in some Linux distros. If
> developers on other platforms could at least build the win32 port, it
> would make it easier for those developers to check whether their changes
> break the w32 build. It might also simplify building tarballs and binary
> releases.

I agree that this might be good goals, but could please such
development be kept in a separate branch, not on the trunk?

Re: Making gnulib imports build on Windows

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Thu, 27 Jan 2011 15:34:16 +0100
> Cc: emacs-devel@gnu.org
> 
> On Thu, Jan 27, 2011 at 3:22 PM, Andy Moreton <andrewjmoreton@gmail.com> wrote:
> >
> > Sorry if I was unclear. Initially the w32 build needs to be repaired,
> > but the longer term aim should surely be reducing the differences
> > between w32 build scripts ans the rest of emacs. This includes allowing
> > for additional toolchains (e.g. Mingw-w64, llvm).
> >
> > Mingw32 and wingw-w64 cross toolchains are available in some Linux distros. If
> > developers on other platforms could at least build the win32 port, it
> > would make it easier for those developers to check whether their changes
> > break the w32 build. It might also simplify building tarballs and binary
> > releases.
> 
> I agree that this might be good goals, but could please such
> development be kept in a separate branch, not on the trunk?

If you assume that this development (if indeed Andy decides to work on
it) will somehow get in the way of the native Windows build, then your
assumption is wrong.  See my response to Andy.

Re: Making gnulib imports build on Windows

[Lennart Borgman]

On Thu, Jan 27, 2011 at 4:13 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >
>> > Mingw32 and wingw-w64 cross toolchains are available in some Linux distros. If
>> > developers on other platforms could at least build the win32 port, it
>> > would make it easier for those developers to check whether their changes
>> > break the w32 build. It might also simplify building tarballs and binary
>> > releases.
>>
>> I agree that this might be good goals, but could please such
>> development be kept in a separate branch, not on the trunk?
>
> If you assume that this development (if indeed Andy decides to work on
> it) will somehow get in the way of the native Windows build, then your
> assumption is wrong.  See my response to Andy.

Yes, I did assume that. If not that is fine.

Re: Making gnulib imports build on Windows

[Sam Steingold]

> * Eli Zaretskii <ryvm@tah.bet> [2011-01-26 21:29:59 +0200]:
>
> I don't like asking every end user to have Sed, especially since many
> Windows ports of Sed are notoriously broken (I had to port it myself
> to get a reliable tool).  And since 3) sounds like wheel reinvention,
> I tend to alternative 2).

I think 1 is the best long-term approach.

First of all, "windows culture" does not include building one's tools,
so, IMO, most windows Emacs users use the pre-built executables, thus
this change will affect only a few people.

Second, cygwin has been around for quite some time and it stable enough
for most purposes. I bet that most people who build Emacs on windows
already have cygwin installed. Yes, Eli is one important exception, but
I doubt that there are many (any?) others.
And those who hate cygwin can get gnuwin32.

Third, and most important: it is best to have one build system for all
platforms instead of separate build systems for unixes and windows.
I am talking from personal experience: GNU CLISP configures and builds
with the exact same command line on all platforms (well, you have to add
"--with-mingw" to configure on windows) and this has saved me a lot of
maintenance time and effort.
It is inevitable that more tools in addition to sed will seep into the
Emacs build process (especially since Emacs now relies on gnulib);
working around that will take Eli's time from real development into
maintaining a byzantine structure.

Yes, the price is installing gnuwin or cygwin.
However, I claim this is not a big deal: checking out Emacs bzr tree
consumes 0.5+G and cygwin/gnuwin take about that much each.

Yes, the up-front cost is higher: discarding all the configure.bat gunk
and switching to autoconf on windows will take more effort than simply
distributing pre-generated getopt.h and whatnot.
However, I claim that this cost will be paid eventually, sooner or
later, and the later this is done, the more effort this will require.
You cannot hold Emacs/windows build system with duct tape forever.

I have made this suggestion in the past and I know that Eli does not
like it. However, I am making it again here so that in a few years, when
Eli inevitably sees the light and relents, I can say "I told you so"
(provided I am still around :-).

-- 
Sam Steingold (http://sds.podval.org/) on CentOS release 5.3 (Final)
http://pmw.org.il http://www.memritv.org http://thereligionofpeace.com
http://palestinefacts.org http://camera.org http://truepeace.org
The software said it requires Windows 3.1 or better, so I installed Linux.

Re: Making gnulib imports build on Windows

[Lennart Borgman]

On Thu, Jan 27, 2011 at 5:43 PM, Sam Steingold <sds@gnu.org> wrote:
>
> I have made this suggestion in the past and I know that Eli does not
> like it. However, I am making it again here so that in a few years, when
> Eli inevitably sees the light and relents, I can say "I told you so"
> (provided I am still around :-).

One way to convince Eli might be to actually try it out and show the
solution. But I am not entirely sure you will survive long enough to
succeed with finding the solution... - it might include more work than
you expect, trying to fix some bugs here and some bugs there. And
convincing those responsible for the tools to include your changes so
that everything is stable (you might get the message they do not care
that much about w32 to change anything...).

But it would good if you did succeed.

Re: Making gnulib imports build on Windows

[Sam Steingold]

On Thu, Jan 27, 2011 at 11:58 AM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Thu, Jan 27, 2011 at 5:43 PM, Sam Steingold <sds@gnu.org> wrote:
>>
>> I have made this suggestion in the past and I know that Eli does not
>> like it. However, I am making it again here so that in a few years, when
>> Eli inevitably sees the light and relents, I can say "I told you so"
>> (provided I am still around :-).
>
> One way to convince Eli might be to actually try it out and show the
> solution.

ATM I don't have time even for clisp, let alone emacs.
You can look at the clisp sources, which build using the standard
autoconf tool (configure &c) on windows OOTB.

-- 
Sam Steingold <http://sds.podval.org>

Re: Making gnulib imports build on Windows

[Juanma Barranquero]

On Thu, Jan 27, 2011 at 17:43, Sam Steingold <sds@gnu.org> wrote:

> Second, cygwin has been around for quite some time and it stable enough
> for most purposes. I bet that most people who build Emacs on windows
> already have cygwin installed. Yes, Eli is one important exception, but
> I doubt that there are many (any?) others.

Yes, there are others. I don't have cygwin on my computer, and I won't.

    Juanma

Re: Making gnulib imports build on Windows

[martin rudalics]

 >> Second, cygwin has been around for quite some time and it stable enough
 >> for most purposes. I bet that most people who build Emacs on windows
 >> already have cygwin installed. Yes, Eli is one important exception, but
 >> I doubt that there are many (any?) others.
 >
 > Yes, there are others. I don't have cygwin on my computer, and I won't.

Same here.  I had cygwin installed on another machine and utterly
disliked it.

martin

Re: Making gnulib imports build on Windows

[Sam Steingold]

>>> Second, cygwin has been around for quite some time and it stable enough
>>> for most purposes. I bet that most people who build Emacs on windows
>>> already have cygwin installed. Yes, Eli is one important exception, but
>>> I doubt that there are many (any?) others.
>>
>> Yes, there are others. I don't have cygwin on my computer, and I won't.
>
> Same here.  I had cygwin installed on another machine and utterly
> disliked it.

So how do you all live without find/grep/sed/bash/&c&c&c?!

At any rate, this is getting too close to a religious dispute, I am bailing out...

-- 
Sam Steingold (http://sds.podval.org/) on CentOS release 5.3 (Final)
http://jihadwatch.org http://truepeace.org http://www.memritv.org
http://palestinefacts.org http://mideasttruth.com
Single tasking: Just Say No.

Re: Making gnulib imports build on Windows

[Juanma Barranquero]

On Fri, Jan 28, 2011 at 16:12, Sam Steingold <sds@gnu.org> wrote:

> So how do you all live without find/grep/sed/bash/&c&c&c?!

Some of those I have, from GnuWin32 (grep, for example). Most others
(like bash) I don't need.

    Juanma

Re: Making gnulib imports build on Windows

[Sean Sieger]

martin rudalics <rudalics@gmx.at> writes:

    >> Second, cygwin has been around for quite some time and it stable enough
    >> for most purposes. I bet that most people who build Emacs on windows
    >> already have cygwin installed. Yes, Eli is one important exception, but
    >> I doubt that there are many (any?) others.
    >
    > Yes, there are others. I don't have cygwin on my computer, and I won't.

    Same here.  I had cygwin installed on another machine and utterly
    disliked it.

Me either.  Just MinG, GnuWin32 and the stuff I hand-built using them.

Re: Making gnulib imports build on Windows

[Sam Steingold]

> * Sean Sieger <frna.fvrtre@tznvy.pbz> [2011-01-28 12:32:06 -0500]:
>
> martin rudalics <rudalics@gmx.at> writes:
>
>     >> Second, cygwin has been around for quite some time and it stable enough
>     >> for most purposes. I bet that most people who build Emacs on windows
>     >> already have cygwin installed. Yes, Eli is one important exception, but
>     >> I doubt that there are many (any?) others.
>     >
>     > Yes, there are others. I don't have cygwin on my computer, and I won't.
>
>     Same here.  I had cygwin installed on another machine and utterly
>     disliked it.
>
> Me either.  Just MinG, GnuWin32 and the stuff I hand-built using them.

okay, so what you already have is probably enough to use autoconf :-)

-- 
Sam Steingold (http://sds.podval.org/) on CentOS release 5.3 (Final)
http://jihadwatch.org http://thereligionofpeace.com http://www.memritv.org
http://iris.org.il http://pmw.org.il http://www.PetitionOnline.com/tap12009/
Those who value Life above Freedom are destined to lose both.

Re: Making gnulib imports build on Windows

[Martin Stemplinger]

Sam Steingold schrieb am 28.01.2011:

>> Me either.  Just MinG, GnuWin32 and the stuff I hand-built using them.
>
> okay, so what you already have is probably enough to use autoconf :-)

There is even an autoconf package available in Gnuwin32. It's version
2.63 compared to version 2.68 on the autoconf homepage. Would this be
recent enough to use in building emacs?

Martin

Re: Making gnulib imports build on Windows

[Lennart Borgman]

On Fri, Jan 28, 2011 at 9:44 PM, Martin Stemplinger <mstemplinger@gmx.de> wrote:
> Sam Steingold schrieb am 28.01.2011:
>
>>> Me either.  Just MinG, GnuWin32 and the stuff I hand-built using them.
>>
>> okay, so what you already have is probably enough to use autoconf :-)
>
> There is even an autoconf package available in Gnuwin32. It's version
> 2.63 compared to version 2.68 on the autoconf homepage. Would this be
> recent enough to use in building emacs?


Are there any tests you can check this with? Does these tests pass on w32?

Re: Making gnulib imports build on Windows

[Martin Stemplinger]

Sam Steingold schrieb am 27.01.2011:

> First of all, "windows culture" does not include building one's tools,
> so, IMO, most windows Emacs users use the pre-built executables, thus
> this change will affect only a few people.
Ok then I'm one of the few ;-). I like to run the current development
version so I compile quite frequently.
>
> Second, cygwin has been around for quite some time and it stable enough
> for most purposes. I bet that most people who build Emacs on windows
> already have cygwin installed. Yes, Eli is one important exception, but
> I doubt that there are many (any?) others.

For sure I'm not important but I neither have cygwin installed. Up to
now mingw+gnuwin32 were all I needed to build on Windows. 

> And those who hate cygwin can get gnuwin32.

Gnuwin is needed anyway if you want image support. So at least for me
it would be no big deal to install also the sed package from gnuwin to
build. But I only talk about the windows build on XP, vista and
windows 7 not about the MS-DOS build that Eli also maintains. Things
may be completely different there.

> Third, and most important: it is best to have one build system for all
> platforms instead of separate build systems for unixes and windows.
> I am talking from personal experience: GNU CLISP configures and builds
> with the exact same command line on all platforms (well, you have to add
> "--with-mingw" to configure on windows) and this has saved me a lot of
> maintenance time and effort.

Fully agreed. E.g. make dist is currently broken because some of the
files like BUGS, COPYING, README, INSTALL are not copied to the
install dir. At least I alsways need to copy those manually after make
install for make dist to succeed.

> I have made this suggestion in the past and I know that Eli does not
> like it. However, I am making it again here so that in a few years, when
> Eli inevitably sees the light and relents, I can say "I told you so"
> (provided I am still around :-).
Eli has probably struggled a lot to get the various windows versions
to work and I really appreciate this very much because it helps me to
have the true editor also on windows. 

Regards,
Martin

Re: Making gnulib imports build on Windows

[Eli Zaretskii]

> From: Martin Stemplinger <mstemplinger@gmx.de>
> Date: Thu, 27 Jan 2011 21:32:35 +0100
> 
> > And those who hate cygwin can get gnuwin32.
> 
> Gnuwin is needed anyway if you want image support. So at least for me
> it would be no big deal to install also the sed package from gnuwin to
> build. But I only talk about the windows build on XP, vista and
> windows 7 not about the MS-DOS build that Eli also maintains. Things
> may be completely different there.

Things _are_ completely different with the MS-DOS port: Sed is already
required to build it.  And, unlike with GnuWin32, the DOS
(a.k.a. DJGPP) port of Sed is rock-solid and up to date (as generally
are all the other DJGPP ports of GNU software).

As for requiring Sed for the Windows build: those who don't mind
installing it will be at advantage, in that they will be able to
regenerate getopt.h whenever a new version of getopt.in.h gets
imported from gnulib.  Those who don't have Sed will have to wait for
someone who does to update the repository with the regenerated files.

> E.g. make dist is currently broken because some of the
> files like BUGS, COPYING, README, INSTALL are not copied to the
> install dir. At least I alsways need to copy those manually after make
> install for make dist to succeed.

Please submit bug reports about any such omissions, thanks.

> > I have made this suggestion in the past and I know that Eli does not
> > like it. However, I am making it again here so that in a few years, when
> > Eli inevitably sees the light and relents, I can say "I told you so"
> > (provided I am still around :-).
> Eli has probably struggled a lot to get the various windows versions
> to work and I really appreciate this very much because it helps me to
> have the true editor also on windows. 

Thanks, but the credit is not mine to take.  The current build system
on Windows was done by others (see nt/ChangeLog), I'm just trying not
to break it ;-)