windows build help

windows build help

[Phillip Lord]

I was just reading 28601@debbugs.gnu.org. I have a slightly related
problem and wonder if anyone could help.

Currently, I am building the windows binaries for Emacs using an
out-of-source build with a directory structure like this:

git/emacs-26.0.50

build/emacs-26.0.50/x86_64
build/emacs-26.0.50/i686

install/emacs-26.0.50/x86_64
install/emacs-26.0.50/i686

This works nicely, but some of the process is manual. I wanted to script
everything up to and including the creation of the final zip files --
currently, the entire build takes about an hour or two.

But I do the x86_64 building using the MINGW64 shell
(i.e. C:\msys64\mingw64.exe) and the i686 using MINGW32
(i.e. C:\msys64\mingw64.exe). You might have thought I could just used
one -- the build for the i686 would just be cross compiling. But, this
doesn't work because I get this from ./configure:

checking for struct ifreq.ifr_addr.sa_len... no
checking whether gcc understands -MMD -MF... no
checking for long file names... yes
checking for X... no
checking whether Windows API headers are recent enough... no
configure: error: the Windows API headers are too old to support this build.

So, I cannot do both builds with a single command -- I have to come back
in the middle. I don't really understand the difference between the two
shells nor why I get this error.

Another solution would be to run both builds in parallel; this fails,
unfortunately, because the "out-of-source" build is not entirely out of
source -- the .elc files are created in-source. If I launch both a 64
and 32 bit build at the same time, they fight for whole compiles what,
until one or both crash.

It's not a major issue, but I'd quite like to start providing emacs-27
snapshots every couple of weeks. It's not going to happen, if it takes
too much effort.

Any thoughts welcome.

Phil

Re: windows build help

[Óscar Fuentes]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> So, I cannot do both builds with a single command -- I have to come back
> in the middle. I don't really understand the difference between the two
> shells nor why I get this error.

Look at the scripts that start each shell. They setup some environment
variables. Mimic that on your script and you should be fine.

Re: windows build help

[Stephen Leake]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Another solution would be to run both builds in parallel; this fails,
> unfortunately, because the "out-of-source" build is not entirely out of
> source -- the .elc files are created in-source. If I launch both a 64
> and 32 bit build at the same time, they fight for whole compiles what,
> until one or both crash.

Have two source checkouts. then you could also build in-source, although
I doubt that makes much difference.


-- 
-- Stephe

Re: windows build help

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Another solution would be to run both builds in parallel; this fails,
>> unfortunately, because the "out-of-source" build is not entirely out of
>> source -- the .elc files are created in-source. If I launch both a 64
>> and 32 bit build at the same time, they fight for whole compiles what,
>> until one or both crash.
>
> Have two source checkouts. then you could also build in-source, although
> I doubt that makes much difference.

I thought about that, but the reason against it is that it takes much
longer, because I have to build the elc twice.

Phil

Re: windows build help

[Stephen Leake]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> phillip.lord@russet.org.uk (Phillip Lord) writes:
>>
>>> Another solution would be to run both builds in parallel; this fails,
>>> unfortunately, because the "out-of-source" build is not entirely out of
>>> source -- the .elc files are created in-source. If I launch both a 64
>>> and 32 bit build at the same time, they fight for whole compiles what,
>>> until one or both crash.
>>
>> Have two source checkouts. then you could also build in-source, although
>> I doubt that makes much difference.
>
> I thought about that, but the reason against it is that it takes much
> longer, because I have to build the elc twice.

Hmm. You are implying that the elc built for 32 bit emacs are identical
to the elc built for 64 bit emacs. 

Is that true? I would have guessed that most objects are either
32 bit or 64 bit; elisp manual 3.1 says the integer range depends on the
machine, and pointers should also.

If they are the same, then you should be able to write a script that
does:

1) parallel make to build elc using 32 bit
2) parallel make to build everything else, both 32 bit and 64 bit.

-- 
-- Stephe

Re: windows build help

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>>
>>> phillip.lord@russet.org.uk (Phillip Lord) writes:
>>>
>>>> Another solution would be to run both builds in parallel; this fails,
>>>> unfortunately, because the "out-of-source" build is not entirely out of
>>>> source -- the .elc files are created in-source. If I launch both a 64
>>>> and 32 bit build at the same time, they fight for whole compiles what,
>>>> until one or both crash.
>>>
>>> Have two source checkouts. then you could also build in-source, although
>>> I doubt that makes much difference.
>>
>> I thought about that, but the reason against it is that it takes much
>> longer, because I have to build the elc twice.
>
> Hmm. You are implying that the elc built for 32 bit emacs are identical
> to the elc built for 64 bit emacs. 
>
> Is that true? I would have guessed that most objects are either
> 32 bit or 64 bit; elisp manual 3.1 says the integer range depends on the
> machine, and pointers should also.

Yes, I believe that they are. In fact, the source tar-ball includes the
ELC files, when it's possible to build from source on many different
platforms.

I use to build both 32 and 64 totally form source (i.e. without ELC),
but haven't done for I think all of Emacs-25.


> If they are the same, then you should be able to write a script that
> does:
>
> 1) parallel make to build elc using 32 bit
> 2) parallel make to build everything else, both 32 bit and 64 bit.

I have this now, thanks to comments in this thread. I do, parallel build
of 64 bit, then parallel build of 32 bit. The ELC build happens during
the 64, although it could happen either way.

Phil

Re: windows build help

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 14 Oct 2017 10:15:41 -0500
> 
> Hmm. You are implying that the elc built for 32 bit emacs are identical
> to the elc built for 64 bit emacs. 
> 
> Is that true?

Yes.

Re: windows build help

[Paul Eggert]

Stephen Leake wrote:
> You are implying that the elc built for 32 bit emacs are identical
> to the elc built for 64 bit emacs.

Although one can write Elisp code that compiles to different .elc files 
depending whether the compiling Emacs is built for a 32 or 64 bit machine, such 
Elisp code is unportable and the GNU Emacs source code does not do that, as far 
as I know. (It might be nice if we routinely tested for such portability bugs.)

Re: windows build help

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Although one can write Elisp code that compiles to different .elc files 
  > depending whether the compiling Emacs is built for a 32 or 64 bit machine, 

Just for curiosity's sake, would anyone like to tell me what circumstances
this happens in?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: windows build help

[Paul Eggert]

>    > one can write Elisp code that compiles to different .elc files
>    > depending whether the compiling Emacs is built for a 32 or 64 bit machine,
> 
> Just for curiosity's sake, would anyone like to tell me what circumstances
> this happens in?

For example, given this source code:

(defconst x (* 32768 32768))

On a 64-bit machine, byte-compiling generates an .elc file containing this:

(defconst x 1073741824)

On a 32-bit machine, byte-compiling the same source code generates an .elc file 
containing this instead:

(defconst x 0)

due to multiplication overflow in this Emacs's integers, which use 30-bit two's 
complement. Each .elc file is correct for the Emacs it is built with, but 
incorrect for the other Emacs.

GNU Emacs source code should not do this sort of thing, since the .elc files 
distributed in its tarballs are supposed to be portable.

Re: windows build help

[Stefan Monnier]

> (defconst x (* 32768 32768))

Indeed.  This one is arguably undesirable.
But there are more direct ones, where you can't similarly argue that
it's a limitation in Elisp:

    (defconst foo-build-system (eval-when-compile system-configuration))


-- Stefan

Re: windows build help

[Noam Postavsky]

On Fri, Oct 13, 2017 at 2:52 PM, Phillip Lord
<phillip.lord@russet.org.uk> wrote:

> But I do the x86_64 building using the MINGW64 shell
> (i.e. C:\msys64\mingw64.exe) and the i686 using MINGW32
> (i.e. C:\msys64\mingw64.exe). You might have thought I could just used
> one -- the build for the i686 would just be cross compiling.

AFAIK, emacs doesn't support cross compiling.

> So, I cannot do both builds with a single command -- I have to come back
> in the middle. I don't really understand the difference between the two
> shells nor why I get this error.

I think the only difference is PATH has /mingw32/bin or /mingw64/bin,
and MSYSTEM is MINGW32 or MINGW64

At least for me, after doing

PATH=/mingw32/bin:$PATH
MSYSTEM=MINGW32

in a mingw64 shell, doing ./configure succeeds and I end up with a 32bit emacs.

Re: windows build help

[Phillip Lord]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> On Fri, Oct 13, 2017 at 2:52 PM, Phillip Lord
> <phillip.lord@russet.org.uk> wrote:
>
>> But I do the x86_64 building using the MINGW64 shell
>> (i.e. C:\msys64\mingw64.exe) and the i686 using MINGW32
>> (i.e. C:\msys64\mingw64.exe). You might have thought I could just used
>> one -- the build for the i686 would just be cross compiling.
>
> AFAIK, emacs doesn't support cross compiling.

I assumed that only GCC needed to do this (assuming that the emacs
produced can run on the current system).

>
>> So, I cannot do both builds with a single command -- I have to come back
>> in the middle. I don't really understand the difference between the two
>> shells nor why I get this error.
>
> I think the only difference is PATH has /mingw32/bin or /mingw64/bin,
> and MSYSTEM is MINGW32 or MINGW64
>
> At least for me, after doing
>
> PATH=/mingw32/bin:$PATH
> MSYSTEM=MINGW32
>
> in a mingw64 shell, doing ./configure succeeds and I end up with a
> 32bit emacs.

Yep, that's all you need. Thank you very much!

Phil

Re: windows build help

[Angelo Graziosi]

Phillip Lord wrote:

> But I do the x86_64 building using the MINGW64 shell
> (i.e. C:\msys64\mingw64.exe) and the i686 using MINGW32

in my script I added this:

  MSYSTEM="MINGW64"
  PATH="/mingw64/bin:$PATH"

Also this should work:

  MSYSTEM=MINGW64 PATH="/mingw64/bin:$PATH" ~/emacs-w64-build.sh

> the entire build takes about an hour or two

On which machine? Here (an Athlon X2, 10 years old), with 'make -j3', the build [1] takes about 35-40 minutes [2].

Ciao,
 Angelo.


-----
[1] From http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-master.tar.gz
[2] Download of [1], unpacking, build, and tar.xz creation.

Re: windows build help

[Phillip Lord]

Angelo Graziosi <angelo.g0@libero.it> writes:

> Phillip Lord wrote:
>
>> But I do the x86_64 building using the MINGW64 shell
>> (i.e. C:\msys64\mingw64.exe) and the i686 using MINGW32
>
> in my script I added this:
>
>   MSYSTEM="MINGW64"
>   PATH="/mingw64/bin:$PATH"
>
> Also this should work:
>
>   MSYSTEM=MINGW64 PATH="/mingw64/bin:$PATH" ~/emacs-w64-build.sh
>
>> the entire build takes about an hour or two
>
> On which machine? Here (an Athlon X2, 10 years old), with 'make -j3',
> the build [1] takes about 35-40 minutes [2].

The majority of the time is the elc compilation. So, the first build
takes a long time, second much (much) less.

There is also quite a lot of baggage on top of this. After the zip files
have been build, I have to unzip them, so I check they run cleanly. This
takes around 5 minutes per zip. For Emacs-26, I'm making four zips. Then
the uploads. I normally do it over two days.

Phil