* msys2 building issue
@ 2018-03-27 21:20 Phillip Lord
2018-03-29 12:22 ` Eli Zaretskii
0 siblings, 1 reply; 5+ messages in thread
From: Phillip Lord @ 2018-03-27 21:20 UTC (permalink / raw)
To: emacs-devel
I'm struggling with an issue with the snapshot build for windows. I have
a small script that in turn runs admin/nt/dist-build/build-zips.sh. The
build times (based on when the output files are created) make no sense.
x86_64
no-deps zip 17min
Deps-zip 3min
installer 25min
i686
no deps zip 31min
deps zip 47min
installer 61min
So, the installer is really slow to build (probably because of the
compression I guess). But the second i686 step is really slow,
especially the building the zips and installer.
I don't think this is the Emacs build per se; it looks to me like a
memory leak; the build process just gets slower and slower. In fact,
I've had to reduce the parallalism of make or the whole process crashes
with a resource allocation error. But Process Manager reports that the
VM has memory left.
I ask about all this, because I have no idea how to debug this further,
and I thought someone might have an idea.
My launch script is below, and build-zips.sh is in the repo.
#!/bin/bash
set -o errexit
## List the ssh-keys -- this has the crashes the script if there are none
ssh-add -l
pushd master/admin/nt/dist-build
## Update repo
git pull
./build-zips.sh -sn | tee ../../../../build.log
popd
pushd master
## Make the source so we can distribute it, but don't update because
## we've build in a different directory. --no-changelog is also needed
## because otherwise we run make which runs configure
./make-dist --snapshot --no-check --no-changelog
rename .tar _`date +%Y-%m-%d`.tar emacs*.tar.gz
mv emacs-*.tar.gz ~/emacs-upload
popd
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: msys2 building issue
2018-03-27 21:20 msys2 building issue Phillip Lord
@ 2018-03-29 12:22 ` Eli Zaretskii
2018-04-10 12:00 ` Phillip Lord
0 siblings, 1 reply; 5+ messages in thread
From: Eli Zaretskii @ 2018-03-29 12:22 UTC (permalink / raw)
To: Phillip Lord; +Cc: emacs-devel
> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Tue, 27 Mar 2018 22:20:34 +0100
>
> x86_64
> no-deps zip 17min
> Deps-zip 3min
> installer 25min
>
> i686
> no deps zip 31min
> deps zip 47min
> installer 61min
>
> So, the installer is really slow to build (probably because of the
> compression I guess). But the second i686 step is really slow,
> especially the building the zips and installer.
Does this happen only when building both the 64-bit and the 32-bit
zips? What if you build only the 32-bit one -- do you get the same
slow times?
My next suspect is the way MSYS2 lets you build 32-bit binaries. I
know nothing about that, all I see is that you set 2 environment
variables, but could that cause Bash or Make or some other component
to run out of memory due to a bug or a memory leak?
> I don't think this is the Emacs build per se; it looks to me like a
> memory leak; the build process just gets slower and slower. In fact,
> I've had to reduce the parallalism of make or the whole process crashes
> with a resource allocation error.
Which program reports the error?
> But Process Manager reports that the VM has memory left.
What does Task Manager say about memory consumption of the various
programs involved in the build, like Bash, Make, and Python? Do you
see their memory growing to values that are unreasonably large?
The slow-down could be caused by memory consumption, so it's possible
the actual problem is with memory, not with slow builds.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: msys2 building issue
2018-03-29 12:22 ` Eli Zaretskii
@ 2018-04-10 12:00 ` Phillip Lord
2018-04-10 15:17 ` Eli Zaretskii
0 siblings, 1 reply; 5+ messages in thread
From: Phillip Lord @ 2018-04-10 12:00 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: emacs-devel
Eli Zaretskii <eliz@gnu.org> writes:
>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Date: Tue, 27 Mar 2018 22:20:34 +0100
>>
>> x86_64
>> no-deps zip 17min
>> Deps-zip 3min
>> installer 25min
>>
>> i686
>> no deps zip 31min
>> deps zip 47min
>> installer 61min
>>
>> So, the installer is really slow to build (probably because of the
>> compression I guess). But the second i686 step is really slow,
>> especially the building the zips and installer.
>
> Does this happen only when building both the 64-bit and the 32-bit
> zips? What if you build only the 32-bit one -- do you get the same
> slow times?
>
> My next suspect is the way MSYS2 lets you build 32-bit binaries. I
> know nothing about that, all I see is that you set 2 environment
> variables, but could that cause Bash or Make or some other component
> to run out of memory due to a bug or a memory leak?
I've tested this out since. And I am having a hard time making head nor
tail of it. Building on it's own, the i686 will build fast
enough. Building x86_64 sometimes give very inconsistent behaviour
between the two builds (with the second slower). I could, of course,
build the x86 and i686 separately, but I want to launch the build once
and let it finish.
I've had occasions when the build time goes up to many hours.
>> I don't think this is the Emacs build per se; it looks to me like a
>> memory leak; the build process just gets slower and slower. In fact,
>> I've had to reduce the parallalism of make or the whole process crashes
>> with a resource allocation error.
>
> Which program reports the error?
>
>> But Process Manager reports that the VM has memory left.
>
> What does Task Manager say about memory consumption of the various
> programs involved in the build, like Bash, Make, and Python? Do you
> see their memory growing to values that are unreasonably large?
>
> The slow-down could be caused by memory consumption, so it's possible
> the actual problem is with memory, not with slow builds.
Actually, no Task Manager reports that CPU is 100%, memory is between 50
and 70%.
I'm coming to the conclusion that it's an oddity of the VM
infrastructure. I think it's going to be hard to test out further,
because the slow build makes experiments a bit of a PITA.
Never mind, even if it is slow it always seems to build now that I am
using only dual threaded build. I guess that is ultimately all I care
about.
Phil
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-04-10 20:51 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-03-27 21:20 msys2 building issue Phillip Lord
2018-03-29 12:22 ` Eli Zaretskii
2018-04-10 12:00 ` Phillip Lord
2018-04-10 15:17 ` Eli Zaretskii
[not found] ` <873702sxly.fsf@russet.org.uk>
[not found] ` <83muya3n6s.fsf@gnu.org>
2018-04-10 20:51 ` Phillip Lord
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.