Building Guile with ‘-j1’?

Building Guile with ‘-j1’?

[Ludovic Courtès]

Hi!

As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
are no exception since Guile builds are non-reproducible, despite work
done a few years ago:

  https://issues.guix.gnu.org/20272

Until it’s fixed in Guile proper, what do you think of building Guile
2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
‘core-updates’ now.

That would work around the problem for Guile itself.  It would increase
build times, but probably not that much since the most expensive part
(compiling the first few files) is sequential anyway.  IIRC this is what
Vagrant did for the Debian packages.

We could also disable parallel builds in ‘guile-build-system’.  It’s
only used for small packages so the extra build time is probably OK.

Thoughts?

Ludo’.

Re: Building Guile with ‘-j1’?

[zimoun]

Hi Ludo,

On Wed, 20 Jan 2021 at 09:41, Ludovic Courtès <ludo@gnu.org> wrote:

> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.

[...]

> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.

Good idea.


One question: will “guix pull” be slowed?  (if it makes sense)


Cheers,
simon

Re: Building Guile with ‘-j1’?

[Gábor Boskovits]

Hello,

Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2021. jan. 20., Sze, 9:42):
>
> Hi!
>
> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272
>
> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.
>
> That would work around the problem for Guile itself.  It would increase
> build times, but probably not that much since the most expensive part
> (compiling the first few files) is sequential anyway.  IIRC this is what
> Vagrant did for the Debian packages.
>
> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.
>
> Thoughts?

Let's go for it.

>
> Ludo’.
>

Regards,
g_bor
-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

Re: Building Guile with ‘-j1’?

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello,

> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272

Ugh.  Thanks for bringing this up again.  Would you consider disabling
parallel builds in Guile by default, adding something like a
"--enable-somewhat-faster-non-reproducible-build" configure option until
this is fixed?

> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.
>
> That would work around the problem for Guile itself.  It would increase
> build times, but probably not that much since the most expensive part
> (compiling the first few files) is sequential anyway.  IIRC this is what
> Vagrant did for the Debian packages.

Yes, I would support at least doing this.  

> We could also disable parallel builds in ‘guile-build-system’.  It’s
> only used for small packages so the extra build time is probably OK.

Are packages using guile-build-system expected to build non-reproducible
too?  In that case, I would certainly support doing it there.

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Building Guile with ‘-j1’?

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
> are no exception since Guile builds are non-reproducible, despite work
> done a few years ago:
>
>   https://issues.guix.gnu.org/20272
>
> Until it’s fixed in Guile proper, what do you think of building Guile
> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
> ‘core-updates’ now.

Sounds good to me.  Given the nature of Guile's module system, it's
still not clear to me how to ensure reproducible builds of Guile modules
unless the order of compilation is deterministic.  I haven't thought
about it in a few years, though.

       Mark

Re: Building Guile with ‘-j1’?

[Maxim Cournoyer]

Hi,

Mark H Weaver <mhw@netris.org> writes:

> Hi Ludovic,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
>> are no exception since Guile builds are non-reproducible, despite work
>> done a few years ago:
>>
>>   https://issues.guix.gnu.org/20272
>>
>> Until it’s fixed in Guile proper, what do you think of building Guile
>> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
>> ‘core-updates’ now.

I'm not too found of it.  It'll make the already slow Guile build much
slower, making the lower strata of core-updates packages more painfully
slow to build and test.  It'll also enable us to overlook the issue for
years to come (similarly to the fact that the testsuite of the Guix
package itself hasn't been run in parallel for the last 6 years or so
:-)).

It also won't fix the issue of Guix modules compiled in parallel
suffering from the same problem.

So I'd keep it as it is for now, as a reminder that this is a serious
problem in need of a fix.

Thanks,

Maxim

Re: Building Guile with ‘-j1’?

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Hi Ludovic,
>>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
>>> are no exception since Guile builds are non-reproducible, despite work
>>> done a few years ago:
>>>
>>>   https://issues.guix.gnu.org/20272
>>>
>>> Until it’s fixed in Guile proper, what do you think of building Guile
>>> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
>>> ‘core-updates’ now.

I pushed that as 2ea52f90141974fb5d88b8c6b1785817c4203da4.

> I'm not too found of it.  It'll make the already slow Guile build much
> slower, making the lower strata of core-updates packages more painfully
> slow to build and test.

Yes, it’s suboptimal.  OTOH, as I wrote, I think the slowdown is not as
important as one might think: a large fraction of the Guile build time
goes into building ice-9/eval.scm and the first few files, which is
already sequential (enforced by Guile’s makefile).

> It'll also enable us to overlook the issue for years to come
> (similarly to the fact that the testsuite of the Guix package itself
> hasn't been run in parallel for the last 6 years or so :-)).

It remains and bug to fix in Guile, no doubt.  But we also have to be
pragmatic IMO.

> It also won't fix the issue of Guix modules compiled in parallel
> suffering from the same problem.

Definitely.

> So I'd keep it as it is for now, as a reminder that this is a serious
> problem in need of a fix.

There’s also the problem that a fix in Guile takes time to deploy
(basically a ‘core-updates’ cycle).  It’s not the first time we work
around a Guile bug until it’s fixed in Guile proper.

Thanks,
Ludo’.

Re: Building Guile with ‘-j1’?

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> Hi Ludovic,
>>>
>>> Ludovic Courtès <ludo@gnu.org> writes:
>>>
>>>> As the saying goes, “the cobbler’s children go barefoot”.  Guile/Guix
>>>> are no exception since Guile builds are non-reproducible, despite work
>>>> done a few years ago:
>>>>
>>>>   https://issues.guix.gnu.org/20272
>>>>
>>>> Until it’s fixed in Guile proper, what do you think of building Guile
>>>> 2.0/2.2/3.0 with #:parallel-build? #f ?  We could do that in
>>>> ‘core-updates’ now.
>
> I pushed that as 2ea52f90141974fb5d88b8c6b1785817c4203da4.
>
>> I'm not too found of it.  It'll make the already slow Guile build much
>> slower, making the lower strata of core-updates packages more painfully
>> slow to build and test.
>
> Yes, it’s suboptimal.  OTOH, as I wrote, I think the slowdown is not as
> important as one might think: a large fraction of the Guile build time
> goes into building ice-9/eval.scm and the first few files, which is
> already sequential (enforced by Guile’s makefile).

I've built Guile many times on core-updates since and it seems to be
true, that the build time didn't change noticeably.

>> It'll also enable us to overlook the issue for years to come
>> (similarly to the fact that the testsuite of the Guix package itself
>> hasn't been run in parallel for the last 6 years or so :-)).
>
> It remains and bug to fix in Guile, no doubt.  But we also have to be
> pragmatic IMO.

I had the feeling this partial fix would cause build the build time to
explode, but given it didn't, I'm happy with it.

Thanks,

Maxim