all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#46650: Extending package transformation options
@ 2021-02-19 23:24 Carl Dong
  2021-02-20  0:17 ` Leo Famulari
  2021-02-20 13:57 ` Ludovic Courtès
  0 siblings, 2 replies; 9+ messages in thread
From: Carl Dong @ 2021-02-19 23:24 UTC (permalink / raw)
  To: 46650

Hi Guix!

TL;DR Shall we add a `--without-tests` option (or something similar) for `guix time-machine`?

Our release workflow in Bitcoin is planned to shift over to Guix for the next release, and as such we have a lot more testers who are running Guix as we speak on their machines! Exciting!

Our build workflow basically boils down to something like:

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=<pinned-guix> … \
       -- environment --manifest=manifest.scm … \
          -- <build-command>
--8<---------------cut here---------------end--------------->8---

The fact that this is even possible (and allows reproducibility across time) speaks to the awesomeness of Guix! :-)

One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:
1. Spurious tests
2. “Expiring” tests (like the gnutls one)
3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)

I think that the package transformation options are the perfect “hack-around” solution for this, namely `--without-tests`. In fact, since those who experience these problems are building from source, it makes no difference that the store item would be change: they would need to build everything anyway!

However, in the case of our workflow above, there is no way to specify `--without-tests=` to `guix time-machine`, which insists on building the broken `gnutls` package and running its tests. I’ve tried isolating the command to just:

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=<pinned-guix> … \
       -- --help
--8<---------------cut here---------------end--------------->8—

To make sure that it’s `guix time-machine` which is trying to build gnutls and not the time-machine’d `guix environment`.

I’m not too familiar with how package transformation options work internally, but I’m guessing all we need here is a `--without-tests` option (or something similar) for `guix time-machine`? Is there anything else that’s necessary to successfully skip tests for my workflow?

Cheers,
Carl Dong
contact@carldong.me
"I fight for the users"





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46650: Extending package transformation options
  2021-02-19 23:24 bug#46650: Extending package transformation options Carl Dong
@ 2021-02-20  0:17 ` Leo Famulari
  2021-02-20 14:00   ` Ludovic Courtès
  2021-02-20 13:57 ` Ludovic Courtès
  1 sibling, 1 reply; 9+ messages in thread
From: Leo Famulari @ 2021-02-20  0:17 UTC (permalink / raw)
  To: Carl Dong; +Cc: 46650

On Fri, Feb 19, 2021 at 06:24:57PM -0500, Carl Dong wrote:
> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:
> 1. Spurious tests
> 2. “Expiring” tests (like the gnutls one)
> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)

I hate to throw a wrench in your plans, but I think that using package
transformation options on the command-line militates against many of the
benefits you are seeking to take advantage of, in terms of an
introspectable and bootstrappable software system.

Normally when using Guix, our Git tree provides a convenient and
comprehensive view of the entire software system. It also provides an
entry point into the system for making changes.

When using these command-line package transformations, you diverge from
the Guix Git tree and enter uncharted and untracked territory. If you
install some software or build some system using these transformations,
there is no Git tree you can check out and begin exploring. For example,
there will be no package definition of GnuTLS-without-tests that can be
referred to and shared.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46650: Extending package transformation options
  2021-02-19 23:24 bug#46650: Extending package transformation options Carl Dong
  2021-02-20  0:17 ` Leo Famulari
@ 2021-02-20 13:57 ` Ludovic Courtès
  2021-02-23 22:22   ` zimoun
  1 sibling, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-20 13:57 UTC (permalink / raw)
  To: Carl Dong; +Cc: 46650

Hi,

Carl Dong <contact@carldong.me> skribis:

> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:
> 1. Spurious tests
> 2. “Expiring” tests (like the gnutls one)
> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)

Yes, these flaky/uncontrolled problems were one of the motivations for
‘--without-tests’.

> However, in the case of our workflow above, there is no way to specify `--without-tests=` to `guix time-machine`, which insists on building the broken `gnutls` package and running its tests. I’ve tried isolating the command to just:
>
> --8<---------------cut here---------------start------------->8---
> $ guix time-machine --commit=<pinned-guix> … \
>        -- --help
> --8<---------------cut here---------------end--------------->8—
>
> To make sure that it’s `guix time-machine` which is trying to build gnutls and not the time-machine’d `guix environment`.

Package transformation options currently only apply to packages
specified on the command line of ‘guix install’, ‘guix build’, etc.
These options are not available to ‘guix system’, ‘guix pull’, and other
non-package-oriented commands.

Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on
GnuTLS.  So here, we have a special case: to build Guix itself, you need
to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at
that level.

I’m not sure how to change that because unfortunately, ‘guix pull’ and
‘time-machine’ don’t even know that Guix depends on GnuTLS; only the
target Guix knows about this.  :-/

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46650: Extending package transformation options
  2021-02-20  0:17 ` Leo Famulari
@ 2021-02-20 14:00   ` Ludovic Courtès
  0 siblings, 0 replies; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-20 14:00 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 46650, Carl Dong

Leo Famulari <leo@famulari.name> skribis:

> On Fri, Feb 19, 2021 at 06:24:57PM -0500, Carl Dong wrote:
>> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:
>> 1. Spurious tests
>> 2. “Expiring” tests (like the gnutls one)
>> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)
>
> I hate to throw a wrench in your plans, but I think that using package
> transformation options on the command-line militates against many of the
> benefits you are seeking to take advantage of, in terms of an
> introspectable and bootstrappable software system.
>
> Normally when using Guix, our Git tree provides a convenient and
> comprehensive view of the entire software system. It also provides an
> entry point into the system for making changes.
>
> When using these command-line package transformations, you diverge from
> the Guix Git tree and enter uncharted and untracked territory. If you
> install some software or build some system using these transformations,
> there is no Git tree you can check out and begin exploring. For example,
> there will be no package definition of GnuTLS-without-tests that can be
> referred to and shared.

I think that’s OK: you’re building a different GnuTLS derivation, true,
but how that derivation is computed is fully under control as the
transformation option appears on the command line or in your manifest.

It’s really equivalent to forking Guix and adding #:tests? #f, only more
flexible.

Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46650: Extending package transformation options
  2021-02-20 13:57 ` Ludovic Courtès
@ 2021-02-23 22:22   ` zimoun
  2021-02-24 11:11     ` Ludovic Courtès
  0 siblings, 1 reply; 9+ messages in thread
From: zimoun @ 2021-02-23 22:22 UTC (permalink / raw)
  To: Ludovic Courtès, Carl Dong; +Cc: 46650

Hi,

On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:

> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on
> GnuTLS.  So here, we have a special case: to build Guix itself, you need
> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at
> that level.
>
> I’m not sure how to change that because unfortunately, ‘guix pull’ and
> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the
> target Guix knows about this.  :-/

If  «“guix pull” and ’time-machine’ don’t even know that Guix depends on
GnuTLS» and «to build Guix itself, you need to build GnuTLS first», how
does it work?  How does «you» know that they needs to build GnuTLS
first?

Well, I do not know if it is viable, one hack should to use the package
Guix to collect this missing knowledge.  Why is it not possible to build
all the package dependencies of Guix with the option --without-tests?


Cheers,
simon




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46650: Extending package transformation options
  2021-02-23 22:22   ` zimoun
@ 2021-02-24 11:11     ` Ludovic Courtès
  2021-02-24 13:43       ` zimoun
  0 siblings, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-24 11:11 UTC (permalink / raw)
  To: zimoun; +Cc: 46650, Carl Dong

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on
>> GnuTLS.  So here, we have a special case: to build Guix itself, you need
>> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at
>> that level.
>>
>> I’m not sure how to change that because unfortunately, ‘guix pull’ and
>> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the
>> target Guix knows about this.  :-/
>
> If  «“guix pull” and ’time-machine’ don’t even know that Guix depends on
> GnuTLS» and «to build Guix itself, you need to build GnuTLS first», how
> does it work?  How does «you» know that they needs to build GnuTLS
> first?

‘guix pull’ builds Guix using the method implemented in (guix self).
The entry point there is ‘compiled-guix’, which returns a compiled Guix.

It’s not a package so the package API doesn’t apply.

HTH,
Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46650: Extending package transformation options
  2021-02-24 11:11     ` Ludovic Courtès
@ 2021-02-24 13:43       ` zimoun
  2021-02-24 17:21         ` Ludovic Courtès
  0 siblings, 1 reply; 9+ messages in thread
From: zimoun @ 2021-02-24 13:43 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 46650, Carl Dong

Hi Ludo,

On Wed, 24 Feb 2021 at 12:11, Ludovic Courtès <ludo@gnu.org> wrote:
> > On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:
> >
> >> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on
> >> GnuTLS.  So here, we have a special case: to build Guix itself, you need
> >> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at
> >> that level.
> >>
> >> I’m not sure how to change that because unfortunately, ‘guix pull’ and
> >> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the
> >> target Guix knows about this.  :-/
> >
> > If  «“guix pull” and ’time-machine’ don’t even know that Guix depends on
> > GnuTLS» and «to build Guix itself, you need to build GnuTLS first», how
> > does it work?  How does «you» know that they needs to build GnuTLS
> > first?
>
> ‘guix pull’ builds Guix using the method implemented in (guix self).
> The entry point there is ‘compiled-guix’, which returns a compiled Guix.

Thanks.  It perfectly answer to my question. :-)

> It’s not a package so the package API doesn’t apply.

I am confused.  The 'specification->package' returns a regular
package, right?.  And then,

--8<---------------cut here---------------start------------->8---
  (define gnutls
    (specification->package "gnutls"))

  (define dependencies
    (append-map transitive-package-dependencies
                (list guile-gcrypt gnutls guile-git guile-avahi
                      guile-json guile-semver guile-ssh guile-sqlite3
                      guile-zlib guile-lzlib guile-zstd)))
--8<---------------cut here---------------end--------------->8---

So here, a transformation could be applied.  For example, build all
the dependencies without testing them;  at the 'define gnutls' (&co)
step or in the map.  Why not?

Well, just to be able to travel back in time if some expiration changed.


Cheers,
simon




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46650: Extending package transformation options
  2021-02-24 13:43       ` zimoun
@ 2021-02-24 17:21         ` Ludovic Courtès
  2021-02-26 21:44           ` zimoun
  0 siblings, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-24 17:21 UTC (permalink / raw)
  To: zimoun; +Cc: 46650, Carl Dong

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> I am confused.  The 'specification->package' returns a regular
> package, right?.  And then,
>
>   (define gnutls
>     (specification->package "gnutls"))
>
>   (define dependencies
>     (append-map transitive-package-dependencies
>                 (list guile-gcrypt gnutls guile-git guile-avahi
>                       guile-json guile-semver guile-ssh guile-sqlite3
>                       guile-zlib guile-lzlib guile-zstd)))
>
> So here, a transformation could be applied.  For example, build all
> the dependencies without testing them;  at the 'define gnutls' (&co)
> step or in the map.  Why not?

Sure, but how do you get a user-specified transformation to this line?

Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46650: Extending package transformation options
  2021-02-24 17:21         ` Ludovic Courtès
@ 2021-02-26 21:44           ` zimoun
  0 siblings, 0 replies; 9+ messages in thread
From: zimoun @ 2021-02-26 21:44 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 46650, Carl Dong

Hi,

On Wed, 24 Feb 2021 at 18:21, Ludovic Courtès <ludo@gnu.org> wrote:

>> So here, a transformation could be applied.  For example, build all
>> the dependencies without testing them;  at the 'define gnutls' (&co)
>> step or in the map.  Why not?
>
> Sure, but how do you get a user-specified transformation to this line?

I do not know (yet?). :-)

First, we are only speaking about “guix time-machine” and second, only
about corner cases which could break in the future, as expiration.
Right?
We are not talking about general transformations applied to self (even
if it could be great).

So we could imagine something like ’set-build-from-command-line’ or
’set-build-option’ to tweak self only for the inferior.  I have not
traced (yet) on the stack of calls from “guix time-machine” to self via
the inferiors, but somehow it looks like passing arguments.

Well, be able to tweak how to build Guix via the time-machine appears to
me better than configuring a build machine ahead of time.  It is like
predicting the future and instead we should try to have a flexible
present in order to fix the past once in the unpredictable future. :-)


Cheers,
simon




^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-02-26 21:46 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-19 23:24 bug#46650: Extending package transformation options Carl Dong
2021-02-20  0:17 ` Leo Famulari
2021-02-20 14:00   ` Ludovic Courtès
2021-02-20 13:57 ` Ludovic Courtès
2021-02-23 22:22   ` zimoun
2021-02-24 11:11     ` Ludovic Courtès
2021-02-24 13:43       ` zimoun
2021-02-24 17:21         ` Ludovic Courtès
2021-02-26 21:44           ` zimoun

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.