Removing #:skip-build? from the crate importer?

Removing #:skip-build? from the crate importer?

[Liliana Marie Prikler]

Hi Maxime,

similar discussions have been had in the past, but thanks for raising
them again.  As far as I'm aware, the "consensus" – if it can be called
such – is that due to Rust packages often being a giant nothing burger
both in terms of what they actually do and in what can be used for
subsequent builds, but still generating a lot of hot air literally when
evaluated on CI, we ought to skip the build in intermediate packages.

There are a few counter-proposals.  One suggestion that has been
raised, but not yet implemented, would be to make it so that build
results can actually be reused.  This is the most promising
conceptually, but from what I can gather from those working on it might
not be that easy to pull off.  Another one I recently thought about
myself would be to demote these packages to origins, which they in fact
are.  This could fairly easily be done by simply allowing an optional
license field in origins, but we'd have to apply a special trick to get
back the recursive unpacking.  Basically, what we'd have to do is for
e.g. rust-serde, (define rust-serde (append (list rust-serde-origin)
other-origins ...) where other-origins ... are the "packages" rust-
serde pulls in, add those to inputs via append and then unpack all of
them in an "unpack-all-those-crates" phase to be added to cargo-build-
system.  I am probably missing further proposals that have been made
over time.

In any case, #:skip-build? is far from being the right thing we
typically aim for, and I support every attempt to remove it that allows
us to meet our climate goals.

Cheers

Re: Removing #:skip-build? from the crate importer?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

Liliana Marie Prikler schreef op do 31-03-2022 om 21:07 [+0200]:
> Hi Maxime,
> 
> similar discussions have been had in the past, but thanks for raising
> them again.  As far as I'm aware, the "consensus" – if it can be called
> such – is that due to Rust packages often being a giant nothing burger
> both in terms of what they actually do and in what can be used for
> subsequent builds, but still generating a lot of hot air literally when
> evaluated on CI, we ought to skip the build in intermediate packages.

I was not aware that it has been discussed in the past ,thanks.

> There are a few counter-proposals.  One suggestion that has been
> raised, but not yet implemented, would be to make it so that build
> results can actually be reused.  This is the most promising
> conceptually, but from what I can gather from those working on it might
> not be that easy to pull off.

Yes, that would be nice.  Are there already some concrete patches that
can be tested?  I know there are some (ideas for?) patches to move
development-inputs->native-inputs but I don't think I've seen any
patches for actually reusing build results yet ...

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Removing #:skip-build? from the crate importer?

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

Am 31.03.22 um 21:14 schrieb Maxime Devos:
>> There are a few counter-proposals.  One suggestion that has been
>> raised, but not yet implemented, would be to make it so that build
>> results can actually be reused.  This is the most promising
>> conceptually, but from what I can gather from those working on it might
>> not be that easy to pull off.
> Yes, that would be nice.

AFAIK Effraim and me tried to make cargo pick up artifacts build in 
another package. I gave somewhen: cargo did not pick up the artifacts, 
due to much „magic“ which I was unable to understand and work around.

One of the topics here are „features“ (like compile-time options), 
timestamps, exact absolute pathname, etc. All of this seems to go into 
some hash, which will determine which files will be rebuild. If anybody 
is interested, I could share the code of my last try. (Maybe the road to 
go would be to make cargo less strict here. But this requires 
understanding rust code and cargo.)


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          |h.goebel@crazy-compilers.com                |
|www.crazy-compilers.com  | compilers which you thought are impossible |

[-- Attachment #2: Type: text/html, Size: 1929 bytes --]

Re: Removing #:skip-build? from the crate importer?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

Hartmut Goebel schreef op za 02-04-2022 om 11:45 [+0200]:
> [checksum things]
> (Maybe the road to go would be to make cargo less strict here. But
> this requires understanding rust code and cargo.)

FWIW, in
<https://notabug.org/maximed/cargoless-rust-experiments/src/master/antioxidant.scm#L31>
and
<https://notabug.org/maximed/cargoless-rust-experiments/src/master/antioxidant.scm#L97>,
the Cargo.toml is parsed.
By using 'toml.dump' from python-toml, it should be feasible to
automatically strip out all mentions of optional dependencies.  That's
another form of ‘less strict’ though.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Removing #:skip-build? from the crate importer?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

Hi guix,

Often, when new rust package definitions are submitted at guix-
patches@, I see #:skip-build? #false.  Apparently it's added by default
in (guix import cargo), with some exceptions.  However, ‘(guix)Rust
Crates’ states:

   Care should be taken to ensure the correct version of dependencies
are used; to this end we try to refrain from skipping the tests or
using ‘#:skip-build?’ when possible. Of course this is not always
possible [...]

so I respond by asking to remove #:skip-build?.

As such, WDYT of removing #:skip-build? #false from (guix import
crate)?  FWIW, this was added in commit
269c1db41bd82f93c7ae5c62a4969a423e556183 to (guix import crate)?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Removing #:skip-build? from the crate importer?

[Hartmut Goebel]

Hi,

since rust does not support anything like static or dynamic libraries, 
building (intermediate) crates is useless like a hole in my head. Any 
output on any intermediate crate will just be thrown away.

> Often, when new rust package definitions are submitted at guix-
> patches@, I see #:skip-build? #false.  Apparently it's added by default
> in (guix import cargo), with some exceptions.
The idea behind is to have #:skip-buiild #f for all "top level" crates, 
which are assumed to be programs. Thus, only crates imported recursively 
will get get #:skip-buiild #t. If one imports a single crate, it well 
get #:skip-buiild #f — which is what you experience.

>    However, ‘(guix)Rust
> Crates’ states:
>
>     Care should be taken to ensure the correct version of dependencies
> are used; to this end we try to refrain from skipping the tests or
> using ‘#:skip-build?’ when possible. Of course this is not always
> possible [...]

This text is from 2020-02-17 (written by Effraim) and predates 
269c1db41bd8 (committed 2020-12-02).

While I understand the intention of this, I'm not convinces about it. 
Primary this will lead to a huge wast of time and electrical power - 
just to trash the results. This will not only effect our own build farm, 
but also each user.

Please be aware, that with #:skip-buiild #t, every crate will be build 
again by every other crate using it. So if crate AA is used by B1 and B2 
and C1 depends on B1 and B2, AA will be build 4 times!


> As such, WDYT of removing #:skip-build? #false from (guix import
> crate)?  FWIW, this was added in commit

I would propose the opposite: Keep it and make #t the default.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |