Packaging rust crates without compiling dependencies again for every dependent (wasting build time)

Packaging rust crates without compiling dependencies again for every dependent (wasting build time)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1947 bytes --]

Hi various distributions packaging rust things (^),

Currently, most (all?) distributions seem to package ‘source-only
crates’, which when ‘compiled’, just copy some source code to a
location.  This works, but the downside is that if both a package X and
a package Y depends on rust-Z, then rust-Z will effectively be compiled
twice: when building X, but also when building Y.

This seems a waste of compilation time to me, so I've been writing a
build tool ‘antioxidant’ (*) for the distribution Guix that avoids
double compilation, at
<https://notabug.org/maximed/cargoless-rust-experiments>).

To avoid the double compilation things, antioxidant calls the
underlying rustc by itself instead of cargo, and it reads and
interprets Cargo.toml by itself instead of letting cargo do that.

Currently, it's rather Guix-specific, but in principle it should be
possible to turn it in some binary 'antioxidant' that could be used on
other distributions as well if someone is interested in that, e.g.
something like

$ # (not implemented yet!)
$ antioxidant configure --prefix=$HOME/extra-software -Lcrate=/usr/lib/crates -L/usr/lib -Lcrate=$HOME/extra-software/lib/crates --features=x,y,z
$ antioxidant make
$ antioxidant make check
$ antioxidant make install DESTDIR=... # this does not install source code, only the compiled crates

(*) A ‘build system’, in Guix terminology.

Currently, it can succesfully compile 'hexyl', 'sniffglue' and all
their dependencies.  I'm working on addressing limitations that
prevent antioxidant from compiling other binaries (currently trying
to make 'agate' and all its dependencies compile).

Greetings,
Maxime.

(^) If you are reading this from guix-devel@: some of the distributions I
intend to contact have a no-crossposting policy, so I'm first sending this to
guix-devel@, then sending some mails to individual distributions with a link
to this e-mail.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Packaging rust crates without compiling dependencies again for every dependent (wasting build time)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2814 bytes --]

On Sat, Apr 30, 2022 at 11:29:25AM +0200, Maxime Devos wrote:
> Hi various distributions packaging rust things (^),
> 
> Currently, most (all?) distributions seem to package ‘source-only
> crates’, which when ‘compiled’, just copy some source code to a
> location.  This works, but the downside is that if both a package X and
> a package Y depends on rust-Z, then rust-Z will effectively be compiled
> twice: when building X, but also when building Y.
> 
> This seems a waste of compilation time to me, so I've been writing a
> build tool ‘antioxidant’ (*) for the distribution Guix that avoids
> double compilation, at
> <https://notabug.org/maximed/cargoless-rust-experiments>).
> 
> To avoid the double compilation things, antioxidant calls the
> underlying rustc by itself instead of cargo, and it reads and
> interprets Cargo.toml by itself instead of letting cargo do that.
> 
> Currently, it's rather Guix-specific, but in principle it should be
> possible to turn it in some binary 'antioxidant' that could be used on
> other distributions as well if someone is interested in that, e.g.
> something like
> 
> $ # (not implemented yet!)
> $ antioxidant configure --prefix=$HOME/extra-software -Lcrate=/usr/lib/crates -L/usr/lib -Lcrate=$HOME/extra-software/lib/crates --features=x,y,z
> $ antioxidant make
> $ antioxidant make check
> $ antioxidant make install DESTDIR=... # this does not install source code, only the compiled crates
> 
> (*) A ‘build system’, in Guix terminology.
> 
> Currently, it can succesfully compile 'hexyl', 'sniffglue' and all
> their dependencies.  I'm working on addressing limitations that
> prevent antioxidant from compiling other binaries (currently trying
> to make 'agate' and all its dependencies compile).
> 
> Greetings,
> Maxime.
> 
> (^) If you are reading this from guix-devel@: some of the distributions I
> intend to contact have a no-crossposting policy, so I'm first sending this to
> guix-devel@, then sending some mails to individual distributions with a link
> to this e-mail.

This is really really cool! Thanks for sharing it.

I noticed that there are a number of crates that you remove
unconditionally, would it help if we patched them out of the crates that
are in Guix?

I'm glad you found a use for (target-64bit?), it seemed too useful to
not have in Guix :)

It took me less than 3 minutes to download and build all the sources for
the three packages! It looks to be faster than building the packages
straight, and with the benefit of incremental compilation.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging rust crates without compiling dependencies again for every dependent (wasting build time)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

Efraim Flashner schreef op ma 09-05-2022 om 13:02 [+0300]:
> This is really really cool! Thanks for sharing it.
> 
> I noticed that there are a number of crates that you remove
> unconditionally, would it help if we patched them out of the crates
> that
> are in Guix?

This is for rust-winapi and the like?  Unfortunately IIUC, that's
currently not yet possible in general, because Cargo requires optional
dependencies to be available, see
<https://github.com/rust-lang/cargo/issues/4544>.  Unless the
Cargo.toml are patched appropriately as well.

(I don't know if this applies to dev-dependencies as well, or only to
'dependencies' ...)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]