Building hexyl (a rust app) without cargo, with antioxidant-build-system

Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Hi,

antioxidant-build-system can now be used for some ‘real’ software -- it
compiles 'hexyl'.  To test, download
<https://notabug.org/maximed/cargoless-rust-experiments> (commit:
d09fd93750ac6d77e0c85623286b45cf5c3b055b) and run
"guix build -L . -f guix.scm" and then

$ cat guix.scm | /gnu/store/[...]-hexyl-0.8.0/bin/hexyl
> lots of coloured hex output

Some features of antioxidant-build-system:

  * no copying source code of dependencies
  * no compiling dependencies again -- old artifacts are reused
  * all dependencies use the usual package input system
    (native-inputs, inputs, propagated-inputs)

Limitations:

  * no support for linking to arbitrary shared libraries yet
    (only rust deps)
  * makes a few assumptions on the source layout (can be fixed
    by using more info from Cargo.toml)
  * no tests
  * no cross-compilation yet
  * no shared libraries (just replacing 'rlib' by 'dylib' causes problems)
  * code is a bit messy
  * no cdylib yet (probably needed for librsvg)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Brendan Tildesley]

[-- Attachment #1: Type: text/plain, Size: 180 bytes --]

I would have called it car/gone./

Do you believe sidestepping cargo all together like this is a good long 
term strategy?
In particular that importing packages will still be easy.

[-- Attachment #2: Type: text/html, Size: 347 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]

Brendan Tildesley schreef op ma 04-04-2022 om 15:10 [+1000]:
>  I would have called it cargone.
>  
>  Do you believe sidestepping cargo all together like this is a good
> long term strategy?  In particular that importing packages will still
> be easy. 

The package definitions look almost the same, except for:

  * #:cargo-inputs and #:cargo-development-inputs are moved to
    'inputs'/'propagated-inputs' and 'native-inputs'
  * For a few packages, #:features ~'(...) needs to be added
  * antioxidant-build-system doesn't look at version numbers,
    so the numbers of variants of packages that needs to be defined
    could perhaps be reduced
  * Cycles are not supported

The fourth point might make things difficult though I have some ideas
on how to resolve it (without simply disabling tests)  (TBI!). 
Otherwise, I don't see much complications with importing packages.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2291 bytes --]

Maxime Devos schreef op za 02-04-2022 om 17:18 [+0200]:
>   * makes a few assumptions on the source layout (can be fixed
>     by using more info from Cargo.toml)

This has now been partially resolved.  It now looks in Cargo.toml to
determine where Rusts's equivalent of 'configure' (build.rs) is
located, and the Cargo.toml can now override the default src/lib.rs.

Some other improvements that weren't announced previously:

  * The code now automatically detects cycles and reports which
    packages are involved in the cycle.
  * --edition (cf. -std=c11/c99/...) is set appropriately, depending on
    Cargo.toml, fixing some build failures.
  * All default 'features' (in Cargo.toml) are now enabled by default.
  * Package definitions can request non-default features to be built
    anyway.

    A difference from cargo-build-system: features are set in the
    package of the rust crate, not the package using the rust crate.
  * build.rs is now run, which reduces the number of features that need
    to be set manually.
  * proc-macro crates are now named .so instead of .rlib, apparently
    naming them .rlib even though they aren't .rlibs confuses rustc
    or something.
  * #[deny(warnings)] (rust's equivalent of -Werrors) is now ignored
    (FWIW this is done by cargo as well).
  * Conversion from cargo-build-system to antioxidate-build-system is
    now automated by a recursive 'vitaminate/auto' procedure and some
    ad-hoc cycle breaking.

Some things that could perhaps be moved to (upstream) Guix:

  * Sometimes test and build.rs dependencies are listed in
    #:cargo-inputs and #:cargo-development-inputs (e.g.
    rust-quickcheck, rust-cc).  I deleted the test dependencies
    (because they contribute to cycles and for now running tests
    isn't supported anyways).
   
    Maybe they can be moved to #:cargo-development-inputs upstream
    (untested)?  Extending 'check-inputs-should-be-native' would also
    be nice too ...

  * There's a cycle between rust-backtrace-sys and rust-cc can be
    eliminated by deleting build.rs and tweaking Cargo.toml, at cost
    of losing some support for Android (which is not supported by Guix
    except in the sense that Linux is supported).

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[raingloom]

On Mon, 04 Apr 2022 11:26:45 +0200
Maxime Devos <maximedevos@telenet.be> wrote:

> Brendan Tildesley schreef op ma 04-04-2022 om 15:10 [+1000]:
> >  I would have called it cargone.
> >  
> >  Do you believe sidestepping cargo all together like this is a good
> > long term strategy?  In particular that importing packages will
> > still be easy.   
> 
> The package definitions look almost the same, except for:
> 
>   * #:cargo-inputs and #:cargo-development-inputs are moved to
>     'inputs'/'propagated-inputs' and 'native-inputs'
>   * For a few packages, #:features ~'(...) needs to be added
>   * antioxidant-build-system doesn't look at version numbers,
>     so the numbers of variants of packages that needs to be defined
>     could perhaps be reduced
>   * Cycles are not supported
> 
> The fourth point might make things difficult though I have some ideas
> on how to resolve it (without simply disabling tests)  (TBI!). 
> Otherwise, I don't see much complications with importing packages.
> 
> Greetings,
> Maxime.

I'm also a bit worried with the trend of Guix trying to duplicate
functionality already present in language package managers and config
file formats. It creates a two-sources-of-truth situation. Trying to
keep one up to date with the other can be an issue, this is why I
didn't create a custom record type for Yggdrasil config files and just
used a generic JSON converter. Which paid off, since there were in fact
changes in the config fields between versions.
Will this build system avoid that issue as well?
I guess if the data it operates on has truly stable semantics, then
writing an independent implementation is not as big a problem, since
once written and debugged, it won't need to change.
(And getting rid of cargo would be nice for Rust dev on Guix.)

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 4102 bytes --]

raingloom schreef op wo 06-04-2022 om 00:28 [+0200]:
> I'm also a bit worried with the trend of Guix trying to duplicate
> functionality already present in language package managers and config
> file formats. It creates a two-sources-of-truth situation.

For service configurations, there is only one source of truth: the
configuration record.  It just happens to be converted to another
format.  The underlying file format, TOML is not Cargo-specific
(https://toml.io/en/) and parsed with python-toml, presumably python-
toml parses it correctly.  (Whether antioxidant-build-system interprets
it correctly, is of course a different matter.)

Also, Cargo is both a package manager and a build system.  The package
manager part is useless to Guix, we even have to work-around it (see
e.g. patch-cargo-checksums, crate-closure).  Unfortunately, there does
not seem to be a way to only use the build system part of Cargo and
ignore the package manager part, so the build system part has to be
reimplemented ...

> Trying to keep one up to date with the other can be an issue, this is why I
> didn't create a custom record type for Yggdrasil config files and just
> used a generic JSON converter. Which paid off, since there were in fact
> changes in the config fields between versions.

The configuration record fields does not have to support every new
thing, it just haves to be _sufficiently_ up to date for users.  And if
a configuration field that's needed is missing, then it's almost
trivial to add it.

Additionally, some services have an 'extra-content' escape hatch. 
yggrasil-configuration could have one too, then the user could have
both write most things as Scheme (benefit: largely unified interface,
more automation possible) _and_ be able to use the latest features not
yet directly supported by the configuraiton fields, via the escape
hatch.

> Will this build system avoid that issue as well?

antioxidant-build-system is a build system, not a configuration format.

I've found that most Cargo.toml are rather tame.  They often just set:

  * name
  * Some metadata like version, authors, home-page, include, exclude
    ... that antioxidant-build-system doesn't need.
  * sometimes 'build' or 'path'
  * dependencies  / dev-dependencies / build-dependencies, also
    unneeded by antioxidant-build-system (the importer might need them,
    but that's a separate component)

Looking at <https://doc.rust-lang.org/cargo/reference/manifest.html>,
there are some more things are things that antioxidant-build-system
doesn't need:

  * resolver
  * badges, [[bench]]
  * the [patch] and [replace]  --- just use a variant package in the
    package inputs

Now, there are some things that aren't yet supported by antioxidant-
build-system but probably need to be:

  * autobins / autoexamples / autotests / [lib] / [bin] / [example]
    / [links]
  * setting various environment variables before build.rs
    (there's probably some list somewhere ...)

But aside from that, it seems that the essential parts of Cargo.toml
are rather tame and unlikely to be added to.  And if something
essential is added anyway and needs to be supported, then it should be
easy to modify antioxidant-build-system appropriately (long-term)
and/or work-around it by setting some environment variables locally or
maybe some substitute* (short-term).

> I guess if the data it operates on has truly stable semantics, then
> writing an independent implementation is not as big a problem, since
> once written and debugged, it won't need to change.
> (And getting rid of cargo would be nice for Rust dev on Guix.)

AFAICT, the parts important to Guix are sufficiently stable -- not
immutable (sometimes changes do happen).  And even if it was unstable,
it still seems much better than just using Cargo (due to much shorter
build times, less strict about versions, not having to add winapi
crates when compiling for Linux, no copying tarballs around, no
#:cargo-inputs instead of [propagated-]inputs ...).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Hartmut Goebel]

Am 05.04.22 um 18:10 schrieb Maxime Devos:
> Some other improvements that weren't announced previously: 

Wow! Impressive!

I'd be eager to tr it with sequoia-openpgp. Please drop me a note when 
you think your work might be able to build that beast.


>    * Package definitions can request non-default features to be built
>      anyway.
>
>      A difference from cargo-build-system: features are set in the
>      package of the rust crate, not the package using the rust crate.

How is this intended to work?

Package 1 has features AAA (= default) and BBB. So a .rlib is build for 
each feature (package1-AAA.rlib, package1-BBB.rlib) Or will one need to 
define two guix packages (package1+aaa and package1+bbb) and make the 
build-system build the respective feature?

I personally would prefer the former. Thus package2 would pick up the 
pre-compiled rlib for the respective feature.

Rational: If there are more features to be combined, the number of 
packages to be build can a order of square. So defining all these 
packages becomes a burden quickly. This is a computer's job :-) The 
build-system could easily build all combinations. suffixing each rlib 
with a short hash over the feature names,


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

Hartmut Goebel schreef op wo 06-04-2022 om 17:49 [+0200]:

> [...]
> >     * Package definitions can request non-default features to be built
> >       anyway.
> > 
> >       A difference from cargo-build-system: features are set in the
> >       package of the rust crate, not the package using the rust crate.
> 
> How is this intended to work?
> 
> Package 1 has features AAA (= default) and BBB. So a .rlib is build for 
> each feature (package1-AAA.rlib, package1-BBB.rlib) Or will one need to 
> define two guix packages (package1+aaa and package1+bbb) and make the 
> build-system build the respective feature?
> 
> I personally would prefer the former. Thus package2 would pick up the 
> pre-compiled rlib for the respective feature.
> 
> Rational: If there are more features to be combined, the number of 
> packages to be build can a order of square. So defining all these 
> packages becomes a burden quickly. This is a computer's job :-) The 
> build-system could easily build all combinations. suffixing each rlib 
> with a short hash over the feature names,

My third solution: no package variants, no multiple .rlib variants.
Instead, by default enable the ‘default features’ mentioned in the
Cargo.toml, and (manually) add the extra features to the package
definition of the Rust dependency if they turn out to be needed by the
dependent.

This is like adding --enable-this --enable-that, ... to ffmpeg's
#:configure-flags instead of building a variant of ffmpeg for each
dependent of ffmpeg that needs a different feature set.

So far, I have not encountered incompatible features ... Let's see if
it will become a problem.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

On Sat, Apr 02, 2022 at 05:18:55PM +0200, Maxime Devos wrote:
> Hi,
> 
> antioxidant-build-system can now be used for some ‘real’ software -- it
> compiles 'hexyl'.  To test, download
> <https://notabug.org/maximed/cargoless-rust-experiments> (commit:
> d09fd93750ac6d77e0c85623286b45cf5c3b055b) and run
> "guix build -L . -f guix.scm" and then
> 
> $ cat guix.scm | /gnu/store/[...]-hexyl-0.8.0/bin/hexyl
> > lots of coloured hex output
> 
> Some features of antioxidant-build-system:
> 
>   * no copying source code of dependencies
>   * no compiling dependencies again -- old artifacts are reused
>   * all dependencies use the usual package input system
>     (native-inputs, inputs, propagated-inputs)
> 
> Limitations:
> 
>   * no support for linking to arbitrary shared libraries yet
>     (only rust deps)
>   * makes a few assumptions on the source layout (can be fixed
>     by using more info from Cargo.toml)
>   * no tests
>   * no cross-compilation yet
>   * no shared libraries (just replacing 'rlib' by 'dylib' causes problems)
>   * code is a bit messy
>   * no cdylib yet (probably needed for librsvg)

Something that might help with that would be to also include the source
of the crate somewhere in the output. Then at the worst we could just
put the rust inputs of librsvg as regular inputs and let librsvg do its
own special build thing. We'd then still keep the build dependency tree
you've got working.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[raingloom]

On Mon, 30 May 2022 11:23:21 +0300
Efraim Flashner <efraim@flashner.co.il> wrote:

> On Sat, Apr 02, 2022 at 05:18:55PM +0200, Maxime Devos wrote:
> > Hi,
> > 
> > antioxidant-build-system can now be used for some ‘real’ software
> > -- it compiles 'hexyl'.  To test, download
> > <https://notabug.org/maximed/cargoless-rust-experiments> (commit:
> > d09fd93750ac6d77e0c85623286b45cf5c3b055b) and run
> > "guix build -L . -f guix.scm" and then
> > 
> > $ cat guix.scm | /gnu/store/[...]-hexyl-0.8.0/bin/hexyl  
> > > lots of coloured hex output  
> > 
> > Some features of antioxidant-build-system:
> > 
> >   * no copying source code of dependencies
> >   * no compiling dependencies again -- old artifacts are reused
> >   * all dependencies use the usual package input system
> >     (native-inputs, inputs, propagated-inputs)
> > 
> > Limitations:
> > 
> >   * no support for linking to arbitrary shared libraries yet
> >     (only rust deps)
> >   * makes a few assumptions on the source layout (can be fixed
> >     by using more info from Cargo.toml)
> >   * no tests
> >   * no cross-compilation yet
> >   * no shared libraries (just replacing 'rlib' by 'dylib' causes
> > problems)
> >   * code is a bit messy
> >   * no cdylib yet (probably needed for librsvg)  
> 
> Something that might help with that would be to also include the
> source of the crate somewhere in the output. Then at the worst we
> could just put the rust inputs of librsvg as regular inputs and let
> librsvg do its own special build thing. We'd then still keep the
> build dependency tree you've got working.
> 
> 

Maybe put the source in an extra output, not the main "out" one.
Splitting them up later for size reduction will be much more difficult
than doing this from the get-go.

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

Efraim Flashner schreef op ma 30-05-2022 om 11:23 [+0300]:
> Something that might help with that would be to also include the
> source of the crate somewhere in the output. Then at the worst we
> could just put the rust inputs of librsvg as regular inputs and let
> librsvg do its own special build thing. We'd then still keep the
> build dependency tree you've got working.

Maybe, but that kind of defeats the point of

 * no compiling dependencies again -- old artifacts are reused

and librsvg's build system seems to use Cargo, so then we again have
the problem that Cargo wants Windows crates etc. that are useless in
Guix and its overly-pickyness in versioning, we lose the ability to let
crates be shared libraries (not yet implemented, but could be
implemented in antioxidant) ..., so I'd rather just implement cdylib
and patch the build system appropriately.

(It's build system is more complicated than ordinary crates, but the
additional Makefile.am isn't too long).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]