Removing #:skip-build? from the crate importer?

Removing #:skip-build? from the crate importer?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

Hi guix,

Often, when new rust package definitions are submitted at guix-
patches@, I see #:skip-build? #false.  Apparently it's added by default
in (guix import cargo), with some exceptions.  However, ‘(guix)Rust
Crates’ states:

   Care should be taken to ensure the correct version of dependencies
are used; to this end we try to refrain from skipping the tests or
using ‘#:skip-build?’ when possible. Of course this is not always
possible [...]

so I respond by asking to remove #:skip-build?.

As such, WDYT of removing #:skip-build? #false from (guix import
crate)?  FWIW, this was added in commit
269c1db41bd82f93c7ae5c62a4969a423e556183 to (guix import crate)?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Removing #:skip-build? from the crate importer?

[Hartmut Goebel]

Hi,

since rust does not support anything like static or dynamic libraries, 
building (intermediate) crates is useless like a hole in my head. Any 
output on any intermediate crate will just be thrown away.

> Often, when new rust package definitions are submitted at guix-
> patches@, I see #:skip-build? #false.  Apparently it's added by default
> in (guix import cargo), with some exceptions.
The idea behind is to have #:skip-buiild #f for all "top level" crates, 
which are assumed to be programs. Thus, only crates imported recursively 
will get get #:skip-buiild #t. If one imports a single crate, it well 
get #:skip-buiild #f — which is what you experience.

>    However, ‘(guix)Rust
> Crates’ states:
>
>     Care should be taken to ensure the correct version of dependencies
> are used; to this end we try to refrain from skipping the tests or
> using ‘#:skip-build?’ when possible. Of course this is not always
> possible [...]

This text is from 2020-02-17 (written by Effraim) and predates 
269c1db41bd8 (committed 2020-12-02).

While I understand the intention of this, I'm not convinces about it. 
Primary this will lead to a huge wast of time and electrical power - 
just to trash the results. This will not only effect our own build farm, 
but also each user.

Please be aware, that with #:skip-buiild #t, every crate will be build 
again by every other crate using it. So if crate AA is used by B1 and B2 
and C1 depends on B1 and B2, AA will be build 4 times!


> As such, WDYT of removing #:skip-build? #false from (guix import
> crate)?  FWIW, this was added in commit

I would propose the opposite: Keep it and make #t the default.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Compiling rust things without cargo (super WIP POC)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1671 bytes --]

Hartmut Goebel schreef op do 31-03-2022 om 21:47 [+0200]:
since rust does not support anything like static or dynamic libraries, 
building (intermediate) crates is useless like a hole in my head. Any 
output on any intermediate crate will just be thrown away.

In my experiments, it looks like the rust compiler actually _does_
support static libraries, though perhaps cargo doesn't.

I invite you to take a look at <https://notabug.org/maximed/cargoless-rust-experiments>.
It contains a minimal rust library (libhello) and a minimal 'hello
world'-style application that uses 'libhello'.

To simulate how Guix compiles C software (but here applied to Rust), 
the Makefile does the following:

  * Run 'rustc --crate-type=lib libhello/hello.rs -o out/libhello/lib/libhello.rlib',
    to compile the library and install it in 'out/libhello/lib/libhello.rlib'
    (cf. /gnu/store/...-libhello.../lib/hello.so).

  * Run 'rustc -Lout/libhello/lib hello-app/main.rs -o out/hello-oxygen/bin/hello'
    to compile the application.  By default, rustc will fail because it cannot find
    the library.  However, if -Lout/libhello/lib is passed, then it does find it!

    (cf. LIBRARY_PATH=/gnu/store/.../lib & gcc -L/gnu/store/.../lib)

This is a rather basic example (no transitive dependencies, no test dependencies,
no macros ...), but it seems like there are some possibilities here ...

As a next step, maybe I could try writing a Guix package definition for libhello
and hello-oxygen, gradually making things more complicated (macros, transitive
dependencies, some non-toy Rust dependencies, a Guix build system ...)?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Compiling rust things without cargo (super WIP POC)

[Brendan Tildesley]

Recently I did the bevy (game engine) hello world tutorial[1].
To facilitate less slow iteration time when developing, it supports 
dynamic linking,
disabling some optimizations, and using the lld linker for fast linking, 
although using lld failed for me.

When running cargo build, only your project is recompiled quickly and 
linked.

I was wondering how this dynamic linking might be facilitated in Guix?


Note: I had to update cargo to 0.60 to compile bevy with edition2021

[1] https://bevyengine.org/learn/book/getting-started/

Re: Compiling rust things without cargo (super WIP POC)

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> I invite you to take a look at <https://notabug.org/maximed/cargoless-rust-experiments>.
> It contains a minimal rust library (libhello) and a minimal 'hello
> world'-style application that uses 'libhello'.
>
> To simulate how Guix compiles C software (but here applied to Rust), 
> the Makefile does the following:
>
>   * Run 'rustc --crate-type=lib libhello/hello.rs -o out/libhello/lib/libhello.rlib',
>     to compile the library and install it in 'out/libhello/lib/libhello.rlib'
>     (cf. /gnu/store/...-libhello.../lib/hello.so).
>
>   * Run 'rustc -Lout/libhello/lib hello-app/main.rs -o out/hello-oxygen/bin/hello'
>     to compile the application.  By default, rustc will fail because it cannot find
>     the library.  However, if -Lout/libhello/lib is passed, then it does find it!
>
>     (cf. LIBRARY_PATH=/gnu/store/.../lib & gcc -L/gnu/store/.../lib)
>
> This is a rather basic example (no transitive dependencies, no test dependencies,
> no macros ...), but it seems like there are some possibilities here ...

Interesting.

> As a next step, maybe I could try writing a Guix package definition for libhello
> and hello-oxygen, gradually making things more complicated (macros, transitive
> dependencies, some non-toy Rust dependencies, a Guix build system ...)?

I guess the whole question is whether that technique can be made to work
for the vast majority of Rust packages.  I’d suggest working in that
direction: writing a build system as a first step, using it in all the
Rust packages, and analyzing the kinds of problems encountered, with the
goal of estimating the effort it would take to make it work for every
single package.  Easier said than done, I guess.

Overall, I’m afraid Rust packaging is getting out of hands and we’re all
looking elsewhere.  For example, that Rust packages live in their own
separate world means there’s no tooling available, leading to poor QA, a
proliferation of versions of the same packages that never get removed,
and so on.  I think addressing that, for instance with something as I
proposed in <https://issues.guix.gnu.org/53127>, should be high
priority.

Thoughts?

Ludo’.

Re: Compiling rust things without cargo (super WIP POC)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 709 bytes --]

Ludovic Courtès schreef op vr 01-04-2022 om 11:10 [+0200]:
> Overall, I’m afraid Rust packaging is getting out of hands and we’re all
> looking elsewhere.  For example, that Rust packages live in their own
> separate world means there’s no tooling available, leading to poor QA, a
> proliferation of versions of the same packages that never get removed,
> and so on.  I think addressing that, for instance with something as I
> proposed in <https://issues.guix.gnu.org/53127>, should be high
> priority.

I think I'll look into the cycle issue, maybe cargo.toml can be
modified to not use rust-cfg-il and maybe the solution, if any, could
be used for other cycles.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Compiling rust things without cargo (super WIP POC)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

Ludovic Courtès schreef op vr 01-04-2022 om 11:10 [+0200]:
> I guess the whole question is whether that technique can be made to work
> for the vast majority of Rust packages.  I’d suggest working in that
> direction: writing a build system as a first step, using it in all the
> Rust packages, and analyzing the kinds of problems encountered, with the
> goal of estimating the effort it would take to make it work for every
> single package.  Easier said than done, I guess.

Do you know a ‘real’ Rust applications with few transitive dependencies
(say, 3 or so) with preferably few rust source files?  That would be a
practical target for the ad-hoc Makefile and eventually, the new
antioxidant-build-system.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Compiling rust things without cargo (super WIP POC)

[Hartmut Goebel]

Am 01.04.22 um 12:08 schrieb Maxime Devos:
> Do you know a ‘real’ Rust applications with few transitive dependencies
> (say, 3 or so) with preferably few rust source files?

For my tests I used „roxmltree“, and by searching I just discovered 
https://crates.io/crates/hello_exercism.

HTH

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Compiling rust things without cargo (super WIP POC)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

Maxime Devos schreef op do 31-03-2022 om 22:06 [+0200]:
> As a next step, maybe I could try writing a Guix package definition
> for libhello
> and hello-oxygen, gradually making things more complicated (macros,
> transitive
> dependencies, some non-toy Rust dependencies, a Guix build system
> ...)?

Update: it can now build rust-cfg-if, libhello and rust-hello with
Guix!

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Compiling rust things without cargo (super WIP POC)

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Maxime Devos schreef op do 31-03-2022 om 22:06 [+0200]:
>> As a next step, maybe I could try writing a Guix package definition
>> for libhello
>> and hello-oxygen, gradually making things more complicated (macros,
>> transitive
>> dependencies, some non-toy Rust dependencies, a Guix build system
>> ...)?
>
> Update: it can now build rust-cfg-if, libhello and rust-hello with
> Guix!

Woohoo, keep up the good work!  :-)

Ludo’.

Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Hi,

antioxidant-build-system can now be used for some ‘real’ software -- it
compiles 'hexyl'.  To test, download
<https://notabug.org/maximed/cargoless-rust-experiments> (commit:
d09fd93750ac6d77e0c85623286b45cf5c3b055b) and run
"guix build -L . -f guix.scm" and then

$ cat guix.scm | /gnu/store/[...]-hexyl-0.8.0/bin/hexyl
> lots of coloured hex output

Some features of antioxidant-build-system:

  * no copying source code of dependencies
  * no compiling dependencies again -- old artifacts are reused
  * all dependencies use the usual package input system
    (native-inputs, inputs, propagated-inputs)

Limitations:

  * no support for linking to arbitrary shared libraries yet
    (only rust deps)
  * makes a few assumptions on the source layout (can be fixed
    by using more info from Cargo.toml)
  * no tests
  * no cross-compilation yet
  * no shared libraries (just replacing 'rlib' by 'dylib' causes problems)
  * code is a bit messy
  * no cdylib yet (probably needed for librsvg)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2291 bytes --]

Maxime Devos schreef op za 02-04-2022 om 17:18 [+0200]:
>   * makes a few assumptions on the source layout (can be fixed
>     by using more info from Cargo.toml)

This has now been partially resolved.  It now looks in Cargo.toml to
determine where Rusts's equivalent of 'configure' (build.rs) is
located, and the Cargo.toml can now override the default src/lib.rs.

Some other improvements that weren't announced previously:

  * The code now automatically detects cycles and reports which
    packages are involved in the cycle.
  * --edition (cf. -std=c11/c99/...) is set appropriately, depending on
    Cargo.toml, fixing some build failures.
  * All default 'features' (in Cargo.toml) are now enabled by default.
  * Package definitions can request non-default features to be built
    anyway.

    A difference from cargo-build-system: features are set in the
    package of the rust crate, not the package using the rust crate.
  * build.rs is now run, which reduces the number of features that need
    to be set manually.
  * proc-macro crates are now named .so instead of .rlib, apparently
    naming them .rlib even though they aren't .rlibs confuses rustc
    or something.
  * #[deny(warnings)] (rust's equivalent of -Werrors) is now ignored
    (FWIW this is done by cargo as well).
  * Conversion from cargo-build-system to antioxidate-build-system is
    now automated by a recursive 'vitaminate/auto' procedure and some
    ad-hoc cycle breaking.

Some things that could perhaps be moved to (upstream) Guix:

  * Sometimes test and build.rs dependencies are listed in
    #:cargo-inputs and #:cargo-development-inputs (e.g.
    rust-quickcheck, rust-cc).  I deleted the test dependencies
    (because they contribute to cycles and for now running tests
    isn't supported anyways).
   
    Maybe they can be moved to #:cargo-development-inputs upstream
    (untested)?  Extending 'check-inputs-should-be-native' would also
    be nice too ...

  * There's a cycle between rust-backtrace-sys and rust-cc can be
    eliminated by deleting build.rs and tweaking Cargo.toml, at cost
    of losing some support for Android (which is not supported by Guix
    except in the sense that Linux is supported).

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Hartmut Goebel]

Am 05.04.22 um 18:10 schrieb Maxime Devos:
> Some other improvements that weren't announced previously: 

Wow! Impressive!

I'd be eager to tr it with sequoia-openpgp. Please drop me a note when 
you think your work might be able to build that beast.


>    * Package definitions can request non-default features to be built
>      anyway.
>
>      A difference from cargo-build-system: features are set in the
>      package of the rust crate, not the package using the rust crate.

How is this intended to work?

Package 1 has features AAA (= default) and BBB. So a .rlib is build for 
each feature (package1-AAA.rlib, package1-BBB.rlib) Or will one need to 
define two guix packages (package1+aaa and package1+bbb) and make the 
build-system build the respective feature?

I personally would prefer the former. Thus package2 would pick up the 
pre-compiled rlib for the respective feature.

Rational: If there are more features to be combined, the number of 
packages to be build can a order of square. So defining all these 
packages becomes a burden quickly. This is a computer's job :-) The 
build-system could easily build all combinations. suffixing each rlib 
with a short hash over the feature names,


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

Hartmut Goebel schreef op wo 06-04-2022 om 17:49 [+0200]:

> [...]
> >     * Package definitions can request non-default features to be built
> >       anyway.
> > 
> >       A difference from cargo-build-system: features are set in the
> >       package of the rust crate, not the package using the rust crate.
> 
> How is this intended to work?
> 
> Package 1 has features AAA (= default) and BBB. So a .rlib is build for 
> each feature (package1-AAA.rlib, package1-BBB.rlib) Or will one need to 
> define two guix packages (package1+aaa and package1+bbb) and make the 
> build-system build the respective feature?
> 
> I personally would prefer the former. Thus package2 would pick up the 
> pre-compiled rlib for the respective feature.
> 
> Rational: If there are more features to be combined, the number of 
> packages to be build can a order of square. So defining all these 
> packages becomes a burden quickly. This is a computer's job :-) The 
> build-system could easily build all combinations. suffixing each rlib 
> with a short hash over the feature names,

My third solution: no package variants, no multiple .rlib variants.
Instead, by default enable the ‘default features’ mentioned in the
Cargo.toml, and (manually) add the extra features to the package
definition of the Rust dependency if they turn out to be needed by the
dependent.

This is like adding --enable-this --enable-that, ... to ffmpeg's
#:configure-flags instead of building a variant of ffmpeg for each
dependent of ffmpeg that needs a different feature set.

So far, I have not encountered incompatible features ... Let's see if
it will become a problem.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

On Sat, Apr 02, 2022 at 05:18:55PM +0200, Maxime Devos wrote:
> Hi,
> 
> antioxidant-build-system can now be used for some ‘real’ software -- it
> compiles 'hexyl'.  To test, download
> <https://notabug.org/maximed/cargoless-rust-experiments> (commit:
> d09fd93750ac6d77e0c85623286b45cf5c3b055b) and run
> "guix build -L . -f guix.scm" and then
> 
> $ cat guix.scm | /gnu/store/[...]-hexyl-0.8.0/bin/hexyl
> > lots of coloured hex output
> 
> Some features of antioxidant-build-system:
> 
>   * no copying source code of dependencies
>   * no compiling dependencies again -- old artifacts are reused
>   * all dependencies use the usual package input system
>     (native-inputs, inputs, propagated-inputs)
> 
> Limitations:
> 
>   * no support for linking to arbitrary shared libraries yet
>     (only rust deps)
>   * makes a few assumptions on the source layout (can be fixed
>     by using more info from Cargo.toml)
>   * no tests
>   * no cross-compilation yet
>   * no shared libraries (just replacing 'rlib' by 'dylib' causes problems)
>   * code is a bit messy
>   * no cdylib yet (probably needed for librsvg)

Something that might help with that would be to also include the source
of the crate somewhere in the output. Then at the worst we could just
put the rust inputs of librsvg as regular inputs and let librsvg do its
own special build thing. We'd then still keep the build dependency tree
you've got working.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[raingloom]

On Mon, 30 May 2022 11:23:21 +0300
Efraim Flashner <efraim@flashner.co.il> wrote:

> On Sat, Apr 02, 2022 at 05:18:55PM +0200, Maxime Devos wrote:
> > Hi,
> > 
> > antioxidant-build-system can now be used for some ‘real’ software
> > -- it compiles 'hexyl'.  To test, download
> > <https://notabug.org/maximed/cargoless-rust-experiments> (commit:
> > d09fd93750ac6d77e0c85623286b45cf5c3b055b) and run
> > "guix build -L . -f guix.scm" and then
> > 
> > $ cat guix.scm | /gnu/store/[...]-hexyl-0.8.0/bin/hexyl  
> > > lots of coloured hex output  
> > 
> > Some features of antioxidant-build-system:
> > 
> >   * no copying source code of dependencies
> >   * no compiling dependencies again -- old artifacts are reused
> >   * all dependencies use the usual package input system
> >     (native-inputs, inputs, propagated-inputs)
> > 
> > Limitations:
> > 
> >   * no support for linking to arbitrary shared libraries yet
> >     (only rust deps)
> >   * makes a few assumptions on the source layout (can be fixed
> >     by using more info from Cargo.toml)
> >   * no tests
> >   * no cross-compilation yet
> >   * no shared libraries (just replacing 'rlib' by 'dylib' causes
> > problems)
> >   * code is a bit messy
> >   * no cdylib yet (probably needed for librsvg)  
> 
> Something that might help with that would be to also include the
> source of the crate somewhere in the output. Then at the worst we
> could just put the rust inputs of librsvg as regular inputs and let
> librsvg do its own special build thing. We'd then still keep the
> build dependency tree you've got working.
> 
> 

Maybe put the source in an extra output, not the main "out" one.
Splitting them up later for size reduction will be much more difficult
than doing this from the get-go.

Re: Building hexyl (a rust app) without cargo, with antioxidant-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

Efraim Flashner schreef op ma 30-05-2022 om 11:23 [+0300]:
> Something that might help with that would be to also include the
> source of the crate somewhere in the output. Then at the worst we
> could just put the rust inputs of librsvg as regular inputs and let
> librsvg do its own special build thing. We'd then still keep the
> build dependency tree you've got working.

Maybe, but that kind of defeats the point of

 * no compiling dependencies again -- old artifacts are reused

and librsvg's build system seems to use Cargo, so then we again have
the problem that Cargo wants Windows crates etc. that are useless in
Guix and its overly-pickyness in versioning, we lose the ability to let
crates be shared libraries (not yet implemented, but could be
implemented in antioxidant) ..., so I'd rather just implement cdylib
and patch the build system appropriately.

(It's build system is more complicated than ordinary crates, but the
additional Makefile.am isn't too long).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Compiling rust things without cargo (super WIP POC)

[Hartmut Goebel]

Am 31.03.22 um 22:06 schrieb Maxime Devos:
> In my experiments, it looks like the rust compiler actually_does_
> support static libraries, though perhaps cargo doesn't.

AFAIU this assumption is correct.


> I invite you to take a look at<https://notabug.org/maximed/cargoless-rust-experiments>.
> It contains a minimal rust library (libhello) and a minimal 'hello
> world'-style application that uses 'libhello'.
Impressive!
> As a next step, maybe I could try writing a Guix package definition for libhello
> and hello-oxygen, gradually making things more complicated (macros, transitive
> dependencies, some non-toy Rust dependencies, a Guix build system ...)?

Here is my challenge :-) 
<https://gitlab.com/sequoia-pgp/sequoia/-/blob/main/openpgp/Cargo.toml>: 
different dependencies per feature, os, target-arch and target-os as 
well as passing on features to dependencies.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |