Re: antioxidant-build-system can be tested as a channel, + > GTK app 'castor' builds

Re: antioxidant-build-system can be tested as a channel, + > GTK app 'castor' builds

[kiasoc5]

Hi Maxime,

> 
> Non-goals:
> 
> * Produce exactly the same binaries with exactly the same dependencies as with
> Cargo. If you want to reproduce a binary produced with Cargo, use Cargo.
> 

If I compile project P to produce binary A (with antioxidant) and binary C (with Cargo), will A and C have the same functionality?

And for dependencies not being the same, is this just limited to using the latest version of crates instead of following the versions in Cargo.toml?

Re: antioxidant-build-system can be tested as a channel, + > GTK app 'castor' builds

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

kiasoc5@disroot.org schreef op di 31-05-2022 om 17:45 [+0000]:
> Hi Maxime,
> 
> > 
> > Non-goals:
> > 
> > * Produce exactly the same binaries with exactly the same dependencies as with
> > Cargo. If you want to reproduce a binary produced with Cargo, use Cargo.
> > 
> 
> If I compile project P to produce binary A (with antioxidant) and binary C
> (with Cargo), will A and C have the same functionality?

If nothing weird is going on, yes.  Though antioxidant and Cargo have
different methods for selecting ‘features’, so a creative Rust app
author could engineer things such that when compiled with Cargo, the
app prints "hello cargo" and when compiled with antioxidant, it prints
"hello antioxidant".

I expect such situations to be the exception though, and if such a
thing happens, a $GUIX_PACKAGER could manually choose the right set of
‘features’ and put an outdated dependency in inputs/native-inputs if
needed.

That said, I haven't ‘tried out’ many of the Rust apps, mostly only
built them.

> And for dependencies not being the same, is this just limited to using the latest version of crates instead of following the versions in Cargo.toml?

If you feed antioxidant-build-system the exact same versions
of dependencies as specified in Cargo, in principle you should
get something equivalent (except maybe for feature flags because
antioxidant has a different procedure for automatically selecting
feature flags, but if you really want to, you as writer of the
package definition could manually pick the features such that they
match 100% with those that Cargo would have picked).

However, the package transformation procedures in the antioxidant
channel for turning Cargo packages into antioxidant packages currently
often just select the latest version.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

antioxidant-build-system can be tested as a channel, + GTK app 'castor' builds

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

Hi,

I've set up the channel things for the antioxidant repo,
so now you can put the following in channels.scm:

(use-modules (guix ci))

(cons
 (channel
  (name 'antioxidated-packages)
  (url "https://notabug.org/maximed/cargoless-rust-experiments")
  (introduction
   (make-channel-introduction
    "020851ad649480ee4769b77a947642e993ea5956"
    (openpgp-fingerprint
     "C1F3 3EE2 0C52 8FDB 7DD7  011F 49E3 EE22 1917 25EE"))))
 (list (channel-with-substitutes-available
            %default-guix-channel
            "https://ci.guix.gnu.org")))


do "guix pull" and "guix shell antioxidated-castor -- castor" to
run a 'castor' build (*) with antioxidant-build-system.  (Warning:
some crates were updated or added without checking the source code
diff for malware!)

(*) A graphical web browser for Gemini using the GTK stack,
implemented in Rust.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: antioxidant-build-system can be tested as a channel, + GTK app 'castor' builds

[Ludovic Courtès]

Hello,

Maxime Devos <maximedevos@telenet.be> skribis:

> I've set up the channel things for the antioxidant repo,
> so now you can put the following in channels.scm:
>
> (use-modules (guix ci))
>
> (cons
>  (channel
>   (name 'antioxidated-packages)
>   (url "https://notabug.org/maximed/cargoless-rust-experiments")
>   (introduction
>    (make-channel-introduction
>     "020851ad649480ee4769b77a947642e993ea5956"
>     (openpgp-fingerprint
>      "C1F3 3EE2 0C52 8FDB 7DD7  011F 49E3 EE22 1917 25EE"))))
>  (list (channel-with-substitutes-available
>             %default-guix-channel
>             "https://ci.guix.gnu.org")))
>
>
> do "guix pull" and "guix shell antioxidated-castor -- castor" to
> run a 'castor' build (*) with antioxidant-build-system.  (Warning:
> some crates were updated or added without checking the source code
> diff for malware!)

Nice!  (Unfortunately notabug.org is down since a few days ago.)

To make sure share a common understanding, could you post a summary of:

  1. the goals;
  2. the status;
  3. pros and cons over the status quo and other options (if any!);
  4. the next steps.

I think you already explained #1 but it can’t hurt to rehash.  ;-)

It’d be great to address the major shortcoming of the current solution,
which is its isolation from the rest of Guix.

Thanks,
Ludo’.

Re: antioxidant-build-system can be tested as a channel, + GTK app 'castor' builds

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 250 bytes --]

Ludovic Courtès schreef op ma 30-05-2022 om 17:26 [+0200]:
> (Unfortunately notabug.org is down since a few days ago.)

For now, the fallback location
https://github.com/emixa-d/antioxidant-fallback/
can be used.  (rest of the response later)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: antioxidant-build-system can be tested as a channel, + GTK app 'castor' builds

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 4885 bytes --]

Ludovic Courtès schreef op ma 30-05-2022 om 17:26 [+0200]:
[...]
To make sure share a common understanding, could you post a summary of:

  1. the goals;
  2. the status;
  3. pros and cons over the status quo and other options (if any!);
  4. the next steps.
> [...]

(notabug.org is down, so I've made a copy at <https://github.com/emixa-d/antioxidant-fallback/>.)

Goals:

  * Reduce rift between Guix and Rust by making Rust packages work
    nicely in Guix.
  * Eliminating #:skip-build? and making compile time faster,
    by not re-compiling dependencies and by reusing compiled dependencies
    in dependents.  (solved)
  * Allow crates to be shared libraries, such that grafting can work
    and maybe for space usage. (not solved yet)
  * regular inputs/native-inputs (solved)

Non-goals:

  * Produce exactly the same binaries with exactly the same dependencies as with
    Cargo.  If you want to reproduce a binary produced with Cargo, use Cargo.
  * Cross-compilation is left for later.

Side-goals: (not the main goal, but partially solved along the way)

  * Having multiple versions of the same dependency is inconvenient for
    antioxidant, so often only the latest version is preserved.
    Less package variants -> less space usage, less compilation time, less
    opportunities for sneaking malware through, less old buggy versions.
  * As part of resolving build failures caused by using antioxidant,
    some crates were updated or added.

Solved issues:

  * <https://issues.guix.gnu.org/25327>
    ‘cargo build-system should be able to filter out target.cfg(windows) dependencies’
  * <https://issues.guix.gnu.org/53127>
    ‘[PATCH RFC] Turning Rust/Cargo inputs into “regular” inputs?’
  * <https://issues.guix.gnu.org/40412>
    ‘cargo-build-system hides the package dependency graph’

    (at least, after transforming the packages appropriately)

  * <https://issues.guix.gnu.org/50015>
    ‘Rust packages are not reproducible’

    (at least, "guix build --check" hasn't failed for me yet)

Avoided issues:
  * <https://issues.guix.gnu.org/42049>
    ‘[PATCH 0/4] build-system/cargo: Propagations across the crate closure.’
  * <https://issues.guix.gnu.org/36470>
    ‘Cargo-build-system guix-vendor does not provide packages correctly 2 levels down’
  * <https://issues.guix.gnu.org/40413>
    ‘Handling source-only dependencies in a generic way’

Pros/cons of antioxidant:
  * pro: less compilation time: when rust-foo is compiled, antioxidant
    only compiles rust-foo and not its dependencies rust-bar0 ... rust-bar200.
  * pro: theoretically less space usage (source code is not copied to the #$output
    and build results of dependencies are linked to).  However: not measured,
    and we're not setting LTO and -Os compilation flags yet.
  * pro: antioxidant just uses inputs/native-inputs, so "guix graph", "guix refresh -l"
    will be useful after transforming existing package definitions.
  * cons: probably not everything of Cargo.toml is implemented yet

Status:
  * Unless they have been broken, the following packages are known to build:

    antioxidated-castor
    antioxidated-dutree
    antioxidated-diffr
    antioxidated-git-absorb
    antioxidated-hex
    antioxidated-hexyl
    antioxidated-ripgrep
    antioxidated-rtss
    antioxidated-sniffglue
    antioxidated-swayhide
    antioxidated-tealdeer

 * I wouldn't know how many fail to build

Next steps :

(a) The antioxidant channel has information on how to transform
    package definitions using cargo-build-sstem to antioxidant-build-system.
    Maybe "guix style" could be taught to perform these changes to
    the source code?

(b) To get a good grasp on what builds/what not, it would be useful
    to build the channel at ci.guix.gnu.org or such.  More concretely,
    the CI would grab all cargo-build-system rust apps from (gnu packages ...),
    feed them to the package transformation procedure 'vitaminate/auto',
    and build the results.  Then we can have a look at a dashboard like
    <https://ci.guix.gnu.org/eval/361030/dashboard>.

(c) Maybe there could be some kind of transition period, where
    'antioxidant.scm' and 'antioxidant-packages.scm' is added
    to the Guix repo, known-to-build packages are replaced by
    their antioxidated variant (using vitaminate/auto) and gradually
    all rust apps are becoming supported by antioxidant.  Finally,
    when all are supported, the "guix style" from (a) could be run
    to clean up the source code?

    Though for now, I would prefer to do something like (b) first.

(d) Implement support for "cdylib" such that things like librsvg can be built.

(e) Make sure the optimisation settings are reasonable (LTO, -Os etc)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: antioxidant-build-system can be tested as a channel, + GTK app 'castor' builds

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 280 bytes --]

Maxime Devos schreef op di 31-05-2022 om 11:27 [+0200]:
> (d) Implement support for "cdylib" such that things like librsvg can
> be built.

Some basic support now implemented in
7040ce32840ba74f948fe9d243e1eb393daec4fc, though probably more remains
to be done for librsvg.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: antioxidant-build-system can be tested as a channel, + GTK app 'castor' builds

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

Maxime Devos schreef op di 31-05-2022 om 11:27 [+0200]:
(b) To get a good grasp on what builds/what not, it would be useful
    to build the channel at ci.guix.gnu.org or such.  More
concretely,
    the CI would grab all cargo-build-system rust apps from (gnu
packages ...),
    feed them to the package transformation procedure
'vitaminate/auto',
    and build the results.  Then we can have a look at a dashboard
like
    <https://ci.guix.gnu.org/eval/361030/dashboard>.

I've added a file antioxidant-ci.scm to the antioxidant repo which
automatically produces a list of antioxidated packages (except for
those for that the vitaminate/auto procedure currently cannot grok).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: antioxidant-build-system can be tested as a channel, + GTK app 'castor' builds

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 588 bytes --]

Maxime Devos schreef op di 31-05-2022 om 11:27 [+0200]:
>   * cons: probably not everything of Cargo.toml is implemented yet

Another downside: in Cargo-land, the leaf package declares what
‘features’ (= configure flags for enabling things) it requires from
dependencies.  This works because Cargo can recompile dependencies.
In antioxidant, recompiling dependencies would defeat the point of
antioxidant.  So if there's a non-default feature to enable, the
packager needs to manually enable that feature in the package
definition of the dependency.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]