Why does Guix duplicate dependency versions from Cargo.toml?

Why does Guix duplicate dependency versions from Cargo.toml?

[Jonas Møller]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

Hi Guix! Why does cargo-build-system need #:cargo-inputs specified in the package definition? This seems like a big mistake for a couple of reasons.

- It is completely redundant, it should match what is in Cargo.toml. I know `guix import crate` exists to automate this process, but I don't understand the rationale for duplicating this information.
- It is bad practice for Guix to override Cargo.lock if it exists, this means that Guix is building a different binary to the one the developers of the packaged Rust application are seeing on their end, this is a much bigger problem. This can and will cause spurious build failures, or bugs that are unknown to the developers of the Rust programs that Guix packages.

[-- Attachment #2: Type: text/html, Size: 1065 bytes --]

Re: Why does Guix duplicate dependency versions from Cargo.toml?

[(]

Jonas Møller <jonas@moller.systems> writes:
> Hi Guix! Why does cargo-build-system need #:cargo-inputs specified in the package definition? This seems like a
> big mistake for a couple of reasons.
>
> 1 It is completely redundant, it should match what is in Cargo.toml. I know `guix import crate` exists to
>  automate this process, but I don't understand the rationale for duplicating this information.

Because it'd be literally impossible to do otherwise.  You'd have to
contort Guix into some pretty weird shapes to change a package
derivation based on what was downloaded as the source.

  -- (

Re: Why does Guix duplicate dependency versions from Cargo.toml?

[Jonas Møller]

Interesting, Guix already has git/url-fetch, what is keeping Guix from simply fetching a cargo project and then running `cargo build` in the fetched source directory?

If the problem is that the build daemon is sandboxed and doesn't have internet access, it is also feasible to have one stage of the build process download all the resources specified in Cargo.lock (and cache this in /gnu/store) and rewrite `version = "x.y.z"` to `path = "x/y/z"` before everything is passed to the build daemon.

I just don't see why this can't be ad-hoc.



------- Original Message -------
On Thursday, August 24th, 2023 at 09:05, ( <paren@disroot.org> wrote:


> 
> 
> Jonas Møller jonas@moller.systems writes:
> 
> > Hi Guix! Why does cargo-build-system need #:cargo-inputs specified in the package definition? This seems like a
> > big mistake for a couple of reasons.
> > 
> > 1 It is completely redundant, it should match what is in Cargo.toml. I know `guix import crate` exists to
> > automate this process, but I don't understand the rationale for duplicating this information.
> 
> 
> Because it'd be literally impossible to do otherwise. You'd have to
> contort Guix into some pretty weird shapes to change a package
> derivation based on what was downloaded as the source.
> 
> -- (

Re: Why does Guix duplicate dependency versions from Cargo.toml?

[(]

Hi,

Sorry if I came off a bit harsh in the initial reply :)  I didn't intend
for it to read as a "ugh, how don't you understand this" sort of thing
but that's what it appeared to be looking at it later.

(Communication: It's Hard™)

Aaaaaaanyway....

Jonas Møller <jonas@moller.systems> writes:
> Interesting, Guix already has git/url-fetch, what is keeping Guix from simply
> fetching a cargo project and then running `cargo build` in the fetched source
> directory?

Okay, this will require a bit of explanation about how Guix's (and
Nix's) derivations work.

As I understand it, there are broadly two types of derivations:

  - fixed-output, used for things like <ORIGIN>
  - whatever-the-opposite-is-called, used for normal things like
    packages

The reason fixed-output derivations are called that is because their
hashes are *known before the derivation is built*, or at least their
expected hashes; so Guix will download the file/repo, and if it doesn't
match the given hash, it'll throw an error.

Now, this means there's no reproducibility issue with internet access.
If produced outputs O1 and and O2 are different, then either one or both
will fail the hash check, and thus the output will never be built.
There can never *be* a reproducibility issue because if there was one on
the server side Guix would catch a hash-mismatch before the consequences
of that irreproducibility were ever felt.

Thus, *it is safe to allow internet access in a fixed-output build*,
because reproducibility issues become null and void.  Or, at least,
that's how I understand it.  On the flip side, of course, regular build
scripts are not allowed to access anything (other than stuff we can't
seem to figure out how to block, like system time) that could affect
reproducibility.

> If the problem is that the build daemon is sandboxed and doesn't have internet
> access, it is also feasible to have one stage of the build process download all
> the resources specified in Cargo.lock (and cache this in /gnu/store) and rewrite
> `version = "x.y.z"` to `path = "x/y/z"` before everything is passed to the build
> daemon.

This is actually *extraordinarily* close to what we already do.  The
cargo-build-system, when building a library, copies its entire source
into the output directory (I know, I know... But without writing our own
Rust build system, there's no alternative.)

We use this source when building packages that depend on it; the sources
of the #:CARGO-INPUTS of a package in the process of being built are
copied into a 'guix-vendor' directory, and then we pass this flag to
Cargo which makes it treat the vendor directory like a local package
registry that takes precedence over crates.io.

Hopefully that clears things up :)

  -- (

Re: Why does Guix duplicate dependency versions from Cargo.toml?

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 2376 bytes --]


Jonas Møller <jonas@moller.systems> writes:

> Hi Guix! Why does cargo-build-system need #:cargo-inputs specified in the package definition? This seems like a big
> mistake for a couple of reasons.

Just like the nice people in mail list explained, when building a
package, Guix builders are not allowed to connect to network, so the
crates should be prefetched.

and AFIAK, Maxime Devos is working on new build system called
"Antioxidant", which can build rust application without cargo (Yes,
invoke rustc directly!), The new build system will cache the rlib
intermediate result of crate and share between different builds. 

> 1 It is completely redundant, it should match what is in Cargo.toml. I know `guix import crate` exists to automate
>  this process, but I don't understand the rationale for duplicating
>  this information.

Not only Guix do this, Debian also package Rust crates [1]. 

> 2 It is bad practice for Guix to override Cargo.lock if it exists, this means that Guix is building a different binary to the
>  one the developers of the packaged Rust application are seeing on their end, this is a much bigger problem.  

OK, IMO almost all software packaged by Linux distro are "different"
from upstream at binary level. A notable example is, software developer
may prefeer to add bundled 3rd party dependencies to ease the build of
package. But distro maintainers want to ensure every software use the
library provided by distro[2]. And there maybe distro specific patches
(Redhat backports security fixes to the old version of package for their
RHEL)

>  This can and will cause spurious build failures, or bugs that are unknown to the developers of the Rust programs that
>  Guix packages.

The Rust crate in Guix is used to package rust application, if user have
problem with the rust application on Guix (e.g. ripgrep, fd, xxd, bat
...) They should report to Guix first, so it's Guix developer's
responsibility to smooth out those differences. 

If user want to develop a Rust crate/application, they can still use
"cargo" command and fetch crates from crates.io registry.

 

[1]: https://packages.debian.org/sid/librust-bytecount-dev
[2]: https://wiki.gentoo.org/wiki/Why_not_bundle_dependencies
-- 
Retrieve my PGP public key:

  gpg --recv-keys B3EBC086AB0EBC0F45E0B4D433DB374BCEE4D9DC

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Why does Guix duplicate dependency versions from Cargo.toml?

[(]

Zhu Zihao <all_but_last@163.com> writes:
> and AFIAK, Maxime Devos is working on new build system called
> "Antioxidant", which can build rust application without cargo (Yes,
> invoke rustc directly!), The new build system will cache the rlib
> intermediate result of crate and share between different builds. 

Sadly, I think that's been abandoned :(

  -- (

Re: Why does Guix duplicate dependency versions from Cargo.toml?

[Andreas Enge]

Am Fri, Aug 25, 2023 at 03:56:56PM +0100 schrieb (:
> Zhu Zihao <all_but_last@163.com> writes:
> > and AFIAK, Maxime Devos is working on new build system called
> > "Antioxidant", which can build rust application without cargo (Yes,
> > invoke rustc directly!), The new build system will cache the rlib
> > intermediate result of crate and share between different builds. 
> Sadly, I think that's been abandoned :(

My impression is that Maxime has left the Guix project, so that the
work was naturally abandoned, but there was no conscious decision that
it should be dropped. So it would be nice if someone knowledgeable about
the topic could pick it up and finish the build system.

Andreas

Re: Why does Guix duplicate dependency versions from Cargo.toml?

[Attila Lendvai]

i may not understand this well enough, but with that in mind...

the nix crowd allows something that they call vendoring: they use the native tools of the language ecosystem to fetch the transitive closure of the dependencies, as specified by their own package management descriptions. then they compute a hash on the entire directory, and record it in the leaf package's definition. i think this vendoring dir/archive then even gets cached by their substitute servers (for prosperity).

IIUC, this method is rejected by guix on principle.

if someone wants to test their mailing list search-fu, then there was a similar discussion about golang in the past.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“As children go, so go nations. It's that simple.”
	— Carol Bellamy