Overhauling the cargo-build-system

Overhauling the cargo-build-system

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3555 bytes --]

I've spent a lot of time over the past few months hacking at packaging
rust crates. I've found it time-consuming, soothing and, ultimately,
likely a waste of time. When we started with the cargo-build-system the
premise was 'packages are both libraries and source, we need them in
source form to build the next library in the chain' and the example was
our first three rust- packages, rust-quote, rust-syn and
rust-proc-macro2, all three of which depend on the other two.

I'd like to challenge the assumption that packages are both libraries
and source. A 'library' in rust compiles into one of three types: a
static library (libfoo.a), a shared library (libfoo.so), or a
'rust-library' (libfoo.rlib). A 'rust library', an rlib file, is like a
shared library but with rust-specific metadata. I haven't looked into
what exactly that means, but I can tell you practically how it works.
Through some in depth searching online it seems it is possible to
pre-compile rlibs and link them to a final binary, but it is not
possible to link an rlib to another rlib. What this means is currently
we have $(guix package -A ^rust- | wc -l) ~> 192 rust libraries packaged
that no one will ever use in any form EXCEPT as source inputs for a
future library or binary.

Let me repeat that. We have 192 rust packages that no one needs or
wants, except in source form.

We have a couple of packages/variables which are defined in source-form
only. We normally see them as inputs or native inputs as
("foo" ,(origin .... This is basically what we have with the rust
packages. Upstream to build a binary they call 'cargo build' and it
downloads the sources of the dependent libraries, and some of their
dependants based on the use-case, compiles them, and then compiles the
binary. In Guix, when we eventually have packaged enough libraries to
get to a target binary, we'll be pulling in hundreds of extra libraries
(transitive inputs of inputs) so a few dozen can be compiled.

PROPOSAL:
Change all the rust packages we have now to be source-only. Rename them
from rust-foo to rust-foo-src or rust-src-foo. The package 'alacritty'
is perhaps not a great example choice, but it's Cargo.lock file, which
describes the dependencies and their versions, lists 278 dependencies.
(Generally you can care only about the major and minor part of the
version.) It's package definition would explicitly list the 278
dependent libraries it wanted so they could be put in the 'guix-vendor'
directory like now. It makes for a very large package definition, but we
wouldn't have to ensure thousands of other rust libraries built so we
can throw away the results and build alacritty in the end. Another
package, starship, would have 124 rust dependencies, but I'm betting
there's a fair amount of overlap between them.

Since we never actually use the output of any of the rust packages we
currently have it doesn't really make sense to 'build them' per se as we
currently do.

$ tree $(guix build rust-proc-macro2)
/gnu/store/jjinj0dvvzqnlpn9li3nxcyrpv1fbbfp-rust-proc-macro2-0.4.30
`-- share
    `-- doc
         `-- rust-proc-macro2-0.4.30
             |-- LICENSE-APACHE
             `-- LICENSE-MIT

My plan is to hack on the cargo-build-system on a separate branch and
see if I can pull all of this together.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Overhauling the cargo-build-system

[Ludovic Courtès]

Hello!

Efraim Flashner <efraim@flashner.co.il> skribis:

> I'd like to challenge the assumption that packages are both libraries
> and source. A 'library' in rust compiles into one of three types: a
> static library (libfoo.a), a shared library (libfoo.so), or a
> 'rust-library' (libfoo.rlib).

Why don’t we create .so files, then?  They have NEEDED and RUNPATH, so
that could work like for C, no?

> Let me repeat that. We have 192 rust packages that no one needs or
> wants, except in source form.

Ouch!  So the rlib file is never actually used?!

You said “it is not possible to link an rlib to another rlib”, but
that’s not necessarily a problem, it’s like .a libraries, no?

> PROPOSAL:
> Change all the rust packages we have now to be source-only. Rename them
> from rust-foo to rust-foo-src or rust-src-foo.

In the current scheme, can you actually do, say:

  guix environment --ad-hoc rust rust-foo rust-bar

and then (pseudo syntax):

  rustc mystuff.rust -lfoo -lbar

?

Thanks,
Ludo’.

Re: Overhauling the cargo-build-system

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 6322 bytes --]

On Fri, Oct 11, 2019 at 12:33:18AM +0200, Ludovic Courtès wrote:
> Hello!
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > I'd like to challenge the assumption that packages are both libraries
> > and source. A 'library' in rust compiles into one of three types: a
> > static library (libfoo.a), a shared library (libfoo.so), or a
> > 'rust-library' (libfoo.rlib).
> 
> Why don’t we create .so files, then?  They have NEEDED and RUNPATH, so
> that could work like for C, no?
> 

It is possible to force a library to produce .so files, but then we'd
have to force all the other packages to link against them, and then at
that point we're basically rewriting 'cargo build'. For some packages,
like icecat, that does seem to be what they do.

> > Let me repeat that. We have 192 rust packages that no one needs or
> > wants, except in source form.
> 
> Ouch!  So the rlib file is never actually used?!
> 

As I understand it, it gets treated as a build artifact. It'll be
compiled as libfoo_XXXXXXXXX.rlib and then, if some other program comes
around as asks for "foo with feature baz" it'll be reused. At some point
it gets compiled directly into the final binary, so it's really more of
an intermediary stage.

> You said “it is not possible to link an rlib to another rlib”, but
> that’s not necessarily a problem, it’s like .a libraries, no?
> 

if libfoo depends on libbar, then pre-building libbar_XXXXX.rlib doesn't
allow us to call "rustc --crate-name foo src/lib.rs --crate-type lib
--extern (string-append bar=(assoc-ref %build-inputs bar)
/lib/libbar.rlib)" (note the --crate-type lib in there), it'll just not
work and want to recompile libbar for just the few functions it uses
from it. The only difference I've seen is if our foo is also a "binary
crate", ie produces a binary, then it'll sometimes accept rlibs for its
own rlib lib output.

Going back to forcing it to produce .so files, if we changed the
crate-type to dylib then we'd get libfoo.so. The two parts of the
problem is now we need to change all the other libraries and programs
that depend on foo to take our libfoo.so and not expect an rlib, and
that if foo has 3 options then we have to decide how to build libfoo so
it's useful for everything (obviously all 3) but without dragging in
everything (easiest with none of the 3), or to have some combination,
which becomes a nightmare very quickly.

That sentence was way too long.

> > PROPOSAL:
> > Change all the rust packages we have now to be source-only. Rename them
> > from rust-foo to rust-foo-src or rust-src-foo.
> 
> In the current scheme, can you actually do, say:
> 
>   guix environment --ad-hoc rust rust-foo rust-bar
> 
> and then (pseudo syntax):
> 
>   rustc mystuff.rust -lfoo -lbar
> 
> ?

Ignoring that we don't actually install the libraries here's a copy of
a check phase I wrote trying to use installed libraries. I think in the
end it did consume the rlibs as requested, but it never did link to
them, it just absorbed them into the final binary.

(replace 'check
  (lambda* (#:key inputs #:allow-other-keys)
    (let ((ndarray    (assoc-ref inputs "ndarray"))
          (rand       (assoc-ref inputs "rand"))
          (rayon      (assoc-ref inputs "rayon"))
          (serde      (assoc-ref inputs "serde"))
          (serde-json (assoc-ref inputs "serde-json"))
          (structopt  (assoc-ref inputs "structopt")))
      (invoke "rustc" "--edition=2018" "--crate-name" "qtlreaper"
              "src/lib.rs"
              "-C" "opt-level=3"
              "--test" "-C" "metadata=test" "-C" "extra-filename=-test"
              "--out-dir" "target/release/deps"
              "--extern" (string-append "ndarray=" ndarray "/lib/libndarray.rlib")
              "--extern" (string-append "rand=" rand "/lib/librand.rlib")
              "--extern" (string-append "rayon=" rayon "/lib/librayon.rlib")
              "--extern" (string-append "serde=" serde "/lib/libserde.rlib")
              "--extern" (string-append "serde_json=" serde-json "/lib/libserde_json.rlib")
              "--extern" (string-append "structopt=" structopt "/lib/structopt.rlib")
              "--cap-lints" "allow" "-C" "rpath" "-C" "prefer-dynamic")
      (invoke "rustc" "--edition=2018" "--crate-name" "qtlreaper"
              "src/main.rs"
              "-C" "opt-level=3"
              "--test" "-C" "metadata=test" "-C" "extra-filename=-test"
              "--out-dir" "target/release/deps"
              "--extern" "qtlreaper=target/release/deps/libqtlreaper-test.rlib"
              "--extern" (string-append "ndarray=" ndarray "/lib/libndarray.rlib")
              "--extern" (string-append "rand=" rand "/lib/librand.rlib")
              "--extern" (string-append "rayon=" rayon "/lib/librayon.rlib")
              "--extern" (string-append "serde=" serde "/lib/libserde.rlib")
              "--extern" (string-append "serde_json=" serde-json "/lib/libserde_json.rlib")
              "--extern" (string-append "structopt=" structopt "/lib/structopt.rlib")
              "--cap-lints" "allow" "-C" "rpath" "-C" "prefer-dynamic")
      (invoke "./target/release/deps/qtlreaper-test")
      (invoke "rustdoc" "--edition=2018" "--test" "src/lib" "--crate-name" "qtlreaper"
              "--extern" "qtlreaper=target/release/deps/libqtlreaper-test.rlib"
              "--extern" (string-append "ndarray=" ndarray "/libndarray.rlib")
              "--extern" (string-append "rand=" rand "/librand.rlib")
              "--extern" (string-append "rayon=" rayon "/lib/librayon.rlib")
              "--extern" (string-append "serde=" serde "/lib/libserde.rlib")
              "--extern" (string-append "serde_json=" serde-json "/lib/libserde_json.rlib")
              "--extern" (string-append "structopt=" structopt "/lib/structopt.rlib"))))))))

> 
> Thanks,
> Ludo’.

I was thinking of making the build phases for the cargo-build-system use
the --verbose flag, I found it very informative to see what it was doing
along the way.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Overhauling the cargo-build-system

[Martin Becze]

Sorry for digging up and old issue, but i just saw commit
86e443c71d4d19e6f80cad9ca15b9c3a301c738c

> It makes for a very large package definition, but we
wouldn't have to ensure thousands of other rust libraries built so we

The whole point of package management is that you can use module
building blocks. By having to specify the sub-dependencies in a top
level definition kinda breaks the whole modular thing. In commit
86e443c71d4d19e6f80cad9ca15b9c3a301c738c all the inputs got removed for
all the libraries. So if I'm trying to use guix as package manager for a
rust project and I want to use one of the rust libraries in
crates-io.scm, how am i suppose to do this? I can't just include it as
an input to my project because now I have to look up all of it
dependencies as well?

> > PROPOSAL:
> > Change all the rust packages we have now to be source-only. Rename them
> > from rust-foo to rust-foo-src or rust-src-foo.

Is there no way we can do this and keep the have a dependancy graph?
 
On 2019-10-11 14:13, Efraim Flashner wrote:
> On Fri, Oct 11, 2019 at 12:33:18AM +0200, Ludovic Courtès wrote:
>> Hello!
>>
>> Efraim Flashner <efraim@flashner.co.il> skribis:
>>
>> > I'd like to challenge the assumption that packages are both libraries
>> > and source. A 'library' in rust compiles into one of three types: a
>> > static library (libfoo.a), a shared library (libfoo.so), or a
>> > 'rust-library' (libfoo.rlib).
>>
>> Why don’t we create .so files, then?  They have NEEDED and RUNPATH, so
>> that could work like for C, no?
>>
> 
> It is possible to force a library to produce .so files, but then we'd
> have to force all the other packages to link against them, and then at
> that point we're basically rewriting 'cargo build'. For some packages,
> like icecat, that does seem to be what they do.
> 
>> > Let me repeat that. We have 192 rust packages that no one needs or
>> > wants, except in source form.
>>
>> Ouch!  So the rlib file is never actually used?!
>>
> 
> As I understand it, it gets treated as a build artifact. It'll be
> compiled as libfoo_XXXXXXXXX.rlib and then, if some other program comes
> around as asks for "foo with feature baz" it'll be reused. At some point
> it gets compiled directly into the final binary, so it's really more of
> an intermediary stage.
> 
>> You said “it is not possible to link an rlib to another rlib”, but
>> that’s not necessarily a problem, it’s like .a libraries, no?
>>
> 
> if libfoo depends on libbar, then pre-building libbar_XXXXX.rlib doesn't
> allow us to call "rustc --crate-name foo src/lib.rs --crate-type lib
> --extern (string-append bar=(assoc-ref %build-inputs bar)
> /lib/libbar.rlib)" (note the --crate-type lib in there), it'll just not
> work and want to recompile libbar for just the few functions it uses
> from it. The only difference I've seen is if our foo is also a "binary
> crate", ie produces a binary, then it'll sometimes accept rlibs for its
> own rlib lib output.
> 
> Going back to forcing it to produce .so files, if we changed the
> crate-type to dylib then we'd get libfoo.so. The two parts of the
> problem is now we need to change all the other libraries and programs
> that depend on foo to take our libfoo.so and not expect an rlib, and
> that if foo has 3 options then we have to decide how to build libfoo so
> it's useful for everything (obviously all 3) but without dragging in
> everything (easiest with none of the 3), or to have some combination,
> which becomes a nightmare very quickly.
> 
> That sentence was way too long.
> 
>> > PROPOSAL:
>> > Change all the rust packages we have now to be source-only. Rename them
>> > from rust-foo to rust-foo-src or rust-src-foo.
>>
>> In the current scheme, can you actually do, say:
>>
>>   guix environment --ad-hoc rust rust-foo rust-bar
>>
>> and then (pseudo syntax):
>>
>>   rustc mystuff.rust -lfoo -lbar
>>
>> ?
> 
> Ignoring that we don't actually install the libraries here's a copy of
> a check phase I wrote trying to use installed libraries. I think in the
> end it did consume the rlibs as requested, but it never did link to
> them, it just absorbed them into the final binary.
> 
> (replace 'check
>   (lambda* (#:key inputs #:allow-other-keys)
>     (let ((ndarray    (assoc-ref inputs "ndarray"))
>           (rand       (assoc-ref inputs "rand"))
>           (rayon      (assoc-ref inputs "rayon"))
>           (serde      (assoc-ref inputs "serde"))
>           (serde-json (assoc-ref inputs "serde-json"))
>           (structopt  (assoc-ref inputs "structopt")))
>       (invoke "rustc" "--edition=2018" "--crate-name" "qtlreaper"
>               "src/lib.rs"
>               "-C" "opt-level=3"
>               "--test" "-C" "metadata=test" "-C" "extra-filename=-test"
>               "--out-dir" "target/release/deps"
>               "--extern" (string-append "ndarray=" ndarray
> "/lib/libndarray.rlib")
>               "--extern" (string-append "rand=" rand "/lib/librand.rlib")
>               "--extern" (string-append "rayon=" rayon "/lib/librayon.rlib")
>               "--extern" (string-append "serde=" serde "/lib/libserde.rlib")
>               "--extern" (string-append "serde_json=" serde-json
> "/lib/libserde_json.rlib")
>               "--extern" (string-append "structopt=" structopt
> "/lib/structopt.rlib")
>               "--cap-lints" "allow" "-C" "rpath" "-C" "prefer-dynamic")
>       (invoke "rustc" "--edition=2018" "--crate-name" "qtlreaper"
>               "src/main.rs"
>               "-C" "opt-level=3"
>               "--test" "-C" "metadata=test" "-C" "extra-filename=-test"
>               "--out-dir" "target/release/deps"
>               "--extern" "qtlreaper=target/release/deps/libqtlreaper-test.rlib"
>               "--extern" (string-append "ndarray=" ndarray
> "/lib/libndarray.rlib")
>               "--extern" (string-append "rand=" rand "/lib/librand.rlib")
>               "--extern" (string-append "rayon=" rayon "/lib/librayon.rlib")
>               "--extern" (string-append "serde=" serde "/lib/libserde.rlib")
>               "--extern" (string-append "serde_json=" serde-json
> "/lib/libserde_json.rlib")
>               "--extern" (string-append "structopt=" structopt
> "/lib/structopt.rlib")
>               "--cap-lints" "allow" "-C" "rpath" "-C" "prefer-dynamic")
>       (invoke "./target/release/deps/qtlreaper-test")
>       (invoke "rustdoc" "--edition=2018" "--test" "src/lib"
> "--crate-name" "qtlreaper"
>               "--extern" "qtlreaper=target/release/deps/libqtlreaper-test.rlib"
>               "--extern" (string-append "ndarray=" ndarray "/libndarray.rlib")
>               "--extern" (string-append "rand=" rand "/librand.rlib")
>               "--extern" (string-append "rayon=" rayon "/lib/librayon.rlib")
>               "--extern" (string-append "serde=" serde "/lib/libserde.rlib")
>               "--extern" (string-append "serde_json=" serde-json
> "/lib/libserde_json.rlib")
>               "--extern" (string-append "structopt=" structopt
> "/lib/structopt.rlib"))))))))
> 
>>
>> Thanks,
>> Ludo’.
> 
> I was thinking of making the build phases for the cargo-build-system use
> the --verbose flag, I found it very informative to see what it was doing
> along the way.

Re: Overhauling the cargo-build-system

[John Soo]

Hi all,

I agree with this:

> The whole point of package management is that you can use module
> building blocks. By having to specify the sub-dependencies in a top
> level definition kinda breaks the whole modular thing.

It is quite frustrating to have to specify transitive dependencies at the top level.  I’m not sure what else is to be done, though.

- John

Re: Overhauling the cargo-build-system

[Martin Becze]

> It is quite frustrating to have to specify transitive dependencies at
> the top level.  I’m not sure what else is to be done, though.

How about having something  like "propagated-input" for rust source
dependencies?  The top level package could use the source deps as
development-inputs so the source files wouldn't end up polluting the
environment. Would that make sense?

Re: Overhauling the cargo-build-system

[Ludovic Courtès]

Hello!

Martin Becze <mjbecze@riseup.net> skribis:

> Sorry for digging up and old issue, but i just saw commit
> 86e443c71d4d19e6f80cad9ca15b9c3a301c738c
>
>> It makes for a very large package definition, but we
> wouldn't have to ensure thousands of other rust libraries built so we
>
> The whole point of package management is that you can use module
> building blocks. By having to specify the sub-dependencies in a top
> level definition kinda breaks the whole modular thing. In commit
> 86e443c71d4d19e6f80cad9ca15b9c3a301c738c all the inputs got removed for
> all the libraries. So if I'm trying to use guix as package manager for a
> rust project and I want to use one of the rust libraries in
> crates-io.scm, how am i suppose to do this? I can't just include it as
> an input to my project because now I have to look up all of it
> dependencies as well?

I agree that removing all the dependencies from Rust packages feels
wrong.

What I would have liked is to somehow replace the #:cargo-inputs
argument (which is build-system-specific and thus “opaque”) with regular
‘native-inputs’ or ‘inputs’ field.

I know it’s not that easy with Rust and Cargo, I just never manage to
fully grasp why :-), but at least that should be our horizon IMO.

WDYT, Efraim, Martin, and other Rusty people?  :-)

Thanks,
Ludo’.

Re: Overhauling the cargo-build-system

[Martin Becze]

> What I would have liked is to somehow replace the #:cargo-inputs
> argument (which is build-system-specific and thus “opaque”) with regular
> ‘native-inputs’ or ‘inputs’ field.

That would be lovely! I always wondered why "#:cargo-inputs" existed.

Re: Overhauling the cargo-build-system

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3278 bytes --]

On Sat, Nov 16, 2019 at 10:33:32PM +0100, Ludovic Courtès wrote:
> Hello!
> 
> Martin Becze <mjbecze@riseup.net> skribis:
> 
> > Sorry for digging up and old issue, but i just saw commit
> > 86e443c71d4d19e6f80cad9ca15b9c3a301c738c
> >
> >> It makes for a very large package definition, but we
> > wouldn't have to ensure thousands of other rust libraries built so we
> >
> > The whole point of package management is that you can use module
> > building blocks. By having to specify the sub-dependencies in a top
> > level definition kinda breaks the whole modular thing. In commit
> > 86e443c71d4d19e6f80cad9ca15b9c3a301c738c all the inputs got removed for
> > all the libraries. So if I'm trying to use guix as package manager for a
> > rust project and I want to use one of the rust libraries in
> > crates-io.scm, how am i suppose to do this? I can't just include it as
> > an input to my project because now I have to look up all of it
> > dependencies as well?
> 
> I agree that removing all the dependencies from Rust packages feels
> wrong.
> 
> What I would have liked is to somehow replace the #:cargo-inputs
> argument (which is build-system-specific and thus “opaque”) with regular
> ‘native-inputs’ or ‘inputs’ field.
> 
> I know it’s not that easy with Rust and Cargo, I just never manage to
> fully grasp why :-), but at least that should be our horizon IMO.
> 
> WDYT, Efraim, Martin, and other Rusty people?  :-)
> 

The big problems are the recursive dependencies, the partial
dependencies and the versioning. There are some that are easy to figure
out, serde always needs serde-derive, winapi always needs the
winapi-[i686|x86_64] crates, rayon -> rayon-core, etc. The real problem
is that each crate has a number of features, and only needs certain
inputs based on those features. If we include too few then we're stuck
specifying dependencies anyway. If we include all of them then we're
stuck packaging way more versions of things than we actually need. This
by itself might not be a problem; we could just say "this is what the
importer gave us based on upstream, if you're missing something you're
on your own to add it."

I have one package that I was working on, a custom rust package (which
is unfortunately still missing a license file). In the end I needed
about 70 crates. Before the change I had it building with about 200
packaged crates but the file itself failed to compile with a 'guix pull'
because of all the still unpackaged crates which were referenced by
other packaged crates. I still haven't reached the bottom of that tree,
and I have more than 500 crates that I haven't upstreamed yet.

The current situation definitely could be better. Without opening the
repo there's no way to know which crates are packaged. When updating a
crate there's no way to know which packages will be rebuilt.

I suppose one way to work around some of the issues is to make it so
that the crates "build" by copying the source to %out/share/guix-vendor
or something.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Overhauling the cargo-build-system

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> The big problems are the recursive dependencies, the partial
> dependencies and the versioning. There are some that are easy to figure
> out, serde always needs serde-derive, winapi always needs the
> winapi-[i686|x86_64] crates, rayon -> rayon-core, etc.

Do you mean that the crate importer returns partial dependency info?
That alone would be OK, many importers return incomplete dependency
info, but we can fill that out when making the package.

> I suppose one way to work around some of the issues is to make it so
> that the crates "build" by copying the source to %out/share/guix-vendor
> or something.

So the core issue is that there’s nothing like shared libraries, is that
correct?  This, in turn, means that there’s nothing to actually build,
and thus a crate doesn’t really map to a package in the usual sense of
the word, right?

In that case, what you suggest (copying the source in the package
output) sounds like it could work.  It would be an improvement over what
we have now: the package graph would correspond to the crate graph.

Thanks,
Ludo’.

Re: Overhauling the cargo-build-system

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2181 bytes --]

On Sun, Nov 17, 2019 at 10:22:21PM +0100, Ludovic Courtès wrote:
> Hi,
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > The big problems are the recursive dependencies, the partial
> > dependencies and the versioning. There are some that are easy to figure
> > out, serde always needs serde-derive, winapi always needs the
> > winapi-[i686|x86_64] crates, rayon -> rayon-core, etc.
> 
> Do you mean that the crate importer returns partial dependency info?
> That alone would be OK, many importers return incomplete dependency
> info, but we can fill that out when making the package.

It returns incomplete information. The way our packaging works it
says that it wants the latest version of a dependency, but often it's
looking for a different version.

> 
> > I suppose one way to work around some of the issues is to make it so
> > that the crates "build" by copying the source to %out/share/guix-vendor
> > or something.
> 
> So the core issue is that there’s nothing like shared libraries, is that
> correct?  This, in turn, means that there’s nothing to actually build,
> and thus a crate doesn’t really map to a package in the usual sense of
> the word, right?

We can build it and run the tests, but it's not entirely useful. I have
a blog post I'm working on, but basically if binary A wants B, and
library B wants C, D, and E, A might only want B with features from D.
So if B doesn't build because of C it doesn't prevent A from building.

Plus logistically it doesn't really work to link libraries to binaries
in the manner we're used to.

> 
> In that case, what you suggest (copying the source in the package
> output) sounds like it could work.  It would be an improvement over what
> we have now: the package graph would correspond to the crate graph.
> 

It would also make it easier to patch the sources to remove vendored C
libraries or patch Cargo.toml files.

> Thanks,
> Ludo’.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Overhauling the cargo-build-system

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Sun, Nov 17, 2019 at 10:22:21PM +0100, Ludovic Courtès wrote:
>> Hi,
>> 
>> Efraim Flashner <efraim@flashner.co.il> skribis:
>> 
>> > The big problems are the recursive dependencies, the partial
>> > dependencies and the versioning. There are some that are easy to figure
>> > out, serde always needs serde-derive, winapi always needs the
>> > winapi-[i686|x86_64] crates, rayon -> rayon-core, etc.
>> 
>> Do you mean that the crate importer returns partial dependency info?
>> That alone would be OK, many importers return incomplete dependency
>> info, but we can fill that out when making the package.
>
> It returns incomplete information. The way our packaging works it
> says that it wants the latest version of a dependency, but often it's
> looking for a different version.

Maybe we can fix “guix import crate” then, because the info provided by
the HTTP API is rather detailed (see (guix import crate), which exposes
some of that as Scheme records).

>> > I suppose one way to work around some of the issues is to make it so
>> > that the crates "build" by copying the source to %out/share/guix-vendor
>> > or something.
>> 
>> So the core issue is that there’s nothing like shared libraries, is that
>> correct?  This, in turn, means that there’s nothing to actually build,
>> and thus a crate doesn’t really map to a package in the usual sense of
>> the word, right?
>
> We can build it and run the tests, but it's not entirely useful. I have
> a blog post I'm working on, but basically if binary A wants B, and
> library B wants C, D, and E, A might only want B with features from D.
> So if B doesn't build because of C it doesn't prevent A from building.

Hmm!

> Plus logistically it doesn't really work to link libraries to binaries
> in the manner we're used to.

I’ll wait for your blog post to see what you mean by “doesn’t really
work”.  :-)

Thanks for explaining,
Ludo’.

Re: Overhauling the cargo-build-system

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 9400 bytes --]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> What I would have liked is to somehow replace the #:cargo-inputs
> argument (which is build-system-specific and thus “opaque”) with regular
> ‘native-inputs’ or ‘inputs’ field.

That would be nice.  However, it doesn't seem possible to express
Cargo's "dependencies" and "dev-dependencies" concepts using Guix's
current package DSL.

Consider the proc-macro2 and quote crates.  We added these two crates in
commit 2444abd9c124cc55f8f19a0462e06a2094f25a9d, in the same patch
series where we added #:cargo-inputs and #:cargo-development-inputs:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35318

Here is the Cargo.toml file for proc-macro2:

  https://github.com/alexcrichton/proc-macro2/blob/master/Cargo.toml

  [dev-dependencies]
  quote = { version = "1.0", default_features = false }

And here is the Cargo.toml file for quote:

  https://github.com/dtolnay/quote/blob/master/Cargo.toml

  [dependencies]
  proc-macro2 = { version = "1.0", default-features = false }

Here is a diagram of their dependency relationship:

  +---------------+
  |     quote     | <+
  +---------------+  |
    |                |
    | dependencies   | dev-dependencies
    v                |
  +---------------+  |
  |  proc-macro2  | -+
  +---------------+

To Cargo, this cycle is not a problem, since "dev-dependencies" are
treated differently from "dependencies":

  https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html

  "Dev-dependencies are not used when compiling a package for building,
  but are used for compiling tests, examples, and benchmarks.

  These dependencies are not propagated to other packages which depend
  on this package."

The reason proc-macro2 declares a "dev-dependency" on quote is because
proc-macro2 uses quote in its doc tests:

  https://github.com/alexcrichton/proc-macro2/blob/e82e8571460a0a0e00f52f011a74a5e0359acf3e/src/lib.rs#L785

This relationship between proc-macro2 and quote cannot be readily
expressed using the current package DSL in Guix.  If you try to model
"dependencies" and "dev-dependencies" as "inputs" (or "native-inputs",
or some combination of the two), Guix will fail due to the cycle.

Presumably, proc-macro2 just needs the source of quote (and the source
of proc-macro2's other dependency, unicode-xid).  When Cargo builds
proc-macro2, it will take care of building quote and making it available
during proc-macro2's tests.  Guix "just" needs to provide proc-macro2
with the quote source.  You might think this poses a bootstrapping
problem for Cargo, but I guess it doesn't.  As long as Cargo has the
source for proc-macro2, quote, and unicode-xid, I guess it can build
proc-macro2 and quote in any order.

Unless we missed something in our discussion of patch 35318, there is no
easy way to express the relationship between proc-macro2 and quote
without changing (or mis-using) the existing package DSL.  In the same
way that the package DSL introduced "native-inputs" and "inputs" as
concepts to facilitate cross-compilation, one way to solve this problem
might be to introduce a new concept to the package DSL that would make
it possible for Guix to express this kind of relationship correctly.

However, in the discussion of patch 35318, everyone (myself included)
seemed opposed to changing the package DSL if we didn't have to.  For
example, in response to an earlier version of the patch series in which
we tried to map "dependencies" and "dev-dependencies" onto the "inputs"
and "native-inputs" concepts (which was probably an abuse of the package
DSL, since "native-inputs" is a cross-compilation concept), you said: "I
don't understand yet why you change the role of 'inputs' compared to how
it is in the rest of Guix."  Ultimately, we decided not to modify the
package DSL or the meaning of "inputs".  Instead, we decided to encode
the necessary information about dependencies in the cargo-build-system's
arguments.  That is how we arrived at #:cargo-inputs and
#:cargo-development-inputs.

By introducing #:cargo-inputs and #:cargo-development-inputs as package
arguments to the cargo-build-system, we were able to solve the cyclic
dependency problem in one specific way.  Perhaps there are better ways.
I agree it would be nice if it were integrated into the package DSL.  I
think that changing the package DSL to suit our needs might work, but
I'm not sure how to change it without making it too Cargo-specific.

> I know it’s not that easy with Rust and Cargo, I just never manage to
> fully grasp why :-), but at least that should be our horizon IMO.

Well, you're not alone!  I'm not (yet!) an expert in Rust, but I find
these problems difficult to understand, too.  Cyclic dependencies are
just one issue.  There are other problems, too.

One problem that Efraim has mentioned is that every crate wants all of
its sources to be available at build time.  It's as if each crate wants
all of its source crates (and all of their source crates, transitively)
to be propagated into the build environment, even if not all of them are
used.

Another problem, which Efraim has also mentioned, is that it seems hard
to "cache" the result of building a crate.  So every crate wants to
build its transitive closure of dependencies from scratch, every time.
In a traditional GNU/Linux system, I guess cargo must cache the results
of these builds somehow, but I don't know how.  In the case of C
libraries, we can just produce .so files that other builds can receive
as inputs, and all is well...but in Rust, it seems hard to do something
like that.  I have to admit, I don't know a lot about this, but based on
what I've heard, it sounds like we would basically have to re-implement
a lot of what Cargo is already doing, in order to get the behavior we
want.  Maybe that's the right path; I don't know.

Ludovic Courtès <ludo@gnu.org> writes:

>> I suppose one way to work around some of the issues is to make it so
>> that the crates "build" by copying the source to %out/share/guix-vendor
>> or something.
>
> So the core issue is that there’s nothing like shared libraries, is that
> correct?  This, in turn, means that there’s nothing to actually build,
> and thus a crate doesn’t really map to a package in the usual sense of
> the word, right?
>
> In that case, what you suggest (copying the source in the package
> output) sounds like it could work.  It would be an improvement over what
> we have now: the package graph would correspond to the crate graph.

Yes, you could install the source into the "out" output, or to a
separate output such as "src".  You could also define a bunch of
"proc-macro2-src" and "quote-src" packages that only build the source.
These possibilities sound similar to the plan we were originally
considering.  We discussed that plan here:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35155

In short, we hoped to build a crate's source and install it (the source)
into a specific output.  Then for any given crate we want to build, we
would add other crates to its inputs, and the cargo-build-system would
take care of populating the vendor directory with the transitive closure
of source found in the inputs.  However, we couldn't implement that plan
because Guix's current DSL doesn't work with the cycles introduced by
Cargo's "dependencies" and "dev-dependencies".

Perhaps if we defined all crates as "proc-macro2-src" and "quote-src"
etc., we could then define a crate that builds an actual artifact (e.g.,
ripgrep) by dropping the "src" suffix.  So for ripgrep, we'd have
"ripgrep-src" and also "ripgrep".  The former would just copy the source
into the output, and the latter ("ripgrep") would list the former as an
input in order to actually build the program.  In this model, I guess
every "*-src" package would have no inputs and just one output.  I guess
any package that produces an artifact, for example the "ripgrep"
package, would list a bunch of "*-src" packages as inputs: one for every
crate in the transitive closure of dependencies and dev-dependencies of
the ripgrep crate.  That might solve the problem of cyclic dependencies,
and it might reduce (but not eliminate) the amount of excessive building
performed by cargo-build-system.  However, it would make some package
definitions large, it would introduce duplication of inputs across
packages that need the same "*-src" inputs, and it would create a lot of
"*-src" packages.  On the plus side, tools like "guix graph" would work
as-is; currently, "guix graph" has not been taught to understand
#:cargo-inputs and #:cargo-development-inputs for cargo-build-system
packages.

Maybe that really is a better way.  If I'm not mistaken, this seems to
be the direction Efraim has begun to take things with commit
86e443c71d4d19e6f80cad9ca15b9c3a301c738c.  Thank you for taking
initiative to try to improve things, Efraim!  I'm in favor of
experimenting with it to see if it works out.  If it proves to be useful
and we want to stick with it, then we should consider removing the logic
from cargo-build-system that implemented the #:cargo-inputs and
#cargo-development-inputs arguments.

In the end, I just want to be able to use Rust on Guix!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Overhauling the cargo-build-system

[Martin Becze]

On 2019-12-09 04:45, Chris Marusich wrote:

> input in order to actually build the program.  In this model, I guess
> every "*-src" package would have no inputs and just one output.  I guess
> any package that produces an artifact, for example the "ripgrep"
> package, would list a bunch of "*-src" packages as inputs: one for every
> crate in the transitive closure of dependencies and dev-dependencies of
> the ripgrep crate.  That might solve the problem of cyclic dependencies,
> and it might reduce (but not eliminate) the amount of excessive building
> performed by cargo-build-system.  However, it would make some package
> definitions large, it would introduce duplication of inputs across
> packages that need the same "*-src" inputs, and it would create a lot of
> "*-src" packages.  On the plus side, tools like "guix graph" would work
> as-is; currently, "guix graph" has not been taught to understand
> #:cargo-inputs and #:cargo-development-inputs for cargo-build-system
> packages.

I don't understand why just adding a (skip-build?) to the source rust
packages, dropping their cargo-dev-deps AND leaving the cargo-inputs for
all the rest of the dependencies wouldn't accomplish the same thing. The
thing I object to is having to specify the transient packages at the top
level, since that would be unusable for actually rust development. Is
there any problem with the following example for rust source packages?

(define-public rust-home
  (package
    (name "rust-home")
    (version "0.5.1")
    (source
      (origin
        (method url-fetch)
        (uri (crate-uri "home" version))
        (file-name
          (string-append name "-" version ".crate"))
        (sha256
          (base32
            "1a4wcnadw2sarmisb5bz7gs4qwnijalvbf5gf7kg0wdxyxa3jxd3"))))
    (build-system cargo-build-system)
    (arguments
      `(#:skip-build?
        #t
        #:cargo-inputs
        (("rust-scopeguard" ,rust-scopeguard)
         ("rust-winapi" ,rust-winapi))))
    (home-page "https://github.com/brson/home")
    (synopsis
      "Shared definitions of home directories")
    (description
      "Shared definitions of home directories")
    (license (list license:expat license:asl2.0))))

(define-public rust-winapi
  (package
    (name "rust-winapi")
    (version "0.3.8")
    (source
      (origin
        (method url-fetch)
        (uri (crate-uri "winapi" version))
        (file-name
          (string-append name "-" version ".crate"))
        (sha256
          (base32
            "1ii9j9lzrhwri0902652awifzx9fpayimbp6hfhhc296xcg0k4w0"))))
    (build-system cargo-build-system)
    (arguments
      `(#:skip-build?
        #t
        #:cargo-inputs
        (("rust-winapi-i686-pc-windows-gnu"
          ,rust-winapi-i686-pc-windows-gnu)
         ("rust-winapi-x86-64-pc-windows-gnu"
          ,rust-winapi-x86-64-pc-windows-gnu))))
    (home-page
      "https://github.com/retep998/winapi-rs")
    (synopsis
      "Raw FFI bindings for all of Windows API.")
    (description
      "Raw FFI bindings for all of Windows API.")
    (license #f)))


Also see patch 38465
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38465) for a complete
example. I would like too know if there is an problem with this patch
to, before I start packaging the other rust programs this way.

Thanks,
-Martin

Re: Overhauling the cargo-build-system

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> What I would have liked is to somehow replace the #:cargo-inputs
>> argument (which is build-system-specific and thus “opaque”) with regular
>> ‘native-inputs’ or ‘inputs’ field.
>
> That would be nice.  However, it doesn't seem possible to express
> Cargo's "dependencies" and "dev-dependencies" concepts using Guix's
> current package DSL.
>
> Consider the proc-macro2 and quote crates.  We added these two crates in
> commit 2444abd9c124cc55f8f19a0462e06a2094f25a9d, in the same patch
> series where we added #:cargo-inputs and #:cargo-development-inputs:
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35318
>
> Here is the Cargo.toml file for proc-macro2:
>
>   https://github.com/alexcrichton/proc-macro2/blob/master/Cargo.toml
>
>   [dev-dependencies]
>   quote = { version = "1.0", default_features = false }
>
> And here is the Cargo.toml file for quote:
>
>   https://github.com/dtolnay/quote/blob/master/Cargo.toml
>
>   [dependencies]
>   proc-macro2 = { version = "1.0", default-features = false }
>
> Here is a diagram of their dependency relationship:
>
>   +---------------+
>   |     quote     | <+
>   +---------------+  |
>     |                |
>     | dependencies   | dev-dependencies
>     v                |
>   +---------------+  |
>   |  proc-macro2  | -+
>   +---------------+
>
> To Cargo, this cycle is not a problem, since "dev-dependencies" are
> treated differently from "dependencies":
>
>   https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html
>
>   "Dev-dependencies are not used when compiling a package for building,
>   but are used for compiling tests, examples, and benchmarks.
>
>   These dependencies are not propagated to other packages which depend
>   on this package."
>
> The reason proc-macro2 declares a "dev-dependency" on quote is because
> proc-macro2 uses quote in its doc tests:
>
>   https://github.com/alexcrichton/proc-macro2/blob/e82e8571460a0a0e00f52f011a74a5e0359acf3e/src/lib.rs#L785

I see.

> This relationship between proc-macro2 and quote cannot be readily
> expressed using the current package DSL in Guix.  If you try to model
> "dependencies" and "dev-dependencies" as "inputs" (or "native-inputs",
> or some combination of the two), Guix will fail due to the cycle.

True, but we have the same problem with many non-Rust packages, which we
address in various way—e.g., via an intermediate “-minimal” variant.

> Presumably, proc-macro2 just needs the source of quote (and the source
> of proc-macro2's other dependency, unicode-xid).  When Cargo builds
> proc-macro2, it will take care of building quote and making it available
> during proc-macro2's tests.  Guix "just" needs to provide proc-macro2
> with the quote source.  You might think this poses a bootstrapping
> problem for Cargo, but I guess it doesn't.  As long as Cargo has the
> source for proc-macro2, quote, and unicode-xid, I guess it can build
> proc-macro2 and quote in any order.

In that case, would it work to turn “dev-dependencies” into dependencies
on the source rather than on the package?

> Unless we missed something in our discussion of patch 35318, there is no
> easy way to express the relationship between proc-macro2 and quote
> without changing (or mis-using) the existing package DSL.  In the same
> way that the package DSL introduced "native-inputs" and "inputs" as
> concepts to facilitate cross-compilation, one way to solve this problem
> might be to introduce a new concept to the package DSL that would make
> it possible for Guix to express this kind of relationship correctly.
>
> However, in the discussion of patch 35318, everyone (myself included)
> seemed opposed to changing the package DSL if we didn't have to.  For
> example, in response to an earlier version of the patch series in which
> we tried to map "dependencies" and "dev-dependencies" onto the "inputs"
> and "native-inputs" concepts (which was probably an abuse of the package
> DSL, since "native-inputs" is a cross-compilation concept), you said: "I
> don't understand yet why you change the role of 'inputs' compared to how
> it is in the rest of Guix."  Ultimately, we decided not to modify the
> package DSL or the meaning of "inputs".  Instead, we decided to encode
> the necessary information about dependencies in the cargo-build-system's
> arguments.  That is how we arrived at #:cargo-inputs and
> #:cargo-development-inputs.

OK, thanks for the reminder.  :-)

I’d be curious to hear what Ivan and others think of
<https://issues.guix.gnu.org/issue/35318> in hindsight.

> By introducing #:cargo-inputs and #:cargo-development-inputs as package
> arguments to the cargo-build-system, we were able to solve the cyclic
> dependency problem in one specific way.  Perhaps there are better ways.
> I agree it would be nice if it were integrated into the package DSL.  I
> think that changing the package DSL to suit our needs might work, but
> I'm not sure how to change it without making it too Cargo-specific.

Still, the notion of dependency definitely exists in Cargo, so it would
seem natural to use ‘inputs’ or ‘native-inputs’ to express that.

I realize my understanding of Crates is still too limited, but I think
our goal should be to somehow have something that’s closer to normal
<package> objects.  We’ll probably still need Cargo-specific extensions,
but it’d be nice if the <package> graph for Crates was meaningful.

Also, ‘guix import crates’ has all the info so we can always tweak it to
generate something different.

Thanks!

Ludo’.

Re: Overhauling the cargo-build-system

[Ludovic Courtès]

Hi Martin,

Martin Becze <mjbecze@riseup.net> skribis:

> Also see patch 38465
> (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38465) for a complete
> example. I would like too know if there is an problem with this patch
> to, before I start packaging the other rust programs this way.

It’d be great if the people in Cc: could comment.  :-)

Ludo’.

Re: Overhauling the cargo-build-system

[John Soo]

Hi all,

I am working on ripgrep and I was wondering if we could add a key to inputs for cargo inputs instead of using the arguments field.  Is there a downside to saying something like 

`(inputs
    (("rust-loom" ,rust-loom-0.2 #rust-build)
     ("rust-quickcheck" ,rust-quickcheck-0.9 #rust-dev))

Just a thought I had so I could be wrong. I certainly see the downside of not using the inputs fields the way they work everywhere else.

Thanks,

John