(not) testing Rust packages?!

(not) testing Rust packages?!

[Hartmut Goebel]

[-- Attachment #1.1: Type: text/plain, Size: 756 bytes --]

Hi,

I discovered that most packages in crates-io.scm are not build. I
understand that this is done since they do not produce any useful output.

But as an side-effect, the packages are not tested either - which might
leave issues undiscovered.

Is this intended?

My experience when packaging KDE libraries showed that is does make
sense to actually test the libraries to detect issues early. And there
are quite some possible issue in libraries, e.g. ladoing dynamic libs
from /usr/lib, searching executables in PATH or even worth in a
hard-coded path.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: (not) testing Rust packages?!

[Martin Becze]

Part of the reason is that bringing tests for a given library can bring 
in a massive amount of dependencies. So testing only the top level 
package seemed like the way to go. Maybe we could add testing when 
encountering problematic rust packages that break often?

On 1/25/20 8:38 AM, Hartmut Goebel wrote:
> Hi,
> 
> I discovered that most packages in crates-io.scm are not build. I
> understand that this is done since they do not produce any useful output.
> 
> But as an side-effect, the packages are not tested either - which might
> leave issues undiscovered.
> 
> Is this intended?
> 
> My experience when packaging KDE libraries showed that is does make
> sense to actually test the libraries to detect issues early. And there
> are quite some possible issue in libraries, e.g. ladoing dynamic libs
> from /usr/lib, searching executables in PATH or even worth in a
> hard-coded path.
> 

Re: (not) testing Rust packages?!

[John Soo]

Hi Hartmut and Martin,

I think it makes sense to run tests now.

> Part of the reason is that bringing tests for a given library can bring in a massive amount of dependencies.

I think that we are getting close to having complete dependencies for most rust packages we have and most are declared in the package definition. 

Furthermore since most rust libraries we have are not executables, we could still skip the build and run the tests I think. Aren’t the two phases completely separate for cargo?

Other downsides I see for not skipping the build are really increasing the store size.  Would skipping builds but still running tests increase the store size at all?

I like the idea of having tests, too.  Plus I’d like to see the cargo build system come closer to the standard package definition.

John

Re: (not) testing Rust packages?!

[Martin Becze]

On 1/25/20 11:46 AM, John Soo wrote:
> Hi Hartmut and Martin,
> 
> I think it makes sense to run tests now.
> 
>> Part of the reason is that bringing tests for a given library can bring in a massive amount of dependencies.
> 
> I think that we are getting close to having complete dependencies for most rust packages we have and most are declared in the package definition.

Yeah really good point there (and good work on getting all those pkgs in!)

> Furthermore since most rust libraries we have are not executables, we could still skip the build and run the tests I think. Aren’t the two phases completely separate for cargo?

Yes, will can skip the build and just test in the (cargo-build-system) 
but `cargo test` will cause most everything to build anyways i believe.


> I like the idea of having tests, too.  Plus I’d like to see the cargo build system come closer to the standard package definition.


agreed!

Re: (not) testing Rust packages?!

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1521 bytes --]

On Sat, Jan 25, 2020 at 08:46:48AM -0800, John Soo wrote:
> Hi Hartmut and Martin,
> 
> I think it makes sense to run tests now.
> 
> > Part of the reason is that bringing tests for a given library can bring in a massive amount of dependencies.
> 
> I think that we are getting close to having complete dependencies for most rust packages we have and most are declared in the package definition. 
> 
> Furthermore since most rust libraries we have are not executables, we could still skip the build and run the tests I think. Aren’t the two phases completely separate for cargo?
> 
> Other downsides I see for not skipping the build are really increasing the store size.  Would skipping builds but still running tests increase the store size at all?
> 
> I like the idea of having tests, too.  Plus I’d like to see the cargo build system come closer to the standard package definition.
> 

In addition, since we don't actually need any of the results of the
crates, it does provide a quick way to notice if we've provided the
wrong version of a dependency. I also noticed that we're missing
rust-average-0.9 which is definitely needed for rust-rand-0.6 and I'm
sure others.

If we enable building and tests and are lenient with skipping tests,
that alone would be an improvement.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: (not) testing Rust packages?!

[Andreas Rottmann]

Martin Becze <mjbecze@riseup.net> writes:

> Yeah really good point there (and good work on getting all those pkgs
> in!)
>
Being a newcomer to guix, I must say I'm really amazed about the state
of Rust packaging!

>> Furthermore since most rust libraries we have are not executables, we
>> could still skip the build and run the tests I think. Aren’t the two
>> phases completely separate for cargo?
>
> Yes, will can skip the build and just test in the (cargo-build-system)
> but `cargo test` will cause most everything to build anyways i
> believe.
>
`cargo test` will always build the crate a second time, even if `cargo
build` already ran. This is due to the config attribute `test` being set
(similar a to C preprocessor #define), and thus the actual code being
compiled may be different. In addition, `cargo test` will bring in the
`dev-dependencies`, which can indeed be _much_ more massive than the
regular `dependencies`.

Re: (not) testing Rust packages?!

[John Soo]

Hey Andreas,

> `cargo test` will always build the crate a second time, even if `cargo
> build` already ran. This is due to the config attribute `test` being set
> (similar a to C preprocessor #define), and thus the actual code being
> compiled may be different.

Just to make sure, does that mean we can safely run tests without
having unwanted store outputs?
I can see that maybe the test compile step might not catch everything
or compile the whole package, but wouldn't you say that some testing
is better than none at all?

> In addition, `cargo test` will bring in the
> `dev-dependencies`, which can indeed be _much_ more massive than the
> regular `dependencies`.

Can you provide an example? The largest sets of test dependencies I
can find are using the insta library which we have packaged already.
Do you think the number of dependencies at the top level package was
due to the old style of hiding crates packages?
Also we have quickcheck and criterion packaged now which should also
reduce the required work to create a package now.

Thanks,

John

Re: (not) testing Rust packages?!

[Andreas Rottmann]

John Soo <jsoo1@asu.edu> writes:

> Hey Andreas,
>
>> `cargo test` will always build the crate a second time, even if `cargo
>> build` already ran. This is due to the config attribute `test` being set
>> (similar a to C preprocessor #define), and thus the actual code being
>> compiled may be different.
>
> Just to make sure, does that mean we can safely run tests without
> having unwanted store outputs?
>
I'm a new to Guix, and am not sure what you mean by "safely" and
"unwanted store outputs". Running `cargo test` takes the crate source,
and the closure of any `dependencies` and `dev-dependencies`, and
produces no real artifacts that make sense to put in the store, as far
as I can see. The only noteworthy artifact is the stdout/stderr ouput
produced, as well as the exit status, but I guess that's not relevant to
the store.

> I can see that maybe the test compile step might not catch everything
> or compile the whole package, but wouldn't you say that some testing
> is better than none at all?
>
Having tests run would be great: it's a service to the Rust ecosystem,
and can also help catch issues in packaging, as was already pointed
out. How likely catching issues is depends on the testsuite quality of
the crate in question, but that's not at all an argument against running
tests, of course!

>> In addition, `cargo test` will bring in the
>> `dev-dependencies`, which can indeed be _much_ more massive than the
>> regular `dependencies`.
>
> Can you provide an example? [...]
>
This may have come across wrong: I just wanted to point out that it may
mean more packaging effort, but it seems Guix is in a good position here
already.

Re: (not) testing Rust packages?!

[John Soo]

Hi Andreas,

> On Jan 29, 2020, at 11:01 AM, Andreas Rottmann <mail@r0tty.org> wrote:
> 
> I'm a new to Guix, and am not sure what you mean by "safely" and
> "unwanted store outputs". Running `cargo test` takes the crate source,
> and the closure of any `dependencies` and `dev-dependencies`, and
> produces no real artifacts that make sense to put in the store, as far
> as I can see. The only noteworthy artifact is the stdout/stderr ouput
> produced, as well as the exit status, but I guess that's not relevant to
> the store.

Oh I could see how “safely” would be confusing. I meant that we would not populate anything in the store as you suggest.

> Having tests run would be great: it's a service to the Rust ecosystem,
> and can also help catch issues in packaging, as was already pointed
> out. How likely catching issues is depends on the testsuite quality of
> the crate in question, but that's not at all an argument against running
> tests, of course!

Agreed.

> This may have come across wrong: I just wanted to point out that it may
> mean more packaging effort, but it seems Guix is in a good position here
> already.

Ah yeah I probably came across wrong, too. You are definitely right that there will be more packaging effort. I think at some point it should get easier once the popular test dependencies get in.

What do you think?

John

Re: (not) testing Rust packages?!

[Hartmut Goebel]

Am 31.01.20 um 19:50 schrieb John Soo:
>> On Jan 29, 2020, at 11:01 AM, Andreas Rottmann <mail@r0tty.org> wrote:
>>
>> I'm a new to Guix, and am not sure what you mean by "safely" and
>> "unwanted store outputs". Running `cargo test` takes the crate source,
>> and the closure of any `dependencies` and `dev-dependencies`, and
>> produces no real artifacts that make sense to put in the store, as far
>> as I can see. The only noteworthy artifact is the stdout/stderr ouput
>> produced, as well as the exit status, but I guess that's not relevant to
>> the store.
> Oh I could see how “safely” would be confusing. I meant that we would not populate anything in the store as you suggest.

If `cargo test` does not "install" any file - and this is what I would
expect it to do, since this is what `cargo install` - running `cargo
test` is save.

Neither the exit status nor the stderr/stdout output are effecting the
store.


> What do you think? 


Go ahead! :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |