Idea for packaging rust apps

Idea for packaging rust apps

[Murilo]

Hello, I hope this is the right place for this, apologies if it isn't. 

I'm working with a friend on a cargo importer that lowers the entry barrier 
and the maintainability costs for packaging rust apps in general, without 
sacrificing Guix dependency tracking and reproducibility of rust packages. 

When you get used to the tool, you can pretty much package rust apps with 
all the dependency chain very easily (I just packaged [1] texlab for my 
channel earlier this morning in less than 5 minutes, and i can easily 
update apps to the latest version in less than a minute). 

Progress is being tracked in [2] if anyone wants to check it out or 
contribute to it. It is currently missing a lot of features, but we hope 
to improve the user experience of the tool in the near future. 

It is a very simple tool, it essentially parses the Cargo.lock file and 
extracts all the relevant information for constructing the rust package 
definitions of all the cargo-inputs and the package itself, and outputs 
to stdio a guile module containing all the needed cargo-input chain as 
guix packages, with all the cargo-inputs self-contained and versions all 
sorted out for a working final package build. 

This way a packager only needs to focus on the actual package they are 
trying to build, instead of worrying about its cargo-inputs. 

Due to how cargo-inputs are organized in gnu/packages/crates-*.scm, and 
some current packaging guidelines for crates on Guix, we cannot simply 
contribute these self-contained packages generated directly from the 
Cargo.lock, thus requiring to use the guix crate importer and spending 
a lot of time fixing dependencies and worrying about other packages 
breaking in the process. 

I would like to propose some discussion around a better way of organizing 
the rust packages and its cargo-inputs in (gnu packages) for building 
rust apps that only need sources as cargo-inputs: 

1) Create a new directory at gnu/packages/rust/ in which a contributor 
can commit self-contained rust apps scm modules. 
2.1) Add a new module at gnu/packages/rust/my-rust-app-1.scm 
2.2) Add a new module at gnu/packages/rust/my-rust-app-2.scm 
3) All package definitions inside gnu/packages/rust/*.scm which are 
source-only (#:skip-build? #t) should not be exported. 
4.1) gnu/packages/crates-*.scm will not cease to exist, existing rust 
apps packages that have a Cargo.lock could gradually be migrated to the 
new organization 
4.2) libs which need to be built can still live in 
gnu/packages/crates-*.scm 
5) A (define-public my-rust-app-1 (@@ (gnu packages rust my-rust-app-1) 
my-rust-app-1)) or equivalent could be done in a (gnu packages category) 
module to export the rust app in the desired category. 
6) Unlike nix (which also parses the Cargo.lock in the build system), 
we don't lose the ability to make snippets for sources this way. 
7) For updating/maintaining a rust package defined this way, one would 
be able to simply re-run the guix tool, and replace the 
gnu/packages/rust/my-rust-app.scm file, only copying over the final 
relevant package definition for the rust app with its tweaks for building, 
and passing over the new cargo-inputs generated by the guix tool. 

I believe that by only changing the way things are organized and having 
a guix tool that generates self-contained package definitions from 
Cargo.lock, it would be possible to greatly improve the time required 
for contributing new rust apps packages and maintaining them on Guix. 

Things don't need to be the way I described here, these are just my 
initial thoughts after several failed attempts and wasted time trying 
to contribute rust apps to Guix, I'd like to discuss workarounds and if 
the benefits are greater than the disavantages for an approach like this. 

The tool we made works really well for packaging for our personal channels, 
I am very satisfied with how easy it is, and I think Guix could benefit a 
lot by adopting a similar approach. 

What am I missing here? Are there any disavantages to this approach? 
Anything that would break from it if adopted on Guix? 
Any questions or suggestions? 

Murilo 

[1] https://codeberg.org/look/saayix/commit/c7643943545d62baba80cccee1650ebf94362858 
[2] https://git.sr.ht/~look/cargo2guix

Re: Idea for packaging rust apps

[MSavoritias]

On 5/22/24 05:04, Murilo wrote:

> Hello, I hope this is the right place for this, apologies if it isn't.
>
> I'm working with a friend on a cargo importer that lowers the entry barrier
> and the maintainability costs for packaging rust apps in general, without
> sacrificing Guix dependency tracking and reproducibility of rust packages.
>
> When you get used to the tool, you can pretty much package rust apps with
> all the dependency chain very easily (I just packaged [1] texlab for my
> channel earlier this morning in less than 5 minutes, and i can easily
> update apps to the latest version in less than a minute).
>
> Progress is being tracked in [2] if anyone wants to check it out or
> contribute to it. It is currently missing a lot of features, but we hope
> to improve the user experience of the tool in the near future.
>
> It is a very simple tool, it essentially parses the Cargo.lock file and
> extracts all the relevant information for constructing the rust package
> definitions of all the cargo-inputs and the package itself, and outputs
> to stdio a guile module containing all the needed cargo-input chain as
> guix packages, with all the cargo-inputs self-contained and versions all
> sorted out for a working final package build.
>
> This way a packager only needs to focus on the actual package they are
> trying to build, instead of worrying about its cargo-inputs.
>
> Due to how cargo-inputs are organized in gnu/packages/crates-*.scm, and
> some current packaging guidelines for crates on Guix, we cannot simply
> contribute these self-contained packages generated directly from the
> Cargo.lock, thus requiring to use the guix crate importer and spending
> a lot of time fixing dependencies and worrying about other packages
> breaking in the process.
>
> I would like to propose some discussion around a better way of organizing
> the rust packages and its cargo-inputs in (gnu packages) for building
> rust apps that only need sources as cargo-inputs:
>
> 1) Create a new directory at gnu/packages/rust/ in which a contributor
> can commit self-contained rust apps scm modules.
> 2.1) Add a new module at gnu/packages/rust/my-rust-app-1.scm
> 2.2) Add a new module at gnu/packages/rust/my-rust-app-2.scm
> 3) All package definitions inside gnu/packages/rust/*.scm which are
> source-only (#:skip-build? #t) should not be exported.
> 4.1) gnu/packages/crates-*.scm will not cease to exist, existing rust
> apps packages that have a Cargo.lock could gradually be migrated to the
> new organization
> 4.2) libs which need to be built can still live in
> gnu/packages/crates-*.scm
> 5) A (define-public my-rust-app-1 (@@ (gnu packages rust my-rust-app-1)
> my-rust-app-1)) or equivalent could be done in a (gnu packages category)
> module to export the rust app in the desired category.
> 6) Unlike nix (which also parses the Cargo.lock in the build system),
> we don't lose the ability to make snippets for sources this way.
> 7) For updating/maintaining a rust package defined this way, one would
> be able to simply re-run the guix tool, and replace the
> gnu/packages/rust/my-rust-app.scm file, only copying over the final
> relevant package definition for the rust app with its tweaks for building,
> and passing over the new cargo-inputs generated by the guix tool.
>
> I believe that by only changing the way things are organized and having
> a guix tool that generates self-contained package definitions from
> Cargo.lock, it would be possible to greatly improve the time required
> for contributing new rust apps packages and maintaining them on Guix.
>
> Things don't need to be the way I described here, these are just my
> initial thoughts after several failed attempts and wasted time trying
> to contribute rust apps to Guix, I'd like to discuss workarounds and if
> the benefits are greater than the disavantages for an approach like this.
>
> The tool we made works really well for packaging for our personal channels,
> I am very satisfied with how easy it is, and I think Guix could benefit a
> lot by adopting a similar approach.
>
> What am I missing here? Are there any disavantages to this approach?
> Anything that would break from it if adopted on Guix?
> Any questions or suggestions?
>
> Murilo
>
> [1] https://codeberg.org/look/saayix/commit/c7643943545d62baba80cccee1650ebf94362858
> [2] https://git.sr.ht/~look/cargo2guix

Hey Murilo,


Thank you for taking the initiative to do something about the rust 
packaging situation in Guix. Which currently less than optimal.

I wanted to ask, are you also aware of the antioxidant effort? 
https://notabug.org/maximed/cargoless-rust-experiments

I was wondering of the differences since your build system seems to 
still be using cargo under the hood instead of rustc.


MSavoritias

Re: Idea for packaging rust apps

[Murilo]

Hi MSavoritias,

> I wanted to ask, are you also aware of the antioxidant effort? 
> https://notabug.org/maximed/cargoless-rust-experiments

I was not aware of it at all, thank you for enlightening me about the 
effort.

> I was wondering of the differences since your build system seems to 
> still be using cargo under the hood instead of rustc.

Yes, you are correct. It really does nothing special except making 
packaging of rust apps for the end user extremely easy. This approach 
still uses the cargo-build-system, it is not a replacement for it, 
thus carrying all the cargo (and cargo-build-system) flaws along with 
it.

The antioxidant-build-system, on the other hand, aims to mainly solve 
the inefficiencies of the cargo-build-system and cargo itself. After 
looking at some mail exchange on the antioxidant effort, it will be 
hopefully much nicer to make rust packages once the rust-build-system 
is merged, and it will be a much better approach for Guix than what I 
am currently suggesting.

I believe my approach with cargo2guix and transitively generating guix 
package definitions from Cargo.lock will mainly target user's Guix 
channels because of the simplicity of the packaging process. While it 
is better than what we currently have on Guix, it is nowhere ideal for 
Guix as the antioxidant effort is, because of CI and the amount of 
packages Guix has.

Thank you once again for making me aware of antioxidant and the future 
rust overhaul on Guix.

--
Best regards,
Murilo