Packaging a rust program with a lot of crates

Packaging a rust program with a lot of crates

[Paul Collignan]

Hi,

First of all I would like to apologize if the answers to my questions are obvious. This is a first for me in many areas. First mailing list, first time I want to contribute to a free project, first time I have to write something in a programming language, first time I use git, etc. I'm not a computer scientist at all. At best you could call me a GNU/Linux end user for some time, but only to consume, never to produce. I would like to contribute a little, and for that I want to start with guix.

So I discovered guix last week, and spent the last few days reading the documentation and playing with it. I would like to package a program that I use on my computer but which is not in the repositories. It turns out to be a program written in Rust, with lots of dependencies. If I were to copy/paste all of what guix import -r returns the patch would be over 3000 lines long.

I would like to know what are the best practices to adopt in this case. There are simple additions, updates, and additions with inheritance. I guess I shouldn't send a patch with all of this mixed up.

Should I group crates by "logical patches", say slicing by origin url (like a first patch with all related to microsoft/windows-rs), wait for that patch to be accepted, then send the next one? Or maybe I should send one patch per package update first, then one patch per package with inheritance, then one patch with all the simple additions?
Or a mixture of these two proposals?

Also, in this kind of case, I think that adding the program will take  weeks when you're a beginner like me. Did I miss something? For example, is it possible to automate package inheritance during an update to a major version of a crate, or does it have to be done by hand?

Last question, for my culture, is there a plan to "clean up" old packages and dependencies that are no longer used, or will the scm files grow indefinitely?

Thanks for your help,
Paul

Re: Packaging a rust program with a lot of crates

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 3511 bytes --]

Hi Paul!

> I'm not a computer scientist at all. At best you could call me a
> GNU/Linux end user for some time, but only to consume, never to
> produce. I would like to contribute a little, and for that I want to
> start with guix.

That's cool! Please keep in mind, tho that Guix is not an "easy" distro
for novice users. Be ready to spend even more time learning stuff :)

Having that said, the facilities Guix provides do allow for packages to
be created and maintained with relatively little time effort. I'm sure
learning will pay off!

> I would like to package a program that I use on my computer but which
> is not in the repositories. It turns out to be a program written in
> Rust, with lots of dependencies. If I were to copy/paste all of what
> guix import -r returns the patch would be over 3000 lines long.
> 
> I would like to know what are the best practices to adopt in this
> case. There are simple additions, updates, and additions with
> inheritance. I guess I shouldn't send a patch with all of this mixed
> up.

I'm also a novice when it comes to sending to guix-patches but I
believe the perfect approach in this case would be to:
- add each package with a separate git commit
- make sure the final package builds and works properly on your PC
- create a patch series comprising all the additions with `git
  format-patch` as described in the documentation
- describe in general what you did in series' cover letter that `git
  format-patch` generated for you
- send the cover letter to guix-patches mailing list with `git
  send-email` as described in the documentation; this will cause the
  debbugs instance to open a new bug
- wait for an email response from debbugs with your new bug's number
- send the rest of your patch series with one invokation of `git
  send-email --to=<BUG_NUMBER>@debbugs.gnu.org` as described in the
  documentation

This way the Guix committers will see all the package additions as
separate patches but grouped together in one debbugs issue. There's no
need to wait for one patch to be accepted before sending another in
this case.

> Also, in this kind of case, I think that adding the program will take
>  weeks when you're a beginner like me. Did I miss something? For
> example, is it possible to automate package inheritance during an
> update to a major version of a crate, or does it have to be done by
> hand?

I'm not sure what you mean by this question.

Anyway, you just touched one unfortunate truth — when you're a
beginner, any serious task can easily take weeks.

My suggestion is that you start with something easier first. Perhaps an
application that only has 1 or 2 dependencies?

> Last question, for my culture, is there a plan to "clean up" old
> packages and dependencies that are no longer used, or will the scm
> files grow indefinitely?

There might be some misunderstanding here. Guix does allow
1. for multiple versions of the same package to coexist
2. and for multiple versions of the same package to share most of the
   packaging code via inheritance.
However, the possibility 1. is only exercised for some strategic
packages like gcc. For casual packages, when upstream releases a new
version, some kind Guix contributor sends a patch that changes the
definition in the .scm file to now describe the new version. The old
version need not be explicitly deleted — its place is taken by the new
version :)

Good luck and happy hacking ^^
Wojtek

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Packaging a rust program with a lot of crates

[Paul Collignan]

Le Sat, Jul 15, 2023 at 04:06:56PM +0200, Wojtek Kosior a écrit :
> Hi Paul!
> 
> That's cool! Please keep in mind, tho that Guix is not an "easy" distro
> for novice users. Be ready to spend even more time learning stuff :)

Hi, thank you for your answer.
 
> This way the Guix committers will see all the package additions as
> separate patches but grouped together in one debbugs issue. There's no
> need to wait for one patch to be accepted before sending another in
> this case.

Looks like I missed the cookbook. All this is described in it, I will look into it seriously before asking another question.
Thank you for your summary which will make it easier for me to understand the steps to follow.
 
> Anyway, you just touched one unfortunate truth — when you're a
> beginner, any serious task can easily take weeks.
> My suggestion is that you start with something easier first. Perhaps an
> application that only has 1 or 2 dependencies?

Will do, for sure.

> > Last question, for my culture, is there a plan to "clean up" old
> > packages and dependencies that are no longer used, or will the scm
> > files grow indefinitely?
> 
> There might be some misunderstanding here. Guix does allow
> 1. for multiple versions of the same package to coexist
> 2. and for multiple versions of the same package to share most of the
>    packaging code via inheritance.
> However, the possibility 1. is only exercised for some strategic
> packages like gcc. For casual packages, when upstream releases a new
> version, some kind Guix contributor sends a patch that changes the
> definition in the .scm file to now describe the new version. The old
> version need not be explicitly deleted — its place is taken by the new
> version :)

It is a little bit different for Rust packages, as described in the documentation:
"In the rust ecosystem it is common for multiple incompatible versions of a package to be used at any given time, so all package definitions should have a versioned suffix. The versioned suffix is the left-most non-zero digit"
If I understand this correctly, one should keep all packages from 0.1 to 0.46 (for example, if the versions are numbered like this). But from 1.1 to 1.46 you only keep the last one.

> Good luck and happy hacking ^^

Thanks again.

I don't know why and how to explain it, but from a novice point of view guix is very appealing. Everything seems well organized, well explained, and it seems to be a very good framework to start learning.

Re: Packaging a rust program with a lot of crates

[Wojtek Kosior via]

[-- Attachment #1: Type: text/plain, Size: 2783 bytes --]

> > There might be some misunderstanding here. Guix does allow
> > 1. for multiple versions of the same package to coexist
> > 2. and for multiple versions of the same package to share most of the
> >    packaging code via inheritance.
> > However, the possibility 1. is only exercised for some strategic
> > packages like gcc. For casual packages, when upstream releases a new
> > version, some kind Guix contributor sends a patch that changes the
> > definition in the .scm file to now describe the new version. The old
> > version need not be explicitly deleted — its place is taken by the new
> > version :)  
> 
> It is a little bit different for Rust packages, as described in the documentation:
> "In the rust ecosystem it is common for multiple incompatible
> versions of a package to be used at any given time, so all package
> definitions should have a versioned suffix. The versioned suffix is
> the left-most non-zero digit" If I understand this correctly, one
> should keep all packages from 0.1 to 0.46 (for example, if the
> versions are numbered like this). But from 1.1 to 1.46 you only keep
> the last one.

OK, my bad 😅 Didn't know that

-- (sig_start)
website: https://koszko.org/koszko.html
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
follow me on Fediverse: https://friendica.me/profile/koszko/profile

♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
-- (sig_end)


On Sat, 15 Jul 2023 17:10:59 +0200 Paul Collignan <paul.collignan@aquilenet.fr> wrote:

> > There might be some misunderstanding here. Guix does allow
> > 1. for multiple versions of the same package to coexist
> > 2. and for multiple versions of the same package to share most of the
> >    packaging code via inheritance.
> > However, the possibility 1. is only exercised for some strategic
> > packages like gcc. For casual packages, when upstream releases a new
> > version, some kind Guix contributor sends a patch that changes the
> > definition in the .scm file to now describe the new version. The old
> > version need not be explicitly deleted — its place is taken by the new
> > version :)  
> 
> It is a little bit different for Rust packages, as described in the documentation:
> "In the rust ecosystem it is common for multiple incompatible versions of a package to be used at any given time, so all package definitions should have a versioned suffix. The versioned suffix is the left-most non-zero digit"
> If I understand this correctly, one should keep all packages from 0.1 to 0.46 (for example, if the versions are numbered like this). But from 1.1 to 1.46 you only keep the last one.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Packaging a rust program with a lot of crates

[wolf]

[-- Attachment #1: Type: text/plain, Size: 4108 bytes --]

On 2023-07-15 16:06:56 +0200, Wojtek Kosior via wrote:
> Hi Paul!
> 
> > I'm not a computer scientist at all. At best you could call me a
> > GNU/Linux end user for some time, but only to consume, never to
> > produce. I would like to contribute a little, and for that I want to
> > start with guix.
> 
> That's cool! Please keep in mind, tho that Guix is not an "easy" distro
> for novice users. Be ready to spend even more time learning stuff :)
> 
> Having that said, the facilities Guix provides do allow for packages to
> be created and maintained with relatively little time effort. I'm sure
> learning will pay off!
> 
> > I would like to package a program that I use on my computer but which
> > is not in the repositories. It turns out to be a program written in
> > Rust, with lots of dependencies. If I were to copy/paste all of what
> > guix import -r returns the patch would be over 3000 lines long.
> > 
> > I would like to know what are the best practices to adopt in this
> > case. There are simple additions, updates, and additions with
> > inheritance. I guess I shouldn't send a patch with all of this mixed
> > up.
> 
> I'm also a novice when it comes to sending to guix-patches but I
> believe the perfect approach in this case would be to:
> - add each package with a separate git commit
> - make sure the final package builds and works properly on your PC
> - create a patch series comprising all the additions with `git
>   format-patch` as described in the documentation
> - describe in general what you did in series' cover letter that `git
>   format-patch` generated for you
> - send the cover letter to guix-patches mailing list with `git
>   send-email` as described in the documentation; this will cause the
>   debbugs instance to open a new bug
> - wait for an email response from debbugs with your new bug's number
> - send the rest of your patch series with one invokation of `git
>   send-email --to=<BUG_NUMBER>@debbugs.gnu.org` as described in the
>   documentation
> 
> This way the Guix committers will see all the package additions as
> separate patches but grouped together in one debbugs issue. There's no
> need to wait for one patch to be accepted before sending another in
> this case.
> 
> > Also, in this kind of case, I think that adding the program will take
> >  weeks when you're a beginner like me. Did I miss something? For
> > example, is it possible to automate package inheritance during an
> > update to a major version of a crate, or does it have to be done by
> > hand?
> 
> I'm not sure what you mean by this question.
> 
> Anyway, you just touched one unfortunate truth — when you're a
> beginner, any serious task can easily take weeks.
> 
> My suggestion is that you start with something easier first. Perhaps an
> application that only has 1 or 2 dependencies?
> 
> > Last question, for my culture, is there a plan to "clean up" old
> > packages and dependencies that are no longer used, or will the scm
> > files grow indefinitely?
> 
> There might be some misunderstanding here. Guix does allow
> 1. for multiple versions of the same package to coexist
> 2. and for multiple versions of the same package to share most of the
>    packaging code via inheritance.
> However, the possibility 1. is only exercised for some strategic
> packages like gcc. For casual packages, when upstream releases a new
> version, some kind Guix contributor sends a patch that changes the
> definition in the .scm file to now describe the new version. The old
> version need not be explicitly deleted — its place is taken by the new
> version :)

Is there a way to mark a package as a dependency?  I can very well imagine that
new version of some golang program would require less dependencies, but it is
very easy to forget to remove the package definition.  Is there a reasonable way
to clean this up?

> 
> Good luck and happy hacking ^^
> Wojtek

W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging a rust program with a lot of crates

[Hartmut Goebel]

Am 15.07.23 um 17:10 schrieb Paul Collignan:
> It is a little bit different for Rust packages, as described in the documentation:
> "In the rust ecosystem it is common for multiple incompatible versions of a package to be used at any given time, so all package definitions should have a versioned suffix. The versioned suffix is the left-most non-zero digit"
> If I understand this correctly, one should keep all packages from 0.1 to 0.46 (for example, if the versions are numbered like this). But from 1.1 to 1.46 you only keep the last one.

This is correct. Anyhow, please keep in mind that the variable name for 
the 1.x package will stay "some-crate-1".

This follows the idea of semantic versioning, which rust follows.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |