Go Package with multiple subpackage

Go Package with multiple subpackage

[Superfly Johnson]

The Azure SDK for Go 
(https://github.com/Azure/azure-sdk-for-go/releases) has many 
sub-packages within the same directory and the guix import method won't 
work directly. I think the best solution for packaging the requirements 
for rclone would be to make the sub-packages individual guix packages 
using the url-fetch method instead of the git method. Each also depends 
on several sub-packages.

Thoughts?

Re: Go Package with multiple subpackage

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]

Hi,

Sorry for the delay,

On Sun, 22 Sep 2024 13:21:30 -0400
Superfly Johnson <superfly.johnson@yahoo.com> wrote:

> The Azure SDK for Go 
> (https://github.com/Azure/azure-sdk-for-go/releases) has many 
> sub-packages within the same directory and the guix import method
> won't work directly. I think the best solution for packaging the
> requirements for rclone would be to make the sub-packages individual
> guix packages using the url-fetch method instead of the git method.
> Each also depends on several sub-packages.
I had a similar issue with the matterbridge package which has about 500
dependencies that are not in Guix.

I verified most of the licenses for the dependencies with a combination
of recursive guix import and manually looking for the ones that weren't
detected.

And given the number of dependencies I was told that it was okay to
have them bundled in.

As I understand, packaging too many dependencies would create
complications for the maintenance.

Though the current situation is far from ideal as checking the license
of ~500 dependencies is also very time consuming and if problems
appears in newer releases it would be difficult to detect them.

Also note that we didn't know about the dependencies issues of
matterbridge when it got in, so maybe it played a role in the decisions
that was taken at the time.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Go Package with multiple subpackage

[Andreas Enge]

Am Fri, Oct 18, 2024 at 11:43:28PM +0200 schrieb Denis 'GNUtoo' Carikli:
> I verified most of the licenses for the dependencies with a combination
> of recursive guix import and manually looking for the ones that weren't
> detected.
> 
> And given the number of dependencies I was told that it was okay to
> have them bundled in.
> 
> As I understand, packaging too many dependencies would create
> complications for the maintenance.

Is that true? It looks opposite to the general Guix philosophy;
once you have invested all the work of checking the licenses, it would
seem a progress to submit the corresponding packages. But maybe Go is
special in that respect; it would be nice to have the Go team's opinion.

Andreas

Re: Go Package with multiple subpackage

[Samuel Christie via Development of GNU Guix and the GNU System distribution.]

Andreas Enge <andreas@enge.fr> writes: 
> Am Fri, Oct 18, 2024 at 11:43:28PM +0200 schrieb Denis 'GNUtoo' 
> Carikli: 
>> And given the number of dependencies I was told that it was 
>> okay to have them bundled in.
>>
>> As I understand, packaging too many dependencies would create 
>> complications for the maintenance.
>
> Is that true? It looks opposite to the general Guix philosophy; 
> once you have invested all the work of checking the licenses, it 
> would seem a progress to submit the corresponding packages.

I am slightly interested in this question as well.

On the one hand, it seems like taking reproducible builds 
seriously requires not only pinning all of the sources but also 
archival etc. Guix seems to handle that right now by absorbing 
dependencies as packages.  However, this implies that Guix has to 
assimilate all packages from all other package managers, which 
doesn't sound sustainable.

On the other hand, assuming the foreign package repository 
properly supports pinning and retrieving exact older versions of 
the source, it seems plausible to fetch those dependencies as a 
bundle during preparation. This would simplify packaging, but 
bundling dependencies may result in duplication. (Unless each 
dependency can be fetched as a separate store entry to 
deduplicate?) Unlike old NPM though, we wouldn't be recursively 
pulling in dependencies, just once per application.

It sounds to me like doing this properly would require 
build-system support, and possibly explicitly pulling in each 
foreign dependency for pinning and deduplication. The challenge is 
ensuring that all of those dependencies are properly licensed 
source code; it would be easy to leak in binaries as shortcuts.

I'm sure this has been discussed before and I just missed it. Are 
there clear guidelines on when/how to bundle dev dependencies vs. 
packaging them separately?

--
Samuel H. Christie V

Re: Go Package with multiple subpackage

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 2015 bytes --]

On Mon, 21 Oct 2024 16:18:56 +0200
Andreas Enge <andreas@enge.fr> wrote:
> > As I understand, packaging too many dependencies would create
> > complications for the maintenance.
> 
> Is that true? It looks opposite to the general Guix philosophy;
> once you have invested all the work of checking the licenses, it would
> seem a progress to submit the corresponding packages. But maybe Go is
> special in that respect; it would be nice to have the Go team's
> opinion.
One of the issue is also that I didn't find a way to have a path where
things are done step by step, so adding about 500 extra packages just
for the matterbridge package would be complicated I guess.

But in another hand using bundled in dependencies doesn't look great
either.

This also bring in more complicated questions as there is also some
tradeoffs made here. 

For instance here things are not fine with matterbridge but it's not
something a user can immediately see. So should we keep the package?
What would be the quality of the maintenance in the long run with
about 500 packages to update?

I've also no idea about how many go packages use bundled dependencies,
so maybe if there is a way to somehow un-bundle part of the
dependencies it could be a road to improve the situations as the
maintenance the dependencies shared by many go packages could be shared
somehow (assuming people do check that when updating things it doesn't
break other packages).

If packages also patch some dependencies, it would not prevent from
using non-bundled dependencies.

Another issue is that all that is statically built but that's part of
the default go compiler if I understood well, though given how Guix
works, it might easier to somehow use shared libraries (compared to
more standard distributions) if some compilers that support that since
we don't need very strict/strong ABI guarantees with Guix, and thanks
to that, reduce build times and resources consumption (like RAM, space,
etc).

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Go Package with multiple subpackage

[Nicolas Graves]

On 2024-09-22 13:21, Superfly Johnson wrote:

> The Azure SDK for Go 
> (https://github.com/Azure/azure-sdk-for-go/releases) has many 
> sub-packages within the same directory and the guix import method won't 
> work directly. I think the best solution for packaging the requirements 
> for rclone would be to make the sub-packages individual guix packages 
> using the url-fetch method instead of the git method. Each also depends 
> on several sub-packages.
>
> Thoughts?

There are possibilities for some build-systems to work on a workspace of
packages rather than a single package.

I remember I implemented that on a version of the rust-build-system
(never released, would take me quite some time to begin working on that
again), and the Node also considers working with packages.

When the design is good, you can even consider everything as a
workspace, since a package can be "a workspace of one package", and have
a single build-system being able to build all packages inside a repo,
much like submodules. In this case you could have a single guix package
with multiple outputs for each workspace's package. In my case I had for
instance a single pull of a repo with multiple crates such as
https://github.com/RustCrypto/elliptic-curves and multiple outputs, each
one for a single package.


I don't know if a modification of the go-build-system would be able to
handle that, but it seems that go also have a way to handle workspaces :
https://go.dev/blog/get-familiar-with-workspaces

I think it's a great direction to ease package update on Guix, since all
the complexity is handled in the build-system and upstream rather than
at the level of guix maintenance / user. And if a rust or node workspace
behaves poorly for some reason, then for this one we can still go back
to packages for this workspace. Although it's a big more tricky when
regarding things like searching packages, because it introduces a new
kind of "output-package". Maybe using something like

(define-public my-package
  (package
    (name ...)
    (source `(,workspace "output-name"))
    (build-system copy-build-system)
    (description ...)
    ...))

could be a solution to handle this.


Now to also answer on the issue raised by Denis, I agree that it's
better not to debundle when there are a lot of dependencies and when the
dependencies are not already packaged in Guix. Rationale is that there
is no space gain in debundling something that is not present elsewhere,
and indeed leaving certain things bundled might make it easier to build.
Also we can debundle partly based on what makes the more sense regarding
both space gains and maintainability : replace a few big library
dependencies but leave a few small ones not present elsewhere in Guix in
the source (ungoogled-chromium does that for instance).

-- 
Best regards,
Nicolas Graves

Go Package with multiple subpackage

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]


Hi,

That issue is resolved on go-team branch which it's prepared for the
merge right now.

See 
- https://git.savannah.gnu.org/cgit/guix.git/commit/?h=go-team&id=45cbf468d25ddb0075db14f372caea8ba1add26c
- https://git.savannah.gnu.org/cgit/guix.git/commit/?h=go-team&id=d691b09392fa0034d4ccbcd5b1d9b5b71af609d9
- https://git.savannah.gnu.org/cgit/guix.git/commit/?h=go-team&id=23dbf8d073fa0eae4f3422d2997078c44d295074

I'm about to refresh/import all dependencies to upgrade/unbundle:

- go-afero:201
- go-boxo:223
- go-chezmoi:38
- go-matterbridge:300+
- go-rclone:352
- go-restic:225
- go-viper:227
- go-xtaci-kcp-go-v5:203

Feel free to check go-branch and how the import works now.
It also has fix for embed files via new key parameter #:embed-files and
additional #:test-flags.

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Go Package with multiple subpackage

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 3091 bytes --]


Hi Denis,

Adding to Andreas's thesis - it's hard to be the first =) and extending
your concern - start with small; small series, small contribution,
small steps, but be persistent in your efforts and keep the attitude
(refering to Chris Hadfield's "An Astronaut's Guide to Life on Earth").

To ease you first step - just don't think about overwhelming complexity
of bringing all 500+ (or more) packages into Guix at one go, start with
short logical series say 5-10. Contribution in small series is,
personally as a commiter, easy to review and merge quickly than wait for
months when someone has a free time slot to review long chain (I
try to find time between family duties and main job).

Sending small series from larger count of involved contributors makes
propagation of new packages much faster, e.g. you may pack 10 projects
which will be in inputs for others sent by someone else etc.

Take a look at the current effort of unbundle (gnu packages ipfs) -
kubo. Bit by bit and we nearly unweaved the rainbow.

--8<---------------cut here---------------start------------->8---
I've also no idea about how many go packages use bundled dependencies,
so maybe if there is a way to somehow un-bundle part of the
dependencies it could be a road to improve the situations as the
maintenance the dependencies shared by many go packages could be shared
somehow (assuming people do check that when updating things it doesn't
break other packages).
--8<---------------cut here---------------end--------------->8---

This part may be addressed in go-build-system with default phase,
removing "vendor" directory - the only way to bundle in Go ecosystem.

--8<---------------cut here---------------start------------->8---
Another issue is that all that is statically built but that's part of
the default go compiler if I understood well, though given how Guix
works, it might easier to somehow use shared libraries (compared to
more standard distributions) if some compilers that support that since
we don't need very strict/strong ABI guarantees with Guix, and thanks
to that, reduce build times and resources consumption (like RAM, space,
etc).
--8<---------------cut here---------------end--------------->8---

There is nearly 0 linking to any libraries in Go ecosystem, each
"library" is a bunch of code which compiler use to build the final
binary, nothing is linked in terminology of C/C++ e.g. - library from
the Golang point of view is a source code (text, not blob)

--8<---------------cut here---------------start------------->8---
What would be the quality of the maintenance in the long run with
about 500 packages to update?
--8<---------------cut here---------------end--------------->8---

No need to update all the chain to make a fresh version of final binary
in most cases it's enough to refresh 2-3 packages to make build/test
green. Golang does not implement any pin mechanics for dependencie
versions like Rust/Cargo does.

So... let's get packages be packed and review in small consistence flow,
keep your altitude and let the curiosity be with you!

Keep hacking.

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Go Package with multiple subpackage

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

On Sat, 26 Oct 2024 23:11:17 +0100
Sharlatan Hellseher <sharlatanus@gmail.com> wrote:

> Take a look at the current effort of unbundle (gnu packages ipfs) -
> kubo. Bit by bit and we nearly unweaved the rainbow.
Thanks a lot, that was the part that I lacked. I assumed I needed to do
all 500 at once because go didn't provide a way to partially unbundle
dependencies.

I'll try to reproduce that with matterbridge. I assume that it's OK if
I add you in the list of reviewers if/when I get something ready.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Go Package with multiple subpackage

[Andreas Enge]

Am Sun, Oct 27, 2024 at 04:14:24PM +0100 schrieb Denis 'GNUtoo' Carikli:
> I'll try to reproduce that with matterbridge. I assume that it's OK if
> I add you in the list of reviewers if/when I get something ready.

$ ./etc/teams.scm show go
id: go
name: Go team
description: <none>
scope:
+ gnu/packages/configuration-management.scm
+ gnu/packages/golang(-.+|)\.scm$
+ gnu/packages/syncthing.scm
+ gnu/packages/terraform.scm
+ guix/build-system/go.scm
+ guix/build/go-build-system.scm
+ guix/import/go.scm
+ guix/scripts/import/go.scm
+ tests/go.scm
members:
+ Katherine Cox-Buday <cox.katherine.e+guix@gmail.com>
+ Sharlatan Hellseher <sharlatanus@gmail.com>

So it should be okay to cc the listed persons, they have volunteered to be
on the team :) Also, there is some magic in etc/git/gitconfig in the Guix
repository: When you do a "git send-email" to guix-patches@gnu.org with
a commit touching any of the Go team files above, the team members will
be automatically cc-ed.

Andreas

Re: Go Package with multiple subpackage

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

On Sun, 27 Oct 2024 18:56:11 +0100
Andreas Enge <andreas@enge.fr> wrote:
> Also, there is some magic in etc/git/gitconfig in the Guix
> repository: When you do a "git send-email" to guix-patches@gnu.org
> with a commit touching any of the Go team files above, the team
> members will be automatically cc-ed.
Yes, this is exactly why I didn't notice that Sharlatan was part of the
team: this is done almost automatically now.

Thanks for the information.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]