Go Package with multiple subpackage

Go Package with multiple subpackage

[Superfly Johnson]

The Azure SDK for Go 
(https://github.com/Azure/azure-sdk-for-go/releases) has many 
sub-packages within the same directory and the guix import method won't 
work directly. I think the best solution for packaging the requirements 
for rclone would be to make the sub-packages individual guix packages 
using the url-fetch method instead of the git method. Each also depends 
on several sub-packages.

Thoughts?

Go Package with multiple subpackage

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]


Hi,

That issue is resolved on go-team branch which it's prepared for the
merge right now.

See 
- https://git.savannah.gnu.org/cgit/guix.git/commit/?h=go-team&id=45cbf468d25ddb0075db14f372caea8ba1add26c
- https://git.savannah.gnu.org/cgit/guix.git/commit/?h=go-team&id=d691b09392fa0034d4ccbcd5b1d9b5b71af609d9
- https://git.savannah.gnu.org/cgit/guix.git/commit/?h=go-team&id=23dbf8d073fa0eae4f3422d2997078c44d295074

I'm about to refresh/import all dependencies to upgrade/unbundle:

- go-afero:201
- go-boxo:223
- go-chezmoi:38
- go-matterbridge:300+
- go-rclone:352
- go-restic:225
- go-viper:227
- go-xtaci-kcp-go-v5:203

Feel free to check go-branch and how the import works now.
It also has fix for embed files via new key parameter #:embed-files and
additional #:test-flags.

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Go Package with multiple subpackage

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]

Hi,

Sorry for the delay,

On Sun, 22 Sep 2024 13:21:30 -0400
Superfly Johnson <superfly.johnson@yahoo.com> wrote:

> The Azure SDK for Go 
> (https://github.com/Azure/azure-sdk-for-go/releases) has many 
> sub-packages within the same directory and the guix import method
> won't work directly. I think the best solution for packaging the
> requirements for rclone would be to make the sub-packages individual
> guix packages using the url-fetch method instead of the git method.
> Each also depends on several sub-packages.
I had a similar issue with the matterbridge package which has about 500
dependencies that are not in Guix.

I verified most of the licenses for the dependencies with a combination
of recursive guix import and manually looking for the ones that weren't
detected.

And given the number of dependencies I was told that it was okay to
have them bundled in.

As I understand, packaging too many dependencies would create
complications for the maintenance.

Though the current situation is far from ideal as checking the license
of ~500 dependencies is also very time consuming and if problems
appears in newer releases it would be difficult to detect them.

Also note that we didn't know about the dependencies issues of
matterbridge when it got in, so maybe it played a role in the decisions
that was taken at the time.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Go Package with multiple subpackage

[Andreas Enge]

Am Fri, Oct 18, 2024 at 11:43:28PM +0200 schrieb Denis 'GNUtoo' Carikli:
> I verified most of the licenses for the dependencies with a combination
> of recursive guix import and manually looking for the ones that weren't
> detected.
> 
> And given the number of dependencies I was told that it was okay to
> have them bundled in.
> 
> As I understand, packaging too many dependencies would create
> complications for the maintenance.

Is that true? It looks opposite to the general Guix philosophy;
once you have invested all the work of checking the licenses, it would
seem a progress to submit the corresponding packages. But maybe Go is
special in that respect; it would be nice to have the Go team's opinion.

Andreas

Re: Go Package with multiple subpackage

[Nicolas Graves]

On 2024-09-22 13:21, Superfly Johnson wrote:

> The Azure SDK for Go 
> (https://github.com/Azure/azure-sdk-for-go/releases) has many 
> sub-packages within the same directory and the guix import method won't 
> work directly. I think the best solution for packaging the requirements 
> for rclone would be to make the sub-packages individual guix packages 
> using the url-fetch method instead of the git method. Each also depends 
> on several sub-packages.
>
> Thoughts?

There are possibilities for some build-systems to work on a workspace of
packages rather than a single package.

I remember I implemented that on a version of the rust-build-system
(never released, would take me quite some time to begin working on that
again), and the Node also considers working with packages.

When the design is good, you can even consider everything as a
workspace, since a package can be "a workspace of one package", and have
a single build-system being able to build all packages inside a repo,
much like submodules. In this case you could have a single guix package
with multiple outputs for each workspace's package. In my case I had for
instance a single pull of a repo with multiple crates such as
https://github.com/RustCrypto/elliptic-curves and multiple outputs, each
one for a single package.


I don't know if a modification of the go-build-system would be able to
handle that, but it seems that go also have a way to handle workspaces :
https://go.dev/blog/get-familiar-with-workspaces

I think it's a great direction to ease package update on Guix, since all
the complexity is handled in the build-system and upstream rather than
at the level of guix maintenance / user. And if a rust or node workspace
behaves poorly for some reason, then for this one we can still go back
to packages for this workspace. Although it's a big more tricky when
regarding things like searching packages, because it introduces a new
kind of "output-package". Maybe using something like

(define-public my-package
  (package
    (name ...)
    (source `(,workspace "output-name"))
    (build-system copy-build-system)
    (description ...)
    ...))

could be a solution to handle this.


Now to also answer on the issue raised by Denis, I agree that it's
better not to debundle when there are a lot of dependencies and when the
dependencies are not already packaged in Guix. Rationale is that there
is no space gain in debundling something that is not present elsewhere,
and indeed leaving certain things bundled might make it easier to build.
Also we can debundle partly based on what makes the more sense regarding
both space gains and maintainability : replace a few big library
dependencies but leave a few small ones not present elsewhere in Guix in
the source (ungoogled-chromium does that for instance).

-- 
Best regards,
Nicolas Graves

Re: Go Package with multiple subpackage

[Samuel Christie via Development of GNU Guix and the GNU System distribution.]

Andreas Enge <andreas@enge.fr> writes: 
> Am Fri, Oct 18, 2024 at 11:43:28PM +0200 schrieb Denis 'GNUtoo' 
> Carikli: 
>> And given the number of dependencies I was told that it was 
>> okay to have them bundled in.
>>
>> As I understand, packaging too many dependencies would create 
>> complications for the maintenance.
>
> Is that true? It looks opposite to the general Guix philosophy; 
> once you have invested all the work of checking the licenses, it 
> would seem a progress to submit the corresponding packages.

I am slightly interested in this question as well.

On the one hand, it seems like taking reproducible builds 
seriously requires not only pinning all of the sources but also 
archival etc. Guix seems to handle that right now by absorbing 
dependencies as packages.  However, this implies that Guix has to 
assimilate all packages from all other package managers, which 
doesn't sound sustainable.

On the other hand, assuming the foreign package repository 
properly supports pinning and retrieving exact older versions of 
the source, it seems plausible to fetch those dependencies as a 
bundle during preparation. This would simplify packaging, but 
bundling dependencies may result in duplication. (Unless each 
dependency can be fetched as a separate store entry to 
deduplicate?) Unlike old NPM though, we wouldn't be recursively 
pulling in dependencies, just once per application.

It sounds to me like doing this properly would require 
build-system support, and possibly explicitly pulling in each 
foreign dependency for pinning and deduplication. The challenge is 
ensuring that all of those dependencies are properly licensed 
source code; it would be easy to leak in binaries as shortcuts.

I'm sure this has been discussed before and I just missed it. Are 
there clear guidelines on when/how to bundle dev dependencies vs. 
packaging them separately?

--
Samuel H. Christie V