Re: packaging a golang package

Re: packaging a golang package

[Katherine Cox-Buday]

Hello again, François! I've redirected this thread to guix-devel, and
the bug since we've begun discussing implementation details.

JOULAUD François <Francois.JOULAUD@radiofrance.com> writes:

> First is to use vendored dependencies (when upstream provides them). This
> one has the merits of simplicity (as we just have to download the
> source and launch `go build` or whatever is needed) and less risks of
> bugs. The downsides are known: difficult to audit Licences, duplication
> of code source, difficulty to follow security bugs, etc.

-1 to this approach (explanation below).

> Second is to re-vendor package from go.mod (coming directly from code
> source or synthetized from source) by creating a source derivation
> including all dependencies. This is the strategy followed by Nixpkgs
> as explained in [1]. (It seems to me this is the route followed in
> the patches to  from Helio in [2] but I did not take the time to read
> them.) With this approach we still have some of the downsides of using
> vendored upstream but it is at least easier to verify the existence
> of upstream.

I don't fully understand this approach, so I don't understand how this
is different to approach three? Does this mean we create a pseudo,
non-public package which states all the dependencies as inputs? If so,
why wouldn't we just go with option three?

> Third is to package every dependencies. This is the most transparent way
> and the one that leads to less code duplication but the downside is the
> difficulty to scale with the number of packages (even if the objective of
> patch at [3] is to automate it) and the need to have only one reference
> version for each package in the distribution which have its own share of
> problems (one of them being that we don't use those tested by upstream
> as stated in [4]).

I think this is the eternal conflict between distributions and
code-bases. As a software engineer, I am a fan of vendoring. It removes
entire classes of issues:

- Dependency on remote servers to be up to fetch dependencies and
  perform builds
- Requiring some kind of system to pin versions
- Needing to stay on top of releases even if that doesn't meet your
  schedule or needs

As a packager for a distribution, I dislike vendoring because of the
reasons you outlined above, _but_ I also dislike building upstream
software with versions of dependencies that weren't approved, tested,
and verified, upstream. It seems to me like that's a recipe for
unstable, maybe even insecure, software.

Normally, distributions are forced to do this because their packaging
and build systems are only set up to support one version at a time. Guix
can theoretically support all versions, but doesn't want to take on the
dependency graph explosion this approach would cause.

If we go on historical precedent, and how Guix handles the other
language ecosystems, we should only have one version of each dependency,
and build all software which relies on that dependency on the version we
have packaged. Guix has already begun taking on the difficult challenges
with this approach, including patching upstream to work with versions of
dependencies upstream was not built for.

I dislike it, but I also don't think we should try to solve the broader
class of issues while trying to implement an importer. It should be a
larger discussion within the Guix community across _all_ language
packages about how we might achieve upstream parity while still
maintaining our goals as a distribution, all while not crippling our
infrastructure and people :)

That's my viewpoint, but I think we should all defer to some of the
longer-term maintainers who have helped guide Guix.

Thanks for writing up these options.

--
Katherine

Re: packaging a golang package

[Timmy Douglas]

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Hello again, François! I've redirected this thread to guix-devel, and
> the bug since we've begun discussing implementation details.
>
> JOULAUD François <Francois.JOULAUD@radiofrance.com> writes:
>
>> First is to use vendored dependencies (when upstream provides them). This
>> one has the merits of simplicity (as we just have to download the
>> source and launch `go build` or whatever is needed) and less risks of
>> bugs. The downsides are known: difficult to audit Licences, duplication
>> of code source, difficulty to follow security bugs, etc.
>
> -1 to this approach (explanation below).

Seems like there could be two sub-scenarios here: the code could use go
modules or not.

>> Second is to re-vendor package from go.mod (coming directly from code
>> source or synthetized from source) by creating a source derivation
>> including all dependencies. This is the strategy followed by Nixpkgs
>> as explained in [1]. (It seems to me this is the route followed in
>> the patches to  from Helio in [2] but I did not take the time to read
>> them.) With this approach we still have some of the downsides of using
>> vendored upstream but it is at least easier to verify the existence
>> of upstream.
>
> I don't fully understand this approach, so I don't understand how this
> is different to approach three? Does this mean we create a pseudo,
> non-public package which states all the dependencies as inputs? If so,
> why wouldn't we just go with option three?

I think this approach is like saying you git clone the upstream repo,
run go mod vendor or go mod download, then go build. Or whatever
buildGoModule does in nix (https://github.com/NixOS/nixpkgs/issues/84826)

I read through that issue and would personally vote for this approach of
using `go` to restore the code.

I think trying to reimplement go module restore process with Guix
packages is bordering on Not Invented Here. There would be a never
ending battle of trying to reimplement the go module restore process and
the amount of source packages would really clutter things up. I think
some of the issues with the distro wanting to change a package could be
solved with a feature to patch go.mod before calling `go` to restore.

>> Third is to package every dependencies. This is the most transparent way
>> and the one that leads to less code duplication but the downside is the
>> difficulty to scale with the number of packages (even if the objective of
>> patch at [3] is to automate it) and the need to have only one reference
>> version for each package in the distribution which have its own share of
>> problems (one of them being that we don't use those tested by upstream
>> as stated in [4]).
>
> I think this is the eternal conflict between distributions and
> code-bases. As a software engineer, I am a fan of vendoring. It removes
> entire classes of issues:
>
> - Dependency on remote servers to be up to fetch dependencies and
>   perform builds
> - Requiring some kind of system to pin versions
> - Needing to stay on top of releases even if that doesn't meet your
>   schedule or needs
>
> As a packager for a distribution, I dislike vendoring because of the
> reasons you outlined above, _but_ I also dislike building upstream
> software with versions of dependencies that weren't approved, tested,
> and verified, upstream. It seems to me like that's a recipe for
> unstable, maybe even insecure, software.
>
> Normally, distributions are forced to do this because their packaging
> and build systems are only set up to support one version at a time. Guix
> can theoretically support all versions, but doesn't want to take on the
> dependency graph explosion this approach would cause.
>
> If we go on historical precedent, and how Guix handles the other
> language ecosystems, we should only have one version of each dependency,
> and build all software which relies on that dependency on the version we
> have packaged. Guix has already begun taking on the difficult challenges
> with this approach, including patching upstream to work with versions of
> dependencies upstream was not built for.

That sounds like a pretty difficult challenge. Seems like it could
quickly become untenable if a commonly used library had some sort of
breaking change between versions and different versions of it were used
by different packages.

I don't think anything is stopping Guix from having multiple versions,
but hand-assigning them would feel pretty manual and error-prone. On the
other hand, like you said, the dependency graph explosion from importing
everything would be overwhelming.

> I dislike it, but I also don't think we should try to solve the broader
> class of issues while trying to implement an importer. It should be a
> larger discussion within the Guix community across _all_ language
> packages about how we might achieve upstream parity while still
> maintaining our goals as a distribution, all while not crippling our
> infrastructure and people :)
>
> That's my viewpoint, but I think we should all defer to some of the
> longer-term maintainers who have helped guide Guix.

You wrote up both sides pretty well, but I couldn't tell if you had a
strong opinion on having go restore dependencies vs Guix packaging
dependencies. You had a lot of strong points for vendoring, but you're
writing an importer...

> Thanks for writing up these options.

+1

Re: packaging a golang package

[adfeno--- via]

[-- Attachment #1.1: Type: text/plain, Size: 3708 bytes --]

If by vendoring we mean bundling and also make users fetch data from places not explicitly committed to the GNU FSDG, then allow me to jump in to add some important notes.

Em 27/01/2021 11:31, Katherine Cox-Buday escreveu:
> As a packager for a distribution, I dislike vendoring because of the
> reasons you outlined above, _but_ I also dislike building upstream
> software with versions of dependencies that weren't approved, tested,
> and verified, upstream. It seems to me like that's a recipe for
> unstable, maybe even insecure, software.

I also agree that this would be problematic, but I fear that if we surrender to vendoring, we might defeat the purpose of GNU Guix.

Besides, since GNU Guix is committed to GNU FSDG, the package maintainers (or reviewers at least) would *theoretically* be obligated to observe the GNU FSDG requirements, otherwise the package is considered buggy, and one of those requirements is to guarantee that all functional/practical data/work are free/libre and that non-functional/non-practical data/works grant at least freedom 2 in full (to share and sell unlimited number of copies of the original to anyone for any purpose), all this would require at least a pass/check on the source files, the same check that is normally used in the processes of unbundling/unvendoring. So we might as well cut the path short.

> I dislike it, but I also don't think we should try to solve the broader
> class of issues while trying to implement an importer. It should be a
> larger discussion within the Guix community across _all_ language
> packages about how we might achieve upstream parity while still
> maintaining our goals as a distribution, all while not crippling our
> infrastructure and people :)

I'm OK with the importer approach but, *in my opinion*, I don't think this tackles the true issue described on the 4th paragraph of the “License Rules” described on the GNU FSDG ([1]), this is why I opened Guix bug #45450 ([2]).

If Guix could make at least the suggestion (a) from Guix bug #45450 ([2]) work, I would be amazed, since it would remove the repositories from the individual Guix recipes that provide the single-language package managers.

Despite not being a developer focused on the plethora of single-language package managers out there, and not even being a developer per see, I'm also in favor of coordinating an effort between Guix and other package managers, but I think expressed/explicit commitment to the GNU FSDG by those single-language package managers is a sine qua non for all free/libre system distributions, including GuixSD, which GNU Guix also maintains.


# References


[1]: https://www.gnu.org/distros/free-system-distribution-guidelines.html#license-rules .

[2]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45450#5 .


-- 
* Ativista do software livre
	* https://libreplanet.org/wiki/User:Adfeno
	* Membro dos grupos avaliadores de
		* Software (Free Software Directory)
		* Distribuições de sistemas (FreedSoftware)
		* Sites (Free JavaScript Action Team)
	* Não sou advogado e não fomento os não livres
* Sempre veja o spam/lixo eletrônico do teu e-mail
	* Ou coloque todos os recebidos na caixa de entrada
* Sempre assino e-mails com OpenPGP
	* Chave pública: vide endereço anterior
	* Qualquer outro pode ser fraude
	* Se não tens OpenPGP, ignore o anexo "signature.asc"
* Ao enviar anexos
	* Docs., planilhas e apresentações: use OpenDocument
	* Outros tipos: vide endereço anterior
* Use protocolos de comunicação federadas
	* Vide endereço anterior
* Mensagens secretas somente via
	* XMPP com OMEMO
	* E-mail criptografado e assinado com OpenPGP


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: packaging a golang package

[Ludovic Courtès]

Hi,

adfeno--- via <help-guix@gnu.org> skribis:

> If by vendoring we mean bundling and also make users fetch data from places not explicitly committed to the GNU FSDG, then allow me to jump in to add some important notes.
>
> Em 27/01/2021 11:31, Katherine Cox-Buday escreveu:
>> As a packager for a distribution, I dislike vendoring because of the
>> reasons you outlined above, _but_ I also dislike building upstream
>> software with versions of dependencies that weren't approved, tested,
>> and verified, upstream. It seems to me like that's a recipe for
>> unstable, maybe even insecure, software.
>
> I also agree that this would be problematic, but I fear that if we surrender to vendoring, we might defeat the purpose of GNU Guix.

I sympathize with that feeling.

It’s definitely a hard problem.  Even Debian, which has been a
lighthouse for many on these matters, recently gave up:

  https://lwn.net/Articles/843313/

I think both Katherine’s concerns and yours are valid.

IMO, the importer should be able to import things recursively and assume
we’re not going to bundle anything.  It’d be up to the packager, then,
to opt out and selectively use bundled copies of dependencies, if and
when that appears necessary.

> I'm OK with the importer approach but, *in my opinion*, I don't think this tackles the true issue described on the 4th paragraph of the “License Rules” described on the GNU FSDG ([1]), this is why I opened Guix bug #45450 ([2]).

IMO, ‘guix import’ does not “steer users towards obtaining any nonfree
information” any more than wget does.  It’s a tool for packagers that
returns a package definition or template thereof, and it’s up to the
packager to decide what to do with it.

Thanks,
Ludo’.

Re: packaging a golang package

[adfeno--- via]

[-- Attachment #1.1: Type: text/plain, Size: 1731 bytes --]

Em 28/01/2021 13:03, Ludovic Courtès escreveu:
> IMO, ‘guix import’ does not “steer users towards obtaining any nonfree
> information” any more than wget does.  It’s a tool for packagers that
> returns a package definition or template thereof, and it’s up to the
> packager to decide what to do with it.

I do agree with you, sorry if for some reason it sounded otherwise. My intention is not to censor the user on that matter. Let me make it clear what I meant in the previous message:

From the bug report I referenced, one can see that what I find strange is that the cargo provided by Guix (installable through `guix package -i rust:cargo') has cargo's default repository enabled. The bug report referenced has some ideas to try to solve this (although I didn't make extensive test to see if they are all possible and doable).


-- 
* Ativista do software livre
	* https://libreplanet.org/wiki/User:Adfeno
	* Membro dos grupos avaliadores de
		* Software (Free Software Directory)
		* Distribuições de sistemas (FreedSoftware)
		* Sites (Free JavaScript Action Team)
	* Não sou advogado e não fomento os não livres
* Sempre veja o spam/lixo eletrônico do teu e-mail
	* Ou coloque todos os recebidos na caixa de entrada
* Sempre assino e-mails com OpenPGP
	* Chave pública: vide endereço anterior
	* Qualquer outro pode ser fraude
	* Se não tens OpenPGP, ignore o anexo "signature.asc"
* Ao enviar anexos
	* Docs., planilhas e apresentações: use OpenDocument
	* Outros tipos: vide endereço anterior
* Use protocolos de comunicação federadas
	* Vide endereço anterior
* Mensagens secretas somente via
	* XMPP com OMEMO
	* E-mail criptografado e assinado com OpenPGP


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: packaging a golang package

[Elais Player]

[-- Attachment #1: Type: text/plain, Size: 3151 bytes --]

Hey I just want to throw out there that I have a very WIP go module importer[1] sitting on my channel. It uses the proxy.golang.org[2] api to fetch the module's dependencies and source. The single module importer works for the most part but I need to fix it so that the recursive importer doesn't fail when it tries to fetch a dependency in a project's go.mod file that isn't proxied.

There are a few problems that should be looked at for my importer.

1.) it does not include metadata or licensing information, so after importing this way a prudent packager will have to go through and add this information for everything that was pulled down. 

2.) I don't think I'm doing a good job of filtering out similar versions of packages, i tried to follow the examples in the cargo importer but I'm not quite sure what all is going on in there. 

3.) It uses a call to `guix environment --ad-hoc go -- go mod edit -json` to parse module dependencies from the go.mod file, maybe this can be done in a more idiomatic way?

I think its a good start if anyone would like to help get it over the hump.

[1] https://git.sr.ht/~elais/orange/tree/master/item/guix/import/go.scm

[2] https://proxy.golang.org

On Thu, Jan 28, 2021, at 21:10, adfeno--- via Development of GNU Guix and the GNU System distribution. wrote:
> Em 28/01/2021 13:03, Ludovic Courtès escreveu:
> > IMO, ‘guix import’ does not “steer users towards obtaining any nonfree
> > information” any more than wget does.  It’s a tool for packagers that
> > returns a package definition or template thereof, and it’s up to the
> > packager to decide what to do with it.
> 
> I do agree with you, sorry if for some reason it sounded otherwise. My intention is not to censor the user on that matter. Let me make it clear what I meant in the previous message:
> 
> From the bug report I referenced, one can see that what I find strange is that the cargo provided by Guix (installable through `guix package -i rust:cargo') has cargo's default repository enabled. The bug report referenced has some ideas to try to solve this (although I didn't make extensive test to see if they are all possible and doable).
> 
> 
> -- 
> * Ativista do software livre
> * https://libreplanet.org/wiki/User:Adfeno
> * Membro dos grupos avaliadores de
> * Software (Free Software Directory)
> * Distribuições de sistemas (FreedSoftware)
> * Sites (Free JavaScript Action Team)
> * Não sou advogado e não fomento os não livres
> * Sempre veja o spam/lixo eletrônico do teu e-mail
> * Ou coloque todos os recebidos na caixa de entrada
> * Sempre assino e-mails com OpenPGP
> * Chave pública: vide endereço anterior
> * Qualquer outro pode ser fraude
> * Se não tens OpenPGP, ignore o anexo "signature.asc"
> * Ao enviar anexos
> * Docs., planilhas e apresentações: use OpenDocument
> * Outros tipos: vide endereço anterior
> * Use protocolos de comunicação federadas
> * Vide endereço anterior
> * Mensagens secretas somente via
> * XMPP com OMEMO
> * E-mail criptografado e assinado com OpenPGP
> 
> 
> 
> *Attachments:*
>  * signature.asc

[-- Attachment #2: Type: text/html, Size: 4146 bytes --]