unofficial mirror of help-guix@gnu.org 
 help / color / mirror / Atom feed
From: Timmy Douglas <mail@timmydouglas.com>
To: "Katherine Cox-Buday" <cox.katherine.e@gmail.com>,
	44178@debbugs.gnu.org,
	"JOULAUD François" <Francois.JOULAUD@radiofrance.com>
Cc: guix-devel@gnu.org, "help-guix@gnu.org" <help-guix@gnu.org>,
	Helio Machado <0x2b3bfa0@gmail.com>
Subject: Re: packaging a golang package
Date: Thu, 28 Jan 2021 00:18:31 -0800	[thread overview]
Message-ID: <87ft2l4gq0.fsf@timmydouglas.com> (raw)
In-Reply-To: <87wnvymoxe.fsf@gmail.com>

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Hello again, François! I've redirected this thread to guix-devel, and
> the bug since we've begun discussing implementation details.
>
> JOULAUD François <Francois.JOULAUD@radiofrance.com> writes:
>
>> First is to use vendored dependencies (when upstream provides them). This
>> one has the merits of simplicity (as we just have to download the
>> source and launch `go build` or whatever is needed) and less risks of
>> bugs. The downsides are known: difficult to audit Licences, duplication
>> of code source, difficulty to follow security bugs, etc.
>
> -1 to this approach (explanation below).

Seems like there could be two sub-scenarios here: the code could use go
modules or not.

>> Second is to re-vendor package from go.mod (coming directly from code
>> source or synthetized from source) by creating a source derivation
>> including all dependencies. This is the strategy followed by Nixpkgs
>> as explained in [1]. (It seems to me this is the route followed in
>> the patches to  from Helio in [2] but I did not take the time to read
>> them.) With this approach we still have some of the downsides of using
>> vendored upstream but it is at least easier to verify the existence
>> of upstream.
>
> I don't fully understand this approach, so I don't understand how this
> is different to approach three? Does this mean we create a pseudo,
> non-public package which states all the dependencies as inputs? If so,
> why wouldn't we just go with option three?

I think this approach is like saying you git clone the upstream repo,
run go mod vendor or go mod download, then go build. Or whatever
buildGoModule does in nix (https://github.com/NixOS/nixpkgs/issues/84826)

I read through that issue and would personally vote for this approach of
using `go` to restore the code.

I think trying to reimplement go module restore process with Guix
packages is bordering on Not Invented Here. There would be a never
ending battle of trying to reimplement the go module restore process and
the amount of source packages would really clutter things up. I think
some of the issues with the distro wanting to change a package could be
solved with a feature to patch go.mod before calling `go` to restore.

>> Third is to package every dependencies. This is the most transparent way
>> and the one that leads to less code duplication but the downside is the
>> difficulty to scale with the number of packages (even if the objective of
>> patch at [3] is to automate it) and the need to have only one reference
>> version for each package in the distribution which have its own share of
>> problems (one of them being that we don't use those tested by upstream
>> as stated in [4]).
>
> I think this is the eternal conflict between distributions and
> code-bases. As a software engineer, I am a fan of vendoring. It removes
> entire classes of issues:
>
> - Dependency on remote servers to be up to fetch dependencies and
>   perform builds
> - Requiring some kind of system to pin versions
> - Needing to stay on top of releases even if that doesn't meet your
>   schedule or needs
>
> As a packager for a distribution, I dislike vendoring because of the
> reasons you outlined above, _but_ I also dislike building upstream
> software with versions of dependencies that weren't approved, tested,
> and verified, upstream. It seems to me like that's a recipe for
> unstable, maybe even insecure, software.
>
> Normally, distributions are forced to do this because their packaging
> and build systems are only set up to support one version at a time. Guix
> can theoretically support all versions, but doesn't want to take on the
> dependency graph explosion this approach would cause.
>
> If we go on historical precedent, and how Guix handles the other
> language ecosystems, we should only have one version of each dependency,
> and build all software which relies on that dependency on the version we
> have packaged. Guix has already begun taking on the difficult challenges
> with this approach, including patching upstream to work with versions of
> dependencies upstream was not built for.

That sounds like a pretty difficult challenge. Seems like it could
quickly become untenable if a commonly used library had some sort of
breaking change between versions and different versions of it were used
by different packages.

I don't think anything is stopping Guix from having multiple versions,
but hand-assigning them would feel pretty manual and error-prone. On the
other hand, like you said, the dependency graph explosion from importing
everything would be overwhelming.

> I dislike it, but I also don't think we should try to solve the broader
> class of issues while trying to implement an importer. It should be a
> larger discussion within the Guix community across _all_ language
> packages about how we might achieve upstream parity while still
> maintaining our goals as a distribution, all while not crippling our
> infrastructure and people :)
>
> That's my viewpoint, but I think we should all defer to some of the
> longer-term maintainers who have helped guide Guix.

You wrote up both sides pretty well, but I couldn't tell if you had a
strong opinion on having go restore dependencies vs Guix packaging
dependencies. You had a lot of strong points for vendoring, but you're
writing an importer...

> Thanks for writing up these options.

+1


  reply	other threads:[~2021-01-28  8:18 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-08  7:01 packaging a golang package Timmy Douglas
2021-01-08 18:33 ` raingloom
2021-01-08 19:01 ` Leo Famulari
2021-01-10  0:32   ` Timmy Douglas
2021-01-11  6:09     ` Timmy Douglas
2021-01-17 13:31       ` Helio Machado
2021-01-25  7:18         ` Timmy Douglas
2021-01-25 20:49         ` Francois.JOULAUD--- via
2021-01-25 23:38           ` Helio Machado
2021-01-27 14:31           ` Katherine Cox-Buday
2021-01-28  8:18             ` Timmy Douglas [this message]
2021-01-28 10:32             ` adfeno--- via
2021-01-28 16:03               ` Ludovic Courtès
2021-01-28 21:10                 ` adfeno--- via
  -- strict thread matches above, loose matches on Subject: below --
2021-01-20  3:27 jgart

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87ft2l4gq0.fsf@timmydouglas.com \
    --to=mail@timmydouglas.com \
    --cc=0x2b3bfa0@gmail.com \
    --cc=44178@debbugs.gnu.org \
    --cc=Francois.JOULAUD@radiofrance.com \
    --cc=cox.katherine.e@gmail.com \
    --cc=guix-devel@gnu.org \
    --cc=help-guix@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).