From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 8NhWMG5zEmAUTgAA0tVLHw (envelope-from ) for ; Thu, 28 Jan 2021 08:18:54 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id AHlGLG5zEmD+RwAA1q6Kng (envelope-from ) for ; Thu, 28 Jan 2021 08:18:54 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E9BC59403AE for ; Thu, 28 Jan 2021 08:18:53 +0000 (UTC) Received: from localhost ([::1]:55644 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l52Vw-0004bb-O8 for larch@yhetil.org; Thu, 28 Jan 2021 03:18:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53536) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l52Vo-0004aA-T5 for help-guix@gnu.org; Thu, 28 Jan 2021 03:18:44 -0500 Received: from smtp-out-4.mxes.net ([198.205.123.69]:42753) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l52Vm-00043S-Pz for help-guix@gnu.org; Thu, 28 Jan 2021 03:18:44 -0500 Received: from Customer-MUA (mua.mxes.net [IPv6:fd::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 4CEA97597A; Thu, 28 Jan 2021 03:18:33 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mxes.net; s=mta; t=1611821915; bh=mHV9Pyfs/DRlNaJpz13c+Cj/M+HxQrKEYFzNhSiqRo4=; h=From:To:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=Jzdi8vHLJLuy4ipcbBRAYH4lIu97W5IQaM1eJXY+idFDF7vYuex/1bj+EN75XMVjL MqQX47UrLR7MVNCIE/pG04vna+09hLYcwvir5YZIKzCMr8L8djMVmm5svi1lSdXF9K /3kTFt3RFIqMMMpD7wQWkJQtBN5rZdSyqF8eQt5c= From: Timmy Douglas To: Katherine Cox-Buday , 44178@debbugs.gnu.org, JOULAUD =?utf-8?Q?Fran=C3=A7ois?= Subject: Re: packaging a golang package In-Reply-To: <87wnvymoxe.fsf@gmail.com> References: <87h7nrud2a.fsf@timmydouglas.com> <4bdbc469-ad45-4739-b001-739ad3a60adc@www.fastmail.com> <87a6thtyvm.fsf@timmydouglas.com> <87bldw0ztb.fsf@timmydouglas.com> <20210125204534.ovhvt7rzj7tbqrnt@fjo-extia-HPdeb.example.avalenn.eu> <87wnvymoxe.fsf@gmail.com> Date: Thu, 28 Jan 2021 00:18:31 -0800 Message-ID: <87ft2l4gq0.fsf@timmydouglas.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Sent-To: Received-SPF: none client-ip=198.205.123.69; envelope-from=mail+G7=8ddd25ee@timmydouglas.com; helo=smtp-out-4.mxes.net X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_NONE=0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org, "help-guix@gnu.org" , Helio Machado <0x2b3bfa0@gmail.com> Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.05 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=mxes.net header.s=mta header.b=Jzdi8vHL; dmarc=none; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Queue-Id: E9BC59403AE X-Spam-Score: -1.05 X-Migadu-Scanner: scn1.migadu.com X-TUID: AXBpP73/q0Fm Katherine Cox-Buday writes: > Hello again, Fran=C3=A7ois! I've redirected this thread to guix-devel, and > the bug since we've begun discussing implementation details. > > JOULAUD Fran=C3=A7ois writes: > >> First is to use vendored dependencies (when upstream provides them). This >> one has the merits of simplicity (as we just have to download the >> source and launch `go build` or whatever is needed) and less risks of >> bugs. The downsides are known: difficult to audit Licences, duplication >> of code source, difficulty to follow security bugs, etc. > > -1 to this approach (explanation below). Seems like there could be two sub-scenarios here: the code could use go modules or not. >> Second is to re-vendor package from go.mod (coming directly from code >> source or synthetized from source) by creating a source derivation >> including all dependencies. This is the strategy followed by Nixpkgs >> as explained in [1]. (It seems to me this is the route followed in >> the patches to from Helio in [2] but I did not take the time to read >> them.) With this approach we still have some of the downsides of using >> vendored upstream but it is at least easier to verify the existence >> of upstream. > > I don't fully understand this approach, so I don't understand how this > is different to approach three? Does this mean we create a pseudo, > non-public package which states all the dependencies as inputs? If so, > why wouldn't we just go with option three? I think this approach is like saying you git clone the upstream repo, run go mod vendor or go mod download, then go build. Or whatever buildGoModule does in nix (https://github.com/NixOS/nixpkgs/issues/84826) I read through that issue and would personally vote for this approach of using `go` to restore the code. I think trying to reimplement go module restore process with Guix packages is bordering on Not Invented Here. There would be a never ending battle of trying to reimplement the go module restore process and the amount of source packages would really clutter things up. I think some of the issues with the distro wanting to change a package could be solved with a feature to patch go.mod before calling `go` to restore. >> Third is to package every dependencies. This is the most transparent way >> and the one that leads to less code duplication but the downside is the >> difficulty to scale with the number of packages (even if the objective of >> patch at [3] is to automate it) and the need to have only one reference >> version for each package in the distribution which have its own share of >> problems (one of them being that we don't use those tested by upstream >> as stated in [4]). > > I think this is the eternal conflict between distributions and > code-bases. As a software engineer, I am a fan of vendoring. It removes > entire classes of issues: > > - Dependency on remote servers to be up to fetch dependencies and > perform builds > - Requiring some kind of system to pin versions > - Needing to stay on top of releases even if that doesn't meet your > schedule or needs > > As a packager for a distribution, I dislike vendoring because of the > reasons you outlined above, _but_ I also dislike building upstream > software with versions of dependencies that weren't approved, tested, > and verified, upstream. It seems to me like that's a recipe for > unstable, maybe even insecure, software. > > Normally, distributions are forced to do this because their packaging > and build systems are only set up to support one version at a time. Guix > can theoretically support all versions, but doesn't want to take on the > dependency graph explosion this approach would cause. > > If we go on historical precedent, and how Guix handles the other > language ecosystems, we should only have one version of each dependency, > and build all software which relies on that dependency on the version we > have packaged. Guix has already begun taking on the difficult challenges > with this approach, including patching upstream to work with versions of > dependencies upstream was not built for. That sounds like a pretty difficult challenge. Seems like it could quickly become untenable if a commonly used library had some sort of breaking change between versions and different versions of it were used by different packages. I don't think anything is stopping Guix from having multiple versions, but hand-assigning them would feel pretty manual and error-prone. On the other hand, like you said, the dependency graph explosion from importing everything would be overwhelming. > I dislike it, but I also don't think we should try to solve the broader > class of issues while trying to implement an importer. It should be a > larger discussion within the Guix community across _all_ language > packages about how we might achieve upstream parity while still > maintaining our goals as a distribution, all while not crippling our > infrastructure and people :) > > That's my viewpoint, but I think we should all defer to some of the > longer-term maintainers who have helped guide Guix. You wrote up both sides pretty well, but I couldn't tell if you had a strong opinion on having go restore dependencies vs Guix packaging dependencies. You had a lot of strong points for vendoring, but you're writing an importer... > Thanks for writing up these options. +1