From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id iPPmC/1wQWNoegEAbAwnHQ (envelope-from ) for ; Sat, 08 Oct 2022 14:45:49 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id OMT3C/1wQWMUkwAA9RJhRA (envelope-from ) for ; Sat, 08 Oct 2022 14:45:49 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C71113B663 for ; Sat, 8 Oct 2022 14:45:48 +0200 (CEST) Received: from localhost ([::1]:38074 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oh9D9-0002iz-Uv for larch@yhetil.org; Sat, 08 Oct 2022 08:45:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55618) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ogSRF-0004Je-6P for guix-devel@gnu.org; Thu, 06 Oct 2022 11:05:35 -0400 Received: from relay1-d.mail.gandi.net ([2001:4b98:dc4:8::221]:58917) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ogSR4-0003UW-JX for guix-devel@gnu.org; Thu, 06 Oct 2022 11:05:21 -0400 Received: (Authenticated sender: francois@avalenn.eu) by mail.gandi.net (Postfix) with ESMTPSA id 533CF240017; Thu, 6 Oct 2022 15:05:02 +0000 (UTC) Date: Thu, 6 Oct 2022 17:01:19 +0200 From: =?utf-8?B?RnJhbsOnb2lz?= To: Katherine Cox-Buday Cc: Sarah Morgensen , guix-devel@gnu.org Subject: Re: Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags) Message-ID: <20221006150119.my4yibvqr3hway7g@noop.avalenn.eu> References: <87a6jwr5kq.fsf@gmail.com> <86bl4au4zm.fsf@mgsn.dev> <87fstmcc77.fsf_-_@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87fstmcc77.fsf_-_@gmail.com> Received-SPF: pass client-ip=2001:4b98:dc4:8::221; envelope-from=francois@avalenn.eu; helo=relay1-d.mail.gandi.net X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Sat, 08 Oct 2022 08:44:50 -0400 X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1665233148; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=NJkzpdeiktUEgZ0/zQVvVKMeHvehpMdisMk6+7/iEnE=; b=QNQ16/rMIJtSpuHdHZx6pu2h1FK8MfzmLRPdMM6ldWosSQhRz5XMcCcI0lIBiwzC4sdOGs MT3bIsVRaQMHasM49ZW7vU7BLHT8nsI+jQ0mXVc13F5/fkt+NCowGbBwMR6oo2YXjpZi/b iQHNZqzzSGzr5V4ZHfgJOMo6AHoF5w+ojZMomZn4y0cFwxp8S4w7Z2+0nEeOtr0aKex2Vv ntPFa8UmS0urDpz+lOMwfoeJM9/qzWr4niT0/22mwRQ6R1dNheeV1YRSZ0bA/3uEkA8+Yj 4AfTMgpU6VLD/uQZdsMOCEkmRDTcF4itGNjdR8Nv9sgX00oXWT08sBi4TA+xTw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1665233148; a=rsa-sha256; cv=none; b=mGSJ7QaJk2EHXAIB7FJWJaWF3OE/5KVTd270e9DSPmVX0SsVZH4O0QzA8C7OGmrgKk8H7a iYsXpuw+w7bOV6lfZJc6pEoKO6MnnkibKRUWkCk23psdnl2OUeZzsGe6ekbW8w4M2Accqg Rx5HdVTIaMuPsiZR4KknueIYnnrgMzXXiW0OrWIhkh4Lu402iWUUuHOCS9KAdyoDidGrfl 7+p3uJtOe1PZ0lM43LuBCdTDfXUqgvDP3flZqpDm2p9VJ5lJl+D0fV8DUTy8d1wuW4sLxj 7cuSHyoTXEZvdCGb/dV3yrSkPyVMI5JoOetmBgcApSYe1UEu1sZCs0Kv+PADAQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=avalenn.eu (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 3.22 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=avalenn.eu (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: C71113B663 X-Spam-Score: 3.22 X-Migadu-Scanner: scn1.migadu.com X-TUID: QZtLAYskbqei Hello, After a hiatus I am trying to package several softwares written on Golang (I would like to have terraform and go-jsonnet for example) and I have some problems with the current implementation so I am resurrecting this old mail from my Draft folder. Looking at it I think it is mostly reformulations of what Katherine and Sarah already wrote on the thread but several months later it could be a useful reminder. I have not really the energy at the moment to work on this alone but I would be interested for team work. On Thu, Sep 30, 2021 at 10:31:08AM -0500, Katherine Cox-Buday wrote: > Sarah Morgensen writes: > > >> IMO, module resolution and graph dependencies in Go are complicated enough > >> that I'd much rather take the effort we put in trying to replicate and keep > >> up with the Go toolchain's behavior, and spend that effort elsewhere. > >> > >> Does anyone have opinions on this? > > > > Hmmm. Setting aside whether or not we want to use external tools in > > general... > > > > If we use the Go tool, we shift the problem domain into "translating between > > `go' and Guix." I felt uneasy when we reimplemented some inherently Go-specific pieces of software like go.mod file parsing and dependency resolution. It seems so brittle. So I think that for importer specifically we should be able to use external programs like the Go toolchain to be in $PATH. And, given the Go tool would have proper interfaces, we would indeed just have to do some sort of translation. But as to the exact "how" it is not easy. Ideally we could split the tasks in severals steps. 1. identify all dependencies on form of go modules paths 2. identify source repositories (generally identified by a git-ref) for each of those modules 3. download content of source repository 4. populate a ready-to-use source code local cache 5. build As I see it in Guix we want : - 1 and 2 in the importer; - 3 is the "origin" method; - 4 and 5 in the build system. `go mod download` does all of 1, 2 and 3. The translations problems are how to extract data from this output to isolate dependency management on the importer and having the necessary info for the "origin" method. Nix bypass completely the problem by using the complete go mod cache as its "origin" (or more recently "vendored" dependencies see this Nix change to go mod vendor[4]) for each Go package. This solution is not viable for us as we want to be able to reuse Go modules. For the build system part I think we want to use the GOMODCACHE format as output of Guix package and to be able to do union-dir of all dependencies when we want to build a leaf package (e.g. using GOPROXY=file:/// or something like that). > > For example: when Go downloads a module, it only saves the module, not the > > whole repository*, so we can't use that to generate the hash. (* Except it > > does if you used GOPROXY=direct, but that's in the module cache, and only a > > bare repository.) > > We could use the ~GOMODCACHE~ environment variable[0] to specify where these files are placed. Yes we want to do that but it leaves open as to how we create this directory structure. I think it is good to use Go tooling at "guix import" time but not at build time. In consequence the source must be downloaded either from upstream git repo (bare or not I do not know) or as zips from proxy.golang.org. The zip solution could just parse the directory contents of GOMODCACHE as described in [5]. I would rather import git repositories closer to real source but it seems more difficult to do. Even with zip, it leaves open the problem of mapping source zips with Guix packages and to handle the dependencies correctly. Any interest on the matter ? François [4]: https://github.com/NixOS/nixpkgs/pull/86376/commits/9761128d2da7bf4d878982918242e43ae28f9b94 [5]: https://github.com/golang/go/issues/35922#issuecomment-560464243 cd go/src/golang.org/x/tools/gopls # or any other module export GOPATH=$(mktemp -d) go mod download find $GOPATH/pkg/mod/cache/download -type f | \ grep '\.\(mod\|info\|zip\)$' | \ sed -e "s,$GOPATH/pkg/mod/cache/download,https://proxy.golang.org,"