From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id PWA4OVWmDGBmZQAA0tVLHw (envelope-from ) for ; Sat, 23 Jan 2021 22:42:29 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id QN6xNFWmDGAvWwAAB5/wlQ (envelope-from ) for ; Sat, 23 Jan 2021 22:42:29 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3B40494053A for ; Sat, 23 Jan 2021 22:42:29 +0000 (UTC) Received: from localhost ([::1]:50504 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l3Rbv-0003x5-VQ for larch@yhetil.org; Sat, 23 Jan 2021 17:42:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:57608) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l3RbX-0003wy-DO for guix-patches@gnu.org; Sat, 23 Jan 2021 17:42:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:52245) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l3RbW-0002vm-Lv for guix-patches@gnu.org; Sat, 23 Jan 2021 17:42:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l3RbW-0007sR-IQ for guix-patches@gnu.org; Sat, 23 Jan 2021 17:42:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#44178] [PATCH] Create importer for Go modules Resent-From: Katherine Cox-Buday Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 23 Jan 2021 22:42:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44178 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: JOULAUD =?UTF-8?Q?Fran=C3=A7ois?= Received: via spool by 44178-submit@debbugs.gnu.org id=B44178.161144169830246 (code B ref 44178); Sat, 23 Jan 2021 22:42:02 +0000 Received: (at 44178) by debbugs.gnu.org; 23 Jan 2021 22:41:38 +0000 Received: from localhost ([127.0.0.1]:35558 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l3Rb0-0007rf-1n for submit@debbugs.gnu.org; Sat, 23 Jan 2021 17:41:38 -0500 Received: from mail-io1-f43.google.com ([209.85.166.43]:33310) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l3Rax-0007rR-9N for 44178@debbugs.gnu.org; Sat, 23 Jan 2021 17:41:29 -0500 Received: by mail-io1-f43.google.com with SMTP id q129so19009727iod.0 for <44178@debbugs.gnu.org>; Sat, 23 Jan 2021 14:41:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=AfIQgd61Sh0kD6UzUU9JSc0rKkmRCaaBFD1ebOlxmbY=; b=gxEzTPMSm/IdJcxoS6huIgC2Xx1lQow/faxXqn05fPaXQeRj5pxB+/WIo8DdppzOdL tWChQjZ1cXdUxQrsb2CdpII7L3zghkbGAmogPuP0ZEnLz8I4UgwXdrs7ZmouBJ03Yxp6 SlZjGN0lGjkM1awuMh5Q5+ExlwnrVhaiWuuTVopaiVTX4uT2NHVLIXWzmhnRp1cmtXnd OXwB3WRdWIrl4rWXWZUnoE9hHa68X98NZ1ytmAn5DUGT+kGQ3dS2kouJKcshXpdPRacQ s3T8u3hUkDFpvwkpFKhwip4eIsjtm69F4XQ9rVLuPkHW2BqWgGj2oQBz2X1oZJXFyOhU onKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=AfIQgd61Sh0kD6UzUU9JSc0rKkmRCaaBFD1ebOlxmbY=; b=tdxCRjn168qfer+d+aOtI22gQwSUABlYpH1obqRcX+J9D5JwMCD5+P/0O11cVvsklb 7kC0dgA8JNQmr8lYUKU9fR/8Mdr6RnRduxaaDMN9U1lbjIzGfKCw/+5twwsDkpoM5yAe ujjvdUS6ZF1FDLNxihQHTfBRpsJPSHSZGhdB1HCSsOVIdkIHiQeZEIFwIJkFnl/s+uqw C+Pj16zrIJ4ts41UAxoyXXiXq21XoTsKmvUv8x6lKGQGYVGn6rGqwkrSdavABtPMjQ2r eSaEcQMoTN6ILbgTZjknx3WPTyt99yFRgEUNbYvFfPNm+iqQQJfVtd3o4Rz+jfIj8x0P 0TUg== X-Gm-Message-State: AOAM532Q38fwWK76wuVoASpWGd4NHua1vihoG4HeXehU9gFwxmiy5OIr mYeP5C83jE/r/y+WfTK2aAM= X-Google-Smtp-Source: ABdhPJwH0j51zZ4xfe52L9k6xwXb1ysRv9DBuw4UmFEh2AK46CrDeZYuhAJzEREPM+NxWG7u+qu8Gg== X-Received: by 2002:a05:6e02:1c85:: with SMTP id w5mr4897234ill.253.1611441681068; Sat, 23 Jan 2021 14:41:21 -0800 (PST) Received: from washu-v4 (172-221-246-205.res.spectrum.com. [172.221.246.205]) by smtp.gmail.com with ESMTPSA id b17sm9221685ilr.1.2021.01.23.14.41.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 23 Jan 2021 14:41:19 -0800 (PST) From: Katherine Cox-Buday References: <20210123212742.m2thdeuzdvgpkgeo@fjo-extia-HPdeb.example.avalenn.eu> Date: Sat, 23 Jan 2021 16:41:18 -0600 In-Reply-To: <20210123212742.m2thdeuzdvgpkgeo@fjo-extia-HPdeb.example.avalenn.eu> ("JOULAUD =?UTF-8?Q?Fran=C3=A7ois?="'s message of "Sat, 23 Jan 2021 21:35:08 +0000") Message-ID: <87r1mb6zu9.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "44178@debbugs.gnu.org" <44178@debbugs.gnu.org>, Helio Machado <0x2b3bfa0@gmail.com> Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.25 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=gxEzTPMS; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 3B40494053A X-Spam-Score: -1.25 X-Migadu-Scanner: scn1.migadu.com X-TUID: S0XxIruVYek7 Thanks so much for the patches, Helio, Joulaud! I apologize for the long delay before looking at this again. My time right now is extremely limited due to COVID-19 related childcare and activities. I was negligent and left a patch to bitrot on my computer[1]. This patch supersedes it. In addition to the things this patch corrects, I was/am working on a few other bugs: - There are valid Go Module paths which when queried will not serve the requisite meta tag. I had modified `fetch-module-meta-data` to recursively walk up the module path searching for a valid meta tag (see diff[1]). - I think Joulaud's patch covers this, but replacements relative to a module are a possibility. - For reasons Joulaud calls out, a simple line-parser of the HTML for a module is not sufficient. Since we are pulling pages down from the wider internet, we should fall back on a library made for parsing HTML so we handle any edge-cases (e.g. meta tags split across lines). I am currently looking at `sxml`, and if that doesn't pan out `htmlprag` - Some module homepages issue HTTP redirects. Last time I tested this, `http-fetch` does not handle this properly. I think that covers everything. I have pushed everything (including Joulaud's patch with appropriate attribution) here[2]. I am admittedly new at using email to organize code changes, but using a forge seems easier. [1] https://github.com/guix-mirror/guix/commit/cce35c6d68a9bddf9558e85d2cb8= 8be323da9247 [2] https://github.com/kat-co/guix/tree/create-go-importer Can I suggest we coordinate there, or is that too much of an imposition? - Katherine JOULAUD Fran=C3=A7ois writes: > This patch add a `guix import go` command. > > It was tested with several big repositories and seems to mostly work for > the import part (because building Guix packages is an other story). There > is still bugs blocking e.g. use of any k8s.io modules. > > * guix/import/go.scm: Created Go Importer > * guix/scripts/import.scm: Created Go Importer Subcommand > * guix/import/go.scm (importers): Added Go Importer Subcommand > > Signed-off-by: Francois Joulaud > --- > The patch is a rebased and modified version of the one proposed by > Katherine Cox-Buday. > > Notable modifications are=C2=A0: > - move from (guix json) to (json) > - new parse-go.mod with no "set!" and parsing some go.mod which were in > error before > - adding comments (maybe too much comments) > - renamed SCS to VCS to be in accordance with vocabulary in use in Guix > and in Go worlds > - replacing escape-capital-letters by Helio Machado's go-path-escape > - no pruning of major version in go module names as they are considered > as completely different artefacts by Go programmers > - fixed recursive-import probably broken by the rebase > - force usage of url-fetch from (guix build download) > > I would be happy to hear about problems and perspective for this patch and > will now focus on my next step which is actually building any package. > > Hope I CCed the right persons, I am not really aware of applicable > netiquette here. > > Interdiff=C2=A0: > diff --git a/guix/import/go.scm b/guix/import/go.scm > index 61009f3565..7f5f300f0a 100644 > --- a/guix/import/go.scm > +++ b/guix/import/go.scm > @@ -1,5 +1,7 @@ > ;;; GNU Guix --- Functional package management for GNU > ;;; Copyright =C2=A9 2020 Katherine Cox-Buday > +;;; Copyright =C2=A9 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> > +;;; Copyright =C2=A9 2021 Fran=C3=A7ois Joulaud > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -16,6 +18,21 @@ > ;;; You should have received a copy of the GNU General Public License > ;;; along with GNU Guix. If not, see . >=20=20=20=20 > +;;; (guix import golang) wants to make easier to create Guix package > +;;; declaration for Go modules. > +;;; > +;;; Modules in Go are "collection of related Go packages" which are > +;;; "the unit of source code interchange and versioning". > +;;; Modules are generally hosted in a repository. > +;;; > +;;; At this point it should handle correctly modules which > +;;; - have only Go dependencies; > +;;; - use go.mod; > +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). > +;;; > +;;; We translate Go module paths to a Guix package name under the > +;;; assumption that there will be no collision. > + > (define-module (guix import go) > #:use-module (ice-9 match) > #:use-module (ice-9 rdelim) > @@ -23,7 +40,7 @@ > #:use-module (ice-9 regex) > #:use-module (srfi srfi-1) > #:use-module (srfi srfi-9) > - #:use-module (guix json) > + #:use-module (json) > #:use-module ((guix download) #:prefix download:) > #:use-module (guix import utils) > #:use-module (guix import json) > @@ -33,88 +50,129 @@ > #:use-module ((guix licenses) #:prefix license:) > #:use-module (guix base16) > #:use-module (guix base32) > - #:use-module (guix build download) > + #:use-module ((guix build download) #:prefix build-download:) > #:use-module (web uri) >=20=20=20=20 > #:export (go-module->guix-package > go-module-recursive-import > infer-module-root)) >=20=20=20=20 > -(define (escape-capital-letters s) > - "To avoid ambiguity when serving from case-insensitive file systems,= the > -$module and $version elements are case-encoded by replacing every uppe= rcase > -letter with an exclamation mark followed by the corresponding lower-ca= se > -letter." > - (let ((escaped-string (string))) > - (string-for-each-index > - (lambda (i) > - (let ((c (string-ref s i))) > - (set! escaped-string > - (string-concatenate > - (list escaped-string > - (if (char-upper-case? c) "!" "") > - (string (char-downcase c))))))) > - s) > - escaped-string)) > +(define (go-path-escape path) > + "Escape a module path by replacing every uppercase letter with an ex= clamation > +mark followed with its lowercase equivalent, as per the module Escaped= Paths > +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_P= aths" > + (define (escape occurrence) > + (string-append "!" (string-downcase (match:substring occurrence)))) > + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) > + >=20=20=20=20 > (define (fetch-latest-version goproxy-url module-path) > "Fetches the version number of the latest version for MODULE-PATH fr= om the > given GOPROXY-URL server." > (assoc-ref > (json-fetch (format #f "~a/~a/@latest" goproxy-url > - (escape-capital-letters module-path))) > + (go-path-escape module-path))) > "Version")) >=20=20=20=20 > (define (fetch-go.mod goproxy-url module-path version file) > "Fetches go.mod from the given GOPROXY-URL server for the given MODU= LE-PATH > and VERSION." > - (url-fetch (format #f "~a/~a/@v/~a.mod" goproxy-url > - (escape-capital-letters module-path) > - (escape-capital-letters version)) > - file > - #:print-build-trace? #f)) > + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url > + (go-path-escape module-path) > + (go-path-escape version)))) > + (parameterize ((current-output-port (current-error-port))) > + (build-download:url-fetch url > + file > + #:print-build-trace? #f)))) >=20=20=20=20 > (define (parse-go.mod go.mod-path) > - "Parses a go.mod file and returns an alist of module path to version= ." > + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of > +requirements from it." > + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-= grammar > + ;; which we think necessary for our use case. > + (define (toplevel results) > + "Main parser, RESULTS is a pair of alist serving as accumulator for > + all encountered requirements and replacements." > + (let ((line (read-line))) > + (cond > + ((eof-object? line) > + ;; parsing ended, give back the result > + results) > + ((string=3D? line "require (") > + ;; a require block begins, delegate parsing to IN-REQUIRE > + (in-require results)) > + ((string-prefix? "require " line) > + ;; a require directive by itself > + (let* ((stripped-line (string-drop line 8)) > + (new-results (require-directive results stripped-line))) > + (toplevel new-results))) > + ((string-prefix? "replace " line) > + ;; a replace directive by itself > + (let* ((stripped-line (string-drop line 8)) > + (new-results (replace-directive results stripped-line))) > + (toplevel new-results))) > + (#t > + ;; unrecognised line, ignore silently > + (toplevel results))))) > + (define (in-require results) > + (let ((line (read-line))) > + (cond > + ((eof-object? line) > + ;; this should never happen here but we ignore silently > + results) > + ((string=3D? line ")") > + ;; end of block, coming back to toplevel > + (toplevel results)) > + (#t > + (in-require (require-directive results line)))))) > + (define (replace-directive results line) > + "Extract replaced modules and new requirements from replace direct= ive > + in LINE and add to RESULTS." > + ;; ReplaceSpec =3D ModulePath [ Version ] "=3D>" FilePath newline > + ;; | ModulePath [ Version ] "=3D>" ModulePath Version = newline . > + (let* ((requirements (car results)) > + (replaced (cdr results)) > + (re (string-concatenate > + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" > + "[[:blank:]]+" "=3D>" "[[:blank:]]+" > + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) > + (match (string-match re line)) > + (module-path (match:substring match 1)) > + (version (match:substring match 3)) > + (new-module-path (match:substring match 4)) > + (new-version (match:substring match 6)) > + (new-replaced (acons module-path version replaced)) > + (new-requirements > + (if (string-match "^\\.?\\./" new-module-path) > + requirements > + (acons new-module-path new-version requirements)))) > + (cons new-requirements new-replaced))) > + (define (require-directive results line) > + "Extract requirement from LINE and add it to RESULTS." > + (let* ((requirements (car results)) > + (replaced (cdr results)) > + ;; A line in a require directive is composed of a module pa= th and > + ;; a version separated by whitespace and an optionnal '//' = comment at > + ;; the end. > + (re (string-concatenate > + '("^[[:blank:]]*" > + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" > + "([[:blank:]]+//.*)?"))) > + (match (string-match re line)) > + (module-path (match:substring match 1)) > + (version (match:substring match 2))) > + (cons (acons module-path version requirements) replaced))) > (with-input-from-file go.mod-path > (lambda () > - (let ((in-require? #f) > - (requirements (list))) > - (do ((line (read-line) (read-line))) > - ((eof-object? line)) > - (set! line (string-trim line)) > - ;; The parser is either entering, within, exiting, or after = the > - ;; require block. The Go toolchain is trustworthy so edge-ca= ses like > - ;; double-entry, etc. need not complect the parser. > - (cond > - ((string=3D? line "require (") > - (set! in-require? #t)) > - ((and in-require? (string=3D? line ")")) > - (set! in-require? #f)) > - (in-require? > - (let* ((requirement (string-split line #\space)) > - ;; Modules should be unquoted > - (module-path (string-delete #\" (car requirement))) > - (version (list-ref requirement 1))) > - (set! requirements (acons module-path version requiremen= ts)))) > - ((string-prefix? "replace" line) > - (let* ((requirement (string-split line #\space)) > - (module-path (list-ref requirement 1)) > - (new-module-path (list-ref requirement 3)) > - (version (list-ref requirement 4))) > - (set! requirements (assoc-remove! requirements module-pa= th)) > - (set! requirements (acons new-module-path version requir= ements)))))) > - requirements)))) > - > -(define (module-path-without-major-version module-path) > - "Go modules can be appended with a major version indicator, > -e.g. /v3. Sometimes it is desirable to work with the root module path.= For > -instance, for a module path github.com/foo/bar/v3 this function returns > -github.com/foo/bar." > - (let ((m (string-match "(.*)\\/v[0-9]+$" module-path))) > - (if m > - (match:substring m 1) > - module-path))) > + (let* ((results (toplevel '(() . ()))) > + (requirements (car results)) > + (replaced (cdr results))) > + ;; At last we remove replaced modules from the requirements li= st > + (fold > + (lambda (replacedelem requirements) > + (alist-delete! (car replacedelem) requirements)) > + requirements > + replaced))))) >=20=20=20=20 > (define (infer-module-root module-path) > "Go modules can be defined at any level of a repository's tree, but = querying > @@ -124,38 +182,42 @@ root path from its path. For a set of well-known = forges, the pattern of what > consists of a module's root page is known before hand." > ;; See the following URL for the official Go equivalent: > ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea= 21b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 > - (define-record-type > - (make-scs url-prefix root-regex type) > - scs? > - (url-prefix scs-url-prefix) > - (root-regex scs-root-regex) > - (type scs-type)) > - (let* ((known-scs > + ;; > + ;; FIXME: handle module path with VCS qualifier as described in > + ;; https://golang.org/ref/mod#vcs-find and > + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths > + (define-record-type > + (make-vcs url-prefix root-regex type) > + vcs? > + (url-prefix vcs-url-prefix) > + (root-regex vcs-root-regex) > + (type vcs-type)) > + (let* ((known-vcs > (list > - (make-scs > + (make-vcs > "github.com" > "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Z= a-z0-9_.\\-]+)*$" > 'git) > - (make-scs > + (make-vcs > "bitbucket.org" > "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(= /[A-Za-z0-9_.\\-]+)*$`" > 'unknown) > - (make-scs > + (make-vcs > "hub.jazz.net/git/" > "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Z= a-z0-9_.\\-]+)*$" > 'git) > - (make-scs > + (make-vcs > "git.apache.org" > "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\= \-]+)*$" > 'git) > - (make-scs > + (make-vcs > "git.openstack.org" > "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-= ]+)(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" > 'git))) > - (scs (find (lambda (scs) (string-prefix? (scs-url-prefix scs)= module-path)) > - known-scs))) > - (if scs > - (match:substring (string-match (scs-root-regex scs) module-pat= h) 1) > + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs)= module-path)) > + known-vcs))) > + (if vcs > + (match:substring (string-match (vcs-root-regex vcs) module-pat= h) 1) > module-path))) >=20=20=20=20 > (define (to-guix-package-name module-path) > @@ -164,8 +226,7 @@ consists of a module's root page is known before ha= nd." > (string-append "go-" > (string-replace-substring > (string-replace-substring > - ;; Guix has its own field for version > - (module-path-without-major-version module-path) > + module-path > "." "-") > "/" "-")))) >=20=20=20=20 > @@ -173,7 +234,9 @@ consists of a module's root page is known before ha= nd." > "Fetches module meta-data from a module's landing page. This is nece= ssary > because goproxy servers don't currently provide all the information ne= eded to > build a package." > - (let* ((port (http-fetch (string->uri (format #f "https://~a?go-get= =3D1" module-path)))) > + ;; FIXME: This code breaks on k8s.io which have a meta tag splitted > + ;; on several lines > + (let* ((port (build-download:http-fetch (string->uri (format #f "htt= ps://~a?go-get=3D1" module-path)))) > (module-metadata #f) > (meta-tag-prefix " (meta-tag-prefix-length (string-length meta-tag-prefix))) > @@ -185,7 +248,7 @@ build a package." > (let* ((start (+ meta-tag-index meta-tag-prefix-length)) > (end (string-index line #\" start))) > (set! module-metadata > - (string-split (substring/shared line start end) #\space)= ))))) > + (string-split (substring/shared line start end) #\sp= ace)))))) > (close-port port) > module-metadata)) >=20=20=20=20 > @@ -244,7 +307,7 @@ control system is being used." > (dependencies (map car (parse-go.mod temp))) > (guix-name (to-guix-package-name module-path)) > (root-module-path (infer-module-root module-path)) > - ;; SCS type and URL are not included in goproxy informatio= n. For > + ;; VCS type and URL are not included in goproxy informatio= n. For > ;; this we need to fetch it from the official module page. > (meta-data (fetch-module-meta-data root-module-path)) > (scs-type (module-meta-data-scs meta-data)) > @@ -268,9 +331,10 @@ control system is being used." >=20=20=20=20 > (define* (go-module-recursive-import package-name > #:key (goproxy-url "https://proxy= .golang.org")) > - (recursive-import package-name #f > - #:repo->guix-package > - (lambda (name _) > - (go-module->guix-package name > - #:goproxy-url goproxy-u= rl)) > - #:guix-name to-guix-package-name)) > + (recursive-import > + package-name > + #:repo->guix-package (lambda* (name . _) > + (go-module->guix-package > + name > + #:goproxy-url goproxy-url)) > + #:guix-name to-guix-package-name)) > > guix/import/go.scm | 340 +++++++++++++++++++++++++++++++++++++ > guix/scripts/import.scm | 2 +- > guix/scripts/import/go.scm | 118 +++++++++++++ > 3 files changed, 459 insertions(+), 1 deletion(-) > create mode 100644 guix/import/go.scm > create mode 100644 guix/scripts/import/go.scm > > diff --git a/guix/import/go.scm b/guix/import/go.scm > new file mode 100644 > index 0000000000..7f5f300f0a > --- /dev/null > +++ b/guix/import/go.scm > @@ -0,0 +1,340 @@ > +;;; GNU Guix --- Functional package management for GNU > +;;; Copyright =C2=A9 2020 Katherine Cox-Buday > +;;; Copyright =C2=A9 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> > +;;; Copyright =C2=A9 2021 Fran=C3=A7ois Joulaud > +;;; > +;;; This file is part of GNU Guix. > +;;; > +;;; GNU Guix is free software; you can redistribute it and/or modify it > +;;; under the terms of the GNU General Public License as published by > +;;; the Free Software Foundation; either version 3 of the License, or (at > +;;; your option) any later version. > +;;; > +;;; GNU Guix is distributed in the hope that it will be useful, but > +;;; WITHOUT ANY WARRANTY; without even the implied warranty of > +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +;;; GNU General Public License for more details. > +;;; > +;;; You should have received a copy of the GNU General Public License > +;;; along with GNU Guix. If not, see . > + > +;;; (guix import golang) wants to make easier to create Guix package > +;;; declaration for Go modules. > +;;; > +;;; Modules in Go are "collection of related Go packages" which are > +;;; "the unit of source code interchange and versioning". > +;;; Modules are generally hosted in a repository. > +;;; > +;;; At this point it should handle correctly modules which > +;;; - have only Go dependencies; > +;;; - use go.mod; > +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). > +;;; > +;;; We translate Go module paths to a Guix package name under the > +;;; assumption that there will be no collision. > + > +(define-module (guix import go) > + #:use-module (ice-9 match) > + #:use-module (ice-9 rdelim) > + #:use-module (ice-9 receive) > + #:use-module (ice-9 regex) > + #:use-module (srfi srfi-1) > + #:use-module (srfi srfi-9) > + #:use-module (json) > + #:use-module ((guix download) #:prefix download:) > + #:use-module (guix import utils) > + #:use-module (guix import json) > + #:use-module (guix packages) > + #:use-module (guix upstream) > + #:use-module (guix utils) > + #:use-module ((guix licenses) #:prefix license:) > + #:use-module (guix base16) > + #:use-module (guix base32) > + #:use-module ((guix build download) #:prefix build-download:) > + #:use-module (web uri) > + > + #:export (go-module->guix-package > + go-module-recursive-import > + infer-module-root)) > + > +(define (go-path-escape path) > + "Escape a module path by replacing every uppercase letter with an excl= amation > +mark followed with its lowercase equivalent, as per the module Escaped P= aths > +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Pat= hs" > + (define (escape occurrence) > + (string-append "!" (string-downcase (match:substring occurrence)))) > + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) > + > + > +(define (fetch-latest-version goproxy-url module-path) > + "Fetches the version number of the latest version for MODULE-PATH from= the > +given GOPROXY-URL server." > + (assoc-ref > + (json-fetch (format #f "~a/~a/@latest" goproxy-url > + (go-path-escape module-path))) > + "Version")) > + > +(define (fetch-go.mod goproxy-url module-path version file) > + "Fetches go.mod from the given GOPROXY-URL server for the given MODULE= -PATH > +and VERSION." > + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url > + (go-path-escape module-path) > + (go-path-escape version)))) > + (parameterize ((current-output-port (current-error-port))) > + (build-download:url-fetch url > + file > + #:print-build-trace? #f)))) > + > +(define (parse-go.mod go.mod-path) > + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of > +requirements from it." > + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-gr= ammar > + ;; which we think necessary for our use case. > + (define (toplevel results) > + "Main parser, RESULTS is a pair of alist serving as accumulator for > + all encountered requirements and replacements." > + (let ((line (read-line))) > + (cond > + ((eof-object? line) > + ;; parsing ended, give back the result > + results) > + ((string=3D? line "require (") > + ;; a require block begins, delegate parsing to IN-REQUIRE > + (in-require results)) > + ((string-prefix? "require " line) > + ;; a require directive by itself > + (let* ((stripped-line (string-drop line 8)) > + (new-results (require-directive results stripped-line))) > + (toplevel new-results))) > + ((string-prefix? "replace " line) > + ;; a replace directive by itself > + (let* ((stripped-line (string-drop line 8)) > + (new-results (replace-directive results stripped-line))) > + (toplevel new-results))) > + (#t > + ;; unrecognised line, ignore silently > + (toplevel results))))) > + (define (in-require results) > + (let ((line (read-line))) > + (cond > + ((eof-object? line) > + ;; this should never happen here but we ignore silently > + results) > + ((string=3D? line ")") > + ;; end of block, coming back to toplevel > + (toplevel results)) > + (#t > + (in-require (require-directive results line)))))) > + (define (replace-directive results line) > + "Extract replaced modules and new requirements from replace directive > + in LINE and add to RESULTS." > + ;; ReplaceSpec =3D ModulePath [ Version ] "=3D>" FilePath newline > + ;; | ModulePath [ Version ] "=3D>" ModulePath Version ne= wline . > + (let* ((requirements (car results)) > + (replaced (cdr results)) > + (re (string-concatenate > + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" > + "[[:blank:]]+" "=3D>" "[[:blank:]]+" > + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) > + (match (string-match re line)) > + (module-path (match:substring match 1)) > + (version (match:substring match 3)) > + (new-module-path (match:substring match 4)) > + (new-version (match:substring match 6)) > + (new-replaced (acons module-path version replaced)) > + (new-requirements > + (if (string-match "^\\.?\\./" new-module-path) > + requirements > + (acons new-module-path new-version requirements)))) > + (cons new-requirements new-replaced))) > + (define (require-directive results line) > + "Extract requirement from LINE and add it to RESULTS." > + (let* ((requirements (car results)) > + (replaced (cdr results)) > + ;; A line in a require directive is composed of a module path= and > + ;; a version separated by whitespace and an optionnal '//' co= mment at > + ;; the end. > + (re (string-concatenate > + '("^[[:blank:]]*" > + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" > + "([[:blank:]]+//.*)?"))) > + (match (string-match re line)) > + (module-path (match:substring match 1)) > + (version (match:substring match 2))) > + (cons (acons module-path version requirements) replaced))) > + (with-input-from-file go.mod-path > + (lambda () > + (let* ((results (toplevel '(() . ()))) > + (requirements (car results)) > + (replaced (cdr results))) > + ;; At last we remove replaced modules from the requirements list > + (fold > + (lambda (replacedelem requirements) > + (alist-delete! (car replacedelem) requirements)) > + requirements > + replaced))))) > + > +(define (infer-module-root module-path) > + "Go modules can be defined at any level of a repository's tree, but qu= erying > +for the meta tag usually can only be done at the webpage at the root of = the > +repository. Therefore, it is sometimes necessary to try and derive a mod= ule's > +root path from its path. For a set of well-known forges, the pattern of = what > +consists of a module's root page is known before hand." > + ;; See the following URL for the official Go equivalent: > + ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21= b99f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 > + ;; > + ;; FIXME: handle module path with VCS qualifier as described in > + ;; https://golang.org/ref/mod#vcs-find and > + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths > + (define-record-type > + (make-vcs url-prefix root-regex type) > + vcs? > + (url-prefix vcs-url-prefix) > + (root-regex vcs-root-regex) > + (type vcs-type)) > + (let* ((known-vcs > + (list > + (make-vcs > + "github.com" > + "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-= z0-9_.\\-]+)*$" > + 'git) > + (make-vcs > + "bitbucket.org" > + "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[= A-Za-z0-9_.\\-]+)*$`" > + 'unknown) > + (make-vcs > + "hub.jazz.net/git/" > + "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-= z0-9_.\\-]+)*$" > + 'git) > + (make-vcs > + "git.apache.org" > + "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-= ]+)*$" > + 'git) > + (make-vcs > + "git.openstack.org" > + "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+= )(\\.git)?(/[A-Za-z0-9_.\\-]+)*$" > + 'git))) > + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) m= odule-path)) > + known-vcs))) > + (if vcs > + (match:substring (string-match (vcs-root-regex vcs) module-path)= 1) > + module-path))) > + > +(define (to-guix-package-name module-path) > + "Converts a module's path to the canonical Guix format for Go packages= ." > + (string-downcase > + (string-append "go-" > + (string-replace-substring > + (string-replace-substring > + module-path > + "." "-") > + "/" "-")))) > + > +(define (fetch-module-meta-data module-path) > + "Fetches module meta-data from a module's landing page. This is necess= ary > +because goproxy servers don't currently provide all the information need= ed to > +build a package." > + ;; FIXME: This code breaks on k8s.io which have a meta tag splitted > + ;; on several lines > + (let* ((port (build-download:http-fetch (string->uri (format #f "https= ://~a?go-get=3D1" module-path)))) > + (module-metadata #f) > + (meta-tag-prefix " + (meta-tag-prefix-length (string-length meta-tag-prefix))) > + (do ((line (read-line port) (read-line port))) > + ((or (eof-object? line) > + module-metadata)) > + (let ((meta-tag-index (string-contains line meta-tag-prefix))) > + (when meta-tag-index > + (let* ((start (+ meta-tag-index meta-tag-prefix-length)) > + (end (string-index line #\" start))) > + (set! module-metadata > + (string-split (substring/shared line start end) #\spac= e)))))) > + (close-port port) > + module-metadata)) > + > +(define (module-meta-data-scs meta-data) > + "Return the source control system specified by a module's meta-data." > + (string->symbol (list-ref meta-data 1))) > + > +(define (module-meta-data-repo-url meta-data goproxy-url) > + "Return the URL where the fetcher which will be used can download the = source > +control." > + (if (member (module-meta-data-scs meta-data) '(fossil mod)) > + goproxy-url > + (list-ref meta-data 2))) > + > +(define (source-uri scs-type scs-repo-url file) > + "Generate the `origin' block of a package depending on what type of so= urce > +control system is being used." > + (case scs-type > + ((git) > + `(origin > + (method git-fetch) > + (uri (git-reference > + (url ,scs-repo-url) > + (commit (string-append "v" version)))) > + (file-name (git-file-name name version)) > + (sha256 > + (base32 > + ,(guix-hash-url file))))) > + ((hg) > + `(origin > + (method hg-fetch) > + (uri (hg-reference > + (url ,scs-repo-url) > + (changeset ,version))) > + (file-name (format #f "~a-~a-checkout" name version)))) > + ((svn) > + `(origin > + (method svn-fetch) > + (uri (svn-reference > + (url ,scs-repo-url) > + (revision (string->number version)) > + (recursive? #f))) > + (file-name (format #f "~a-~a-checkout" name version)) > + (sha256 > + (base32 > + ,(guix-hash-url file))))) > + (else > + (raise-exception (format #f "unsupported scs type: ~a" scs-type))))) > + > +(define* (go-module->guix-package module-path #:key (goproxy-url "https:= //proxy.golang.org")) > + (call-with-temporary-output-file > + (lambda (temp port) > + (let* ((latest-version (fetch-latest-version goproxy-url module-pat= h)) > + (go.mod-path (fetch-go.mod goproxy-url module-path latest-ve= rsion > + temp)) > + (dependencies (map car (parse-go.mod temp))) > + (guix-name (to-guix-package-name module-path)) > + (root-module-path (infer-module-root module-path)) > + ;; VCS type and URL are not included in goproxy information.= For > + ;; this we need to fetch it from the official module page. > + (meta-data (fetch-module-meta-data root-module-path)) > + (scs-type (module-meta-data-scs meta-data)) > + (scs-repo-url (module-meta-data-repo-url meta-data goproxy-u= rl))) > + (values > + `(package > + (name ,guix-name) > + ;; Elide the "v" prefix Go uses > + (version ,(string-trim latest-version #\v)) > + (source > + ,(source-uri scs-type scs-repo-url temp)) > + (build-system go-build-system) > + ,@(maybe-inputs (map to-guix-package-name dependencies)) > + ;; TODO(katco): It would be nice to make an effort to fetch t= his > + ;; from known forges, e.g. GitHub > + (home-page ,(format #f "https://~a" root-module-path)) > + (synopsis "A Go package") > + (description ,(format #f "~a is a Go package." guix-name)) > + (license #f)) > + dependencies))))) > + > +(define* (go-module-recursive-import package-name > + #:key (goproxy-url "https://proxy.g= olang.org")) > + (recursive-import > + package-name > + #:repo->guix-package (lambda* (name . _) > + (go-module->guix-package > + name > + #:goproxy-url goproxy-url)) > + #:guix-name to-guix-package-name)) > diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm > index 0a3863f965..1d2b45d942 100644 > --- a/guix/scripts/import.scm > +++ b/guix/scripts/import.scm > @@ -77,7 +77,7 @@ rather than \\n." > ;;; >=20=20 > (define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa= " "gem" > - "cran" "crate" "texlive" "json" "opam")) > + "go" "cran" "crate" "texlive" "json" "opam")) >=20=20 > (define (resolve-importer name) > (let ((module (resolve-interface > diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm > new file mode 100644 > index 0000000000..fde7555973 > --- /dev/null > +++ b/guix/scripts/import/go.scm > @@ -0,0 +1,118 @@ > +;;; GNU Guix --- Functional package management for GNU > +;;; Copyright =C2=A9 2020 Katherine Cox-Buday > +;;; > +;;; This file is part of GNU Guix. > +;;; > +;;; GNU Guix is free software; you can redistribute it and/or modify it > +;;; under the terms of the GNU General Public License as published by > +;;; the Free Software Foundation; either version 3 of the License, or (at > +;;; your option) any later version. > +;;; > +;;; GNU Guix is distributed in the hope that it will be useful, but > +;;; WITHOUT ANY WARRANTY; without even the implied warranty of > +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +;;; GNU General Public License for more details. > +;;; > +;;; You should have received a copy of the GNU General Public License > +;;; along with GNU Guix. If not, see . > + > +(define-module (guix scripts import go) > + #:use-module (guix ui) > + #:use-module (guix utils) > + #:use-module (guix scripts) > + #:use-module (guix import go) > + #:use-module (guix scripts import) > + #:use-module (srfi srfi-1) > + #:use-module (srfi srfi-11) > + #:use-module (srfi srfi-37) > + #:use-module (ice-9 match) > + #:use-module (ice-9 format) > + #:export (guix-import-go)) > + > + > +;;; > +;;; Command-line options. > +;;; > + > +(define %default-options > + '()) > + > +(define (show-help) > + (display (G_ "Usage: guix import go PACKAGE-PATH > +Import and convert the Go module for PACKAGE-PATH.\n")) > + (display (G_ " > + -h, --help display this help and exit")) > + (display (G_ " > + -V, --version display version information and exit")) > + (display (G_ " > + -r, --recursive generate package expressions for all Go modules\ > + that are not yet in Guix")) > + (display (G_ " > + -p, --goproxy=3DGOPROXY specify which goproxy server to use")) > + (newline) > + (show-bug-report-information)) > + > +(define %options > + ;; Specification of the command-line options. > + (cons* (option '(#\h "help") #f #f > + (lambda args > + (show-help) > + (exit 0))) > + (option '(#\V "version") #f #f > + (lambda args > + (show-version-and-exit "guix import go"))) > + (option '(#\r "recursive") #f #f > + (lambda (opt name arg result) > + (alist-cons 'recursive #t result))) > + (option '(#\p "goproxy") #t #f > + (lambda (opt name arg result) > + (alist-cons 'goproxy > + (string->symbol arg) > + (alist-delete 'goproxy result)))) > + %standard-import-options)) > + > + > +;;; > +;;; Entry point. > +;;; > + > +(define (guix-import-go . args) > + (define (parse-options) > + ;; Return the alist of option values. > + (args-fold* args %options > + (lambda (opt name arg result) > + (leave (G_ "~A: unrecognized option~%") name)) > + (lambda (arg result) > + (alist-cons 'argument arg result)) > + %default-options)) > + > + (let* ((opts (parse-options)) > + (args (filter-map (match-lambda > + (('argument . value) > + value) > + (_ #f)) > + (reverse opts)))) > + (match args > + ((module-name) > + (if (assoc-ref opts 'recursive) > + (map (match-lambda > + ((and ('package ('name name) . rest) pkg) > + `(define-public ,(string->symbol name) > + ,pkg)) > + (_ #f)) > + (go-module-recursive-import module-name > + #:goproxy-url > + (or (assoc-ref opts 'goproxy) > + "https://proxy.golang.or= g"))) > + (let ((sexp (go-module->guix-package module-name > + #:goproxy-url > + (or (assoc-ref opts 'gop= roxy) > + "https://proxy.golan= g.org")))) > + (unless sexp > + (leave (G_ "failed to download meta-data for module '~a'~= %") > + module-name)) > + sexp))) > + (() > + (leave (G_ "too few arguments~%"))) > + ((many ...) > + (leave (G_ "too many arguments~%")))))) --=20 Katherine