From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id sNd1Km/D82DUYAAAgWs5BA (envelope-from ) for ; Sun, 18 Jul 2021 08:00:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id izguJm/D82AcYwAA1q6Kng (envelope-from ) for ; Sun, 18 Jul 2021 06:00:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EA67B1D800 for ; Sun, 18 Jul 2021 08:00:14 +0200 (CEST) Received: from localhost ([::1]:47926 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m4zqY-0003X0-0F for larch@yhetil.org; Sun, 18 Jul 2021 02:00:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58828) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m4zqM-0003Ta-Hp for guix-patches@gnu.org; Sun, 18 Jul 2021 02:00:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:43509) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m4zqM-00060K-9C for guix-patches@gnu.org; Sun, 18 Jul 2021 02:00:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1m4zqM-00069F-78 for guix-patches@gnu.org; Sun, 18 Jul 2021 02:00:02 -0400 Subject: bug#49602: [PATCH] import: go: Upgrade go.mod parser. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-To: guix-patches@gnu.org Resent-Date: Sun, 18 Jul 2021 06:00:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: cc-closed 49602 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Sarah Morgensen Cc: 49602-done@debbugs.gnu.org Mail-Followup-To: 49602@debbugs.gnu.org, maxim.cournoyer@gmail.com, iskarian@mgsn.dev Received: via spool by 49602-done@debbugs.gnu.org id=D49602.162658799823590 (code D ref 49602); Sun, 18 Jul 2021 06:00:01 +0000 Received: (at 49602-done) by debbugs.gnu.org; 18 Jul 2021 05:59:58 +0000 Received: from localhost ([127.0.0.1]:55053 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m4zqC-00068K-Kh for submit@debbugs.gnu.org; Sun, 18 Jul 2021 01:59:58 -0400 Received: from mail-qk1-f181.google.com ([209.85.222.181]:44771) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m4zqA-000687-PG for 49602-done@debbugs.gnu.org; Sun, 18 Jul 2021 01:59:52 -0400 Received: by mail-qk1-f181.google.com with SMTP id a80so1990751qkg.11 for <49602-done@debbugs.gnu.org>; Sat, 17 Jul 2021 22:59:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=EZuvC/6atMEYFimCXia/D65CMz0lLb/+Ul2WqyYI8+o=; b=HK4FWe8+zOdiqWq0WYHSLhEWceb81WSLiDmZ121VfU+KdB0l67f650BY3Nx/nmae3k +06lu6VuNt6SQ7g/8ApSrO5OpXxKi1fFwB2ePxo8UPazoJLfP0KX54EA/HYsqlf4EznW RiqeLskFNcJAonXK23fqCfDWQmJFauAdW048fiDZjkXycLt3iCXoxlXXiNvBUXiK1+Ju CWYfqzHYBd26gbjZ81jo5OhhZ7zSK5uE8itqc4o9IhjusuRG8a0zrR8PJO60LT5GoE8o 0/YJgLSM2OQB4m7cqy8TE+lSNlQj9v3trkw5dcK0H1eemcqMuvYgGXl2/iPDUsBX7Xau i6fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=EZuvC/6atMEYFimCXia/D65CMz0lLb/+Ul2WqyYI8+o=; b=egIgB2kkHcBjnGDH9P7uMjWNqeWLI7JQYFE7OlOtF3DIp8w/ch71dJwL7x6dT4Fcca SJS+4G/zIgkyTYq5vhR7cGdr3ywYKfhG2r1zw6XB3XBXJALsfuCmWnx64+rAq7eiHiFG 2E3kiK7gssyvg18CfIJJeOdoNbepQZpmLq77J78tzmNoqwk5O3SA/mcImcG4ATO38y4m mDr9uuer8QwXNhV7cWo8G/VXjo2U+sU3rlVqdoD1i35k38KdgTHd6rf8AEyNlLwhYEat FGmleFGnmZITXLB3aqtsY+uYcio7BZ+br2zOZheDt5UZ/Y1N2cNpCw8JTFu22UNak2uw bE5A== X-Gm-Message-State: AOAM5306gf3G+26jZVnm2WcmhTOeT2Xmh13MF1mYwGN/PtUhBcsZk8g+ R5gYGXV7XVqmS7U6OsPxyVHW4yHbh5rjhw== X-Google-Smtp-Source: ABdhPJxwi77ZOOhtJ+3v51697gD0YAdA3MuP0EjH79hp8pVriFW1w1Ork08r/qpnEs16MBLKv7aYWw== X-Received: by 2002:a37:b586:: with SMTP id e128mr18046827qkf.43.1626587984986; Sat, 17 Jul 2021 22:59:44 -0700 (PDT) Received: from hurd (dsl-148-66.b2b2c.ca. [66.158.148.66]) by smtp.gmail.com with ESMTPSA id m80sm3424473qke.98.2021.07.17.22.59.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Jul 2021 22:59:44 -0700 (PDT) From: Maxim Cournoyer References: <8a452809f56c9b62b46a7639ff89148ffc845b3e.1626493564.git.iskarian@mgsn.dev> Date: Sun, 18 Jul 2021 01:59:43 -0400 In-Reply-To: <8a452809f56c9b62b46a7639ff89148ffc845b3e.1626493564.git.iskarian@mgsn.dev> (Sarah Morgensen's message of "Fri, 16 Jul 2021 21:01:28 -0700") Message-ID: <87tuksjh68.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1626588015; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-to:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=EZuvC/6atMEYFimCXia/D65CMz0lLb/+Ul2WqyYI8+o=; b=iHfkBHY1gK0AAugr54PxvpGDGjjuWvayJpLdCyJhSAO3NlI46r6rOWqoLOXKasW4netXNf N0ATnOpTNzfjbDEF1aThuLPLTtw/k76YadvF8SjRLe2iX9jv/DZPO5V9TtDo0U+CiMQGDy vA/gKRejIthKpDsxcIMYcSfCilL63NdGZuxpltUXAwcjKBQyX1+uBLbW6OqSR1L13+CqpX oU+qa8sAx+PkxnZ7U6zO5q7P0CN+/yjuoGjItX5cp3RDlSFikMuKkO92xfN1bgj5ZJXHNV FDOCAhjtV/FyRrYrcQa/lcBS9CwcEuANQaM8dg1dlvk/qc2sHbrh3m6S3ZdzUg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1626588015; a=rsa-sha256; cv=none; b=NsGBBGtLM/vJQoU+wXzRjBNlLb9hkSIOORrMRCsAhKE9PEEp1ct/bhucx9FxFhM77oF06p klFPU/wguYBHIYMvtuXIoYB3P37BR9Mka4TSeK82RRH4pjFbzWIn1Wpy2oe+LhSrRlXdKe zgPn9WuI5+/5hnWaOgzfO9L8slNbSOTa0qtJbvKtHh1QbLBuDSqzWiwObXPdiyPEa8K1sU UIRTuCkw43ufNYIRPns5axIf7gqN/S5f2urBlYpYmdDUiDF0dEj+Lo/YtNk3+wUfxO0UPt UXt6TgpeWdH3Fzvs9kBxlviLaAkCY9kXGfPw8HMHfF3f20NCJGhSfksAHuyF8g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=HK4FWe8+; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.31 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=HK4FWe8+; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: EA67B1D800 X-Spam-Score: -1.31 X-Migadu-Scanner: scn0.migadu.com X-TUID: Ydww8aNMKK92 Hi! Sarah Morgensen writes: > Upgrade the go.mod parser to handle the full go.mod spec, and to > gracefully handle unexpected/malformed syntax. Restructure parser ^ There is a convention in Guix to add two spaces to separate the the ending (period) of a sentence from the beginning of the next one (which was inherited from GNU Standards, AFAICT: "Please put two spaces after the end of a sentence in your comments, so that the Emacs sentence commands will work.", c.f. info "(standards) Comments"). > usage, making the parse tree available for other uses. > > guix/import/go.scm (parse-go.mod): Parse using (ice-9 peg) instead of > regex matching for more robustness. Return a list of directives. > (go.mod-directives): New procedure. > (go.mod-requirements): New procedure. ^Likewise. > (go-module->guix-package): Use it. > (%go.mod-require-directive-rx) The above line can be removed, as it is duplicated below. > (%go.mod-replace-directive-rx): Remove unused variable. > tests/go.scm (testing-parse-mod): Adjust accordingly. > (go.mod-requirements) > (fixture-go-mod-unparseable) > (fixture-go-mod-retract) > (fixture-go-mod-strings): New variable. ^s. > ("parse-go.mod: simple") > ("parse-go.mod: comments and unparseable lines") > ("parse-go.mod: retract") > ("parse-go.mod: raw strings and quoted strings") > ("parse-go.mod: complete"): New tests. > --- > Hello Guix, > > This one is pretty self-explanatory. This parser handles the full go.mod spec > (notably: comments, extra whitespace, retract/exclude) and gracefully handles > things it doesn't understand. It is a bit more complex, I suppose. WDYT? This looks really neat! I'm happy to see more use of (ice-9 peg). I think the added complexity is worth it, given the benefits it provides. > -- Sarah > > guix/import/go.scm | 247 +++++++++++++++++++++++---------------------- > tests/go.scm | 132 +++++++++++++++++++++++- > 2 files changed, 260 insertions(+), 119 deletions(-) > > diff --git a/guix/import/go.scm b/guix/import/go.scm > index d8f838f635..69e71d01e5 100644 > --- a/guix/import/go.scm > +++ b/guix/import/go.scm > @@ -41,6 +41,7 @@ > #:autoload (guix base32) (bytevector->nix-base32-string) > #:autoload (guix build utils) (mkdir-p) > #:use-module (ice-9 match) > + #:use-module (ice-9 peg) > #:use-module (ice-9 rdelim) > #:use-module (ice-9 receive) > #:use-module (ice-9 regex) > @@ -244,129 +245,139 @@ and VERSION and return an input port." > (go-path-escape version)))) > (http-fetch* url))) > > -(define %go.mod-require-directive-rx > - ;; A line in a require directive is composed of a module path and > - ;; a version separated by whitespace and an optionnal '//' comment at > - ;; the end. > - (make-regexp > - (string-append > - "^[[:blank:]]*([^[:blank:]]+)[[:blank:]]+" ;the module path > - "([^[:blank:]]+)" ;the version > - "([[:blank:]]+//.*)?"))) ;an optional comment > - > -(define %go.mod-replace-directive-rx > - ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline > - ;; | ModulePath [ Version ] "=>" ModulePath Version newline . > - (make-regexp > - (string-append > - "([^[:blank:]]+)" ;the module path > - "([[:blank:]]+([^[:blank:]]+))?" ;optional version > - "[[:blank:]]+=>[[:blank:]]+" > - "([^[:blank:]]+)" ;the file or module path > - "([[:blank:]]+([^[:blank:]]+))?"))) ;the version (if a module path) > > (define (parse-go.mod content) > - "Parse the go.mod file CONTENT, returning a list of requirements." > - ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-grammar > - ;; which we think necessary for our use case. > - (define (toplevel requirements replaced) > - "This is the main parser. The results are accumulated in THE REQUIREMENTS > -and REPLACED lists." > - (let ((line (read-line))) > - (cond > - ((eof-object? line) > - ;; parsing ended, give back the result > - (values requirements replaced)) > - ((string=? line "require (") > - ;; a require block begins, delegate parsing to IN-REQUIRE > - (in-require requirements replaced)) > - ((string=? line "replace (") > - ;; a replace block begins, delegate parsing to IN-REPLACE > - (in-replace requirements replaced)) > - ((string-prefix? "require " line) > - ;; a require directive by itself > - (let* ((stripped-line (string-drop line 8))) > - (call-with-values > - (lambda () > - (require-directive requirements replaced stripped-line)) > - toplevel))) > - ((string-prefix? "replace " line) > - ;; a replace directive by itself > - (let* ((stripped-line (string-drop line 8))) > - (call-with-values > - (lambda () > - (replace-directive requirements replaced stripped-line)) > - toplevel))) > - (#t > - ;; unrecognised line, ignore silently > - (toplevel requirements replaced))))) > - > - (define (in-require requirements replaced) > - (let ((line (read-line))) > - (cond > - ((eof-object? line) > - ;; this should never happen here but we ignore silently > - (values requirements replaced)) > - ((string=? line ")") > - ;; end of block, coming back to toplevel > - (toplevel requirements replaced)) > - (#t > - (call-with-values (lambda () > - (require-directive requirements replaced line)) > - in-require))))) > - > - (define (in-replace requirements replaced) > - (let ((line (read-line))) > - (cond > - ((eof-object? line) > - ;; this should never happen here but we ignore silently > - (values requirements replaced)) > - ((string=? line ")") > - ;; end of block, coming back to toplevel > - (toplevel requirements replaced)) > - (#t > - (call-with-values (lambda () > - (replace-directive requirements replaced line)) > - in-replace))))) > - > - (define (replace-directive requirements replaced line) > - "Extract replaced modules and new requirements from the replace directive > -in LINE and add them to the REQUIREMENTS and REPLACED lists." > - (let* ((rx-match (regexp-exec %go.mod-replace-directive-rx line)) > - (module-path (match:substring rx-match 1)) > - (version (match:substring rx-match 3)) > - (new-module-path (match:substring rx-match 4)) > - (new-version (match:substring rx-match 6)) > - (new-replaced (cons (list module-path version) replaced)) > - (new-requirements > - (if (string-match "^\\.?\\./" new-module-path) > - requirements > - (cons (list new-module-path new-version) requirements)))) > - (values new-requirements new-replaced))) > - > - (define (require-directive requirements replaced line) > - "Extract requirement from LINE and augment the REQUIREMENTS and REPLACED > -lists." > - (let* ((rx-match (regexp-exec %go.mod-require-directive-rx line)) > - (module-path (match:substring rx-match 1)) > - ;; Double-quoted strings were seen in the wild without escape > - ;; sequences; trim the quotes to be on the safe side. > - (module-path (string-trim-both module-path #\")) > - (version (match:substring rx-match 2))) > - (values (cons (list module-path version) requirements) replaced))) > - > - (with-input-from-string content > - (lambda () > - (receive (requirements replaced) > - (toplevel '() '()) > - ;; At last remove the replaced modules from the requirements list. > - (remove (lambda (r) > - (assoc (car r) replaced)) > - requirements))))) > + "Parse the go.mod file CONTENT, returning a list of directives, comments, > +and unknown lines. Each sublist begins with a symbol (go, module, require, > +replace, exclude, retract, comment, or unknown) and is followed by one or more > +sublists. Each sublist begins with a symbol (module-path, version, file-path, > +comment, or unknown) and is followed by the indicated data." > + ;; https://golang.org/ref/mod#go-mod-file-grammar > + (define-peg-pattern NL none "\n") > + (define-peg-pattern WS none (or " " "\t" "\r")) > + (define-peg-pattern => none (and (* WS) "=>")) > + (define-peg-pattern punctuation none (or "," "=>" "[" "]" "(" ")")) > + (define-peg-pattern comment all > + (and (ignore "//") (* WS) (* (and (not-followed-by NL) peg-any)))) > + (define-peg-pattern EOL body (and (* WS) (? comment) NL)) > + (define-peg-pattern block-start none (and (* WS) "(" EOL)) > + (define-peg-pattern block-end none (and (* WS) ")" EOL))w > + (define-peg-pattern any-line body > + (and (* WS) (* (and (not-followed-by NL) peg-any)) EOL)) > + > + ;; Strings and identifiers > + (define-peg-pattern identifier body > + (+ (and (not-followed-by (or NL WS punctuation)) peg-any))) > + (define-peg-pattern string-raw body > + (and (ignore "`") (+ (and (not-followed-by "`") peg-any)) (ignore "`"))) > + (define-peg-pattern string-quoted body > + (and (ignore "\"") > + (+ (or (and (ignore "\\") peg-any) > + (and (not-followed-by "\"") peg-any))) > + (ignore "\""))) > + (define-peg-pattern string-or-ident body ^ Perhaps prefer the fully spelled out 'string-or-identifier' variable name. > + (and (* WS) (or string-raw string-quoted identifier))) > + > + (define-peg-pattern version all string-or-ident) > + (define-peg-pattern module-path all string-or-ident) > + (define-peg-pattern file-path all string-or-ident) > + > + ;; Non-directive lines > + (define-peg-pattern unknown all any-line) > + (define-peg-pattern block-line body > + (or EOL (and (not-followed-by block-end) unknown))) > + > + ;; GoDirective = "go" GoVersion newline . > + (define-peg-pattern go all (and (ignore "go") version EOL)) > + > + ;; ModuleDirective = "module" ( ModulePath | "(" newline ModulePath newline ")" ) newline . > + (define-peg-pattern module all > + (and (ignore "module") (or (and block-start module-path EOL block-end) > + (and module-path EOL)))) > + > + ;; The following directives may all be used solo or in a block > + ;; RequireSpec = ModulePath Version newline . > + (define-peg-pattern require all (and module-path version EOL)) > + (define-peg-pattern require-top body > + (and (ignore "require") > + (or (and block-start (* (or require block-line)) block-end) require))) > + > + ;; ExcludeSpec = ModulePath Version newline . > + (define-peg-pattern exclude all (and module-path version EOL)) > + (define-peg-pattern exclude-top body > + (and (ignore "exclude") > + (or (and block-start (* (or exclude block-line)) block-end) exclude))) > + > + ;; ReplaceSpec = ModulePath [ Version ] "=>" FilePath newline > + ;; | ModulePath [ Version ] "=>" ModulePath Version newline . > + (define-peg-pattern original all (or (and module-path version) module-path)) > + (define-peg-pattern with all (or (and module-path version) file-path)) > + (define-peg-pattern replace all (and original => with EOL)) > + (define-peg-pattern replace-top body > + (and (ignore "replace") > + (or (and block-start (* (or replace block-line)) block-end) replace))) > + > + ;; RetractSpec = ( Version | "[" Version "," Version "]" ) newline . > + (define-peg-pattern range all > + (and (* WS) (ignore "[") version > + (* WS) (ignore ",") version (* WS) (ignore "]"))) > + (define-peg-pattern retract all (and (or range version) EOL)) > + (define-peg-pattern retract-top body > + (and (ignore "retract") > + (or (and block-start (* (or retract block-line)) block-end) retract))) > + > + (define-peg-pattern go-mod body > + (* (and (* WS) (or go module require-top exclude-top replace-top > + retract-top EOL unknown)))) > + > + (let ((tree (peg:tree (match-pattern go-mod content))) > + (keywords '(go module require replace exclude retract comment unknown))) > + (keyword-flatten keywords tree))) Fun! > ;; Prevent inlining of this procedure, which is accessed by unit tests. > (set! parse-go.mod parse-go.mod) > > +(define (go.mod-directives go.mod directive) > + "Return the list of top-level directive bodies in GO.MOD matching the symbol > +DIRECTIVE." > + (filter-map (match-lambda > + (((? (cut eq? <> directive) head) . rest) rest) > + (_ #f)) > + go.mod)) > + > +(define (go.mod-requirements go.mod) > + "Compute and return the list of requirements specified by GO.MOD." > + (define (replace directive requirements) > + (define (maybe-replace module-path new-requirement) > + ;; Do not allow version updates for indirect dependencies > + ;; TODO: Is this correct behavior? It's in the go.mod for a > reason... According to [1], it seems that yes: "replace directives only apply in the main module's go.mod file and are ignored in other modules.", IIUC. [1] https://golang.org/ref/mod#go-mod-file-replace > + (if (and (equal? module-path (first new-requirement)) > + (not (assoc-ref requirements module-path))) > + requirements > + (cons new-requirement (alist-delete module-path requirements)))) > + > + (match directive > + ((('original ('module-path module-path) . _) with . _) > + (match with > + (('with ('file-path _) . _) > + (alist-delete module-path requirements)) > + (('with ('module-path new-module-path) ('version new-version) . _) > + (maybe-replace module-path > + (list new-module-path new-version))))))) > + > + (define (require directive requirements) > + (match directive > + ((('module-path module-path) ('version version) . _) > + (cons (list module-path version) requirements)))) > + > + (let* ((requires (go.mod-directives go.mod 'require)) > + (replaces (go.mod-directives go.mod 'replace)) > + (requirements (fold require '() requires))) > + (fold replace requirements replaces))) > + > +;; Prevent inlining of this procedure, which is accessed by unit tests. > +(set! go.mod-requirements go.mod-requirements) > + > (define-record-type > (%make-vcs url-prefix root-regex type) > vcs? > @@ -588,7 +599,7 @@ When VERSION is unspecified, the latest version available is used." > hint: use one of the following available versions ~a\n" > version* available-versions)))) > (content (fetch-go.mod goproxy module-path version*)) > - (dependencies+versions (parse-go.mod content)) > + (dependencies+versions (go.mod-requirements (parse-go.mod content))) > (dependencies (if pin-versions? > dependencies+versions > (map car dependencies+versions))) > diff --git a/tests/go.scm b/tests/go.scm > index b088ab50d2..b16965d941 100644 > --- a/tests/go.scm > +++ b/tests/go.scm > @@ -1,5 +1,6 @@ > ;;; GNU Guix --- Functional package management for GNU > ;;; Copyright 2021 Franois Joulaud > +;;; Copyright 2021 Sarah Morgensen > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -31,6 +32,9 @@ > #:use-module (srfi srfi-64) > #:use-module (web response)) > > +(define go.mod-requirements > + (@@ (guix import go) go.mod-requirements)) > + > (define parse-go.mod > (@@ (guix import go) parse-go.mod)) > > @@ -96,6 +100,40 @@ replace ( > > ") > > +(define fixture-go-mod-unparseable > + "module my/thing > +go 1.12 // avoid feature X > +require other/thing v1.0.2 > +// Security issue: CVE-XXXXX > +exclude old/thing v1.2.3 > +new-directive another/thing yet-another/thing > +replace ( > + bad/thing v1.4.5 => good/thing v1.4.5 > + // Unparseable > + bad/thing [v1.4.5, v1.9.7] => good/thing v2.0.0 > +) > +") > + > +(define fixture-go-mod-retract > + "retract v0.9.1 > + > +retract ( > + v1.9.2 > + [v1.0.0, v1.7.9] > +) > +") > + > +(define fixture-go-mod-strings > + "require `example.com/\"some-repo\"` v1.9.3 > +require ( > + `example.com/\"another.repo\"` v1.0.0 > + \"example.com/special!repo\" v9.3.1 > +) > +replace \"example.com/\\\"some-repo\\\"\" => `launchpad.net/some-repo` v1.9.3 > +replace ( > + \"example.com/\\\"another.repo\\\"\" => launchpad.net/another-repo v1.0.0 > +) > +") > > > (define fixture-latest-for-go-check > @@ -185,7 +223,7 @@ require github.com/kr/pretty v0.2.1 > (string (test-equal name > (sort expected inf?) > - (sort ((@@ (guix import go) parse-go.mod) input) inf?))) > + (sort (go.mod-requirements (parse-go.mod input)) inf?))) > > (testing-parse-mod "parse-go.mod-simple" > '(("good/thing" "v1.4.5") > @@ -221,6 +259,98 @@ require github.com/kr/pretty v0.2.1 > ("github.com/go-check/check" "v0.0.0-20140225173054-eb6ee6f84d0a")) > fixture-go-mod-complete) > > +(test-equal "parse-go.mod: simple" > + `((module (module-path "my/thing")) > + (go (version "1.12")) > + (require (module-path "other/thing") (version "v1.0.2")) > + (require (module-path "new/thing/v2") (version "v2.3.4")) > + (exclude (module-path "old/thing") (version "v1.2.3")) > + (replace (original (module-path "bad/thing") (version "v1.4.5")) > + (with (module-path "good/thing") (version "v1.4.5")))) > + (parse-go.mod fixture-go-mod-simple)) > + > +(test-equal "parse-go.mod: comments and unparseable lines" > + `((module (module-path "my/thing")) > + (go (version "1.12") (comment "avoid feature X")) > + (require (module-path "other/thing") (version "v1.0.2")) > + (comment "Security issue: CVE-XXXXX") > + (exclude (module-path "old/thing") (version "v1.2.3")) > + (unknown "new-directive another/thing yet-another/thing") > + (replace (original (module-path "bad/thing") (version "v1.4.5")) > + (with (module-path "good/thing") (version "v1.4.5"))) > + (comment "Unparseable") > + (unknown "bad/thing [v1.4.5, v1.9.7] => good/thing v2.0.0")) > + (parse-go.mod fixture-go-mod-unparseable)) > + > +(test-equal "parse-go.mod: retract" > + `((retract (version "v0.9.1")) > + (retract (version "v1.9.2")) > + (retract (range (version "v1.0.0") (version "v1.7.9")))) > + (parse-go.mod fixture-go-mod-retract)) > + > +(test-equal "parse-go.mod: raw strings and quoted strings" > + `((require (module-path "example.com/\"some-repo\"") (version "v1.9.3")) > + (require (module-path "example.com/\"another.repo\"") (version "v1.0.0")) > + (require (module-path "example.com/special!repo") (version "v9.3.1")) > + (replace (original (module-path "example.com/\"some-repo\"")) > + (with (module-path "launchpad.net/some-repo") (version "v1.9.3"))) > + (replace (original (module-path "example.com/\"another.repo\"")) > + (with (module-path "launchpad.net/another-repo") (version "v1.0.0")))) > + (parse-go.mod fixture-go-mod-strings)) > + > +(test-equal "parse-go.mod: complete" > + `((module (module-path "M")) > + (go (version "1.13")) > + (replace (original (module-path "github.com/myname/myproject/myapi")) > + (with (file-path "./api"))) > + (replace (original (module-path "github.com/mymname/myproject/thissdk")) > + (with (file-path "../sdk"))) > + (replace (original (module-path "launchpad.net/gocheck")) > + (with (module-path "github.com/go-check/check") > + (version "v0.0.0-20140225173054-eb6ee6f84d0a"))) > + (require (module-path "github.com/user/project") > + (version "v1.1.11")) > + (require (module-path "github.com/user/project/sub/directory") > + (version "v1.1.12")) > + (require (module-path "bitbucket.org/user/project") > + (version "v1.11.20")) > + (require (module-path "bitbucket.org/user/project/sub/directory") > + (version "v1.11.21")) > + (require (module-path "launchpad.net/project") > + (version "v1.1.13")) > + (require (module-path "launchpad.net/project/series") > + (version "v1.1.14")) > + (require (module-path "launchpad.net/project/series/sub/directory") > + (version "v1.1.15")) > + (require (module-path "launchpad.net/~user/project/branch") > + (version "v1.1.16")) > + (require (module-path "launchpad.net/~user/project/branch/sub/directory") > + (version "v1.1.17")) > + (require (module-path "hub.jazz.net/git/user/project") > + (version "v1.1.18")) > + (require (module-path "hub.jazz.net/git/user/project/sub/directory") > + (version "v1.1.19")) > + (require (module-path "k8s.io/kubernetes/subproject") > + (version "v1.1.101")) > + (require (module-path "one.example.com/abitrary/repo") > + (version "v1.1.111")) > + (require (module-path "two.example.com/abitrary/repo") > + (version "v0.0.2")) > + (require (module-path "quoted.example.com/abitrary/repo") > + (version "v0.0.2")) > + (replace (original (module-path "two.example.com/abitrary/repo")) > + (with (module-path "github.com/corp/arbitrary-repo") > + (version "v0.0.2"))) > + (replace (original (module-path "golang.org/x/sys")) > + (with (module-path "golang.org/x/sys") > + (version "v0.0.0-20190813064441-fde4db37ae7a")) > + (comment "pinned to release-branch.go1.13")) > + (replace (original (module-path "golang.org/x/tools")) > + (with (module-path "golang.org/x/tools") > + (version "v0.0.0-20190821162956-65e3620a7ae7")) > + (comment "pinned to release-branch.go1.13"))) > + (parse-go.mod fixture-go-mod-complete)) > + > ;;; End-to-end tests for (guix import go) > (define (mock-http-fetch testcase) > (lambda (url . rest) > > base-commit: 3ee0f170c8bd883728d8abb2c2e00f445c13f17d LGTM! Pushed with commit 793ba333c6, after fixing up some indents and substituting the TODO for an explanatory comment with a reference to the replace directive, and a few more cosmetic changes. Thanks for the continued improvements to the Go importer :-). Closing. Maxim