From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id CCEFBHRDFGC8awAA0tVLHw (envelope-from ) for ; Fri, 29 Jan 2021 17:18:44 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id KKhnO3NDFGByFAAA1q6Kng (envelope-from ) for ; Fri, 29 Jan 2021 17:18:43 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2FA399403C9 for ; Fri, 29 Jan 2021 17:18:43 +0000 (UTC) Received: from localhost ([::1]:40636 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5XPt-0001zp-QS for larch@yhetil.org; Fri, 29 Jan 2021 12:18:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42970) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X14-0002II-9U for guix-patches@gnu.org; Fri, 29 Jan 2021 11:53:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:41325) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l5X14-0007nL-0Q for guix-patches@gnu.org; Fri, 29 Jan 2021 11:53:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l5X13-0001AA-UW for guix-patches@gnu.org; Fri, 29 Jan 2021 11:53:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#44178] [PATCHv2] Create importer for Go modules Resent-From: JOULAUD =?UTF-8?Q?Fran=C3=A7ois?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 29 Jan 2021 16:53:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44178 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: "44178@debbugs.gnu.org" <44178@debbugs.gnu.org> Received: via spool by 44178-submit@debbugs.gnu.org id=B44178.16119391394404 (code B ref 44178); Fri, 29 Jan 2021 16:53:01 +0000 Received: (at 44178) by debbugs.gnu.org; 29 Jan 2021 16:52:19 +0000 Received: from localhost ([127.0.0.1]:52871 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5X0D-00018e-C1 for submit@debbugs.gnu.org; Fri, 29 Jan 2021 11:52:19 -0500 Received: from mx07-00115501.pphosted.com ([185.132.182.48]:41916 helo=mx08-00115501.pphosted.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5X0A-00018S-Sd for 44178@debbugs.gnu.org; Fri, 29 Jan 2021 11:52:08 -0500 Received: from pps.filterd (m0001505.ppops.net [127.0.0.1]) by mx07-00115501.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 10TGjF1R022815 for <44178@debbugs.gnu.org>; Fri, 29 Jan 2021 17:52:05 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=radiofrance.com; h=from : to : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=radiofrance20190306; bh=A2tAJyG7PGX6eZE66F0SE7XwtkHHRJ63IU6ic7Q+oE4=; b=L6/E/6UyJndijpRIARoLlxgTetCPgchC2JGKxryJmiqDXbsGW5zSSVZwPGXNav+gDQZ4 rf0rnZgpg4oXMkv+LOrggrOQha0lh07i9Wqk2kwc7gNiFf5MFf4kkvkzx+z32fNZjTjD +I7jRYpYdPv/L1D2C7Uydi0k5EM8xsDXtuFvVPWG7cEqbdtLo5NaNpm2BzN69v2+NcFt p+x5HhFpJe0T2qsyzqNu0HyXyW7sD0k0GECA51K1addapckyz8jNNcm2n78gQwaqbgW/ pohyfsw/bf1BjHZ2swUbB5Fc+FjB19IgrK68XklHlUsk5OYshiou6dwP74V06ekgZtF5 4g== Received: from fra01-pr2-obe.outbound.protection.outlook.com (mail-pr2fra01on0108.outbound.protection.outlook.com [104.47.24.108]) by mx07-00115501.pphosted.com with ESMTP id 36895rjwty-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for <44178@debbugs.gnu.org>; Fri, 29 Jan 2021 17:52:04 +0100 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=O5fWt0xE0sjizVYjQNrSVrFA8ewM2NQJofTDAgDSh4eslQdFLbWXfPK019j8Jhiz0AFHE8lRRVPyzlsKcrd45LG8/p3uxjI8JjckDtcV8u2BclcY8ztSfb0USNQyDcIoUyTZ/2zTz7iDcYO44Gdnfa+z+UAxGV+6dMVUNurcyx9oRBB2P8tqDQT50uM2DupzQQEKblI1in0r1DeF+DDRCEdlYvFgV/gTechQYu45PKh2YPFbIDUpW2sgHNQlJ50mC4+9+gsWkRt6aapABl8BGO+ezL3Td/oGFs7OsaXzyuVJXUVqhXq0drMMJiu98qlsWLGZYrVdESfj9RhCoFW26A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=A2tAJyG7PGX6eZE66F0SE7XwtkHHRJ63IU6ic7Q+oE4=; b=hYmV/HKoS+nJIJQaIeCBJHyEaf0G4i/F5vtgX8sQT72nOWqZoVm/oUd4qogAtmR8uriI5z3tK6g5w031K0ViNv2zdW1LL+U2uT2MDnczpUmEs9IWQswi9fED15R39lHyLNhH1P7yQ8SxiCkofv/rvw8kf4pdYhI5bUPShdpHXI4VTDZBwVULkIrqWXDTpgYmS0MhoBYJ8FeouhXvNiVc4G9790J5j07b6pJ5N43yJkiz+j7f3bvTK3F6FPO0bEp0BOAM2gwXY4uuxZhA7cMEC2J581Bb4KkrChUgxR/9XvnzDeR7kkG0YaWnbKl4F4kEd5TlLIFS6RFLHhxG9O05XA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=radiofrance.com; dmarc=pass action=none header.from=radiofrance.com; dkim=pass header.d=radiofrance.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RFonline.onmicrosoft.com; s=selector2-RFonline-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=A2tAJyG7PGX6eZE66F0SE7XwtkHHRJ63IU6ic7Q+oE4=; b=ZwHqKLQChaXxjaVtjen4QAAmIdL2TQPdiCwhdN/G/dqB9UF6OinQpyqmS7GeCgY5EgtVRlkHi3YVTjsVQaa170LGQM9KBb4fNvC9GyPVO0tRCDsavrgsBkBs35MiN3ngEM1uigWEy0MB3xS8lLOrKRDWHIm7ER9AP6BHNY0jsDQ= Received: from PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM (2603:10a6:100:b::20) by PR0P264MB0572.FRAP264.PROD.OUTLOOK.COM (2603:10a6:100:4::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3742.12; Fri, 29 Jan 2021 16:52:03 +0000 Received: from PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM ([fe80::51bc:289d:ff1a:6b58]) by PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM ([fe80::51bc:289d:ff1a:6b58%6]) with mapi id 15.20.3784.017; Fri, 29 Jan 2021 16:52:03 +0000 Thread-Topic: [PATCHv2] Create importer for Go modules Thread-Index: AQHW9l8RwPhX3/Q56k6NvVPLE/X+iw== Date: Fri, 29 Jan 2021 16:52:02 +0000 Message-ID: <20210129164827.vrrty5gmi4paf7xv@fjo-extia-HPdeb.example.avalenn.eu> References: <20210129163945.irrdlm3updejkcsg@fjo-extia-HPdeb.example.avalenn.eu> In-Reply-To: <20210129163945.irrdlm3updejkcsg@fjo-extia-HPdeb.example.avalenn.eu> Accept-Language: fr-FR, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [88.126.13.52] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3b60d1ec-9488-4a48-add1-08d8c47633fb x-ms-traffictypediagnostic: PR0P264MB0572: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4502; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Rjs3Xx1RQMyvZ+se3YobzuhAeRpJkM05hqJ46dhuy7XbKjCKYVbs6+fnsRv602gz+9dsEJTwvDLuEmng1msyeIqJaF69tBKDWF7RCYuUcFLEohsc9mvi/sGCQij/RHLd+qh7IJ/r0NJOduNxZVXH1Jg+dbhkPij+myGKQxZ0KioDfRu9UFF6d38jitMUjNR6phB5WGUJsyza44Y4bqWu2hezN8RX60l+eV6mSu+Kt9SylPBsUVy9mbZn9LXsTtTjJcgQ+MqOJYzzYZr687JuSEeBQtCj3Rw/1RiiWOEFmrR2MsglZeKu0hXeTUXNvfNikxYQVV1wWHpmsR5SAmmAs/fipGBhlEy12YHhCB2jNl80o/Mcf/mnGwKTvYSeEef6RBs9uk3GeXMEzMDO5wf9V1Zwb+ar8OeviNGp/Yb9q4uArG3WvlOmQKBJKJEW4bQsi+CJ6jdMioB5FSmvopbKu5pnhPojIdhm7WPZOwyih3grKypCmDAlEqeD/VKm5DgfhNTnUgRub+nA1f6ZQly9uWDny7jYOkSwyeOyowYplN5KtRrSMXuf59+J4lP97FXANNro3np88SoJ1eV8IanKs34RTWKz8HrBbDb2Wf12vO/ixphI3N33LGfrVS9jvsOv6WMLjK2Sab3AVMkoqIk3tQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(4636009)(376002)(39860400002)(396003)(346002)(136003)(366004)(66476007)(66574015)(76116006)(66946007)(6486002)(66556008)(966005)(5660300002)(186003)(478600001)(86362001)(66446008)(26005)(71200400001)(8936002)(2906002)(316002)(64756008)(9686003)(6512007)(83380400001)(30864003)(6506007)(1076003)(6916009)(8676002)(2004002)(579004)(559001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: 5Rot5tK4sjd+E4mAXcKmoTIxGxs+xHdsPq/o/YFO7lIx8XZ4X16sO7D0C0+S4FdHUoZ+LDckJP8ZJUEZNozCHoaYr7oJYQUocx+6/qaCAeW44iMxRfXZ31btcvOGVMWKTa3tNHvlPzKSUN+5dDMhrSbNBAIZ8s0W3jr30snhp42j47aubQUuYFb+pP1bB+qIzJe50Xxtv9EhHs0ZYC38GuUZdUl5Es1hfNdeMdYlG7UT0oW8f709sx4DQ8xxbxHOSQ2zhDIYpsnKy10PZI4odsL2ZInTSyLBmpdqGBeQmrL84hJBD5WdomTqmS2SozlsZOAbwOvXK+vakYKNcilbU6kUQMeShfXEoTDuKFpnQBOQzWXhacaIiyqCzRuA8jwXJHyrR3kq1E19Dm+SzgKqs266uOcaXyh4JcTmS0b0wWftQB9stQLMaD/U29bBOJovH6sos8Wtf5YM6iQpXBnJ2LFf6/wQxxTeFQe+G33WdtPaiAw1VoJUYzi47nD38UWCwA/AuvoIWt3ee8xkKE4oN7bgBqq+u2K7R0+7TK/oY7gTdJR/nbzEhUOt8Ido2roLaJJAZvbyj9TS1j/yH1yimIUqyuRBQd/rm/3A3ECyyC9/Bhvf04wdCiNPEqpwMdQmBqSVq9RaTxgyjy40Rp0Jl0P+4b45gTwowWtt/XAfOEJvkgfudVz689yTxZ9t5a6Z2ZHSAMx0sPJnLcubBPLvkMyNgho7U3C7WcuTZoeHi4hSUdPLVzdmSPFQg/xz4PUlHgpRC1GID+xA3y+TE7SKB5neBVY2NZsRLA9bDsyT2q63wznbTWmxVHeFD60b0agXAanBPzDQmCa4zPDkEQycvQ/fVibixG0ahMHFfGq7TCmpS1g8BdMzJrGu7Y9JvjbCAa8DKU4DMV6nzxbx80HOMIdiSs109cJvPZFqj2Yo4H2+D1PYWBoiZqugjqfIaNSeQn SXcf8oCruXeHunlv8Jl8LKmtCSRqzNCppcCIiV5mwzCF6/TUfNN0S1OH8dCnp4rALEHGZbr+GRxz5hb9MhygXIhqTvlHLWHh+bDCaYhNb3HrNqhfBiPIlCjtRjg8/c Content-Type: text/plain; charset="iso-8859-1" Content-ID: <24A251FD2E97CE4A826BCA2AB96C6DA5@FRAP264.PROD.OUTLOOK.COM> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: radiofrance.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PR0P264MB0425.FRAP264.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 3b60d1ec-9488-4a48-add1-08d8c47633fb X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jan 2021 16:52:02.9592 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 1d643b07-8cf5-4e2d-ad1e-86d6b948fc3b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: v3OzlCL/s/lNiQeH4SZ8eDFQLvq9ReNt3/5PZjY+8SofwRnnqnyTeSJp6USc4e0H5UczxPzkrYPhoRRPoArOSYtky4F3t3cnwsvIgXNQQoYSwbqmZXAnBJp/lQ9t1P/i X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR0P264MB0572 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.737 definitions=2021-01-29_06:2021-01-29, 2021-01-29 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 clxscore=1015 mlxscore=0 suspectscore=0 adultscore=0 impostorscore=0 priorityscore=1501 spamscore=0 phishscore=0 lowpriorityscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101290082 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" Reply-to: JOULAUD =?UTF-8?Q?Fran=C3=A7ois?= X-ACL-Warn: , JOULAUD =?UTF-8?Q?Fran=C3=A7ois?= via Guix-patches From: guix-patches--- via X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -0.85 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=radiofrance.com header.s=radiofrance20190306 header.b="L6/E/6Uy"; dkim=fail ("headers rsa verify failed") header.d=RFonline.onmicrosoft.com header.s=selector2-RFonline-onmicrosoft-com header.b=ZwHqKLQC; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:reject}"); dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 2FA399403C9 X-Spam-Score: -0.85 X-Migadu-Scanner: scn1.migadu.com X-TUID: Izv/4harg1wP This patch add a `guix import go` command. It was tested with several big repositories and seems to mostly work for the import part (because building Guix packages is an other story). * guix/import/go.sc: Created Go Importerm * guix/scripts/import.scm: Added Go Importer Subcommand * guix/scripts/import/go.scm: Created Go Importer Subcommand * doc/guix.texi: add a paragraph about `guix import go` * tests/import-go.scm: tests for parse-go.mod procedure Signed-off-by: Francois Joulaud --- doc/guix.texi | 25 +++ guix/import/go.scm | 384 +++++++++++++++++++++++++++++++++++++ guix/scripts/import.scm | 2 +- guix/scripts/import/go.scm | 118 ++++++++++++ tests/import-go.scm | 143 ++++++++++++++ 5 files changed, 671 insertions(+), 1 deletion(-) create mode 100644 guix/import/go.scm create mode 100644 guix/scripts/import/go.scm create mode 100644 tests/import-go.scm diff --git a/doc/guix.texi b/doc/guix.texi index 6ea782fd23..d77e2811ae 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -860,6 +860,10 @@ substitutes (@pxref{Invoking guix publish}). @uref{https://ngyro.com/software/guile-semver.html, Guile-Semver} for the @code{crate} importer (@pxref{Invoking guix import}). =20 +@item +@uref{https://www.nongnu.org/guile-lib/doc/ref/htmlprag/, guile-lib} for +the @code{crate} importer (@pxref{Invoking guix import}). + @item When @url{http://www.bzip.org, libbz2} is available, @command{guix-daemon} can use it to compress build logs. @@ -11370,6 +11374,27 @@ Select the given repository (a repository name). = Possible values include: of coq packages. @end itemize @end table + +@item go +@cindex go +Import metadata for a Go module using +@uref{https://proxy.golang.org, proxy.golang.org}. + +This importer is highly experimental. + +@example +guix import go gopkg.in/yaml.v2 +@end example + +Additional options include: + +@table @code +@item --recursive +@itemx -r +Traverse the dependency graph of the given upstream package recursively +and generate package expressions for all those packages that are not yet +in Guix. +@end table @end table =20 The structure of the @command{guix import} code is modular. It would be diff --git a/guix/import/go.scm b/guix/import/go.scm new file mode 100644 index 0000000000..cf2d31ce12 --- /dev/null +++ b/guix/import/go.scm @@ -0,0 +1,384 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =A9 2020 Katherine Cox-Buday +;;; Copyright =A9 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> +;;; Copyright =A9 2021 Fran=E7ois Joulaud +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;;; (guix import golang) wants to make easier to create Guix package +;;; declaration for Go modules. +;;; +;;; Modules in Go are "collection of related Go packages" which are +;;; "the unit of source code interchange and versioning". +;;; Modules are generally hosted in a repository. +;;; +;;; At this point it should handle correctly modules which +;;; - have only Go dependencies; +;;; - use go.mod; +;;; - and are accessible from proxy.golang.org (or configured GOPROXY). +;;; +;;; We translate Go module paths to a Guix package name under the +;;; assumption that there will be no collision. + +(define-module (guix import go) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) + #:use-module (ice-9 regex) + #:use-module (htmlprag) + #:use-module (sxml xpath) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-9) + #:use-module (srfi srfi-11) + #:use-module (json) + #:use-module ((guix download) #:prefix download:) + #:use-module (guix import utils) + #:use-module (guix import json) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix base16) + #:use-module (guix base32) + #:use-module ((guix build download) #:prefix build-download:) + #:use-module (web uri) + + #:export (go-module->guix-package + go-module-recursive-import + infer-module-root)) + +(define (go-path-escape path) + "Escape a module path by replacing every uppercase letter with an exclam= ation +mark followed with its lowercase equivalent, as per the module Escaped Pat= hs +specification. https://godoc.org/golang.org/x/mod/module#hdr-Escaped_Paths= " + (define (escape occurrence) + (string-append "!" (string-downcase (match:substring occurrence)))) + (regexp-substitute/global #f "[A-Z]" path 'pre escape 'post)) + + +(define (fetch-latest-version goproxy-url module-path) + "Fetches the version number of the latest version for MODULE-PATH from t= he +given GOPROXY-URL server." + (assoc-ref + (json-fetch (format #f "~a/~a/@latest" goproxy-url + (go-path-escape module-path))) + "Version")) + +(define (fetch-go.mod goproxy-url module-path version file) + "Fetches go.mod from the given GOPROXY-URL server for the given MODULE-P= ATH +and VERSION." + (let ((url (format #f "~a/~a/@v/~a.mod" goproxy-url + (go-path-escape module-path) + (go-path-escape version)))) + (parameterize ((current-output-port (current-error-port))) + (build-download:url-fetch url + file + #:print-build-trace? #f)))) + +(define (parse-go.mod go.mod-path) + (parse-go.mod-port (open-input-file go.mod-path))) + +(define (parse-go.mod-port go.mod-port) + "PARSE-GO.MOD takes a filename in GO.MOD-PATH and extract a list of +requirements from it." + ;; We parse only a subset of https://golang.org/ref/mod#go-mod-file-gram= mar + ;; which we think necessary for our use case. + (define (toplevel results) + "Main parser, RESULTS is a pair of alist serving as accumulator for + all encountered requirements and replacements." + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; parsing ended, give back the result + results) + ((string=3D? line "require (") + ;; a require block begins, delegate parsing to IN-REQUIRE + (in-require results)) + ((string=3D? line "replace (") + ;; a replace block begins, delegate parsing to IN-REPLACE + (in-replace results)) + ((string-prefix? "require " line) + ;; a require directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (require-directive results stripped-line))) + (toplevel new-results))) + ((string-prefix? "replace " line) + ;; a replace directive by itself + (let* ((stripped-line (string-drop line 8)) + (new-results (replace-directive results stripped-line))) + (toplevel new-results))) + (#t + ;; unrecognised line, ignore silently + (toplevel results))))) + (define (in-require results) + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; this should never happen here but we ignore silently + results) + ((string=3D? line ")") + ;; end of block, coming back to toplevel + (toplevel results)) + (#t + (in-require (require-directive results line)))))) + (define (in-replace results) + (let ((line (read-line))) + (cond + ((eof-object? line) + ;; this should never happen here but we ignore silently + results) + ((string=3D? line ")") + ;; end of block, coming back to toplevel + (toplevel results)) + (#t + (in-replace (replace-directive results line)))))) + (define (replace-directive results line) + "Extract replaced modules and new requirements from replace directive + in LINE and add to RESULTS." + ;; ReplaceSpec =3D ModulePath [ Version ] "=3D>" FilePath newline + ;; | ModulePath [ Version ] "=3D>" ModulePath Version newl= ine . + (let* ((requirements (car results)) + (replaced (cdr results)) + (re (string-concatenate + '("([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?" + "[[:blank:]]+" "=3D>" "[[:blank:]]+" + "([^[:blank:]]+)([[:blank:]]+([^[:blank:]]+))?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 3)) + (new-module-path (match:substring match 4)) + (new-version (match:substring match 6)) + (new-replaced (acons module-path version replaced)) + (new-requirements + (if (string-match "^\\.?\\./" new-module-path) + requirements + (acons new-module-path new-version requirements)))) + (cons new-requirements new-replaced))) + (define (require-directive results line) + "Extract requirement from LINE and add it to RESULTS." + (let* ((requirements (car results)) + (replaced (cdr results)) + ;; A line in a require directive is composed of a module path a= nd + ;; a version separated by whitespace and an optionnal '//' comm= ent at + ;; the end. + (re (string-concatenate + '("^[[:blank:]]*" + "([^[:blank:]]+)[[:blank:]]+([^[:blank:]]+)" + "([[:blank:]]+//.*)?"))) + (match (string-match re line)) + (module-path (match:substring match 1)) + (version (match:substring match 2))) + (cons (acons module-path version requirements) replaced))) + (with-input-from-port go.mod-port + (lambda () + (let* ((results (toplevel '(() . ()))) + (requirements (car results)) + (replaced (cdr results))) + ;; At last we remove replaced modules from the requirements list + (fold + (lambda (replacedelem requirements) + (alist-delete! (car replacedelem) requirements)) + requirements + replaced))))) + +(define (infer-module-root module-path) + "Go modules can be defined at any level of a repository's tree, but quer= ying +for the meta tag usually can only be done at the webpage at the root of th= e +repository. Therefore, it is sometimes necessary to try and derive a modul= e's +root path from its path. For a set of well-known forges, the pattern of wh= at +consists of a module's root page is known before hand." + ;; See the following URL for the official Go equivalent: + ;; https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b9= 9f910/src/cmd/go/internal/vcs/vcs.go#L1026-L1087 + ;; + ;; TODO: handle module path with VCS qualifier as described in + ;; https://golang.org/ref/mod#vcs-find and + ;; https://golang.org/cmd/go/#hdr-Remote_import_paths + (define-record-type + (make-vcs url-prefix root-regex type) + vcs? + (url-prefix vcs-url-prefix) + (root-regex vcs-root-regex) + (type vcs-type)) + (let* ((known-vcs + (list + (make-vcs + "github.com" + "^(github\\.com/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0= -9_.\\-]+)*$" + 'git) + (make-vcs + "bitbucket.org" + "^(bitbucket\\.org/([A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+))(/[A-= Za-z0-9_.\\-]+)*$" + 'unknown) + (make-vcs + "hub.jazz.net/git/" + "^(hub\\.jazz\\.net/git/[a-z0-9]+/[A-Za-z0-9_.\\-]+)(/[A-Za-z0= -9_.\\-]+)*$" + 'git) + (make-vcs + "git.apache.org" + "^(git\\.apache\\.org/[a-z0-9_.\\-]+\\.git)(/[A-Za-z0-9_.\\-]+= )*$" + 'git) + (make-vcs + "git.openstack.org" + "^(git\\.openstack\\.org/[A-Za-z0-9_.\\-]+/[A-Za-z0-9_.\\-]+)(= \\.git)?(/[A-Za-z0-9_.\\-]+)*$" + 'git))) + (vcs (find (lambda (vcs) (string-prefix? (vcs-url-prefix vcs) mod= ule-path)) + known-vcs))) + (if vcs + (match:substring (string-match (vcs-root-regex vcs) module-path) 1= ) + module-path))) + +(define (go-module->guix-package-name module-path) + "Converts a module's path to the canonical Guix format for Go packages." + (string-downcase + (string-append "go-" + (string-replace-substring + (string-replace-substring + module-path + "." "-") + "/" "-")))) + +(define-record-type + (make-module-meta import-prefix vcs repo-root) + module-meta? + (import-prefix module-meta-import-prefix) + ;; VCS field is a symbol + (vcs module-meta-vcs) + (repo-root module-meta-repo-root)) + +(define (fetch-module-meta-data module-path) + "Fetches module meta-data from a module's landing page. This is + necessary because goproxy servers don't currently provide all the + information needed to build a package." + ;; + (define (meta-go-import->module-meta text) + "Takes the content of the go-import meta tag as TEXT and gives back + a MODULE-META record" + (define (get-component s start) + (let* + ((start (string-skip s char-set:whitespace start)) + (end (string-index s char-set:whitespace start)) + (end (if end end (string-length s))) + (result (substring s start end))) + (values result end))) + (let*-values (((import-prefix end) (get-component text 0)) + ((vcs end) (get-component text end)) + ((repo-root end) (get-component text end))) + (make-module-meta import-prefix (string->symbol vcs) repo-root))) + (define (html->meta-go-import port) + "Read PORT with HTML content. Find the go-import meta tag and gives + back its content as a string." + (let* ((parsedhtml (html->sxml port)) + (extract-content (node-join + (select-kids (node-typeof? 'html)) + (select-kids (node-typeof? 'head)) + (select-kids (node-typeof? 'meta)) + (select-kids (node-typeof? '@)) + (node-self + (node-join + (select-kids (node-typeof? 'name)) + (select-kids (node-equal? "go-import")))) + (select-kids (node-typeof? 'content)) + (select-kids (lambda (_) #t)))) + (content (car (extract-content parsedhtml)))) + content)) + (let* ((port (build-download:http-fetch (string->uri (format #f "https:/= /~a?go-get=3D1" module-path)))) + (meta-go-import (html->meta-go-import port)) + (module-metadata (meta-go-import->module-meta meta-go-import))) + (close-port port) + module-metadata)) + +(define (module-meta-data-repo-url meta-data goproxy-url) + "Return the URL where the fetcher which will be used can download the so= urce +control." + (if (member (module-meta-vcs meta-data)'(fossil mod)) + goproxy-url + (module-meta-repo-root meta-data))) + +(define (source-uri vcs-type vcs-repo-url file) + "Generate the `origin' block of a package depending on what type of sour= ce +control system is being used." + (case vcs-type + ((git) + `(origin + (method git-fetch) + (uri (git-reference + (url ,vcs-repo-url) + (commit (string-append "v" version)))) + (file-name (git-file-name name version)) + (sha256 + (base32 + ,(guix-hash-url file))))) + ((hg) + `(origin + (method hg-fetch) + (uri (hg-reference + (url ,vcs-repo-url) + (changeset ,version))) + (file-name (format #f "~a-~a-checkout" name version)))) + ((svn) + `(origin + (method svn-fetch) + (uri (svn-reference + (url ,vcs-repo-url) + (revision (string->number version)) + (recursive? #f))) + (file-name (format #f "~a-~a-checkout" name version)) + (sha256 + (base32 + ,(guix-hash-url file))))) + (else + (raise-exception (format #f "unsupported vcs type: ~a" vcs-type))))) + +(define* (go-module->guix-package module-path #:key (goproxy-url "https://= proxy.golang.org")) + (call-with-temporary-output-file + (lambda (temp port) + (let* ((latest-version (fetch-latest-version goproxy-url module-path)= ) + (go.mod-path (fetch-go.mod goproxy-url module-path latest-vers= ion + temp)) + (dependencies (map car (parse-go.mod temp))) + (guix-name (go-module->guix-package-name module-path)) + (root-module-path (infer-module-root module-path)) + ;; VCS type and URL are not included in goproxy information. F= or + ;; this we need to fetch it from the official module page. + (meta-data (fetch-module-meta-data root-module-path)) + (vcs-type (module-meta-vcs meta-data)) + (vcs-repo-url (module-meta-data-repo-url meta-data goproxy-url= ))) + (values + `(package + (name ,guix-name) + ;; Elide the "v" prefix Go uses + (version ,(string-trim latest-version #\v)) + (source + ,(source-uri vcs-type vcs-repo-url temp)) + (build-system go-build-system) + ,@(maybe-inputs (map go-module->guix-package-name dependencies)= ) + ;; TODO(katco): It would be nice to make an effort to fetch thi= s + ;; from known forges, e.g. GitHub + (home-page ,(format #f "https://~a" root-module-path)) + (synopsis "A Go package") + (description ,(format #f "~a is a Go package." guix-name)) + (license #f)) + dependencies))))) + +(define* (go-module-recursive-import package-name + #:key (goproxy-url "https://proxy.gol= ang.org")) + (recursive-import + package-name + #:repo->guix-package (lambda* (name . _) + (go-module->guix-package + name + #:goproxy-url goproxy-url)) + #:guix-name go-module->guix-package-name)) diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm index 0a3863f965..1d2b45d942 100644 --- a/guix/scripts/import.scm +++ b/guix/scripts/import.scm @@ -77,7 +77,7 @@ rather than \\n." ;;; =20 (define importers '("gnu" "nix" "pypi" "cpan" "hackage" "stackage" "elpa" = "gem" - "cran" "crate" "texlive" "json" "opam")) + "go" "cran" "crate" "texlive" "json" "opam")) =20 (define (resolve-importer name) (let ((module (resolve-interface diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm new file mode 100644 index 0000000000..fde7555973 --- /dev/null +++ b/guix/scripts/import/go.scm @@ -0,0 +1,118 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =A9 2020 Katherine Cox-Buday +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix scripts import go) + #:use-module (guix ui) + #:use-module (guix utils) + #:use-module (guix scripts) + #:use-module (guix import go) + #:use-module (guix scripts import) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) + #:use-module (srfi srfi-37) + #:use-module (ice-9 match) + #:use-module (ice-9 format) + #:export (guix-import-go)) + + +;;; +;;; Command-line options. +;;; + +(define %default-options + '()) + +(define (show-help) + (display (G_ "Usage: guix import go PACKAGE-PATH +Import and convert the Go module for PACKAGE-PATH.\n")) + (display (G_ " + -h, --help display this help and exit")) + (display (G_ " + -V, --version display version information and exit")) + (display (G_ " + -r, --recursive generate package expressions for all Go modules\ + that are not yet in Guix")) + (display (G_ " + -p, --goproxy=3DGOPROXY specify which goproxy server to use")) + (newline) + (show-bug-report-information)) + +(define %options + ;; Specification of the command-line options. + (cons* (option '(#\h "help") #f #f + (lambda args + (show-help) + (exit 0))) + (option '(#\V "version") #f #f + (lambda args + (show-version-and-exit "guix import go"))) + (option '(#\r "recursive") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive #t result))) + (option '(#\p "goproxy") #t #f + (lambda (opt name arg result) + (alist-cons 'goproxy + (string->symbol arg) + (alist-delete 'goproxy result)))) + %standard-import-options)) + + +;;; +;;; Entry point. +;;; + +(define (guix-import-go . args) + (define (parse-options) + ;; Return the alist of option values. + (args-fold* args %options + (lambda (opt name arg result) + (leave (G_ "~A: unrecognized option~%") name)) + (lambda (arg result) + (alist-cons 'argument arg result)) + %default-options)) + + (let* ((opts (parse-options)) + (args (filter-map (match-lambda + (('argument . value) + value) + (_ #f)) + (reverse opts)))) + (match args + ((module-name) + (if (assoc-ref opts 'recursive) + (map (match-lambda + ((and ('package ('name name) . rest) pkg) + `(define-public ,(string->symbol name) + ,pkg)) + (_ #f)) + (go-module-recursive-import module-name + #:goproxy-url + (or (assoc-ref opts 'goproxy) + "https://proxy.golang.org"= ))) + (let ((sexp (go-module->guix-package module-name + #:goproxy-url + (or (assoc-ref opts 'gopro= xy) + "https://proxy.golang.= org")))) + (unless sexp + (leave (G_ "failed to download meta-data for module '~a'~%"= ) + module-name)) + sexp))) + (() + (leave (G_ "too few arguments~%"))) + ((many ...) + (leave (G_ "too many arguments~%")))))) diff --git a/tests/import-go.scm b/tests/import-go.scm new file mode 100644 index 0000000000..7c59bf2d7c --- /dev/null +++ b/tests/import-go.scm @@ -0,0 +1,143 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =A9 2021 Fran=E7ois Joulaud +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;;; Summary +;; Tests for guix/import/go.scm + +(define-module (test-import-go) + #:use-module (guix import go) + #:use-module (guix base32) + ;#:use-module (guix tests) + #:use-module (ice-9 iconv) + #:use-module (ice-9 match) + #:use-module (srfi srfi-64)) + +(define fixture-go-mod-simple + "module my/thing +go 1.12 +require other/thing v1.0.2 +require new/thing/v2 v2.3.4 +exclude old/thing v1.2.3 +replace bad/thing v1.4.5 =3D> good/thing v1.4.5 +") + +(define fixture-go-mod-with-block + "module M + +require ( + A v1 + B v1.0.0 + C v1.0.0 + D v1.2.3 + E dev +) + +exclude D v1.2.3 +") + + +(define fixture-go-mod-complete + "module M + +go 1.13 + +replace github.com/myname/myproject/myapi =3D> ./api + +replace github.com/mymname/myproject/thissdk =3D> ../sdk + +replace launchpad.net/gocheck =3D> github.com/go-check/check v0.0.0-201402= 25173054-eb6ee6f84d0a + +require ( + github.com/user/project v1.1.11 + github.com/user/project/sub/directory v1.1.12 + bitbucket.org/user/project v1.11.20 + bitbucket.org/user/project/sub/directory v1.11.21 + launchpad.net/project v1.1.13 + launchpad.net/project/series v1.1.14 + launchpad.net/project/series/sub/directory v1.1.15 + launchpad.net/~user/project/branch v1.1.16 + launchpad.net/~user/project/branch/sub/directory v1.1.17 + hub.jazz.net/git/user/project v1.1.18 + hub.jazz.net/git/user/project/sub/directory v1.1.19 + k8s.io/kubernetes/subproject v1.1.101 + one.example.com/abitrary/repo v1.1.111 + two.example.com/abitrary/repo v0.0.2 +) + +replace two.example.com/abitrary/repo =3D> github.com/corp/arbitrary-repo = v0.0.2 + +replace ( + golang.org/x/sys =3D> golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a= // pinned to release-branch.go1.13 + golang.org/x/tools =3D> golang.org/x/tools v0.0.0-20190821162956-65e3620a= 7ae7 // pinned to release-branch.go1.13 +) + +") + +(test-begin "import go") + +(test-equal "go-path-escape" + "github.com/!azure/!avere" + ((@@ (guix import go) go-path-escape) "github.com/Azure/Avere")) + + + +;; We define a function for all similar tests with different go.mod files +(define (testing-parse-mod name expected input) + (define (inf? p1 p2) + (string