From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id 6DEDGqaqFWcO8gAA62LTzQ:P1 (envelope-from ) for ; Mon, 21 Oct 2024 01:13:10 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id 6DEDGqaqFWcO8gAA62LTzQ (envelope-from ) for ; Mon, 21 Oct 2024 03:13:10 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=KO91L13q; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b="RYI/CwWD"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1729473190; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=fAG5n4qrmQA/1eWTastbiYgR9QMT6fXQmvaV1cTBRXc=; b=NhXNqqoIea2XREN4E4xcKyEFQO9mWlc8vGqx+zD92c+TJuziemA4L1d+w9V0R5W53KfEph tdZPIPdaN1uN5HQxzRdCQ060uxngZ/UENDKe0t3u59N7GhmPxCys3PXatgzLnAM8uCoZnD 4MEe8s+IqAwFhmt0VXiogpbWpFMg+lstLB0/yiow192c9kFAbFLWLJ8wk6yAYbCggOMp5W HguCEwGokJZ8y9wJD0dC3Sg3Q9dkpFhWZMzoI/lYFBbqey7BcU5KeoWIJ0IOCPOyTyC+RC 4lHxUWw2+5831wbFvQwFOyNl8MZ+vKQ/at9uWnGfOzR3DtuGMFhfD4laZKxfiQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=KO91L13q; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b="RYI/CwWD"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1729473190; a=rsa-sha256; cv=none; b=HqsUCDa4kmJiSLqhvgzYs9CteOPEWOwY2PKwZyJTnuqX7xYWfHW54ukJJ7rL2lR2fQfabb hXu5Kvz8CDk89p5dYE/lAFiBV/5D5L3nakgBUOOluFp9h7eVYPC95CZ1q4PTA0zkOgb4AS I3v1f5a4RavUAAPF9pBZZ3gbCwsAQ8QsMn1HVyV1fXYmfrOkUcQlz5Su+hbcjsaWTgY6Db Cxm4A/aqFjkagDzGhn7RXRs26NBm6SmXgNZOxfYDXYcU/1KuqTC9FfXb3qjZl2/QEUtOSe hgNKUzpUqxPyzPqHY6Rnolc43IRog/wbTiSbQiwUUUqtTQzncAofqKzDpEyBFw== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B0AF28DDC1 for ; Mon, 21 Oct 2024 03:13:09 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t2gyT-0002sU-CO; Sun, 20 Oct 2024 21:12:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t2gyR-0002sI-NS for guix-patches@gnu.org; Sun, 20 Oct 2024 21:12:43 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t2gyQ-0004Zj-OP; Sun, 20 Oct 2024 21:12:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:From:To:Subject; bh=fAG5n4qrmQA/1eWTastbiYgR9QMT6fXQmvaV1cTBRXc=; b=KO91L13qIYIno9ec00Ts/lBXiZV/UpUXyVhlTqvYCYI2voUdz0MeUjQqAljh4b3hPrFxV1VL/r8EJnyNZVQZrARD4slve14mOIfpnYJ4ZinuAlXWrmMwlXW+tRukn5DKAOJogOzYe85EFHd9zgD3pR4Xp/0apGmqoQrw2txj0p82JiL7bjUOqDecgxePNUrqeq8QhUghhpgrIgLNKp++lSJ9TjgU92300Kmo+2qpl7ydenccljKNjAnjCsm0GlST6yo1rfnm+TKhqe2+kACJN1OQbIbL/Udxq8j3vXg5xavcNeiCo6egwyPUyXK0dn1w9HTNuuomK3UOM8B4S3P/RA==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1t2gyj-0000vp-Uh; Sun, 20 Oct 2024 21:13:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#73921] [PATCH] svn-download: Add support for partial checkout. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, maxim.cournoyer@gmail.com, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Mon, 21 Oct 2024 01:13:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 73921 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 73921@debbugs.gnu.org Cc: Maxim Cournoyer , Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Maxim Cournoyer , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Maxim Cournoyer , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by submit@debbugs.gnu.org id=B.17294731403513 (code B ref -1); Mon, 21 Oct 2024 01:13:01 +0000 Received: (at submit) by debbugs.gnu.org; 21 Oct 2024 01:12:20 +0000 Received: from localhost ([127.0.0.1]:49408 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1t2gy3-0000ub-84 for submit@debbugs.gnu.org; Sun, 20 Oct 2024 21:12:19 -0400 Received: from lists.gnu.org ([209.51.188.17]:41510) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1t2gy0-0000uS-NU for submit@debbugs.gnu.org; Sun, 20 Oct 2024 21:12:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t2gxa-0002JD-1X for guix-patches@gnu.org; Sun, 20 Oct 2024 21:11:50 -0400 Received: from mail-pg1-x52c.google.com ([2607:f8b0:4864:20::52c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t2gxY-0004Xk-4B for guix-patches@gnu.org; Sun, 20 Oct 2024 21:11:49 -0400 Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-7d4fa972cbeso3038701a12.2 for ; Sun, 20 Oct 2024 18:11:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729473105; x=1730077905; darn=gnu.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=fAG5n4qrmQA/1eWTastbiYgR9QMT6fXQmvaV1cTBRXc=; b=RYI/CwWDZRL03CAM+VOEndXlNg36REHAP7OqzUT5TICG9ot2fRUdRx3BSwl5Vvv+oT iQsUgzEHJcgMff9kjaRCi90fC45OjOPEG0UAkWc814lSuhXo7qMEBoLxnpHWCSQfuNye 0mqiBKvx8QfaL0KMV2eA0NhvpjRr1rnRbWUzyzF/PldHTfB206sQlh5cdaGjoFiH0Cbg mM2fEK0TyhVd33qTygdZrXyRCAwQG8rVdYnhaMyHW9AwwUKwRBp/qnGXn5+Op0rBLEHE j/lzPW7Wc/isi0Ht5Jxte9d3ih14AyteB6+F+9EL5m2nAxkrNmca2XJiN/Nz7dEZ5Aep ADyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729473105; x=1730077905; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=fAG5n4qrmQA/1eWTastbiYgR9QMT6fXQmvaV1cTBRXc=; b=vL2dVd7Oj8nOYdoUzfKXDOgSLhd16yICHXUwR0uN7Wa2I4tbErtL/IANypUWxVZO9f x1wq/DQpIPm4h0RWa1VW9WMEpQQ0nSU/byZjhdZ1xVW6EQuA0gsMD6GWuLA8ExBVzf8l G0wPcIJ4SO3Zlh+gTVrawVixWsbZj9lSD3SXUf/P/9wKrtfm3QM/XPeQ07/IqeKKMww+ 9V8M5sGRp8GyK7ak6HMdi5WhcNZfmPr/kZA9p3V4tOR0XToN86H7oKymh3PC3E3pvP+h aj+zhVmlFcJ4aN3bLCtI8qWo0j/rRILtI+Uw3PX2jtA8O0EaOnWozZzLhVpCGZa69qTJ /d9w== X-Gm-Message-State: AOJu0YxlKWSDCfK6+wc2Sjd3boYzLtDB1fS1WyK/qJEjqlNcA6eMuQZi ED8e6k3gSXFUpotw2PYmkUzKndFPDepKS6g76JwVcLbtzi0wco9KwcQH0g== X-Google-Smtp-Source: AGHT+IECiYmY0QjTGmOtH9J0om6qaCPBboW4nIhCKhOU21AN7jIqh6FaBGLpqgyDUhRorerJkb3Sgg== X-Received: by 2002:a05:6a21:2d88:b0:1d8:a13d:d6b2 with SMTP id adf61e73a8af0-1d92c4a5349mr15835878637.10.1729473104792; Sun, 20 Oct 2024 18:11:44 -0700 (PDT) Received: from localhost.localdomain ([2405:6586:be0:0:c8ff:1707:9b9:af89]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7eaeab20470sm1531256a12.34.2024.10.20.18.11.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 18:11:44 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Oct 2024 10:08:46 +0900 Message-ID: X-Mailer: git-send-email 2.46.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::52c; envelope-from=maxim.cournoyer@gmail.com; helo=mail-pg1-x52c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Scanner: mx11.migadu.com X-Migadu-Spam-Score: 4.74 X-Spam-Score: 4.74 X-Migadu-Queue-Id: B0AF28DDC1 X-TUID: UcIHX/B2yyT3 * guix/svn-download.scm () [subdirectories]: New field. (svn-fetch): Set "svn subdirectories" environment variable when subdirectories are provided. (svn-fetch-builder): Pass subdirectories to svn-fetch. * guix/build/svn.scm (svn-fetch) [subdirectories]: New keyword argument. Implement alternative code path when it's provided. * doc/guix.texi (svn-reference): Document the new field. Change-Id: I21ca96bc48d26dafca82b26daccef0d324f79dc5 --- Note: At first I was planning to use this to checkout exactly one subdirectory of a game data SVN repository, but later I realized SVN allows for checking out any subdirectory, which made this addition unnecessary. I guess it could still be useful when wanting to select more than one subdirectory, but the perhaps svn-multi-fetch could be used instead, to combine multiple SVN checkouts into one (IIUC). Anyway, I'm sending this to guix-devel; if anyone find it useful it could be considered for guix-patches. doc/guix.texi | 3 ++ guix/build/svn.scm | 74 ++++++++++++++++++++++++++++--------------- guix/svn-download.scm | 20 +++++++++--- 3 files changed, 68 insertions(+), 29 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index b9f71527a3..2d1e5bdd2c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -8312,6 +8312,9 @@ origin Reference @item @code{password} (default: @code{#f}) Password to access the Subversion repository, if required. + +@item @code{subdirectories} (default: @code{#f}) +Only recurse into subdirectories, resulting in a partial checkout. @end table @end deftp diff --git a/guix/build/svn.scm b/guix/build/svn.scm index 875d3c50ca..1b1c9b1b5c 100644 --- a/guix/build/svn.scm +++ b/guix/build/svn.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2014 Sree Harsha Totakura ;;; Copyright © 2018 Mark H Weaver ;;; Copyright © 2020 Simon Tournier +;;; Copyright © 2024 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -21,6 +22,9 @@ (define-module (guix build svn) #:use-module (guix build utils) + #:use-module (ice-9 match) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:export (svn-fetch)) @@ -33,35 +37,55 @@ (define-module (guix build svn) (define* (svn-fetch url revision directory #:key (svn-command "svn") - (recursive? #t) - (user-name #f) - (password #f)) + recursive? + user-name + password + subdirectories) "Fetch REVISION from URL into DIRECTORY. REVISION must be an integer, and a -valid Subversion revision. Return #t on success, #f otherwise." +valid Subversion revision. If SUBDIRECTORIES is provided, only these +subdirectories will be fully fetched (partial checkout). Return #t on +success, #f otherwise." + (define base-options + `("--non-interactive" + ;; Trust the server certificate. This is OK as we + ;; verify the checksum later. This can be removed when + ;; ca-certificates package is added. + "--trust-server-cert" + "-r" ,(number->string revision) + ,@(if (and user-name password) + (list (string-append "--username=" user-name) + (string-append "--password=" password)) + '()) + ,@(if recursive? + '() + (list "--ignore-externals")))) + (guard (c ((invoke-error? c) (report-invoke-error c) #f)) - (apply invoke svn-command - "export" "--non-interactive" - ;; Trust the server certificate. This is OK as we - ;; verify the checksum later. This can be removed when - ;; ca-certificates package is added. - "--trust-server-cert" "-r" (number->string revision) - - ;; Disable keyword substitutions (keywords are CVS-like strings - ;; like "$Date$", "$Id$", and so on) for two reasons: (1) some - ;; expansions depend on the local time zone, and (2) SWH disables - ;; it in its archive for this very reason. - "--ignore-keywords" - - `(,@(if (and user-name password) - (list (string-append "--username=" user-name) - (string-append "--password=" password)) - '()) - ,@(if recursive? - '() - (list "--ignore-externals")) - ,url ,directory)) + (match subdirectories + (#f + (apply invoke svn-command "export" + (append base-options + `(;; Disable keyword substitutions (keywords are CVS-like strings + ;; like "$Date$", "$Id$", and so on) for two reasons: (1) some + ;; expansions depend on the local time zone, and (2) SWH disables + ;; it in its archive for this very reason. + "--ignore-keywords" + ,url ,directory)))) + (_ + (apply invoke svn-command "checkout" + (append base-options + `(,@(if subdirectories + (list "--depth" "immediates") + '()) + ,url ,directory))) + (with-directory-excursion directory + (apply invoke svn-command "update" + (append base-options + (append-map (cut list "--set-depth" "infinity" <>) + subdirectories))) + (delete-file-recursively ".svn")))) #t)) ;;; svn.scm ends here diff --git a/guix/svn-download.scm b/guix/svn-download.scm index b20cdc79d1..e263498f30 100644 --- a/guix/svn-download.scm +++ b/guix/svn-download.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2014-2016, 2019, 2021-2024 Ludovic Courtès ;;; Copyright © 2014 Sree Harsha Totakura ;;; Copyright © 2017, 2019, 2021 Ricardo Wurmus +;;; Copyright © 2024 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -35,6 +36,7 @@ (define-module (guix svn-download) svn-reference? svn-reference-url svn-reference-revision + svn-reference-subdirectories svn-reference-recursive? svn-reference-user-name svn-reference-password @@ -65,6 +67,8 @@ (define-record-type* svn-reference? (url svn-reference-url) ; string (revision svn-reference-revision) ; number + (subdirectories svn-reference-subdirectories ;list or #f + (default #f)) (recursive? svn-reference-recursive? (default #f)) (user-name svn-reference-user-name (default #f)) (password svn-reference-password (default #f))) @@ -84,7 +88,7 @@ (define (svn-fetch-builder svn hash-algo) (define guile-gnutls (module-ref (resolve-interface '(gnu packages tls)) 'guile-gnutls)) - (define tar+gzip ;for (guix swh) + (define tar+gzip ;for (guix swh) (list (module-ref (resolve-interface '(gnu packages compression)) 'gzip) (module-ref (resolve-interface '(gnu packages base)) @@ -96,7 +100,7 @@ (define (svn-fetch-builder svn hash-algo) (guix build download-nar) (guix build utils) (guix swh))) - (with-extensions (list guile-json guile-gnutls ;for (guix swh) + (with-extensions (list guile-json guile-gnutls ;for (guix swh) guile-lzlib) #~(begin (use-modules (guix build svn) @@ -105,7 +109,8 @@ (define (svn-fetch-builder svn hash-algo) (guix build download-nar) (guix build utils) (guix swh) - (ice-9 match)) + (ice-9 match) + (srfi srfi-26)) ;; Add tar and gzip to $PATH so ;; 'swh-download-directory-by-nar-hash' can invoke them. @@ -120,7 +125,9 @@ (define (svn-fetch-builder svn hash-algo) ("yes" #t) (_ #f)) #:user-name (getenv "svn user name") - #:password (getenv "svn password"))) + #:password (getenv "svn password") + #:subdirectories (and=> (getenv "svn subdirectories") + (cut string-split <> #\:)))) (and (download-method-enabled? 'nar) (download-nar #$output)) (and (download-method-enabled? 'swh) @@ -164,6 +171,11 @@ (define* (svn-fetch ref hash-algo hash `(("svn password" . ,(svn-reference-password ref))) '()) + ,@(if (svn-reference-subdirectories ref) + `(("svn subdirectories" + . ,(string-join + (svn-reference-subdirectories ref) ":"))) + '()) ,@(match (getenv "GUIX_DOWNLOAD_METHODS") (#f '()) (value base-commit: 503919fcf01d7eb8d550df5c3993aee9a966ba9b -- 2.46.0