From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id SMhLOts8q2WDfwAAe85BDQ:P1 (envelope-from ) for ; Sat, 20 Jan 2024 04:24:12 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id SMhLOts8q2WDfwAAe85BDQ (envelope-from ) for ; Sat, 20 Jan 2024 04:24:12 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=Uh8lITUx; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1705721051; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=YV5bQkU6I643bcNzV5lhfOvZ7wMtT3B8YpiUOiDaTuQ=; b=HjZQuzu9VfS7/Mm0CtEsc8AO5xalnzC9fvSRfYjGCT3sUidgV5RLXPCeiemXOfT1NtN/OK o1o3ie5RrQKKdLm0U1T4cwns03MFZd5rZOq0D69VtrCkYph4fVEcFsndsiRy3qlJABY+4S LfxP6Yzm1T57OAPF621rRER5tHq6SGdfWnG9PLL2LcZKZRwpDBiXUZBNcjZf5H5C6gaN/f +U0DJfEPkIOAZ+dPga/GJyVKA4HAkzbAh4mWDk70Ir6t9uANoD0rtR6ehIZiEbs7tjxANG oYryv4RMGgC2lC9GV5ayA18QfbxA6pSxvug4AsAhG48W3wCBZRgU6xfxNi4uOQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=Uh8lITUx; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1705721051; a=rsa-sha256; cv=none; b=F1IMfVqZxtoDMH8BUcxRXkEKD3T0YNlXkdLh/sA9+6frH1Kt0MdpMfVjjhXRXiU1mt9W1K nD/kJwpKzfUXQdUSQyRyX0saJS1Cq8bnfFEN4coHLgD/ryELaeaj3iaitdeKRMp0hQYndd Po+ZvsjsPwAozJIxGJ3/lk1flzAiLpKpGzoznxNrsKHNBoIBAsHXU8YJDaDYH3HjWyo5HI M41vvWfAQAtV8NY/+CE0pl4fpQzKzkVpxzNpP5MbWW2KW+IBNjj6qEndXoNQ2Y0IYipBO1 F3lTSWbrNahwwiv+7X0zwtR4A0L+wff3YMH2RdNlg7xrCWbuwaQPDoPkADaZBg== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 892FF39D49 for ; Sat, 20 Jan 2024 04:24:11 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rR1xi-0004gU-EF; Fri, 19 Jan 2024 22:24:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rR1xg-0004g5-6r for guix-patches@gnu.org; Fri, 19 Jan 2024 22:24:00 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rR1xf-0005nk-V1 for guix-patches@gnu.org; Fri, 19 Jan 2024 22:23:59 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rR1xi-0000IM-8r for guix-patches@gnu.org; Fri, 19 Jan 2024 22:24:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#68405] [PATCH v4] guix: download: Add support for git repositories. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 20 Jan 2024 03:24:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 68405 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Romain GARBAGE Cc: 68405@debbugs.gnu.org, Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 68405-submit@debbugs.gnu.org id=B68405.17057209941057 (code B ref 68405); Sat, 20 Jan 2024 03:24:02 +0000 Received: (at 68405) by debbugs.gnu.org; 20 Jan 2024 03:23:14 +0000 Received: from localhost ([127.0.0.1]:60602 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rR1ww-0000Gy-1b for submit@debbugs.gnu.org; Fri, 19 Jan 2024 22:23:14 -0500 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]:53682) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rR1wt-0000Gj-I8 for 68405@debbugs.gnu.org; Fri, 19 Jan 2024 22:23:12 -0500 Received: by mail-qk1-x72f.google.com with SMTP id af79cd13be357-783293278adso104149985a.3 for <68405@debbugs.gnu.org>; Fri, 19 Jan 2024 19:23:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705720983; x=1706325783; darn=debbugs.gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=YV5bQkU6I643bcNzV5lhfOvZ7wMtT3B8YpiUOiDaTuQ=; b=Uh8lITUxnhQ1o7v7+Zpufa6UnSj1Yi4AUg2RpdZB6heGq2jn1fF6CDVpR1lTX7CPua EgwBMhLMdxK1k0WMYqm/YOLWsOYIIPYOjW2BwtU5cE7/Md0OcP2R3uB51WUTkHYb7BCt vr0YT/EndRhEN5iWqZroHfCxlQ4nq1WVwlgRuBEVUQQfuUa3wqxHaQHLNbYOZI6WgtVa tjNYLcrWIxcUOXn3Rw7aCUPJUanUZ6v5VqrlzVUPXRLwqLPDPmzSWHYhrNoMbtGU2WB+ 4bw371NoMPujtvLP6Vjx0Wl6qz6uRIUsN83Zhp2jC9BKv0Rs9OUEPaqqBkYZsNILYXfV 0/VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705720983; x=1706325783; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YV5bQkU6I643bcNzV5lhfOvZ7wMtT3B8YpiUOiDaTuQ=; b=G9DUggzYUGeQt/YTPo+ZGhhDBBb7EYbgtkXWw2K33YFklhg1LToTKdQNprfUPpfhyX xTNCL0zq6n1rqySU55TzkJqdShNpox1dArVXPjcq9dFNHworO4VdvSluYsVtoak8K0iH mskZwgc4nkHqh6vTu5MoQshxZnzvdh35cCa9UTWwYGl/pstTh+1lKtziWuN7j6/++cZG A5iExH5Tmc5c/6Ml1fI6AKdJGNftsyEhwa8H+ksp1Q67MYu34HFPpsRmQ6mPisuRge2Z fzE4VeLRSWYLCgwQkMrsYE91uThmjq/ScjZ4pFkYTcDp92ApDIRt7EXkkJ1OrmcUyaas h70A== X-Gm-Message-State: AOJu0Yy6dreF1SZ7nt8sQ04n8hFPetDUV/CzVP+ZRhmDn0dYicxYZIUb Iy9fRfNxnGOAgCgwt7/Y4Dgp9CKM4HprWEI6EV5TeBiPKTWu7ibw X-Google-Smtp-Source: AGHT+IHSJdmc+tRpnlUfMnyEqL0tLs6fN14nlCbI2w5ayxzc7S6QEyQP1p8fXyC9pAe6Sm2WnL8WTg== X-Received: by 2002:ae9:e918:0:b0:783:54b4:7763 with SMTP id x24-20020ae9e918000000b0078354b47763mr983026qkf.87.1705720983220; Fri, 19 Jan 2024 19:23:03 -0800 (PST) Received: from hurd (dsl-10-132-160.b2b2c.ca. [72.10.132.160]) by smtp.gmail.com with ESMTPSA id vy4-20020a05620a490400b00781df19c062sm251944qkn.59.2024.01.19.19.23.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Jan 2024 19:23:02 -0800 (PST) From: Maxim Cournoyer In-Reply-To: <20240119102417.17155-1-romain.garbage@inria.fr> (Romain GARBAGE's message of "Fri, 19 Jan 2024 11:19:23 +0100") References: <20240119102417.17155-1-romain.garbage@inria.fr> Date: Fri, 19 Jan 2024 22:23:01 -0500 Message-ID: <87v87ohene.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: 2.33 X-Migadu-Scanner: mx13.migadu.com X-Spam-Score: 2.33 X-Migadu-Queue-Id: 892FF39D49 X-TUID: vq508cu8H367 Hello! Romain GARBAGE writes: > * guix/scripts/download.scm (git-download-to-store*): Add new variable. > (copy-recursively-without-dot-git): New variable. > (git-download-to-file): Add new variable. > (show-help): Add 'git', 'commit', 'branch' and 'recursive'options > help message. > (%default-options): Add default value for 'git-reference' and > 'recursive' options. > (%options): Add 'git', 'commit', 'branch' and 'recursive' command > line options. > (guix-download) [hash]: Compute hash with 'file-hash*' instead of > 'port-hash' from (gcrypt hash) module. This allows us to compute > hashes for directories. > * doc/guix.texi (Invoking guix-download): Add @item entries for > `git', `commit', `branch' and `recursive' options. Add a paragraph in > the introduction. > * tests/guix-download.sh: New tests. Move variables and trap definition > to the top of the file. > > Change-Id: Ic2c428dca4cfcb0d4714ed361a4c46609339140a [...] > +(define* (git-download-to-store* url > + reference > + recursive? > + #:key (verify-certificate? #t)) > + "Download the git repository at URL to the store, checked out at REFERENCE. > +URL must specify a protocol (i.e https:// or file://), REFERENCE must be a > +pair argument as understood by 'latest-repository-commit'." > + ;; Ensure the URL string is properly formatted when using the 'file' > + ;; protocol: URL is generated using 'uri->string', which returns > + ;; "file:/path/to/file" instead of "file:///path/to/file", which in turn > + ;; makes 'git-download-to-store' fail. > + (let* ((file? (string-prefix? "file:" url)) > + (url (if (and file? > + (not (string-prefix? "file:///" url))) > + (string-append "file://" (string-drop url (string-length "file:"))) > + url))) > + (with-store store > + ;; TODO: Verify certificate support and deactivation. > + (with-git-error-handling > + (latest-repository-commit store url #:recursive? recursive? #:ref reference))))) The above contains too long lines still :-). > + > (define %default-options > ;; Alist of default option values. > `((format . ,bytevector->nix-base32-string) > (hash-algorithm . ,(hash-algorithm sha256)) > (verify-certificate? . #t) > + (git-reference . #f) > + (recursive? . #f) > (download-proc . ,download-to-store*))) > > (define (show-help) > @@ -97,6 +180,19 @@ (define (show-help) > do not validate the certificate of HTTPS servers ")) > (format #t (G_ " > -o, --output=FILE download to FILE")) > + (format #t (G_ " > + -g, --git download the default branch's latest commit of the > + Git repository at URL")) > + (format #t (G_ " > + --commit=COMMIT-OR-TAG > + download the given commit or tag of the Git > + repository at URL")) > + (format #t (G_ " > + --branch=BRANCH download the given branch of the Git repository > + at URL")) > + (format #t (G_ " > + -r, --recursive download a Git repository recursively")) > + > (newline) > (display (G_ " > -h, --help display this help and exit")) > @@ -105,6 +201,13 @@ (define (show-help) > (newline) > (show-bug-report-information)) > > +(define (add-git-download-option result) > + (alist-cons 'download-proc > + ;; XXX: #:verify-certificate? currently ignored. > + (lambda* (url #:key verify-certificate? ref recursive?) > + (git-download-to-store* url ref recursive?)) > + (alist-delete 'download result))) > + > (define %options > ;; Specifications of the command-line options. > (list (option '(#\f "format") #t #f > @@ -136,11 +239,36 @@ (define fmt-proc > (alist-cons 'verify-certificate? #f result))) > (option '(#\o "output") #t #f > (lambda (opt name arg result) > - (alist-cons 'download-proc > - (lambda* (url #:key verify-certificate?) > - (download-to-file url arg)) > - (alist-delete 'download result)))) > - > + (let* ((git > + (assoc-ref result 'git-reference))) > + (if git > + (alist-cons 'download-proc > + (lambda* (url #:key verify-certificate? ref recursive?) > + (git-download-to-file url arg (assoc-ref result 'git-reference) recursive?)) > + (alist-delete 'download result)) > + (alist-cons 'download-proc > + (lambda* (url #:key verify-certificate? #:allow-other-keys) > + (download-to-file url arg)) > + (alist-delete 'download result)))))) Here as well. Otherwise, I've tested it with: --8<---------------cut here---------------start------------->8--- ./pre-inst-env guix download -gr https://git.jami.net/savoirfairelinux/jami-client-qt -o /tmp/jami --8<---------------cut here---------------end--------------->8--- and it worked as advertised; very nice! Some idea for the future: the --recurse option could take an optional argument that'd be a comma-separated list of submodules to fetch, e.g. --8<---------------cut here---------------start------------->8--- ./pre-inst-env guix download --git \ --recurse=daemon,3rdparty/SortFilterProxyModel \ https://git.jami.net/savoirfairelinux/jami-client-qt -o /tmp/jami --8<---------------cut here---------------end--------------->8--- and it'd recurse *only* the listed submodules. This would be useful as some projects contain submodules for windows or other platforms we do not care about and they may be very large (heavy) to download. The same idea could be implemented for our git-reference, where recursive? could accept a git submodule names list. But back to the current scope: Reviewed-by: Maxim Cournoyer -- Thanks, Maxim