From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id oF84LDh0WWBKDgAA0tVLHw (envelope-from ) for ; Tue, 23 Mar 2021 04:53:12 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 6M79Jzh0WWB5MwAAbx9fmQ (envelope-from ) for ; Tue, 23 Mar 2021 04:53:12 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E84311C208 for ; Tue, 23 Mar 2021 05:53:11 +0100 (CET) Received: from localhost ([::1]:52624 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOZ2U-0003F2-IG for larch@yhetil.org; Tue, 23 Mar 2021 00:53:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60924) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOZ2O-0003Ek-4Q for bug-guix@gnu.org; Tue, 23 Mar 2021 00:53:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:47710) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOZ2M-00063O-5Q for bug-guix@gnu.org; Tue, 23 Mar 2021 00:53:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOZ2M-00026W-3f for bug-guix@gnu.org; Tue, 23 Mar 2021 00:53:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#47336: [PATCH 1/2] swh: Add a directory download procedure. References: <87eeg6o50b.fsf@ngyro.com> In-Reply-To: <87eeg6o50b.fsf@ngyro.com> Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Mar 2021 04:53:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.16164751648063 (code B ref 47336); Tue, 23 Mar 2021 04:53:02 +0000 Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 04:52:44 +0000 Received: from localhost ([127.0.0.1]:59255 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZ24-00025z-Ds for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:52:44 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:42087) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZ23-00025o-5S for 47336@debbugs.gnu.org; Tue, 23 Mar 2021 00:52:43 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 5C2E14CE; Tue, 23 Mar 2021 00:52:37 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:52:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=9GhkL9aQDX+o8K1Ib LLy06+cd1v9DpMcsGDadXOnnPk=; b=mD0rvrRxnVCL73RGYUbbdN8818yUnGh58 WE9F+dIqu3kz3MZxL1mHEkGX5ba3xkVlxU0Jk++lJ93qngoCNF5t9lxuMudGhDgG YaIBxdv+jeNcLz36AbuwJgtQCT4he/E9bBtXImaqdWFyLyJ8DNCZb6c+j9wX/c2g Ys/CmVIQkVKME2MiVs1SdTS3AL9gzTsGMAow2k9UkdKWb4RKX0AT7QziTP8bokdi a6wpQbxcNE2qz1LgmAZX195fBJt4JbWUNUgyxIpJtMMixqgBz7SPR+lhrAQxolMh oU+h6LKZRvuT6cOaXWFNYCE3s59m1uWyFi6I1CsN1qsafElYVts0g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgggfestdekredtre dttdenucfhrhhomhepvfhimhhothhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhg hihrohdrtghomheqnecuggftrfgrthhtvghrnhepvdehudduieektedugeevgfeutdffff dujeeifffhudehvdelveduvedtfefhtdejnecukfhppeejgedrudduiedrudekiedrgeeg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshgrmh hplhgvthesnhhghihrohdrtghomh X-ME-Proxy: Received: from mrblack.lan (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 738071080054; Tue, 23 Mar 2021 00:52:36 -0400 (EDT) From: Timothy Sample Date: Tue, 23 Mar 2021 00:52:12 -0400 Message-Id: <20210323045213.9419-1-samplet@ngyro.com> X-Mailer: git-send-email 2.31.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616475192; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=9GhkL9aQDX+o8K1IbLLy06+cd1v9DpMcsGDadXOnnPk=; b=CuBAoQM3o3n6kVBs7lX2V4rYpgEAnfpzn10ql8lm+uB79R/XIzu/i1bEENRab1i7AgfKRr lqdI1mFJn6Bz1VUkH9/IcL74h9wjWCU55mHFMnYQ8zj1hHQel744+0X4Wit5m1E8LAfiLI jDtC0wCW7i+MwLQ65TPrT0ls21AizsNhccBrb6kdNGi6kWXsFInJaAeEklt3N+GN9OAy+G kLhrwLscOO0v39AmpVZzO1mZO2w3GGqb2pQAfwB8i+2Huada0QvwtDLM8+1zRGCYrQ8f6d 0ZxT1DAoqMy60i5kQww9FWWyr5kZPX1Llf0eEmb+vcH52e8PxtUp8IPSUFQUZQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616475192; a=rsa-sha256; cv=none; b=eUwvU4KDflPIf8Ey3zd5N0jH4yqHQ3DBZ56BX50X05M1C3JDyQrY45xfFueVd1aNtHIX8G bT96w9XBk8dHyVBaJnWPCPnPtRUNCIYGL9R8iRZQHVnYGV/YUq86W2BsXtQxVI7PMpN1Z7 eLU9N+vatiV4m653CM41zyyuTAidpVJINJzr3NQOwUpcikhsRC465bgXuLzAO/OrI6y3bi /xKL3p47IoXvrhbq3FU1MhPDMQ2tVqShJP1qUzBhCMqg20H2MzJeaSlykEjG6S+4t01gt8 IQB/rZxtFt5K8PtNZxx+PPsg+KK78OkmpnpzuUARrTIW2z7vTz1vSoq3wnUbeQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=mD0rvrRx; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: 3.58 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=mD0rvrRx; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: E84311C208 X-Spam-Score: 3.58 X-Migadu-Scanner: scn0.migadu.com X-TUID: NLULLZz39Z8p * guix/swh.scm (swh-directory-download): New procedure (with implementation extracted from 'swh-download'). (swh-download): Use it to download the revision directory. --- guix/swh.scm | 65 +++++++++++++++++++++++++++++----------------------- 1 file changed, 36 insertions(+), 29 deletions(-) diff --git a/guix/swh.scm b/guix/swh.scm index f11b7ea2d5..2402ec98e6 100644 --- a/guix/swh.scm +++ b/guix/swh.scm @@ -108,6 +108,7 @@ commit-id? + swh-download-directory swh-download)) ;;; Commentary: @@ -558,12 +559,6 @@ requested bundle cooking, waiting for completion...~%")) ;;; High-level interface. ;;; -(define (commit-id? reference) - "Return true if REFERENCE is likely a commit ID, false otherwise---e.g., if -it is a tag name. This is based on a simple heuristic so use with care!" - (and (= (string-length reference) 40) - (string-every char-set:hex-digit reference))) - (define (call-with-temporary-directory proc) ;FIXME: factorize "Call PROC with a name of a temporary directory; close the directory and delete it when leaving the dynamic extent of this call." @@ -577,6 +572,39 @@ delete it when leaving the dynamic extent of this call." (lambda () (false-if-exception (delete-file-recursively tmp-dir)))))) +(define* (swh-download-directory id output + #:key (log-port (current-error-port))) + "Download from Software Heritage the directory with the given ID, and +unpack it to OUTPUT. Return #t on success and #f on failure" + (call-with-temporary-directory + (lambda (directory) + (match (vault-fetch id 'directory #:log-port log-port) + (#f + (format log-port + "SWH: directory ~a could not be fetched from the vault~%" + id) + #f) + ((? port? input) + (let ((tar (open-pipe* OPEN_WRITE "tar" "-C" directory "-xzvf" "-"))) + (dump-port input tar) + (close-port input) + (let ((status (close-pipe tar))) + (unless (zero? status) + (error "tar extraction failure" status))) + + (match (scandir directory) + (("." ".." sub-directory) + (copy-recursively (string-append directory "/" sub-directory) + output + #:log (%make-void-port "w")) + #t)))))))) + +(define (commit-id? reference) + "Return true if REFERENCE is likely a commit ID, false otherwise---e.g., if +it is a tag name. This is based on a simple heuristic so use with care!" + (and (= (string-length reference) 40) + (string-every char-set:hex-digit reference))) + (define* (swh-download url reference output #:key (log-port (current-error-port))) "Download from Software Heritage a checkout of the Git tag or commit @@ -593,28 +621,7 @@ wait until it becomes available, which could take several minutes." (format log-port "SWH: found revision ~a with directory at '~a'~%" (revision-id revision) (swh-url (revision-directory-url revision))) - (call-with-temporary-directory - (lambda (directory) - (match (vault-fetch (revision-directory revision) 'directory - #:log-port log-port) - (#f - (format log-port - "SWH: directory ~a could not be fetched from the vault~%" - (revision-directory revision)) - #f) - ((? port? input) - (let ((tar (open-pipe* OPEN_WRITE "tar" "-C" directory "-xzvf" "-"))) - (dump-port input tar) - (close-port input) - (let ((status (close-pipe tar))) - (unless (zero? status) - (error "tar extraction failure" status))) - - (match (scandir directory) - (("." ".." sub-directory) - (copy-recursively (string-append directory "/" sub-directory) - output - #:log (%make-void-port "w")) - #t)))))))) + (swh-download-directory (revision-directory revision) output + #:log-port log-port)) (#f #f))) -- 2.31.0