From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id GOvfK3Sin2AZZAEAgWs5BA (envelope-from ) for ; Sat, 15 May 2021 12:29:08 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id IKJ/J3Sin2AhVQAAB5/wlQ (envelope-from ) for ; Sat, 15 May 2021 10:29:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 090BB21EC0 for ; Sat, 15 May 2021 12:29:08 +0200 (CEST) Received: from localhost ([::1]:37032 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhrXe-0003ly-V5 for larch@yhetil.org; Sat, 15 May 2021 06:29:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37288) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhrXa-0003lq-Lw for guix-patches@gnu.org; Sat, 15 May 2021 06:29:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:36136) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lhrXa-0005HY-EV for guix-patches@gnu.org; Sat, 15 May 2021 06:29:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lhrXa-0001zJ-9f for guix-patches@gnu.org; Sat, 15 May 2021 06:29:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#48437] [PATCH] lint: archival: Lookup content in Disarchive database. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 15 May 2021 10:29:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 48437 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 48437@debbugs.gnu.org Cc: Timothy Sample , Ludovic =?UTF-8?Q?Court=C3=A8s?= X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16210745057585 (code B ref -1); Sat, 15 May 2021 10:29:02 +0000 Received: (at submit) by debbugs.gnu.org; 15 May 2021 10:28:25 +0000 Received: from localhost ([127.0.0.1]:47682 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhrWy-0001yG-Se for submit@debbugs.gnu.org; Sat, 15 May 2021 06:28:25 -0400 Received: from lists.gnu.org ([209.51.188.17]:38636) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhrWx-0001y9-9m for submit@debbugs.gnu.org; Sat, 15 May 2021 06:28:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37250) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhrWx-0003iH-3D for guix-patches@gnu.org; Sat, 15 May 2021 06:28:23 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40358) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhrWw-0004sS-PL; Sat, 15 May 2021 06:28:22 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51246 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lhrWw-00037e-G3; Sat, 15 May 2021 06:28:22 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Sat, 15 May 2021 12:28:14 +0200 Message-Id: <20210515102814.5944-1-ludo@gnu.org> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1621074548; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=ROCZBQ++4DGDmZs53kUa0ECKnwF+1iCAiMgWkaQOeVs=; b=qnf0iCaUlNecC+j0RVeH7uEAHBw6M9XWHFWeDAHD5JqoxVk9hxokvIkzrHpzmVFBa9e7f6 J5vJJ4oxtNiYbiaQ6iSGO1b/F04u45BlTc4pck9cd9FYBd+XomA8Y0+lDf6IxZqA1GvQVU ZjFkYIp6DoGvcw+V24JzdAfAleIzfZrRJaKRUx0aYTXw/Yy+QiZkxmhtR21k50pTNave5x 7JYqSqO2CsgWOe6DpqtBKakCQUvLTU8/jlIVg9Gy4LgVw+2MBZoNf0waGAj3/XS0PSTRX/ UQo94kRCRB7TfiLZJNNoKyc+ERCuYD+vujHnLy3VNacaKNIy1K32YZgHDGhs8Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1621074548; a=rsa-sha256; cv=none; b=KikXyVmdadKz8Wfe7G6sIySmP8Wsaj10z1XtiwTL9xlkNtaWV7qywdrYoMp/e1YtTMZ3XL KXmXD11cLA/uXH+QLotmYKr5VsAoT08A9AedVJg1lTZJWaI3B9uFqsNb4a9riSWPEmyDxt OHAuzGvfPnOXk0u/a9apoavXUmwy/aVM+WG+z7l4zPjrTL/lFoY+Mz4Z8EgUrxG6N0cxBE KcUxN0UMEZ8jv/X6vX7YP/Hpe8lqemjqNfOt/i+YkhZHuItoQSPh8gojxcxM4K48mzvdxQ hmPiX9JA0Xx0GkPkR7Yvt0fQOkNTNLAIAqucrpQL+gBG/f4AwT/WVD7eaa77vg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.95 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 090BB21EC0 X-Spam-Score: -1.95 X-Migadu-Scanner: scn0.migadu.com X-TUID: uTvoeykTThke * guix/lint.scm (lookup-disarchive-spec): New procedure. (check-archival): When 'lookup-content' returns #f, call 'lookup-disarchive-spec'. * guix/download.scm (%disarchive-mirrors): Make public. --- guix/download.scm | 1 + guix/lint.scm | 31 +++++++++++++++++++++++++++---- 2 files changed, 28 insertions(+), 4 deletions(-) Hello! This patch makes the ‘archival’ checker check the Disarchive database(s) when SWH ‘lookup-content’ returns #f. For example, before the patch, we get: $ guix lint -c archival guile-json gnu/packages/guile.scm:622:12: guile-json@4.5.2: source not archived on Software Heritage After the patch, we get nothing (success) thanks to Disarchive metadata available at: https://disarchive.ngyro.com/sha256/1ab046ec36b1c44c041ac275568d818784d71fab9a5d95f9128cfe8a25051933 It assumes that the swhid found in the Disarchive metadata is valid, a reasonable assumption IMO. Thoughts? Ludo’. diff --git a/guix/download.scm b/guix/download.scm index 72094e7318..b6eb97e6fa 100644 --- a/guix/download.scm +++ b/guix/download.scm @@ -35,6 +35,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) #:export (%mirrors + %disarchive-mirrors (url-fetch* . url-fetch) url-fetch/executable url-fetch/tarbomb diff --git a/guix/lint.scm b/guix/lint.scm index 1bebfe03d3..c6ad54ddeb 100644 --- a/guix/lint.scm +++ b/guix/lint.scm @@ -30,6 +30,7 @@ (define-module (guix lint) #:use-module (guix store) + #:autoload (guix base16) (bytevector->base16-string) #:use-module (guix base32) #:use-module (guix diagnostics) #:use-module (guix download) @@ -1227,6 +1228,23 @@ upstream releases") #:field 'source))))))) +(define (lookup-disarchive-spec hash) + "Return true if Disarchive mirrors have a spec for HASH, false otherwise." + (any (lambda (mirror) + (with-networking-fail-safe + (format #f (G_ "failed to access Disarchive database at ~a") + mirror) + #f + (let* ((url (string-append mirror + (symbol->string + (content-hash-algorithm hash)) + "/" + (bytevector->base16-string + (content-hash-value hash)))) + (response (http-head url))) + (= 200 (response-code response))))) + %disarchive-mirrors)) + (define (check-archival package) "Check whether PACKAGE's source code is archived on Software Heritage. If it's not, and if its source code is a VCS snapshot, then send a \"save\" @@ -1302,10 +1320,15 @@ try again later") (symbol->string (content-hash-algorithm hash))) (#f - (list (make-warning package - (G_ "source not archived on Software \ -Heritage") - #:field 'source))) + ;; If SWH doesn't have HASH as is, it may be because it's + ;; a hand-crafted tarball. In that case, check whether + ;; the Disarchive database has an entry for that tarball. + (if (lookup-disarchive-spec hash) + '() + (list (make-warning package + (G_ "source not archived on Software \ +Heritage and missing from the Disarchive database") + #:field 'source)))) ((? content?) '()))) '())))) -- 2.31.1