From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id GIS9N1dpGGSCgQEASxT56A (envelope-from ) for ; Mon, 20 Mar 2023 15:10:32 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id KHXMNldpGGREQAEAG6o9tA (envelope-from ) for ; Mon, 20 Mar 2023 15:10:31 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9440F11944 for ; Mon, 20 Mar 2023 15:10:27 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=inria.fr header.s=dc header.b=PXyR6HVL; dmarc=fail reason="SPF not aligned (relaxed)" header.from=inria.fr (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1679321427; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=ZXgWW4DhDQrUfTaQOnOoMqLhzH2AVg6/Tt35qHUcGk0=; b=l7ZIcttA3BgCw5EXWs8+P4Ap5tESOxNUlDjyzoOzh4ghvmME9Jkeyq5+kUWcCa62bHOXap wL6cHJc6tTesbfiKMF7Xy0UDKAGMHUBYDNTaHVeDDjJ316qXR2EuBIHebwFO2EdYjTb5tN erWITl7aPz3yAqVWoRUbgptX04ngLuiX/v8CP7QhTrUe6/zgc3gJetPhK51//D61rQi+ZU arq8PRybA4hGaXqu6/ng7HcpFy/rKbZA4WKjDUQzD9lbttyxW78UgrGN6n/qNhyuf/1YT3 3yNuIiUtVZNWKbRHVAKU3XSzpJk8olFIQCT8hRU3ytKdYbev/aPjxns41Mz8Ig== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=inria.fr header.s=dc header.b=PXyR6HVL; dmarc=fail reason="SPF not aligned (relaxed)" header.from=inria.fr (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1679321427; a=rsa-sha256; cv=none; b=j+7xRbax0JD2n7okD/34aWc0smlhS6uG81xikPN7TfAmiMRLKjImStDuL2zC1o2+l4HI0g 2l36RSyUDm9Cbe6lkPFOUUWpN+kgtoi2KFbxGd8gdfZSJP9g7UJ74lMH66dz8NnjdiNO/0 V2lKWpZlNeFsLwy4EQdb3Q8EsOMgmKsXWR0mLr5uqjMtBC/GnUOHaZpnHYRcock/TQW/zM cvQFQ0FYfM4096FEG9EUNxpP7D4lHCKjPRoetF9ww0Uj6TUENYPgM/DZimlYUjSq199M3D xNt3VfC6IDN8HcAY+n3swUX7AQhZUQ4UyQxoCWzNfBc1f4xVEeremLYLhjk9KA== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1peGD8-0008K1-Mp; Mon, 20 Mar 2023 10:10:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1peGD6-0008Ik-C3 for guix-patches@gnu.org; Mon, 20 Mar 2023 10:10:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1peGD4-0004Cj-Hp for guix-patches@gnu.org; Mon, 20 Mar 2023 10:10:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1peGD4-0004DZ-Ar for guix-patches@gnu.org; Mon, 20 Mar 2023 10:10:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43442] [PATCH] Fixes init of #42162: gforge.inria.fr down Dec. 2020 Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 20 Mar 2023 14:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43442 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: zimoun Cc: 43442@debbugs.gnu.org Received: via spool by 43442-submit@debbugs.gnu.org id=B43442.167932136116148 (code B ref 43442); Mon, 20 Mar 2023 14:10:02 +0000 Received: (at 43442) by debbugs.gnu.org; 20 Mar 2023 14:09:21 +0000 Received: from localhost ([127.0.0.1]:55443 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1peGCO-0004CN-Vw for submit@debbugs.gnu.org; Mon, 20 Mar 2023 10:09:21 -0400 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:51064) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1peGCM-0004C2-DC for 43442@debbugs.gnu.org; Mon, 20 Mar 2023 10:09:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=ZXgWW4DhDQrUfTaQOnOoMqLhzH2AVg6/Tt35qHUcGk0=; b=PXyR6HVLP05J1oetvEKzKq8zZoISabxVafitfcnSvHIXQJe9iXpqb0uy oninQB2KYeepWSAYlmNFGSHR3HpazwUDh2lkieI2ANMJqLtpLfYD7JiVL e1eiNTjz4BxFZrTQ1qAmEdUOXgjmW/P9RlwcfZ3Wuy42v6d1gKwg3DM2m A=; X-IronPort-AV: E=Sophos;i="5.98,274,1673910000"; d="scan'208";a="50699762" Received: from unknown (HELO ribbon) ([193.50.110.213]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2023 15:09:12 +0100 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200916081411.5801-1-zimon.toutoune@gmail.com> <20200916081642.6716-1-zimon.toutoune@gmail.com> <87o8lyj090.fsf@gnu.org> <86zh5iol1n.fsf@gmail.com> <87h7rocvkt.fsf@gnu.org> <877dsi44lu.fsf@gnu.org> <87y2knhei3.fsf@gnu.org> Date: Mon, 20 Mar 2023 15:09:11 +0100 In-Reply-To: <87y2knhei3.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Sat, 03 Oct 2020 10:59:00 +0200") Message-ID: <87jzzbms54.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: X-Migadu-Queue-Id: 9440F11944 X-Spam-Score: -1.16 X-Migadu-Spam-Score: -1.16 X-Migadu-Scanner: scn0.migadu.com List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-TUID: g6MdzKN0nxLf --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi! Ludovic Court=C3=A8s skribis: > Ah yes, under =E2=80=9Cextra_headers=E2=80=9D there=E2=80=99s the SVN rev= ision number. (guix > swh) doesn=E2=80=99t expose =E2=80=9Cextra_headers=E2=80=9D yet, but once= it does, we can walk > snapshots until we find the SVN revision we=E2=80=99re looking for. > > scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/= svn/mpfi/") > $3 =3D #< visits-url: "https://archive.softwareheritage.org/api/1= /origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visits/" type: #f url:= "https://scm.gforge.inria.fr/anonscm/svn/mpfi"> > scheme@(guile-user)> (origin-visits $3) > $4 =3D (#< date: # origin: "https://scm.gf= orge.inria.fr/anonscm/svn/mpfi" url: "https://archive.softwareheritage.org/= api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visit/1/" snapsho= t-url: "https://archive.softwareheritage.org/api/1/snapshot/e7fdd4dc6230f71= 0dbc55c1b308804fa1b5f51f0/" status: full number: 1>) > scheme@(guile-user)> (visit-snapshot (car $4)) > $5 =3D #< branches: (#< name: "HEAD" target-type: revis= ion target-url: "https://archive.softwareheritage.org/api/1/revision/f7b445= a6bdc38bf075f29265120ca49824f698ea/">)> > > So the next step is to augment (guix swh) with a > =E2=80=98lookup-subversion-revision=E2=80=99 procedure. The attached patch does that: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> (lookup-subversion-revision "https://scm.gforge.inria.= fr/anonscm/svn/mpfi" 680) $12 =3D #< id: "72102de7605a2459ebcb016338ebbf1a997e8c8d" date: #= directory: "5c89c025a4cd9d16befdfec12dfc23f7318d0d5= b" directory-url: "https://archive.softwareheritage.org/api/1/directory/5c8= 9c025a4cd9d16befdfec12dfc23f7318d0d5b/" parents-ids: ("16da41f1848d77a93aec= 565320b72b460c429b61") extra-headers: (("svn_repo_uuid" . "e2f78e0c-bb60-47= 09-9413-9660a31d4696") ("svn_revision" . "680"))> scheme@(guile-user)> (lookup-subversion-revision "https://scm.gforge.inria.= fr/anonscm/svn/mpfi" 666) $13 =3D #< id: "148eb1e7206b111af4075c73c656e54c9efed6cb" date: #= directory: "ed7b0bd7019fb85cd86d948a97c23b9d43aa8728= " directory-url: "https://archive.softwareheritage.org/api/1/directory/ed7b= 0bd7019fb85cd86d948a97c23b9d43aa8728/" parents-ids: ("0ba2aa7e0d3fc0a1eb3ba= 72b32094515415ae47a") extra-headers: (("svn_repo_uuid" . "e2f78e0c-bb60-470= 9-9413-9660a31d4696") ("svn_revision" . "666"))> --8<---------------cut here---------------end--------------->8--- The implementation is pretty bad though, because it walks the revision history until it finds the right revision number=E2=80=94so you=E2=80=99re = likely to reach the bandwidth rate limit before you=E2=80=99ve found the revision you= =E2=80=99re looking for. More importantly, most svn origins cannot be found, or at least not by passing the URL as-is: https://sympa.inria.fr/sympa/arc/swh-devel/2023-03/msg00009.html This whole hack looks like a dead end. It would be ideal if SWH would compute nar hashes, as you proposed: https://gitlab.softwareheritage.org/swh/meta/-/issues/4538 As a stopgap, I wonder if we could use =E2=80=9Cdouble hashing=E2=80=9D on = our side, but only for svn: we=E2=80=99d store both the nar sha256 as we currently do, pl= us the swhid. It still seems to me that it=E2=80=99d be hard to scale and to maintain that over time, even if it=E2=80=99s limited to svn. Plus, there= =E2=80=99d still be the problem of =E2=80=98svn-multi-fetch=E2=80=99, which is what mo= st TeX=C2=A0Live packages use. Thoughts? Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable diff --git a/guix/swh.scm b/guix/swh.scm index c7c1c873a2..a65635b1db 100644 --- a/guix/swh.scm +++ b/guix/swh.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright =C2=A9 2018, 2019, 2020, 2021 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2018, 2019, 2020, 2021, 2023 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2020 Jakub K=C4=85dzio=C5=82ka ;;; Copyright =C2=A9 2021 Xinglu Chen ;;; Copyright =C2=A9 2021 Simon Tournier @@ -75,8 +75,10 @@ (define-module (guix swh) revision-id revision-date revision-directory + revision-parents lookup-revision lookup-origin-revision + lookup-subversion-revision =20 content? content-checksums @@ -207,6 +209,14 @@ (define string* ((? null?) #f) ;Guile-JSON 3.x ('null #f))) ;Guile-JSON 4.x =20 +(define pair-vector->alist + (match-lambda + ('null '()) + ((=3D vector->list lst) + (map (match-lambda + (#(key value) (cons key value))) + lst)))) + (define %allow-request? ;; Takes a URL and method (e.g., the 'http-get' procedure) and returns t= rue ;; to keep going. This can be used to disallow requests when @@ -346,7 +356,14 @@ (define-json-mapping make-revision revision? (id revision-id) (date revision-date "date" (maybe-null string->date*)) (directory revision-directory) - (directory-url revision-directory-url "directory_url")) + (directory-url revision-directory-url "directory_url") + (parents-ids revision-parent-ids "parents" + (lambda (vector) + (map (lambda (alist) + (assoc-ref alist "id")) + (vector->list vector)))) + (extra-headers revision-extra-headers ;alist--e.g., with "svn_revis= ion" + "extra_headers" pair-vector->alist)) =20 ;; (define-json-mapping make-content content? @@ -524,6 +541,50 @@ (define (lookup-origin-revision url tag) (() #f))))) =20 +(define (revision-parents revision) + "Return the parent revision(s) of REVISION." + (filter-map lookup-revision (revision-parent-ids revision))) + +(define (lookup-subversion-revision-in-history revision revision-number) + "Look for Subversion REVISION-NUMBER starting from REVISION and going ba= ck +in history." + (let loop ((revision revision)) + (let ((number (and=3D> (assoc-ref (revision-extra-headers revision) + "svn_revision") + string->number))) + (and number + (cond ((=3D number revision-number) + ;; Found it! + revision) + ((< number revision-number) + ;; REVISION is ancestor of REVISION-NUMBER, so stop here. + #f) + (else + ;; Check the parent(s) of REVISION. + (any loop (revision-parents revision)))))))) + +(define (lookup-subversion-revision url revision-number) + "Return either #f or the revision of the Subversion repository once +available at URL with the given REVISION-NUMBER." + (match (lookup-origin url) + (#f #f) + (origin + (match (filter (lambda (visit) + ;; Return #f if (visit-snapshot VISIT) would return= #f. + (and (visit-snapshot-url visit) + (eq? 'full (visit-status visit)))) + (origin-visits origin)) + (() + #f) + ((visit . _) + (any (lambda (branch) + (match (branch-target branch) + ((? revision? revision) + (lookup-subversion-revision-in-history revision + revision-number)) + (_ #f))) + (snapshot-branches (visit-snapshot visit)))))))) + (define (release-target release) "Return the revision that is the target of RELEASE." (match (release-target-type release) --=-=-=--