From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id aP5VOTfKcmGR5wAAgWs5BA (envelope-from ) for ; Fri, 22 Oct 2021 16:27:03 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id oOnrNDfKcmFUTAAAB5/wlQ (envelope-from ) for ; Fri, 22 Oct 2021 14:27:03 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id AC390311C1 for ; Fri, 22 Oct 2021 16:27:03 +0200 (CEST) Received: from localhost ([::1]:46972 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mdvVe-0002Jo-SO for larch@yhetil.org; Fri, 22 Oct 2021 10:27:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35992) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mdvOJ-0007o8-Qz for guix-devel@gnu.org; Fri, 22 Oct 2021 10:19:29 -0400 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:32891) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mdvOG-0007HL-PL; Fri, 22 Oct 2021 10:19:27 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 265A33200E60; Fri, 22 Oct 2021 10:19:19 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Fri, 22 Oct 2021 10:19:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=8qq+mwOcJNxS7pn0BmokmEa+mXTK2X/noD7o6L8eG 6k=; b=QpmDCFSG1tx05FmGXwwj71UWSsp1qQfMlkOe2dCJhmIdZy31hWSEETKCK hQsLjeQ+ZcHnm1Kk5q0Pd/f3LBqHgO5lmgYfwfpXpnrSXFAHpE87Dxt4+++pZd8h 1FCC1RSCCI5GRcpwHdVvs7WSDtAEsrM0eOYy7GiJ5aRP3H0AeQHZn/tQ6BoViKmO NcnZPthKbguYirh8gYGj24SJ0uNjPzlwZIYuCgB8BLoVHTFQQYjbRz6+BmNSTP/S dX2LSj72FDU6P7k22n9l2hUSjOAp/cw6ZcOzLvDEA/kGKWCkriVdaz3xNQDolMbw 7/xS+Tl3QDSuAqiQ6qyCEhfa5ujXA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrvddvkedgjedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufhfffgjkfgfgggtgfesthhqredttderjeenucfhrhhomhepvfhimhho thhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhghihrohdrtghomheqnecuggftrf grthhtvghrnhepveevgfeuudevhedtvdeiffffkedtteegvdehkeevkeeugfetffeuvedu vdeigeeunecuffhomhgrihhnpehnghihrhhordgtohhmpdhsohhfthifrghrvghhvghrih htrghgvgdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhl fhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 22 Oct 2021 10:19:18 -0400 (EDT) From: Timothy Sample To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Preservation of Guix Report References: <87o87jjx54.fsf@ngyro.com> <87a6j2w1et.fsf@gnu.org> Date: Fri, 22 Oct 2021 10:19:17 -0400 In-Reply-To: <87a6j2w1et.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Thu, 21 Oct 2021 22:47:22 +0200") Message-ID: <87tuh9cfbu.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=64.147.123.24; envelope-from=samplet@ngyro.com; helo=wout1-smtp.messagingengine.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1634912823; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=8qq+mwOcJNxS7pn0BmokmEa+mXTK2X/noD7o6L8eG6k=; b=QbdcOd82EBXTgwL7vq5Iz8JEBzfpuQc74XqxjuR55RylBgyQ6cmipKEm6zzTufQEXyjVO3 jlHwONzpElhGU2/u6CEqjuGI2/ypQZwH70qLLL9g6SBl0NNAEmGgjzCHfvAqFCE2NIpQ0S SNg8MsMqJQ8CRDqnRojhdE+L090/Px2YolWgbHXrXaXxYX9KPtbtAG62VnuJjc/drUjZVt XoutPuJEQuvm/1kKV2KHqrkCUOayO4sYkBXi1lCSMzWS0mNMWEYQL1K61O0u/el2abZrIm biiDBfaUIXqjFpr6zTrnRUyZQW5186eNFcygif2WnEc6sZzgKWfOdtpgYDgAVw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1634912823; a=rsa-sha256; cv=none; b=SInPAxJVLnDIJUfCHoUjRdcjoXOVh09oQi7j1iWm5zYwn+N5DE2g4B4PJV4aaRMecy9klL fsVd47aPPz2P+W36okqf/OLRyu7UcWgxRLCQdmlJwpcuxvUra4h/yKBrxJKfRoPu7iVtru 4w2y+KloFb+rUFucgCbN3xMUxqjNBvaUpTrXPrxQzd2h/bp9LbzmieRnRgpyPTFmSYNCmC R+ek4OyFYhsZo5nnPRcosPZtNOUz3XJYZpI5AJzb6xUuE56EowV8+W4RzIA7aVF6pq/Odj vLJw8EwmkUVwfxwTMjrUW34ls9yTN3asInDRg4YKeO2a2ao34d++/WlfU36gdg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=QpmDCFSG; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.63 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=QpmDCFSG; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: AC390311C1 X-Spam-Score: -2.63 X-Migadu-Scanner: scn0.migadu.com X-TUID: ItqD04BmNSha Hey, Ludovic Court=C3=A8s writes: > Timothy Sample skribis: > >> Early this summer I did a bunch of work trying to figure out which Guix >> sources are preserved by the SWH archive. I=E2=80=99m finally ready to = share >> some preliminary results! >> >> https://ngyro.com/pog-reports/2021-10-20/ >> >> This report is already quite outdated, though. It only covers commits >> up to the end of May, and sometime in June is when the sources were >> checked against the SWH archive. I=E2=80=99m sharing it now to avoid any >> further delays. > > This is truly awesome! (Did you manage to grab all that info with the > default rate limit?!) Yes, but I have another trick. The =E2=80=9Cknown=E2=80=9D endpoint [1]. = If you already know the SWHIDs you want to check, you can check 1,000 per call. With the anonymous rate limit, I can check 120,000 every hour, which is plenty. [1] https://docs.softwareheritage.org/devel/swh-web/uri-scheme-api.html#get= --api-1-content-known-(sha1)[,(sha1),%20...,(sha1)]- > I can=E2=80=99t wait for the updated report now that Simon and yourself h= ave > identified that SWHID computation bug! I=E2=80=99m computing SWHIDs while writing this. Not long now! > Some of our refer to tags, not commits. How do you > determine whether they=E2=80=99re saved? The short answer is =E2=80=9Celbow grease=E2=80=9D. Basically, I=E2=80=99m= taking a =E2=80=9Cwork harder, not smarter=E2=80=9D approach. :p I go out and obtain the source, verify it with Guix=E2=80=99s hash, and then compute the SWHID. This is an= other thing we could move to the CI infrastructure, but I think there might be some hiccoughs. For git-references, I believe we can=E2=80=99t just comput= e the ID after the download derivation =E2=80=93 we would have to change the down= load derivation itself. Maybe add an =E2=80=98swhid=E2=80=99 output? It=E2=80= =99s a little more complicated than just throwing up some scripts, anyway. > =E2=80=98guix lint -c archival=E2=80=99 uses =E2=80=98lookup-origin-revis= ion=E2=80=99, which is a good > approximation, but it=E2=80=99s not 100% reliable because tags can be mod= ified > and that procedure only tells you that a same-named tag was found, not > that it=E2=80=99s the commit you were expecting. (And really, we should = stop > referring to tags.) Like zimoun said elsewhere in this thread, having an explicit mapping from Guix hash to SHWID will improve reliability quite a bit. It=E2=80=99s= hard to get to 100%, though! With the reports, we will eventually be able to check everything. However, there=E2=80=99s still a small possibility of bu= gs and false positives. Ultimately, I=E2=80=99m hoping the reports will help detect small problems (some specific source is missing) and guide our efforts on big problems (xz support in Disarchive or support for more version control systems, etc.). -- Tim