From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id yItjL2017GGe8wAAgWs5BA (envelope-from ) for ; Sat, 22 Jan 2022 17:48:45 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id gLk0KG017GGnaQEAG6o9tA (envelope-from ) for ; Sat, 22 Jan 2022 17:48:45 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1E6B72B525 for ; Sat, 22 Jan 2022 17:48:45 +0100 (CET) Received: from localhost ([::1]:57548 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nBJZE-0006Fz-9e for larch@yhetil.org; Sat, 22 Jan 2022 11:48:44 -0500 Received: from eggs.gnu.org ([209.51.188.92]:46276) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nBJYu-0006Fq-3K for guix-devel@gnu.org; Sat, 22 Jan 2022 11:48:24 -0500 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:41773) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nBJYr-0004Yu-Nj; Sat, 22 Jan 2022 11:48:23 -0500 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 1E7F55C00A6; Sat, 22 Jan 2022 11:48:19 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Sat, 22 Jan 2022 11:48:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=u6EXdPoi5S75PIvq/ AH965KEG7VmIJbPNAZMAzFHrBI=; b=i5MKlsLnafQZM8/v2p188Tn2UTf+IZKCl VK6CR4wARozn92GsOyWaJ025/gyfbmCvleeFEm9bE3cwVA1BYN12xqirItCBjLpv JJvPgmMc2PbeQWwh8/XkyiddlpqNGcbSFrFBmSW+UXN+vC4zPEPFhzt5usS577Nl AnvpAVJ5P+DYYShzsZo+BwcnWyYcd9oqJCnm4tr9Svk/el/FzEz6Ox4RECo71bRD J2f5TaAVjpI0EpBD7MeC3CvD6Y65hBY5GZhbnuRl0X3il8vtvPVbzRXVKEDMAQjR IB4euTgu86hs3fG/ZjRi4kXiFEBkS5ajwS9E6wtmxtBwE87jEMH8g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrvddvgdelgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufhffjgfkfgggtgesmhdtreertderjeenucfhrhhomhepvfhimhhothhh hicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhghihrohdrtghomheqnecuggftrfgrth htvghrnhepvefgleekieffieejvefggeevudevkeetheehleeuleekfeetffethfevudek kefgnecuffhomhgrihhnpehnghihrhhordgtohhmnecuvehluhhsthgvrhfuihiivgeptd enucfrrghrrghmpehmrghilhhfrhhomhepshgrmhhplhgvthesnhhghihrohdrtghomh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 22 Jan 2022 11:48:18 -0500 (EST) From: Timothy Sample To: zimoun Subject: Re: ImageMagick from 2020? References: <87tuebwlrc.fsf@gmail.com> <87k0f71ndv.fsf@ngyro.com> <03cbf268883ae93ae45b20e9ab8dee49ca5e7dd0.camel@telenet.be> <87a6ftglbx.fsf@gnu.org> <87k0ewau6y.fsf@gnu.org> Date: Sat, 22 Jan 2022 11:48:17 -0500 In-Reply-To: (zimoun's message of "Wed, 19 Jan 2022 12:30:52 +0100") Message-ID: <87pmoj7m3y.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=66.111.4.29; envelope-from=samplet@ngyro.com; helo=out5-smtp.messagingengine.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1642870125; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=u6EXdPoi5S75PIvq/AH965KEG7VmIJbPNAZMAzFHrBI=; b=twa4RWIDJ2YPohIlTiBsGhLoXvDyWpsG+XksNL35LGNF4mJE00vMIh2MFBaPHP/YDjQiOq V1eTWOg/4f/ynv0pQCAQkxhjnuoVwWipLqUFsWeKjFIACvQ7sAlsauK8HExZHUVWivj/6w 9/Zmg90PxwCIhBkrAu5QKx34TwZdQCFhnhPjoDeYflLKxRF9JIVIqMvesfr4rZvsIo1wbh DxZBmNCtVExXKvxw0gWKwX/uOfubC4pttj9HqBwuGlCyNq599M18pH53JU9/2LLVj1uAOI rhGU9sxIURdp6w5dI28vDH5ZLDF3KUp39q7tnVd9SRhhqLAUhRvs6bwaVRZvsQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1642870125; a=rsa-sha256; cv=none; b=lulIRKbFzC7ATiQ8RuTUDwgn2m7AikFUOnBns24SWFYiVzQuNBjlNn0oU41qsBOdz+82z7 JalFsD54ZnTToQtqZyxgtbhC/5RRlsfwmFCCJdJ3M+cS8P8Iyq+ir6uKl3zLDS37tM8i0z hJPAf1K+VKL0KdvDbsPmkDv43FCxsG4lIMgOq7HLCLW5QL6df4AP8jtbWqM3tyE6TUPdNN QwgZgvm2RMqFm5DIY5aO7FLvyxfhdb5wZMDv7LsgOhlUE1HYJ8R4qD/ipAw3SbO5SQi8wi ufrncU2sL5/oerUv7TFKIlGbTuOtyiSDGXmMuiAQgqHV05tUyY2Uu9u6aKb/XA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm1 header.b=i5MKlsLn; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.73 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm1 header.b=i5MKlsLn; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 1E6B72B525 X-Spam-Score: -1.73 X-Migadu-Scanner: scn0.migadu.com X-TUID: ksGoNMOQQT6G --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hey, zimoun writes: > On Wed, 19 Jan 2022 at 11:36, Ludovic Court=C3=A8s wrote: > >> Oh right, so we=E2=80=99ll need to feed them historical =E2=80=98sources= .json=E2=80=99 files >> eventually, I think Timothy was planning to do that eventually. > > From my side, what I would like to achieve soon: > > [...] > > Then what is also missing is: > > 3- have a collection of sources.json per Guix revision -- at least > some determined by us; I=E2=80=99ve attached a script that makes a =E2=80=9Csources.json=E2=80=9D = per commit from the PoG database. It only lists regularly downloaded sources (no VCS sources), since that=E2=80=99s all the SWH loader supports so far. I also played around with it and came up with https://ngyro.com/pog-reports/2022-01-16/missing-sources.json This is a =E2=80=9Csources.json=E2=80=9D that only lists the =E2=80=9Cmissi= ng=E2=80=9D and =E2=80=9Cunknown=E2=80=9D sources from the PoG report. It lists sources across all commits (since 1.0.0). This might be the easiest thing for SWH to handle, since it omits nearly 20k sources that they definitely already have. Since they don=E2=80=99t have the tarball hashes, they have no way to skip downloading= and processing tarballs that they already have by hash. Hence, filtering it with the extra data we have through the PoG projects should be something that they welcome! If they want, they could point a loader task at https://ngyro.com/pog-reports/latest/missing-sources.json and I could publish updates when I publish new PoG reports. There=E2=80=99s one other thing to think about. Some of our sources are arguably unsuitable for SWH. For instance, our bootstrap binaries. I bet we have a bunch of other borderline things, too, like game assets. Of course, if they are indiscriminately ingesting Github, I=E2=80=99m sure they=E2=80=99ve loaded plenty of garbage. Mostly, I think about these thin= gs because I believe it=E2=80=99s important to maintain the Guix-SWH relations= hip. Here=E2=80=99s the per-commit script. You can run it like this: $ guile sources.scm pog.db output-directory --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=sources.scm (use-modules (gcrypt base64) (guix base32) (guix build download) ((guix download) #:select (%mirrors)) (ice-9 match) (json) (sqlite3) (srfi srfi-1) (srfi srfi-9 gnu) (srfi srfi-19) (web uri)) (define-immutable-record-type (make-commit push-time hash) commit? (push-time commit-push-time) (hash commit-hash)) (define lookup-commits-query "\ SELECT c.push_time, c.hash FROM commits c ORDER BY c.push_time DESC") (define (lookup-commits db) (define (record->commit rec) (match-let ((#(push-time hash) rec)) (make-commit (and push-time (make-time time-utc 0 push-time)) hash))) (define (kons rec acc) (cons (record->commit rec) acc)) (let* ((stmt (sqlite-prepare db lookup-commits-query)) (commits (sqlite-fold kons '() stmt))) (sqlite-finalize stmt) commits)) (define lookup-sources-query "\ SELECT f.hash, fr.reference FROM commits c JOIN fod_commit_links fcl USING (commit_id) JOIN fods f USING (fod_id) JOIN fod_references fr USING (fod_id) WHERE c.hash = ? AND f.algorithm = 'sha256' AND (fr.reference LIKE '\"%' OR fr.reference LIKE '(\"%') AND NOT fr.is_error") (define (nix-base32-sha256->subresource-integrity digest) "Convert the Nix-style base32-encoded SHA-256 hash DIGEST into a Subresource Integrity metadata value." (define bv (nix-base32-string->bytevector digest)) (define b64 (base64-encode bv)) (string-append "sha256-" b64)) (define (web-reference-urls reference) (define uris (match (call-with-input-string reference read) ((urls ...) (map string->uri urls)) (url (list (string->uri url))))) (append-map (lambda (uri) (map uri->string (maybe-expand-mirrors uri %mirrors))) uris)) (define (lookup-sources db commit) (define (record->url-source rec) (match-let ((#(digest reference) rec)) (let ((urls (web-reference-urls reference)) (integrity (nix-base32-sha256->subresource-integrity digest))) `(("type" . "url") ("urls" . ,(list->vector urls)) ("integrity" . ,integrity))))) (define (kons rec acc) (cons (record->url-source rec) acc)) (let* ((stmt (sqlite-prepare db lookup-sources-query)) (_ (sqlite-bind-arguments stmt commit)) (sources (sqlite-fold kons '() stmt))) (sqlite-finalize stmt) sources)) (define (commit-sources-name directory commit) (string-append directory "/" (date->string (time-utc->date (commit-push-time commit)) "~Y-~m-~d") "-" (string-take (commit-hash commit) 7) "-sources.json")) (match (program-arguments) ((_ db-file directory) (mkdir directory) (let* ((db (sqlite-open db-file)) (commits (lookup-commits db))) (for-each (lambda (commit) (call-with-output-file (commit-sources-name directory commit) (lambda (out) (let* ((hash (commit-hash commit)) (sources (lookup-sources db hash))) (scm->json `(("version" . "1") ("revision" . ,hash) ("sources" . ,(list->vector sources))) out) (newline out))))) (list (car commits))) (sqlite-close db) (exit EXIT_SUCCESS))) (_ (display "usage: sources.scm DB-FILE\n" (current-error-port)) (exit EXIT_FAILURE))) --=-=-= Content-Type: text/plain -- Tim --=-=-=--