From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id UKt1AmCRVmDuDgAA0tVLHw (envelope-from ) for ; Sun, 21 Mar 2021 00:20:48 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id V8HsOV+RVmBdNgAA1q6Kng (envelope-from ) for ; Sun, 21 Mar 2021 00:20:47 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 584C920AC3 for ; Sun, 21 Mar 2021 01:20:47 +0100 (CET) Received: from localhost ([::1]:52856 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lNlpk-0007Qb-0P for larch@yhetil.org; Sat, 20 Mar 2021 20:20:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54242) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lNlp3-0007ON-O6 for guix-devel@gnu.org; Sat, 20 Mar 2021 20:20:01 -0400 Received: from mail-wr1-x42f.google.com ([2a00:1450:4864:20::42f]:45798) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lNlp1-0001bZ-8C for guix-devel@gnu.org; Sat, 20 Mar 2021 20:20:01 -0400 Received: by mail-wr1-x42f.google.com with SMTP id 61so12849288wrm.12 for ; Sat, 20 Mar 2021 17:19:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=EL9R30mbV7cmZMiAY88Y7FsmmfWImisSMDR4cn7Y6eY=; b=DabOto231IKvliyfQW0N91vZuhJIw/i9cuAD7l4DUTTlNTh1A8yVPhmfE095cuV6Pn 8xk3strNh8h4jP4MOSSVxhCWzQMzwxBGdRdyyrwbBfTYdlQ4xc+Yv3zoHXeBVXiaheID MMqfExf7GVWkJL0FTesbgGqK8dFLYRrsjyLk+8NXhUFJI4QD0qA/xR8rZ8aRedcPHBK2 pavy8/q0Pqlqs4GouRgIJ9ptjUIhKQYqcEMGki/JPHN1FmAl8xFJ50C49zcD+oAL1T9N IGZ/I01t+Xjqk14fcGTQpRnlYIYysKiHuax5TPsQr6v6fjUcdUtzyJLipeNfR3tSjBrc cUtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=EL9R30mbV7cmZMiAY88Y7FsmmfWImisSMDR4cn7Y6eY=; b=OMaHNKP+p1431cULCAraRoFE3DvlmfDMEhUoQMbVH3eG6U4Ub1cavyyKukSQZZ7Lg7 78D0UKvWTXSaZhWJlXozYmx79mls4VM6Hru6Hdo4/+GlSxgymeuirKG0ZDzROGICE5tk CmCclYexFPhsumR8gHDhLhwmMzRnFzDkWhklX288sE6j6Bocb7N8Mnew+g4tPJX8AhYg RV+UlSYbeGAA8hYRemBACP5fSCfdY94GyccmHFkf5uxHWHBjc4NkKhcEAvS61MffBFt3 ASE5tDkzpTVR3uNHM8fdTh/dZaahgxS5iZoBcf5Qe3MQ9veB5a8fkfd/gX2vW0RvXtCJ 7jMg== X-Gm-Message-State: AOAM531H1FpgzOBgMiB9m20jFjTW9s5NBjFnUrfjuRZLwwvFCAVk16e3 L24o7o43kgu5JwCBRiCxlT7+BZcNV7g= X-Google-Smtp-Source: ABdhPJyA5nXo4m8XhZhXgzfAB6aIGDX4/fp9C2FQ1R06NrTDRqSkfczcgsTATuvAU9Dhm6/O+YZP3g== X-Received: by 2002:adf:dfc3:: with SMTP id q3mr11590975wrn.121.1616285997403; Sat, 20 Mar 2021 17:19:57 -0700 (PDT) Received: from lili ([2a01:e0a:925:4d20:1002:72b2:7492:ea32]) by smtp.gmail.com with ESMTPSA id l15sm14796403wru.38.2021.03.20.17.19.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 20 Mar 2021 17:19:56 -0700 (PDT) From: zimoun To: Luis Felipe Subject: Re: Release 1.2.1: status In-Reply-To: References: <86zgz0tu21.fsf@gmail.com> Date: Sun, 21 Mar 2021 01:16:57 +0100 Message-ID: <86sg4puzra.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::42f; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x42f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616286047; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=EL9R30mbV7cmZMiAY88Y7FsmmfWImisSMDR4cn7Y6eY=; b=hrWKcawq/9TCRig+cjRPnh/Xl/BnFNO6Tf+oacgEAddAyfTerSvRmtgUYW4tyh/NNGTv7F XZgLc9WaEPvvNGryxaRaCGIcSZfI8rwbP7u6kgIXXZHweLtoyKK+hqiYDsheDazzHJjF7I HUBHTeQ4bPKQG6gQxbCMybJRlQBVCrCpmkqn9JAd1vf/DjD8/Hawb00HwavZrqhNIypcrz jzALWjAZMUMP+FUZ0CwOLR0zrwEXqL2q22a5WwHzimCdsoq3mGkpmMgaKbZbLOzoj5cfvo cr6OsqwdN33uzN4RFwJgIwmncIXyh15volifqvS9cHvPveFW9bFQHT1enTnYgg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616286047; a=rsa-sha256; cv=none; b=ijJ3pJOz0cH5mUE0EkAue5JbVhQbAatN+x5nT6WEKFdyRpHzCQD+rXe7iMzQB26dekUCWC ziL+Cfz05o3YNUFwrp8JshjxPAfWWhfiQMJ2X0cGVS3ojR10S26TajOpO1/zEKYFi9JdDH ac/8n7PPnEPYQjknkXSV1YNmhsY15o1PJ0XEwKR74jNxnn7jzy4ORbn/ilfxtV4CkwF4dx Udq2aMHm+OlJkGJPTfOFT3G9pWNx/KnuCxVQh73iqbjq+ZcfelTDirMN61NuIcyG2eMMQg 5wxJXE2ygQVWY4VItS0hkxcr1GcU7sm0L8p02jeqLVwCH0+nR/41jckaQ9YACw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=DabOto23; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.11 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=DabOto23; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 584C920AC3 X-Spam-Score: -3.11 X-Migadu-Scanner: scn0.migadu.com X-TUID: 3sjaTBHmc1gT Hi Luis, Thanks for testings and reporting. On Fri, 19 Mar 2021 at 22:26, Luis Felipe w= rote: > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original = Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 > On Thursday, March 18, 2021 2:28 PM, zimoun wr= ote: > > [...] > >> We are still missing a good story to monitor what is archived on >> Software Heritage and what is not. Because for now there is rate lim= it, >> I am not able to automate=E2=80=A6 Well, it is a long WIP. :-) >> >> To help and avoid this rate limit, if all of us simply run: >> >> for pkg in $(guix package -I | cut -f1); >> do >> guix lint -c archival $pkg >> done >> >> for all our profiles, it will ensure at least a coverage for the >> packages using git-fetch that we individually care. Note the option >> --manifest for =E2=80=9Cguix lint=E2=80=9D is missing=E2=80=A6 shoul= d happen soon if no one >> beats me. :-) > > I tried this with my user profile and the script hit the Software > heritage limit after reporting the status of 33 packages (most of > which are not archived). So, for the rest of the packages, you are > asked to try again later (I have 95 packages in my profile). There is 2 rate limit: one for saving and one for requesting. Each time you do =E2=80=9Cguix lint -c archival =E2=80=9D, Guix reques= ts to SWH via their API [1] if the package is already in. AFAIR, it is 120 requests per hour. Then if it is not, Guix saves to SWH via their API. And this rate is very low, maybe 10 per hour. Well, if I remember correctly. 1: > Is "guix lint -c archival $pkg" supposed to poke Software Heritage to > archive the $pkg if it is not archived yet? I ask because I ran the > script later and I got the same output from the first run, i.e., > packages reported to be missing from Software Heritage were still > reported as such, instead of being planned for archive. Currently, the request/save via =E2=80=9Cguix lint -c archival=E2=80=9D is = not optimal. For instance, the source of the package =E2=80=9Clibvirt=E2=80=9C is url-fe= tch so requesting for it is not necessary because it cannot be saved via their API. And I not remember exactly how the =E2=80=98tarball=E2=80=99 request = is counted. BTW, the packages using =E2=80=99url-fetch=E2=80=99 should be ingested by S= WH via their nixguix loader reading the sources.json [2]. And for example: --8<---------------cut here---------------start------------->8--- $ guix lint -c archival libvirt gnu/packages/virtualization.scm:1070:5: libvirt@5.8.0: source not archived = on Software Heritage $ guix download -H sha256 -f base64 https://libvirt.org/sources/libvirt-5.8= .0.tar.xz Starting download of /tmp/guix-file.XWwdFj >From https://libvirt.org/sources/libvirt-5.8.0.tar.xz... libvirt-5.8.0.tar.xz 12.5MiB 678KiB/s 00:19 [##################] 1= 00.0% /gnu/store/1pgi1bl8p7jv2mhk83kv7raak2b4k1w5-libvirt-5.8.0.tar.xz 4jMoKJsYve3B6Wb2wmQCspgxScZg7YvVLNpv6rDCDFU=3D --8<---------------cut here---------------end--------------->8--- and in the same time, the sources.json contains:=20 --8<---------------cut here---------------start------------->8--- { "type": "url", "urls": [ "https://libvirt.org/sources/libvirt-5.8.0.tar.xz" ], "integrity": "sha256-4jMoKJsYve3B6Wb2wmQCspgxScZg7YvVLNpv6rDCDFU=3D" }, --8<---------------cut here---------------end--------------->8--- Well, 2 possible explanations: 1) The tarball is not in SWH; because their loader fails on it or for whatever else reasons =20 2) Or the tarball is archived by SWH but they use another hashing (SWH-ID) than the NAR. Well, with the information in the package, Guix is not able to ask to SWH with the correct hash. That=E2=80=99s the main motivation behind disarchive [3]. Last, Guix is not able to deal with hg-fetch or svn-fetch. In this message [4] and the 2 follow-up in the thread, there is some explanation to implement =E2=80=98lookup-subversion-revision=E2=80=99 in (guix swh). Ma= ybe for the next release on Nov. ;-) Well, the dance SWH needs some love. :-) Thanks for trying! It really helps to have this kind of feedback. 2: 3: 4: > Also, I got a backtrace when checking icecat: > > =E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85= =E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85 > Backtrace:cecat@78.8.0-guix0-preview1 [archival]... [...] > ice-9/boot-9.scm:1667:16: In procedure raise-exception: > In procedure bv-length: Wrong type argument in position 1 (expecting byte= vector): #f > =E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85= =E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85=E2=98=85 Indeed, there is a bug. Because the source of =E2=80=99icecat=E2=80=99 rai= ses a case that is not handled by =E2=80=99check-archival=E2=80=99 in (guix lint). Basically in the snippet: --8<---------------cut here---------------start------------->8--- (match (lookup-content (content-hash-value hash) (symbol->string (content-hash-algorithm hash))) --8<---------------cut here---------------end--------------->8--- =E2=80=99lookup-content=E2=80=99 expect a bytevector for =E2=80=99content-h= ash=E2=80=99 and in the case of =E2=80=99icecat=E2=80=99, it returns #f. Then raises the backtrace. Co= uld you open a bug report for that? Just to not forget to fix it. :-) For the record, compare =E2=80=99icecat=E2=80=99 with =E2=80=99hello=E2=80= =99: --8<---------------cut here---------------start------------->8--- $ guix repl GNU Guile 3.0.5 Copyright (C) 1995-2021 Free Software Foundation, Inc. Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. This program is free software, and you are welcome to redistribute it under certain conditions; type `,show c' for details. Enter `,help' for help. scheme@(guix-user)> ,use(guix packages) scheme@(guix-user)> ,use(guix swh) scheme@(guix-user)> ,use(gnu packages gnuzilla) scheme@(guix-user)> (content-hash-value (origin-hash (package-source icecat= ))) $1 =3D #f scheme@(guix-user)> (lookup-content (content-hash-value (origin-hash (packa= ge-source icecat))) "sha256") ice-9/boot-9.scm:1669:16: In procedure raise-exception: In procedure bv-length: Wrong type argument in position 1 (expecting byteve= ctor): #f Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. scheme@(guix-user) [1]> ,q scheme@(guix-user)> ,use(gnu packages base) scheme@(guix-user)> (content-hash-value (origin-hash (package-source hello)= )) $3 =3D #vu8(49 224 102 19 122 150 38 118 232 159 105 209 182 83 130 222 149= 167 239 125 145 75 140 185 86 244 30 167 46 15 81 107) scheme@(guix-user)> (lookup-content (content-hash-value (origin-hash (packa= ge-source hello))) "sha256") $2 =3D #< checksums: (("sha1" . #vu8(247 190 191 111 156 98 162 41= 94 136 159 102 224 92 233 191 174 217 172 227)) ("blake2s256" . #vu8(4 255= 253 50 132 65 210 22 201 36 146 173 114 211 115 136 216 199 120 137 136 11= 6 145 81 41 135 134 253 72 216 137)) ("sha1_git" . #vu8(202 230 179 60 195= 63 170 253 45 107 216 108 107 66 115 249 51 140 105 194)) ("sha256" . #vu8= (49 224 102 19 122 150 38 118 232 159 105 209 182 83 130 222 149 167 239 12= 5 145 75 140 185 86 244 30 167 46 15 81 107))) data-url: "https://archive.s= oftwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95= a7ef7d914b8cb956f41ea72e0f516b/raw/" file-type-url: "https://archive.softwa= reheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7= d914b8cb956f41ea72e0f516b/filetype/" language-url: "https://archive.softwar= eheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d= 914b8cb956f41ea72e0f516b/language/" length: 725946 license-url: "https://ar= chive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65= 382de95a7ef7d914b8cb956f41ea72e0f516b/license/"> --8<---------------cut here---------------end--------------->8--- Cheers, simon