From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id qDTaEZIucWP90QAAbAwnHQ (envelope-from ) for ; Sun, 13 Nov 2022 18:51:14 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id 0EDPEJIucWNCWQAAG6o9tA (envelope-from ) for ; Sun, 13 Nov 2022 18:51:14 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id BFEA323393 for ; Sun, 13 Nov 2022 18:51:13 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ouH8J-0004pL-QM; Sun, 13 Nov 2022 12:51:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ouH8I-0004ot-DG for guix-patches@gnu.org; Sun, 13 Nov 2022 12:51:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ouH8H-0000UI-SM for guix-patches@gnu.org; Sun, 13 Nov 2022 12:51:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ouH8H-0004iY-OE for guix-patches@gnu.org; Sun, 13 Nov 2022 12:51:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#59078] [PATCH] lint: Split the derivation lint checker by system. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 13 Nov 2022 17:51:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 59078 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 59078@debbugs.gnu.org Received: via spool by 59078-submit@debbugs.gnu.org id=B59078.166836182718086 (code B ref 59078); Sun, 13 Nov 2022 17:51:01 +0000 Received: (at 59078) by debbugs.gnu.org; 13 Nov 2022 17:50:27 +0000 Received: from localhost ([127.0.0.1]:52462 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ouH7j-0004he-9G for submit@debbugs.gnu.org; Sun, 13 Nov 2022 12:50:27 -0500 Received: from mira.cbaines.net ([212.71.252.8]:41834) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ouH7g-0004hU-3d for 59078@debbugs.gnu.org; Sun, 13 Nov 2022 12:50:26 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:3a91:a0a4:ecee:f157]) by mira.cbaines.net (Postfix) with ESMTPSA id A205B27BBE9; Sun, 13 Nov 2022 17:50:22 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 94e9cc66; Sun, 13 Nov 2022 17:50:21 +0000 (UTC) References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> User-agent: mu4e 1.8.9; emacs 28.1 From: Christopher Baines Date: Sun, 13 Nov 2022 17:27:48 +0000 In-reply-to: <87k041dui7.fsf@gnu.org> Message-ID: <87bkpau4kl.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1668361874; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=9jW296RKVD0wPoGuT4YdEoxVxNafy18i7s0cxusXjzk=; b=qDjJpoQGQsgnddT50AvEMQ1J8jmtiRRzcETcwFUb4K+dux5W1CtTWHhfZ8WPiteNM7+Gv+ yIP37YP/45kouBOB6d9BaQ9511AS/ik0M4VxXkHACrGDxTAkJmqAM5jTLDLwz0TUxfBQJ+ +61Hgqt6Rs3J0x0ILAwwXaHbofRvrQv5Eb8+5ekQCee09v3DGvTEhL/2KGA8FFAfjD+hwf sv+ge/TVjXGZ3kk65gkJFr0oClHz30GwJxcYwnDcAWxcC/rk+A3mvrhYfWDHSGi4mwIzoN vr98fSfopjjvB2bghsY+4eXxEGDsQUvyhsQyQ0UjKVofY09y0Cy756qNm4gMGg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1668361874; a=rsa-sha256; cv=none; b=rwHHT74ZTsbeP7YGmD9f+FmWmnf1G9j5pW8gj+TpjvPc3VlGaFL94PYZ/Ct2R1R+DsDN4X zKf5wUf0n7OBT0SLQl1sEzuQ/3UhGeTraXzG6jrfcIAXztamDtxUuLurd977/GLQZbAXJE CuJV5h2VWQf45oxM8TR8oIjluX7qROC8pIxvq5KUZSkGSzNXd621Bu28vy/GFrDhwR2RXt FspZhIlCKIR1cQFVqYaTNvjSLzp2rqbJYOLU5pnSNw/XIAFQ5TBf75go3lUmI9N2/hTSbl 0wBiINMd2WrkOcvxXmMaN+rl5BJvvAWwMDaBg089g+pXYGWeVEcFRChmjb5A6g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.25 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: BFEA323393 X-Spam-Score: -5.25 X-Migadu-Scanner: scn0.migadu.com X-TUID: qdm3TvWB8fCf --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> Currently, if you attempt to run the derivation checker on all packages,= the >> Guile process will run out of memory. I think a contributing factor to t= his is >> that the checker forces an inefficient order when you want to generate >> derivations for all the supported systems of each package, constantly >> switching system then package. >> >> This problem also impacts the Guix Data Service, since it tries to run t= he >> derivation checker for all packages. >> >> The changes in this commit to split the derivation lint checker in to se= veral, >> one for each system, means that you can now treat each system separately, >> which should be better for caching purposes. >> >> If it's desirable to keep some notion of checking all supported systems = for a >> single package, I think lint checker groups could be added, so that you = could >> ask for the "derivation" checker, and this would run all the derivation >> checkers. > > The =E2=80=98derivation=E2=80=99 checker was added for this purpose: maki= ng sure that a > package=E2=80=99s derivation can be computed for all the supported system= s. > Previously it was easy to overlook that kind of breakage. > > I think it=E2=80=99s important to keep a =E2=80=98derivation=E2=80=99 che= cker that does this. What aspect of it do you think is important? I realise just splitting the checker in to several doesn't quite pick up on all the same problems, but an additional checker could be added that flags when packages support systems that don't appear as platforms. I think that would catch everything that the derivation checker currently does. If it's a "we need a singular checker" kind of problem, I was thinking of having some kind of lint checker groups. So you could have a group called "derivation", which runs all of the relevant checkers. > Now, the memory consumption you report is unacceptable and this needs to > be addressed. > > Most (all?) caches are now per-session (associated with > ). Since =E2=80=98guix lint=E2=80=99 uses a single ses= sion, those > caches keep growing because there=E2=80=99s no eviction mechanism in plac= e. > > A hack like the one below should work around that. Could you check how > well it works for you? I tried in the Guix Data Service processing packages in chunks of 1000 plus closing the store connection after each batch, and that led to a heap size of 3090MiB. But this is still higher than 1778MiB heap usage I got just by splitting the derivation linter. Fundamentally, I think it's still going to be more memory/time efficient to move towards processing the derivations by system, rather than by package. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmNxLlpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XezfA/9FHf6b7aHZTFe1eu4v2YibxOX8g0s87O1 2GVyMEG5qc3ome2130/430dATjtdQu20SIyswavSHLTQTPchFuwTi3AE0z5un15l doeMEHtM+PHDgTl0alxE9UAfWIRuliUnGzbtjJco921zbVr+pPQ9qT+//XFlAfZx X64XVcvlvGELVww0i5mrLVbukeI8rErcsHW0EcLkBw5El4JxN+54CDr3t6KnF+2O yh63HVgfWj9GOZUDQH1iuEktAoRccZl5CWKHO8qHVaeW4nt/0wrvAj/XO3XI4xXp elqg7WRZdj5fPFHqvwBtrFnMa/QI2+vBsJ7qvN1XDLWhxfeSgiDF7Hoh20gVowHt 3KcEWp0tN092ciyTFr1UrQeLeXxz+rxmo55gytQzINqJxzVAOWDQKgTq2v1bx4+w /DGS32Br7aZQiTBEhVaoQQVGX7rZHhFEKLL1algNHCKKdAhROkKIANBV43Go54SP M2/lRY2j0CvSRcbwxAo0nWNcwM1/zMK/knjenQ1GUpd1W9ysKlfcXw7emjwAR1TU SmxlMUKqlnoMTYj0lt/HG+ueAOpvpmGUaGSM5311us0CdJ7x5c8VqQm8lNfAfBGB RspmDeyYm0cO3lgngq4SD5aCMPLtDRK6S5lVQ9SFumXPBfkNmYe21jOO9fpOo814 xXLfKg+C/LI= =/WZa -----END PGP SIGNATURE----- --=-=-=--