From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:403:478a::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id SMsHJpjd7WQyPwEAG6o9tA:P1 (envelope-from ) for ; Tue, 29 Aug 2023 13:59:20 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:478a::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id SMsHJpjd7WQyPwEAG6o9tA (envelope-from ) for ; Tue, 29 Aug 2023 13:59:20 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3B34F60D6D for ; Tue, 29 Aug 2023 13:59:20 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=jpoiret.xyz header.s=dkim header.b=IzuSOjvS; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=reject) header.from=jpoiret.xyz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1693310360; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=aGZgNTFQrcN9EndvtQWBfqPxRxyjTF78xPgga26kgTE=; b=GVQs9QyeNnfOG9hXQUYAvhHmFaVrhS5jtAOyU1J2n6xjmIjw/yK/zs9TmiAENsAg8aU8oz TYG4W5Pc3toToK3kyBXZtk8Y1b5pWjcDGb9nlak8pLX2R7qIgAav5rVhPZbT6LS1UopZ71 CvMMMnc+MkaJLR1PBN5/FapGZTT3LJ970Wm+Y2QgyM/T570x3aIzJgi2Tow30s6Q89aFrP 8jXVFU3tT3vURnpUaam3fD6g4HCl/fVgAZenO3FxERIi0maHDr/sQOAQlva1JjixTL8UtH 0R9P6MRbR+jxYmB/GeLuDnEa4mrHpTqWo7RvVRl8/lQgc+fNwT1KPrFgdiHqAQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=jpoiret.xyz header.s=dkim header.b=IzuSOjvS; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=reject) header.from=jpoiret.xyz ARC-Seal: i=1; s=key1; d=yhetil.org; t=1693310360; a=rsa-sha256; cv=none; b=jH3jZOynvEBikjGJbuTVnH8YWZhScxkMm6R4pWilgmbDpVUfofSq4g2HkTifvUz2lwk5oI m+80ZMBOgWmtoX77ZUjr3R7u4F0HH9WxtTdOdiqWr0uqvS7ICJH08MnR/zHWjI6vDjor4e 9DFICsvMHSDd9bhit01VJC455peDxRuMzWf2DYiA00cG2ogG0TItEgLbI4nPj9Psbguhra xDpqFucuv/J6WsHMYlAJUpGX0YHWWTJXVjybE7CZpql5doKYSp/0lrlnI7bNeCmK1i/dRD 43ish/ngTSV0LUZxISFzptaPEDep99bg0iAuEh0VSgV3vEFdwgPbF98olmu+Ow== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qaxMn-00077i-Tt; Tue, 29 Aug 2023 07:58:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qaxMl-000774-SE for guix-devel@gnu.org; Tue, 29 Aug 2023 07:58:39 -0400 Received: from jpoiret.xyz ([206.189.101.64]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qaxMh-0000JI-Er for guix-devel@gnu.org; Tue, 29 Aug 2023 07:58:38 -0400 Received: from authenticated-user (jpoiret.xyz [206.189.101.64]) by jpoiret.xyz (Postfix) with ESMTPA id E39D918543A for ; Tue, 29 Aug 2023 11:58:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jpoiret.xyz; s=dkim; t=1693310311; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=aGZgNTFQrcN9EndvtQWBfqPxRxyjTF78xPgga26kgTE=; b=IzuSOjvSpXfykpOkfEzdsT+/Wc8dLH3Cjc1VQ5kcWBXm+1f/2GbSxfHJOvSJXLuEar1Fc7 KA6BfIoGnZw8ZrOsU0Ikf5xu7jppv/ss/anqKNtDf1CY5PLU0t1nDk25meNgbJEpxc6A2e IFLLufYccsO+oL6QZRWcYgpnJ3g0gdmXLWyLR5YVQAubRzhKVkgJ781r2yBQfu9aoiTdhR WeCIXwe+zQg7ZvxrpQynX3KOMhcI+F+7kkiYt9EY7xWKQQGsR4m5xi6wxCIcYtnFhGHQJq x7me8Gn3eTcdkhPDVotlybFyW//iPBG6dwP6HS7uGCaQgIGkyzIXzK3HWS/Wag== From: Josselin Poiret To: guix-devel@gnu.org Subject: Guix pull speed Date: Tue, 29 Aug 2023 13:58:13 +0200 Message-ID: <87wmxe3vsa.fsf@jpoiret.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spamd-Bar: / Received-SPF: pass client-ip=206.189.101.64; envelope-from=dev@jpoiret.xyz; helo=jpoiret.xyz X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx0.migadu.com X-Spam-Score: -11.36 X-Migadu-Queue-Id: 3B34F60D6D X-Migadu-Spam-Score: -11.36 X-TUID: USDaogpzzcrp --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hi everyone, After looking a bit more into guix pull speed, or to be more precise the "Computing Guix derivation..." step, which is not substitutable. I've come to the conclusion that the thing that takes the majority of the time is loading the files that define the packages that the new Guix needs to build itself. These files are not compiled yet, and worse, loading just (gnu packages guile) ends up loading 361 other package files. You can generate a package graph in GraphML with `guix graph -t module -b graphml guile`, and use e.g. networkx to analyze it. You can compare with a compiled check-out of guix by just running the following in a `guix repl`: =2D-8<---------------cut here---------------start------------->8--- (use-modules (guix self) (guix monad-repl)) ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1) =2D-8<---------------cut here---------------end--------------->8--- which takes at most 5 seconds on my laptop. One idea I had was to move all the packages that are looked up in (guix self) to their own little bubble, to avoid having to load extra stuff. However, this is not currently possible because some of them do have non-trivial dependency graphs. I've identified these problematic inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it pulls in all other dependencies itself) po4a graphviz What could be done about this? Another solution would be to somehow build Guix without any of the dependencies and then add them in later, similar to what is done with build-aux/build-self.scm to be able to load (guix self) in the first place. That seems quite complex though. Best, =2D-=20 Josselin Poiret --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQHEBAEBCgAuFiEEOSSM2EHGPMM23K8vUF5AuRYXGooFAmTt3VYQHGRldkBqcG9p cmV0Lnh5egAKCRBQXkC5Fhcais36DACGRD1hWyPMpXEoQZ4VFddJIQiIgrOO7k9Y 0lWLizB0LOCa/2DQD7SFU1wabZWF+MhvdeBIcXcjojkB2qmknNm5unitDz/8moq2 bCTjECAOxbsDPNdHIZCi/n+ssS+csHrGW+ESjX/RGfMFYD/SLZDvJ05PY1wbqp1a Fx52sojH+4c6Ggo1KBKTPsU155E0kOANXkwVaXPINQCWDLPgYlASuMar7XTv0hIN ASWRwAKVHeGhA4fOOVyn7BS0/ITKaU5M+P7ZLVy4Df8AcOqnFDrRAnhyb4yfxgqR J05E8NTzI/e7QsdJMdrTZWNkLTGFAXdIXvvW4fQ1FsYxcTrS0u5JsBy+Jcdnvqkn guz+wkRoe0EA4PaPltfD0Pk4zv92O83Rs/QYZbLIAZMx9cWwG4XP8kioVUOJFOrs vNLQIwVq0OGp3XH/6uWdDK0lPsK7K8rknnrIBbIZ2Yrjrs5d9gQHxc028H3a88/2 tsAygUjRlfm/+YD1NCUcCdPl6igquvI= =yv0t -----END PGP SIGNATURE----- --=-=-=--