From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id WFoIMW3ZAmVwVgEAauVa8A:P1 (envelope-from ) for ; Thu, 14 Sep 2023 11:59:09 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id WFoIMW3ZAmVwVgEAauVa8A (envelope-from ) for ; Thu, 14 Sep 2023 11:59:09 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5CEAD64ECF for ; Thu, 14 Sep 2023 11:59:09 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=gWzyqiNA; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1694685549; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=59sKrKl/sIWFwGKS5iuJcV4VviZCFfN19NYjsV7ymlc=; b=eBQWTFIOxuL0vRM2jSttovLgEwMIekST8EdJE8D7RVyfZGGj+8rhveZTSPQ+cWsre8OiTg utjBrOzOfQ9SBg6zzhK4RxVYxxg/9dnjWUu09Yfd86vPBalU6WPhP5/cAyoa8XAE9S9tyS 611OXx5+Ut9mudSPyDnTAErC7P7ciuF9yzykSIinPHKUpb5ABng2eZ27jZhqy5ikm+/D2M /2zmSfoiq9gZ8S4e+ny6tWafori2ZXWf61NU+iquVZ2sbA8MHTh27XFlyG3/Mm1osjxQAZ LgQUCmzTwH0ElJ4iyEWy+cwpf24HTuO/pXblXX5O+3HKM/eMXfXfvguYtpvNJA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=gWzyqiNA; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1694685549; a=rsa-sha256; cv=none; b=nQQW5ipvbCgn2zXXfOvLncV7zuv/ZGKbsvIr06Tgvtg+O6TTLUqFoISCnHO3EUKHvHuStw qcOc3PSzn5ls2sg6hbfbwC+jeI5VRPnm2ORBsvOAcX6zXYVvo6SWIfJykMEeW6uFX4OtN2 sxNqH45bxA2DA4ZxSWcTarFeFcUIrRdI5VRXBM249QIqtnTYg/TUNEOQPBILhL/jzKPKoy kwAP8IQPfuvHMKD7zozOvrdeu/H1ndCJhyaz4nZHTKAbou/HUEC0CJMjaJvnABBPewfHzS b5m0n6+Z8+kHzWiJR32tWJS1utoruCY3sZow50B+VJQIZOzyaQ2rJAMl0LDQKg== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgj7S-00059E-Ub; Thu, 14 Sep 2023 05:58:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgj7B-00058d-6C for guix-devel@gnu.org; Thu, 14 Sep 2023 05:58:25 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgj7A-0004Lm-L8; Thu, 14 Sep 2023 05:58:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=59sKrKl/sIWFwGKS5iuJcV4VviZCFfN19NYjsV7ymlc=; b=gWzyqiNAHvQ9fWBREiIN 3e0ONXzsWqOdTiUBzygeGcDCbZkcqIhKsqr9D0MGBOwGqVVNsn3X9v/y4us2Q2NwO1aLCv/yt5G// K+zaF+W6OAnbz9jK87W2TKLUVNiC79zAuyHTd8EQw9g89soNXKYjvJEhJGbWB8YCYNuWXLX+P2SdQ LfTZPi7QECkbpoAi4K8ZpvHuPMyxq+SXQWD/9Vc3cUlMy5MKrSx/pZesSoP1TsTGoebv7MArko0EP 2NupZA3b4YpmMUBhBAi3Sm+Z+mlqCVyoHJNRbT22+vu5Z9OHfk3ULSQu/z7MBWUcSzCEWbGHC4l/J e5qRFF0u/iZCww==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Josselin Poiret Cc: guix-devel@gnu.org Subject: Re: Guix pull speed References: <87wmxe3vsa.fsf@jpoiret.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Octidi 28 Fructidor an 231 de la =?utf-8?Q?R=C3=A9vo?= =?utf-8?Q?lution=2C?= jour du =?utf-8?Q?Ma=C3=AFs?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 14 Sep 2023 11:58:18 +0200 In-Reply-To: <87wmxe3vsa.fsf@jpoiret.xyz> (Josselin Poiret's message of "Tue, 29 Aug 2023 13:58:13 +0200") Message-ID: <87il8ddqkl.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx1.migadu.com X-Migadu-Spam-Score: -3.81 X-Spam-Score: -3.81 X-Migadu-Queue-Id: 5CEAD64ECF X-TUID: agtdx/CRiiAH Hello! Josselin Poiret skribis: > After looking a bit more into guix pull speed, or to be more precise the > "Computing Guix derivation..." step, which is not substitutable. I've > come to the conclusion that the thing that takes the majority of the > time is loading the files that define the packages that the new Guix > needs to build itself. These files are not compiled yet, and worse, > loading just (gnu packages guile) ends up loading 361 other package > files. You can generate a package graph in GraphML with `guix graph -t > module -b graphml guile`, and use e.g. networkx to analyze it. Nice (we should add this trick under =E2=80=9CInvoking guix graph=E2=80=9D = for instance; we already mention xdot there.) > You can compare with a compiled check-out of guix by just running the > following in a `guix repl`: > > (use-modules (guix self) (guix monad-repl)) > ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1) > > which takes at most 5 seconds on my laptop. > > One idea I had was to move all the packages that are looked up in (guix > self) to their own little bubble, to avoid having to load extra stuff. > However, this is not currently possible because some of them do have > non-trivial dependency graphs. I've identified these problematic > inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it > pulls in all other dependencies itself) po4a graphviz Yeah, I don=E2=80=99t think the idea of an independent subset of the module graph is achievable, because sooner or later glibc starts depending on Python, which starts depending on Rust, which pulls in Qt. Compiling with =E2=80=98-Wunused-modules=E2=80=99 might allow us to clean up the modu= le graph a bit though. > What could be done about this? Another solution would be to somehow > build Guix without any of the dependencies and then add them in later, > similar to what is done with build-aux/build-self.scm to be able to load > (guix self) in the first place. That seems quite complex though. Most of the time is spent evaluating (gnu packages =E2=80=A6) modules; evaluation is pretty slow, and from what I remember of previous profiling sessions, this is largely due to psyntax. We should concentrate on that. Another approach: the resulting .drv is a deterministic result. Wouldn=E2=80=99t it be nice if we could make the whole =E2=80=9CComputing G= uix derivation=E2=80=9D process a derivation itself? That way it could be substituted. Nix introduced (or considered introducing) =E2=80=9Crecursive Nix=E2=80=9D,= whereby a derivation build process can talk to the daemon =E2=80=9Cfrom the inside=E2= =80=9D, creating additional derivations on the way. This could be one possibility, but it=E2=80=99s a lot of work with unclear outcomes. In Guix, we can use the (guix =E2=80=A6) modules inside the build process, = and we have almost everything we need to compute a set of derivations =E2=80=9C= in the abstract=E2=80=9D (we=E2=80=99d need a backend that,= instead of making RPCs, would store .drv and =E2=80=9Csources=E2=80=9D in memory or in= a file). So we could create a derivation that returns let=E2=80=99s say a nar containin= g the closure of the .drv (the =E2=80=9CGuix derivation=E2=80=9D), which we would= then import. There=E2=80=99s =E2=80=9Cjust=E2=80=9D one gotcha with this plan: grafts. = To compute grafts, we potentially need to build things, and for that we=E2=80=99d need to actually talk to the daemon. Maybe there are other possibilities, such as adding a =E2=80=9Cbuiltin:build-self=E2=80=9D derivation builder in the daemon. Food for thought! Ludo=E2=80=99.