From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id yLMQGavNHGJTCQEAgWs5BA (envelope-from ) for ; Mon, 28 Feb 2022 14:27:07 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id KPDCFavNHGJqSwAAauVa8A (envelope-from ) for ; Mon, 28 Feb 2022 14:27:07 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2AA1E386EB for ; Mon, 28 Feb 2022 14:27:07 +0100 (CET) Received: from localhost ([::1]:46966 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nOg3O-0004L3-5G for larch@yhetil.org; Mon, 28 Feb 2022 08:27:06 -0500 Received: from eggs.gnu.org ([209.51.188.92]:58752) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nOfub-000496-Lw for guix-devel@gnu.org; Mon, 28 Feb 2022 08:18:02 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:34166) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nOfuY-0004ZX-TQ for guix-devel@gnu.org; Mon, 28 Feb 2022 08:18:00 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id A62575E8; Mon, 28 Feb 2022 14:17:56 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5M0fOzX0wdsq; Mon, 28 Feb 2022 14:17:54 +0100 (CET) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 5BF2E482; Mon, 28 Feb 2022 14:17:54 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxime Devos Subject: Re: Faster "guix pull" by incremental compilation and non-circular modules? References: <2067ba1e606855eace261fd0b0ae9721b369bbd5.camel@telenet.be> <878ru5epyv.fsf@elephly.net> <753ba5897ed397b5e95175cd139137975245945b.camel@telenet.be> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 10 =?utf-8?Q?Vent=C3=B4se?= an 230 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 28 Feb 2022 14:17:53 +0100 In-Reply-To: <753ba5897ed397b5e95175cd139137975245945b.camel@telenet.be> (Maxime Devos's message of "Sun, 20 Feb 2022 13:19:07 +0100") Message-ID: <875yozdta6.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / X-Rspamd-Server: hera X-Rspamd-Queue-Id: A62575E8 X-Spamd-Result: default: False [0.90 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] Received-SPF: softfail client-ip=185.233.100.1; envelope-from=ludo@gnu.org; helo=hera.aquilenet.fr X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.665, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Wurmus , guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1646054827; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=DFVOlhahKKzD/xPbcpkSIeplAWdt2YVrKcepoE0GFKQ=; b=qqvSeJzW052Qt0Ft+b9e7gAiaebCaNbInz3XhevfdMKW3wfnTq3+wlw/GHBPVkBIDSe3Ue WKH70AuR2BGYgY3JOZiWOnBs2a13I5yK6GpO3v1ClGQ/9kdEJBuSPQcCX5/kbsjtrLAqL1 XwRfcBrpBEboqUaU1npNQj0rW0tRVaMj5F+pkHVw2lqxK3U07SI7pjcfhbOnQSCV8qqxhm Rwb1Fdkq3B7EyObXXU1ZGHlyvZT58fR9S8pVMijtMpTsmXpx58XPjt4tHsEAnTyjrwVkRf jWNFH/UffUMIm0FWSkhX1mc0pePFDC9GY+scG/vq+buScDGE9fogqhX8H40zIw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1646054827; a=rsa-sha256; cv=none; b=VpD684d4YvmnE+S+zNSaxp6yeMqq57wKjhLNVl2xgF1qptBbOJDqVlKwhmRu8e/oKIb5aW cFoITbhXELTpExZr3ix9U5LF7fAIOjNpmFs4zNBSnl/gmWShObL8m9xPKrTpTcEFEvWryX cvxWTRVz98MLR1JhqiwctUa+KTeFQj+GWbREdQTPVnLLbwVe3rzXWxC4A5tIEnciS8+P0c +LxXPbOabVnn/IhHpGsE7Ib4VWwhsq4l5RSK+ax+FL4hB3GyUnvBkKMBAY5bkYgk1CKsDt uvvpQykl+b5R63J1gupQJwyz2oZWmjsLuDNNJ1ttjfRh0XT2iUX7ioXsuswKYw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.60 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 2AA1E386EB X-Spam-Score: -3.60 X-Migadu-Scanner: scn0.migadu.com X-TUID: X1GJIMQqK4TS Hi, Maxime Devos skribis: > 2. Instead of building all of Guix as a single derivation, > create a DAG of derivations. More concretely: > > First read the *.scm files to determine which module imports > which modules. Then to compile, say, (gnu packages acl), > a derivation taking gnu/packages/acl.scm and its dependencies > gnu/packages/attr.go, gnu/packages/base.go, ... is made > compiling gnu/packages/acl.scm to a gnu/packages/acl.go. > > Then to build all of Guix, 'union-build' or 'file-union' is used. This is what (guix self), used by =E2=80=98guix pull=E2=80=99, is already d= oing. However, currently, package modules are split in just two groups: the =E2=80=9Cbase=E2=80=9D group is the closure of (guix packages base), and th= e second group has all the rest: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> ,use(guix modules) scheme@(guile-user)> (length (source-module-closure '((gnu packages base)))) $4 =3D 417 scheme@(guile-user)> ,use(ice-9 ftw) scheme@(guile-user)> (length (scandir "gnu/packages" (lambda (f) (string-su= ffix? ".scm" f)))) $5 =3D 537 scheme@(guile-user)> (- $5 $4) $6 =3D 120 --8<---------------cut here---------------end--------------->8--- This is clearly uneven, especially considering that the biggest file, crates-io.scm, is in the first group. At its core though, the situation pretty much reflects the free software situation: there are low-level packages (glibc, GCC, GTK, etc.) that might depend on high-level packages (Python, Pandoc, Rust, etc.). It=E2=80=99s not easy to split this spaghetti ball in smaller groups. Thoughts? Ludo=E2=80=99.