From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id OLwbIlfhL2DCXgAA0tVLHw (envelope-from ) for ; Fri, 19 Feb 2021 16:03:35 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id wNjzHVfhL2CLXQAA1q6Kng (envelope-from ) for ; Fri, 19 Feb 2021 16:03:35 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 052F9CB7F for ; Fri, 19 Feb 2021 17:03:35 +0100 (CET) Received: from localhost ([::1]:58944 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lD8Fh-0002FZ-QS for larch@yhetil.org; Fri, 19 Feb 2021 11:03:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:52562) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lD8Dw-0001cD-9U for guix-devel@gnu.org; Fri, 19 Feb 2021 11:01:51 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51288) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lD8Dv-0001T1-Pr for guix-devel@gnu.org; Fri, 19 Feb 2021 11:01:43 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=50086 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lD8Du-0005ld-7W for guix-devel@gnu.org; Fri, 19 Feb 2021 11:01:43 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Guix-devel Subject: Will 2021 be the year of build systems on gexps? X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 1 =?utf-8?Q?Vent=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 19 Feb 2021 17:01:38 +0100 Message-ID: <87im6ojbbx.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.87 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 052F9CB7F X-Spam-Score: -1.87 X-Migadu-Scanner: scn1.migadu.com X-TUID: KbcEvlYNctDs Hello Guix! Over the last few days I=E2=80=99ve been head-down working on =E2=80=98wip-build-systems-gexp=E2=80=99, the mythical branch that brings g= exps to build systems and packages, so we can say goodbye to =E2=80=98build-expression->derivation=E2=80=99. And=E2=80=A6 it=E2=80=99s = quite a ride! I rebased it on =E2=80=98core-updates=E2=80=99, which was a bit tedious (it= hadn=E2=80=99t been touched in 3 years or so). There are still test failures and build systems not yet converted, but that=E2=80=99s the easy part, although both = are time-consuming. The more difficult part is performance. On current =E2=80=98core-updates= =E2=80=99 I get: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=3Dgc time ./pre-inst-env guix build qemu -d --no-grafts /gnu/store/z27l6plrxr5wm7818xhj9mdll99jcqz3-qemu-5.1.0.drv Garbage collection statistics: heap size: 80.52 MiB allocated: 191.14 MiB GC times: 15 time spent in GC: 0.66 seconds (32% of user time) 2.05user 0.12system 0:01.93elapsed 112%CPU (0avgtext+0avgdata 242872maxresi= dent)k 0inputs+0outputs (0major+34228minor)pagefaults 0swaps --8<---------------cut here---------------end--------------->8--- but on =E2=80=98wip-build-systems-gexp=E2=80=99 I get: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=3Dgc time ./pre-inst-env guix build qemu -d --no-grafts /gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv Garbage collection statistics: heap size: 80.52 MiB allocated: 249.89 MiB GC times: 16 time spent in GC: 0.75 seconds (34% of user time) 2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresi= dent)k 0inputs+0outputs (0major+34304minor)pagefaults 0swaps --8<---------------cut here---------------end--------------->8--- In short, 30% more garbage allocated and an 8% slowdown. I=E2=80=99ve made progress identifying and mitigating what seems to be the = main cause of this (the fact that it=E2=80=99s possible to ungexp a list and that list will be scanned in its entirety in search of file-like objects=E2=80=94convenient but expensive) but as you can see, there=E2=80= =99s still a lot to do. Statprof and gcprof are not super helpful here; I feel a need for better tools here, or better metrics. Anyway, that=E2=80=99s the situation. If you have ideas or if you=E2=80=99= re in a profiling & optimization mood, now=E2=80=99s the time to unleash your creativity. :-) Ludo=E2=80=99.