From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id kKfcGBXE2F5jWAAA0tVLHw (envelope-from ) for ; Thu, 04 Jun 2020 09:51:17 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id yDyqFBXE2F6/XgAAbx9fmQ (envelope-from ) for ; Thu, 04 Jun 2020 09:51:17 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E2712940B0C for ; Thu, 4 Jun 2020 09:51:16 +0000 (UTC) Received: from localhost ([::1]:33260 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jgmWp-0002ZR-Qd for larch@yhetil.org; Thu, 04 Jun 2020 05:51:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58374) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jgmWh-0002YZ-MT for guix-devel@gnu.org; Thu, 04 Jun 2020 05:51:07 -0400 Received: from fanzine.igalia.com ([178.60.130.6]:60760) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jgmWd-0005MW-Nd; Thu, 04 Jun 2020 05:51:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-ID:In-Reply-To:Date:References:Subject:Cc:To:From; bh=azdNbyUSK6HwkIHbepHv/H3VY4zw1vCnqqNGzSQm4M0=; b=GQLqUAWid0L3V58iz4FfZLHf35QvwpweOWN0LqqdwySjdOY1ofQAWGsH7AkVYxZ6l3GaPlHhsoysMy1ki84DjHo/IrNPyRQJnyUeW4TMysdLt/50bwH25mIPcPKLuBgB1MF2tSjljhnU+S8lh5m1oYR7kkM70xQKHdHHR4cuW3FCP4fNQRm10qw5ovlPN5bmkzQT3CVOcnMAMG/gj2b0VoZFsFiyiu7W4m8Z2K9UaeNOtloqCx6e0FUBdslUCJSb460X4DO4sXBLVfKe0gaI8pHGlQ0E6T4ceuHWWHEdBfv94+MWp4xTCgj/uYcJG4wxhQNyLjqWlNTehrfl9canuQ==; Received: from 82-65-63-215.subs.proxad.net ([82.65.63.215] helo=milano) by fanzine.igalia.com with esmtpsa (Cipher TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim) id 1jgmWG-0002v2-Qe; Thu, 04 Jun 2020 11:50:41 +0200 From: Andy Wingo To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Compilation time with Guile 3.0.3-to-be References: <87sgfbgthj.fsf@inria.fr> Date: Thu, 04 Jun 2020 11:50:29 +0200 In-Reply-To: <87sgfbgthj.fsf@inria.fr> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22\?\= \=\?utf-8\?Q\?'s\?\= message of "Thu, 04 Jun 2020 09:50:16 +0200") Message-ID: <87ftbbrwgq.fsf@igalia.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=178.60.130.6; envelope-from=wingo@igalia.com; helo=fanzine.igalia.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/04 05:50:41 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x (no timestamps) [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=igalia.com header.s=20170329 header.b=GQLqUAWi; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: 0.29 X-TUID: 56XIp2gmkvgG Hi :) On Thu 04 Jun 2020 09:50, Ludovic Court=C3=A8s writes: > With the attached patch I=E2=80=99ve run =E2=80=98make as-derivation=E2= =80=99 (equivalent to > =E2=80=98guix pull=E2=80=99) and timed the builds of guix-packages-base.d= rv (279 files) > and guix-packages.drv (217 files) on my 4-core i7 laptop: > > =E2=80=A2 guix-packages-base.drv: 1m30s (was 4m) > > =E2=80=A2 guix-packages.drv: 30s (was 1m8s) Nice!! Thanks for testing :-) > For the record, the optimizations currently used in (guix build compile) > are between the new -O0 and -O1: > > (cond ((or (string-contains file "gnu/packages/") > (string-contains file "gnu/tests/")) > ;; Level 0 is good enough but partial evaluation helps preserve = the > ;; "macro writer's bill of rights". > (override-option #:partial-eval? #t > (optimizations-for-level 0))) Here fwiw I would use -O1. It is basically the same as -O0 except that it adds partial evaluation and it inlines primcalls. If you are willing to do partial evaluation, you are probably willing to inline primcalls too; I think it typically leads to less code and the compilation time is similar to -O0. > ((string-contains file "gnu/services/") > ;; '-O2 -Ono-letrectify' compiles about ~20% faster than '-O2' f= or > ;; large files like gnu/services/mail.scm. > (override-option #:letrectify? #f > (optimizations-for-level 2))) Interesting. I think this is probably a bug of some sort that we'll have to keep working on. > With the new -O1, the scheme->tree-il conversion (mostly macro > expansion) accounts for half of the build time on large files: > > scheme@(guile-user)> ,use(system base optimize) > scheme@(guile-user)> ,time (compile-file "gnu/packages/python-xyz.scm" #:= opts (optimizations-for-level 1)) > $1 =3D "/data/src/guile-3.0/cache/guile/ccache/3.0-LE-8-4.3/home/ludo/src= /guix/gnu/packages/python-xyz.scm.go" > ;; 4.154311s real time, 5.604945s run time. 2.538106s spent in GC. Interesting data :) Note that in 3.0.3 there is also a new phase called "lowering". Before a compiler from Tree-IL to language X is called, the tree-IL program is "lowered" -- meaning canonicalized and optionally optimized. (define (lower-exp exp env optimization-level opts) (let ((make-lowerer (language-lowerer (lookup-language 'tree-il)))) ((make-lowerer optimization-level opts) exp env))) Similarly there is an analysis pass for warnings, which runs before lowering: (define (analyze-exp exp env warning-level warnings) (let ((make-analyzer (language-analyzer (lookup-language 'tree-il)))) ((make-analyzer warning-level warnings) exp env))) These can be interesting to test different phases of the tree-il -> bytecode path. > The profile looks like this: > > scheme@(guile-user)> ,pr (define t (call-with-input-file "gnu/packages/p= ython-xyz.scm" (lambda (port) (read-and-compile port #:to 'tree-il)))) > % cumulative self=20=20=20=20=20=20=20=20=20=20=20=20=20 > time seconds seconds procedure > 13.16 0.45 0.40 anon #x1136458 > 10.53 0.35 0.32 ice-9/popen.scm:168:0:reap-pipes > 7.89 0.24 0.24 anon #x1132af8 > 6.14 0.35 0.19 ice-9/boot-9.scm:3128:0:module-gensym > 5.26 0.21 0.16 ice-9/boot-9.scm:2201:0:%load-announce > 4.39 0.19 0.13 ice-9/psyntax.scm:749:8:search > 3.51 0.69 0.11 ice-9/psyntax.scm:2964:6:match* > 3.51 0.11 0.11 anon #x11334e8 > 3.51 0.11 0.11 anon #x1136428 > 2.63 0.08 0.08 anon #x113a258 > 1.75 0.05 0.05 ice-9/psyntax.scm:3017:12:$sc-dispatch > 1.75 0.05 0.05 anon #x1139e68 > 0.88 277.30 0.03 ice-9/boot-9.scm:220:5:map1 > > It=E2=80=99d be great to waive the anonymity of that first lambda. :-) I think I just fixed it :) > I was wondering what fraction of that time was spent running Guix macros > (=E2=80=98package=E2=80=99, =E2=80=98base32=E2=80=99, and so on), but it= =E2=80=99s difficult to answer that > question here. Probably something to investigate so we can make further > progress! I think the reap-pipes call is a pretty bad sign, incidentally! Cheers, Andy