From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id CC15GjLou2ApXAAAgWs5BA (envelope-from ) for ; Sat, 05 Jun 2021 23:10:10 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 0Jz1FTLou2CGGAAA1q6Kng (envelope-from ) for ; Sat, 05 Jun 2021 21:10:10 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D9D69314BC for ; Sat, 5 Jun 2021 23:10:09 +0200 (CEST) Received: from localhost ([::1]:60814 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lpdYW-0006ob-SZ for larch@yhetil.org; Sat, 05 Jun 2021 17:10:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59500) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lpdYQ-0006oS-PF for guix-patches@gnu.org; Sat, 05 Jun 2021 17:10:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:38633) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lpdYQ-0006Bz-IZ for guix-patches@gnu.org; Sat, 05 Jun 2021 17:10:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lpdYP-0004hq-Vm for guix-patches@gnu.org; Sat, 05 Jun 2021 17:10:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#48853] [PATCH] profiles: Move some of the work to the build side. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Jun 2021 21:10:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 48853 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 48853@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.162292735218021 (code B ref -1); Sat, 05 Jun 2021 21:10:01 +0000 Received: (at submit) by debbugs.gnu.org; 5 Jun 2021 21:09:12 +0000 Received: from localhost ([127.0.0.1]:50179 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lpdXc-0004gb-3u for submit@debbugs.gnu.org; Sat, 05 Jun 2021 17:09:12 -0400 Received: from lists.gnu.org ([209.51.188.17]:48270) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lpdXX-0004gQ-WE for submit@debbugs.gnu.org; Sat, 05 Jun 2021 17:09:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59278) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lpdXX-0006jF-QC for guix-patches@gnu.org; Sat, 05 Jun 2021 17:09:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:59130) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lpdXX-0005au-8a; Sat, 05 Jun 2021 17:09:07 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=53896 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lpdXX-0005L2-0I; Sat, 05 Jun 2021 17:09:07 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Sat, 5 Jun 2021 23:08:56 +0200 Message-Id: <20210605210856.10481-1-ludo@gnu.org> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1622927410; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=3QPBYMyP4TV5s1qamHPlzrsJKADSdGeK0987XFEaWHg=; b=iz9lMYmJWZVJs2KJqgjaQVNUk8V+oF7cMLp96a84k3gJoaWE2zYfgP6fS63VhjeNIn8Xlr IlGAORmfeUKrZECjPXl4ou+hi2JUA8vNhLF2FkUefiK25IzxljqIeXt96bla8D3l4yWZIZ TYExScrdRrWG0FU5Kyv9u2J4vE8hEdErGgyM8pgMe8eK7zPv1Rnnh/L591D79whK/GDHyJ K4BUxzZmjUKLYxy3K7hmNBKC1waRhYoQH3EJlQuI5MzL3+vBIRsTHU/ebkUPP4TKYICBVt G+nDvktORArcNAytAIh5UkwIQMQTcDAY9uFPjZU6t0kFcJDvcXXPOUWAMtPsdQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1622927410; a=rsa-sha256; cv=none; b=XelQgqfBeIcD1LGZK+Ck9W3nN7QpFRFbUHMlePe6/IdgGCeS0ahh/dmlYCG9W5juFnzElj YrGwTlcEnEIuxb0xb3XrlT619Hqy3epuEeDPwQ62d13BYINm/0VnDqRrBiyRWcbIc128PJ geSYuFgku/PrdAZYMgo/bos3AsKYmDZM0oDcpC27MSiXVHW2a9xC2aqU8lypv358c+0ooA iEUZLj0wlngQ19iXB/xG+8caDKz8isqShhO5InfH9A9jB7LRtY7hdsb92w452oJzvY1cFp 2/QI46fFM+ZfwNEPvyzPex0CQC6S57vgwA9AF1cqS8RQO7/5wTLK3OAkJEObFQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: 3.57 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: D9D69314BC X-Spam-Score: 3.57 X-Migadu-Scanner: scn0.migadu.com X-TUID: zTwfAdtUkzfE When running: guix environment --ad-hoc gnome --no-grafts --search-paths this reduces wall-clock time by ~5%. The number of object cache lookups goes down from 96K to 89K. (Note that 'gnome' is an interesting example because it has many propagated inputs, which themselves have propagated inputs too, which would lead to a long input list and a long manifest in the 'profile-derivation' gexp.) * guix/profiles.scm (profile-derivation)[inputs, search-paths]: Remove. [extra-inputs]: New variable. [builder]: Adjust call to 'build-profile'. * guix/build/profiles.scm (manifest-sexp->inputs+search-paths): New procedure. (build-profile): Remove 'inputs' parameter; make 'manifest' the 2nd positional parameter and add #:extra-inputs. Call 'manifest-sexp->inputs+search-paths' to obtain 'inputs' and 'search-paths'. --- guix/build/profiles.scm | 86 +++++++++++++++++++++++++++-------------- guix/profiles.scm | 25 ++++-------- 2 files changed, 64 insertions(+), 47 deletions(-) Hi! This is a rather modest optimization of the code that computes the derivation of a profile, probably only measurable for profiles with lots of packages or lots of propagated inputs (as can happen with R/Python packages). Let me know what difference it makes for your use cases! Thanks, Ludo'. diff --git a/guix/build/profiles.scm b/guix/build/profiles.scm index a40c3f96de..9249977bed 100644 --- a/guix/build/profiles.scm +++ b/guix/build/profiles.scm @@ -20,6 +20,8 @@ #:use-module (guix build union) #:use-module (guix build utils) #:use-module (guix search-paths) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (ice-9 ftw) #:use-module (ice-9 match) @@ -143,45 +145,71 @@ instead make DIRECTORY a \"real\" directory containing symlinks." directory)))) (apply throw args)))))) -(define* (build-profile output inputs - #:key manifest search-paths - (symlink symlink)) - "Build a user profile from INPUTS in directory OUTPUT, using SYMLINK to -create symlinks. Write MANIFEST, an sexp, to OUTPUT/manifest. Create -OUTPUT/etc/profile with Bash definitions for -all the variables listed in -SEARCH-PATHS." +(define (manifest-sexp->inputs+search-paths manifest) + "Parse MANIFEST, an sexp as produced by 'manifest->gexp', and return two +values: the list of store items of its manifest entries, and the list of +search path specifications." + (match manifest ;this must match 'manifest->gexp' + (('manifest ('version 3) + ('packages (entries ...))) + (let loop ((entries entries) + (inputs '()) + (search-paths '())) + (match entries + (((name version output item + ('propagated-inputs deps) + ('search-paths paths) _ ...) . rest) + (loop (append deps rest) + (cons item inputs) + (append paths search-paths))) + (() + (values inputs + (delete-duplicates + (cons $PATH + (map sexp->search-path-specification + search-paths)))))))))) + +(define* (build-profile output manifest + #:key (extra-inputs '()) (symlink symlink)) + "Build a user profile from MANIFEST, an sexp, and EXTRA-INPUTS, a list of +store items, in directory OUTPUT, using SYMLINK to create symlinks. Create +OUTPUT/etc/profile with Bash definitions for all the variables listed in the +search paths of MANIFEST's entries." (define manifest-file (string-append output "/manifest")) - ;; Make the symlinks. - (union-build output inputs - #:symlink symlink - #:log-port (%make-void-port "w")) + (let-values (((inputs search-paths) + (manifest-sexp->inputs+search-paths manifest))) - ;; If one of the INPUTS provides a '/manifest' file, delete it. That can - ;; happen if MANIFEST contains something such as a Guix instance, which is - ;; ultimately built as a profile. - (when (file-exists? manifest-file) - (delete-file manifest-file)) + ;; Make the symlinks. + (union-build output (append extra-inputs inputs) + #:symlink symlink + #:log-port (%make-void-port "w")) - ;; Store meta-data. - (call-with-output-file manifest-file - (lambda (p) - (display "\ + ;; If one of the INPUTS provides a '/manifest' file, delete it. That can + ;; happen if MANIFEST contains something such as a Guix instance, which is + ;; ultimately built as a profile. + (when (file-exists? manifest-file) + (delete-file manifest-file)) + + ;; Store meta-data. + (call-with-output-file manifest-file + (lambda (p) + (display "\ ;; This file was automatically generated and is for internal use only. ;; It cannot be passed to the '--manifest' option. ;; Run 'guix package --export-manifest' if you want to export a file ;; suitable for '--manifest'.\n\n" - p) - (pretty-print manifest p))) + p) + (pretty-print manifest p))) - ;; Make sure we can write to 'OUTPUT/etc'. 'union-build' above could have - ;; made 'etc' a symlink to a read-only sub-directory in the store so we need - ;; to work around that. - (ensure-writable-directory (string-append output "/etc") - #:symlink symlink) + ;; Make sure we can write to 'OUTPUT/etc'. 'union-build' above could have + ;; made 'etc' a symlink to a read-only sub-directory in the store so we + ;; need to work around that. + (ensure-writable-directory (string-append output "/etc") + #:symlink symlink) - ;; Write 'OUTPUT/etc/profile'. - (build-etc/profile output search-paths)) + ;; Write 'OUTPUT/etc/profile'. + (build-etc/profile output search-paths))) ;;; profile.scm ends here diff --git a/guix/profiles.scm b/guix/profiles.scm index ed5c10315a..8cbffa4d2b 100644 --- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -1812,12 +1812,10 @@ are cross-built for TARGET." (mapm/accumulate-builds (lambda (hook) (hook manifest)) hooks)))) - (define inputs - (append (filter-map (lambda (drv) - (and (derivation? drv) - (gexp-input drv))) - extras) - (manifest-inputs manifest))) + (define extra-inputs + (filter-map (lambda (drv) + (and (derivation? drv) (gexp-input drv))) + extras)) (define glibc-utf8-locales ;lazy reference (module-ref (resolve-interface '(gnu packages base)) @@ -1851,20 +1849,11 @@ are cross-built for TARGET." #+(if locales? set-utf8-locale #t) - (define search-paths - ;; Search paths of MANIFEST's packages, converted back to their - ;; record form. - (map sexp->search-path-specification - (delete-duplicates - '#$(map search-path-specification->sexp - (manifest-search-paths manifest))))) - - (build-profile #$output '#$inputs + (build-profile #$output '#$(manifest->gexp manifest) + #:extra-inputs '#$extra-inputs #:symlink #$(if relative-symlinks? #~symlink-relative - #~symlink) - #:manifest '#$(manifest->gexp manifest) - #:search-paths search-paths)))) + #~symlink))))) (gexp->derivation name builder #:system system -- 2.31.1