From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id mBY1Dcvj+mDlFAAAgWs5BA (envelope-from ) for ; Fri, 23 Jul 2021 17:44:11 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 2E3MCMvj+mBGNQAAbx9fmQ (envelope-from ) for ; Fri, 23 Jul 2021 15:44:11 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 76A65DA49 for ; Fri, 23 Jul 2021 17:44:10 +0200 (CEST) Received: from localhost ([::1]:56430 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6xLN-0001vx-HZ for larch@yhetil.org; Fri, 23 Jul 2021 11:44:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40842) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6xLG-0001tq-Tj for guix-patches@gnu.org; Fri, 23 Jul 2021 11:44:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:32808) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m6xLG-0005Jb-Gi for guix-patches@gnu.org; Fri, 23 Jul 2021 11:44:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1m6xLG-0004gm-9m for guix-patches@gnu.org; Fri, 23 Jul 2021 11:44:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 23 Jul 2021 15:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39258 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: zimoun Cc: arunisaac@systemreboot.net, 39258@debbugs.gnu.org Received: via spool by 39258-submit@debbugs.gnu.org id=B39258.162705500417973 (code B ref 39258); Fri, 23 Jul 2021 15:44:02 +0000 Received: (at 39258) by debbugs.gnu.org; 23 Jul 2021 15:43:24 +0000 Received: from localhost ([127.0.0.1]:44354 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m6xKd-0004fo-Ma for submit@debbugs.gnu.org; Fri, 23 Jul 2021 11:43:24 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42858) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m6xKb-0004fb-TH for 39258@debbugs.gnu.org; Fri, 23 Jul 2021 11:43:22 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:59328) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6xKV-0004wY-95; Fri, 23 Jul 2021 11:43:15 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=42914 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6xKV-0003pT-0s; Fri, 23 Jul 2021 11:43:15 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20210715073328.212123-1-zimon.toutoune@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 5 Thermidor an 229 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 23 Jul 2021 17:43:12 +0200 In-Reply-To: <20210715073328.212123-1-zimon.toutoune@gmail.com> (zimoun's message of "Thu, 15 Jul 2021 09:33:26 +0200") Message-ID: <875yx1ave7.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1627055050; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=BFDluSaz6v4Z8QE/p3AH1sDA18lp80ymocfkig3vlDI=; b=DvEIDV+Ca3XoVI8sXA9/KgGQjaybziqa6TKcmQpv/mN2bX/lLMSuDtZUegCcIYGxsg29Va Q5cl3l/hMW7J6snHII5nLspNExBpcPhZco/Ziy8rOYwcRkpES6H+L6b001u5i21J9ANKfo 2EUt9k2ho9M0CFa8N2NRtMDxFuqS3lDEnqHSAaYIzuwz9efX+xCY2W/qGkZNQAdbwm/jYA 51EaYfQbLQ+fv8lH/ASRwLAA7PqXSFX0C8upHeRWUVrL5VaYtr2Y6T6F4dvcly4+/vYEc+ WSKhuwCeNoyvWfUNoPDep9N2rOSmbSxZrBrthca+mKs4Fcwn6LcWZmEdDYI6ag== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1627055050; a=rsa-sha256; cv=none; b=oBIsw/NR47kW08yj+4BmkECepVAcuwkkwwB/U2hGzZvzMW2EyrSDxsC5HWhdYBcTqfuTX4 D+8BJn6N0sumXniCohBHF5P97LQbEbbTji2auSzHYHe5/CCKC1+lmUodTuE2FEmoYqbOFf OG+hdMrfylmh0rBYHst8G+0hDkE8hp7txUvq4MYHi9nVk6/HFo/lu5tEhvirmPfYKWpORO hGQS6jyk7xdrYBqjqtCmbYMw5CelsSSsDtmEZkvRIX/WDAJmTdMv22RD4ViaE7f+LaerLV ZwfDGKLtMDIiAmoxbCZ9k255jTxgqRWWvOfCIRiORdHKaSSAqknTWQW9233zHw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.92 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 76A65DA49 X-Spam-Score: -2.92 X-Migadu-Scanner: scn0.migadu.com X-TUID: JeljvTJlKkHI Hi! zimoun skribis: > This is an attempt to improve the performance of "guix search". It is st= ill > half baked but it allows to discuss further the idea about expanding the > current '/lib/guix/package.cache' and avoids to forget an IRL discussion.= ;-) Thanks for resuming this discussion. :-) > From my understanding, the issue that 'package-relevance' accepts a 'pack= age' > (and then all the chain until displaying) and 'fold-avaibale-packages' do= es > not return a package. Well, I do not know; especially where to put somet= hing > similiar to 'read-package-from'. Yeah that=E2=80=99s annoying. Perhaps we need or . With some trickery we could have record type inheritance or something, maybe. Dunno. It would be good if we could arrange so that =E2=80=98fold-available-packag= es=E2=80=99 doesn=E2=80=99t allocate anything though. > Let compare only for cold cache and time this cache building (Guix 7db8fd= 6): > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix build --check $(guix gc --derivers $(readlink -f ~/.config/gu= ix/current/lib/guix/package.cache)) > > real 0m28,848s > user 0m1,481s > sys 0m0,252s > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib= /guix/package.cache)) > > real 0m40,279s > user 0m1,582s > sys 0m0,232s > > It seems longer but compared to the time of "guix pull" completion, it se= ems > acceptable. Both the initial timing and the target are waaay too much. :-/ On my i7 laptop I have: --8<---------------cut here---------------start------------->8--- $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-pac= kage-cache "/tmp/t.cache")' real 0m20.738s user 0m44.413s sys 0m0.341s --8<---------------cut here---------------end--------------->8--- It=E2=80=99s CPU-bound; we should probably start by optimizing that. In Guile 3.0.7 there was a change that improved this noticeably: https://git.savannah.gnu.org/cgit/guile.git/commit/?id=3D05614f792bfabbc3= 3798863edd0bb67c744e9299 We should prolly look for similar optimization opportunities in the assembler=E2=80=A6 > Let compare for some queries: > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > time guix search game | recsel -C -P name | wc -l > 371 > > real 0m7,561s > user 0m3,525s > sys 0m0,391s I think you should run: time guix search game > /dev/null otherwise Bash=E2=80=99s built-in =E2=80=98time=E2=80=99 shows the wall-clo= ck time of the whole pipeline, and the processing time of =E2=80=98recsel=E2=80=99 is probably n= ot negligible here. [...] > Last, two minor remarks about previous comments. > > 1. Ludo commented: > > > Therefore the cache '/lib/guix/package.cache' contains more > > information. > > This breaks the binary interface, so we=E2=80=99ll have to analyz= e the impact of > such a change and devise a strategy. > > > > and after some checking, this should be fine, IIUC. The '--news' is ok > because of '#:allow-other-keys'. And other parts are modified accordingl= y. > Guix revision N creates a cache that Guix revision N+1 will read but it s= hould > not be an issue; see 'inferior-available-packages'. > > 2. And Ludo wrote: > > I realize the other cache also has that problem, but it would be = nice to add a > version tag to the cache. Basically emit something like: > > (package-metadata-cache (version 0) VECTOR =E2=80=A6) > > instead of just: > > (VECTOR =E2=80=A6) > > > > which is, after discussions, not necessary. Versioning does not make sen= se > here because the cache is read by the Guix which generates it. Therefore, > specify a version is extraneous here. I confirm! :-) Thanks, Ludo=E2=80=99.