From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id QG27Mvjk72AqjwAAgWs5BA (envelope-from ) for ; Thu, 15 Jul 2021 09:34:16 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id wKG3LPjk72BMSwAA1q6Kng (envelope-from ) for ; Thu, 15 Jul 2021 07:34:16 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4CE668353 for ; Thu, 15 Jul 2021 09:34:16 +0200 (CEST) Received: from localhost ([::1]:43260 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m3vst-0007Fb-BR for larch@yhetil.org; Thu, 15 Jul 2021 03:34:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37000) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m3vsg-0007DA-Gy for guix-patches@gnu.org; Thu, 15 Jul 2021 03:34:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:35121) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m3vsg-0000xJ-8k for guix-patches@gnu.org; Thu, 15 Jul 2021 03:34:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1m3vsg-0006wP-6L for guix-patches@gnu.org; Thu, 15 Jul 2021 03:34:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances References: In-Reply-To: Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 15 Jul 2021 07:34:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39258 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: 39258@debbugs.gnu.org Cc: ludo@gnu.org, arunisaac@systemreboot.net, zimoun Received: via spool by 39258-submit@debbugs.gnu.org id=B39258.162633442526643 (code B ref 39258); Thu, 15 Jul 2021 07:34:02 +0000 Received: (at 39258) by debbugs.gnu.org; 15 Jul 2021 07:33:45 +0000 Received: from localhost ([127.0.0.1]:46663 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m3vsP-0006ve-76 for submit@debbugs.gnu.org; Thu, 15 Jul 2021 03:33:45 -0400 Received: from mail-wr1-f45.google.com ([209.85.221.45]:34658) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m3vsM-0006vE-T0 for 39258@debbugs.gnu.org; Thu, 15 Jul 2021 03:33:43 -0400 Received: by mail-wr1-f45.google.com with SMTP id u1so6464491wrs.1 for <39258@debbugs.gnu.org>; Thu, 15 Jul 2021 00:33:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=SxulHMwv2ho0c/VIBIpkJ+f2dRMjwqwllobobw5/ykY=; b=q8NHSwUSYRMexyfN28eeHqFGVPswoIrKK4Lmd2/U69ZMnw9c0C7rAQkUpPwT+8Dz+3 K8ykKL0tkz32E4eI3RadbhR76WbKT1KQ56h4DaMFo6Bwrp1EdYpTgxmQMblLVTabqkg8 sV1ud6rNAjQssxHIkB9AXijzPu0Pp0N0exTIkOlGvjAXvPrFKB/2Ff4aGzWpplNJtUmV pi9I8iW/GyIOKsYsBvuu0j86kIwW213rEWYBVMJgcu32QLI9DWi1nhw1mWIZwYfxjvm9 OcKVbFKG/xBQj7bzjKJ1hSOXpPAAXZtk9rNFCkV6PPGVeBd/qEIMlvE8//suDDl1fQYq S3Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=SxulHMwv2ho0c/VIBIpkJ+f2dRMjwqwllobobw5/ykY=; b=PM2LZNhpgT+008Wmc2hzjHSUf/SRL/I33qcE7oaWThRmgAW/SybzeoCOTIrqso7s1Q GzDOTyay0EKn6k21oH8AEuqbYK1C6b1aFaKP4wGUBD597huwgFvElxRr+Ovw/axGN3Z/ BB0u5/fR0ui2OE+uw3N/sO9hv3hNW/nqsBiGb9udMfNDbk2rlEn68y74BSGTpZZkiVaT T/y8Q0dDBMgvyZ5TTQijms3ups3gufO4VYflNYf1fNW/0Pc+lSY0EI+oSqdes7v9XgZb n36mNypsNtElYMgkxSraz9zUysL2lcJmv/6d29LV3VMLwjjF8duv5JRc3Jw43tX8YkLZ lKcA== X-Gm-Message-State: AOAM533nkQYykn7n7IQTz0aR0Vk+rNsmdoqgzaTfmyUEvrZgjGXZis3c MA3fEmp/Wq9meHIc491Lr8TUrZaBMlg= X-Google-Smtp-Source: ABdhPJw4Q2xPor1BtBZX6x+gzqeqK691ET7saEFmkVH3dSlzxQgUow16apstSIeenNb51QJ+M1D4zw== X-Received: by 2002:a05:6000:1248:: with SMTP id j8mr3655805wrx.391.1626334417042; Thu, 15 Jul 2021 00:33:37 -0700 (PDT) Received: from localhost.localdomain ([193.48.40.117]) by smtp.gmail.com with ESMTPSA id y3sm5376205wrh.16.2021.07.15.00.33.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Jul 2021 00:33:36 -0700 (PDT) From: zimoun Date: Thu, 15 Jul 2021 09:33:26 +0200 Message-Id: <20210715073328.212123-1-zimon.toutoune@gmail.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1626334456; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=SxulHMwv2ho0c/VIBIpkJ+f2dRMjwqwllobobw5/ykY=; b=VeKghnC608a8NNR9d2EymqL6N2lM2MqKYiuhRTlrgb9w1LG7s5f2C6WAbTK4v5F+3mKYsQ ziRa8qk9YzC2SNrLM3lnPHhT2xW9SRR7etJJ2AwLXpEFFA/3/zbyEqG4g4WBsVgvhK0fe0 NwQsUPbiTtu+V8mxJ9ZtYNZBbny0hmI/jFmzvfFC9LzFy7qx8tPmbpTa8day3VhcpqNUKC TNtQaVS16Vepf2bnQkwmt6FcJYnOALoWUvSRne2BAC5lOMwqMATiCiMY1QJIdjXWA2mQnh XmdYxp7Dk8MbLw0NWEARD5k5Y5wxZc8igFnBbexrI85ErN1cOxBAdeGXqy9hSQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1626334456; a=rsa-sha256; cv=none; b=SUHsxsQMoGBiBl5ZV/GC2Ue3UDk+3J7bgv0/zDm3JFgGf3SUYL9js8a97QBvt5T6MBMYuo junl4BMT2coXu+8K8vBHyZmJ00dGk2OGyIuvdU6Bn7gHWiHFDgZCrRLY0knQ/vRO+MNxzw Y1iruv4R5NhL/7tR5vzNMG5eE/0cTD/OlbbZUwEwhTE1Fz85LDtBX3mg/Ji2DRLlAiVFke crxvq04r7PtpkX4S7A1DfoS32Ekp+hnsi9GC4wUP1P+izpMqyGCNatMxZhhxp4T0OoaBnF mM9XGizaHTrKWqP6lS8hD3T0+4m0pNm16QzgkZnanLWHakDyXjDeOIPX9jWdjg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=q8NHSwUS; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: 0.20 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=q8NHSwUS; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 4CE668353 X-Spam-Score: 0.20 X-Migadu-Scanner: scn0.migadu.com X-TUID: 78Jqz9Qfg1iJ Hi, This is an attempt to improve the performance of "guix search". It is still half baked but it allows to discuss further the idea about expanding the current '/lib/guix/package.cache' and avoids to forget an IRL discussion. ;-) Let start by what needs to be improved: the part when cache is not authoritative. It is slower than the current approach because the package is read twice, i.e., the module is indeed loaded twice, once by 'fold-available-packages' via 'fold-module-public-variables*' and then again by 'find-packages-by-description' via 'read-package-from'. The issue is to have a common interface for both cases (cache and no-cache). More thoughts are required. ;-) Then, using the cache is slower than expected. Therefore, something is maybe twisted -- quick implementation before holidays ;-) -- with the use of 'fold-avaibale-packages' as proposed by Ludo [1]. Note that instead another 'fold-packages' (say 'fold-packages*') using the new cache should be used. As it is done with v4 and the performances were as expected: 1: >From my understanding, the issue that 'package-relevance' accepts a 'package' (and then all the chain until displaying) and 'fold-avaibale-packages' does not return a package. Well, I do not know; especially where to put something similiar to 'read-package-from'. To test, after applying the patches, the command is: ./pre-inst-env guix pull --allow-downgrades --disable-authentication \ --url=$(pwd) --branch=search-v6 -p /tmp/new Let compare only for cold cache and time this cache building (Guix 7db8fd6): sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix build --check $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache)) real 0m28,848s user 0m1,481s sys 0m0,252s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib/guix/package.cache)) real 0m40,279s user 0m1,582s sys 0m0,232s It seems longer but compared to the time of "guix pull" completion, it seems acceptable. However, maybe it could become an issue when running a lot of "guix time-machine"... Well, hard trade-off. ;-) Let compare for some queries: --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search game | recsel -C -P name | wc -l 371 real 0m7,561s user 0m3,525s sys 0m0,391s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search game | recsel -C -P name | wc -l 371 real 0m9,814s user 0m3,240s sys 0m0,363s --8<---------------cut here---------------end--------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search strategy game | recsel -C -P name | wc -l 16 real 0m8,565s user 0m2,803s sys 0m0,430s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search strategy game | recsel -C -P name | wc -l 16 real 0m9,679s user 0m2,370s sys 0m0,334s --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search strategy game caesar | recsel -C -P name | wc -l 0 real 0m8,307s user 0m2,388s sys 0m0,366s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search strategy game caesar | recsel -C -P name | wc -l 0 real 0m3,626s user 0m0,948s sys 0m0,101s --8<---------------cut here---------------end--------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search game strategy the | recsel -C -P name | wc -l 4 real 0m8,776s user 0m2,903s sys 0m0,454s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search game strategy the | recsel -C -P name | wc -l 4 real 0m9,495s user 0m2,546s sys 0m0,313s --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search the game strategy | recsel -C -P name | wc -l 4 real 0m8,502s user 0m2,534s sys 0m0,388s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search the game strategy | recsel -C -P name | wc -l 4 real 0m9,508s user 0m2,254s sys 0m0,363s --8<---------------cut here---------------end--------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search crypto library | recsel -C -P name | grep libb2 libb2 real 0m8,744s user 0m2,875s sys 0m0,374s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search crypto library | recsel -C -P name | grep libb2 libb2 real 0m9,229s user 0m2,448s sys 0m0,397s --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search cuirass integration | recsel -C -P name | wc -l 1 real 0m8,132s user 0m2,343s sys 0m0,407s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search cuirass integration | recsel -C -P name | wc -l 1 real 0m8,940s user 0m2,036s sys 0m0,369s --8<---------------cut here---------------end--------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search cuirass | recsel -C -P name | wc -l 2 real 0m8,240s user 0m2,461s sys 0m0,367s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search cuirass | recsel -C -P name | wc -l 2 real 0m8,863s user 0m2,019s sys 0m0,377s --8<---------------cut here---------------start------------->8--- sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time guix search cuirass integration foo | recsel -C -P name | wc -l 0 real 0m8,258s user 0m2,418s sys 0m0,521s sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search cuirass integration foo | recsel -C -P name | wc -l 0 real 0m3,358s user 0m0,867s sys 0m0,139s --8<---------------cut here---------------end--------------->8--- This last example suggests that 'read-package-from' is the slowdown. (On a side note, maybe I am doing wrong, but there is no improvement by the recent introduction of 'cut' for multi-terms as the query "the game strategy" and "game strategy the". Another story. :-)) When cache is not authoritative, it is worse, as expected: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' time /tmp/new/bin/guix search -L /tmp/my-pkgs cuirass integration foo | recsel -C -P name | wc -l 0 real 0m12,503s user 0m7,807s sys 0m0,529s and note that currently the performances of "guix search" is the same for both cases (authoritative and not authoritative); i.e., see previous timing. Last, two minor remarks about previous comments. 1. Ludo commented: > Therefore the cache '/lib/guix/package.cache' contains more > information. This breaks the binary interface, so we’ll have to analyze the impact of such a change and devise a strategy. and after some checking, this should be fine, IIUC. The '--news' is ok because of '#:allow-other-keys'. And other parts are modified accordingly. Guix revision N creates a cache that Guix revision N+1 will read but it should not be an issue; see 'inferior-available-packages'. 2. And Ludo wrote: I realize the other cache also has that problem, but it would be nice to add a version tag to the cache. Basically emit something like: (package-metadata-cache (version 0) VECTOR …) instead of just: (VECTOR …) which is, after discussions, not necessary. Versioning does not make sense here because the cache is read by the Guix which generates it. Therefore, specify a version is extraneous here. Comments are welcome for this work-in-progress. ;-) Cheers, simon zimoun (2): DRAFT packages: Add fields to packages cache. DRAFT scripts: package: Use cache in 'find-packages-by-description'. gnu/packages.scm | 52 +++++++++++++++++++++++++++------------- guix/scripts/package.scm | 46 +++++++++++++++++++++++++---------- 2 files changed, 70 insertions(+), 28 deletions(-) base-commit: 4196087f3d6fc254db5b4c47658e5679c835516f -- 2.32.0