From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id cJT4KuYwd2SjNQEASxT56A (envelope-from ) for ; Wed, 31 May 2023 13:35:02 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id SDjDKuYwd2Tp5QAA9RJhRA (envelope-from ) for ; Wed, 31 May 2023 13:35:02 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8AE8CDDCA for ; Wed, 31 May 2023 13:35:02 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4K6A-0004IC-Da; Wed, 31 May 2023 07:34:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4K68-0004Ht-Ux for guix-devel@gnu.org; Wed, 31 May 2023 07:34:36 -0400 Received: from mx1.riseup.net ([198.252.153.129]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4K66-00008z-Q7; Wed, 31 May 2023 07:34:36 -0400 Received: from fews01-sea.riseup.net (fews01-sea-pn.riseup.net [10.0.1.109]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx1.riseup.net (Postfix) with ESMTPS id 4QWRxh5Yc2zDqDx; Wed, 31 May 2023 11:34:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1685532872; bh=QlbpvTqmmGbQlC2Juycr1ovWYPgGvZvLaWx+8rPefoo=; h=References:From:To:Cc:Subject:Date:In-reply-to:From; b=KvaB5GRxebRxqR/Zuj+3M/XnuiyjngPO/WlSnBwGfypzxzAgGc+mpI2sFvDPcNVTi l7Z7ElhYzFf2KuvJTHvRmmH4tJUMeTcV57fthtlXg8Fr3YlWvKkxJ7ouri7tN5jD+M zQA5FYOJ1fGn0o3ZHqGKmfEGSu/OGzZ7nCyD9eb8= X-Riseup-User-ID: 1778AA80CDE1A89323A3D1BD49EB54D97ADCB0DAE8477B193BA4CCF900665083 Received: from [127.0.0.1] (localhost [127.0.0.1]) by fews01-sea.riseup.net (Postfix) with ESMTPSA id 4QWRxh018FzJntP; Wed, 31 May 2023 11:34:31 +0000 (UTC) References: <875y9jzl9m.fsf@gnu.org> <874jot19fd.fsf_-_@gnu.org> <87fs7rvv5s.fsf_-_@gnu.org> <878rddooy4.fsf@gnu.org> <87r0r4uv4x.fsf@gmail.com> <87ttvzhxm9.fsf@gnu.org> <87cz2it3yk.fsf@gmail.com> <87r0qxvd9q.fsf@riseup.net> <86y1l5oric.fsf@gmail.com> From: Csepp To: Simon Tournier Cc: Csepp , Ludovic =?utf-8?Q?Court=C3=A8s?= , Andreas Enge , guix-devel@gnu.org Subject: Re: Faster =?utf-8?Q?=E2=80=9Cguix_search=E2=80=9D?= (was Re: How many bytes do we add (closure of guix) when adding one new package?) Date: Wed, 31 May 2023 13:10:09 +0200 In-reply-to: <86y1l5oric.fsf@gmail.com> Message-ID: <87ttvska4n.fsf@riseup.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=198.252.153.129; envelope-from=raingloom@riseup.net; helo=mx1.riseup.net X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1685532902; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=hOf4+boxSE+mL5BiDdX2+OtVebbBdFfpHbYNGuX88Xw=; b=Jtx0IPMVvPTolqtTbztD/d6q/D4jBzCvNwcFfzHSHGpXS4Qn7wdBCz/2fO54bsv2ZZAbIg G4+zyfCYg55YaNLyn7dJn2ELw1grW2r1TyZrcrrb9+Uupk3HnoRGa0YVIYKNzpRXRIiMPX IHz5BLrIbQdh5LNbs9Mf9lJP82tnIRnSAeCEMSWNEhZwsx7y0zv1EceGUhJCtOkgu+s7g6 7WAWFcaD479xewk9+Y2dyLwChnq93PdSH4ZSiSaECx2CL9ujO0lmwa7ie+Ju1iwd3f4i0K 9ECS9FIWf9OJR9qjxRBnUcIIiGm7B8Aks0eJYLDM/hI4xMzsHyzwnQFlu/ehFg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=riseup.net header.s=squak header.b=KvaB5GRx; dmarc=pass (policy=none) header.from=riseup.net; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1685532902; a=rsa-sha256; cv=none; b=LE/FfnM56ay5mitEa6C/AA3W1OT7hUA/KJMMzIcwbbQiU0seBxXQVLgf5Q3IFBdsgqtM83 fqlAb2iWVjOAhkvBlV5tbPazHmDEdqvmdH9otJKXTDrgfU87olrTFcUuJIvAAxzqobO5jx ZerDZQfK8qitYu56GYEj0UtAvIjHCy6Hkfxa2eu0MnaAsnJ+zYIMCORL1fi1spPQ8+lqtk bLX+DyZ9ZoN7EfxuHAx8o2vJ3kFp5rcFC5U7sbUVP5BNfiVIxu09qR4w5+F7dJ8hwCuKSH jTs3jCgCwvyyir9ae6xOZJysJhQDPF4pyWfOPQ4Qa1zZ24APfJeFb6y1x72N1g== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -4.97 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=riseup.net header.s=squak header.b=KvaB5GRx; dmarc=pass (policy=none) header.from=riseup.net; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 8AE8CDDCA X-Spam-Score: -4.97 X-TUID: kgk2dnlPPs/N Simon Tournier writes: > Hi, > > On Tue, 30 May 2023 at 21:10, Csepp wrote: > >> It makes zero sense to load full package definitions from >> disk for most queries, such as guix search, with an SoA representation >> we could load only the fields that we care about. > > That=E2=80=99s already the case; see > ~/.config/guix/current/lib/guix/package.cache. > > For instance, =E2=80=9Cguix package -A=E2=80=9D exploits it and the perfo= rmances are > acceptable. Two past summers, wow already! I tried to augment it and > exploit it for =E2=80=9Cguix search=E2=80=9D. The implementation and ben= chmark is in > #39258 [1]. Well, the whole thread of #39258 appears to me worth to > consider because it spots various bottleneck specific to =E2=80=9Cguix se= arch=E2=80=9D > and explains why the improvement is not straightforward. That's a good improvement, but it's in addition to the ELF files, so it doesn't save any space, and as far as I know it doesn't speed up non-textual queries, like searching for packages that use a specific build system. > Well, I have started months ago to write a Guix extension using > guile-xapian. My aim is to tackle two annoyances: 1. the speed and > 2. the relevance. > > About the relevance #2, the issue is that the current scoring considers > only the local information of one package without considering the global > information of all the others. Well, see [2,3,4] for some details. :-) > > 1: https://issues.guix.gnu.org/39258#119 > 2: https://yhetil.org/guix/CAJ3okZ3E3bhZ5pROZS68wEKdKOcZ8SpXsvdi-bnB=3D9J= z3mPahA@mail.gmail.com > 3: https://yhetil.org/guix/CAJ3okZ3+hn0nJP98OhnZYLWJvhLGpdTUK+jB0hoM5JArQ= xO=3Dzw@mail.gmail.com > 4: https://yhetil.org/guix/CAJ3okZ0LaJzWDBA7bjqZew_jAmtt1rj9PJhevwrtBiA_C= OXENg@mail.gmail.com Thanks for the links, gonna read them in more detail later. >> ps.: Now I'm even more glad that I'm using a file system with >> transparent compression on all my Guix systems. > > Did you benchmarked the performances for some Guix operations on these > compressed vs uncompressed file system? I haven't, but I have recently tried to move to a larger drive and accidentally did a btrfs send without compression and the system didn't fit.