From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:1008:1e59::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id 4IX4MisNTmYBcwAAA41jLg (envelope-from ) for ; Wed, 22 May 2024 17:20:11 +0200 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id kNoFMCsNTmZsGAEA62LTzQ (envelope-from ) for ; Wed, 22 May 2024 17:20:11 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YOqNZ8ld; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1716391211; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=g4nmP4m7QbVBHgO/+V6t0bBshc84HUcOZws/JXLxWIs=; b=jefh8DNhiDC/AozO4dliEM4bvami2au6J/AvUuQFciRF00/l2U09xM1ug92Ti+zKYZs+qb ulQ3YysYKNhGbwoQOFQ2GNLvXG28PKJTSIvXqiFS9EPpYRaQUYkoMoo1cmwoy8gbFL53tW ISbI+7wuLfJPIg5c2VoJYljhNUIt42mI2D2+TCCSmc3RHGynYbuJkGeqyUdY/pFI2dnhfm 2YmiSYhF/Tvmglpx7qUOIv6nie6QsQWtTBgw1lyWFFNVexyunwAOTWW87XpnGZBg/oU2c3 DssgSz1isSjN8j7e7fNIhU16JDepFCpFcV3zNTYS7Jy44YzBRawiGueQXI54Rw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YOqNZ8ld; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=key1; d=yhetil.org; t=1716391211; a=rsa-sha256; cv=none; b=kcqG9rHxUyf1mO9LzOsSs1cId5tjxkk/5zboP1ZdASYYthw/fD8by9qQ6PrmvhnRFL/ywN w26pNFWHtMIL9mnCQXI8vMsQpPof+lFNJpegIZCE05K/HQy90X3QN9hpQB0PIBC2NpS6Bx lGIvGIQivkIG2yu+yMcd8okIIbjChpETLQixu8ENGvZkObioCbswbUQeU6/2tgDw1+eSps U+YSsHPZvpZaCThoQWvE22WANCuJb143iVN3gbyYeSU7IVDmw7BluUS7mr3dVEYALD1FUi czWHbMIZf/z6rdZlHr7MfM6Onzu6n6ufVz7yVAXpVTSikiFFOF0vxNvJ76+gjw== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 394415181A for ; Wed, 22 May 2024 17:20:11 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s9nkm-0006lT-Qe; Wed, 22 May 2024 11:19:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9nki-0006kt-Vt for guix-devel@gnu.org; Wed, 22 May 2024 11:19:41 -0400 Received: from mail-wr1-x42d.google.com ([2a00:1450:4864:20::42d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s9nkh-0003og-3V for guix-devel@gnu.org; Wed, 22 May 2024 11:19:40 -0400 Received: by mail-wr1-x42d.google.com with SMTP id ffacd0b85a97d-354ba9ae540so149400f8f.3 for ; Wed, 22 May 2024 08:19:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716391177; x=1716995977; darn=gnu.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=g4nmP4m7QbVBHgO/+V6t0bBshc84HUcOZws/JXLxWIs=; b=YOqNZ8ldnDlKAtwPS+FLKSnvOMqR0CyVfgBqFF2d3+PezLGNvZb/ZglViQQKxl8vwi wZYaOyNsVBAp0axekpC2gaTBj7TG3M0Wzw6tMdJz+YKGkFZDp9U9yDhxdD60r92PiK8s ODYMszw4uShLRa01p5HYho1nFsuaoBVOSE65ZLeCEVmLES2uWBaI5eA44bZpDYnEkSwt 7SK5JKxa+lbJODBx73aQFg+ZgPLrIrdmm+MNmy2xV07nWKKCAFa9RuCYVIc7exRWYhiO eL4czroD4Zkg+b4EX2RmUVDyxpHYgCSFuIMiwzmflOf+f4RQdmKfrT5KrRWmrU0QcjRz S0vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716391177; x=1716995977; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=g4nmP4m7QbVBHgO/+V6t0bBshc84HUcOZws/JXLxWIs=; b=hSrcnti11JTke4yuzyuzw5+Mf7rYPXtU2ZIeleyW0zsb7eqHJVd3t+4U/+vMLX0V9B 5OvA2MgfCF+/gpNOcJ0kvRTASoSJdtoYaWWw7hejqw4ilJ4WvhHIp4aRGlj0zjJJ5Dw6 9wz30r8/Kuq2txLJ5dNElQKS6fdQ7eOixH09zZkEdqKuh8nW8VDLisBvRHdJS6Emjk/K MkSlZnG688cJz5qcY2JIIN55HGPhDkOyiz13nvAosjGGzu9vHYhko+YxGuA+TOppB15F dOUHmYzjqJEXD7OIuTihdievV18OCcSUSUQvjKSBQ9THCz4BRfp+tmBaazra32iS9UEr mq6w== X-Gm-Message-State: AOJu0YzUyJbOI5EhficcurFwVblbC5BqeCS1EeA1giFCwQACcoholzKR XHqP8U8x8Qji/OoGuI9RRfphk1f/JeCRt+fkG+dfN8XXw2coMMa/FadmuQ== X-Google-Smtp-Source: AGHT+IGnNNW5RB+veCskMN43RgyCKjH/Zlv8xlslAPUtGfv9Np+ZPmYCboBDgqcrSXJ+PkdFQAT7DQ== X-Received: by 2002:adf:e3c4:0:b0:354:e778:3665 with SMTP id ffacd0b85a97d-354e77837c1mr1179178f8f.5.1716391176396; Wed, 22 May 2024 08:19:36 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:c865:f7ad:a866:6eca]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-352b66ba0c5sm15087805f8f.27.2024.05.22.08.19.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 May 2024 08:19:35 -0700 (PDT) From: Simon Tournier To: Guix Devel Cc: Arun Isaac Subject: Extension for improving Guix search? Date: Wed, 22 May 2024 12:15:09 +0200 Message-ID: <87msoi40gi.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::42d; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x42d.google.com X-Spam_score_int: -4 X-Spam_score: -0.5 X-Spam_bar: / X-Spam_report: (-0.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_03_06=1.592, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -8.68 X-Spam-Score: -8.68 X-Migadu-Queue-Id: 394415181A X-Migadu-Scanner: mx13.migadu.com X-TUID: APsbETFfjLTv Hi, Guile-Xapian@0.4.0 is out! That=E2=80=99s a fun opportunity for revisiting =E2=80=9Cguix search=E2=80=9D. Well, =E2=80=9Cguix search=E2=80=9D is curr= ently an alias of =E2=80=9Cguix package --search=E2=80=9D and it can be considered as =E2=80=9Cslow=E2=80= =9D. The main reason is from the lack of a good cache. Although it is not a fair comparison, =E2=80=9Cguix package --list-availabl= e=E2=80=9D is much faster than =E2=80=9Cguix package --search=E2=80=9D. Obviously, they = do not process and format the same content. Well, the point is that =E2=80=99list-available=E2=80=99 relies on a cache, namely ~/.config/guix/current/lib/guix/package.cache. It had been discussed to expand this cache (or create another one) for helping =E2=80=9Cguix search=E2=80=9D. Instead of walking through Guile mo= dules, just read a cache. However, this implies a slower =E2=80=9Cguix pull=E2=80=9D = =E2=80=93 and =E2=80=9Cguix time-machine=E2=80=9D too =E2=80=93; the penalty depends on your machine an= d from my opinion, I can wait some seconds more after waiting many minutes, but that had not been the consensus. All that said, there is a very long thread where different approaches had been tried; see all the subthreads. ;-) 1. Rely on SQL [1] 2. Rely on Xapian [2] 3. Extend package.cache [3] 4. Improving string search algorithm [4] 5. Rely on package.cache and load the required Guile [5] On the top of that, it is not handy to find which Guix revision contains which package version. For a description of the problem [6]. For instance, I do stuff as: $ git -C ~/src/guix/guix log --format=3D"%h %s" | grep 'gnu: bowtie:' a47a90b900 gnu: bowtie: Remove reference to %outputs. f336cc4fe7 gnu: bowtie: Replace invalid characters. e5a26a1f02 gnu: bowtie: Remove trailing #T. 2ec601580b gnu: bowtie: Use TBB 2020. 21c837405a gnu: bowtie: Update to 2.3.4.3. 06e372360e gnu: bowtie: Use 'modify-phases'. d6e63cf31c gnu: bowtie: Update to 2.3.2. 2642231b39 gnu: bowtie: Update to 2.2.9. 0047d26a22 gnu: bowtie: Update to 2.2.6. 241e122193 gnu: bowtie: fix build errors which is not super handy. Well, it was somehow an idea behind this Outreachy internship [7]. It=E2=80=99s another direction: exploit the form= at of Guix commit messages. Anyway. Therefore, all in all, I think it=E2=80=99s time to have a Guix extension t= hat improves the situation on both sides: a) faster search b) search inside all the history And I think that Xapian could offer a very great framework. Well, I have started an embryo months (years?) ago and it is still an embryo. This issue deserve more love. :-) Adding Xapian as Guix dependencies is too much. But it is possible to extend Guix with GUIX_EXTENSIONS_PATH. This appears to me the best solution here: write an extension relying on Xapian for indexing. Still reading? I think this fit a very good topic if you are looking for something to work on. Maybe it could be turned on some GSoC =E2=80=93 = hum I am a bit lost with all the deadlines. Feel free to ping. :-) Cheers, simon 1: [bug#39258] Faster guix search using an sqlite cache Arun Isaac Fri, 24 Jan 2020 01:21:57 +0530 id:cu7pnfaar36.fsf@systemreboot.net https://issues.guix.gnu.org/39258 https://issues.guix.gnu.org/msgid/cu7pnfaar36.fsf@systemreboot.net https://yhetil.org/guix/cu7pnfaar36.fsf@systemreboot.net 2: [bug#39258] [PATCH v2 0/3] Xapian for Guix package search Arun Isaac Sat, 07 Mar 2020 19:01:13 +0530 id:20200307133116.11443-1-arunisaac@systemreboot.net https://issues.guix.gnu.org/39258 https://issues.guix.gnu.org/msgid/20200307133116.11443-1-arunisaac@systemre= boot.net https://yhetil.org/guix/20200307133116.11443-1-arunisaac@systemreboot.net 3: [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) zimoun Sun, 03 May 2020 17:01:51 +0200 id:20200503150154.26532-1-zimon.toutoune@gmail.com https://issues.guix.gnu.org/39258 https://issues.guix.gnu.org/msgid/20200503150154.26532-1-zimon.toutoune@gma= il.com https://yhetil.org/guix/20200503150154.26532-1-zimon.toutoune@gmail.com 4: [bug#39258] KMP string search algorithm? zimoun Mon, 01 Jun 2020 12:11:52 +0200 id:CAJ3okZ2ZNYaH=3D_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@mail.gmail.com https://issues.guix.gnu.org/39258 https://issues.guix.gnu.org/msgid/CAJ3okZ2ZNYaH=3D_OjP2Axvq-anY6Q9CkVmjUVLP= x-Z6+kZTVsdg@mail.gmail.com https://yhetil.org/guix/CAJ3okZ2ZNYaH=3D_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVs= dg@mail.gmail.com 5: [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances zimoun Thu, 15 Jul 2021 09:33:26 +0200 id:20210715073328.212123-1-zimon.toutoune@gmail.com https://issues.guix.gnu.org/39258 https://issues.guix.gnu.org/msgid/20210715073328.212123-1-zimon.toutoune@gm= ail.com https://yhetil.org/guix/20210715073328.212123-1-zimon.toutoune@gmail.com 6: Re: List available versions of package. Philippe Veber Tue, 11 Jun 2019 09:43:08 +0200 id:CAOOOohSzUezKvm=3DRO0bXRGH3m0eo2x0cOTvd--vARxWoqtceaQ@mail.gmail.com https://lists.gnu.org/archive/html/help-guix/2019-06 https://yhetil.org/guix/CAOOOohSzUezKvm=3DRO0bXRGH3m0eo2x0cOTvd--vARxWoqtce= aQ@mail.gmail.com 7: https://guix.gnu.org/en/blog/2021/outreachy-guix-git-log-internship-wrap-up/ 8: Re: Mechanism for helping in multi-channels configuration (and Xapian in= dex) Simon Tournier Mon, 06 May 2024 14:05:50 +0200 id:87pltzp2ld.fsf@gmail.com https://lists.gnu.org/archive/html/guix-devel/2024-05 https://yhetil.org/guix/87pltzp2ld.fsf@gmail.com