From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:42833) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jB2Xn-0002z8-Kg for guix-patches@gnu.org; Sun, 08 Mar 2020 16:29:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jB2Xm-0002Ex-KQ for guix-patches@gnu.org; Sun, 08 Mar 2020 16:29:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:43364) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jB2Xm-0002Eo-Hm for guix-patches@gnu.org; Sun, 08 Mar 2020 16:29:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jB2Xm-0004iH-F1 for guix-patches@gnu.org; Sun, 08 Mar 2020 16:29:02 -0400 Subject: [bug#39258] [PATCH v2 0/3] Xapian for Guix package search Resent-Message-ID: From: Arun Isaac In-Reply-To: <875zffcc87.fsf@gnu.org> References: <20200307133116.11443-1-arunisaac@systemreboot.net> <87sgijgb1v.fsf@gnu.org> <875zffcc87.fsf@gnu.org> Date: Mon, 09 Mar 2020 01:57:40 +0530 Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: mail@ambrevar.xyz, 39258@debbugs.gnu.org, zimon.toutoune@gmail.com --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable >> This could be accomplished even with pre-rendering. Xapian provides >> "slots" to store arbitrary strings with a document. Instead of storing >> the pre-rendered document as a whole, we could store pre-rendered fields >> in separate slots. Then, during `guix search` time, we can assemble the >> result from these pre-rendered fields. > > I=E2=80=99m not sure I understand. The index wouldn=E2=80=99t store pre-= rendered > strings for every possible terminal width, right? No, it wouldn't. It would store a partially pre-rendered string, that is without fill-paragraph. We run fill-paragraph at `guix search` time to complete the rendering. > I think we need to take the whole user experience into account, not > just =E2=80=98guix search=E2=80=99. =E2=80=98guix pull=E2=80=99 already = feels very slow, and it=E2=80=99s a > fairly common operation. Conversely, =E2=80=98guix search=E2=80=99 takes= roughly > between 0.5 and 2 seconds and is an uncommon operation on a =E2=80=9Cslow > path=E2=80=9D (in the sense that when you=E2=80=99re searching for softwa= re, you=E2=80=99ll > probably have to spend more than a couple of seconds to find what > you=E2=80=99re looking for.) I agree we can't compromise too much on `guix pull` performance. > To me, adding 20=E2=80=9350 seconds on =E2=80=98guix pull=E2=80=99 would = be undesirable. :-/ Maybe I'm missing something here. guix pull takes around 40 minutes on my machine. In comparison to that, is another 20-50 seconds (roughly 1 minute) a big deal? How much time would it be acceptable to spend on building the Xapian index? Also, is it possible to somehow provide substitutes for the Xapian index so that the user does not have to actually build it locally during `guix pull` time? > I=E2=80=99m not sufficiently familiar with Xapian=E2=80=99s query languag= e. The > examples I had in mind were: > It=E2=80=99s not so much about regexps than it is about selecting individ= ual > fields. I have totally not tested this, but I imagine that equivalent Xapian queries might look something like: > guix search | recsel -p name -e 'license ~ "LGPL 3"' guix search license:LGPL3 > guix search crypto library | \ > recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis guix search crypto library AND (NOT ghc) AND (NOT perl) AND (NOT python) AND (NOT ruby) > What I meant was that we could use (statprof) to see whether/how Texinfo > rendering/parsing can be optimized. Oh, ok. I'll try this if we decide not to pre-render. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEf3MDQ/Lwnzx3v3nTLiXui2GAK7MFAl5lVTwACgkQLiXui2GA K7MufAf+PuuYkEHruEk5UrGI/8ofRfpKs0DL5GUdhc09xLrYI7GnBu3IkOB/2ock UfNDd1Vkgs0UKPs6rbkxer/IJ8Vf/Eat5B7r+RlDoUxByICgj/IRhgBYz42IUrfS oZuM5XxeBS158WvNvQcFfYgrXrkUl/WpxyoWZP64xEZBhgSEJYbV783Mf6trYiOY TLwuRi4vzZQru1RAHu8L+GoAUE5SGTf6jLb5xPZ5cYdNXwlRq2zSZcm8+UWv7RCR v9MVlydIsOtb++TSI/n/xwCL9eeSl8Sz5VTphVpHrZ74VKKakI0DEiubg3bTzxR9 WD9peO5+8BTwuI3kI/hmmKDx3vmQQw== =XhPo -----END PGP SIGNATURE----- --=-=-=--