From: zimoun <zimon.toutoune@gmail.com>
To: Arun Isaac <arunisaac@systemreboot.net>
Cc: "Ludovic Courtès" <ludo@gnu.org>, 39258@debbugs.gnu.org
Subject: [bug#39258] [PATCH 3/4] gnu: Generate xapian package search index.
Date: Tue, 3 Mar 2020 19:29:57 +0100 [thread overview]
Message-ID: <CAJ3okZ2TxN=mxQKhP7S7-yeAmBhYFiBkGHDYbzjD2kO_X7igFQ@mail.gmail.com> (raw)
In-Reply-To: <20200227204150.30985-4-arunisaac@systemreboot.net>
Hi Arun,
In the commit message, I would capitalize Xapian.
On Thu, 27 Feb 2020 at 21:42, Arun Isaac <arunisaac@systemreboot.net> wrote:
>
> * gnu/packages.scm (%package-search-index): New variable.
> (generate-package-search-index): New function.
> * guix/channels.scm (package-search-index): New function.
> (%channel-profile-hooks): Add package-search-index.
> ---
> gnu/packages.scm | 29 ++++++++++++++++++++++++++++-
> guix/channels.scm | 34 +++++++++++++++++++++++++++++++++-
> 2 files changed, 61 insertions(+), 2 deletions(-)
>
> diff --git a/gnu/packages.scm b/gnu/packages.scm
> index d22c992bb1..e91753e2a8 100644
> --- a/gnu/packages.scm
> +++ b/gnu/packages.scm
> @@ -4,6 +4,7 @@
> ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
> ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com>
> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
> +;;; Copyright © 2020 Arun Isaac <arunisaac@systemreboot.net>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -43,6 +44,7 @@
> #:use-module (srfi srfi-34)
> #:use-module (srfi srfi-35)
> #:use-module (srfi srfi-39)
> + #:use-module (xapian xapian)
> #:export (search-patch
> search-patches
> search-auxiliary-file
> @@ -64,7 +66,8 @@
> specification->location
> specifications->manifest
>
> - generate-package-cache))
> + generate-package-cache
> + generate-package-search-index))
>
> ;;; Commentary:
> ;;;
> @@ -426,6 +429,30 @@ reducing the memory footprint."
> #:opts '(#:to-file? #t)))))
> cache-file)
>
> +(define %package-search-index
> + ;; Location of the package search-index
> + "/lib/guix/package-search.index")
> +
> +(define (generate-package-search-index directory)
> + "Generate under DIRECTORY a xapian index of all the available packages."
Xapian with capital.
> + (define db-path
> + (string-append directory %package-search-index))
> +
> + (mkdir-p (dirname db-path))
> + (call-with-writable-database db-path
> + (lambda (db)
> + (fold-packages (lambda (package _)
> + (let* ((idterm (string-append "Q" (package-name package)))
> + (doc (make-document #:data (package-name package)
> + #:terms `((,idterm . 0))))
> + (term-generator (make-term-generator #:stem (make-stem "en")
> + #:document doc)))
> + (index-text! term-generator (package-description package))
Instead, this:
(index-term! term-generator (string-append (package-synopsis package)
(package-description package)))
should index both 'synopsis' and 'description'.
Is (make-stem "en") for the locale?
> + (replace-document! db idterm doc)))
> + #f)))
> +
> + db-path)
> +
>
> (define %sigint-prompt
> ;; The prompt to jump to upon SIGINT.
> diff --git a/guix/channels.scm b/guix/channels.scm
> index f0261dc2da..c70c70938c 100644
> --- a/guix/channels.scm
> +++ b/guix/channels.scm
> @@ -2,6 +2,7 @@
> ;;; Copyright © 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
> ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
> ;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
> +;;; Copyright © 2020 Arun Isaac <arunisaac@systemreboot.net>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -581,9 +582,40 @@ be used as a profile hook."
> (hook . package-cache))
> #:local-build? #t)))
>
> +(define (package-search-index manifest)
> + "Build a package search index for the instance in MANIFEST. This is meant
> +to be used as a profile hook."
> + (mlet %store-monad ((profile (profile-derivation manifest
> + #:hooks '())))
> +
> + (define build
> + #~(begin
> + (use-modules (gnu packages))
> +
> + (if (defined? 'generate-package-search-index)
> + (begin
> + ;; Delegate package search index generation to the inferior.
> + (format (current-error-port)
> + "Generating package search index for '~a'...~%"
> + #$profile)
> + (generate-package-search-index #$output))
> + (mkdir #$output))))
> +
> + (gexp->derivation-in-inferior "guix-package-search-index" build
> + profile
> +
> + ;; If the Guix in PROFILE is too old and
> + ;; lacks 'guix repl', don't build the cache
> + ;; instead of failing.
> + #:silent-failure? #t
> +
> + #:properties '((type . profile-hook)
> + (hook . package-search-index))
> + #:local-build? #t)))
> +
package-search-index and package-cache-file could be refactored
because they share all the same code.
> (define %channel-profile-hooks
> ;; The default channel profile hooks.
> - (cons package-cache-file %default-profile-hooks))
> + (cons* package-cache-file package-search-index %default-profile-hooks))
>
> (define (channel-instances->derivation instances)
> "Return the derivation of the profile containing INSTANCES, a list of
> --
> 2.23.0
>
next prev parent reply other threads:[~2020-03-03 18:31 UTC|newest]
Thread overview: 126+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-23 19:51 [bug#39258] Faster guix search using an sqlite cache Arun Isaac
2020-01-29 23:33 ` zimoun
2020-01-30 13:48 ` Arun Isaac
2020-01-31 12:48 ` zimoun
2020-02-02 21:16 ` Arun Isaac
2020-02-04 10:19 ` zimoun
2020-02-06 1:58 ` Arun Isaac
2020-02-11 16:29 ` Ludovic Courtès
2020-02-11 18:21 ` zimoun
2020-02-11 18:39 ` Ludovic Courtès
2020-02-11 19:07 ` Arun Isaac
2020-02-11 20:20 ` zimoun
2020-02-15 14:50 ` Arun Isaac
2020-02-11 20:13 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 0/4] Xapian for Guix package search Arun Isaac
2020-02-27 20:41 ` [bug#39258] [PATCH 1/4] gnu: Add guile-xapian Arun Isaac
2020-03-03 16:29 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 2/4] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-02-27 20:41 ` [bug#39258] [PATCH 3/4] gnu: Generate xapian package search index Arun Isaac
2020-02-28 8:04 ` Pierre Neidhardt
2020-03-05 20:26 ` Arun Isaac
2020-03-03 18:29 ` zimoun [this message]
2020-02-27 20:41 ` [bug#39258] [PATCH 4/4] gnu: Use xapian index for package search Arun Isaac
2020-02-28 8:11 ` Pierre Neidhardt
2020-03-03 19:21 ` zimoun
2020-03-03 19:51 ` zimoun
2020-02-28 8:13 ` [bug#39258] [PATCH 0/4] Xapian for Guix " Pierre Neidhardt
2020-02-28 12:39 ` zimoun
2020-02-28 12:49 ` Pierre Neidhardt
2020-02-28 15:36 ` Arun Isaac
2020-02-28 16:04 ` Arun Isaac
2020-03-02 18:37 ` zimoun
2020-03-02 19:13 ` zimoun
2020-03-03 20:04 ` zimoun
2020-02-29 8:25 ` Arun Isaac
2020-03-02 18:27 ` zimoun
2020-02-28 12:36 ` zimoun
2020-03-05 16:46 ` Ludovic Courtès
2020-03-07 13:31 ` [bug#39258] [PATCH v2 0/3] " Arun Isaac
2020-03-07 13:31 ` [bug#39258] [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-03-09 18:14 ` zimoun
2020-03-09 23:40 ` Jonathan Brielmaier
2020-03-10 5:24 ` Arun Isaac
2020-03-07 13:31 ` [bug#39258] [PATCH v2 2/3] gnu: Generate Xapian package search index Arun Isaac
2020-03-09 18:19 ` zimoun
2020-03-07 13:31 ` [bug#39258] [PATCH v2 3/3] gnu: Use Xapian index for package search Arun Isaac
2020-03-07 20:33 ` [bug#39258] [PATCH v2 0/3] Xapian for Guix " Ludovic Courtès
2020-03-08 9:01 ` Arun Isaac
2020-03-08 11:33 ` Ludovic Courtès
2020-03-08 20:27 ` Arun Isaac
2020-03-09 7:42 ` Pierre Neidhardt
2020-03-09 12:50 ` zimoun
2020-03-09 10:35 ` Ludovic Courtès
2020-03-10 14:17 ` Arun Isaac
2020-03-10 14:33 ` zimoun
2020-03-11 13:50 ` Ludovic Courtès
2020-03-13 5:37 ` Arun Isaac
2020-03-15 20:40 ` Ludovic Courtès
2020-03-09 7:50 ` Pierre Neidhardt
2020-03-09 10:28 ` Ludovic Courtès
2020-03-09 13:03 ` zimoun
2020-03-09 12:53 ` zimoun
2020-03-09 12:47 ` zimoun
2020-03-09 12:40 ` zimoun
2020-03-09 12:34 ` zimoun
2020-03-08 20:27 ` zimoun
2020-03-08 20:40 ` Arun Isaac
2020-03-09 12:28 ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Arun Isaac
2020-03-27 16:26 ` [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache Arun Isaac
2020-04-24 20:48 ` Ludovic Courtès
2020-04-26 9:48 ` zimoun
2020-04-26 14:35 ` Ludovic Courtès
2020-04-26 14:54 ` Pierre Neidhardt
2020-04-26 15:33 ` Ludovic Courtès
2020-04-26 15:05 ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 2/3] guix: Search " Arun Isaac
2020-04-24 20:58 ` Ludovic Courtès
2020-03-27 16:26 ` [bug#39258] [PATCH v3 3/3] guix: Use package metadata cache for package search Arun Isaac
2020-04-24 21:03 ` Ludovic Courtès
2020-04-05 14:08 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Ludovic Courtès
2020-04-24 21:05 ` Ludovic Courtès
2020-04-26 3:54 ` [bug#39258] benchmark search: default vs v2 vs v3 zimoun
2020-04-26 7:29 ` Pierre Neidhardt
2020-04-26 15:49 ` Ludovic Courtès
2020-04-26 17:01 ` zimoun
2020-04-26 20:22 ` Ludovic Courtès
2020-04-30 13:10 ` zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 1/3] DRAFT packages: Add fields to packages cache zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 2/3] DRAFT packages: Add new procedure 'fold-packages*' zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 3/3] DRAFT guix package: Use cache in 'find-packages-by-description' zimoun
2020-05-03 16:43 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) Ludovic Courtès
2020-05-03 18:10 ` zimoun
2020-05-03 19:49 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 0/4] Optimize guix search Arun Isaac
2020-06-01 0:00 ` [bug#39258] [PATCH 1/4] ui: Cut off search early if any regexp does not match Arun Isaac
2020-06-09 8:29 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings Arun Isaac
2020-06-09 8:33 ` Ludovic Courtès
2020-06-09 9:55 ` zimoun
2020-06-13 12:37 ` Arun Isaac
2020-06-13 13:36 ` zimoun
2020-06-13 17:21 ` Arun Isaac
2020-06-14 19:14 ` zimoun
2020-06-13 19:32 ` Ludovic Courtès
2020-06-15 20:18 ` Arun Isaac
2020-06-01 0:00 ` [bug#39258] [PATCH 3/4] ui: Do not translate package synopsis a second time Arun Isaac
2020-06-09 8:33 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 4/4] ui: Use package-description-string Arun Isaac
2020-06-09 8:34 ` Ludovic Courtès
2020-06-01 1:25 ` [bug#39258] [PATCH v5 0/4] Optimize guix search zimoun
2020-06-01 2:24 ` Arun Isaac
2020-06-01 10:01 ` zimoun
2020-06-01 10:11 ` [bug#39258] KMP string search algorithm? zimoun
2020-06-01 22:24 ` Leo Famulari
2020-06-01 23:48 ` Arun Isaac
2020-06-02 8:49 ` Ludovic Courtès
2021-07-15 7:33 ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances zimoun
2021-07-15 7:33 ` [bug#39258] [PATCH v6 1/2] DRAFT packages: Add fields to packages cache zimoun
2021-07-17 8:31 ` Arun Isaac
2021-07-23 15:30 ` Ludovic Courtès
2021-08-17 14:03 ` zimoun
2021-07-15 7:33 ` [bug#39258] [PATCH v6 2/2] DRAFT scripts: package: Use cache in 'find-packages-by-description' zimoun
2021-07-23 15:43 ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances Ludovic Courtès
2021-08-20 15:42 ` zimoun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAJ3okZ2TxN=mxQKhP7S7-yeAmBhYFiBkGHDYbzjD2kO_X7igFQ@mail.gmail.com' \
--to=zimon.toutoune@gmail.com \
--cc=39258@debbugs.gnu.org \
--cc=arunisaac@systemreboot.net \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).