From: Arun Isaac <arunisaac@systemreboot.net>
To: 39258@debbugs.gnu.org
Cc: Arun Isaac <arunisaac@systemreboot.net>,
mail@ambrevar.xyz, ludo@gnu.org, zimon.toutoune@gmail.com
Subject: [bug#39258] [PATCH v2 2/3] gnu: Generate Xapian package search index.
Date: Sat, 7 Mar 2020 19:01:15 +0530 [thread overview]
Message-ID: <20200307133116.11443-3-arunisaac@systemreboot.net> (raw)
In-Reply-To: <20200307133116.11443-1-arunisaac@systemreboot.net>
* guix/ui.scm: Export %package-metrics.
* gnu/packages.scm (%package-search-index): New variable.
(generate-package-search-index): New function.
* guix/channels.scm (package-search-index): New function.
(%channel-profile-hooks): Add package-search-index.
---
gnu/packages.scm | 42 +++++++++++++++++++++++++++++++++++++++++-
guix/channels.scm | 34 +++++++++++++++++++++++++++++++++-
guix/ui.scm | 2 ++
3 files changed, 76 insertions(+), 2 deletions(-)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..c8e221de68 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com>
;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
+;;; Copyright © 2020 Arun Isaac <arunisaac@systemreboot.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -43,6 +44,7 @@
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-35)
#:use-module (srfi srfi-39)
+ #:use-module (xapian xapian)
#:export (search-patch
search-patches
search-auxiliary-file
@@ -64,7 +66,8 @@
specification->location
specifications->manifest
- generate-package-cache))
+ generate-package-cache
+ generate-package-search-index))
;;; Commentary:
;;;
@@ -426,6 +429,43 @@ reducing the memory footprint."
#:opts '(#:to-file? #t)))))
cache-file)
+(define %package-search-index
+ ;; Location of the package search-index
+ "/lib/guix/package-search.index")
+
+(define (generate-package-search-index directory)
+ "Generate under DIRECTORY a Xapian index of all the available packages."
+ (define db-path
+ (string-append directory %package-search-index))
+
+ (mkdir-p (dirname db-path))
+ (call-with-writable-database db-path
+ (lambda (db)
+ (fold-packages (lambda (package _)
+ (let* ((idterm (string-append "Q" (package-name package)))
+ (doc (make-document #:data (string-trim-right
+ (call-with-output-string
+ (cut package->recutils package <>))
+ #\newline)
+ #:terms `((,idterm . 0))))
+ (term-generator (make-term-generator #:stem (make-stem "en")
+ #:document doc)))
+ (for-each (match-lambda
+ ((field . weight)
+ (match (field package)
+ ((? string? str)
+ (index-text! term-generator str
+ #:wdf-increment weight))
+ ((lst ...)
+ (for-each (cut index-text! term-generator <>
+ #:wdf-increment weight)
+ lst)))
+ (replace-document! db idterm doc)))
+ %package-metrics)))
+ #f)))
+
+ db-path)
+
\f
(define %sigint-prompt
;; The prompt to jump to upon SIGINT.
diff --git a/guix/channels.scm b/guix/channels.scm
index f0261dc2da..c70c70938c 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
+;;; Copyright © 2020 Arun Isaac <arunisaac@systemreboot.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -581,9 +582,40 @@ be used as a profile hook."
(hook . package-cache))
#:local-build? #t)))
+(define (package-search-index manifest)
+ "Build a package search index for the instance in MANIFEST. This is meant
+to be used as a profile hook."
+ (mlet %store-monad ((profile (profile-derivation manifest
+ #:hooks '())))
+
+ (define build
+ #~(begin
+ (use-modules (gnu packages))
+
+ (if (defined? 'generate-package-search-index)
+ (begin
+ ;; Delegate package search index generation to the inferior.
+ (format (current-error-port)
+ "Generating package search index for '~a'...~%"
+ #$profile)
+ (generate-package-search-index #$output))
+ (mkdir #$output))))
+
+ (gexp->derivation-in-inferior "guix-package-search-index" build
+ profile
+
+ ;; If the Guix in PROFILE is too old and
+ ;; lacks 'guix repl', don't build the cache
+ ;; instead of failing.
+ #:silent-failure? #t
+
+ #:properties '((type . profile-hook)
+ (hook . package-search-index))
+ #:local-build? #t)))
+
(define %channel-profile-hooks
;; The default channel profile hooks.
- (cons package-cache-file %default-profile-hooks))
+ (cons* package-cache-file package-search-index %default-profile-hooks))
(define (channel-instances->derivation instances)
"Return the derivation of the profile containing INSTANCES, a list of
diff --git a/guix/ui.scm b/guix/ui.scm
index fbe2b70485..3bc82111a5 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -14,6 +14,7 @@
;;; Copyright © 2019 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2019 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2019 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2020 Arun Isaac <arunisaac@systemreboot.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -120,6 +121,7 @@
relevance
package-relevance
display-search-results
+ %package-metrics
with-profile-lock
string->generations
--
2.25.1
next prev parent reply other threads:[~2020-03-07 13:32 UTC|newest]
Thread overview: 126+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-23 19:51 [bug#39258] Faster guix search using an sqlite cache Arun Isaac
2020-01-29 23:33 ` zimoun
2020-01-30 13:48 ` Arun Isaac
2020-01-31 12:48 ` zimoun
2020-02-02 21:16 ` Arun Isaac
2020-02-04 10:19 ` zimoun
2020-02-06 1:58 ` Arun Isaac
2020-02-11 16:29 ` Ludovic Courtès
2020-02-11 18:21 ` zimoun
2020-02-11 18:39 ` Ludovic Courtès
2020-02-11 19:07 ` Arun Isaac
2020-02-11 20:20 ` zimoun
2020-02-15 14:50 ` Arun Isaac
2020-02-11 20:13 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 0/4] Xapian for Guix package search Arun Isaac
2020-02-27 20:41 ` [bug#39258] [PATCH 1/4] gnu: Add guile-xapian Arun Isaac
2020-03-03 16:29 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 2/4] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-02-27 20:41 ` [bug#39258] [PATCH 3/4] gnu: Generate xapian package search index Arun Isaac
2020-02-28 8:04 ` Pierre Neidhardt
2020-03-05 20:26 ` Arun Isaac
2020-03-03 18:29 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 4/4] gnu: Use xapian index for package search Arun Isaac
2020-02-28 8:11 ` Pierre Neidhardt
2020-03-03 19:21 ` zimoun
2020-03-03 19:51 ` zimoun
2020-02-28 8:13 ` [bug#39258] [PATCH 0/4] Xapian for Guix " Pierre Neidhardt
2020-02-28 12:39 ` zimoun
2020-02-28 12:49 ` Pierre Neidhardt
2020-02-28 15:36 ` Arun Isaac
2020-02-28 16:04 ` Arun Isaac
2020-03-02 18:37 ` zimoun
2020-03-02 19:13 ` zimoun
2020-03-03 20:04 ` zimoun
2020-02-29 8:25 ` Arun Isaac
2020-03-02 18:27 ` zimoun
2020-02-28 12:36 ` zimoun
2020-03-05 16:46 ` Ludovic Courtès
2020-03-07 13:31 ` [bug#39258] [PATCH v2 0/3] " Arun Isaac
2020-03-07 13:31 ` [bug#39258] [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-03-09 18:14 ` zimoun
2020-03-09 23:40 ` Jonathan Brielmaier
2020-03-10 5:24 ` Arun Isaac
2020-03-07 13:31 ` Arun Isaac [this message]
2020-03-09 18:19 ` [bug#39258] [PATCH v2 2/3] gnu: Generate Xapian package search index zimoun
2020-03-07 13:31 ` [bug#39258] [PATCH v2 3/3] gnu: Use Xapian index for package search Arun Isaac
2020-03-07 20:33 ` [bug#39258] [PATCH v2 0/3] Xapian for Guix " Ludovic Courtès
2020-03-08 9:01 ` Arun Isaac
2020-03-08 11:33 ` Ludovic Courtès
2020-03-08 20:27 ` Arun Isaac
2020-03-09 7:42 ` Pierre Neidhardt
2020-03-09 12:50 ` zimoun
2020-03-09 10:35 ` Ludovic Courtès
2020-03-10 14:17 ` Arun Isaac
2020-03-10 14:33 ` zimoun
2020-03-11 13:50 ` Ludovic Courtès
2020-03-13 5:37 ` Arun Isaac
2020-03-15 20:40 ` Ludovic Courtès
2020-03-09 7:50 ` Pierre Neidhardt
2020-03-09 10:28 ` Ludovic Courtès
2020-03-09 13:03 ` zimoun
2020-03-09 12:53 ` zimoun
2020-03-09 12:47 ` zimoun
2020-03-09 12:40 ` zimoun
2020-03-09 12:34 ` zimoun
2020-03-08 20:27 ` zimoun
2020-03-08 20:40 ` Arun Isaac
2020-03-09 12:28 ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Arun Isaac
2020-03-27 16:26 ` [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache Arun Isaac
2020-04-24 20:48 ` Ludovic Courtès
2020-04-26 9:48 ` zimoun
2020-04-26 14:35 ` Ludovic Courtès
2020-04-26 14:54 ` Pierre Neidhardt
2020-04-26 15:33 ` Ludovic Courtès
2020-04-26 15:05 ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 2/3] guix: Search " Arun Isaac
2020-04-24 20:58 ` Ludovic Courtès
2020-03-27 16:26 ` [bug#39258] [PATCH v3 3/3] guix: Use package metadata cache for package search Arun Isaac
2020-04-24 21:03 ` Ludovic Courtès
2020-04-05 14:08 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Ludovic Courtès
2020-04-24 21:05 ` Ludovic Courtès
2020-04-26 3:54 ` [bug#39258] benchmark search: default vs v2 vs v3 zimoun
2020-04-26 7:29 ` Pierre Neidhardt
2020-04-26 15:49 ` Ludovic Courtès
2020-04-26 17:01 ` zimoun
2020-04-26 20:22 ` Ludovic Courtès
2020-04-30 13:10 ` zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 1/3] DRAFT packages: Add fields to packages cache zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 2/3] DRAFT packages: Add new procedure 'fold-packages*' zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 3/3] DRAFT guix package: Use cache in 'find-packages-by-description' zimoun
2020-05-03 16:43 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) Ludovic Courtès
2020-05-03 18:10 ` zimoun
2020-05-03 19:49 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 0/4] Optimize guix search Arun Isaac
2020-06-01 0:00 ` [bug#39258] [PATCH 1/4] ui: Cut off search early if any regexp does not match Arun Isaac
2020-06-09 8:29 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings Arun Isaac
2020-06-09 8:33 ` Ludovic Courtès
2020-06-09 9:55 ` zimoun
2020-06-13 12:37 ` Arun Isaac
2020-06-13 13:36 ` zimoun
2020-06-13 17:21 ` Arun Isaac
2020-06-14 19:14 ` zimoun
2020-06-13 19:32 ` Ludovic Courtès
2020-06-15 20:18 ` Arun Isaac
2020-06-01 0:00 ` [bug#39258] [PATCH 3/4] ui: Do not translate package synopsis a second time Arun Isaac
2020-06-09 8:33 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 4/4] ui: Use package-description-string Arun Isaac
2020-06-09 8:34 ` Ludovic Courtès
2020-06-01 1:25 ` [bug#39258] [PATCH v5 0/4] Optimize guix search zimoun
2020-06-01 2:24 ` Arun Isaac
2020-06-01 10:01 ` zimoun
2020-06-01 10:11 ` [bug#39258] KMP string search algorithm? zimoun
2020-06-01 22:24 ` Leo Famulari
2020-06-01 23:48 ` Arun Isaac
2020-06-02 8:49 ` Ludovic Courtès
2021-07-15 7:33 ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances zimoun
2021-07-15 7:33 ` [bug#39258] [PATCH v6 1/2] DRAFT packages: Add fields to packages cache zimoun
2021-07-17 8:31 ` Arun Isaac
2021-07-23 15:30 ` Ludovic Courtès
2021-08-17 14:03 ` zimoun
2021-07-15 7:33 ` [bug#39258] [PATCH v6 2/2] DRAFT scripts: package: Use cache in 'find-packages-by-description' zimoun
2021-07-23 15:43 ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances Ludovic Courtès
2021-08-20 15:42 ` zimoun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200307133116.11443-3-arunisaac@systemreboot.net \
--to=arunisaac@systemreboot.net \
--cc=39258@debbugs.gnu.org \
--cc=ludo@gnu.org \
--cc=mail@ambrevar.xyz \
--cc=zimon.toutoune@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).