Re: [PATCH] profiles: Generate database file for manpages

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 9327 bytes --]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> I've managed to finalize a profile hooks which takes care of generating
>> the manpages database file. This file is used by apropos or man -k to
>> provide results.
>
> Awesome!  I’ve been willing to have this for some time.
>

Thanks for providing feedback on this so quickly!

>> I've tested and found it to be working for the regular user profile. To
>> try it, apply the patch and use a guix command that causes a new profile
>> generation.
>>
>> For example: guix package -r git -i git
>> Then: "apropos git" should return plenty of results.
>
> For some reason I have to use “man -K” (capital):
>
> ~$ apropos guile
> guile: nothing appropriate.
> ~$ man -k guile
> guile: nothing appropriate.
> ~$ man -K guile
> --Man-- next: skribilo(1) [ view (return) | skip (Ctrl-D) | quit (Ctrl-C) ]
> --Man-- next: guile(1) [ view (return) | skip (Ctrl-D) | quit (Ctrl-C) ]
> --Man-- next: guix-pull(1) [ view (return) | skip (Ctrl-D) | quit (Ctrl-C) ]
> --Man-- next: guix-package(1) [ view (return) | skip (Ctrl-D) | quit (Ctrl-C) ]
> --Man-- next: gv(3guile) [ view (return) | skip (Ctrl-D) | quit (Ctrl-C) ]
> ~$ env | grep MANPATH
> MANPATH=/home/ludo/.guix-profile/share/man:/run/current-system/profile/share/man:/home/ludo/.guix-profile/share/man:/run/current-system/profile/share/man:/home/ludo/.opam/system/man
>
>
> Any idea why?
>

Unfortunately some of the simplifications you made appear to break it
:). I've tried your reworked patch and got the same result as you (no
result found except when using man -K). When I use the original patch it
works as I originally described.

>> It requires extra processing time to do the work since the complete
>> database (on my system, there are thousands or manpages being indexed)
>> is recreated everytime a new profile is generated.
>
> On my laptop (with an SSD), I found it to be reasonable:
>
> $ time guix build /gnu/store/rkri628apz2a2i2jvav11ylv2736fvv3-manual-database.drv --check
> @ build-started /gnu/store/rkri628apz2a2i2jvav11ylv2736fvv3-manual-database.drv - x86_64-linux /var/log/guix/drvs/rk//ri628apz2a2i2jvav11ylv2736fvv3-manual-database.drv.bz2
> /gnu/store/g7qxdk1cn7fwkamkaisi4428k6ijg0w2-manual-database
>
> real	0m2.647s
> user	0m0.116s
> sys	0m0.004s
> $ guix package -I |wc -l
> 229
>
>
>> For now it doesn't work for guix environments (yet), but this should be
>> easy to fix (it seems the $GUIX_ENVIRONMENT/share/man should be merged
>> with $HOME/.guix-profile/share/man, but I need to look at it more
>> carefully).
>
> It seems to work but only with capital -K:
>
> $ ./pre-inst-env guix environment --ad-hoc guile man-db --pure -- man -K guile
> man: can't execute cat: No such file or directory
> man: command exited with status 255: sed -e '/^[[:space:]]*$/{ N; /^[[:space:]]*\n[[:space:]]*$/D; }' | (cd <fd 3> && LESS=-ix8RmPm Manual page guile(1) ?ltline %lt?L/%L.:byte %bB?s/%s..?e (END):?pB %pB\%.. (press h for help or q to quit)$PM Manual page guile(1) ?ltline %lt?L/%L.:byte %bB?s/%s..?e (END):?pB %pB\%.. (press h for help or q to quit)$ MAN_PN=guile(1) cat)
>
> (And we should hard-code the file name of ‘cat’ in ‘man’…)
>
>>
>> The mandb database file (index.db) is used by the "apropos" (whatis) or
>> "man -k" commands. This change introduces a profile hook to generate
>> such database file.
>>
>> Co-authored by Ludovic Courtès
>>
>> * guix/profiles.scm (manual-database): New procedure.
>> (%default-profile-hooks): Add it.
>
> I made some simplifications to the patch.  WDYT?
>

Looks good, except for the 2 problematic changes in the reworked patch
identified below.

> Thanks,
> Ludo’.
>
> diff --git a/guix/profiles.scm b/guix/profiles.scm
> index 795c9447f..f565f358b 100644
> --- a/guix/profiles.scm
> +++ b/guix/profiles.scm
> @@ -7,6 +7,7 @@
>  ;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>
>  ;;; Copyright © 2016 Chris Marusich <cmmarusich@gmail.com>
>  ;;; Copyright © 2017 Huang Ying <huang.ying.caritas@gmail.com>
> +;;; Copyright © 2017 Maxim Cournoyer <maxim.cournoyer@gmail.com>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -946,10 +947,73 @@ files for the fonts of the @var{manifest} entries."
>                      #:local-build? #t
>                      #:substitutable? #f))
>  
> +(define (manual-database manifest)
> +  (define man-db                                  ;lazy reference
> +    (module-ref (resolve-interface '(gnu packages man)) 'man-db))
> +
> +  (define build
> +    #~(begin
> +        (use-modules (guix build utils)
> +                     (srfi srfi-1))
> +
> +        (define entries
> +          (filter-map (lambda (directory)
> +                        (let ((man (string-append directory "/share/man")))
> +                          (and (directory-exists? man)
> +                               man)))
> +                      '#$(manifest-inputs manifest)))
> +
> +        (define manpages-collection-dir
> +          (string-append (getenv "PWD") "/manpages-collection"))
> +
> +        (define man-directory
> +          (string-append #$output "/share/man"))
> +
> +        (define (populate-manpages-collection-dir entries)
> +          (let ((manpages (append-map find-files entries)))
> +            (for-each (lambda (manpage)
> +                        (let* ((dest-file (string-append
> +                                           manpages-collection-dir "/"
> +                                           (strip-store-file-name manpage))))

Unfortunately we cannot simply use strip-store-file-name, since what
mandb expects to find in our manpages-collection-dir is a collection of
subdirectories such as man1, man3, man7, etc. such as would be found
under "/usr/share/man" on a traditional distro. This change instead
makes it look like:

bazaar-2.7.0/          gdbm-1.12/              libxau-1.0.8/  ...

I guess this is what broke it.

> +                          (mkdir-p (dirname dest-file))
> +                          (catch 'system-error
> +                            (lambda ()
> +                              (symlink manpage dest-file))
> +                            (lambda args
> +                              ;; Different packages may contain the same
> +                              ;; manpage.  Simply ignore the symlink error.
> +                              #t))))
> +                      manpages)))
> +
> +        (mkdir-p manpages-collection-dir)
> +        (populate-manpages-collection-dir entries)
> +
> +        ;; Create a mandb config file which contains a custom made
> +        ;; manpath. The associated catpath is the location where the database
> +        ;; gets generated.
> +        (copy-file #+(file-append man-db "/etc/man_db.conf")
> +                   "man_db.conf")
> +
> +        (substitute* "man_db.conf"
> +          (("MANDB_MAP\\>.*$")
> +           (string-append "MANDB_MAP " manpages-collection-dir " "
> +                          man-directory "\n")))

This change here will replace every MANDB_MAP entries with duplicated
information:

MANDB_MAP /tmp/guix-build-manual-database.drv-0/manpages-collection /gnu/store/6w3gq10pgi8qmyq9i989glxs8md4miyk-manual-database/share/man
MANDB_MAP /tmp/guix-build-manual-database.drv-0/manpages-collection /gnu/store/6w3gq10pgi8qmyq9i989glxs8md4miyk-manual-database/share/man
MANDB_MAP /tmp/guix-build-manual-database.drv-0/manpages-collection /gnu/store/6w3gq10pgi8qmyq9i989glxs8md4miyk-manual-database/share/man
MANDB_MAP /tmp/guix-build-manual-database.drv-0/manpages-collection /gnu/store/6w3gq10pgi8qmyq9i989glxs8md4miyk-manual-database/share/man
MANDB_MAP /tmp/guix-build-manual-database.drv-0/manpages-collection /gnu/store/6w3gq10pgi8qmyq9i989glxs8md4miyk-manual-database/share/man
MANDB_MAP /tmp/guix-build-manual-database.drv-0/manpages-collection
/gnu/store/6w3gq10pgi8qmyq9i989glxs8md4miyk-manual-database/share/man

I don't think this is desirable. At first I was removing
all the MANDB_MAP entries and then adding only our, but there's nothing
to gain from doing this since the original man_db.conf file (bundled
with mandb) is still somehow read and contains those entries anyway
(which are harmless).

> +
> +        (mkdir-p man-directory)
> +        (setenv "MANPATH" (string-join entries ":"))
> +        (zero? (system* #+(file-append man-db "/bin/mandb")
> +                        "--quiet" "--create"
> +                        "-C" "man_db.conf"))))
> +
> +  (gexp->derivation "manual-database" build
> +                    #:modules '((guix build utils))
> +                    #:local-build? #t))
> +
>  (define %default-profile-hooks
>    ;; This is the list of derivation-returning procedures that are called by
>    ;; default when making a non-empty profile.
>    (list info-dir-file
> +        manual-database
>          fonts-dir-file
>          ghc-package-cache-file
>          ca-certificate-bundle


I have reverted these two problematic changes and attached a new patch. Please
try it out and let me know how it works :)

Thanks,

Maxim

[-- Attachment #1.2: 0001-profiles-Generate-database-file-for-manpages.patch --]
[-- Type: text/x-patch, Size: 4522 bytes --]

From c81fc96da953c867749b56b96324f500ba428f03 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Tue, 28 Mar 2017 09:25:21 -0700
Subject: [PATCH] profiles: Generate database file for manpages
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The mandb database file (index.db) is used by the "apropos" (whatis) or
"man -k" commands. This change introduces a profile hook to generate
such database file.

Co-authored by Ludovic Courtès

* guix/profiles.scm (manual-database): New procedure.
(%default-profile-hooks): Add it.
---
 guix/profiles.scm | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index 795c9447fe..0c719987b8 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -7,6 +7,7 @@
 ;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Chris Marusich <cmmarusich@gmail.com>
 ;;; Copyright © 2017 Huang Ying <huang.ying.caritas@gmail.com>
+;;; Copyright © 2017 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -946,10 +947,76 @@ files for the fonts of the @var{manifest} entries."
                     #:local-build? #t
                     #:substitutable? #f))
 
+(define (manual-database manifest)
+  (define man-db                                  ;lazy reference
+    (module-ref (resolve-interface '(gnu packages man)) 'man-db))
+
+  (define build
+    #~(begin
+        (use-modules (guix build utils)
+                     (srfi srfi-1))
+
+        (define entries
+          (filter-map (lambda (directory)
+                        (let ((man (string-append directory "/share/man")))
+                          (and (directory-exists? man)
+                               man)))
+                      '#$(manifest-inputs manifest)))
+
+        (define manpages-collection-dir
+          (string-append (getenv "PWD") "/manpages-collection"))
+
+        (define man-directory
+          (string-append #$output "/share/man"))
+
+        (define (get-manpage-tail-path manpage-path)
+          (let ((index (string-contains manpage-path "/share/man/")))
+            (substring manpage-path (+ index (string-length "/share/man/")))))
+
+        (define (populate-manpages-collection-dir entries)
+          (let ((manpages (append-map find-files entries)))
+            (for-each (lambda (manpage)
+                        (let* ((dest-file (string-append
+                                           manpages-collection-dir "/"
+                                           (get-manpage-tail-path manpage))))
+                          (mkdir-p (dirname dest-file))
+                          (catch 'system-error
+                            (lambda ()
+                              (symlink manpage dest-file))
+                            (lambda args
+                              ;; Different packages may contain the same
+                              ;; manpage.  Simply ignore the symlink error.
+                              #t))))
+                      manpages)))
+
+        (mkdir-p manpages-collection-dir)
+        (populate-manpages-collection-dir entries)
+
+        ;; Create a mandb config file which contains a custom made
+        ;; manpath. The associated catpath is the location where the database
+        ;; gets generated.
+        (copy-file #+(file-append man-db "/etc/man_db.conf")
+                   "man_db.conf")
+        (substitute* "man_db.conf"
+          (("MANDB_MAP	/usr/man		/var/cache/man/fsstnd")
+           (string-append "MANDB_MAP " manpages-collection-dir " "
+                          man-directory)))
+
+        (mkdir-p man-directory)
+        (setenv "MANPATH" (string-join entries ":"))
+        (zero? (system* #+(file-append man-db "/bin/mandb")
+                        "--quiet" "--create"
+                        "-C" "man_db.conf"))))
+
+  (gexp->derivation "manual-database" build
+                    #:modules '((guix build utils))
+                    #:local-build? #t))
+
 (define %default-profile-hooks
   ;; This is the list of derivation-returning procedures that are called by
   ;; default when making a non-empty profile.
   (list info-dir-file
+        manual-database
         fonts-dir-file
         ghc-package-cache-file
         ca-certificate-bundle
-- 
2.12.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]