Re: 04/06: profiles: Generate database file for man pages.

Re: 04/06: profiles: Generate database file for man pages.

[Leo Famulari]

On Wed, Apr 05, 2017 at 04:56:39PM -0400, Ludovic Court�s wrote:
> civodul pushed a commit to branch master
> in repository guix.
> 
> commit a0b87ef8ec7735aa42cf35d380e9cff04f3236f3
> Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> Date:   Wed Apr 5 01:09:22 2017 -0700
> 
>     profiles: Generate database file for man pages.
>     
>     The mandb database file (index.db) is used by the "apropos" (whatis) or
>     "man -k" commands.  This change introduces a profile hook to generate
>     such database file.
>     
>     * guix/profiles.scm (manual-database): New procedure.
>     (%default-profile-hooks): Add it.
>     
>     Co-authored-by: Ludovic Courtès <ludo@gnu.org>

Recently, I notice messages like this when doing profile operations:

[...]
creating manual page database for 49 packages...
find-files: /gnu/store/0y3qrzzwfxf601g885967wcringvnw69-gimp-2.8.18/share/man/man1/gimp-console.1: No such file or directory
[...]

The file in question is a symlink:
$ ls -l /gnu/store/0y3qrzzwfxf601g885967wcringvnw69-gimp-2.8.18/share/man/man1/gimp-console.1 
lrwxrwxrwx 5 root root 21 Dec 31  1969 /gnu/store/0y3qrzzwfxf601g885967wcringvnw69-gimp-2.8.18/share/man/man1/gimp-console.1 -> gimp-console-2.8.1.gz

Re: 04/06: profiles: Generate database file for man pages.

[myglc2]

On 04/05/2017 at 16:56 Ludovic Courtès writes:

> civodul pushed a commit to branch master
> in repository guix.
>
> commit a0b87ef8ec7735aa42cf35d380e9cff04f3236f3
> Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> Date:   Wed Apr 5 01:09:22 2017 -0700
[...]

So now q'guix package -m' produces messages like ...

guix package -m g1.scm
installing new manifest from 'g1.scm' with 22 entries
substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/smyjh63avay2gb6yq47h3hrlyxfrsajp-profile.drv
   /gnu/store/ygp7n87177fngqqpb5l5bkkba5cawcvh-gtk-icon-themes.drv
   /gnu/store/wrad8m1vhck9bb7dzqqlxb03rq1ifsm8-gtk-im-modules.drv
   /gnu/store/qv12155lx8y195my98970mmfawqpnrcs-info-dir.drv
   /gnu/store/ndiq7ccmi32rlxdyf52f506jil0zn9p0-ca-certificate-bundle.drv
   /gnu/store/90gzasiixp9np5m4nn28vbfqd1zd5azq-fonts-dir.drv
   /gnu/store/3l8kl1b438qr3yk2qca4pzvlpv04q7rc-xdg-mime-database.drv
   /gnu/store/2brridy714shmik1zksysik36kn2iwsf-manual-database.drv
creating manual page database for 23 packages...
22 packages in profile
g1@g1 ~/src$ 

And... watching this, it seems like a long time between ...

creating manual page database for 23 packages...

... and ...

22 packages in profile

Does it really take this long? If not, can we close the message ...

'creating manual page database for 25 packages...DONE'

... so that our gentle users will not be thinking ...

"I don't want this #*&^$*&#@%(*% man database!"

Re: 04/06: profiles: Generate database file for man pages.

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]

Hi Leo!

On Sun, Apr 9, 2017 at 4:29 AM, Leo Famulari <leo@famulari.name> wrote:

> On Wed, Apr 05, 2017 at 04:56:39PM -0400, Ludovic Court�s wrote:
> > civodul pushed a commit to branch master
> > in repository guix.
> >
> > commit a0b87ef8ec7735aa42cf35d380e9cff04f3236f3
> > Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> > Date:   Wed Apr 5 01:09:22 2017 -0700
> >
> >     profiles: Generate database file for man pages.
> >
> >     The mandb database file (index.db) is used by the "apropos" (whatis)
> or
> >     "man -k" commands.  This change introduces a profile hook to generate
> >     such database file.
> >
> >     * guix/profiles.scm (manual-database): New procedure.
> >     (%default-profile-hooks): Add it.
> >
> >     Co-authored-by: Ludovic Courtès <ludo@gnu.org>
>
> Recently, I notice messages like this when doing profile operations:
>
> [...]
> creating manual page database for 49 packages...
> find-files: /gnu/store/0y3qrzzwfxf601g885967wcringvnw
> 69-gimp-2.8.18/share/man/man1/gimp-console.1: No such file or directory
> [...]
>
> The file in question is a symlink:
> $ ls -l /gnu/store/0y3qrzzwfxf601g885967wcringvnw
> 69-gimp-2.8.18/share/man/man1/gimp-console.1
> lrwxrwxrwx 5 root root 21 Dec 31  1969 /gnu/store/
> 0y3qrzzwfxf601g885967wcringvnw69-gimp-2.8.18/share/man/man1/gimp-console.1
> -> gimp-console-2.8.1.gz
>

This find-files (used in the manual-database hook) message is caused by the
fact the gimp-console-2.8.1.gz file doesn`t exist (IOW, the gimp-console.1
symlink is broken). It would be nice to investigate is this Gimp problem
comes from upstream or as a result of our build process.

Maxim

[-- Attachment #2: Type: text/html, Size: 2311 bytes --]

Re: 04/06: profiles: Generate database file for man pages.

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 1838 bytes --]

Hi myglc2!

On Mon, Apr 10, 2017 at 2:28 AM, myglc2 <myglc2@gmail.com> wrote:
>
> On 04/05/2017 at 16:56 Ludovic Courtès writes:
>
> > civodul pushed a commit to branch master
> > in repository guix.
> >
> > commit a0b87ef8ec7735aa42cf35d380e9cff04f3236f3
> > Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> > Date:   Wed Apr 5 01:09:22 2017 -0700
> [...]
>
> So now q'guix package -m' produces messages like ...
>
> guix package -m g1.scm
> installing new manifest from 'g1.scm' with 22 entries
> substitute: updating list of substitutes from '
https://mirror.hydra.gnu.org'... 100.0%
> The following derivations will be built:
>    /gnu/store/smyjh63avay2gb6yq47h3hrlyxfrsajp-profile.drv
>    /gnu/store/ygp7n87177fngqqpb5l5bkkba5cawcvh-gtk-icon-themes.drv
>    /gnu/store/wrad8m1vhck9bb7dzqqlxb03rq1ifsm8-gtk-im-modules.drv
>    /gnu/store/qv12155lx8y195my98970mmfawqpnrcs-info-dir.drv
>    /gnu/store/ndiq7ccmi32rlxdyf52f506jil0zn9p0-ca-certificate-bundle.drv
>    /gnu/store/90gzasiixp9np5m4nn28vbfqd1zd5azq-fonts-dir.drv
>    /gnu/store/3l8kl1b438qr3yk2qca4pzvlpv04q7rc-xdg-mime-database.drv
>    /gnu/store/2brridy714shmik1zksysik36kn2iwsf-manual-database.drv
> creating manual page database for 23 packages...
> 22 packages in profile
> g1@g1 ~/src$
>
> And... watching this, it seems like a long time between ...
>
> creating manual page database for 23 packages...
>
> ... and ...
>
> 22 packages in profile
>
> Does it really take this long? If not, can we close the message ...

FWIR it is CPU bound, so your mileage will vary; in my case on an old
low-power laptop it can take a few seconds for ~65 packages.
I wonder why the odd package count mismatch (23 vs 22) ? I don't recall
observing this while testing (maybe it only happens when using the -m
option) ?

Maxim

[-- Attachment #2: Type: text/html, Size: 2300 bytes --]

Re: 04/06: profiles: Generate database file for man pages.

[myglc2]

On 04/10/2017 at 10:22 Maxim Cournoyer writes:

> Hi myglc2!
>
> On Mon, Apr 10, 2017 at 2:28 AM, myglc2 <myglc2@gmail.com> wrote:
>>
>> On 04/05/2017 at 16:56 Ludovic Courtès writes:
>>
>> > civodul pushed a commit to branch master
>> > in repository guix.
>> >
>> > commit a0b87ef8ec7735aa42cf35d380e9cff04f3236f3
>> > Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
>> > Date: Wed Apr 5 01:09:22 2017 -0700
>> [...]
>>
>> So now q'guix package -m' produces messages like ...
>>
>> guix package -m g1.scm
>> installing new manifest from 'g1.scm' with 22 entries
>> substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
>> The following derivations will be built:
>> /gnu/store/smyjh63avay2gb6yq47h3hrlyxfrsajp-profile.drv
>> /gnu/store/ygp7n87177fngqqpb5l5bkkba5cawcvh-gtk-icon-themes.drv
>> /gnu/store/wrad8m1vhck9bb7dzqqlxb03rq1ifsm8-gtk-im-modules.drv
>> /gnu/store/qv12155lx8y195my98970mmfawqpnrcs-info-dir.drv
>> /gnu/store/ndiq7ccmi32rlxdyf52f506jil0zn9p0-ca-certificate-bundle.drv
>> /gnu/store/90gzasiixp9np5m4nn28vbfqd1zd5azq-fonts-dir.drv
>> /gnu/store/3l8kl1b438qr3yk2qca4pzvlpv04q7rc-xdg-mime-database.drv
>> /gnu/store/2brridy714shmik1zksysik36kn2iwsf-manual-database.drv
>> creating manual page database for 23 packages...
>> 22 packages in profile
>> g1@g1 ~/src$
>>
>> And... watching this, it seems like a long time between ...
>>
>> creating manual page database for 23 packages...
>>
>> ... and ...
>>
>> 22 packages in profile
>>
>> Does it really take this long? If not, can we close the message ...
>
> FWIR it is CPU bound, so your mileage will vary; in my case on an old
> low-power laptop it can take a few seconds for ~65 packages.

It is taking more like 50 seconds on my 3.4 Ghz server for 33 packages.

> I wonder why the odd package count mismatch (23 vs 22) ?

That's my fault. I cut and pasted from two different runs. SORRY! ;-)

> I don't recall observing this while testing (maybe it only happens
> when using the -m option) ?

No, I get the same effect w/ 'guix package -r foo.

Re: 04/06: profiles: Generate database file for man pages.

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 3573 bytes --]

Hello myglc2,

myglc2 <myglc2@gmail.com> writes:

> On 04/10/2017 at 10:22 Maxim Cournoyer writes:
>
>> Hi myglc2!
>>
>> On Mon, Apr 10, 2017 at 2:28 AM, myglc2 <myglc2@gmail.com> wrote:
>>>
>>> On 04/05/2017 at 16:56 Ludovic Courtès writes:
>>>
>>> > civodul pushed a commit to branch master
>>> > in repository guix.
>>> >
>>> > commit a0b87ef8ec7735aa42cf35d380e9cff04f3236f3
>>> > Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
>>> > Date: Wed Apr 5 01:09:22 2017 -0700
>>> [...]
>>>
>>> So now q'guix package -m' produces messages like ...
>>>
>>> guix package -m g1.scm
>>> installing new manifest from 'g1.scm' with 22 entries
>>> substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
>>> The following derivations will be built:
>>> /gnu/store/smyjh63avay2gb6yq47h3hrlyxfrsajp-profile.drv
>>> /gnu/store/ygp7n87177fngqqpb5l5bkkba5cawcvh-gtk-icon-themes.drv
>>> /gnu/store/wrad8m1vhck9bb7dzqqlxb03rq1ifsm8-gtk-im-modules.drv
>>> /gnu/store/qv12155lx8y195my98970mmfawqpnrcs-info-dir.drv
>>> /gnu/store/ndiq7ccmi32rlxdyf52f506jil0zn9p0-ca-certificate-bundle.drv
>>> /gnu/store/90gzasiixp9np5m4nn28vbfqd1zd5azq-fonts-dir.drv
>>> /gnu/store/3l8kl1b438qr3yk2qca4pzvlpv04q7rc-xdg-mime-database.drv
>>> /gnu/store/2brridy714shmik1zksysik36kn2iwsf-manual-database.drv
>>> creating manual page database for 23 packages...
>>> 22 packages in profile
>>> g1@g1 ~/src$
>>>
>>> And... watching this, it seems like a long time between ...
>>>
>>> creating manual page database for 23 packages...
>>>
>>> ... and ...
>>>
>>> 22 packages in profile
>>>
>>> Does it really take this long? If not, can we close the message ...
>>
>> FWIR it is CPU bound, so your mileage will vary; in my case on an old
>> low-power laptop it can take a few seconds for ~65 packages.
>
> It is taking more like 50 seconds on my 3.4 Ghz server for 33 packages.
>
>> I wonder why the odd package count mismatch (23 vs 22) ?
>
> That's my fault. I cut and pasted from two different runs. SORRY! ;-)
>

OK! No problem; thanks for clarifying it!

>> I don't recall observing this while testing (maybe it only happens
>> when using the -m option) ?
>
> No, I get the same effect w/ 'guix package -r foo.

50 seconds for 33 packages on such a powerful machine seems abnormal,
unless you have a specific package(s) which would install an unusually
large amount of manual pages (which would take more time to be indexed).

Ludovic had a way to time the generation of the manual-database
derivation; he was using something like [0]:

--8<---------------cut here---------------start------------->8---
time guix build --check /gnu/store/rkri628apz2a2i2jvav11ylv2736fvv3-manual-database.drv
--8<---------------cut here---------------end--------------->8---

Notice that you need the derivation (.drv) of manual-database rather
than the store item ending by manual-database. Unfortunately I'm not
aware of how we can easily find the derivation of the corresponding
manual-database store item used by the profile (as can be found by: guix
gc -R $(realpath $HOME/.guix-profile) | grep manual-database).

Maybe Ludovic can enlight us?

If you are interested in digging into this further, you could enable
debug output for the man-db command used by manual-database profile hook
defined in (guix profiles) by replacing the "--quiet" option by
--debug. This way you'd see everything man-db might be complaining about.

[0] https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00847.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Performance of the man page database generation

[Ludovic Courtès]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> myglc2 <myglc2@gmail.com> writes:

[...]

>> It is taking more like 50 seconds on my 3.4 Ghz server for 33 packages.
>>
>>> I wonder why the odd package count mismatch (23 vs 22) ?
>>
>> That's my fault. I cut and pasted from two different runs. SORRY! ;-)
>>
>
> OK! No problem; thanks for clarifying it!
>
>>> I don't recall observing this while testing (maybe it only happens
>>> when using the -m option) ?
>>
>> No, I get the same effect w/ 'guix package -r foo.
>
> 50 seconds for 33 packages on such a powerful machine seems abnormal,
> unless you have a specific package(s) which would install an unusually
> large amount of manual pages (which would take more time to be indexed).
>
> Ludovic had a way to time the generation of the manual-database
> derivation; he was using something like [0]:
>
> time guix build --check /gnu/store/rkri628apz2a2i2jvav11ylv2736fvv3-manual-database.drv
>
> Notice that you need the derivation (.drv) of manual-database rather
> than the store item ending by manual-database. Unfortunately I'm not
> aware of how we can easily find the derivation of the corresponding
> manual-database store item used by the profile (as can be found by: guix
> gc -R $(realpath $HOME/.guix-profile) | grep manual-database).
>
> Maybe Ludovic can enlight us?

You can see those .drv names when building the profile.  So you just
need to copy/paste them and run “guix build --check” above.

If you have the profile but not its .drv, you can find out what the .drv
for that profile was but there’s no command-line interface for that (you
have to use ‘valid-derivers’ from (guix store).)

HTH,
Ludo’.

Re: Performance of the man page database generation

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 15901 bytes --]

Hello!

ludo@gnu.org (Ludovic Courtès) writes:

> Hello,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> myglc2 <myglc2@gmail.com> writes:
>
> [...]
>
>>> It is taking more like 50 seconds on my 3.4 Ghz server for 33 packages.
>>>
>>>> I wonder why the odd package count mismatch (23 vs 22) ?
>>>
>>> That's my fault. I cut and pasted from two different runs. SORRY! ;-)
>>>
>>
>> OK! No problem; thanks for clarifying it!
>>
>>>> I don't recall observing this while testing (maybe it only happens
>>>> when using the -m option) ?
>>>
>>> No, I get the same effect w/ 'guix package -r foo.
>>
>> 50 seconds for 33 packages on such a powerful machine seems abnormal,
>> unless you have a specific package(s) which would install an unusually
>> large amount of manual pages (which would take more time to be indexed).
>>
>> Ludovic had a way to time the generation of the manual-database
>> derivation; he was using something like [0]:
>>
>> time guix build --check /gnu/store/rkri628apz2a2i2jvav11ylv2736fvv3-manual-database.drv
>>
>> Notice that you need the derivation (.drv) of manual-database rather
>> than the store item ending by manual-database. Unfortunately I'm not
>> aware of how we can easily find the derivation of the corresponding
>> manual-database store item used by the profile (as can be found by: guix
>> gc -R $(realpath $HOME/.guix-profile) | grep manual-database).
>>
>> Maybe Ludovic can enlight us?
>
> You can see those .drv names when building the profile.  So you just
> need to copy/paste them and run “guix build --check” above.
>
> If you have the profile but not its .drv, you can find out what the .drv
> for that profile was but there’s no command-line interface for that (you
> have to use ‘valid-derivers’ from (guix store).)
>
> HTH,
> Ludo’.

Thanks Ludovic; that was helpful.

I did a small test and it takes about 40 s on my system, with 60
packages or so in my profile. I removed the "--quiet" flag of mandb in
the profile hook (guix profile) to see the output of mandb while it was
indexing the manual pages:

--8<---------------cut here---------------start------------->8---
time guix build --check /gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv
@ build-started /gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv - x86_64-linux /var/log/guix/drvs/vk//10zmd5nm8kppd0f655fraradr019fq-manual-database.drv.bz2
creating manual page database for 62 packages...
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: $MANPATH set, ignoring /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/etc/man_db.conf
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/man7'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection...
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/man3'. Wait...
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_jit_stack_alloc(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_jit_stack_alloc(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_jit_stack_free(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_jit_stack_free(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_study(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_study(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_utf32_to_host_byte_order(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_copy_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_copy_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_dfa_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_dfa_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_utf16_to_host_byte_order(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_get_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_get_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_config(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_config(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: /gnu/store/5h50xmsvxkxj1xjl81vh1shxwprz8gna-pcre-8.40/share/man/man3/pcredemo.3.gz: whatis parse for pcredemo(3) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_free_substring_list(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_free_substring_list(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_version(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_version(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_refcount(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_refcount(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_get_stringnumber(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_get_stringnumber(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_fullinfo(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_fullinfo(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_jit_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_jit_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_get_stringtable_entries(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_get_stringtable_entries(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_free_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_free_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_pattern_to_host_byte_order(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_pattern_to_host_byte_order(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_free_study(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_free_study(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_get_substring_list(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_get_substring_list(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_compile2(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_compile2(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_compile(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_compile(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_maketables(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_maketables(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_assign_jit_stack(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_assign_jit_stack(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_copy_named_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_copy_named_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre32_get_named_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for pcre16_get_named_substring(3)
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/man5'. Wait...
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for gimprc-2.8(5)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: /gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man5/gdbinit.5.gz: whatis parse for gdbinit(5) failed
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/man4'. Wait...
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/man8'. Wait...
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/man1'. Wait...
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for dot(1)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for gimp-console-2.8(1)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: /gnu/store/55qb28n3hwvmyn60hfqwpqyxgd09d0mp-gcc-6.3.0/share/man/man1/g++.1.gz: whatis parse for g++(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: /gnu/store/55qb28n3hwvmyn60hfqwpqyxgd09d0mp-gcc-6.3.0/share/man/man1/gcc.1.gz: whatis parse for gcc(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: /gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gcore.1.gz: whatis parse for gcore(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: /gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gdbserver.1.gz: whatis parse for gdbserver(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: /gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gdb.1.gz: whatis parse for gdb(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: failed to store entry for python3(1)
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/ja/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/ja...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/ru.KOI8-R/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/ru.KOI8-R...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/fr.ISO8859-1/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/fr.ISO8859-1...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/it.ISO8859-1/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/it.ISO8859-1...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/fr/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/fr...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/it/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/it...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/pl.ISO8859-2/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/pl.ISO8859-2...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/pl/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/pl...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/ru.UTF-8/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/ru.UTF-8...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/fr.UTF-8/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/fr.UTF-8...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/it.UTF-8/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/it.UTF-8...
done.
Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpages-collection/pl.UTF-8/man1'. Wait...
Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpages-collection/pl.UTF-8...
done.
18 man subdirectories contained newer manual pages.
3943 manual pages were added.
find-files: /gnu/store/5mk4lg1rdzq2p3148idjd4z0bvif0nf4-gimp-2.8.18/share/man/man1/gimp-console.1: No such file or directory
guix build: error: build failed: derivation `/gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv' may not be deterministic: output `/gnu/store/lmf9rfzj565m31xzhhlam3lk91bjbqsm-manual-database' differs

real    0m39.890s
user    0m0.356s
sys     0m0.040s
--8<---------------cut here---------------end--------------->8---

As you can see, mandb stumbled more than once with some of our packages'
manpages. Those failures are probably attributable to bad packages
manual pages. For example, the gcc manual pages are known to be broken
for example. [0]

By looking at the output as it was generated, it seemed as if those
failures caused mandb to take more time.

Another reason could be that mandb is just inefficient. There's an old
bug about mandb's use of processes being less than optimal ("strace
indicates that mandb is forking on the order of four processes per
page") [1]. It seems that Colin Watson had a good idea of how to improve
things but didn't get around to implementing it yet.

[0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24069
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630799

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Performance of the man page database generation

[myglc2]

On 05/13/2017 at 23:18 Maxim Cournoyer writes:

> Hello!
>
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hello,
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>
>>> myglc2 <myglc2@gmail.com> writes:
>>
[...]
>>> 50 seconds for 33 packages on such a powerful machine seems abnormal,
>>> unless you have a specific package(s) which would install an unusually
>>> large amount of manual pages (which would take more time to be indexed).
>>>
[...]
>>
>> You can see those .drv names when building the profile.  So you just
>> need to copy/paste them and run “guix build --check” above.
>>
>> If you have the profile but not its .drv, you can find out what the .drv
>> for that profile was but there’s no command-line interface for that (you
>> have to use ‘valid-derivers’ from (guix store).)
>>
>> HTH,
>> Ludo’.
>
> Thanks Ludovic; that was helpful.
>
> I did a small test and it takes about 40 s on my system, with 60
> packages or so in my profile. I removed the "--quiet" flag of mandb in
> the profile hook (guix profile) to see the output of mandb while it was
> indexing the manual pages:
>
> time guix build --check /gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv
[...]
> real    0m39.890s
> user    0m0.356s
> sys     0m0.040s
>
[...]

Hi Maxim and Ludo’,

I hacked profiles.scm (please see git diff below) so that start and end
seconds appear in the message like this:

creating manual page database for 23 packages...1494773268...1494773275 DONE

I tested a few cases and man-db typically takes only a few seconds.  It
only _appears_ to take a long time because other processing occurs after
the man-db command.

So, I suggest either ...

1) delete the message altogether, or
2) close the message with a "DONE"

SORRY for the run-around. In my defense, when I first raised the
question I did suggest the idea of closing the message:

http://lists.gnu.org/archive/html/guix-devel/2017-04/msg00196.html

HTH, George

g1@g1 ~/src/guix [env]$ git diff
diff --git a/guix/profiles.scm b/guix/profiles.scm
index eb172ef45..4dbf44a81 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -1011,13 +1011,15 @@ the entries in MANIFEST."
         (mkdir-p man-directory)
         (setenv "MANPATH" (string-join entries ":"))
 
-        (format #t "creating manual page database for ~a packages...~%"
-                (length entries))
+        (format #t "creating manual page database for ~a packages...~a"
+                (length entries)(current-time))
         (force-output)
 
         (zero? (system* #+(file-append man-db "/bin/mandb")
                         "--quiet" "--create"
-                        "-C" "man_db.conf"))))
+                        "-C" "man_db.conf"))
+        (format #t "...~a DONE~%" (current-time))
+        (force-output)))
 
   (gexp->derivation "manual-database" build
                     #:modules '((guix build utils)
g1@g1 ~/src/guix [env]$ 

Re: Performance of the man page database generation

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> 18 man subdirectories contained newer manual pages.
> 3943 manual pages were added.
> find-files: /gnu/store/5mk4lg1rdzq2p3148idjd4z0bvif0nf4-gimp-2.8.18/share/man/man1/gimp-console.1: No such file or directory
> guix build: error: build failed: derivation `/gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv' may not be deterministic: output `/gnu/store/lmf9rfzj565m31xzhhlam3lk91bjbqsm-manual-database' differs
>
> real    0m39.890s
> user    0m0.356s
> sys     0m0.040s

Not good!

> As you can see, mandb stumbled more than once with some of our packages'
> manpages. Those failures are probably attributable to bad packages
> manual pages. For example, the gcc manual pages are known to be broken
> for example. [0]
>
> By looking at the output as it was generated, it seemed as if those
> failures caused mandb to take more time.

That would be surprising no?

> Another reason could be that mandb is just inefficient. There's an old
> bug about mandb's use of processes being less than optimal ("strace
> indicates that mandb is forking on the order of four processes per
> page") [1]. It seems that Colin Watson had a good idea of how to improve
> things but didn't get around to implementing it yet.

Yeah, I noticed that it was forking a lot by looking at the process
graph in gkrellm.  That’s not good.

The database creation is probably not rocket science, so I wonder if we
could implement it more efficiently in Guile using the appropriate gdbm
or bdb bindings.  Certainly we could avoid the forks.

Alternately we optimize mandb itself directly and contribute it
upstream, which would be benefit everyone but might be trickier.

Thoughts?

Ludo’.

Re: Performance of the man page database generation

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 3485 bytes --]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> 18 man subdirectories contained newer manual pages.
>> 3943 manual pages were added.
>> find-files: /gnu/store/5mk4lg1rdzq2p3148idjd4z0bvif0nf4-gimp-2.8.18/share/man/man1/gimp-console.1: No such file or directory
>> guix build: error: build failed: derivation
>> `/gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv'
>> may not be deterministic: output
>> `/gnu/store/lmf9rfzj565m31xzhhlam3lk91bjbqsm-manual-database'
>> differs
>>
>> real    0m39.890s
>> user    0m0.356s
>> sys     0m0.040s
>
> Not good!
>
>> As you can see, mandb stumbled more than once with some of our packages'
>> manpages. Those failures are probably attributable to bad packages
>> manual pages. For example, the gcc manual pages are known to be broken
>> for example. [0]
>>
>> By looking at the output as it was generated, it seemed as if those
>> failures caused mandb to take more time.
>
> That would be surprising no?
>

You're right. The output just tricked me into thinking that. But in the
end it's just the sheer numbers of manpages in my profile which takes
time to get indexed (I have almost 4000). The package which the biggest
collection that I know of is 'libx11', which comes with more than 2000
packages itself!

It can be used as a good test for the manual database profile hook:

--8<---------------cut here---------------start------------->8---
guix environment --ad-hoc libx11
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/1ysz4v6nn4iwpwisgpzb3x92isympxrq-profile.drv
   /gnu/store/4laycbr82asv7k4li1y1fwfgfjfp0qxz-manual-database.drv
substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
Creating manual page database for 5 packages... done in 20.124 s
--8<---------------cut here---------------start------------->8---

>> Another reason could be that mandb is just inefficient. There's an old
>> bug about mandb's use of processes being less than optimal ("strace
>> indicates that mandb is forking on the order of four processes per
>> page") [1]. It seems that Colin Watson had a good idea of how to improve
>> things but didn't get around to implementing it yet.
>
> Yeah, I noticed that it was forking a lot by looking at the process
> graph in gkrellm.  That’s not good.
>
> The database creation is probably not rocket science, so I wonder if we
> could implement it more efficiently in Guile using the appropriate gdbm
> or bdb bindings.  Certainly we could avoid the forks.
>
> Alternately we optimize mandb itself directly and contribute it
> upstream, which would be benefit everyone but might be trickier.
>
> Thoughts?

It'd be nice as a first to check with Colin if he still thinks fixing it
in mandb would be easy. Colin pointed to improving the pipeline with
coroutines or generators [0], which would potentially go in
"libpipeline".

If this stuff is not easy to implement there, then yes, an
implementation in Guile with bindings to gdbm (which is what we are
using currently) would make sense since Guile already comes which
coroutines and generators, IIUC.

Maxim

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630799

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]