Re: Performance of the man page database generation

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3485 bytes --]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> 18 man subdirectories contained newer manual pages.
>> 3943 manual pages were added.
>> find-files: /gnu/store/5mk4lg1rdzq2p3148idjd4z0bvif0nf4-gimp-2.8.18/share/man/man1/gimp-console.1: No such file or directory
>> guix build: error: build failed: derivation
>> `/gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv'
>> may not be deterministic: output
>> `/gnu/store/lmf9rfzj565m31xzhhlam3lk91bjbqsm-manual-database'
>> differs
>>
>> real    0m39.890s
>> user    0m0.356s
>> sys     0m0.040s
>
> Not good!
>
>> As you can see, mandb stumbled more than once with some of our packages'
>> manpages. Those failures are probably attributable to bad packages
>> manual pages. For example, the gcc manual pages are known to be broken
>> for example. [0]
>>
>> By looking at the output as it was generated, it seemed as if those
>> failures caused mandb to take more time.
>
> That would be surprising no?
>

You're right. The output just tricked me into thinking that. But in the
end it's just the sheer numbers of manpages in my profile which takes
time to get indexed (I have almost 4000). The package which the biggest
collection that I know of is 'libx11', which comes with more than 2000
packages itself!

It can be used as a good test for the manual database profile hook:

--8<---------------cut here---------------start------------->8---
guix environment --ad-hoc libx11
substitute: updating list of substitutes from 'https://bayfront.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/1ysz4v6nn4iwpwisgpzb3x92isympxrq-profile.drv
   /gnu/store/4laycbr82asv7k4li1y1fwfgfjfp0qxz-manual-database.drv
substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
Creating manual page database for 5 packages... done in 20.124 s
--8<---------------cut here---------------start------------->8---

>> Another reason could be that mandb is just inefficient. There's an old
>> bug about mandb's use of processes being less than optimal ("strace
>> indicates that mandb is forking on the order of four processes per
>> page") [1]. It seems that Colin Watson had a good idea of how to improve
>> things but didn't get around to implementing it yet.
>
> Yeah, I noticed that it was forking a lot by looking at the process
> graph in gkrellm.  That’s not good.
>
> The database creation is probably not rocket science, so I wonder if we
> could implement it more efficiently in Guile using the appropriate gdbm
> or bdb bindings.  Certainly we could avoid the forks.
>
> Alternately we optimize mandb itself directly and contribute it
> upstream, which would be benefit everyone but might be trickier.
>
> Thoughts?

It'd be nice as a first to check with Colin if he still thinks fixing it
in mandb would be easy. Colin pointed to improving the pipeline with
coroutines or generators [0], which would potentially go in
"libpipeline".

If this stuff is not easy to implement there, then yes, an
implementation in Guile with bindings to gdbm (which is what we are
using currently) would make sense since Guile already comes which
coroutines and generators, IIUC.

Maxim

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630799

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]