Re: Performance of the man page database generation

[ludo@gnu.org (Ludovic Courtès)]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> 18 man subdirectories contained newer manual pages.
> 3943 manual pages were added.
> find-files: /gnu/store/5mk4lg1rdzq2p3148idjd4z0bvif0nf4-gimp-2.8.18/share/man/man1/gimp-console.1: No such file or directory
> guix build: error: build failed: derivation `/gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv' may not be deterministic: output `/gnu/store/lmf9rfzj565m31xzhhlam3lk91bjbqsm-manual-database' differs
>
> real    0m39.890s
> user    0m0.356s
> sys     0m0.040s

Not good!

> As you can see, mandb stumbled more than once with some of our packages'
> manpages. Those failures are probably attributable to bad packages
> manual pages. For example, the gcc manual pages are known to be broken
> for example. [0]
>
> By looking at the output as it was generated, it seemed as if those
> failures caused mandb to take more time.

That would be surprising no?

> Another reason could be that mandb is just inefficient. There's an old
> bug about mandb's use of processes being less than optimal ("strace
> indicates that mandb is forking on the order of four processes per
> page") [1]. It seems that Colin Watson had a good idea of how to improve
> things but didn't get around to implementing it yet.

Yeah, I noticed that it was forking a lot by looking at the process
graph in gkrellm.  That’s not good.

The database creation is probably not rocket science, so I wonder if we
could implement it more efficiently in Guile using the appropriate gdbm
or bdb bindings.  Certainly we could avoid the forks.

Alternately we optimize mandb itself directly and contribute it
upstream, which would be benefit everyone but might be trickier.

Thoughts?

Ludo’.