From mboxrd@z Thu Jan 1 00:00:00 1970 From: Maxim Cournoyer Subject: Re: Performance of the man page database generation Date: Sat, 13 May 2017 23:18:51 -0700 Message-ID: <87fug8j704.fsf@gmail.com> References: <20170405205638.12336.78854@vcs0.savannah.gnu.org> <20170405205640.15AE6210A4@vcs0.savannah.gnu.org> <86y3v9bicq.fsf@gmail.com> <864lxxghmm.fsf@gmail.com> <87a87ibjq1.fsf@gmail.com> <871ssn4ebg.fsf_-_@gnu.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:56416) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d9mrs-0001Sx-R0 for guix-devel@gnu.org; Sun, 14 May 2017 02:19:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d9mrq-00025H-HP for guix-devel@gnu.org; Sun, 14 May 2017 02:19:00 -0400 In-Reply-To: <871ssn4ebg.fsf_-_@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\?\= \=\?utf-8\?Q\?\=22's\?\= message of "Thu, 20 Apr 2017 11:29:39 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel , myglc2 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello! ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Hello, > > Maxim Cournoyer skribis: > >> myglc2 writes: > > [...] > >>> It is taking more like 50 seconds on my 3.4 Ghz server for 33 packages. >>> >>>> I wonder why the odd package count mismatch (23 vs 22) ? >>> >>> That's my fault. I cut and pasted from two different runs. SORRY! ;-) >>> >> >> OK! No problem; thanks for clarifying it! >> >>>> I don't recall observing this while testing (maybe it only happens >>>> when using the -m option) ? >>> >>> No, I get the same effect w/ 'guix package -r foo. >> >> 50 seconds for 33 packages on such a powerful machine seems abnormal, >> unless you have a specific package(s) which would install an unusually >> large amount of manual pages (which would take more time to be indexed). >> >> Ludovic had a way to time the generation of the manual-database >> derivation; he was using something like [0]: >> >> time guix build --check /gnu/store/rkri628apz2a2i2jvav11ylv2736fvv3-manu= al-database.drv >> >> Notice that you need the derivation (.drv) of manual-database rather >> than the store item ending by manual-database. Unfortunately I'm not >> aware of how we can easily find the derivation of the corresponding >> manual-database store item used by the profile (as can be found by: guix >> gc -R $(realpath $HOME/.guix-profile) | grep manual-database). >> >> Maybe Ludovic can enlight us? > > You can see those .drv names when building the profile. So you just > need to copy/paste them and run =E2=80=9Cguix build --check=E2=80=9D abov= e. > > If you have the profile but not its .drv, you can find out what the .drv > for that profile was but there=E2=80=99s no command-line interface for th= at (you > have to use =E2=80=98valid-derivers=E2=80=99 from (guix store).) > > HTH, > Ludo=E2=80=99. Thanks Ludovic; that was helpful. I did a small test and it takes about 40 s on my system, with 60 packages or so in my profile. I removed the "--quiet" flag of mandb in the profile hook (guix profile) to see the output of mandb while it was indexing the manual pages: =2D-8<---------------cut here---------------start------------->8--- time guix build --check /gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-= database.drv @ build-started /gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database= .drv - x86_64-linux /var/log/guix/drvs/vk//10zmd5nm8kppd0f655fraradr019fq-m= anual-database.drv.bz2 creating manual page database for 62 packages... /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : $MANPATH set, ignoring /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db= -2.7.5/etc/man_db.conf Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/man7'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection... Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/man3'. Wait... /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_jit_stack_alloc(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_jit_stack_alloc(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_jit_stack_free(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_jit_stack_free(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_study(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_study(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_utf32_to_host_byte_order(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_copy_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_copy_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_dfa_exec(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_dfa_exec(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_utf16_to_host_byte_order(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_get_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_get_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_config(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_config(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : /gnu/store/5h50xmsvxkxj1xjl81vh1shxwprz8gna-pcre-8.40/share/man/man3/pcre= demo.3.gz: whatis parse for pcredemo(3) failed /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_free_substring_list(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_free_substring_list(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_version(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_version(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_refcount(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_refcount(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_get_stringnumber(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_get_stringnumber(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_fullinfo(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_fullinfo(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_jit_exec(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_jit_exec(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_get_stringtable_entries(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_get_stringtable_entries(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_free_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_free_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_pattern_to_host_byte_order(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_pattern_to_host_byte_order(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_free_study(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_free_study(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_get_substring_list(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_get_substring_list(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_compile2(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_compile2(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_exec(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_exec(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_compile(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_compile(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_maketables(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_maketables(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_assign_jit_stack(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_assign_jit_stack(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_copy_named_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_copy_named_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre32_get_named_substring(3) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for pcre16_get_named_substring(3) Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/man5'. Wait... /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for gimprc-2.8(5) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : /gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man5/gdb= init.5.gz: whatis parse for gdbinit(5) failed Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/man4'. Wait... Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/man8'. Wait... Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/man1'. Wait... /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for dot(1) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for gimp-console-2.8(1) /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : /gnu/store/55qb28n3hwvmyn60hfqwpqyxgd09d0mp-gcc-6.3.0/share/man/man1/g++.= 1.gz: whatis parse for g++(1) failed /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : /gnu/store/55qb28n3hwvmyn60hfqwpqyxgd09d0mp-gcc-6.3.0/share/man/man1/gcc.= 1.gz: whatis parse for gcc(1) failed /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : /gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gco= re.1.gz: whatis parse for gcore(1) failed /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : /gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gdb= server.1.gz: whatis parse for gdbserver(1) failed /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : /gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gdb= .1.gz: whatis parse for gdb(1) failed /gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning= : failed to store entry for python3(1) done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/ja/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/ja... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/ru.KOI8-R/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/ru.KOI8-R... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/fr.ISO8859-1/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/fr.ISO8859-1... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/it.ISO8859-1/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/it.ISO8859-1... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/fr/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/fr... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/it/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/it... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/pl.ISO8859-2/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/pl.ISO8859-2... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/pl/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/pl... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/ru.UTF-8/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/ru.UTF-8... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/fr.UTF-8/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/fr.UTF-8... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/it.UTF-8/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/it.UTF-8... done. Updating index cache for path `/tmp/guix-build-manual-database.drv-0/manpag= es-collection/pl.UTF-8/man1'. Wait... Processing manual pages under /tmp/guix-build-manual-database.drv-0/manpage= s-collection/pl.UTF-8... done. 18 man subdirectories contained newer manual pages. 3943 manual pages were added. find-files: /gnu/store/5mk4lg1rdzq2p3148idjd4z0bvif0nf4-gimp-2.8.18/share/m= an/man1/gimp-console.1: No such file or directory guix build: error: build failed: derivation `/gnu/store/vk10zmd5nm8kppd0f65= 5fraradr019fq-manual-database.drv' may not be deterministic: output `/gnu/s= tore/lmf9rfzj565m31xzhhlam3lk91bjbqsm-manual-database' differs real 0m39.890s user 0m0.356s sys 0m0.040s =2D-8<---------------cut here---------------end--------------->8--- As you can see, mandb stumbled more than once with some of our packages' manpages. Those failures are probably attributable to bad packages manual pages. For example, the gcc manual pages are known to be broken for example. [0] By looking at the output as it was generated, it seemed as if those failures caused mandb to take more time. Another reason could be that mandb is just inefficient. There's an old bug about mandb's use of processes being less than optimal ("strace indicates that mandb is forking on the order of four processes per page") [1]. It seems that Colin Watson had a good idea of how to improve things but didn't get around to implementing it yet. [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D24069 [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D630799 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEFVwTvgZwWPWnh370K8v/MDHLu9sFAlkX9ssACgkQK8v/MDHL u9vbLA/7BDeFtmyBZc7jxBnjUFuNmHwH+6bmDIwp6z4xmXvUctjgPc1DIwUYiUhd Wvrp+NRmke0zuEP45aumbK3UOZb32VkY7J+qTMJYwFh0dHR9Z6+09qvK45tjgwr9 8GrxKdo79spBGIGf9ECC8IBoBoSpkkiCN0kdTGR4aU/mA7N3hmke9S6OchuZsKgG pcAnC9caUKvNiBCgynYN5RVjzKJ6eSamfK9t+DE5+JhQ1zjlegXuCMpynoLSdfN0 DyRAMXzJMJHukIcEABWWdGXgrFfIZ4SXJErog0PwpHHXV8G/7YkXaQYRV9UzW+QC 8P8FBazmXONCrmKRkmorlfWx+FCYf3ivYlruAQxmyKjy51VvJn34JXwB1Xqc3XEo P/7xw1vQqIEbjiAIvb9uZWpTRv4ItCoft1pdP1E+Ho7aEI5B1MuwhLIagVej1ECw t8cN2Xr03CYz05MEZ8l+1TC1pepOwB9N3YwNQd9Ox1knz1yunaMftuwr9COzSTEj aQ55d3OXy//0QbJTS8FN3auUaeRO7tcPRx3ufgp9YRBS2SMBFS1sOL4/cykwhqbv uCeTR4K/p2BixARrt0kca+8EklxgEI2oKNUwidhpINiZfbgLZVHuyxC3zgRt3zDQ ZDjGGT5SawEvVHRMtWNC6J2VYiYrs0ndbKlgPu8X0oSp1mpPRxY= =Krto -----END PGP SIGNATURE----- --=-=-=--