From mboxrd@z Thu Jan 1 00:00:00 1970 From: Maxim Cournoyer Subject: Re: Performance of the man page database generation Date: Tue, 16 May 2017 10:24:45 -0700 Message-ID: <87tw4kzpcy.fsf@gmail.com> References: <20170405205638.12336.78854@vcs0.savannah.gnu.org> <20170405205640.15AE6210A4@vcs0.savannah.gnu.org> <86y3v9bicq.fsf@gmail.com> <864lxxghmm.fsf@gmail.com> <87a87ibjq1.fsf@gmail.com> <871ssn4ebg.fsf_-_@gnu.org> <87fug8j704.fsf@gmail.com> <87tw4lb4am.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:44761) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dAgDN-0000Kk-C9 for guix-devel@gnu.org; Tue, 16 May 2017 13:24:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dAgDL-0001NK-Ux for guix-devel@gnu.org; Tue, 16 May 2017 13:24:53 -0400 In-Reply-To: <87tw4lb4am.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 16 May 2017 10:21:21 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel , myglc2 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Ludovic, ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Hi Maxim, > > Maxim Cournoyer skribis: > >> 18 man subdirectories contained newer manual pages. >> 3943 manual pages were added. >> find-files: /gnu/store/5mk4lg1rdzq2p3148idjd4z0bvif0nf4-gimp-2.8.18/shar= e/man/man1/gimp-console.1: No such file or directory >> guix build: error: build failed: derivation >> `/gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv' >> may not be deterministic: output >> `/gnu/store/lmf9rfzj565m31xzhhlam3lk91bjbqsm-manual-database' >> differs >> >> real 0m39.890s >> user 0m0.356s >> sys 0m0.040s > > Not good! > >> As you can see, mandb stumbled more than once with some of our packages' >> manpages. Those failures are probably attributable to bad packages >> manual pages. For example, the gcc manual pages are known to be broken >> for example. [0] >> >> By looking at the output as it was generated, it seemed as if those >> failures caused mandb to take more time. > > That would be surprising no? > You're right. The output just tricked me into thinking that. But in the end it's just the sheer numbers of manpages in my profile which takes time to get indexed (I have almost 4000). The package which the biggest collection that I know of is 'libx11', which comes with more than 2000 packages itself! It can be used as a good test for the manual database profile hook: =2D-8<---------------cut here---------------start------------->8--- guix environment --ad-hoc libx11 substitute: updating list of substitutes from 'https://bayfront.guixsd.org'= ... 100.0% substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org= '... 100.0% The following derivations will be built: /gnu/store/1ysz4v6nn4iwpwisgpzb3x92isympxrq-profile.drv /gnu/store/4laycbr82asv7k4li1y1fwfgfjfp0qxz-manual-database.drv substitute: updating list of substitutes from 'https://mirror.hydra.gnu.org= '... 100.0% Creating manual page database for 5 packages... done in 20.124 s =2D-8<---------------cut here---------------start------------->8--- >> Another reason could be that mandb is just inefficient. There's an old >> bug about mandb's use of processes being less than optimal ("strace >> indicates that mandb is forking on the order of four processes per >> page") [1]. It seems that Colin Watson had a good idea of how to improve >> things but didn't get around to implementing it yet. > > Yeah, I noticed that it was forking a lot by looking at the process > graph in gkrellm. That=E2=80=99s not good. > > The database creation is probably not rocket science, so I wonder if we > could implement it more efficiently in Guile using the appropriate gdbm > or bdb bindings. Certainly we could avoid the forks. > > Alternately we optimize mandb itself directly and contribute it > upstream, which would be benefit everyone but might be trickier. > > Thoughts? It'd be nice as a first to check with Colin if he still thinks fixing it in mandb would be easy. Colin pointed to improving the pipeline with coroutines or generators [0], which would potentially go in "libpipeline". If this stuff is not easy to implement there, then yes, an implementation in Guile with bindings to gdbm (which is what we are using currently) would make sense since Guile already comes which coroutines and generators, IIUC. Maxim [0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D630799 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEFVwTvgZwWPWnh370K8v/MDHLu9sFAlkbNd0ACgkQK8v/MDHL u9t8rg/+OE4bO08pwld8pYl5TjWPVRd0Q5Zk4+Ct2stpjmmhAU0TnkJId0SxE23K rjGtByExy+QBgrAzvgULZhD1Sg+hPo/nwmrdSfDqETkN6zc8Dq3S036+ITYZwb9h XgIfTsQ8CvZrSyYb3mpE/HP+BmqlcHiOeaC+MnFXDGMoYlWeHSc/yLtoEUbBRxzd 7jMoZ9dDdxurDotVEfSgvSP9y83kyCyQ+Bp0CQZVlFvVNg2n5eLdMkRvcQA2fqmX zmQVPweWDFeIzoP8h/aUCOrhW+7JEDw35SVPjlRveWSNgvcnZI1p7aW7ZqM4A3En ihcNU/QQsBFSAJafqYuDyktLMU3G11BITxKvGYtuyMi2pcnpis+Ov6QqeDgdNvrm w7RVm2s+T5MjdKf46gLVQLrBvQVpg4y7cxSVBfhwe+3j3A4WgVJbu3n6i5+ecU/L BO8Cc+QTcMQt5BhHCD0tUKUZPiFT5SYlsiOb3qKyrCSjxCoaxbQ0NFxJY7UJfTOw lg3qPgJIaUtKkROan0vEDxDEEIgyouqFfmD4flt0ayBCV0eM3J5NyRAyh+PL1BXw pfxBmp2DfpTfRLsU2cmr1SkIBhBdxpv9cUqFHpYdGWzLSJaW7PLATlCH3tnIEP1X /6F5WcQ2KsfPuPYzu+TYWtWW98Rwz27XaFRXqRhUAOnd6VbEpQI= =efqM -----END PGP SIGNATURE----- --=-=-=--