bug#44053: ‘xdg-mime-database’ profile hook is slow

bug#44053: ‘xdg-mime-database’ profile hook is slow

[Joshua Branson via Bug reports for GNU Guix]

I am just posting my performance numbers.  Hopefully you all find it helpful.
I am running Guix System on a Lenovo Thinkpad T400 with 8GB on a hard drive.

#+BEGIN_SRC sh :results output :exports both
guix describe
#+END_SRC

#+RESULTS:
#+begin_example
Generation 49	Oct 20 2020 01:15:33	(current)
  my-personal-packages 547d900
    repository URL: https://notabug.org/jbranso/guix-packages.git
    branch: master
    commit: 547d900c25d2bb9490946162e0fcb315d6bace68
  guix 93c2518
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 93c251818dcd2bf5f60f42e595504293ee83371a

GUIX_PACKAGE_PATH="/home/joshua/prog/gnu/guix/guix-packages/"
#+end_example

The "XDG MIME database" takes a while.

#+BEGIN_SRC sh :results output :exports both
time guix build --check $(guix gc -R $(guix gc --derivers $(readlink -f ~/.guix-profile)) |grep xdg-mime-database.drv)
#+END_SRC

#+RESULTS:
: The following profile hook will be built:
:    /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
: building XDG MIME database...
: successfully built /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
: successfully built /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
: /gnu/store/x8q8g9l0jhrpmjjm3xsh3ib1z8l79cyx-xdg-mime-database
:
: real	0m43.716s
: user	0m3.626s
: sys	0m0.258s

Cold cache:

#+BEGIN_SRC sh :results output :exports both
drv="$(guix gc -R $(guix gc --derivers $(readlink -f ~/.guix-profile)) |grep xdg-mime-database.drv)"
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

time guix build --check "$drv"
#+END_SRC

#+RESULTS:
: The following profile hook will be built:
:    /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
: building XDG MIME database...
: successfully built /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
: successfully built /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
: /gnu/store/x8q8g9l0jhrpmjjm3xsh3ib1z8l79cyx-xdg-mime-database
:
: real	1m29.606s
: user	0m1.401s
: sys	0m0.238s

I have 64 packages in my profile.

#+BEGIN_SRC sh :results output :exports both
guix package -I | wc --lines
#+END_SRC

#+RESULTS:
: 64

If you want to see all packages:

#+BEGIN_SRC sh :results output :exports both
guix package -I
#+END_SRC

#+RESULTS:
#+begin_example
guile-colorized	0.1	out	/gnu/store/5h0jfssgyx49cq82wrmw57an7ngb7kkz-guile-colorized-0.1
font-hack	3.003	out	/gnu/store/52r8anazd4rnkq9m3vxk700jga5h0i74-font-hack-3.003
font-dejavu	2.37	out	/gnu/store/7y3lvk3xf4im8n44337mc6y0ccysvfia-font-dejavu-2.37
font-adobe-source-code-pro	2.030R-ro-1.050R-it	out	/gnu/store/l5m74158njcan3p784gqhkir6zyk0bcp-font-adobe-source-code-pro-2.030R-ro-1.050R-it
sicp	20170703-1.225c172	out	/gnu/store/x2safq6fvcbhp2rljny89qiq74y1xd13-sicp-20170703-1.225c172
unzip	6.0	out	/gnu/store/1zx8knb7h79q6cr4kwhbz6z6v609309w-unzip-6.0
font-google-noto	20171025	out	/gnu/store/g2szydnbvs7qqy2nf7qylba0rapajmd8-font-google-noto-20171025
font-awesome	4.7.0	out	/gnu/store/gknn01i1nr80apmcx42w36cqcz3zzri2-font-awesome-4.7.0
texinfo	6.7	out	/gnu/store/w8k9hcigvhzrlrblv8lgqj77sm3833rs-texinfo-6.7
gcc-toolchain	10.2.0	out	/gnu/store/pknm43xsza6nlc7bn27djip8fis92akd-gcc-toolchain-10.2.0
recutils	1.8	out	/gnu/store/qlx179h9mfdv4zywx2ias2jmkygvs834-recutils-1.8
guile-readline	3.0.2	out	/gnu/store/va6l1ivclww22fi38w5h99pb4ndn99hg-guile-readline-3.0.2
gtypist	2.9.5	out	/gnu/store/03wd707lyz49f2r14cm9ybyzh7vfvi0j-gtypist-2.9.5
scsh	0.0.0-1.1144324	out	/gnu/store/57frd08q6gyd7ysq3aqwlxxmify3w53h-scsh-0.0.0-1.1144324
make	4.3	out	/gnu/store/4k33n2nhsnnaxk2ip75gj7xiqdjns5hq-make-4.3
isync	1.3.3	out	/gnu/store/745qmx9mmnwq0j3zrd50qcdd2bgv5clz-isync-1.3.3
jmacs	27.1	out	/gnu/store/0hqdddkvhi2acxjdssdi5as3wj6cgvnz-jmacs-27.1
glibc-locales	2.31	out	/gnu/store/x9c4dk4zc9mi5j9hm3lq1q539kkdhkln-glibc-locales-2.31
guile	3.0.4	out	/gnu/store/ah16zr8mmfkqy23rr7jy5a842ca1q9h1-guile-3.0.4
guile-json	4.3.2	out	/gnu/store/12y1s0p3585ydf336gq85kjh5217pnzp-guile-json-4.3.2
guile-syntax-highlight	0.1	out	/gnu/store/dwjqscckpypqfq38hc97x5ipc9fplpp9-guile-syntax-highlight-0.1
haunt	0.2.4	out	/gnu/store/ccvqa9a6yclgkpi9izhpgf1vgjaghqvr-haunt-0.2.4
gnutls	3.6.14	out	/gnu/store/0f6agk6xk8rcz99907d39nd1vzmc4abq-gnutls-3.6.14
wget	1.20.3	out	/gnu/store/d7b16jqrgb1jz65hl7vnz261dbxmjh3s-wget-1.20.3
gnupg	2.2.23	out	/gnu/store/89cjqnar4aacg58wl6cm1h41rym9mv4l-gnupg-2.2.23
sassc	3.6.1	out	/gnu/store/6pqn2i0r12ypss13mdvsypskkcpaagji-sassc-3.6.1
qterminal	0.15.0	out	/gnu/store/y7qrfa78y785xfajb3gvvhwcm2d02l81-qterminal-0.15.0
dmenu	5.0	out	/gnu/store/393frjc5xjaab6nvp3wxpajb1rc6gynr-dmenu-5.0
openssh	8.4p1	out	/gnu/store/k8lnv9cqk6m15k18hhzg7w696r1vxx1v-openssh-8.4p1
grim	1.3.1	out	/gnu/store/9v5i9q7kwcmq1q31qri0icjkhh3csh9r-grim-1.3.1
evince	3.36.5	out	/gnu/store/arwvpbrzsch4w6afkpqln6625dfl0bz8-evince-3.36.5
pulseaudio	13.0	out	/gnu/store/19wcjfwdr3hzq1a4wcpld1zdrjichck0-pulseaudio-13.0
libreoffice	6.4.6.2	out	/gnu/store/7rbk52nzqmakd5whds5a6f07xrd2k870-libreoffice-6.4.6.2
termite	15	out	/gnu/store/8lqfy0mxqnq3xg24gwsxvzpxnvv6firh-termite-15
mpv	0.32.0	out	/gnu/store/2i2gzn7fa8qq0f1jkqhiyl0xw63fjr4w-mpv-0.32.0
pinentry	1.1.0	out	/gnu/store/bdsk9yq48lx1f7y7xr9sv4yrmm4ync9b-pinentry-1.1.0
glib	2.62.6	doc	/gnu/store/ql2226knbh4q04bbdiqcppvs5rjyl3ga-glib-2.62.6-doc
pidgin	2.13.0	out	/gnu/store/fd06njmc5maid7b474v224iazy7bc20h-pidgin-2.13.0
gimp	2.10.20	out	/gnu/store/m397hvwhsd0l6v5d0vqxxkm4iyp0p9w3-gimp-2.10.20
sway	1.4	out	/gnu/store/27ggd92in0hpgmkgk79n2iy4angjxwvl-sway-1.4
wf-recorder	0.2.1	out	/gnu/store/glc6y75l4czpfyrgbrbgxg38ig004v1p-wf-recorder-0.2.1
wofi	1.1.2	out	/gnu/store/qaylx0k6wg3awsamn2rqr41q9c8iwypg-wofi-1.1.2
linphoneqt	4.1.1	out	/gnu/store/wlcviacpagxqnw1737x131if0fj7kzjv-linphoneqt-4.1.1
qbittorrent	4.2.5	out	/gnu/store/29cjbdmz809pdvmylbx307l2p7zv0ma9-qbittorrent-4.2.5
pavucontrol	4.0	out	/gnu/store/0ilm4lncr6gpg4h3zdvaxg1pibkm6934-pavucontrol-4.0
git	2.28.0	send-email	/gnu/store/g5x559qk5yxs19jcxqplqv8b9y32vak2-git-2.28.0-send-email
flatpak	1.8.2	out	/gnu/store/ywfsnccvlr0j6afb5vnl3z4ismvcfbz6-flatpak-1.8.2
xdg-desktop-portal	1.7.2	out	/gnu/store/jkjx5dd7s3dp58p2hwsm8n2rm15z90ys-xdg-desktop-portal-1.7.2
xdg-desktop-portal-gtk	1.7.1	out	/gnu/store/zhssisyp66vggjnqi5y8nb1xnk55vnw1-xdg-desktop-portal-gtk-1.7.1
xeyes	1.1.2	out	/gnu/store/4jx8i19xd94d421n4acy2c1p0nn9h6d0-xeyes-1.1.2
emacs-polymode	0.2	out	/gnu/store/gmm1y1xahg44yw9ghsf0kwd40w86q7mf-emacs-polymode-0.2
notmuch	0.31	out	/gnu/store/0js5dwj8nxj0x0hxfllgzz4zxlsmyrz4-notmuch-0.31
msmtp	1.8.12	out	/gnu/store/s1g4nskvnsx9ww0032xg23548h7mnwvq-msmtp-1.8.12
youtube-viewer	3.7.9	out	/gnu/store/jm7q53lpda3a4rs2z7xv7cgig7adpncb-youtube-viewer-3.7.9
guile-curl	0.6	out	/gnu/store/9nnclxjxakwx9yc381arqrk1nhld21b5-guile-curl-0.6
qemu	5.1.0	out	/gnu/store/qzi36f60d2zlrdj9mg6b5y4i00w1fzfq-qemu-5.1.0
icecat	78.3.1-guix0-preview1	out	/gnu/store/71fkykdajhkshsvzwrnqm4lsc6qgycg2-icecat-78.3.1-guix0-preview1
git	2.28.0	out	/gnu/store/j66k3x1wz0bc6nh09dwa6i4jwahqcabi-git-2.28.0
guix	1.1.0-29.4e3ed9b	out	/gnu/store/irirj169rncank4y7i6n8qyiiqbs76qr-guix-1.1.0-29.4e3ed9b
ffmpeg	4.3.1	out	/gnu/store/sg4yq9i0v1lphvs1vx33dw9mn801adrd-ffmpeg-4.3.1
pavucontrol-qt	0.15.0	out	/gnu/store/9d0zp4wps5kpbqsl4yvjb96spb5xwz9b-pavucontrol-qt-0.15.0
clementine	1.3.1-2.4619a4c	out	/gnu/store/5392fysgmmi2jmn98maz9ihzn9ahzhw9-clementine-1.3.1-2.4619a4c
mako	1.4	out	/gnu/store/k83v9wz6szf6k2p2sz1amvwqm687j3h2-mako-1.4
ungoogled-chromium	86.0.4240.75-0.c34a56d	out	/gnu/store/m0s9mrbirbssis5nn9agx8k4sl2ck0y9-ungoogled-chromium-86.0.4240.75-0.c34a56d
#+end_example


Thanks,

--
Joshua Branson
Sent from Emacs and Gnus
https://gnucode.me
https://video.hardlimit.com/accounts/joshua_branson/video-channels
"You can have whatever you want, as long as you help enough other people get what they want." - Zig Ziglar

bug#44053: ‘xdg-mime-database’ profile hook is slow

[Ludovic Courtès]

Hi,

Joshua Branson <jbranso@dismail.de> skribis:

> The "XDG MIME database" takes a while.
>
> #+BEGIN_SRC sh :results output :exports both
> time guix build --check $(guix gc -R $(guix gc --derivers $(readlink -f ~/.guix-profile)) |grep xdg-mime-database.drv)
> #+END_SRC
>
>
> #+RESULTS:
> : The following profile hook will be built:
> :    /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
> : building XDG MIME database...
> : successfully built /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
> : successfully built /gnu/store/lmhklgdscbfp5c6gl81skyz0azfg156m-xdg-mime-database.drv
> : /gnu/store/x8q8g9l0jhrpmjjm3xsh3ib1z8l79cyx-xdg-mime-database
> :
> : real	0m43.716s
> : user	0m3.626s
> : sys	0m0.258s

I found that the MIME database is computed only over the subset of the
packages in your profile that provide ‘share/mime/packages’, plus
‘shared-mime-info’.

In my profile, only LibreOffice provides that directory, so the union is
made over these two directories and that’s what ‘update-mime-database’
works on.

Apart from LibreOffice, I found that ‘share/mime/packages’ is provided
by at least: hugin, gcr, fontforge.  Most GUI packages don’t have it.
So in practice, we’re often rebuilding the exact same database.

Here’s the time taken by ‘update-mime-database’ alone:

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~$ mkdir -p /tmp/mime/share/mime/packages
ludo@ribbon ~$ cd /tmp/mime/share/mime/packages
ludo@ribbon /tmp/mime/share/mime/packages$ for i in $(guix build libreoffice ^C
ludo@ribbon /tmp/mime/share/mime/packages$ for i in ~/.guix-profile/share/mime/packages/* ; do ln -s $i ; done
ludo@ribbon /tmp/mime/share/mime/packages$ ls -l
totalo 8
lrwxrwxrwx 1 ludo users 64 Nov  2 11:46 freedesktop.org.xml -> /home/ludo/.guix-profile/share/mime/packages/freedesktop.org.xml
lrwxrwxrwx 1 ludo users 60 Nov  2 11:46 libreoffice.xml -> /home/ludo/.guix-profile/share/mime/packages/libreoffice.xml
ludo@ribbon /tmp/mime/share/mime/packages$ cd /tmp/mime/
ludo@ribbon /tmp/mime$ time update-mime-database -V /tmp/mime/share/mime
Updating MIME database in /tmp/mime/share/mime...

Parsing source file /tmp/mime/share/mime/packages/freedesktop.org.xml...
Parsing source file /tmp/mime/share/mime/packages/libreoffice.xml...
Wrote 1124 strings at 2c - 6310
Wrote aliases at 6310 - 6be4
Wrote parents at 6be4 - 8468
Wrote literal globs at 8468 - 855c
Wrote suffix globs at 855c - 13514
Wrote full globs at 13514 - 13554
Wrote magic at 13554 - 2065c
Wrote namespace list at 2065c - 20798
Wrote icons list at 20798 - 2079c
Wrote generic icons list at 2079c - 213b8
Wrote types list at 213b8 - 22020

Note that '/tmp/mime/share' is not in the search path
set by the XDG_DATA_HOME and XDG_DATA_DIRS
environment variables, so applications may not
be able to find it until you set them. The
directories currently searched are:

- /home/ludo/.local/share
- /home/ludo/.guix-profile/share
- /run/current-system/profile/share
- /home/ludo/.guix-profile/share
- /run/current-system/profile/share


real    0m2.166s
user    0m0.278s
sys     0m0.056s
--8<---------------cut here---------------end--------------->8---

To be compared with:

--8<---------------cut here---------------start------------->8---
$ drv="$(guix gc -R $(guix gc --derivers $(readlink -f ~/.guix-profile)) |grep xdg-mime-database.drv)"
$ time guix build --check $drv
The following profile hook will be built:
   /gnu/store/jrmwsxpz3wmq37zx29lvb0r9nvcmdviz-xdg-mime-database.drv
building XDG MIME database...
successfully built /gnu/store/jrmwsxpz3wmq37zx29lvb0r9nvcmdviz-xdg-mime-database.drv
/gnu/store/npvj2sr9kxx48znh7zc8zmzwzs6brc90-xdg-mime-database

real    0m3.670s
user    0m1.428s
sys     0m0.029s
--8<---------------cut here---------------end--------------->8---

Thus, the build time itself is entirely taken by ‘update-mime-database’
(the 1.4s above is “overhead” in ‘guix build’ it would seem).

Ludo’.

bug#44053: ‘xdg-mime-database’ profile hook is slow

[Ludovic Courtès]

Hi,

Luis Felipe <luis.felipe.la@protonmail.com> skribis:

> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Tuesday, November 3, 2020 11:32 PM, zimoun <zimon.toutoune@gmail.com> wrote:
>
>> Hi,
>>
>> > https://git.savannah.gnu.org/cgit/guix.git/commit/?id=76ea70bd70aeb76570445c11cea2f98139192b54
>>
>> Clever workaround! What are now the performances on previous examples
>> (same profiles and same packages)?
>
> In my case there seem to be no improvement (using Guix 5e7cf66fb35780f930ad0bc5fe21ac330df4411d).

Please note that the change above addresses only one specific source of
slowness, the ‘xdg-mime-database’ hook, and only in specific cases.

It’s good to look at the overall timing of ‘guix install’, because
that’s what matters in the end, but as we work on optimizing it, we have
to look at specific aspects of it.

> $ time guix package -i perl --max-jobs=1

[...]

> injertando 12 paquetes en /gnu/store/anknpdyhmfirw3rz2k9zm9kiyak8yy1s-cups-filters-1.27.4.drv ...
> construyendo la base de datos MIME XDG...
> injertando 3 paquetes en /gnu/store/xgny7xbl635g8na8x03x4cdr7abiphiw-cups-2.3.3.drv ...
> injertando 20 paquetes en /gnu/store/yhjl68x7kcjbv40v823x4hl8rvv8l50b-gtk+-2.24.32.drv ...
> injertando 21 paquetes en /gnu/store/kq37fnw8335f1hqc3j4hhqqcdnhl371p-gtk+-3.24.20.drv ...
> creando la caché de temas de iconos de GTK+...
> construyendo los ficheros de caché para los métodos de entrada de GTK+...
> construyendo perfil con 86 paquetes...
>
>
> real	8m38,121s
> user	0m2,742s
> sys	0m0,338s

Here it’s likely that grafting is what’s taking the most time on a
spinning disk.

We should hack (guix status) so it can optionally prefix each event with
a timestamp.

As far as ‘xdg-mime-database’ is concerned, it should be down to 0s,
unless your profile contains one of the packages I cited (libreoffice,
gcr, hugin, etc.).

Thanks,
Ludo’.