unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / Atom feed
* bug#44053: Poor profile generation performance on spinning disks
@ 2020-10-18  3:35 Maxim Cournoyer
  2020-10-18 23:05 ` Luis Felipe via Bug reports for GNU Guix
  2020-10-19  8:18 ` zimoun
  0 siblings, 2 replies; 4+ messages in thread
From: Maxim Cournoyer @ 2020-10-18  3:35 UTC (permalink / raw)
  To: 44053

Hello!

I've noticed on multiple occasions that using Guix on traditional
spinning drives can be quite slow.

On my home machine, will is still relying on 2 x 1 TB spinning drives in
RAID1, rebuilding my user profile, which contains 182 entries, takes on
average about 20 minutes, even when there are no packages to be built:

--8<---------------cut here---------------start------------->8--- $ time
guix package -i perl --max-jobs=1 The following package will be
upgraded: perl (dependencies or package changed)

The following derivation will be built:
   /gnu/store/lhywla1z2zcz16df4hbvvvngr9zmswr7-profile.drv

building CA certificate bundle...
building fonts directory...
generating GLib schema cache...
creating GTK+ icon theme cache...
building cache files for GTK+ input methods...
building directory of Info manuals...
building database for manual pages...
building XDG desktop file cache...
building XDG MIME database...
building profile with 182 packages...

real    19m0.126s
user    0m5.648s
sys     0m0.333s
--8<---------------cut here---------------end--------------->8---

Most of the time remains spent after the message 'building profile with
182 package...'.  That part seems IO-bound, with the spinning disks
grinding heavily and the CPU mostly idling.  The rest of the time (3
minutes), was used by the profile hooks.

The same operation on a second, more modern machine equipped with M2
SSDs does much better and takes about 1 minute to accomplish the same,
so it seems the bad performance can be mostly attributed to the much
slower disk seek times of the spinning disks.

On the older machine, two profile hooks are also sticking out w.r.t. the
time they take (they take more than one minute opposed to a few
seconds):

--8<---------------cut here---------------start------------->8---
The following profile hook will be built:
   /gnu/store/08fanpydi7z4i3qnlqbr8iz23zdgsamw-manual-database.drv
building database for manual pages...
Creating manual page database...
[2139/2139] building list of man-db entries...
175322 entries processed in 95.1 s
successfully built /gnu/store/08fanpydi7z4i3qnlqbr8iz23zdgsamw-manual-database.drv
successfully built /gnu/store/08fanpydi7z4i3qnlqbr8iz23zdgsamw-manual-database.drv
/gnu/store/wzp4mk2r7r4ysciw74gqbfkyai0zmrcc-manual-database

real    1m36.378s
user    0m1.674s
sys     0m0.108s

The following profile hook will be built:
   /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
building XDG MIME database...
successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
/gnu/store/j0bznlj2ibnhirijhnwpkkxzz4qfk8wb-xdg-mime-database

real    1m7.344s
user    0m1.331s
sys     0m0.053s
--8<---------------cut here---------------end--------------->8---

So we should profile what's going on while generating the profile (no
pun intended) and try to improve this at first since this is where most
of the time is spent on spinning drives (17 minutes out of the 20 in the
above example).

After that we could look into the two above profile hooks.

Thanks,

Maxim




^ permalink raw reply	[flat|nested] 4+ messages in thread

* bug#44053: Poor profile generation performance on spinning disks
  2020-10-18  3:35 bug#44053: Poor profile generation performance on spinning disks Maxim Cournoyer
@ 2020-10-18 23:05 ` Luis Felipe via Bug reports for GNU Guix
  2020-10-19  8:18 ` zimoun
  1 sibling, 0 replies; 4+ messages in thread
From: Luis Felipe via Bug reports for GNU Guix @ 2020-10-18 23:05 UTC (permalink / raw)
  To: 44053

I've never used SDDs, so I've been experiencing this since I installed Guix (~5 years ago). And yes, building the profile seems to be the more resource intensive operation. I usually can't do anything else while that happens because the computer becomes unresponsive.

Running the same command Maxim mentioned took the following time in my case:


$ time guix package -i perl --max-jobs=1
Se instalará el siguiente paquete:
   perl 5.30.2
Se construirá la siguiente derivación:
   /gnu/store/86g48nh8dhgdhzb8r9bxjk6mmszlssss-profile.drv
construyendo empaquetado de certificados de CA...
construyendo el directorio de tipografías...
generando la caché de esquemas de GLib...
creando la caché de temas de iconos de GTK+...
construyendo los ficheros de caché para los métodos de entrada de GTK+...
construyendo el directorio de manuales Info...
construyendo la base de datos de páginas de manual...
construyendo la caché de ficheros desktop XDG...
construyendo la base de datos MIME XDG...
construyendo perfil con 87 paquetes...


real    8m3,043s
user    0m2,656s
sys    0m0,239s


My computer:

Intel® Core™ i3-8100 CPU @ 3.60GHz × 4
RAM: 4 GiB
1 TB HDD





^ permalink raw reply	[flat|nested] 4+ messages in thread

* bug#44053: Poor profile generation performance on spinning disks
  2020-10-18  3:35 bug#44053: Poor profile generation performance on spinning disks Maxim Cournoyer
  2020-10-18 23:05 ` Luis Felipe via Bug reports for GNU Guix
@ 2020-10-19  8:18 ` zimoun
  2020-10-19 18:18   ` Maxim Cournoyer
  1 sibling, 1 reply; 4+ messages in thread
From: zimoun @ 2020-10-19  8:18 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: 44053

Hi Maxim,

On Sun, 18 Oct 2020 at 05:37, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> On my home machine, will is still relying on 2 x 1 TB spinning drives in
> RAID1, rebuilding my user profile, which contains 182 entries, takes on
> average about 20 minutes, even when there are no packages to be built:

The database build time seems correlated to the number of packages in
the very profile.  Well, I have not benchmarked to see if it is
linear, quadratic; or worse or better.


> --8<---------------cut here---------------start------------->8---
[...]
> building XDG MIME database...
> successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
> successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
> /gnu/store/j0bznlj2ibnhirijhnwpkkxzz4qfk8wb-xdg-mime-database
>
> real    1m7.344s
> user    0m1.331s
> sys     0m0.053s
> --8<---------------cut here---------------end--------------->8---

Well, it is all clear to me, but I do not know if we can do better
because these 2 XDG updates seem relying on the external binaries
"/bin/update-desktop-database" and "/bin/update-mime-database".  I
have not timed the function 'xdg-desktop-database' but all the time
should be spent on these, I bet. :-)


All the best,
simon




^ permalink raw reply	[flat|nested] 4+ messages in thread

* bug#44053: Poor profile generation performance on spinning disks
  2020-10-19  8:18 ` zimoun
@ 2020-10-19 18:18   ` Maxim Cournoyer
  0 siblings, 0 replies; 4+ messages in thread
From: Maxim Cournoyer @ 2020-10-19 18:18 UTC (permalink / raw)
  To: zimoun; +Cc: 44053

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

Hello Simon,

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Sun, 18 Oct 2020 at 05:37, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> On my home machine, will is still relying on 2 x 1 TB spinning drives in
>> RAID1, rebuilding my user profile, which contains 182 entries, takes on
>> average about 20 minutes, even when there are no packages to be built:
>
> The database build time seems correlated to the number of packages in
> the very profile.  Well, I have not benchmarked to see if it is
> linear, quadratic; or worse or better.

It is.  Small profiles operations are much faster than large profiles.
Attached is a slightly trimmed version of the manifest I currently use
for my user profile, if you'd like to benchmark it on your system.

[-- Attachment #2: manifest.scm --]
[-- Type: text/plain, Size: 3639 bytes --]

(use-modules (gnu packages)
             (gnu packages emacs)
             (guix build-system emacs)
             (guix profiles))

(concatenate-manifests
 (list
 ;;; Emacs packages.
  (specifications->manifest
   '("emacs"
     "emacs-auctex"
     "emacs-bash-completion"
     "emacs-bbdb"
     "emacs-cmake-mode"
     "emacs-company"
     "emacs-company-quickhelp"
     "emacs-counsel"
     "emacs-csv-mode"
     "emacs-debbugs"
     "emacs-diff-hl"
     "emacs-el-mock"
     "emacs-elpy"
     "emacs-emms"
     "emacs-ggtags"
     "emacs-go-mode"
     "emacs-grep-a-lot"
     "emacs-guix"
     "emacs-htmlize"
     "emacs-ivy"
     "emacs-magit"
     "emacs-org"
     "emacs-org-reveal"
     "emacs-paredit"
     "emacs-pdf-tools"
     "emacs-qml-mode"
     "emacs-realgud"
     "emacs-string-inflection"
     "emacs-swiper"
     "emacs-w3m"
     "emacs-ws-butler"
     "emacs-yasnippet"
     "emacs-yasnippet-snippets"))

  ;; Other software.
  (specifications->manifest
   '("adb"
     "acpi"
     "alsa-utils"
     "anthy"
     "arc-icon-theme"
     "arc-theme"
     "aspell"
     "aspell-dict-en"
     "aspell-dict-fr"
     "autoconf"
     "automake"
     "autossh"
     "bash"
     "bc"
     "beep"
     "bind:utils"              ;for 'dig'
     "bluez"
     "bridge-utils"
     "cheese"
     "compsize"
     "cqfd"
     "cryptsetup"
     "curl"
     "dbus"
     "dconf"
     "ddcutil"
     "diffoscope"
     "docker-cli"
     "dosfstools"
     "evince"
     "file"
     "font-adobe-source-han-sans"
     "font-dejavu"
     "font-google-roboto"
     "font-hack"
     "gcc-toolchain"
     "gdb"
     "geeqie"
     "ghostscript-with-x"
     "gimp"
     "git"
     "git:send-email"
     "glibc-locales"
     "global"
     "gnome-bluetooth"
     "gnome-boxes"
     "adwaita-icon-theme"
     "hicolor-icon-theme"
     "gnu-standards"
     "gnucash"
     "gnucash:doc"
     "gnupg"
     "graphviz"
     "gtk-engines"
     "guile"
     "guile-readline"
     "guile-sqlite3"
     "guile-ssh"
     "hackneyed-x11-cursors"
     "hicolor-icon-theme"
     "hunspell"
     "hunspell-dict-fr"
     "ibus"
     "ibus-anthy"
     "icecat"
     "imagemagick"
     "inetutils"
     "inkscape"
     "iotop"
     "jack"
     "jami"
     "keepassxc"
     "libjpeg"
     "libmtp"
     "libpcap"
     "libreoffice"
     "libssh"
     "libx11"
     "linphoneqt"
     "lm-sensors"
     "lsof"
     "ltrace"
     "lvm2"                               ;for dmsetup
     "make"
     "man-pages"
     "mesa-utils"
     "mpv"
     "mtr"
     "nmap"
     "openssh"
     "openvpn"
     "parted"
     "pavucontrol"
     "perl"
     "pinentry"
     "pkg-config"
     "poppler"
     "pulseaudio"
     "pv"
     "python"
     "python-wrapper"
     "qemu"
     "recutils"
     "rsync"
     "rtorrent"
     "screen"
     "setxkbmap"
     "shepherd"
     "sicp"
     "smartmontools"
     "spacefm"
     "stow"
     "strace"
     "sysstat"				;for iostat
     "tcpdump"
     "the-silver-searcher"                ;ag
     "time"                             ;aliased to time+
     "transmission"
     "transmission:gui"
     "tree"
     "unzip"
     "vinagre"
     "vorbis-tools"
     "weechat"
     "wget"
     "workrave"
     "wpa-supplicant"
     "xdpyinfo"
     "xdg-utils"
     "xev"
     "xmodmap"
     "xournal"
     "xrandr"
     "xrdb"
     "xsetroot"
     "yelp"
     "gxtuner"
     "shellcheck"))

  ;; Others.
  (specifications->manifest
   '("docker-compose"
     "emacs-adoc-mode"
     "emacs-clang-format"
     "emacs-clang-rename"
     "emacs-feature-mode"
     "picocom"
     "python-git-review"
     "sshpass"
     "ungoogled-chromium"
     "ddrescue"))))

[-- Attachment #3: Type: text/plain, Size: 1721 bytes --]


I'm betting that the part to optimize is:

--8<---------------cut here---------------start------------->8---
  ;; Make the symlinks.
  (union-build output inputs
               #:symlink symlink
               #:log-port (%make-void-port "w"))
--8<---------------cut here---------------end--------------->8---

from the 'build-profile' procedure in (guix build profiles).

>
>> --8<---------------cut here---------------start------------->8---
> [...]
>> building XDG MIME database...
>> successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
>> successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
>> /gnu/store/j0bznlj2ibnhirijhnwpkkxzz4qfk8wb-xdg-mime-database
>>
>> real    1m7.344s
>> user    0m1.331s
>> sys     0m0.053s
>> --8<---------------cut here---------------end--------------->8---
>
> Well, it is all clear to me, but I do not know if we can do better
> because these 2 XDG updates seem relying on the external binaries
> "/bin/update-desktop-database" and "/bin/update-mime-database".  I
> have not timed the function 'xdg-desktop-database' but all the time
> should be spent on these, I bet. :-)

They do, and this is indeed what takes time.  But, this doesn't stop
motivated wizards from implementing our own database generator in Scheme
that would do the same.  Ludovic had done just that for the man-db
database generator.  The custom code in Guix can do it much faster that
it initially could back when it was relying on man-db to do that (that's
also probably the reason why we have issues such as
https://issues.guix.gnu.org/38838; perhaps the generated database is
slightly different -- but that's another topic :-)).

Thanks,

Maxim

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2020-10-19 18:19 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-18  3:35 bug#44053: Poor profile generation performance on spinning disks Maxim Cournoyer
2020-10-18 23:05 ` Luis Felipe via Bug reports for GNU Guix
2020-10-19  8:18 ` zimoun
2020-10-19 18:18   ` Maxim Cournoyer

unofficial mirror of bug-guix@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-bugs/0 guix-bugs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-bugs guix-bugs/ https://yhetil.org/guix-bugs \
		bug-guix@gnu.org
	public-inbox-index guix-bugs

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.bugs
	nntp://news.gmane.io/gmane.comp.gnu.guix.bugs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git