unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
To: zimoun <zimon.toutoune@gmail.com>
Cc: 44053@debbugs.gnu.org
Subject: bug#44053: Poor profile generation performance on spinning disks
Date: Mon, 19 Oct 2020 14:18:29 -0400	[thread overview]
Message-ID: <87d01e5bbe.fsf@gmail.com> (raw)
In-Reply-To: <CAJ3okZ1AMUteo3qpAHb0m6RS3ApawCrdfQwVNkDg6Xtc515LcA@mail.gmail.com> (zimoun's message of "Mon, 19 Oct 2020 10:18:43 +0200")

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

Hello Simon,

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Sun, 18 Oct 2020 at 05:37, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> On my home machine, will is still relying on 2 x 1 TB spinning drives in
>> RAID1, rebuilding my user profile, which contains 182 entries, takes on
>> average about 20 minutes, even when there are no packages to be built:
>
> The database build time seems correlated to the number of packages in
> the very profile.  Well, I have not benchmarked to see if it is
> linear, quadratic; or worse or better.

It is.  Small profiles operations are much faster than large profiles.
Attached is a slightly trimmed version of the manifest I currently use
for my user profile, if you'd like to benchmark it on your system.

[-- Attachment #2: manifest.scm --]
[-- Type: text/plain, Size: 3639 bytes --]

(use-modules (gnu packages)
             (gnu packages emacs)
             (guix build-system emacs)
             (guix profiles))

(concatenate-manifests
 (list
 ;;; Emacs packages.
  (specifications->manifest
   '("emacs"
     "emacs-auctex"
     "emacs-bash-completion"
     "emacs-bbdb"
     "emacs-cmake-mode"
     "emacs-company"
     "emacs-company-quickhelp"
     "emacs-counsel"
     "emacs-csv-mode"
     "emacs-debbugs"
     "emacs-diff-hl"
     "emacs-el-mock"
     "emacs-elpy"
     "emacs-emms"
     "emacs-ggtags"
     "emacs-go-mode"
     "emacs-grep-a-lot"
     "emacs-guix"
     "emacs-htmlize"
     "emacs-ivy"
     "emacs-magit"
     "emacs-org"
     "emacs-org-reveal"
     "emacs-paredit"
     "emacs-pdf-tools"
     "emacs-qml-mode"
     "emacs-realgud"
     "emacs-string-inflection"
     "emacs-swiper"
     "emacs-w3m"
     "emacs-ws-butler"
     "emacs-yasnippet"
     "emacs-yasnippet-snippets"))

  ;; Other software.
  (specifications->manifest
   '("adb"
     "acpi"
     "alsa-utils"
     "anthy"
     "arc-icon-theme"
     "arc-theme"
     "aspell"
     "aspell-dict-en"
     "aspell-dict-fr"
     "autoconf"
     "automake"
     "autossh"
     "bash"
     "bc"
     "beep"
     "bind:utils"              ;for 'dig'
     "bluez"
     "bridge-utils"
     "cheese"
     "compsize"
     "cqfd"
     "cryptsetup"
     "curl"
     "dbus"
     "dconf"
     "ddcutil"
     "diffoscope"
     "docker-cli"
     "dosfstools"
     "evince"
     "file"
     "font-adobe-source-han-sans"
     "font-dejavu"
     "font-google-roboto"
     "font-hack"
     "gcc-toolchain"
     "gdb"
     "geeqie"
     "ghostscript-with-x"
     "gimp"
     "git"
     "git:send-email"
     "glibc-locales"
     "global"
     "gnome-bluetooth"
     "gnome-boxes"
     "adwaita-icon-theme"
     "hicolor-icon-theme"
     "gnu-standards"
     "gnucash"
     "gnucash:doc"
     "gnupg"
     "graphviz"
     "gtk-engines"
     "guile"
     "guile-readline"
     "guile-sqlite3"
     "guile-ssh"
     "hackneyed-x11-cursors"
     "hicolor-icon-theme"
     "hunspell"
     "hunspell-dict-fr"
     "ibus"
     "ibus-anthy"
     "icecat"
     "imagemagick"
     "inetutils"
     "inkscape"
     "iotop"
     "jack"
     "jami"
     "keepassxc"
     "libjpeg"
     "libmtp"
     "libpcap"
     "libreoffice"
     "libssh"
     "libx11"
     "linphoneqt"
     "lm-sensors"
     "lsof"
     "ltrace"
     "lvm2"                               ;for dmsetup
     "make"
     "man-pages"
     "mesa-utils"
     "mpv"
     "mtr"
     "nmap"
     "openssh"
     "openvpn"
     "parted"
     "pavucontrol"
     "perl"
     "pinentry"
     "pkg-config"
     "poppler"
     "pulseaudio"
     "pv"
     "python"
     "python-wrapper"
     "qemu"
     "recutils"
     "rsync"
     "rtorrent"
     "screen"
     "setxkbmap"
     "shepherd"
     "sicp"
     "smartmontools"
     "spacefm"
     "stow"
     "strace"
     "sysstat"				;for iostat
     "tcpdump"
     "the-silver-searcher"                ;ag
     "time"                             ;aliased to time+
     "transmission"
     "transmission:gui"
     "tree"
     "unzip"
     "vinagre"
     "vorbis-tools"
     "weechat"
     "wget"
     "workrave"
     "wpa-supplicant"
     "xdpyinfo"
     "xdg-utils"
     "xev"
     "xmodmap"
     "xournal"
     "xrandr"
     "xrdb"
     "xsetroot"
     "yelp"
     "gxtuner"
     "shellcheck"))

  ;; Others.
  (specifications->manifest
   '("docker-compose"
     "emacs-adoc-mode"
     "emacs-clang-format"
     "emacs-clang-rename"
     "emacs-feature-mode"
     "picocom"
     "python-git-review"
     "sshpass"
     "ungoogled-chromium"
     "ddrescue"))))

[-- Attachment #3: Type: text/plain, Size: 1721 bytes --]


I'm betting that the part to optimize is:

--8<---------------cut here---------------start------------->8---
  ;; Make the symlinks.
  (union-build output inputs
               #:symlink symlink
               #:log-port (%make-void-port "w"))
--8<---------------cut here---------------end--------------->8---

from the 'build-profile' procedure in (guix build profiles).

>
>> --8<---------------cut here---------------start------------->8---
> [...]
>> building XDG MIME database...
>> successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
>> successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
>> /gnu/store/j0bznlj2ibnhirijhnwpkkxzz4qfk8wb-xdg-mime-database
>>
>> real    1m7.344s
>> user    0m1.331s
>> sys     0m0.053s
>> --8<---------------cut here---------------end--------------->8---
>
> Well, it is all clear to me, but I do not know if we can do better
> because these 2 XDG updates seem relying on the external binaries
> "/bin/update-desktop-database" and "/bin/update-mime-database".  I
> have not timed the function 'xdg-desktop-database' but all the time
> should be spent on these, I bet. :-)

They do, and this is indeed what takes time.  But, this doesn't stop
motivated wizards from implementing our own database generator in Scheme
that would do the same.  Ludovic had done just that for the man-db
database generator.  The custom code in Guix can do it much faster that
it initially could back when it was relying on man-db to do that (that's
also probably the reason why we have issues such as
https://issues.guix.gnu.org/38838; perhaps the generated database is
slightly different -- but that's another topic :-)).

Thanks,

Maxim

  reply	other threads:[~2020-10-19 18:19 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-18  3:35 bug#44053: Poor profile generation performance on spinning disks Maxim Cournoyer
2020-10-18 23:05 ` Luis Felipe via Bug reports for GNU Guix
2020-10-19  8:18 ` zimoun
2020-10-19 18:18   ` Maxim Cournoyer [this message]
2022-03-23 12:38 ` zimoun
2022-03-23 16:17   ` Maxim Cournoyer
2022-03-23 16:54     ` zimoun
2023-08-26 22:11 ` jbranso--- via Bug reports for GNU Guix
2023-08-27 17:59   ` Luis Felipe via Bug reports for GNU Guix
2023-08-29  9:19     ` Ludovic Courtès
2023-08-29 21:51       ` Luis Felipe via Bug reports for GNU Guix
2023-09-09 11:02         ` Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87d01e5bbe.fsf@gmail.com \
    --to=maxim.cournoyer@gmail.com \
    --cc=44053@debbugs.gnu.org \
    --cc=zimon.toutoune@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).