From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id cTxFA6PYjV8iIAAA0tVLHw (envelope-from ) for ; Mon, 19 Oct 2020 18:19:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id HiE/OqLYjV/ccQAAbx9fmQ (envelope-from ) for ; Mon, 19 Oct 2020 18:19:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1B3E2940396 for ; Mon, 19 Oct 2020 18:19:13 +0000 (UTC) Received: from localhost ([::1]:52648 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kUZkV-0000D1-2g for larch@yhetil.org; Mon, 19 Oct 2020 14:19:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59810) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kUZkM-0000Cs-C6 for bug-guix@gnu.org; Mon, 19 Oct 2020 14:19:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:58996) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kUZkM-0003DI-2x for bug-guix@gnu.org; Mon, 19 Oct 2020 14:19:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kUZkL-0004ue-Ue for bug-guix@gnu.org; Mon, 19 Oct 2020 14:19:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#44053: Poor profile generation performance on spinning disks Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 19 Oct 2020 18:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44053 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: zimoun Received: via spool by 44053-submit@debbugs.gnu.org id=B44053.160313152318857 (code B ref 44053); Mon, 19 Oct 2020 18:19:01 +0000 Received: (at 44053) by debbugs.gnu.org; 19 Oct 2020 18:18:43 +0000 Received: from localhost ([127.0.0.1]:42309 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kUZk2-0004u5-NM for submit@debbugs.gnu.org; Mon, 19 Oct 2020 14:18:43 -0400 Received: from mail-io1-f46.google.com ([209.85.166.46]:36659) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kUZjx-0004tp-DB for 44053@debbugs.gnu.org; Mon, 19 Oct 2020 14:18:41 -0400 Received: by mail-io1-f46.google.com with SMTP id u19so821316ion.3 for <44053@debbugs.gnu.org>; Mon, 19 Oct 2020 11:18:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=4Qy9tir5c9ssrhdi1MVIkkHwUbROBjPuXtBl0F8Bbqo=; b=tKHWaO1sBTvfsCERlNeRl+PYbQ6PoaDVinOWWjq1SJsmFDsusaEkoQqaWuxS+46utk NP9ckLWDHCFogPL33llRk6xDj2favPfurTr1nHU08x7oM4in20qBOYOl1xVsZcABLskQ inXzSBWjLRof635IdKqalH2Avyqa7lTkABlpT4Y+6sfEJhUkrN5Y0oFSqYepZf8ZEzBL aV06BVDET2W2mGyYMZr8Hi8BOR6O7bzH08IWDJ2ILdwL/8ZJMEd3DTYQ76k4sntjCjnU 35ViCiJiqOSkA71Jf7lm+heyLgoflcLVo3QWoibK4IjFkMNS4ToVivjvgIPBQxksrt3S jldg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=4Qy9tir5c9ssrhdi1MVIkkHwUbROBjPuXtBl0F8Bbqo=; b=DrKCwydRXGa/xx2fKEKbiQSaz0+DShEGhYJRrQ6PL0++6a+aiRqM+lGuI+XSeqFI29 axzc8Tm5iawXC2IoCABK4dhjxsWZhjX9KLsrVlKT2LPAi1WFQdHXNjnBMW0s1/TO7SxW OCQ3aHSuZC9u/I5lu8MZLpr2OSFMKwnqtgqhfq04ZiAMg8gcNcYHwvMMw/8zCiodU9zK FOZyNoWryBwE7Nu54sZ7pUgLdJtsAHvsUOqdm7tM34/PKYR3kkm9xgSEfpdrp/LlT0Lq Zb4YX7jIf5/lfiJIsiGKsoLof/zexpSGrXHum+QMYlPwbicxikAR4TcpTLz+dK5lIBsJ Yr6g== X-Gm-Message-State: AOAM531839xw4RsgU3qMsZOWDQ/GUZwaS6cLpZKKT3RR3g403gEKKaGx KBskpB2Kb2oYEoGmbYrFuXD6dCzDzX4= X-Google-Smtp-Source: ABdhPJxEUd9xr2R2sfAeHNDDZO9QqYNQ32HSiSU9gIG4iXUMnLls8ZnmJ8dbXy1r+XMt4b88s3OlHA== X-Received: by 2002:a02:7685:: with SMTP id z127mr1016513jab.45.1603131511464; Mon, 19 Oct 2020 11:18:31 -0700 (PDT) Received: from hurd (dsl-141-39.b2b2c.ca. [66.158.141.39]) by smtp.gmail.com with ESMTPSA id y16sm324927iow.34.2020.10.19.11.18.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Oct 2020 11:18:29 -0700 (PDT) From: Maxim Cournoyer References: <87zh4k435a.fsf@gmail.com> Date: Mon, 19 Oct 2020 14:18:29 -0400 In-Reply-To: (zimoun's message of "Mon, 19 Oct 2020 10:18:43 +0200") Message-ID: <87d01e5bbe.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 44053@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=gmail.com header.s=20161025 header.b=tKHWaO1s; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: 1.09 X-TUID: lkZmVojTY27I --=-=-= Content-Type: text/plain Hello Simon, zimoun writes: > Hi Maxim, > > On Sun, 18 Oct 2020 at 05:37, Maxim Cournoyer wrote: > >> On my home machine, will is still relying on 2 x 1 TB spinning drives in >> RAID1, rebuilding my user profile, which contains 182 entries, takes on >> average about 20 minutes, even when there are no packages to be built: > > The database build time seems correlated to the number of packages in > the very profile. Well, I have not benchmarked to see if it is > linear, quadratic; or worse or better. It is. Small profiles operations are much faster than large profiles. Attached is a slightly trimmed version of the manifest I currently use for my user profile, if you'd like to benchmark it on your system. --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=manifest.scm (use-modules (gnu packages) (gnu packages emacs) (guix build-system emacs) (guix profiles)) (concatenate-manifests (list ;;; Emacs packages. (specifications->manifest '("emacs" "emacs-auctex" "emacs-bash-completion" "emacs-bbdb" "emacs-cmake-mode" "emacs-company" "emacs-company-quickhelp" "emacs-counsel" "emacs-csv-mode" "emacs-debbugs" "emacs-diff-hl" "emacs-el-mock" "emacs-elpy" "emacs-emms" "emacs-ggtags" "emacs-go-mode" "emacs-grep-a-lot" "emacs-guix" "emacs-htmlize" "emacs-ivy" "emacs-magit" "emacs-org" "emacs-org-reveal" "emacs-paredit" "emacs-pdf-tools" "emacs-qml-mode" "emacs-realgud" "emacs-string-inflection" "emacs-swiper" "emacs-w3m" "emacs-ws-butler" "emacs-yasnippet" "emacs-yasnippet-snippets")) ;; Other software. (specifications->manifest '("adb" "acpi" "alsa-utils" "anthy" "arc-icon-theme" "arc-theme" "aspell" "aspell-dict-en" "aspell-dict-fr" "autoconf" "automake" "autossh" "bash" "bc" "beep" "bind:utils" ;for 'dig' "bluez" "bridge-utils" "cheese" "compsize" "cqfd" "cryptsetup" "curl" "dbus" "dconf" "ddcutil" "diffoscope" "docker-cli" "dosfstools" "evince" "file" "font-adobe-source-han-sans" "font-dejavu" "font-google-roboto" "font-hack" "gcc-toolchain" "gdb" "geeqie" "ghostscript-with-x" "gimp" "git" "git:send-email" "glibc-locales" "global" "gnome-bluetooth" "gnome-boxes" "adwaita-icon-theme" "hicolor-icon-theme" "gnu-standards" "gnucash" "gnucash:doc" "gnupg" "graphviz" "gtk-engines" "guile" "guile-readline" "guile-sqlite3" "guile-ssh" "hackneyed-x11-cursors" "hicolor-icon-theme" "hunspell" "hunspell-dict-fr" "ibus" "ibus-anthy" "icecat" "imagemagick" "inetutils" "inkscape" "iotop" "jack" "jami" "keepassxc" "libjpeg" "libmtp" "libpcap" "libreoffice" "libssh" "libx11" "linphoneqt" "lm-sensors" "lsof" "ltrace" "lvm2" ;for dmsetup "make" "man-pages" "mesa-utils" "mpv" "mtr" "nmap" "openssh" "openvpn" "parted" "pavucontrol" "perl" "pinentry" "pkg-config" "poppler" "pulseaudio" "pv" "python" "python-wrapper" "qemu" "recutils" "rsync" "rtorrent" "screen" "setxkbmap" "shepherd" "sicp" "smartmontools" "spacefm" "stow" "strace" "sysstat" ;for iostat "tcpdump" "the-silver-searcher" ;ag "time" ;aliased to time+ "transmission" "transmission:gui" "tree" "unzip" "vinagre" "vorbis-tools" "weechat" "wget" "workrave" "wpa-supplicant" "xdpyinfo" "xdg-utils" "xev" "xmodmap" "xournal" "xrandr" "xrdb" "xsetroot" "yelp" "gxtuner" "shellcheck")) ;; Others. (specifications->manifest '("docker-compose" "emacs-adoc-mode" "emacs-clang-format" "emacs-clang-rename" "emacs-feature-mode" "picocom" "python-git-review" "sshpass" "ungoogled-chromium" "ddrescue")))) --=-=-= Content-Type: text/plain I'm betting that the part to optimize is: --8<---------------cut here---------------start------------->8--- ;; Make the symlinks. (union-build output inputs #:symlink symlink #:log-port (%make-void-port "w")) --8<---------------cut here---------------end--------------->8--- from the 'build-profile' procedure in (guix build profiles). > >> --8<---------------cut here---------------start------------->8--- > [...] >> building XDG MIME database... >> successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv >> successfully built /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv >> /gnu/store/j0bznlj2ibnhirijhnwpkkxzz4qfk8wb-xdg-mime-database >> >> real 1m7.344s >> user 0m1.331s >> sys 0m0.053s >> --8<---------------cut here---------------end--------------->8--- > > Well, it is all clear to me, but I do not know if we can do better > because these 2 XDG updates seem relying on the external binaries > "/bin/update-desktop-database" and "/bin/update-mime-database". I > have not timed the function 'xdg-desktop-database' but all the time > should be spent on these, I bet. :-) They do, and this is indeed what takes time. But, this doesn't stop motivated wizards from implementing our own database generator in Scheme that would do the same. Ludovic had done just that for the man-db database generator. The custom code in Guix can do it much faster that it initially could back when it was relying on man-db to do that (that's also probably the reason why we have issues such as https://issues.guix.gnu.org/38838; perhaps the generated database is slightly different -- but that's another topic :-)). Thanks, Maxim --=-=-=--