* guix pull performance
@ 2023-10-12 11:03 Emmanuel Beffara
2023-10-12 11:24 ` Ignas Lapėnas
0 siblings, 1 reply; 6+ messages in thread
From: Emmanuel Beffara @ 2023-10-12 11:03 UTC (permalink / raw)
To: help-guix
Hello Guix,
I am having performance issues with Guix operations since a few weeks.
Concretely, any call to `guix pull` takes approximately 10 minutes to finish,
on a reasonably powerful machine (Core i7, 12 cores, 16G RAM, 1 TB SSD drive
with plenty of available space), and this even if rerun `guix pull` when
nothing has changed. I did not note the precise time or context where things
changed, unfortunately, but not long ago this was way under a minute for a
pull to finish, with the same channels.
In the same context, it takes 3 minutes for a `guix home reconfigure` with a
small home profile and 5 minutes for a `guix system reconfigure` with a rather
big profile, when there is nothing to update.
Is this kind of performance normal ? And if not, how can I diagnose the
problem to understand what takes that much time ?
--
Emmanuel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: guix pull performance
2023-10-12 11:03 guix pull performance Emmanuel Beffara
@ 2023-10-12 11:24 ` Ignas Lapėnas
2023-10-12 13:51 ` Emmanuel Beffara
0 siblings, 1 reply; 6+ messages in thread
From: Ignas Lapėnas @ 2023-10-12 11:24 UTC (permalink / raw)
To: Emmanuel Beffara; +Cc: help-guix
Hi,
I’m fairly new to the Guix scene, but recently there was a discussion about it
on devel-guix@gnu.org mailing list. So far it works as designed, because every
time you use guix pull, they have to recompile every package definition.
I’m a complete pleb when it comes to guile development, so take it with a heavy
grain of salt.
You can check for more information in the devel archive:
<https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=Guix+pull+speed&submit=Search%21&idxname=guix-devel&max=20&result=normal&sort=score>
Hope that helps,
Ignas
Emmanuel Beffara <manu@beffara.org> writes:
> Hello Guix,
>
> I am having performance issues with Guix operations since a few weeks.
> Concretely, any call to `guix pull` takes approximately 10 minutes to finish,
> on a reasonably powerful machine (Core i7, 12 cores, 16G RAM, 1 TB SSD drive
> with plenty of available space), and this even if rerun `guix pull` when
> nothing has changed. I did not note the precise time or context where things
> changed, unfortunately, but not long ago this was way under a minute for a
> pull to finish, with the same channels.
>
> In the same context, it takes 3 minutes for a `guix home reconfigure` with a
> small home profile and 5 minutes for a `guix system reconfigure` with a rather
> big profile, when there is nothing to update.
>
> Is this kind of performance normal ? And if not, how can I diagnose the
> problem to understand what takes that much time ?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: guix pull performance
2023-10-12 11:24 ` Ignas Lapėnas
@ 2023-10-12 13:51 ` Emmanuel Beffara
2023-10-12 17:02 ` Simon Tournier
0 siblings, 1 reply; 6+ messages in thread
From: Emmanuel Beffara @ 2023-10-12 13:51 UTC (permalink / raw)
To: Ignas Lapėnas; +Cc: help-guix
De Ignas Lapėnas le 12/10/2023 à 13:24:
> I’m fairly new to the Guix scene, but recently there was a discussion about it
> on devel-guix@gnu.org mailing list. So far it works as designed, because every
> time you use guix pull, they have to recompile every package definition.
> I’m a complete pleb when it comes to guile development, so take it with a heavy
> grain of salt.
>
> You can check for more information in the devel archive:
> <https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=Guix+pull+speed&submit=Search%21&idxname=guix-devel&max=20&result=normal&sort=score>
Thanks for the pointer. Now I think I understand the reason for this
behaviour: Guix has to compile the whole set of modules, including all package
definitions, and it is rather slow at that. So I guess I won't expect
significant improvement in the near future, but at least it's not an issue
with my particular installation!
--
Emmanuel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: guix pull performance
2023-10-12 13:51 ` Emmanuel Beffara
@ 2023-10-12 17:02 ` Simon Tournier
2023-11-28 21:07 ` Emmanuel Beffara
0 siblings, 1 reply; 6+ messages in thread
From: Simon Tournier @ 2023-10-12 17:02 UTC (permalink / raw)
To: Emmanuel Beffara, Ignas Lapėnas; +Cc: help-guix
Hi,
On Thu, 12 Oct 2023 at 15:51, Emmanuel Beffara <manu@beffara.org> wrote:
> Thanks for the pointer. Now I think I understand the reason for this
> behaviour: Guix has to compile the whole set of modules, including all package
> definitions, and it is rather slow at that. So I guess I won't expect
> significant improvement in the near future, but at least it's not an issue
> with my particular installation!
If it is not already the case, you might be interested by:
https://guix.gnu.org/manual/devel/en/guix.html#Channels-with-Substitutes
Cheers,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: guix pull performance
2023-10-12 17:02 ` Simon Tournier
@ 2023-11-28 21:07 ` Emmanuel Beffara
2023-12-01 17:13 ` Simon Tournier
0 siblings, 1 reply; 6+ messages in thread
From: Emmanuel Beffara @ 2023-11-28 21:07 UTC (permalink / raw)
To: help-guix
Hello again,
De Simon Tournier le 12/10/2023 à 19:02:
> On Thu, 12 Oct 2023 at 15:51, Emmanuel Beffara <manu@beffara.org> wrote:
>
> > Thanks for the pointer. Now I think I understand the reason for this
> > behaviour: Guix has to compile the whole set of modules, including all package
> > definitions, and it is rather slow at that. So I guess I won't expect
> > significant improvement in the near future, but at least it's not an issue
> > with my particular installation!
>
> If it is not already the case, you might be interested by:
>
> https://guix.gnu.org/manual/devel/en/guix.html#Channels-with-Substitutes
I didn't answer to that at the time, but I do use channels with substitutes as
described in the documentation. I only have the default channel as desribed
there, plus a small custom channel in a local git repository where I put
definitions for custom packages and for third-party packages that I need
(which I intend to propose for integration in Guix itself eventually).
I actually use another channel with less libre kernels and firmwares, but I
disabled it for testing pull times (and it doesn't change things much).
Now, even if I didn't expect a significant improvement, I am still badly
surprised by the behaviour of Guix: one month ago, any `guix pull` took 10
minutes to complete, and these days it takes 40 minutes (the same command, on
the same machine in the same conditions). When running a second `guix pull`
while the channels are unchanged, it takes "only" 6 minutes of computation
before telling me that there is nothing to do.
How is it possible that things are four times slower over just one month ?
Did the package collection increase that much ?
Were there structural changes that caused a loss of efficiency ?
Or might there actually be someting wrong with my setup, and in this case how
could I diagnose it ?
Updating Guix is becoming painful, and if run times continue with their
exponential growth, it will become simply unusable.
--
Emmanuel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: guix pull performance
2023-11-28 21:07 ` Emmanuel Beffara
@ 2023-12-01 17:13 ` Simon Tournier
0 siblings, 0 replies; 6+ messages in thread
From: Simon Tournier @ 2023-12-01 17:13 UTC (permalink / raw)
To: Emmanuel Beffara, help-guix
Hi,
If I read correctly, your machine is:
a reasonably powerful machine (Core i7, 12 cores, 16G RAM, 1 TB SSD drive
with plenty of available space)
Correct?
On Tue, 28 Nov 2023 at 22:07, Emmanuel Beffara <manu@beffara.org> wrote:
> Now, even if I didn't expect a significant improvement, I am still badly
> surprised by the behaviour of Guix: one month ago, any `guix pull` took 10
> minutes to complete, and these days it takes 40 minutes (the same command, on
> the same machine in the same conditions). When running a second `guix pull`
> while the channels are unchanged, it takes "only" 6 minutes of computation
> before telling me that there is nothing to do.
>
> How is it possible that things are four times slower over just one month ?
> Did the package collection increase that much ?
> Were there structural changes that caused a loss of efficiency ?
Using my desktop machine (i7, 16 cores, 64G RAM), I get:
+ plain “guix pull”: ~10min
+ redo “guix pull” with the same revision: ~2min
--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 29 Oct 25 2023 14:42:15 (current)
guix aabfddb
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: aabfddbe8bc16fa82067e88e7d79c8c1bc802414
$ time guix pull -p /tmp/new
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 842a11f (82 new commits)...
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git 842a11f
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
[...]
config.scm 507B 174KiB/s 00:00 ▕██████████████████▏ 100.0%
git.scm 105B 191KiB/s 00:00 ▕██████████████████▏ 100.0%
hash.scm 130B 105KiB/s 00:00 ▕██████████████████▏ 100.0%
module-import 2KiB 335KiB/s 00:00 ▕██████████████████▏ 100.0%
module-import 2KiB 504KiB/s 00:00 ▕██████████████████▏ 100.0%
module-import-compiled 1.2MiB 2.4MiB/s 00:01 ▕██████████████████▏ 100.0%
module-import-compiled 1.2MiB 3.2MiB/s 00:00 ▕██████████████████▏ 100.0%
building /gnu/store/g83w55vbr1r4p1jbxdxlrd00hsziqyvq-compute-guix-derivation.drv...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
git-minimal-2.41.0 4.6MiB 4.9MiB/s 00:01 ▕██████████████████▏ 100.0%
guix-daemon-1.4.0-15.e0885fc 258KiB 702.4MiB/s 00:00 ▕████▍ ▏ 24.8%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/02czx493wwvilvc8yc38insc94s0l5h7-profile.drv
/gnu/store/j2l9k04mp2wdnkvin6xfqxm9j45i1myd-guix-842a11f1c.drv
/gnu/store/1jf0d5ch5rpjgbbj7hbh7xwmzbvkp6cm-guix-misc.drv
/gnu/store/7q0mghfzrpd1kb0ckcc9hg2skigq2d4i-guix-command.drv
/gnu/store/lvnm196rxcn6d2smz76amj0xj3w73pnk-guix-module-union.drv
/gnu/store/yc9w06iifsb5l3xikkg7idc4jk9ldsw4-guix-842a11f1c-modules.drv
/gnu/store/2qqzddfp48inn38zsq2bi9if69aia1dk-guix-home-modules.drv
/gnu/store/fcrdfvhnm36x3vlh5j87mvl9frwwxxcj-guix-home.drv
/gnu/store/k2mardcbpcsivc9mygzdgx9xyvvxvq0p-guix-packages-base.drv
/gnu/store/vfbxyq50j7brlqyai7dr8npbgnivjlnb-guix-system.drv
/gnu/store/z9acy6mp2zbfvn9jzrqs70f4y7x1im3d-guix-packages.drv
/gnu/store/crwap92xzmvk3crd2nm3mxv2wfy0b6gg-guix-packages-modules.drv
/gnu/store/fnpq63yknym9wfg13wxfqfnc0mdd2ynh-guix-extra-modules.drv
/gnu/store/hbz4z9bq05z8kcmp2j686v76n45l2kfl-guix-system-modules.drv
/gnu/store/ivfpj6mbx12bizh8v4li6as2xd3sxw0l-guix-packages-base-modules.drv
/gnu/store/qs9bax2627scfrv9k5q2k8qa9qfri138-guix-cli-core-modules.drv
/gnu/store/6hb81r74zij9vswbh2zqyy9csifv53cm-guix-cli-core.drv
/gnu/store/qvf2wkgs2jlqkk983jy7qa7wv9mf4mc2-guix-cli-modules.drv
/gnu/store/mn5kjnwym7hikdgqkxz5v6q5icr2knp5-guix-cli.drv
/gnu/store/wjmzwngblydz65ylrh7kkgh0mq62lz0m-guix-config-modules.drv
/gnu/store/4d1ql296bygb0ph375316ny8i6d5m4va-guix-config-source.drv
/gnu/store/0lcmrv5mx312w0h80lrbjzbp25xhl8a3-config.scm.drv
/gnu/store/zvh0ixd433rmganyrcb1prcx4c9q5axb-guix-system-tests-modules.drv
/gnu/store/cbywx3457flfbf135kzn884ik802q2dz-guix-system-tests.drv
/gnu/store/qzhy5lyr56qppryvldghl22db5j66qp6-guix-manual.drv
/gnu/store/wg49p4988b3acgq8mcdrmc7p9gcd7i3y-guix-daemon.drv
/gnu/store/8jxl2mxyj7cqaqlq7rr4qxy0nrhj1mra-inferior-script.scm.drv
/gnu/store/zqpnys3yd7gnljv6nan936hzg7j81ygw-profile.drv
4.0 MB will be downloaded
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
guix-locale-guix 3.0MiB/s 00:00 | 486KiB transferred
git-minimal-2.41.0 4.6MiB 2.8MiB/s 00:02 ▕██████████████████▏ 100.0%
guix-daemon-1.4.0-15.e0885fc 258KiB 3.3MiB/s 00:00 ▕██████████████████▏ 100.0%
guix-config 4KiB 395KiB/s 00:00 ▕██████████████████▏ 100.0%
guix-daemon.cil 3KiB 4.9MiB/s 00:00 ▕██████████████████▏ 100.0%
guix-extra 2.8MiB 6.1MiB/s 00:00 ▕██████████████████▏ 100.0%
guix-translated-texinfo 1.6MiB 2.4MiB/s 00:01 ▕██████████████████▏ 100.0%
building /gnu/store/0lcmrv5mx312w0h80lrbjzbp25xhl8a3-config.scm.drv...
applying 9 grafts for gd-2.3.2 ...
applying 8 grafts for guile-avahi-0.4.1 ...
applying 19 grafts for graphviz-minimal-7.0.1 ...
applying 7 grafts for guile-ssh-0.16.3 ...
building /gnu/store/4d1ql296bygb0ph375316ny8i6d5m4va-guix-config-source.drv...
building /gnu/store/fnpq63yknym9wfg13wxfqfnc0mdd2ynh-guix-extra-modules.drv...
building /gnu/store/wjmzwngblydz65ylrh7kkgh0mq62lz0m-guix-config-modules.drv...
building /gnu/store/qzhy5lyr56qppryvldghl22db5j66qp6-guix-manual.drv...
building /gnu/store/1jf0d5ch5rpjgbbj7hbh7xwmzbvkp6cm-guix-misc.drv...
building /gnu/store/k2mardcbpcsivc9mygzdgx9xyvvxvq0p-guix-packages-base.drv...
building /gnu/store/6hb81r74zij9vswbh2zqyy9csifv53cm-guix-cli-core.drv...
building /gnu/store/ivfpj6mbx12bizh8v4li6as2xd3sxw0l-guix-packages-base-modules.drv...
building /gnu/store/qs9bax2627scfrv9k5q2k8qa9qfri138-guix-cli-core-modules.drv...
building /gnu/store/z9acy6mp2zbfvn9jzrqs70f4y7x1im3d-guix-packages.drv...
building /gnu/store/crwap92xzmvk3crd2nm3mxv2wfy0b6gg-guix-packages-modules.drv...
building /gnu/store/vfbxyq50j7brlqyai7dr8npbgnivjlnb-guix-system.drv...
building /gnu/store/fcrdfvhnm36x3vlh5j87mvl9frwwxxcj-guix-home.drv...
building /gnu/store/hbz4z9bq05z8kcmp2j686v76n45l2kfl-guix-system-modules.drv...
building /gnu/store/mn5kjnwym7hikdgqkxz5v6q5icr2knp5-guix-cli.drv...
building /gnu/store/2qqzddfp48inn38zsq2bi9if69aia1dk-guix-home-modules.drv...
building /gnu/store/qvf2wkgs2jlqkk983jy7qa7wv9mf4mc2-guix-cli-modules.drv...
building /gnu/store/cbywx3457flfbf135kzn884ik802q2dz-guix-system-tests.drv...
building /gnu/store/zvh0ixd433rmganyrcb1prcx4c9q5axb-guix-system-tests-modules.drv...
building /gnu/store/yc9w06iifsb5l3xikkg7idc4jk9ldsw4-guix-842a11f1c-modules.drv...
building /gnu/store/lvnm196rxcn6d2smz76amj0xj3w73pnk-guix-module-union.drv...
building /gnu/store/7q0mghfzrpd1kb0ckcc9hg2skigq2d4i-guix-command.drv...
building /gnu/store/wg49p4988b3acgq8mcdrmc7p9gcd7i3y-guix-daemon.drv...
building /gnu/store/j2l9k04mp2wdnkvin6xfqxm9j45i1myd-guix-842a11f1c.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
building /gnu/store/8jxl2mxyj7cqaqlq7rr4qxy0nrhj1mra-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
hint: Consider setting the necessary environment variables by running:
GUIX_PROFILE="/tmp/new"
. "$GUIX_PROFILE/etc/profile"
Alternately, see `guix package --search-paths -p "/tmp/new"'.
hint: After setting `PATH', run `hash guix' to make sure your shell refers to `/tmp/new/bin/guix'.
real 10m51.145s
user 2m57.646s
sys 0m1.010s
$ /tmp/new/bin/guix describe
Generation 1 Dec 01 2023 16:33:07 (current)
guix 842a11f
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 842a11f1caa1bb929c427722ad9d7b7c1ff65727
$ time /tmp/new/bin/guix pull --commit=842a11f1caa1bb929c427722ad9d7b7c1ff65727 -p /tmp/again
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git 842a11f
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
[...]
building /gnu/store/k99k4dqavfbpk04bahnv9z36374mnjj1-module-import.drv...
module-import 2KiB 134KiB/s 00:00 ▕██████████████████▏ 100.0%
module-import-compiled 1.2MiB 4.3MiB/s 00:00 ▕██████████████████▏ 100.0%
module-import-compiled 1.2MiB 3.3MiB/s 00:00 ▕██████████████████▏ 100.0%
building /gnu/store/y8g46qfpg7yakglc40avby0grgxc2jb6-compute-guix-derivation.drv...
Computing Guix derivation for 'x86_64-linux'... /
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/shrj2p15crrrsj15yd0yancxq0pwi9qi-profile.drv
/gnu/store/8gwgjd9rq23hpi9hzg05p4fd8lcnyd5y-guix-842a11f1c.drv
/gnu/store/7n5jbjagkqi38gmvf0sayfxvsw6r4imz-guix-command.drv
/gnu/store/82f2w8qgliw8db8aaal9hw4v7mgmzgqk-guix-module-union.drv
/gnu/store/f3zl4nnjdbfj9zl04q1cn7hcrpdgdqvz-guix-842a11f1c-modules.drv
/gnu/store/q16d608sr9nn2l4vdhkbr0b5fwf88238-guix-config-modules.drv
/gnu/store/dlk4i3q2x64p4xqi4w628pbx9xir0jha-guix-config.drv
/gnu/store/hnlw8r17x70wdph7lkmj3jqba9cqn9cg-guix-config-source.drv
/gnu/store/lcw7jwv1sav314b3pys380p4vyy9vr8c-config.scm.drv
/gnu/store/m2pba333pg2c1s0rszvzgrfvvqqzbghi-guix-daemon.drv
/gnu/store/906nmg79jw2jxqdb1sbw77wkm21ddmbd-profile.drv
/gnu/store/dmakvrb0nvpvf6g959q6zggygc88nj6c-inferior-script.scm.drv
building /gnu/store/lcw7jwv1sav314b3pys380p4vyy9vr8c-config.scm.drv...
building /gnu/store/hnlw8r17x70wdph7lkmj3jqba9cqn9cg-guix-config-source.drv...
building /gnu/store/dlk4i3q2x64p4xqi4w628pbx9xir0jha-guix-config.drv...
building /gnu/store/q16d608sr9nn2l4vdhkbr0b5fwf88238-guix-config-modules.drv...
building /gnu/store/f3zl4nnjdbfj9zl04q1cn7hcrpdgdqvz-guix-842a11f1c-modules.drv...
building /gnu/store/82f2w8qgliw8db8aaal9hw4v7mgmzgqk-guix-module-union.drv...
building /gnu/store/7n5jbjagkqi38gmvf0sayfxvsw6r4imz-guix-command.drv...
building /gnu/store/m2pba333pg2c1s0rszvzgrfvvqqzbghi-guix-daemon.drv...
building /gnu/store/8gwgjd9rq23hpi9hzg05p4fd8lcnyd5y-guix-842a11f1c.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
building /gnu/store/dmakvrb0nvpvf6g959q6zggygc88nj6c-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
hint: Consider setting the necessary environment variables by running:
GUIX_PROFILE="/tmp/again"
. "$GUIX_PROFILE/etc/profile"
Alternately, see `guix package --search-paths -p "/tmp/again"'.
hint: After setting `PATH', run `hash guix' to make sure your shell refers to `/tmp/again/bin/guix'.
real 2m5.273s
user 3m0.652s
sys 0m1.315s
$ time /tmp/new/bin/guix pull --commit=842a11f1caa1bb929c427722ad9d7b7c1ff65727 -p /tmp/again
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git 842a11f
Computing Guix derivation for 'x86_64-linux'... \
nothing to be done
hint: After setting `PATH', run `hash guix' to make sure your shell refers to `/tmp/again/bin/guix'.
real 1m39.950s
user 2m57.294s
sys 0m1.195s
--8<---------------cut here---------------end--------------->8---
> Or might there actually be someting wrong with my setup, and in this case how
> could I diagnose it ?
Well, ~40min seems something wrong with your setup. Considering “guix
pull”, what is the part (from a quick look) that appears to you the
longest?
Cheers,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-12-01 17:17 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-10-12 11:03 guix pull performance Emmanuel Beffara
2023-10-12 11:24 ` Ignas Lapėnas
2023-10-12 13:51 ` Emmanuel Beffara
2023-10-12 17:02 ` Simon Tournier
2023-11-28 21:07 ` Emmanuel Beffara
2023-12-01 17:13 ` Simon Tournier
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.