guix package is slow

guix package is slow

[swedebugia]

Hi

I would like guix package -A/s to be faster.
There it takes >3s every time I run the command.

Would it be possible to populate an index or something when pulling so
that we have the information in a sqlite-db instead of traversing 40
go-files with guile?

On an older guixsd with about 4k packages the delay was noticeably
shorter. 

What do you think?

Re: guix package is slow

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 348 bytes --]


I've implemented a rather trivial cache in `helm-system-packages' (an
Emacs package).
the cache is created (or refreshed if it's older than the
~/.config/guix/current).
Cache generation takes a few seconds, but from then on everything is
instantaneous.

It does not support searching within descriptions however.

-- 
Pierre Neidhardt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix package is slow

[Maxim Cournoyer]

Hello Pierre,

Pierre Neidhardt <ambrevar@gmail.com> writes:

> I've implemented a rather trivial cache in `helm-system-packages' (an
> Emacs package).
> the cache is created (or refreshed if it's older than the
> ~/.config/guix/current).
> Cache generation takes a few seconds, but from then on everything is
> instantaneous.
>
> It does not support searching within descriptions however.

Could you elaborate on how you create and then use such cache? Is it
useful when used inside Emacs only, say in a *shell* buffer?

If so, what does it bring over using emacs-guix, which caches in memory
the package definitions?

Thanks,

Maxim

Re: guix package is slow

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1135 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Could you elaborate on how you create and then use such cache?

It's very dumb, really: it just drops the output of `guix package
--search "."` to a file :D

> Is it useful when used inside Emacs only, say in a *shell* buffer?

Well, you can use `recsel' or any text processing tool over it, but
beside that, not much I'm afraid.

> If so, what does it bring over using emacs-guix, which caches in memory
> the package definitions?

In terms of caching, nothing, but the Helm interface has some
significant perks over emacs-guix:

- Perform transactions (install/remove) over multiple packages.

- Perform actions over multiple packages: browse home-page, show
(reverse) deps, find-files (with helm-find-files -- awesome! :p).

That's about it.  It's much more basic than emacs-guix but convenient
enough that I still find some use for it :)

That said, this `helm-system-packages-guix.el` was mostly a proof of
concept, it's far from being as neat and complete as emacs-guix.
A proper Helm/Ivy interface for emacs-guix would be much better,
maybe some day.

--
Pierre Neidhardt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix package is slow

[Maxim Cournoyer]

Hello Pierre!

Pierre Neidhardt <ambrevar@gmail.com> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Could you elaborate on how you create and then use such cache?
>
> It's very dumb, really: it just drops the output of `guix package
> --search "."` to a file :D
>
>> Is it useful when used inside Emacs only, say in a *shell* buffer?
>
> Well, you can use `recsel' or any text processing tool over it, but
> beside that, not much I'm afraid.

I see :)

>> If so, what does it bring over using emacs-guix, which caches in memory
>> the package definitions?
>
> In terms of caching, nothing, but the Helm interface has some
> significant perks over emacs-guix:
>
> - Perform transactions (install/remove) over multiple packages.

To be fair, I think you can already accomplish this using emacs-guix by
separating with commas multiple package names :).

> - Perform actions over multiple packages: browse home-page, show
> (reverse) deps, find-files (with helm-find-files -- awesome! :p).

This sounds interesting -- I guess I should read more about helm.

Thank you for taking the time to explain!

Maxim

Re: guix package is slow

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1197 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>> - Perform transactions (install/remove) over multiple packages.
>
> To be fair, I think you can already accomplish this using emacs-guix by
> separating with commas multiple package names :).

Can you explain?  I don't know how to do that.

The point of the Helm interface is that it allows to "batch select"
multiple packages.  I don't think that emacs-guix can do that.

For instance, if I want to install all Emacs packages except, say,
emacs-guix (haha! :p) then I would

- M-x helm-system-packages
- "emacs"
- M-a
- "emacs-guix"
- C-space
- <f2>

In other words, it scales well with big loads of packages.

>> - Perform actions over multiple packages: browse home-page, show
>> (reverse) deps, find-files (with helm-find-files -- awesome! :p).
>
> This sounds interesting -- I guess I should read more about helm.

Yup, it's hard to explain with words and not verbal explanation can
compete with a demo.  If you've never used Helm/Ivy before, wait no
further, it's a must (in my humble opinion, of course!).

> Thank you for taking the time to explain!

My pleasure, let me know if you want to know more.

Cheers!

--
Pierre Neidhardt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix package is slow

[Maxim Cournoyer]

Hi Pierre,

Pierre Neidhardt <ambrevar@gmail.com> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>>> - Perform transactions (install/remove) over multiple packages.
>>
>> To be fair, I think you can already accomplish this using emacs-guix by
>> separating with commas multiple package names :).
>
> Can you explain?  I don't know how to do that.

Sure! It's briefly covered in the Emacs-Guix info manual, in the 'Popup
Interface' section:

--8<---------------cut here---------------start------------->8---
So ‘M-x guix’ command provides a top-level popup interface for all
available guix commands.  When you select an option, you’ll be prompted
for a value in the minibuffer.  Many values have completions, so don’t
hesitate to press ‘<TAB>’ key.  Multiple values (for example, packages
or lint checkers) should be separated by commas.
--8<---------------cut here---------------end--------------->8---

A concrete example would look like:

1. Start Emacs
2. M-x guix
3. p
4. =i
5. emacs-helm,emacs-ivy ENT

You can tab-complete any package names entered at step 5. above.

> The point of the Helm interface is that it allows to "batch select"
> multiple packages.  I don't think that emacs-guix can do that.

You are right, it doesn't scale like this; one would have to resort
using 'guix package -A=some-regexp', calling `guix package -i' on each
element.

> For instance, if I want to install all Emacs packages except, say,
> emacs-guix (haha! :p) then I would
>
> - M-x helm-system-packages
> - "emacs"
> - M-a
> - "emacs-guix"
> - C-space
> - <f2>
>
> In other words, it scales well with big loads of packages.

Neat! Thanks for sharing.

Maxim

Re: guix package is slow

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 867 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> --8<---------------cut here---------------start------------->8---
> So ‘M-x guix’ command provides a top-level popup interface for all
> available guix commands.  When you select an option, you’ll be prompted
> for a value in the minibuffer.  Many values have completions, so don’t
> hesitate to press ‘<TAB>’ key.  Multiple values (for example, packages
> or lint checkers) should be separated by commas.
> --8<---------------cut here---------------end--------------->8---
>
> A concrete example would look like:
>
> 1. Start Emacs
> 2. M-x guix
> 3. p
> 4. =i
> 5. emacs-helm,emacs-ivy ENT
>
> You can tab-complete any package names entered at step 5. above.

Brilliant, I had completely forgotten about that!

> Neat! Thanks for sharing.

Same!

-- 
Pierre Neidhardt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix package is slow

[Oleg Pykhalov]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

Hello,

Pierre Neidhardt <ambrevar@gmail.com> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>>> - Perform transactions (install/remove) over multiple packages.
>>
>> To be fair, I think you can already accomplish this using emacs-guix by
>> separating with commas multiple package names :).
>
> Can you explain?  I don't know how to do that.
>
> The point of the Helm interface is that it allows to "batch select"
> multiple packages.  I don't think that emacs-guix can do that.

[…]

Another way in ‘M-x guix-all-packages’ or any ‘guix-search-by-*’:

  • Hit ‘i’ on not installed packages.
  • Hit ‘d’ on installed package.
  • Hit ‘x’ to apply.


Oleg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: guix package is slow

[Maxim Cournoyer]

Hi,

Oleg Pykhalov <go.wigust@gmail.com> writes:

> Hello,
>
> Pierre Neidhardt <ambrevar@gmail.com> writes:
>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>
>>>> - Perform transactions (install/remove) over multiple packages.
>>>
>>> To be fair, I think you can already accomplish this using emacs-guix by
>>> separating with commas multiple package names :).
>>
>> Can you explain?  I don't know how to do that.
>>
>> The point of the Helm interface is that it allows to "batch select"
>> multiple packages.  I don't think that emacs-guix can do that.
>
> […]
>
> Another way in ‘M-x guix-all-packages’ or any ‘guix-search-by-*’:
>
>   • Hit ‘i’ on not installed packages.
>   • Hit ‘d’ on installed package.
>   • Hit ‘x’ to apply.

Thanks Oleg! I keep forgetting about this interface; there doesn't seem
to be an entry point from M-x guix for easy discovery.

Maxim

Re: guix package is slow

[Alex Kost]

Maxim Cournoyer (2018-07-02 10:55 -0400) wrote:

> Hi,
>
> Oleg Pykhalov <go.wigust@gmail.com> writes:
>
>> Hello,
>>
>> Pierre Neidhardt <ambrevar@gmail.com> writes:
>>
>>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>>
>>>>> - Perform transactions (install/remove) over multiple packages.
>>>>
>>>> To be fair, I think you can already accomplish this using emacs-guix by
>>>> separating with commas multiple package names :).
>>>
>>> Can you explain?  I don't know how to do that.
>>>
>>> The point of the Helm interface is that it allows to "batch select"
>>> multiple packages.  I don't think that emacs-guix can do that.
>>
>> […]
>>
>> Another way in ‘M-x guix-all-packages’ or any ‘guix-search-by-*’:
>>
>>   • Hit ‘i’ on not installed packages.
>>   • Hit ‘d’ on installed package.
>>   • Hit ‘x’ to apply.
>
> Thanks Oleg! I keep forgetting about this interface;

he-he, it is only one of about 10 different interfaces of this kind (for
packages, profiles, generations, services, etc.).  If you use only "M-x
guix", you miss most of Emacs-Guix features.

> there doesn't seem
> to be an entry point from M-x guix for easy discovery.

"M-x guix" can't be an entry point for this or any other interface, it
is only for "guix" *shell* commands.  Actually I'm very surprised
someone uses "M-x guix", I find it unpractical.  I even plan to rename
it to "M-x guix-command", and to make "M-x guix" a real entry point for
the various Emacs-Guix commands (including "guix-command" and all the
interfaces).

-- 
Alex

Re: guix package is slow

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]



On July 2, 2018 10:28:01 PM GMT+02:00, Alex Kost <alezost@gmail.com> wrote:
>Maxim Cournoyer (2018-07-02 10:55 -0400) wrote:
>
>> Hi,
>>
>> Oleg Pykhalov <go.wigust@gmail.com> writes:
>>
>>> Hello,
>>>
>>> Pierre Neidhardt <ambrevar@gmail.com> writes:
>>>
>>>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>>>
>>>>>> - Perform transactions (install/remove) over multiple packages.
>>>>>
>>>>> To be fair, I think you can already accomplish this using
>emacs-guix by
>>>>> separating with commas multiple package names :).
>>>>
>>>> Can you explain?  I don't know how to do that.
>>>>
>>>> The point of the Helm interface is that it allows to "batch select"
>>>> multiple packages.  I don't think that emacs-guix can do that.
>>>
>>> […]
>>>
>>> Another way in ‘M-x guix-all-packages’ or any ‘guix-search-by-*’:
>>>
>>>   • Hit ‘i’ on not installed packages.
>>>   • Hit ‘d’ on installed package.
>>>   • Hit ‘x’ to apply.
>>
>> Thanks Oleg! I keep forgetting about this interface;
>
>he-he, it is only one of about 10 different interfaces of this kind
>(for
>packages, profiles, generations, services, etc.).  If you use only "M-x
>guix", you miss most of Emacs-Guix features.
>
>> there doesn't seem
>> to be an entry point from M-x guix for easy discovery.
>
>"M-x guix" can't be an entry point for this or any other interface, it
>is only for "guix" *shell* commands.  Actually I'm very surprised
>someone uses "M-x guix", I find it unpractical.  I even plan to rename
>it to "M-x guix-command", and to make "M-x guix" a real entry point for
>the various Emacs-Guix commands (including "guix-command" and all the
>interfaces).
>
>-- 
>Alex

Good idea! 
-- 
Cheers Swedebugia 

[-- Attachment #2: Type: text/html, Size: 2791 bytes --]

Re: guix package is slow

[Ludovic Courtès]

Alex Kost <alezost@gmail.com> skribis:

> "M-x guix" can't be an entry point for this or any other interface, it
> is only for "guix" *shell* commands.  Actually I'm very surprised
> someone uses "M-x guix", I find it unpractical.  I even plan to rename
> it to "M-x guix-command", and to make "M-x guix" a real entry point for
> the various Emacs-Guix commands (including "guix-command" and all the
> interfaces).

+1!

Ludo’.

Re: guix package is slow

[Maxim Cournoyer]

Hello!

Alex Kost <alezost@gmail.com> writes:

[...]

>>> Another way in ‘M-x guix-all-packages’ or any ‘guix-search-by-*’:
>>>
>>>   • Hit ‘i’ on not installed packages.
>>>   • Hit ‘d’ on installed package.
>>>   • Hit ‘x’ to apply.
>>
>> Thanks Oleg! I keep forgetting about this interface;
>
> he-he, it is only one of about 10 different interfaces of this kind (for
> packages, profiles, generations, services, etc.).  If you use only "M-x
> guix", you miss most of Emacs-Guix features.

Good to know! I'll investigate the various guix-* commands from now :)

>> there doesn't seem
>> to be an entry point from M-x guix for easy discovery.
>
> "M-x guix" can't be an entry point for this or any other interface, it
> is only for "guix" *shell* commands.  Actually I'm very surprised
> someone uses "M-x guix", I find it unpractical.

My main usage of M-x guix seems to be M-x guix e(dit to find package
definitions or see if a package already exists which is often quicker
than looking at the often overwhelming 'guix package --search' output.

> it to "M-x guix-command", and to make "M-x guix" a real entry point for
> the various Emacs-Guix commands (including "guix-command" and all the
> interfaces).

Sounds good!

Maxim

Re: guix package is slow

[George Clemmer]

Alex Kost <alezost@gmail.com> writes:

> "M-x guix" can't be an entry point for this or any other interface, it
> is only for "guix" *shell* commands.  Actually I'm very surprised
> someone uses "M-x guix", I find it unpractical.

Agreed. FWIW, when I am trying to find an emacs-guix command I type 'M-x
guix-' and search the completions. But sometimes I forget the '-' and
when this pops up I am like ... UGH ;-(

I wanted to use and like the "M-x Guix" pop-up, I really did. I figured
the Guix CLI needed all the help it could get ;-) I tried several times
to learn this, but I just couldn't get the hang of it. I also tried it
in magit and it didn't work for me there.  This makes me wonder: a) Am I
am a dope? b) Do people really use this?

> I even plan to rename it to "M-x guix-command", and to make "M-x guix"
> a real entry point for the various Emacs-Guix commands (including
> "guix-command" and all the interfaces).

That's a great idea.

If "M-x guix" is not widely used you might consider deleting it
altogether.

- George

Re: guix package is slow

[Oleg Pykhalov]

[-- Attachment #1: Type: text/plain, Size: 223 bytes --]

George Clemmer <myglc2@gmail.com> writes:

[…]

> If "M-x guix" is not widely used you might consider deleting it
> altogether.

I guess we couldn't count wide of usage. I use ‘M-x guix’ command. :-)

Oleg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: guix package is slow

[Alex Kost]

Pierre Neidhardt (2018-07-01 21:41 +0200) wrote:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>>> - Perform transactions (install/remove) over multiple packages.
>>
>> To be fair, I think you can already accomplish this using emacs-guix by
>> separating with commas multiple package names :).
>
> Can you explain?  I don't know how to do that.
>
> The point of the Helm interface is that it allows to "batch select"
> multiple packages.  I don't think that emacs-guix can do that.
>
> For instance, if I want to install all Emacs packages except, say,
> emacs-guix (haha! :p) then I would
>
> - M-x helm-system-packages
> - "emacs"
> - M-a
> - "emacs-guix"
> - C-space
> - <f2>

As Oleg wrote, in Emacs-Guix you can use an interface for packages: "M-x
guix-search-by-name emacs", then mark whatever you want to install with
"i" and press "x" to run the operation.

-- 
Alex

Re: guix package is slow

[Maxim Cournoyer]

Hello!

swedebugia@riseup.net writes:

> Hi
>
> I would like guix package -A/s to be faster.
> There it takes >3s every time I run the command.

I agree that Guix is rather slow when comparing it to traditional
package managers such as apt; I think the main reason, as you found, is
that it has to load all the byte-compiled package definition files.

> Would it be possible to populate an index or something when pulling so
> that we have the information in a sqlite-db instead of traversing 40
> go-files with guile?

I remember asking Ludovic in #guix if Guile would be able to compete
with database software in terms of performance, and he said there are
ways we can make it faster. I'm not very knowledgeable myself about
Guile but if we could make it support static linking, to load only one
binary rather than hundreds in the case of Guix, that would be one way
to speed things. Next thing might be to implement native compilation.

> On an older guixsd with about 4k packages the delay was noticeably
> shorter. 

I guess the time it takes grows linearly (I hope!) with the number of
packages/modules it has to load. We're now at double that amount of
packages (IIRC), so Guix might take double the time it used to.

> What do you think?

Note that if you are using emacs-guix, the packages are kept in memory
so the operations there are much faster than when using the CLI.

Using an external databases to speed Guix is an interesting idea, but Id
rather have a lightning fast Guile in the longer term :)

Maxim

Re: guix package is slow

[Ludovic Courtès]

Hello!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> swedebugia@riseup.net writes:
>
>> Hi
>>
>> I would like guix package -A/s to be faster.
>> There it takes >3s every time I run the command.

It’s at 1.5s for me on an SSD and a hot cache, but I agree that overall
it’s too slow.

> I agree that Guix is rather slow when comparing it to traditional
> package managers such as apt; I think the main reason, as you found, is
> that it has to load all the byte-compiled package definition files.

That, and also it generally has “more work” to do (see ‘apt-get update’
vs. ‘guix pull’, or even ‘guix package -i emacs’ vs. ‘apt-get install
emacs’.)  That’s not meant as an excuse, just a clarification of what’s
going on.

>> Would it be possible to populate an index or something when pulling so
>> that we have the information in a sqlite-db instead of traversing 40
>> go-files with guile?
>
> I remember asking Ludovic in #guix if Guile would be able to compete
> with database software in terms of performance, and he said there are
> ways we can make it faster.

I think we could easily optimize package lookups by name.  Instead of
traversing the whole list of package modules to build up a name/package
table, we could have a pre-built cache containing that mapping.  That
way, when typing “guix build emacs”, we could essentially translate that
to “guix build -e '(@ (gnu packages emacs) emacs)'”, which is a little
bit faster.

We could arrange for the cache to be enough for operations such as “guix
package -A”, so that we don’t need to load a single package module.

> I'm not very knowledgeable myself about Guile but if we could make it
> support static linking, to load only one binary rather than hundreds
> in the case of Guix, that would be one way to speed things. Next thing
> might be to implement native compilation.

You might like this:

  https://lists.gnu.org/archive/html/guile-devel/2018-06/msg00026.html

:-)

>> On an older guixsd with about 4k packages the delay was noticeably
>> shorter. 
>
> I guess the time it takes grows linearly (I hope!) with the number of
> packages/modules it has to load. We're now at double that amount of
> packages (IIRC), so Guix might take double the time it used to.

Yeah, that’s why the cache is probably unavoidable.

Thanks,
Ludo’.

Re: guix package is slow

[Maxim Cournoyer]

Hi!

ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> swedebugia@riseup.net writes:
>>
>>> Hi
>>>
>>> I would like guix package -A/s to be faster.
>>> There it takes >3s every time I run the command.
>
> It’s at 1.5s for me on an SSD and a hot cache, but I agree that overall
> it’s too slow.
>
>> I agree that Guix is rather slow when comparing it to traditional
>> package managers such as apt; I think the main reason, as you found, is
>> that it has to load all the byte-compiled package definition files.
>
> That, and also it generally has “more work” to do (see ‘apt-get update’
> vs. ‘guix pull’, or even ‘guix package -i emacs’ vs. ‘apt-get install
> emacs’.)  That’s not meant as an excuse, just a clarification of what’s
> going on.

True, especially for guix pull ;)

>>> Would it be possible to populate an index or something when pulling so
>>> that we have the information in a sqlite-db instead of traversing 40
>>> go-files with guile?
>>
>> I remember asking Ludovic in #guix if Guile would be able to compete
>> with database software in terms of performance, and he said there are
>> ways we can make it faster.
>
> I think we could easily optimize package lookups by name.  Instead of
> traversing the whole list of package modules to build up a name/package
> table, we could have a pre-built cache containing that mapping.  That
> way, when typing “guix build emacs”, we could essentially translate that
> to “guix build -e '(@ (gnu packages emacs) emacs)'”, which is a little
> bit faster.
>
> We could arrange for the cache to be enough for operations such as “guix
> package -A”, so that we don’t need to load a single package module.
>
>> I'm not very knowledgeable myself about Guile but if we could make it
>> support static linking, to load only one binary rather than hundreds
>> in the case of Guix, that would be one way to speed things. Next thing
>> might be to implement native compilation.
>
> You might like this:
>
>   https://lists.gnu.org/archive/html/guile-devel/2018-06/msg00026.html

Interesting developments are coming to Guile, it seems :). I'm happy to
see that Andy is still driving Guile forward. Thanks!

>>> On an older guixsd with about 4k packages the delay was noticeably
>>> shorter. 
>>
>> I guess the time it takes grows linearly (I hope!) with the number of
>> packages/modules it has to load. We're now at double that amount of
>> packages (IIRC), so Guix might take double the time it used to.
>
> Yeah, that’s why the cache is probably unavoidable.

Thanks for sharing your insights on the matter!

Maxim

‘package-transitive-supported-systems’ is slow

[Ludovic Courtès]

(+Cc: Mark, who’s looked at this part of the code before.)

swedebugia@riseup.net skribis:

> I would like guix package -A/s to be faster.
> There it takes >3s every time I run the command.

On closer inspection, ‘package-transitive-supported-systems’ is
responsible for most of the overhead you’re seeing here.

On my laptop, if I define it like this in (guix packages):

  (define package-transitive-supported-systems
    (const '("x86_64-linux")))

then ‘guix package -A’ drops from 1.5s to 0.6s.

The main cost is calling ‘package->bag’, which amounts to doing half of
the work towards computing a package’s derivation.  This particular
change was introduced in 9bf3ced06c42700d6c83ce3a0eda244798104618 so
that ‘package-transitive-supported-systems’ would account for implicit
inputs.

I can’t think of any simple way to optimize it without changing these
semantics.  Mark?

Ludo’.