Including code in a non-Guile language into Guix

Including code in a non-Guile language into Guix

[Evan Cooney]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

Hello everyone,

My name is Evan Cooney. I use GNU Guix but I'm pretty new to it, I'm also
not a professional, I mostly program for fun and I'm mostly self-taught. I
was watching some videos about software performance and optimization, which
brought to mind that, at least on my computer, guix package commands
(particularly search, install, and remove) run much slower than the
equivalents on other distros I've used. I know that Guix is mainly written
in Guile, but has much thought gone into optimizing these commands by
rewriting some of the code in a more performance-oriented language like C?
Guile is designed for interoperability with C code, and many other GNU
programs (gcc, coreutils, gdb, bash, etc.) are written in C or similar
languages. C is also more widely taught and known than Guile, so adding C
might help bring developers to Guix. I would love to try to work on
optimizing the performance of these commands and any help would
be appreciated.

Thanks,
Evan Cooney

[-- Attachment #2: Type: text/html, Size: 1108 bytes --]

Re: Including code in a non-Guile language into Guix

[Daniel Littlewood]

[-- Attachment #1: Type: text/plain, Size: 3470 bytes --]

Hi Evan,

Some guix commands are indeed quite slow. This has been the subject of
discussion on the mailing list previously. I found one example from a year
ago, but I guess there are others:

https://lists.gnu.org/archive/html/guix-devel/2023-06/msg00085.html

That being said, I would recommend caution in your conclusions for two
reasons:

1. Most distributions are not a like-for-like comparison with Guix. The
only distro I would consider directly comparing with in terms of
performance would be NixOS. Installation of software on guix requires
strictly more work than, say, on Debian or Arch, because it provides more
guarantees than those distributions do in terms of reproducibility. Others
on the list can probably speak more to the details of why or how much
slower it should be expected to be. However, Guix *is* noticeably slower
than Nix is, so your observation is still correct.

2. Programme performance is a complex topic, and it is a common
misconception that programs written in C are always, or typically, faster
than counterparts in garbage-collected languages. This is not really true
in many cases, particularly when costs of development are considered.

I assume that having a C core, or splitting out parts of Guix into C, was
considered early on in development. After all it is kind of the point of
guile that it interfaces well with C. Other members of the lost may be able
to give the historical context.

Having said that, I wonder (and could not find out) whether there is
quantitative data on how fast/slow Guix is for common operations. If you
could do some timings of what you think "too slow" means (e.g. for a guix
pull, or guix shell PKG).

Some data from me, using Guix on a foreign distro (Debian)

guix pull ("38k new commits"): 21m45s
guix pull immediately after: 2m25s
guix shell emacs (fresh): 1m49s
guix shell emacs (cached): <1s

apt update: 2s
apt install emacs: 2s

nix-channel --update: 0m23s
nix shell -p emacs (fresh): 0m24s
nix-shell -p emacs (cached): 4s

The apt commands are really not a fair comparison. In particular I noticed
the Guix command had to pull in all dependencies, while apt seemingly had
everything readily available. Note that I did apt remove emacs before
installing, but nothing else.

I'm not sure whether the nix/guix comparisons are fair. I might have more
installed in Guix vs nix.

All the best,
Dan

On Thu, Oct 31, 2024, 06:50 Evan Cooney <evancooney71@gmail.com> wrote:

> Hello everyone,
>
> My name is Evan Cooney. I use GNU Guix but I'm pretty new to it, I'm also
> not a professional, I mostly program for fun and I'm mostly self-taught. I
> was watching some videos about software performance and optimization, which
> brought to mind that, at least on my computer, guix package commands
> (particularly search, install, and remove) run much slower than the
> equivalents on other distros I've used. I know that Guix is mainly written
> in Guile, but has much thought gone into optimizing these commands by
> rewriting some of the code in a more performance-oriented language like C?
> Guile is designed for interoperability with C code, and many other GNU
> programs (gcc, coreutils, gdb, bash, etc.) are written in C or similar
> languages. C is also more widely taught and known than Guile, so adding C
> might help bring developers to Guix. I would love to try to work on
> optimizing the performance of these commands and any help would
> be appreciated.
>
> Thanks,
> Evan Cooney
>

[-- Attachment #2: Type: text/html, Size: 4766 bytes --]

Re: Including code in a non-Guile language into Guix

[Suhail Singh]

Daniel Littlewood <danielittlewood@gmail.com> writes:

> guix pull ("38k new commits"): 21m45s
> guix pull immediately after: 2m25s
> guix shell emacs (fresh): 1m49s
> ...
>
> nix-channel --update: 0m23s
> nix shell -p emacs (fresh): 0m24s

Those are some interesting comparisons.  Is the reason guix pull takes
so long as compared to updating nix-channel primarily due to the
authentication of commits?  Or something else?

-- 
Suhail

Re: Including code in a non-Guile language into Guix

[Ricardo Wurmus]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Daniel Littlewood <danielittlewood@gmail.com> writes:
>
>> guix pull ("38k new commits"): 21m45s
>> guix pull immediately after: 2m25s
>> guix shell emacs (fresh): 1m49s
>> ...
>>
>> nix-channel --update: 0m23s
>> nix shell -p emacs (fresh): 0m24s
>
> Those are some interesting comparisons.  Is the reason guix pull takes
> so long as compared to updating nix-channel primarily due to the
> authentication of commits?  Or something else?

It's "something else".  This is a comparison between apples and
giraffes.  "guix pull" does a different job than "nix-channel"; the
latter only needs to download a new version of inert data whereas the
former computes a trampoline and then updates Guix itself.

The fundamental difference is that Guix is a library, not merely
"package metadata" that would be independent of the Guix executable.

Comparing "guix shell" and "nix shell" is fair game, though, but I
cannot reproduce the above numbers.  Here's my crude test:

    time guix shell --no-substitutes emacs-minimal -- ls
    real	0m2.386s
    user	0m1.489s
    sys 	0m0.141s

This is without the "guix shell" cache, but with Emacs present in /gnu/store.

-- 
Ricardo

Re: Including code in a non-Guile language into Guix

[Ricardo Wurmus]

Evan Cooney <evancooney71@gmail.com> writes:

> I know that Guix is mainly written in Guile, but has much thought gone
> into optimizing these commands

Yes.

> by rewriting some of the
> code in a more performance-oriented language like C?

No.

Guile is not the bottleneck.

> C is also more widely taught and known than Guile, so adding C might
> help bring developers to Guix.

I find this doubtful.

-- 
Ricardo

Re: Including code in a non-Guile language into Guix

[Runciter via Development of GNU Guix and the GNU System distribution.]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Daniel Littlewood <danielittlewood@gmail.com> writes:
>
>> guix pull ("38k new commits"): 21m45s
>> guix pull immediately after: 2m25s
>> guix shell emacs (fresh): 1m49s
>> ...
>>
>> nix-channel --update: 0m23s
>> nix shell -p emacs (fresh): 0m24s
>
> Those are some interesting comparisons.  Is the reason guix pull takes
> so long as compared to updating nix-channel primarily due to the
> authentication of commits?  Or something else?

As far as the local machine computations go, clearly, authenticating the
commits is not the bottleneck. On all machines, indexing the received
git objects locally is much longer than authenticating the commits.

On my X60, when I pull for the first time after I delete the cache, the
indexing step alone takes more than 40 minutes.

The 2m25s that Daniel had for his second git pull, that had to be spent
mostly on computing the Guix derivation. This time is in large part
in-compressible I guess. Not that I know of a lot about this, but by
reading the output, it's clear that every time guix pull has to compute
the whole derivation of the latest commit of all the
channels. Apparently, in a pull where Guix determines that it has
nothing to do, this step is required before Guix can make the
determination that it has nothing to do...

Re: Including code in a non-Guile language into Guix

[Evan Cooney]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

Would it be useful for me to wrote a script to collect performance data for
various stages? Like downloading, computing derivations, building profile
etc?

On Thu, Oct 31, 2024, 11:18 AM Runciter via Development of GNU Guix and the
GNU System distribution. <guix-devel@gnu.org> wrote:

> Suhail Singh <suhailsingh247@gmail.com> writes:
>
> > Daniel Littlewood <danielittlewood@gmail.com> writes:
> >
> >> guix pull ("38k new commits"): 21m45s
> >> guix pull immediately after: 2m25s
> >> guix shell emacs (fresh): 1m49s
> >> ...
> >>
> >> nix-channel --update: 0m23s
> >> nix shell -p emacs (fresh): 0m24s
> >
> > Those are some interesting comparisons.  Is the reason guix pull takes
> > so long as compared to updating nix-channel primarily due to the
> > authentication of commits?  Or something else?
>
> As far as the local machine computations go, clearly, authenticating the
> commits is not the bottleneck. On all machines, indexing the received
> git objects locally is much longer than authenticating the commits.
>
> On my X60, when I pull for the first time after I delete the cache, the
> indexing step alone takes more than 40 minutes.
>
> The 2m25s that Daniel had for his second git pull, that had to be spent
> mostly on computing the Guix derivation. This time is in large part
> in-compressible I guess. Not that I know of a lot about this, but by
> reading the output, it's clear that every time guix pull has to compute
> the whole derivation of the latest commit of all the
> channels. Apparently, in a pull where Guix determines that it has
> nothing to do, this step is required before Guix can make the
> determination that it has nothing to do...
>
>
>

[-- Attachment #2: Type: text/html, Size: 2286 bytes --]

Re: Including code in a non-Guile language into Guix

[Suhail Singh]

Ricardo Wurmus <rekado@elephly.net> writes:

>>> guix pull ("38k new commits"): 21m45s
>>> guix pull immediately after: 2m25s
>>> ...
>>>
>>> nix-channel --update: 0m23s
>>
>> Those are some interesting comparisons.  Is the reason guix pull takes
>> so long as compared to updating nix-channel primarily due to the
>> authentication of commits?  Or something else?
>
> It's "something else".  This is a comparison between apples and
> giraffes.  "guix pull" does a different job than "nix-channel"; the
> latter only needs to download a new version of inert data whereas the
> former computes a trampoline and then updates Guix itself.
>
> The fundamental difference is that Guix is a library, not merely
> "package metadata" that would be independent of the Guix executable.

Thank you for that explanation.

-- 
Suhail