compiling guix is too slow?

compiling guix is too slow?

[Feng Shu]

Now I have found that 'guix pull' is too slow,
I need 3 hours to compile guix, is it possible to speed it ?

-- 

Re: compiling guix is too slow?

[Adonay Felipe Nogueira]

I'm also trying to understand the cause of this.

I still don't have enough time to begin investigating it or to suggest
improvements, but I notice that it gets worse and with *more* resource
usage if you pass the `--cores`/`-c` option with any value to `guix
pull`. Without this option the resource usage is short (in terms of how
long it lasts) and causes less heat.

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.

Re: compiling guix is too slow?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Thu, Jun 29, 2017 at 06:58:28AM +0800, Feng Shu wrote:
> 
> Now I have found that 'guix pull' is too slow,
> I need 3 hours to compile guix, is it possible to speed it ?

Wow, that's really slow :(

I don't know of any easy way to speed it up. Maybe you could build Guix
on a faster computer and use `guix copy` to move it to this slower
computer.

My understanding is that Guile 2.2 traded slower compilation for faster
execution of compiled programs. Hopefully the performance of the
compiler will be improved in later updates to Guile.

I'm curious, what machine are you using where it takes 3 hours? My
slowest machine takes 100 minutes, which is also too slow, but it's much
better than 3 hours.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: compiling guix is too slow?

[Ludovic Courtès]

Hi,

Leo Famulari <leo@famulari.name> skribis:

> My understanding is that Guile 2.2 traded slower compilation for faster
> execution of compiled programs. Hopefully the performance of the
> compiler will be improved in later updates to Guile.

Yes, that’s a good summary.

Most of the code we compile is package definitions that don’t benefit
from all the fancy optimizations, so build-aux/build-self.scm (run by
‘guix pull’) turns those off.

Unfortunately even with this change compilation is slower than with
Guile 2.0, and IMO unacceptably slow for ‘guix pull’ (it takes only a
few minutes at most on my laptop, but that’s a lot if we compare to
‘apt-get update’.)

Andy, do you have other approaches in mind that we could explore to
improve on this?  Evaluating all of gnu/packages instead of compiling it
AOT would probably be bad for memory and CPU consumption.

Thanks,
Ludo’.

Re: compiling guix is too slow?

[Andy Wingo]

On Sat 01 Jul 2017 15:33, ludo@gnu.org (Ludovic Courtès) writes:

> Leo Famulari <leo@famulari.name> skribis:
>
>> My understanding is that Guile 2.2 traded slower compilation for faster
>> execution of compiled programs. Hopefully the performance of the
>> compiler will be improved in later updates to Guile.
>
> Yes, that’s a good summary.
>
> Most of the code we compile is package definitions that don’t benefit
> from all the fancy optimizations, so build-aux/build-self.scm (run by
> ‘guix pull’) turns those off.
>
> Unfortunately even with this change compilation is slower than with
> Guile 2.0, and IMO unacceptably slow for ‘guix pull’ (it takes only a
> few minutes at most on my laptop, but that’s a lot if we compare to
> ‘apt-get update’.)
>
> Andy, do you have other approaches in mind that we could explore to
> improve on this?  Evaluating all of gnu/packages instead of compiling it
> AOT would probably be bad for memory and CPU consumption.

I agree that this is a correct summary.  I think it's possible that even
with improvements that 2.2 would remain slower than 2.0 to compile, but
the current situation is clearly not good and needs improvement.  I
think we need to revisit that synthetic python.scm test you had (can't
find it atm), see where the compiler is spending time, and focus effort
there.

As far as what Guix can do -- there I don't know :/  Nothing immediate
comes to mind.

I would like to help here but can't promise much over the summertime...

Andy

Re: compiling guix is too slow?

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1974 bytes --]

Andy Wingo transcribed 1.5K bytes:
> On Sat 01 Jul 2017 15:33, ludo@gnu.org (Ludovic Courtès) writes:
> 
> > Leo Famulari <leo@famulari.name> skribis:
> >
> >> My understanding is that Guile 2.2 traded slower compilation for faster
> >> execution of compiled programs. Hopefully the performance of the
> >> compiler will be improved in later updates to Guile.
> >
> > Yes, that’s a good summary.
> >
> > Most of the code we compile is package definitions that don’t benefit
> > from all the fancy optimizations, so build-aux/build-self.scm (run by
> > ‘guix pull’) turns those off.
> >
> > Unfortunately even with this change compilation is slower than with
> > Guile 2.0, and IMO unacceptably slow for ‘guix pull’ (it takes only a
> > few minutes at most on my laptop, but that’s a lot if we compare to
> > ‘apt-get update’.)
> >
> > Andy, do you have other approaches in mind that we could explore to
> > improve on this?  Evaluating all of gnu/packages instead of compiling it
> > AOT would probably be bad for memory and CPU consumption.
> 
> I agree that this is a correct summary.  I think it's possible that even
> with improvements that 2.2 would remain slower than 2.0 to compile, but
> the current situation is clearly not good and needs improvement.  I
> think we need to revisit that synthetic python.scm test you had (can't
> find it atm), see where the compiler is spending time, and focus effort
> there.
> 
> As far as what Guix can do -- there I don't know :/  Nothing immediate
> comes to mind.
> 
> I would like to help here but can't promise much over the summertime...
> 
> Andy
> 
> 

What about the sledgehammer approach:
Would it help if we just split the incredible long modules into
smaller modules like python-web, python-library, etc?
-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: compiling guix is too slow?

[Ludovic Courtès]

ng0 <ng0@infotropique.org> skribis:

> What about the sledgehammer approach:
> Would it help if we just split the incredible long modules into
> smaller modules like python-web, python-library, etc?

We can always do that, yes (and it’s probably a good idea independently
of this.)

Any volunteer to get started?

Ludo’.

Re: compiling guix is too slow?

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1052 bytes --]

Ludovic Courtès transcribed 0.3K bytes:
> ng0 <ng0@infotropique.org> skribis:
> 
> > What about the sledgehammer approach:
> > Would it help if we just split the incredible long modules into
> > smaller modules like python-web, python-library, etc?
> 
> We can always do that, yes (and it’s probably a good idea independently
> of this.)

I'd say we split up:

2.0 MiB and bigger:
python         (3.7 MiB go)
perl           (2.3 MiB go)
haskell        (2.1 MiB go)
bioinformatics (2.0 MiB go)

And those we can consider to split up as they are bigger than 1MiB
and smaller than 2MiB:
statistics     (1.4 MiB go)
gnome          (1.4 MiB go)
xorg           (1.3 MiB go)
emacs          (1.2 MiB go)
web            (1.2 MiB go)
ruby           (1.1 MiB go)

games.go and linux.go are border cases for consideration.

> Any volunteer to get started?
> 
> Ludo’.
> 
> 

-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: compiling guix is too slow?

[Ricardo Wurmus]

ng0 <ng0@infotropique.org> writes:

> I'd say we split up:
>
> 2.0 MiB and bigger:
> python         (3.7 MiB go)
> perl           (2.3 MiB go)
> haskell        (2.1 MiB go)
> bioinformatics (2.0 MiB go)

Modules named after languages would probably benefit from splitting
them.  A *lot* of software is written in those languages, and we ought
to move packages to where they make sense (e.g. web, documentation,
admin, etc).

I disagree on splitting up “bioinformatics”; it’s already a “topic
module”.  It would be hard to split by field (e.g. “RNA-seq”), because
many tools cover a wide range.  I’m against some arbitrary split, which
would give us something silly like “(gnu packages bioinformatics a)”,
“(gnu packages bioinformatics b)”, etc.

> And those we can consider to split up as they are bigger than 1MiB
> and smaller than 2MiB:
> statistics     (1.4 MiB go)
> gnome          (1.4 MiB go)
> xorg           (1.3 MiB go)
> emacs          (1.2 MiB go)
> web            (1.2 MiB go)
> ruby           (1.1 MiB go)

The only module I’d split from this list is “ruby” (for the same reasons
as above).  “statistics” will change as soon as we get started with the
much bigger “cran” module — but that’s going to cause more problems, not
solve them :)

Anyway, Andy has already identified a problem with the compilation, so
I’d defer any work on these other modules.  Independent of how this
goes, however, (gnu packages python) ought to be split up.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: compiling guix is too slow?

[ng0]

[-- Attachment #1: Type: text/plain, Size: 2141 bytes --]

Ricardo Wurmus transcribed 1.6K bytes:
> 
> ng0 <ng0@infotropique.org> writes:
> 
> > I'd say we split up:
> >
> > 2.0 MiB and bigger:
> > python         (3.7 MiB go)
> > perl           (2.3 MiB go)
> > haskell        (2.1 MiB go)
> > bioinformatics (2.0 MiB go)
> 
> Modules named after languages would probably benefit from splitting
> them.  A *lot* of software is written in those languages, and we ought
> to move packages to where they make sense (e.g. web, documentation,
> admin, etc).
> 
> I disagree on splitting up “bioinformatics”; it’s already a “topic
> module”.  It would be hard to split by field (e.g. “RNA-seq”), because
> many tools cover a wide range.  I’m against some arbitrary split, which
> would give us something silly like “(gnu packages bioinformatics a)”,
> “(gnu packages bioinformatics b)”, etc.
> 
> > And those we can consider to split up as they are bigger than 1MiB
> > and smaller than 2MiB:
> > statistics     (1.4 MiB go)
> > gnome          (1.4 MiB go)
> > xorg           (1.3 MiB go)
> > emacs          (1.2 MiB go)
> > web            (1.2 MiB go)
> > ruby           (1.1 MiB go)
> 
> The only module I’d split from this list is “ruby” (for the same reasons
> as above).  “statistics” will change as soon as we get started with the
> much bigger “cran” module — but that’s going to cause more problems, not
> solve them :)
> 
> Anyway, Andy has already identified a problem with the compilation, so
> I’d defer any work on these other modules.  Independent of how this
> goes, however, (gnu packages python) ought to be split up.
> 
> --
> Ricardo
> 
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net

Okay.

I would like to be able to run guix pull on servers with
512 MB RAM again, any way this happens at some point is
okay for me. If it is only splitting up python plus
solving the problem with compilation, that's okay.
-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://n0is.noblogs.org/my-keys
https://www.infotropique.org https://krosos.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: compiling guix is too slow?

[Ludovic Courtès]

Andy Wingo <wingo@igalia.com> skribis:

> On Sat 01 Jul 2017 15:33, ludo@gnu.org (Ludovic Courtès) writes:
>
>> Leo Famulari <leo@famulari.name> skribis:
>>
>>> My understanding is that Guile 2.2 traded slower compilation for faster
>>> execution of compiled programs. Hopefully the performance of the
>>> compiler will be improved in later updates to Guile.
>>
>> Yes, that’s a good summary.
>>
>> Most of the code we compile is package definitions that don’t benefit
>> from all the fancy optimizations, so build-aux/build-self.scm (run by
>> ‘guix pull’) turns those off.
>>
>> Unfortunately even with this change compilation is slower than with
>> Guile 2.0, and IMO unacceptably slow for ‘guix pull’ (it takes only a
>> few minutes at most on my laptop, but that’s a lot if we compare to
>> ‘apt-get update’.)
>>
>> Andy, do you have other approaches in mind that we could explore to
>> improve on this?  Evaluating all of gnu/packages instead of compiling it
>> AOT would probably be bad for memory and CPU consumption.
>
> I agree that this is a correct summary.  I think it's possible that even
> with improvements that 2.2 would remain slower than 2.0 to compile, but
> the current situation is clearly not good and needs improvement.  I
> think we need to revisit that synthetic python.scm test you had (can't
> find it atm), see where the compiler is spending time, and focus effort
> there.

It’s here:
<https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.

This one was focusing on memory consumption but there’s obviously a
connection between the two.

I’ll see if I can at least provide you with more precise data and
benchmarks so we have an idea of where the problem lies.

Thank you,
Ludo’.

Re: compiling guix is too slow?

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

"Feng Shu" <tumashu@163.com> writes:

> Now I have found that 'guix pull' is too slow,
> I need 3 hours to compile guix, is it possible to speed it ?

Maybe you can try building from git instead? I used to run
'$ guix pull && guix package --upgrade', but it gets slower as the
number of packages of guix increases. So now I use
'$ git pull && make -j`nproc` && ./pre-inst-env guix package --upgrade'.
You can read the manual[0] for more info.

If you use emacs-guix, you need to tell emacs-guix the location of your
git repository as well. I am unware of how others do it. Here is how I
do it:

  (require 'guix-autoloads)
  (guix-prettify-global-mode)
  (setq-default guix-current-profile
                (file-chase-links "~/.guix-profile" 1))
  (setq-default guix-directory "~/scm/guix/")

  (require 'guix-build-config)
  (let ((guix-src-dir (expand-file-name "~/scm/guix/")))
    (setq-default guix-config-image-directory guix-src-dir)
    (setq-default guix-config-guix-scheme-compiled-directory guix-src-dir))


[0]: https://www.gnu.org/software/guix/manual/html_node/Building-from-Git.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: compiling guix is too slow?

[Alex Kost]

Alex Vong (2017-06-29 12:28 +0800) wrote:

> "Feng Shu" <tumashu@163.com> writes:
>
>> Now I have found that 'guix pull' is too slow,
>> I need 3 hours to compile guix, is it possible to speed it ?
>
> Maybe you can try building from git instead? I used to run
> '$ guix pull && guix package --upgrade', but it gets slower as the
> number of packages of guix increases. So now I use
> '$ git pull && make -j`nproc` && ./pre-inst-env guix package --upgrade'.
> You can read the manual[0] for more info.
>
> If you use emacs-guix, you need to tell emacs-guix the location of your
> git repository as well. I am unware of how others do it. Here is how I
> do it:
>
>   (require 'guix-autoloads)

I always recommend avoid such "hard" requiring, it is better to use
(require 'foo nil t) instead.  With (require 'foo), your emacs config
will be really fragile: once 'foo' feature will disappear (renamed or
'foo' package will be uninstalled, etc.), Emacs will fail loading your
config!  So I would use (require 'guix-autoloads nil t) instead.

>   (guix-prettify-global-mode)

For the same reason, I never call functions from external packages in my
emacs config: what if this function disappear one day? (I don't mean I'm
going to remove it).  If I need to load something on Emacs start, I add
it to 'after-init-hook'.  If it is going to fail, at least it will fail
after your config will be fully loaded.

>   (setq-default guix-current-profile
>                 (file-chase-links "~/.guix-profile" 1))

Hm, I don't think this is needed.  Doesn't Emacs-Guix work with the
default 'guix-current-profile' setting?

>   (require 'guix-build-config)
>   (let ((guix-src-dir (expand-file-name "~/scm/guix/")))
>     (setq-default guix-config-image-directory guix-src-dir)
>     (setq-default guix-config-guix-scheme-compiled-directory guix-src-dir))

I have never had a need to set these variables.  Setting image directory
is definitely not needed (if the default value doesn't work for you,
it's a bug).

As for the directory with Guix compiled files, although you can set it,
it is not really intended for customizing.  Do you have
"~/.config/guix/latest" point to "~/scm/guix"?  Or do you add
"~/scm/guix" to GUILE_LOAD_PATH and GUILE_LOAD_COMPILED_PATH?  Or do you
add it to 'geiser-guile-load-path'?  If anything of this is true, then
you don't need to set that variable.

-- 
Alex

Re: compiling guix is too slow?

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 3000 bytes --]

Alex Kost <alezost@gmail.com> writes:

> Alex Vong (2017-06-29 12:28 +0800) wrote:
>
>> "Feng Shu" <tumashu@163.com> writes:
>>
>>> Now I have found that 'guix pull' is too slow,
>>> I need 3 hours to compile guix, is it possible to speed it ?
>>
>> Maybe you can try building from git instead? I used to run
>> '$ guix pull && guix package --upgrade', but it gets slower as the
>> number of packages of guix increases. So now I use
>> '$ git pull && make -j`nproc` && ./pre-inst-env guix package --upgrade'.
>> You can read the manual[0] for more info.
>>
>> If you use emacs-guix, you need to tell emacs-guix the location of your
>> git repository as well. I am unware of how others do it. Here is how I
>> do it:
>>
>>   (require 'guix-autoloads)
>
> I always recommend avoid such "hard" requiring, it is better to use
> (require 'foo nil t) instead.  With (require 'foo), your emacs config
> will be really fragile: once 'foo' feature will disappear (renamed or
> 'foo' package will be uninstalled, etc.), Emacs will fail loading your
> config!  So I would use (require 'guix-autoloads nil t) instead.
>
>>   (guix-prettify-global-mode)
>
> For the same reason, I never call functions from external packages in my
> emacs config: what if this function disappear one day? (I don't mean I'm
> going to remove it).  If I need to load something on Emacs start, I add
> it to 'after-init-hook'.  If it is going to fail, at least it will fail
> after your config will be fully loaded.
>
Is there an intermediate approch? Is is possible for Emacs to signal a
warning when the required features are not found, but still continue
loading the rest of the config?

>>   (setq-default guix-current-profile
>>                 (file-chase-links "~/.guix-profile" 1))
>
> Hm, I don't think this is needed.  Doesn't Emacs-Guix work with the
> default 'guix-current-profile' setting?
>
>>   (require 'guix-build-config)
>>   (let ((guix-src-dir (expand-file-name "~/scm/guix/")))
>>     (setq-default guix-config-image-directory guix-src-dir)
>>     (setq-default guix-config-guix-scheme-compiled-directory guix-src-dir))
>
> I have never had a need to set these variables.  Setting image directory
> is definitely not needed (if the default value doesn't work for you,
> it's a bug).
>
> As for the directory with Guix compiled files, although you can set it,
> it is not really intended for customizing.  Do you have
> "~/.config/guix/latest" point to "~/scm/guix"?  Or do you add
> "~/scm/guix" to GUILE_LOAD_PATH and GUILE_LOAD_COMPILED_PATH?  Or do you
> add it to 'geiser-guile-load-path'?  If anything of this is true, then
> you don't need to set that variable.

Thanks for the suggestion! It turns out setting NIX_STATE_DIR to
"/usr/local/var/guix/" and making "~/.config/guix/latest" pointing to
"~/scm/guix/" completely resolves the problem. So now only a oneliner is
required:

  (require 'guix-autoloads '() t)

Do you think it is a bug that Emacs-Guix does not set
`guix-state-directory' correctly?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: compiling guix is too slow?

[Alex Kost]

Alex Vong (2017-07-04 16:16 +0800) wrote:

[...]
> Is there an intermediate approch? Is is possible for Emacs to signal a
> warning when the required features are not found, but still continue
> loading the rest of the config?

If you wish to have a message in *Messages* buffer, you can do it like
this:

(unless (require 'foo nil t)
  (message "Error during loading 'foo'!!!"))

or like this:

(with-demoted-errors "%S" (require 'foo))

If you want to have a warning in a pop-up buffer, then:

(unless (require 'foo nil t)
  (display-warning 'oops "Error during loading 'foo'!!!"))

[...]
> Thanks for the suggestion! It turns out setting NIX_STATE_DIR to
> "/usr/local/var/guix/" and making "~/.config/guix/latest" pointing to
> "~/scm/guix/" completely resolves the problem. So now only a oneliner is
> required:
>
>   (require 'guix-autoloads '() t)

Actually, why do you need this line at all?  Such autoloads should be
loaded automatically.  Do you also require autoloads of other emacs
packages?  Do you use Emacs from Guix or from your operating system (not
GuixSD)?

> Do you think it is a bug that Emacs-Guix does not set
> `guix-state-directory' correctly?

I'm not sure what you mean.  It sets this variable to "/var/guix" by
default (or to NIX_STATE_DIR if you use it).  If you use Guix with a
non-standard state directory, how can Emacs-Guix define it?

BTW, don't you have problems using "/usr/local/var" as the state
directory?

If you use guix from a git checkout, I think it is a standard practice
to configure it with "--localstatedir=/var".  It is even mentioned in
the manual a couple of times:

  (info "(guix) Building from Git")
  (info "(guix) Requirements")

-- 
Alex

Re: compiling guix is too slow?

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 2280 bytes --]

Alex Kost <alezost@gmail.com> writes:

> Alex Vong (2017-07-04 16:16 +0800) wrote:
>
> [...]
>> Is there an intermediate approch? Is is possible for Emacs to signal a
>> warning when the required features are not found, but still continue
>> loading the rest of the config?
>
> If you wish to have a message in *Messages* buffer, you can do it like
> this:
>
> (unless (require 'foo nil t)
>   (message "Error during loading 'foo'!!!"))
>
> or like this:
>
> (with-demoted-errors "%S" (require 'foo))
>
> If you want to have a warning in a pop-up buffer, then:
>
> (unless (require 'foo nil t)
>   (display-warning 'oops "Error during loading 'foo'!!!"))
>
OK.

> [...]
>> Thanks for the suggestion! It turns out setting NIX_STATE_DIR to
>> "/usr/local/var/guix/" and making "~/.config/guix/latest" pointing to
>> "~/scm/guix/" completely resolves the problem. So now only a oneliner is
>> required:
F>>
>>   (require 'guix-autoloads '() t)
>
> Actually, why do you need this line at all?  Such autoloads should be
> loaded automatically.  Do you also require autoloads of other emacs
> packages?  Do you use Emacs from Guix or from your operating system (not
> GuixSD)?
>
Indeed, even that line is not needed. I think that was needed when
Emacs-Guix was still part of Guix.

>> Do you think it is a bug that Emacs-Guix does not set
>> `guix-state-directory' correctly?
>
> I'm not sure what you mean.  It sets this variable to "/var/guix" by
> default (or to NIX_STATE_DIR if you use it).  If you use Guix with a
> non-standard state directory, how can Emacs-Guix define it?
>
> BTW, don't you have problems using "/usr/local/var" as the state
> directory?
>
It works on my laptop so far...

> If you use guix from a git checkout, I think it is a standard practice
> to configure it with "--localstatedir=/var".  It is even mentioned in
> the manual a couple of times:
>
>   (info "(guix) Building from Git")
>   (info "(guix) Requirements")

Hmm, I am unaware of this practice. I'll re-install Guix with
"--localstatedir=/var".

Indeed, the manual mentions that, "LOCALSTATEDIR is the state directory
specified via ‘--localstatedir’ at configure time, usually ‘/var’"
in:

  (info "(guix) The Store")

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: compiling guix is too slow?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 706 bytes --]

"Feng Shu" <tumashu@163.com> writes:

> Now I have found that 'guix pull' is too slow,
> I need 3 hours to compile guix, is it possible to speed it ?

3 hours is longer than it usually takes on my Lenovo X200 laptop with 2
cores, an SSD, and 8 GB of RAM.  Usually, for me, it seems to take about
15 minutes.

I once tried to speed up 'guix pull' by offloading to another machine
with comparable specs.  Although I did manage to set up offloading
successfully, the result was that 'guix pull' with offloading was many
times slower than 'guix pull' without offloading.  In fact, it was so
slow, I gave up before it completed.  Instead, I just ran 'guix pull'
without offloading.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Can we speed it up? Prev: compiling guix is too slow?

[Pjotr Prins]

On Wed, Jul 05, 2017 at 08:49:52PM -0700, Chris Marusich wrote:
> "Feng Shu" <tumashu@163.com> writes:
> 
> > Now I have found that 'guix pull' is too slow,
> > I need 3 hours to compile guix, is it possible to speed it ?
> 
> 3 hours is longer than it usually takes on my Lenovo X200 laptop with 2
> cores, an SSD, and 8 GB of RAM.  Usually, for me, it seems to take about
> 15 minutes.

Guix pull takes a while and building Guix in general is slow too. I
find I have to rebuild quite often (updating trunk, moving between
machines) and with having my own guix publish server building
Guix is the bottle neck(!) From our discussion on Friday I gather
it is going to get a lot worse with the growing number of packages. 
It scales probably worse than linear.

What will it be like with 15K packages? We will get there. We can
actually try it now by doubling the package tree - anyone wants to try
and create a simulation? I.e., not just double the tree, make sure
there are cross references between the two graphs by modifying some of
the inputs at random.

This leads to the following thought: why don't we create a 'lazy'
build. There are multiple ways to go about it (I think). One would be
to parse scheme files for package names and only compile those that
are needed when someone invokes a guix command (and have not been
compiled yet). Or generate a meta list for a source tree. Or
subcategorize packages so only those packages get included that are
asked for (assuming there are no deeper dependencies). For example,
few people need the bioinformatics packages. We could have the sub
section of the graph split out and have people do:

  guix package --topic=bio -i samtools

for example and compile the contents gnu/packages/bio/ directory when
that happens the first time for a specific checkout.

This would be a lot more scalable and the main package section could be
fairly small and build fast. 

Sectioning the graph may be hard (you'd be inclined to section off
languages and window managers), but I think it can be dictated by
whether a sub graph can live on its own. And if we allow packages to
reference packages in a sub graph (triggering the build) then anything
goes - the JVM subsystem could be in one and Perl CPAN in another.

I think scalability is a good goal and instant compilation another ;).
A few years back it just took 30 seconds to build Guix.

Pj.

Re: Can we speed it up? Prev: compiling guix is too slow?

[Konrad Hinsen]

On 05/02/2018 08:34, Pjotr Prins wrote:

> compiled yet). Or generate a meta list for a source tree. Or
> subcategorize packages so only those packages get included that are
> asked for (assuming there are no deeper dependencies). For example,
> few people need the bioinformatics packages. We could have the sub
> section of the graph split out and have people do:
> 
>    guix package --topic=bio -i samtools

Or move special-topic packages to separate channels, once they get 
implemented. The hard part is of course *where* to split the graph, not 
how to implement it.

> Sectioning the graph may be hard (you'd be inclined to section off
> languages and window managers), but I think it can be dictated by
> whether a sub graph can live on its own.
I wonder if anyone has analyzed the dependency graphs of software 
packages (not necessarily for Guix, some big distribution like Debian 
would be more interesting), with the goal if identifying good splits 
based on simple criteria.

Konrad.

Re: Can we speed it up? Prev: compiling guix is too slow?

[Pjotr Prins]

On Mon, Feb 05, 2018 at 09:15:37AM +0100, Konrad Hinsen wrote:
> On 05/02/2018 08:34, Pjotr Prins wrote:
> 
> >compiled yet). Or generate a meta list for a source tree. Or
> >subcategorize packages so only those packages get included that are
> >asked for (assuming there are no deeper dependencies). For example,
> >few people need the bioinformatics packages. We could have the sub
> >section of the graph split out and have people do:
> >
> >   guix package --topic=bio -i samtools
> 
> Or move special-topic packages to separate channels, once they get
> implemented. The hard part is of course *where* to split the graph,
> not how to implement it.

Channels are coming. I agree that bioinformatics could become a
channel - that is what bioconda does for Conda. But for other sections
of the graph (say Ruby) we better not split out to channels. That is
why I am bringing this up as an in-trunk solution.

> >Sectioning the graph may be hard (you'd be inclined to section off
> >languages and window managers), but I think it can be dictated by
> >whether a sub graph can live on its own.
> I wonder if anyone has analyzed the dependency graphs of software
> packages (not necessarily for Guix, some big distribution like
> Debian would be more interesting), with the goal if identifying good
> splits based on simple criteria.

Yeah, that would be a neat exercise. Any student here inclined to have
a go?

Pj.

Re: Can we speed it up? Prev: compiling guix is too slow?

[Konrad Hinsen]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

>> I wonder if anyone has analyzed the dependency graphs of software
>> packages (not necessarily for Guix, some big distribution like
>> Debian would be more interesting), with the goal if identifying good
>> splits based on simple criteria.
>
> Yeah, that would be a neat exercise. Any student here inclined to have
> a go?

As was to be expected, a quick search found some promising pointers, e.g.:

  https://arxiv.org/abs/0905.4226

  http://ieeexplore.ieee.org/document/7490780/

Plus a lot more on language-specific dependency analysis, which is less
directly useful but may contain interesting methods that could be
generalized.

So, yes, this would make a good research project (master's level for
example) with the potential for academic recognition.

Konrad.

Re: Can we speed it up? Prev: compiling guix is too slow?

[Pjotr Prins]

On Mon, Feb 05, 2018 at 10:56:54AM +0100, Konrad Hinsen wrote:
> Pjotr Prins <pjotr.public12@thebird.nl> writes:
> 
> >> I wonder if anyone has analyzed the dependency graphs of software
> >> packages (not necessarily for Guix, some big distribution like
> >> Debian would be more interesting), with the goal if identifying good
> >> splits based on simple criteria.
> >
> > Yeah, that would be a neat exercise. Any student here inclined to have
> > a go?
> 
> As was to be expected, a quick search found some promising pointers, e.g.:
> 
>   https://arxiv.org/abs/0905.4226
> 
>   http://ieeexplore.ieee.org/document/7490780/
> 
> Plus a lot more on language-specific dependency analysis, which is less
> directly useful but may contain interesting methods that could be
> generalized.
> 
> So, yes, this would make a good research project (master's level for
> example) with the potential for academic recognition.

Agree. I am thinking we should add a section to the website with such
projects.

Pj.
-- 

Re: Can we speed it up? Prev: compiling guix is too slow?

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> What will it be like with 15K packages? We will get there. We can
> actually try it now by doubling the package tree - anyone wants to try
> and create a simulation? I.e., not just double the tree, make sure
> there are cross references between the two graphs by modifying some of
> the inputs at random.
>
> This leads to the following thought: why don't we create a 'lazy'
> build. There are multiple ways to go about it (I think). One would be
> to parse scheme files for package names and only compile those that
> are needed when someone invokes a guix command (and have not been
> compiled yet). Or generate a meta list for a source tree.

Note that there are several things we could do:

  • Not compile gnu/packages/*.scm at all and instead turn on Guile’s
    auto-compilation.  As things are currently, the first ‘guix package’
    invocation would take ages though.

  • Never compile gnu/packages/*.scm and instead interpret it, though
    that’s currently relatively slow and probably more memory-consuming.

I think with ‘wip-pull-reload’ (I’ll resume work on it, I promise!)
things should already be nicer, and then, we should keep improving the
compiler (Andy already significantly improved CPU consumption in Guile
2.2.3).

> Or subcategorize packages so only those packages get included that are
> asked for (assuming there are no deeper dependencies). For example,
> few people need the bioinformatics packages. We could have the sub
> section of the graph split out and have people do:
>
>   guix package --topic=bio -i samtools
>
> for example and compile the contents gnu/packages/bio/ directory when
> that happens the first time for a specific checkout.

I think that should be the last resort, but yeah.

Another thing we could do to speed up lookup-by-name is to maintain a
cache that maps package names to modules (currently we always traverse
all the package modules with ‘fold-packages’.)

> I think scalability is a good goal and instant compilation another ;).
> A few years back it just took 30 seconds to build Guix.

Yep, scalability is this year’s challenge!

Ludo’.