Caching The Python World: guix shell --container

Caching The Python World: guix shell --container

[jgart]

Hi Guixers, 

I've been developing an app[0] that launches jupyter notebooks in "the
cloud" in guix containers[1]. In other words, the user submits a url for a
git repo containing the jupyter notebook and the app runs the notebook
in a guix container serving it to the user's browser tab.

The app auto-detects a manifest.scm in the root of the jupyter notebook's
project repository. The app then builds the container and all the
dependencies for the notebook specified by the manifest.scm.

The issue is the following: 

Sometimes a manifest with 10 notebook dependencies or even less will take
a long time (> 3 minutes) for Guix to build the container with everything
specified by the manifest.

For example, this repository containing a guix manifest caused the
browser to timeout out today after minutes of waiting for guix to build
the container.

https://github.com/BonfaceKilz/tsaf-analysis-of-bxd-mouse-colonies/blob/main/manifest.scm

Is there a way to pre-cache all the guix packaged binaries or a
subset thereof *ahead of time* so that if the user needs a particular
package/library they can get it near instantaneously[2]/on demand when
the app calls guix shell?

In other words, I am interested in pre-caching "the python world"
packaged by GNU Guix upstream. 

Is there currently a convenient way to tell Guix to build "all python-*
packages" and cache them? One way that comes to mind is dumping all
the python packages to a manifest and regularly updating that manifest
as new packages are added by maintainers. 

Does this sound like a sane way to solve this container build waiting issue?

Any thoughts/advice is much appreciated.

all best,

jgart

[0]: https://git.genenetwork.org/jgart/binderlite
[1]: https://guix.gnu.org/en/blog/2015/container-provisioning-with-guix/
[2]: https://guix.gnu.org/en/blog/2021/from-guix-environment-to-guix-shell/
"hot cache, in 0.1 seconds"

Re: Caching The Python World: guix shell --container

[Ricardo Wurmus]

jgart <jgart@dismail.de> writes:

> Is there currently a convenient way to tell Guix to build "all python-*
> packages" and cache them?

This should do it:

    guix build --keep-going -e '\
      (begin \
        (import (guix packages) (gnu packages) (guix build-system python)) \
        (fold-packages \
          (lambda (item acc) \
            (if (eq? python-build-system (package-build-system item)) \
              (cons item acc) acc)) \
          (list)))'

This will build all packages with python-build-system.

-- 
Ricardo

Re: Caching The Python World: guix shell --container

[jgart]

On Sat, 15 Jan 2022 08:45:57 +0100 Ricardo Wurmus <rekado@elephly.net> wrote:
> 
> jgart <jgart@dismail.de> writes:
> 
> > Is there currently a convenient way to tell Guix to build "all python-*
> > packages" and cache them?
> 
> This should do it:
> 
>     guix build --keep-going -e '\
>       (begin \
>         (import (guix packages) (gnu packages) (guix build-system python)) \
>         (fold-packages \
>           (lambda (item acc) \
>             (if (eq? python-build-system (package-build-system item)) \
>               (cons item acc) acc)) \
>           (list)))'
> 
> This will build all packages with python-build-system.
> 
> -- 
> Ricardo

Oh darn! Ricardo, THNX

You've been giving me some great tips :)

Much appreciated!

all best,

jgart

Re: Caching The Python World: guix shell --container

[Ricardo Wurmus]

jgart <jgart@dismail.de> writes:

> On Sat, 15 Jan 2022 08:45:57 +0100 Ricardo Wurmus <rekado@elephly.net> wrote:
>> 
>> jgart <jgart@dismail.de> writes:
>> 
>> > Is there currently a convenient way to tell Guix to build "all python-*
>> > packages" and cache them?
>> 
>> This should do it:
>> 
>>     guix build --keep-going -e '\
>>       (begin \
>>         (import (guix packages) (gnu packages) (guix build-system python)) \
>>         (fold-packages \
>>           (lambda (item acc) \
>>             (if (eq? python-build-system (package-build-system item)) \
>>               (cons item acc) acc)) \
>>           (list)))'
>> 
>> This will build all packages with python-build-system.

fold-packages is really neat.  You can also add even more filtering
here; by package name, by inputs, anything really.  For example, the
expression above will give you any package built with
python-build-system, but this may be too much, as it includes Python 2
packages.

It is just a fold, though, so all work happens in that procedure that is
given the current package and the accumulator from past invocations of
the procedure.  The only important thing is that the procedure always
returns a value for the accumulator.

It doesn’t have to be a list either; it could be a hash table so you can
easily look up packages you’ve already accumulated so far and base your
decision on that, etc.

-- 
Ricardo