all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Caching The Python World: guix shell --container
@ 2022-01-15  7:30 jgart
  2022-01-15  7:45 ` Ricardo Wurmus
  0 siblings, 1 reply; 4+ messages in thread
From: jgart @ 2022-01-15  7:30 UTC (permalink / raw)
  To: Guix Devel

Hi Guixers, 

I've been developing an app[0] that launches jupyter notebooks in "the
cloud" in guix containers[1]. In other words, the user submits a url for a
git repo containing the jupyter notebook and the app runs the notebook
in a guix container serving it to the user's browser tab.

The app auto-detects a manifest.scm in the root of the jupyter notebook's
project repository. The app then builds the container and all the
dependencies for the notebook specified by the manifest.scm.

The issue is the following: 

Sometimes a manifest with 10 notebook dependencies or even less will take
a long time (> 3 minutes) for Guix to build the container with everything
specified by the manifest.

For example, this repository containing a guix manifest caused the
browser to timeout out today after minutes of waiting for guix to build
the container.

https://github.com/BonfaceKilz/tsaf-analysis-of-bxd-mouse-colonies/blob/main/manifest.scm

Is there a way to pre-cache all the guix packaged binaries or a
subset thereof *ahead of time* so that if the user needs a particular
package/library they can get it near instantaneously[2]/on demand when
the app calls guix shell?

In other words, I am interested in pre-caching "the python world"
packaged by GNU Guix upstream. 

Is there currently a convenient way to tell Guix to build "all python-*
packages" and cache them? One way that comes to mind is dumping all
the python packages to a manifest and regularly updating that manifest
as new packages are added by maintainers. 

Does this sound like a sane way to solve this container build waiting issue?

Any thoughts/advice is much appreciated.

all best,

jgart

[0]: https://git.genenetwork.org/jgart/binderlite
[1]: https://guix.gnu.org/en/blog/2015/container-provisioning-with-guix/
[2]: https://guix.gnu.org/en/blog/2021/from-guix-environment-to-guix-shell/
"hot cache, in 0.1 seconds"



^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Caching The Python World: guix shell --container
  2022-01-15  7:30 Caching The Python World: guix shell --container jgart
@ 2022-01-15  7:45 ` Ricardo Wurmus
  2022-01-15  8:24   ` jgart
  0 siblings, 1 reply; 4+ messages in thread
From: Ricardo Wurmus @ 2022-01-15  7:45 UTC (permalink / raw)
  To: jgart; +Cc: guix-devel


jgart <jgart@dismail.de> writes:

> Is there currently a convenient way to tell Guix to build "all python-*
> packages" and cache them?

This should do it:

    guix build --keep-going -e '\
      (begin \
        (import (guix packages) (gnu packages) (guix build-system python)) \
        (fold-packages \
          (lambda (item acc) \
            (if (eq? python-build-system (package-build-system item)) \
              (cons item acc) acc)) \
          (list)))'

This will build all packages with python-build-system.

-- 
Ricardo


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Caching The Python World: guix shell --container
  2022-01-15  7:45 ` Ricardo Wurmus
@ 2022-01-15  8:24   ` jgart
  2022-01-15 11:25     ` Ricardo Wurmus
  0 siblings, 1 reply; 4+ messages in thread
From: jgart @ 2022-01-15  8:24 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: guix-devel

On Sat, 15 Jan 2022 08:45:57 +0100 Ricardo Wurmus <rekado@elephly.net> wrote:
> 
> jgart <jgart@dismail.de> writes:
> 
> > Is there currently a convenient way to tell Guix to build "all python-*
> > packages" and cache them?
> 
> This should do it:
> 
>     guix build --keep-going -e '\
>       (begin \
>         (import (guix packages) (gnu packages) (guix build-system python)) \
>         (fold-packages \
>           (lambda (item acc) \
>             (if (eq? python-build-system (package-build-system item)) \
>               (cons item acc) acc)) \
>           (list)))'
> 
> This will build all packages with python-build-system.
> 
> -- 
> Ricardo

Oh darn! Ricardo, THNX

You've been giving me some great tips :)

Much appreciated!

all best,

jgart



^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Caching The Python World: guix shell --container
  2022-01-15  8:24   ` jgart
@ 2022-01-15 11:25     ` Ricardo Wurmus
  0 siblings, 0 replies; 4+ messages in thread
From: Ricardo Wurmus @ 2022-01-15 11:25 UTC (permalink / raw)
  To: jgart; +Cc: guix-devel


jgart <jgart@dismail.de> writes:

> On Sat, 15 Jan 2022 08:45:57 +0100 Ricardo Wurmus <rekado@elephly.net> wrote:
>> 
>> jgart <jgart@dismail.de> writes:
>> 
>> > Is there currently a convenient way to tell Guix to build "all python-*
>> > packages" and cache them?
>> 
>> This should do it:
>> 
>>     guix build --keep-going -e '\
>>       (begin \
>>         (import (guix packages) (gnu packages) (guix build-system python)) \
>>         (fold-packages \
>>           (lambda (item acc) \
>>             (if (eq? python-build-system (package-build-system item)) \
>>               (cons item acc) acc)) \
>>           (list)))'
>> 
>> This will build all packages with python-build-system.

fold-packages is really neat.  You can also add even more filtering
here; by package name, by inputs, anything really.  For example, the
expression above will give you any package built with
python-build-system, but this may be too much, as it includes Python 2
packages.

It is just a fold, though, so all work happens in that procedure that is
given the current package and the accumulator from past invocations of
the procedure.  The only important thing is that the procedure always
returns a value for the accumulator.

It doesn’t have to be a list either; it could be a hash table so you can
easily look up packages you’ve already accumulated so far and base your
decision on that, etc.

-- 
Ricardo


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-01-15 11:32 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-15  7:30 Caching The Python World: guix shell --container jgart
2022-01-15  7:45 ` Ricardo Wurmus
2022-01-15  8:24   ` jgart
2022-01-15 11:25     ` Ricardo Wurmus

Code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.