Hi Guixers, I've been developing an app[0] that launches jupyter notebooks in "the cloud" in guix containers[1]. In other words, the user submits a url for a git repo containing the jupyter notebook and the app runs the notebook in a guix container serving it to the user's browser tab. The app auto-detects a manifest.scm in the root of the jupyter notebook's project repository. The app then builds the container and all the dependencies for the notebook specified by the manifest.scm. The issue is the following: Sometimes a manifest with 10 notebook dependencies or even less will take a long time (> 3 minutes) for Guix to build the container with everything specified by the manifest. For example, this repository containing a guix manifest caused the browser to timeout out today after minutes of waiting for guix to build the container. https://github.com/BonfaceKilz/tsaf-analysis-of-bxd-mouse-colonies/blob/main/manifest.scm Is there a way to pre-cache all the guix packaged binaries or a subset thereof *ahead of time* so that if the user needs a particular package/library they can get it near instantaneously[2]/on demand when the app calls guix shell? In other words, I am interested in pre-caching "the python world" packaged by GNU Guix upstream. Is there currently a convenient way to tell Guix to build "all python-* packages" and cache them? One way that comes to mind is dumping all the python packages to a manifest and regularly updating that manifest as new packages are added by maintainers. Does this sound like a sane way to solve this container build waiting issue? Any thoughts/advice is much appreciated. all best, jgart [0]: https://git.genenetwork.org/jgart/binderlite [1]: https://guix.gnu.org/en/blog/2015/container-provisioning-with-guix/ [2]: https://guix.gnu.org/en/blog/2021/from-guix-environment-to-guix-shell/ "hot cache, in 0.1 seconds"
jgart <jgart@dismail.de> writes:
> Is there currently a convenient way to tell Guix to build "all python-*
> packages" and cache them?
This should do it:
guix build --keep-going -e '\
(begin \
(import (guix packages) (gnu packages) (guix build-system python)) \
(fold-packages \
(lambda (item acc) \
(if (eq? python-build-system (package-build-system item)) \
(cons item acc) acc)) \
(list)))'
This will build all packages with python-build-system.
--
Ricardo
On Sat, 15 Jan 2022 08:45:57 +0100 Ricardo Wurmus <rekado@elephly.net> wrote:
>
> jgart <jgart@dismail.de> writes:
>
> > Is there currently a convenient way to tell Guix to build "all python-*
> > packages" and cache them?
>
> This should do it:
>
> guix build --keep-going -e '\
> (begin \
> (import (guix packages) (gnu packages) (guix build-system python)) \
> (fold-packages \
> (lambda (item acc) \
> (if (eq? python-build-system (package-build-system item)) \
> (cons item acc) acc)) \
> (list)))'
>
> This will build all packages with python-build-system.
>
> --
> Ricardo
Oh darn! Ricardo, THNX
You've been giving me some great tips :)
Much appreciated!
all best,
jgart
jgart <jgart@dismail.de> writes:
> On Sat, 15 Jan 2022 08:45:57 +0100 Ricardo Wurmus <rekado@elephly.net> wrote:
>>
>> jgart <jgart@dismail.de> writes:
>>
>> > Is there currently a convenient way to tell Guix to build "all python-*
>> > packages" and cache them?
>>
>> This should do it:
>>
>> guix build --keep-going -e '\
>> (begin \
>> (import (guix packages) (gnu packages) (guix build-system python)) \
>> (fold-packages \
>> (lambda (item acc) \
>> (if (eq? python-build-system (package-build-system item)) \
>> (cons item acc) acc)) \
>> (list)))'
>>
>> This will build all packages with python-build-system.
fold-packages is really neat. You can also add even more filtering
here; by package name, by inputs, anything really. For example, the
expression above will give you any package built with
python-build-system, but this may be too much, as it includes Python 2
packages.
It is just a fold, though, so all work happens in that procedure that is
given the current package and the accumulator from past invocations of
the procedure. The only important thing is that the procedure always
returns a value for the accumulator.
It doesn’t have to be a list either; it could be a hash table so you can
easily look up packages you’ve already accumulated so far and base your
decision on that, etc.
--
Ricardo