A package search engine for a curated list of channels

A package search engine for a curated list of channels

[Mekeor Melire]

Hello,

it would be nice to be able to search for a package (and/or a service)
by name or description etc. through many channels. AFAIK, there's no
such search engine yet. Please correct me if I'm wrong.

I'm considering to implement it. I'd suggest to approach it with a
server where (1) Guix is installed; (2) a long list of channels is
activated; (3) a cron-job or so which regularly runs `guix pull`; (4) a
web-api-service written in Guile which leverages the Guix-library in
order to search for packages (-- I guess it'd be calling the
`guix-package*` function from /guix/scripts/package.scm?); (5) a
simplistic front-end web-site which talks to the web-api.

I post this to help-guix instead of the devel-mailing-list because this
is not about development on Guix itself.

Kindly
Mekeor

Re: A package search engine for a curated list of channels

[Edouard Klein]

That's a great idea :)

I can lend a VPS with guix installed on it for testing and initial
deployment, if you need it.

I can't promise to help with development because of time constraints,
but I'll follow this with eagerness.

Unless I'm mistaken (which I very well my be), pulling from a channel
basically gives the channel authors code execution privileges on the
pulling machine, so as you said in the subject line, hand curation would
be in order.

Cheers,

Edouard

Mekeor Melire <mekeor@posteo.de> writes:

> Hello,
>
> it would be nice to be able to search for a package (and/or a service)
> by name or description etc. through many channels. AFAIK, there's no
> such search engine yet. Please correct me if I'm wrong.
>
> I'm considering to implement it. I'd suggest to approach it with a
> server where (1) Guix is installed; (2) a long list of channels is
> activated; (3) a cron-job or so which regularly runs `guix pull`; (4) a
> web-api-service written in Guile which leverages the Guix-library in
> order to search for packages (-- I guess it'd be calling the
> `guix-package*` function from /guix/scripts/package.scm?); (5) a
> simplistic front-end web-site which talks to the web-api.
>
> I post this to help-guix instead of the devel-mailing-list because this
> is not about development on Guix itself.
>
> Kindly
> Mekeor

Re: A package search engine for a curated list of channels

[Ricardo Wurmus]

Hi Mekeor,

> it would be nice to be able to search for a package (and/or a service)
> by name or description etc. through many channels. AFAIK, there's no
> such search engine yet. Please correct me if I'm wrong.
>
> I'm considering to implement it. I'd suggest to approach it with a
> server where (1) Guix is installed; (2) a long list of channels is
> activated; (3) a cron-job or so which regularly runs `guix pull`; (4) a
> web-api-service written in Guile which leverages the Guix-library in
> order to search for packages (-- I guess it'd be calling the
> `guix-package*` function from /guix/scripts/package.scm?); (5) a
> simplistic front-end web-site which talks to the web-api.

See https://github.com/UMCUGenetics/hpcguix-web.  It handles the
constant updates to all Guix channels and the search.  You’re welcome to
contribute to it, e.g. to add an API, suggest changes in how results are
displayed, etc.

An example deployment can be found at https://guix.mdc-berlin.de

Guix has a service for it that makes deployment easy.

You’ll see that it lists packages from guix-science and other channels,
e.g. https://guix.mdc-berlin.de/package/rstudio-server

-- 
Ricardo

404 errors with hpcguix-web (e.g. for /packages.json)

[Mekeor Melire]

Thank you for the hint. hpcguix-web is awesome! I installed it locally
with the following declaration:

--8<---------------cut here---------------start------------->8---
;; added modules: (guix channels) (gnu services web)
(service hpcguix-web-service-type
  (hpcguix-web-configuration
    (specs
      #~(define site-config
          (hpcweb-configuration
            (channels
              (cons*
                ;; ...
                %default-channels
                )))))))
--8<---------------cut here---------------end--------------->8---

When I open the website (http://127.0.0.1:5000) in the browser, no
packages are listed and the browser console shows 404-errors occurring
for the following routes:

http://127.0.0.1:5000/static/css/code.css
http://127.0.0.1:5000/static/images/h-separator-darker.png
http://127.0.0.1:5000/packages.json
http://127.0.0.1:5000/static/images/favicon.png

Does hpcguix-web depend on a web-server (like nginx) to server a certain
folder? Or what am I missing?



2022-04-27 / 20:37 / rekado@elephly.net:

> Hi Mekeor,
>
>> it would be nice to be able to search for a package (and/or a service)
>> by name or description etc. through many channels. AFAIK, there's no
>> such search engine yet. Please correct me if I'm wrong.
>>
>> I'm considering to implement it. I'd suggest to approach it with a
>> server where (1) Guix is installed; (2) a long list of channels is
>> activated; (3) a cron-job or so which regularly runs `guix pull`; (4) a
>> web-api-service written in Guile which leverages the Guix-library in
>> order to search for packages (-- I guess it'd be calling the
>> `guix-package*` function from /guix/scripts/package.scm?); (5) a
>> simplistic front-end web-site which talks to the web-api.
>
> See https://github.com/UMCUGenetics/hpcguix-web.  It handles the
> constant updates to all Guix channels and the search.  You’re welcome to
> contribute to it, e.g. to add an API, suggest changes in how results are
> displayed, etc.
>
> An example deployment can be found at https://guix.mdc-berlin.de
>
> Guix has a service for it that makes deployment easy.
>
> You’ll see that it lists packages from guix-science and other channels,
> e.g. https://guix.mdc-berlin.de/package/rstudio-server

Show: Minimal Working Example of hpcguix-web

[Mekeor Melire]

By the way, I got it working. Here's a minimal working example of
hpcguix-web (meant to be run in a Docker container):

--8<---------------cut here---------------start------------->8---
(use-modules
  (guix channels)           ;; channel (for hpcguix-web)
  (gnu packages certs)      ;; nss-certs
  (gnu services networking) ;; dhcp-client-service-type
  (gnu services ssh)        ;; openssh-service-type
  (gnu services web)        ;; hpcguix-web-service-type
  (gnu system image)
  (gnu))

(operating-system

  ;; this will be ignored
  (bootloader
    (bootloader-configuration
      (bootloader grub-bootloader)
      (targets '())
      (terminal-outputs '())))

  ;; this will be ignored
  (file-systems '())

  (firmware '())

  (host-name "host-name")

  ;; hpcguix-web requires pulling channels via git which requires nss-cers
  (packages (cons nss-certs %base-packages))

  (services
    (cons*

      (service hpcguix-web-service-type
        (hpcguix-web-configuration
          (specs
            #~(begin
                (use-modules (guix channels))
                (define site-config
                  (hpcweb-configuration
                    (channels
                      (cons*
                        ;; TODO: add channels here
                        %default-guix-channel))))))))

      (service dhcp-client-service-type)

      %base-services))

  ;; TODO: set timezone
  (timezone "Etc/UTC")

  (users
    (list
      ;; un-comment this user when you want to debug this os-declaration:
      ;; (user-account
      ;;   (name "debug")
      ;;   (password "debug")
      ;;   (group "users")
      ;;   (supplementary-groups
      ;;     (list "wheel" "netdev" "audio" "video")))
      )))
--8<---------------cut here---------------end--------------->8---


2022-04-27 / 21:07 / mekeor@posteo.de:

> Thank you for the hint. hpcguix-web is awesome! I installed it locally
> with the following declaration:
>
> --8<---------------cut here---------------start------------->8---
> ;; added modules: (guix channels) (gnu services web)
> (service hpcguix-web-service-type
>   (hpcguix-web-configuration
>     (specs
>       #~(define site-config
>           (hpcweb-configuration
>             (channels
>               (cons*
>                 ;; ...
>                 %default-channels
>                 )))))))
> --8<---------------cut here---------------end--------------->8---
>
> When I open the website (http://127.0.0.1:5000) in the browser, no
> packages are listed and the browser console shows 404-errors occurring
> for the following routes:
>
> http://127.0.0.1:5000/static/css/code.css
> http://127.0.0.1:5000/static/images/h-separator-darker.png
> http://127.0.0.1:5000/packages.json
> http://127.0.0.1:5000/static/images/favicon.png
>
> Does hpcguix-web depend on a web-server (like nginx) to server a certain
> folder? Or what am I missing?
>
>
>
> 2022-04-27 / 20:37 / rekado@elephly.net:
>
>> Hi Mekeor,
>>
>>> it would be nice to be able to search for a package (and/or a service)
>>> by name or description etc. through many channels. AFAIK, there's no
>>> such search engine yet. Please correct me if I'm wrong.
>>>
>>> I'm considering to implement it. I'd suggest to approach it with a
>>> server where (1) Guix is installed; (2) a long list of channels is
>>> activated; (3) a cron-job or so which regularly runs `guix pull`; (4) a
>>> web-api-service written in Guile which leverages the Guix-library in
>>> order to search for packages (-- I guess it'd be calling the
>>> `guix-package*` function from /guix/scripts/package.scm?); (5) a
>>> simplistic front-end web-site which talks to the web-api.
>>
>> See https://github.com/UMCUGenetics/hpcguix-web.  It handles the
>> constant updates to all Guix channels and the search.  You’re welcome to
>> contribute to it, e.g. to add an API, suggest changes in how results are
>> displayed, etc.
>>
>> An example deployment can be found at https://guix.mdc-berlin.de
>>
>> Guix has a service for it that makes deployment easy.
>>
>> You’ll see that it lists packages from guix-science and other channels,
>> e.g. https://guix.mdc-berlin.de/package/rstudio-server

Re: A package search engine for a curated list of channels

[Mekeor Melire]

2022-04-27 / 20:37 / rekado@elephly.net:

> See https://github.com/UMCUGenetics/hpcguix-web. It handles the
> constant updates to all Guix channels and the search. You’re welcome
> to contribute to it, e.g. to add an API, suggest changes in how
> results are displayed, etc.
>
> An example deployment can be found at https://guix.mdc-berlin.de
>
> Guix has a service for it that makes deployment easy.
>
> You’ll see that it lists packages from guix-science and other channels,
> e.g. https://guix.mdc-berlin.de/package/rstudio-server

For the record: hpcguix-web is a really nice program but it does not
offer sandboxing. Thus, it'd be a lot of work to make sure the channels
do not contain code that executes malicious commands on the server which
runs hpcguix-web.

An alternative would be to implement some kind of isolation. But
channels and package declarations are just scheme/guile code, so they
will probably always be able to run arbitrary commands on the server.

Another approach would be isolation. For each channel, we could run
hpcguix-web inside a Docker-container so that there's some isolation.
Then, we'd need to run another web-service which "bundles" the
packages.json files of all single-channel, dockerized hpcguix-web
instances. But:

    (1.) Does Docker really offer sufficient isolation?

    (2.) AFAIK, inside a operating-system-declaration with a
    docker-service, it's not possible to declare further / nested
    operating-systems which shall be run inside docker-containers. This
    would be awesome. We could write something like this then:

--8<---------------cut here---------------start------------->8---
          (operating-system
            (services (list
              ;; ...
              (service docker-service-type (docker-configuration
                (images (map
                  (lambda (chan)
                    (operating-system
                      (services (list
                        ;; ...
                        (service hpcguix-web-service-type
                          (hpcguix-web-configuration
                            (specs
                              #~(begin
                                  (use-modules (guix channels))
                                  (define site-config
                                    (hpcweb-configuration (channels
                                      (list chan))))))))))))
                  some-long-list-of-channels)))))))
--8<---------------cut here---------------end--------------->8---

Re: A package search engine for a curated list of channels

[Ricardo Wurmus]

Mekeor Melire <mekeor@posteo.de> writes:

> An alternative would be to implement some kind of isolation. But
> channels and package declarations are just scheme/guile code, so they
> will probably always be able to run arbitrary commands on the server.

Guile has some sandboxing features.  It would be an option to evaluate
channel modules in a restricted environment with (ice-9 sandbox).  That
would benefit all of Guix.

> Another approach would be isolation. For each channel, we could run
> hpcguix-web inside a Docker-container so that there's some isolation.
> Then, we'd need to run another web-service which "bundles" the
> packages.json files of all single-channel, dockerized hpcguix-web
> instances. But:
>
>     (1.) Does Docker really offer sufficient isolation?

No more than “guix shell -C”.  There’s no good reason to use Docker when
you already have Guix.  The Docker service exists for when you have a
Docker container image that you must use, not because its
containerization is superior to “guix shell -C”.

-- 
Ricardo

Re: A package search engine for a curated list of channels

[zimoun]

Hi Ricardo,

I am late to the party. :-)  This hpcguix-web is really cool is not
enough known.


On Wed, 27 Apr 2022 at 20:37, Ricardo Wurmus <rekado@elephly.net> wrote:

> See https://github.com/UMCUGenetics/hpcguix-web.  It handles the
> constant updates to all Guix channels and the search.  You’re welcome to
> contribute to it, e.g. to add an API, suggest changes in how results are
> displayed, etc.

Since hpcguix-web uses ’inferior’, is it possible to index more
revisions than just the last one?


> You’ll see that it lists packages from guix-science and other channels,
> e.g. https://guix.mdc-berlin.de/package/rstudio-server

Is it possible to display the channel name in the search page listing
the potential candidates?


Cheers,
simon