Functional package interface

Functional package interface

[spacecadet]

Hi guix

I started working on a "functional interface" to guix packages, as an
approach to globally modifying packages in an environment, similar to
nix overlays or the parameterization project.
Right now it's just a syntax rule that you can wrap a bunch of package
definitions in with some inputs in the style of a let.


(define-syntax define-public-package-set
   (syntax-rules ()
     ((define-public-package-set pkgset
       ((input ...)
        ...)
       (package-name package-def)
       ...)
      (begin
        (define-public pkgset
          (lambda* (#:optional packages
                    #:key (input ...)
                          ...
                    #:allow-other-keys)
            (define package-name package-def)
            ...
            (cond ((and packages (list? packages))
                   (map (lambda (package)
                          (cond ((eq? package 'package-name)
                                 `(package-name . ,package-name))
                                ...))
                        packages))
                  (packages
                    (cond ((eq? packages 'package-name)
                           package-name)
                          ...))
                  (else
                    `((package-name . ,package-name)
                      ...)))))
        (set-procedure-property! pkgset 'package-set? #t)
        (define-public package-name
          (pkgset 'package-name))
        ...))))

     
Tried rewriting the guile packages module with it, link to the full
code further down but here's a snip:

     
;; existing module
(define-module (gnu packages guile)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (gnu packages)
...
            
;; package-set definition
(define-public-package-set gnu-packages-guile
   ((bash-minimal bash-minimal) ;; inputs
    (gawk gawk)
    (gmp gmp)
...
    (url-fetch url-fetch))
            
   ;; packages
   (guile-1.8
     (package
       (name "guile")
...


You end up with a public "gnu-packages-guile" function that takes
keyword-argument inputs like:

(gnu-packages-guile #:gawk (@ (gnu packages bioinformatics) bioawk))
;; => all the packages in gnu/packages/guile.scm, but every instance
;; of awk is replaced with bioawk


Then returns an alist of all the packages in the set with the inputs
appropriately "modified". For the keys in the alist I'm using the
variable names as symbols instead of the package names like in inputs,
because I feel that's more generally appropriate in the case where a
package-set might contain something that's not a package record.
Will probably write a helper function that spits out input pairs.

It can also take a single symbol or list of symbols as an optional
argument to produce either a single package or list of requested
packages:


(gnu-packages-guile 'guile-3.0)
;; => #<package guile@3.0.9 ... )
(gnu-packages-guile '(guile-2.2 guile2.2-json))
;; => (#<package guile@2.2.7 ... #<package guile2.2-json@4.7.3 ...)


Packages still get define-public'd, I assume in a way that would
preserve the existing working condition of the repository (although
there is one issue right now, more below) so this could hopefully turn
into a drop-in addition after some more work. I tried to make very few
changes necessary to the existing code to for the same reason.

Eventually, the goal is to be able to create an operating system
definition (or manifest, or anything else) using this, with a
hypothetical case like:


(use-package-modules base network etc.)

(define my-package-set
   (packaget-set-append gnu-packages-base
                        gnu-packages-network ...))

(with-package-set
   (my-package-set #:mesa something-else)
   (operating-system
     (...)))


Adapting more complicated package modules would probably be a task, but
I'm invested enough to keep working on it.

There are also a few problems with the current implementation, and I
think they all boil down to one issue:
If packages A and B are both in the same module, and B is an input to
A, I can't replace B.
There may be a solution, it might be as simple as providing a dummy
input for each package in the module, I don't know, open to ideas.

Another more minor problem is that it won't pull:


In ice-9/boot-9.scm:
    222:29 19 (map1 _)
    222:29 18 (map1 _)
    222:29 17 (map1 _)
    222:29 16 (map1 _)
    222:29 15 (map1 _)
    222:17 14 (map1 (((gnu packages guile))))
   3327:17 13 (resolve-interface (gnu packages guile) #:select _ #:hid
In ice-9/threads.scm:
     390:8 12 (_ _)
In ice-9/boot-9.scm:
   3253:13 11 (_)
In ice-9/threads.scm:
     390:8 10 (_ _)
In ice-9/boot-9.scm:
   3544:20  9 (_)
    2836:4  8 (save-module-excursion _)
   3564:26  7 (_)
In unknown file:
            6 (primitive-load-path "gnu/packages/guile" #<procedure 7f
In ice-9/eval.scm:
     619:8  5 (_ #f)
    626:19  4 (_ #<directory (gnu packages guile) 7fffec4d2dc0>)
     619:8  3 (_ #(#(#<directory (gnu packages guile) 7fffec4d2dc0>) #
    632:34  2 (_ #(#(#<directory (gnu packages guile) 7fffec4d2dc0>) #
    223:20  1 (proc #(#(#<directory (gnu packages guile) 7fffec4d2dc0>
In unknown file:
            0 (%resolve-variable (5 (gnu packages bash) bash-minimal .

ERROR: In procedure %resolve-variable:
error: bash-minimal: unbound variable


So I'm probably doing something bad.
Seems like the first variable in the "inputs" fails to resolve
somewhere? I don't even have a guess what's going wrong.

My code is available at https://gitlab.vulnix.sh/spacecadet/guix
Open to comments or ideas, and if anyone can give me a hand figuring out
the pull issue I'd be grateful.

- sc

Re: Functional package interface

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

Hi spacecadet!

Isn't what you're trying to do already in Guix? Have a look at
package-inputs-rewrite right there :
https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html

I've also been working on general package propagation in RDE, it's still
a work in progress, but it's not far from it :
https://lists.sr.ht/~abcdw/rde-devel/patches/49956

Best regards,
Nicolas

On 2024-04-14 07:02, spacecadet wrote:

> Hi guix
>
> I started working on a "functional interface" to guix packages, as an
> approach to globally modifying packages in an environment, similar to
> nix overlays or the parameterization project.
> Right now it's just a syntax rule that you can wrap a bunch of package
> definitions in with some inputs in the style of a let.
>
>
> (define-syntax define-public-package-set
>    (syntax-rules ()
>      ((define-public-package-set pkgset
>        ((input ...)
>         ...)
>        (package-name package-def)
>        ...)
>       (begin
>         (define-public pkgset
>           (lambda* (#:optional packages
>                     #:key (input ...)
>                           ...
>                     #:allow-other-keys)
>             (define package-name package-def)
>             ...
>             (cond ((and packages (list? packages))
>                    (map (lambda (package)
>                           (cond ((eq? package 'package-name)
>                                  `(package-name . ,package-name))
>                                 ...))
>                         packages))
>                   (packages
>                     (cond ((eq? packages 'package-name)
>                            package-name)
>                           ...))
>                   (else
>                     `((package-name . ,package-name)
>                       ...)))))
>         (set-procedure-property! pkgset 'package-set? #t)
>         (define-public package-name
>           (pkgset 'package-name))
>         ...))))
>
>      
> Tried rewriting the guile packages module with it, link to the full
> code further down but here's a snip:
>
>      
> ;; existing module
> (define-module (gnu packages guile)
>    #:use-module ((guix licenses) #:prefix license:)
>    #:use-module (gnu packages)
> ...
>             
> ;; package-set definition
> (define-public-package-set gnu-packages-guile
>    ((bash-minimal bash-minimal) ;; inputs
>     (gawk gawk)
>     (gmp gmp)
> ...
>     (url-fetch url-fetch))
>             
>    ;; packages
>    (guile-1.8
>      (package
>        (name "guile")
> ...
>
>
> You end up with a public "gnu-packages-guile" function that takes
> keyword-argument inputs like:
>
> (gnu-packages-guile #:gawk (@ (gnu packages bioinformatics) bioawk))
> ;; => all the packages in gnu/packages/guile.scm, but every instance
> ;; of awk is replaced with bioawk
>
>
> Then returns an alist of all the packages in the set with the inputs
> appropriately "modified". For the keys in the alist I'm using the
> variable names as symbols instead of the package names like in inputs,
> because I feel that's more generally appropriate in the case where a
> package-set might contain something that's not a package record.
> Will probably write a helper function that spits out input pairs.
>
> It can also take a single symbol or list of symbols as an optional
> argument to produce either a single package or list of requested
> packages:
>
>
> (gnu-packages-guile 'guile-3.0)
> ;; => #<package guile@3.0.9 ... )
> (gnu-packages-guile '(guile-2.2 guile2.2-json))
> ;; => (#<package guile@2.2.7 ... #<package guile2.2-json@4.7.3 ...)
>
>
> Packages still get define-public'd, I assume in a way that would
> preserve the existing working condition of the repository (although
> there is one issue right now, more below) so this could hopefully turn
> into a drop-in addition after some more work. I tried to make very few
> changes necessary to the existing code to for the same reason.
>
> Eventually, the goal is to be able to create an operating system
> definition (or manifest, or anything else) using this, with a
> hypothetical case like:
>
>
> (use-package-modules base network etc.)
>
> (define my-package-set
>    (packaget-set-append gnu-packages-base
>                         gnu-packages-network ...))
>
> (with-package-set
>    (my-package-set #:mesa something-else)
>    (operating-system
>      (...)))
>
>
> Adapting more complicated package modules would probably be a task, but
> I'm invested enough to keep working on it.
>
> There are also a few problems with the current implementation, and I
> think they all boil down to one issue:
> If packages A and B are both in the same module, and B is an input to
> A, I can't replace B.
> There may be a solution, it might be as simple as providing a dummy
> input for each package in the module, I don't know, open to ideas.
>
> Another more minor problem is that it won't pull:
>
>
> In ice-9/boot-9.scm:
>     222:29 19 (map1 _)
>     222:29 18 (map1 _)
>     222:29 17 (map1 _)
>     222:29 16 (map1 _)
>     222:29 15 (map1 _)
>     222:17 14 (map1 (((gnu packages guile))))
>    3327:17 13 (resolve-interface (gnu packages guile) #:select _ #:hid
> In ice-9/threads.scm:
>      390:8 12 (_ _)
> In ice-9/boot-9.scm:
>    3253:13 11 (_)
> In ice-9/threads.scm:
>      390:8 10 (_ _)
> In ice-9/boot-9.scm:
>    3544:20  9 (_)
>     2836:4  8 (save-module-excursion _)
>    3564:26  7 (_)
> In unknown file:
>             6 (primitive-load-path "gnu/packages/guile" #<procedure 7f
> In ice-9/eval.scm:
>      619:8  5 (_ #f)
>     626:19  4 (_ #<directory (gnu packages guile) 7fffec4d2dc0>)
>      619:8  3 (_ #(#(#<directory (gnu packages guile) 7fffec4d2dc0>) #
>     632:34  2 (_ #(#(#<directory (gnu packages guile) 7fffec4d2dc0>) #
>     223:20  1 (proc #(#(#<directory (gnu packages guile) 7fffec4d2dc0>
> In unknown file:
>             0 (%resolve-variable (5 (gnu packages bash) bash-minimal .
>
> ERROR: In procedure %resolve-variable:
> error: bash-minimal: unbound variable
>
>
> So I'm probably doing something bad.
> Seems like the first variable in the "inputs" fails to resolve
> somewhere? I don't even have a guess what's going wrong.
>
> My code is available at https://gitlab.vulnix.sh/spacecadet/guix
> Open to comments or ideas, and if anyone can give me a hand figuring out
> the pull issue I'd be grateful.
>
> - sc
>

-- 
Best regards,
Nicolas Graves

Re: Functional package interface

[spacecadet]

Hi Nicolas

> Isn't what you're trying to do already in Guix? Have a look at
> package-inputs-rewrite right there :
> https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html

I want to have this but more powerful, I'd like to do something like
define an operating system with all instances of a package replaced.
Maybe something like that is already possible, but I haven't figured
out a way.

As far as I know, with package-input-rewriting you would still have to
manually add it to all the services in your operating-system
definition. Or just use module-set! and manage the environment state
some other way.

You also only have access to the input fields of packages, this is
something parameterize packages tackles too, being able to add inputs
that go anywhere in the package definition. Build phases for example,
to enable common compile options across multiple packages more easily.

> I've also been working on general package propagation in RDE, it's still
> a work in progress, but it's not far from it :
> https://lists.sr.ht/~abcdw/rde-devel/patches/49956

I'll take a look. I knew there had to be more people working on this idea,
but it's not always easy to find these projects.

- sc

Re: Functional package interface

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

On 2024-04-16 01:17, spacecadet wrote:

> Hi Nicolas
>
>> Isn't what you're trying to do already in Guix? Have a look at
>> package-inputs-rewrite right there :
>> https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html
>
> I want to have this but more powerful, I'd like to do something like
> define an operating system with all instances of a package replaced.
> Maybe something like that is already possible, but I haven't figured
> out a way.
>
> As far as I know, with package-input-rewriting you would still have to
> manually add it to all the services in your operating-system
> definition. Or just use module-set! and manage the environment state
> some other way.
>
> You also only have access to the input fields of packages, this is
> something parameterize packages tackles too, being able to add inputs
> that go anywhere in the package definition. Build phases for example,
> to enable common compile options across multiple packages more easily.
>
>> I've also been working on general package propagation in RDE, it's still
>> a work in progress, but it's not far from it :
>> https://lists.sr.ht/~abcdw/rde-devel/patches/49956
>
> I'll take a look. I knew there had to be more people working on this idea,
> but it's not always easy to find these projects.

In the code there, the function propagate-packages will indeed try and
propagate every package through the graph node.

Four tricky points though :

- propagating some packages that are used in build-systems will trigger
massive rebuilds (e.g. go, emacs, python, ocaml, texlive), you'll
probably have to be smart about what you want to avoid to propagate, but
it's great because guix can already hide a package to
package-inputs-rewriting using (hidden-package {package-name}). There are some
examples in the series.

- some packages are present under different names (emacs has
emacs-minimal, emacs, emacs-pgtk, emacs-wide-int...) and that is not
handled by this function.

- this has a cost. I haven't timed it yet, but the slowdown is
  noticeable, though still quite usable.

- you of course have to get the packages into the function from a
  profile. Here's it's done in RDE's code, but that should be quite
  doable in Guix too.  


>
> - sc

-- 
Best regards,
Nicolas Graves

Re: Functional package interface

[spacecadet]

> In the code there, the function propagate-packages will indeed try and
> propagate every package through the graph node.

I looked at your code, it's comprehensive. I wasn't sure such a thing
was possible with input rewriting, although I think there's still a lot
of benefit to the functional approach, like this:
> - some packages are present under different names (emacs has
> emacs-minimal, emacs, emacs-pgtk, emacs-wide-int...) and that is not
> handled by this function.

It should handle it pretty elegantly, since those packages inherit from
emacs, if you replace emacs they'll inherit from your replacement while
keeping their unique traits.

> - propagating some packages that are used in build-systems will trigger
> massive rebuilds (e.g. go, emacs, python, ocaml, texlive), you'll
> probably have to be smart about what you want to avoid to propagate, but
> it's great because guix can already hide a package to
> package-inputs-rewriting using (hidden-package {package-name}). There are some
> examples in the series.

No idea how, but I guess I should implement a #:mask-hidden? argument
to handle this. I have more basic issues for now, but that's a good
point.

> - this has a cost. I haven't timed it yet, but the slowdown is
>    noticeable, though still quite usable.

I'm doing a couple stupid things, it's improved with the last few
commits, but there's still a long way to go. If there's a cost to
syntax rule abuse, you were probably feeling that.

> - you of course have to get the packages into the function from a
>    profile. Here's it's done in RDE's code, but that should be quite
>    doable in Guix too.

Yeah, the implementation model is kinda strange. Having to put extra
code in every single module in the entire repo isn't a great
prerequisite for anything. It could be the point that prevents this
from manifesting. The mile long butt-ugly list of inputs that it needs
doesn't help.

Thanks for the tips. Work is still ongoing, but recursive propagation
now works when giving one package-set another as an input, so the
intended use-case is now possible.

- sc