Pinning package inputs using inferiors?

Pinning package inputs using inferiors?

[Phil]

Hi all,

A change in a package ("dependency" in the below example) in a channel I
own has caused a conflict in another package in the same channel that depends
on it ("test-package" in the below).  Whilst fixing the "test-package"
package is the right solution, this is too complicated in do in the
short-term.  I need to pin "dependency" to v1.0 in test-pacakge's
propagated-inputs. Simultaneously, other packages need the new update to
the "dependency" package to use this functionality to deliver new
functionality that can't wait. 

This isn't a one-off situation; this happens frequently and I'm
interested in how other Guixers resolve this with as little friction to
users as possible?

One brainwave I had was to use inferiors - but this doesn't seem to
work.  Continuing from the above example we could define access to a
historical v1.0 of the dependency package for the test-package like so:

(define dependency-inferior
  ;; An inferior containing dependency v1.0.
  (inferior-for-channels dependency-channels))

If we do this then I can get the below manifest to work, just like the
example in the manual:

(packages->manifest
 (list (specification->package "python") ;; useful for testing
       ;; Remove current dependency add old dependency
       (package/inherit test-package
			(propagated-inputs
			 (modify-inputs (package-propagated-inputs test-package)
					(delete "dependency"))))
       (car (lookup-inferior-packages dependency-inferior "dependency"
 "1.0"))))

But this isn't a practical approach - knocking "dependency" out of
test-package's inputs means the unit tests would fail, so I would also
have to delete the check phase - I don't want to do this, it's too much
compromise.

Instead I was hoping to replace the dependency *inside* the package
definition with an inferior like this, so it's still available whilst
the inherited package is being built.

(packages->manifest
 (list (specification->package "python") ;; useful for testing
       ;; Remove current dependency, add old dependency
       (package/inherit test-package
			(propagated-inputs
			 (modify-inputs (package-propagated-inputs test-package)
					(replace "dependency" (car (lookup-inferior-packages dependency-inferior "dependency" "1.0"))))))))

When I try this, depending what operation I perform on the manifest, I
normally get some type mismatch telling me I've used a package-inferior
when Guile was expecting a package.  Nothing works, alas.

Thus I'm assuming the two types are not completely substitutable, and
this won't work?

Given this, the workaround I am employing is to replace a single package
definition of "dependency" with locally scoped definitions for each
package that uses it.  This is duplication feels suboptimal.

FWIW, a much more involved solution is to store the dependency package
inside test-package's project repo (rather than my channel), and then
automate copying this into the channel at build time.  This is a cool
exercise, but means we are decentralizing the channel's package
definitions and co-locating them across many repos - which feels not in
the spirit of Guix, and more like how requirements.txt files are
co-located in (non-Guix) python projects. 

My questions are:

1. Is my above use of inferiors always doomed, or something we code make
work with changes to Guix core?

2. Is there another way of handling the situation elegantly without
using inferiors or duplicating package definitions at module scope. 


Any advice or comments gladly received!

Cheers,
Phil.

Re: Pinning package inputs using inferiors?

[zimoun]

Hi Phil,

On Thu, 20 Oct 2022 at 22:37, Phil <phil@beadling.co.uk> wrote:

> A change in a package ("dependency" in the below example) in a channel I
> own has caused a conflict in another package in the same channel that depends
> on it ("test-package" in the below).  Whilst fixing the "test-package"
> package is the right solution, this is too complicated in do in the
> short-term.  I need to pin "dependency" to v1.0 in test-pacakge's
> propagated-inputs. Simultaneously, other packages need the new update to
> the "dependency" package to use this functionality to deliver new
> functionality that can't wait.
>
> This isn't a one-off situation; this happens frequently and I'm
> interested in how other Guixers resolve this with as little friction to
> users as possible?

Well, I answer to this question and I do not answer to your question
about inferior…

> One brainwave I had was to use inferiors - but this doesn't seem to
> work.  Continuing from the above example we could define access to a
> historical v1.0 of the dependency package for the test-package like so:
>
> (define dependency-inferior
>   ;; An inferior containing dependency v1.0.
>   (inferior-for-channels dependency-channels))

…instead of defining a complete inferior, why not just define 2
packages.  Something as,

        (define-public foo
          (package
            (name "foo")
            (version "2.0")
        [...]


        (define-public foo-1.6
          (package
            (inherit foo)
            (name "foo")
            (version "1.6")
        [...]

For an example, see python-numpy and python-numpy-next in (gnu packages
python-xyz).

Note that most CLI as “guix install foo” will install the last version.
Hence the ’-next’ trick in the package name. :-)  Depending what you
would like to be the “default“.


Cheers,
simon

Re: Pinning package inputs using inferiors?

[Phil]

Thanks Simon - I've given an example below.

zimoun writes:

> For an example, see python-numpy and python-numpy-next in (gnu packages
> python-xyz).

This was my original way of handling this but in what is perhaps a niche
use of Guix by my department - it ultimately doesn't scale well, for our
use-case.

Originally the department was small enough that there was only a handful
of applications sharing one or two common in-house libraries.

As we've scaled-up we now have the situation where 3 or 4 common
libraries are being used by say 10 applications.

We have rapid release schedules - and want to be able to release the
common libraries on a weekly basis.  But the time to sign-off on a
common library takes a few days per application, so it's not practical for
every project to bump version every week - they have other priorities.

In an ideal world automated unit and regression testing would be
comprehensive enough that we could move aggressively each week, but at
least for now that's not practical given the complex nature of signing
off the libraries and the applications which use the libraries.

So, ideally, what we'd like to do is just have each common library
churn-out releases every week, and have the releases available in Guix,
but without having an obligation on dependent applications to adopt
these changes if they don't want to.

Note all libraries and applications share the same channel - one
solution would be to have each library in their own channel, but this
feels ugly to me.

Our solution (somewhat tricky to explain without a whiteboard -
apologies!) is to co-locate the package definition of the common library
in the common library repo itself - we call it something like
.requirements.scm and it is naturally kept in lockstep with the code in
that repo that the definition will build.  This is very different to
traditional Guix where channels contain definitions separately in a
different repo to the code those definitions describe how to build. 

We then have a job in our CI/CD system that allows us to give a tag on the
common library repo, and the name of an application that uses the common
library.

The job will copy the .requirements.scm into our channel inside a
private module specific to the application that uses the common library.

The idea is that you can have many versions of .requirements.scm private
to every application package definition that references it.

You could even read .requirements.scm using a function that clones the
application repo on-the-fly rather than statically storing it in the
channel - we haven't gone this far yet, as it makes the system even more
complex to reason about.

This is basically the same idea as the python-numpy-next but allows for
many versions of python-numpy to co-exist by keeping them all in private
modules so they don't clash.

It's a cool idea and works pretty well, but requires us to augment Guix
with a set of extra tools to lift and shift these private definitions
around, which complicates our setup considerably.

It feels like wanting to make many versions of a library available at
once isn't an unreasonable way to work at-scale.  However, it also feels
like a departure from the philosophy of Guix to decentralise package
definitions and to allow for a potentially large range of versions to
co-exist in the same channel commit.

We could try to further integrate the idea into guix by writing new guix
commands to support it - we're still working out the details ourselves,
but if it works well we'd love to present it at a future Guix Days or
similar!

In the meantime I was wondering if anyone else had a similar use-case
for Guix and if they had tried something similar or different to handle
many versions in an automated way in the same channel commit?

Apologies that's more than I was intending to write - but hopefully that
makes some sense!  If it doesn't I can try to flesh out specific example?

Re: Pinning package inputs using inferiors?

[david larsson]

On 2022-10-21 23:08, Phil wrote:
[..]
> 
> In the meantime I was wondering if anyone else had a similar use-case
> for Guix and if they had tried something similar or different to handle
> many versions in an automated way in the same channel commit?

To handle many versions in an automated way, I once wrote this: 
https://gitlab.com/methuselah-0/guix-cigmon/-/tree/master

It's a schedulable script that will check for updates in certain git 
repos (specified branches if you want), and create inherited package 
versions for the new commits which it can commit and push to a guix 
channel repository. Probably doesn't solve your problem completely, but 
might be useful.

Example contents of a new file python-nbdev-org-babel-rev.scm with 
versions of python-nbdev-org-babel, automatically added to a guix 
channel:

(define-module (python-nbdev-org-babel-revs) #:use-module 
(python-extras) #:use-module (guix packages) #:use-module (guix 
git-download))
(define-public python-nbdev-org-babel-4f195e9 (package (inherit 
python-nbdev-org-babel)(properties '(("generated-by" . "cigmon")))(name 
"python-nbdev-org-babel-4f195e9")(source (origin (method git-fetch) (uri 
(git-reference (commit "4f195e915eefe5cd4deec3c6aea27e4b61233f33")(url 
"https://github.com/methuselah-0/nbdev-org-babel.git")))(sha256 (base32 
"0ayfxnw1s9rzs1qpqqyqwhf21xk6g8psffsqzfvvl0w5k1j88dqn"))))))
(define-public python-nbdev-org-babel-66079a5 (package (inherit 
python-nbdev-org-babel)(properties '(("generated-by" . "cigmon")))(name 
"python-nbdev-org-babel-66079a5")(source (origin (method git-fetch) (uri 
(git-reference (commit "66079a59aa9ee5e4479bdce597aac0f42f7fb565")(url 
"https://github.com/methuselah-0/nbdev-org-babel.git")))(sha256 (base32 
"0krdpc6zbdljriw6s80g08fywy0d1nq8wi3q07v3qs0b6rfz68n4"))))))
(define-public python-nbdev-org-babel-master (package (inherit 
python-nbdev-org-babel-66079a5)(name "python-nbdev-org-babel-master")))


This way you can pin some packages to have an input like 
mypackage-<1234567>

Would that be useful?

Best regards,
David

Re: Pinning package inputs using inferiors?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 6078 bytes --]

On Fri, Oct 21, 2022 at 10:08:10PM +0100, Phil wrote:
> Thanks Simon - I've given an example below.
> 
> zimoun writes:
> 
> > For an example, see python-numpy and python-numpy-next in (gnu packages
> > python-xyz).
> 
> This was my original way of handling this but in what is perhaps a niche
> use of Guix by my department - it ultimately doesn't scale well, for our
> use-case.
> 
> Originally the department was small enough that there was only a handful
> of applications sharing one or two common in-house libraries.
> 
> As we've scaled-up we now have the situation where 3 or 4 common
> libraries are being used by say 10 applications.
> 
> We have rapid release schedules - and want to be able to release the
> common libraries on a weekly basis.  But the time to sign-off on a
> common library takes a few days per application, so it's not practical for
> every project to bump version every week - they have other priorities.
> 
> In an ideal world automated unit and regression testing would be
> comprehensive enough that we could move aggressively each week, but at
> least for now that's not practical given the complex nature of signing
> off the libraries and the applications which use the libraries.
> 
> So, ideally, what we'd like to do is just have each common library
> churn-out releases every week, and have the releases available in Guix,
> but without having an obligation on dependent applications to adopt
> these changes if they don't want to.
> 
> Note all libraries and applications share the same channel - one
> solution would be to have each library in their own channel, but this
> feels ugly to me.
> 
> Our solution (somewhat tricky to explain without a whiteboard -
> apologies!) is to co-locate the package definition of the common library
> in the common library repo itself - we call it something like
> .requirements.scm and it is naturally kept in lockstep with the code in
> that repo that the definition will build.  This is very different to
> traditional Guix where channels contain definitions separately in a
> different repo to the code those definitions describe how to build. 
> 
> We then have a job in our CI/CD system that allows us to give a tag on the
> common library repo, and the name of an application that uses the common
> library.
> 
> The job will copy the .requirements.scm into our channel inside a
> private module specific to the application that uses the common library.
> 
> The idea is that you can have many versions of .requirements.scm private
> to every application package definition that references it.
> 
> You could even read .requirements.scm using a function that clones the
> application repo on-the-fly rather than statically storing it in the
> channel - we haven't gone this far yet, as it makes the system even more
> complex to reason about.
> 
> This is basically the same idea as the python-numpy-next but allows for
> many versions of python-numpy to co-exist by keeping them all in private
> modules so they don't clash.
> 
> It's a cool idea and works pretty well, but requires us to augment Guix
> with a set of extra tools to lift and shift these private definitions
> around, which complicates our setup considerably.
> 
> It feels like wanting to make many versions of a library available at
> once isn't an unreasonable way to work at-scale.  However, it also feels
> like a departure from the philosophy of Guix to decentralise package
> definitions and to allow for a potentially large range of versions to
> co-exist in the same channel commit.
> 
> We could try to further integrate the idea into guix by writing new guix
> commands to support it - we're still working out the details ourselves,
> but if it works well we'd love to present it at a future Guix Days or
> similar!
> 
> In the meantime I was wondering if anyone else had a similar use-case
> for Guix and if they had tried something similar or different to handle
> many versions in an automated way in the same channel commit?
> 
> Apologies that's more than I was intending to write - but hopefully that
> makes some sense!  If it doesn't I can try to flesh out specific example?

Apologies for not slotting in the reply inline, I'm not sure exactly
where to put it.

This might be a good use for package transformations. Imagine the
following:

;;; Python-dep-1

(define-publid python-dep1-week1
  ...)

(define-publid python-dep1-week2
  ...)

(define-publid python-dep1-week3
  ...)

;;; Python-dep-2

(define-publid python-dep2-week1
  ...)

(define-publid python-dep2-week2
  ...)

(define-publid python-dep2-week3
  ...)

;;; Python package

(define my-python-package-base
  (name "my-python-package-base")
  ...
  (inputs
   (list python-dep1
         python-dep2)))

(define-public my-python-package
  (inherit my-python-package-base)
  (name "my-python-package")
  (inputs
   (modify-inputs (package-inputs python-package-base)
     (replace python-dep1 python-dep1-week3)
     (replace python-dep2 python-dep2-week2))))

or if you wanted to do it recursively

(define package-inputs-for-my-python-package
  (package-input-rewriting/spec
    `(("python-dep1" . ,(const python-dep1-week3))
      ("python-dep2" . ,(const python-dep2-week2)))))

(define-public python-package-with-correct-inputs
  (package
    (inherit (package-inputs-for-ython-package my-python-package-base))
    (name "my-python-package")))


Both ideas use 'placeholder packages' so that you can swap them out for
your actual dependency, the first one only for itself and the second one
working recursively. I use the second one myself at work to replace
tensorflow with a version built for the machine it's running on, and at
home to use some newer golang libraries.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Pinning package inputs using inferiors?

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi,

On Fri, Oct 21, 2022 at 4:51 PM Phil <phil@beadling.co.uk> wrote:
>
> have the releases available ...
> but without ... an obligation on ... applications to adopt
> these changes

While I am relatively new to functional package management, I find it
inconsistent that inputs in Guix are provided by variables.

I believe the inputs should be provided by functions that deliver the
most suitable version of a package. For most packages, that is the
most recent version unless pinned.

A package definition would be the list of available versions rather
than just one version.

One day, those functions could even be combined with importers in the
sense that many versions are potentially available. Substitutes would
only be produced and served for versions actually used, plus the most
recent version.

Kind regards
Felix Lechner

Re: Pinning package inputs using inferiors?

[zimoun]

Hi,

On sam., 22 oct. 2022 at 22:58, Felix Lechner via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> wrote:

> While I am relatively new to functional package management, I find it
> inconsistent that inputs in Guix are provided by variables.

What do you mean by « variables »?


> I believe the inputs should be provided by functions that deliver the
> most suitable version of a package. For most packages, that is the
> most recent version unless pinned.

What do you mean by « inputs »?

Currently, the “new” style refers to symbol as inputs which points to
other package definitions.  A package definition somehow defines a
function to build a package.  At this level, it is hard to have a
precise meaning of « suitable version of a package», IMHO.

At the CLI level, packages are referred by their name field and the most
recent version is used by default; unless a specific version as
’@1.2.3’ is appended to the name.


> A package definition would be the list of available versions rather
> than just one version.

What do you build?  Do you build the matrix of all the combinations?

The hard task of a package manager is to provide a set of packages at
their appropriated versions that works well all together.  Aside some
specific cases where the compatibility across version is guarantee, I
miss how a mutli-version definition could work in practise.

Package transformation is somehow a way to implement package definition
for several versions.  It allows to rewrite the function definition
(package) but in the same time to keep under control the combinations.


Cheers,
simon