Reproducing a Python project environment (using guix inferiors)

Reproducing a Python project environment (using guix inferiors)

[branjam4]

Hello all!

Some time before I started using guix I worked through Python examples
within this book[1] for school. There were some differences in the
packages I installed and the listed requirements for the book (pseudo
manifest at [2]) which made things difficult for me as a (then) new
programmer.

Believing guix could more faithfully/cleanly reproduce the environment
from the book than my former approach using pip, I:
+ imported/wrote definitions for packages not in guix
+ wrote inferiors for older versions of packages that were in guix.

I'm not sure what the rhyme or reason is, but most of my inferior
attempts failed with this backtrace[3]. A couple were successful, as you
can see in this table[4].

Usually I've had good luck consulting the manual, mailing lists, and IRC
logs, but I'm at a bit of a loss this time since the nature of others'
inferior issues seems different from mine (and were probably fixed).
I've tried with the following (non-inferior) commits on a foreign
distro:
+ 71507435225f10d8d944ba183cbcc77ef953e0e5
+ f816deb9055669deaca59ac7652943adf7f11bb1
+ 0d7c95445c (and some commits shortly after this one)

Since the results consistently match[4], I supposed the issue might be
with the commits I've selected for the inferiors. But at this point I
thought I should ask for help in case I'm too far off.

From what I can gather, inferiors are the recommended way to declare
packages defined in older revisions of guix.

But how would I know what a "good" guix inferior commit is for all of
these packages?
Would using the definitions instead of asking guix to reproduce
five different worlds for one package each be an anti-pattern?
Is my issue related to being on a foreign distro, thus Guix System users
wouldn't know much about this problem?

Even though I'm mentioning this specific frustration, I enjoyed the
learning process getting to this point and want to express my
appreciation. Despite interacting with clojure and emacs lisp for at
least a year, guix/guile finally got me to use quoting, let, and lambda
in a non-trivial way. Thus the otherwise tedious work of generating five
different channel/inferior definitions as per the example in the manual
was actually a breeze!

Thanks for your help and hard work,
--Brandon

[1] Complex Network Analysis Using Python, by Dmitry Zinoviev

[2] For some of the versioned packages, I commented the newest commit
prior to guix bumping that package, which I would use for an inferior.
#+BEGIN_SRC scheme
(specifications->manifest ;inferior commit notes
 '("python" ; not trying to build this from scratch.
   "python-matplotlib@2.0.2" ;7e06086522
   "python-nltk" ;need 3.2.5
   "python-pandas@0.22.0" ;ce2cfcabfc
   ;"python-wikipedia@1.4.0" need to guix import
   ;"python-toposort@1.5" need to guix import
   "python-networkx@2.1" ;269f100330
   ;"python-community@0.10" need to guix import
   "python-numpy@1.13.3" ;4d6ed794dd
   "python-pygraphviz" ;want 1.3 but only 1.5 available
   ;;may need graphviz-dev as well
   "python-scipy@1.0.1" ;02ddafef55
   ;"python-louvain@0.14" need to guix import
   ;need to install the author's custom modules))
#+END_SRC
I used the above as a guide while doing the actual imports/inferior
work.

[3]
#+begin_example scheme
In current input:
     12:3 12 (_)
In guix/store.scm:
   623:10 11 (call-with-store _)
  1803:24 10 (run-with-store #<store-connection 256.99 129ec60> _ # _ …)
In guix/inferior.scm:
   681:12  9 (_ _)
In guix/channels.scm:
    527:2  8 (_ _)
    485:2  7 (_ _)
In ./guix/monads.scm:
    482:9  6 (_ _)
In guix/store.scm:
  1673:13  5 (_ _)
In ice-9/eval.scm:
    619:8  4 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
   626:19  3 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
    155:9  2 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
   223:20  1 (proc #(#(#(#<directory (build-self) 1842aa0>) "/…" …) …))
In unknown file:
           0 (%resolve-variable (7 . %guix-register-program) #<direc…>)
#+end_example

[4]
| package of interest | guix commit  | status |
|---------------------+--------------+--------|
| python-matplotlib   | "7e06086522" | bad    |
| python-pandas       | ce2cfcabfc   | bad    |
| python-networkx     | 269f100330   | good   |
| python-numpy        | 4d6ed794dd   | bad    |
| python-scipy        | 02ddafef55   | good   |

Re: Reproducing a Python project environment (using guix inferiors)

[zimoun]

Hi,

On Sat, 28 Nov 2020 at 18:09, branjam4@gmail.com wrote:

> Some time before I started using guix I worked through Python examples
> within this book[1] for school. There were some differences in the
> packages I installed and the listed requirements for the book (pseudo
> manifest at [2]) which made things difficult for me as a (then) new
> programmer.

Cool!  Thank you for your feedback.


> Believing guix could more faithfully/cleanly reproduce the environment
> from the book than my former approach using pip, I:
> + imported/wrote definitions for packages not in guix

Usually, this is done via the option “--load-path” or Custom Channels.

<https://guix.gnu.org/manual/devel/en/guix.html#Creating-a-Channel>

> + wrote inferiors for older versions of packages that were in guix.

You might be interested by this Channel:

  <https://gitlab.inria.fr/guix-hpc/guix-past>


> I'm not sure what the rhyme or reason is, but most of my inferior
> attempts failed with this backtrace[3]. A couple were successful, as you
> can see in this table[4].

I am not sure to understand what you are naming “inferior” here.  From
my experience, one simple way to start is:

  guix time-machine --commit=<old> \
       -- build -L <path/to/local/packages> <your-package>

where <old> is an old Guix commit providing bunch of dependencies used
by <your-package> of interest defined in the folder
<path/to/local/packages>.

A channels.scm file with the option ’--channels/-C’ seems even better.


> But how would I know what a "good" guix inferior commit is for all of
> these packages?

Let’s fix ideas and let’s say you are interested by the package
python-foo at the version 1.2.3.  This package depends on python-bar at
version x.y.z available at commit range 12345-67890 and on python-baz at
version m.n available at commit range ABCDE-FGHIJ.

Then there is no commit where python-bar *and* python-baz are both at
the correct versions (resp. x.y.z and m.n.).  In other words, the 2
commit ranges 12345-67890 and ABCDE-FGHIJ do not overlap.

Is it your use case?

BTW, a tool to find the range of commit where one specific version is
available is the Data Service.  Give a look at:

<https://data.guix.gnu.org/repository/1/branch/master/package/python-scipy/output-history>


> Would using the definitions instead of asking guix to reproduce
> five different worlds for one package each be an anti-pattern?
> Is my issue related to being on a foreign distro, thus Guix System users
> wouldn't know much about this problem?

Being on a foreign distro does not change.  The main difference between
between foreign and native is only about services, not packages.


> Even though I'm mentioning this specific frustration, I enjoyed the
> learning process getting to this point and want to express my
> appreciation. Despite interacting with clojure and emacs lisp for at
> least a year, guix/guile finally got me to use quoting, let, and lambda
> in a non-trivial way. Thus the otherwise tedious work of generating five
> different channel/inferior definitions as per the example in the manual
> was actually a breeze!

Cool!


> [1] Complex Network Analysis Using Python, by Dmitry Zinoviev

I will try to give a look if my library has this reference.


> [2] For some of the versioned packages, I commented the newest commit
> prior to guix bumping that package, which I would use for an inferior.
> #+BEGIN_SRC scheme
> (specifications->manifest ;inferior commit notes
>  '("python" ; not trying to build this from scratch.
>    "python-matplotlib@2.0.2" ;7e06086522
>    "python-nltk" ;need 3.2.5
>    "python-pandas@0.22.0" ;ce2cfcabfc
>    ;"python-wikipedia@1.4.0" need to guix import
>    ;"python-toposort@1.5" need to guix import
>    "python-networkx@2.1" ;269f100330
>    ;"python-community@0.10" need to guix import
>    "python-numpy@1.13.3" ;4d6ed794dd
>    "python-pygraphviz" ;want 1.3 but only 1.5 available
>    ;;may need graphviz-dev as well
>    "python-scipy@1.0.1" ;02ddafef55
>    ;"python-louvain@0.14" need to guix import
>    ;need to install the author's custom modules))
> #+END_SRC
> I used the above as a guide while doing the actual imports/inferior
> work.

I have tried to write a script for my personal needs running via “guix
repl”.  It fetches the JSON file from the Data Service, which give the
commit range for the required version per package.  Then I have not
finished it (yet) but the idea was to be able to spot out the commit
providing all the packages at the required versions, i.e., the
intersection.  Pieces of the necessary material is in guix/git.scm.

<https://git.savannah.gnu.org/cgit/guix.git/tree/guix/git.scm#n444>


> [3]
> #+begin_example scheme
> In current input:
>      12:3 12 (_)
> In guix/store.scm:

[...]

>     619:8  4 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>    626:19  3 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>     155:9  2 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>    223:20  1 (proc #(#(#(#<directory (build-self) 1842aa0>) "/…" …) …))
> In unknown file:
>            0 (%resolve-variable (7 . %guix-register-program) #<direc…>)
> #+end_example

What did you run?  The command and the file(s)?



> [4]
> | package of interest | guix commit  | status |
> |---------------------+--------------+--------|
> | python-matplotlib   | "7e06086522" | bad    |
> | python-pandas       | ce2cfcabfc   | bad    |
> | python-networkx     | 269f100330   | good   |
> | python-numpy        | 4d6ed794dd   | bad    |
> | python-scipy        | 02ddafef55   | good   |

What do you mean by “bad”?
Does it mean:

 1. the package does not build
 2. the package is failing
 3. else

?


All the best,
simon

Re: Reproducing a Python project environment (using guix inferiors)

[Brandon Ellington]

[-- Attachment #1: Type: text/plain, Size: 1690 bytes --]

Thanks for replying simon!

zimoun <zimon.toutoune@gmail.com> writes:

> I am not sure to understand what you are naming “inferior” here.

Sorry for the ambiguity, was not being self-deprecating. I meant
attempts to use certain guix inferiors as described in the guix manual
failed (from Package Management > Inferiors):

> Sometimes you might need to mix packages from the revision of Guix
> you’re currently running with packages available in a different
> revision of Guix.  Guix “inferiors” allow you to achieve that by
> composing different Guix revisions in arbitrary ways.

So answering your questions about what I did:
>> [4]
>> | package of interest | guix commit  | status |
>> |---------------------+--------------+--------|
>> | python-matplotlib   | "7e06086522" | bad    |
>> | python-pandas       | ce2cfcabfc   | bad    |
>> | python-networkx     | 269f100330   | good   |
>> | python-numpy        | 4d6ed794dd   | bad    |
>> | python-scipy        | 02ddafef55   | good   |

> What do you mean by “bad”?

For <package of interest>, guix has a definition for it at commit
<guix commit>. When I declare an inferior channel at <guix commit> and
try to build that channel, I note in the <status> column whether it
builds ("good") or I got the [3] message following a backtrace ("bad").
If it were to build, then I would use (first (lookup-inferior-packages
(<<package-of-interest>>-inferior))) in a manifest for the
package itself.

> What did you run?  The command and the file(s)?
I attached the file I used to define the channels and inferiors. As for
the command, I just load these definitions piecemeal into the repl.


[-- Attachment #2: python-cna-guix-inferiors.scm --]
[-- Type: text/plain, Size: 2152 bytes --]

(define-module (cna-manifests guix-inferiors)
 #:use-module (guix inferior)
 #:use-module (guix channels))

(define python-matplotlib-channel
  ;; This is the old revision from which we want to
  ;; extract python-matplotlib.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
         (commit
          "7e06086522"))))

(define-public python-matplotlib-inferior
   ;; An inferior representing the revision for python-matplotlib.
   (inferior-for-channels python-matplotlib-channel))

(define python-pandas-channel
  ;; This is the old revision from which we want to
  ;; extract python-pandas.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
         (commit
          "ce2cfcabfc"))))

(define-public python-pandas-inferior
   ;; An inferior representing the revision for python-pandas.
   (inferior-for-channels python-pandas-channel))

(define python-networkx-channel
  ;; This is the old revision from which we want to
  ;; extract python-networkx.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
         (commit
          "269f100330"))))

(define-public python-networkx-inferior
   ;; An inferior representing the revision for python-networkx.
   (inferior-for-channels python-networkx-channel))

(define python-numpy-channel
  ;; This is the old revision from which we want to
  ;; extract python-numpy.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
         (commit
          "4d6ed794dd"))))

(define-public python-numpy-inferior
   ;; An inferior representing the revision for python-numpy.
   (inferior-for-channels python-numpy-channel))

(define python-scipy-channel
  ;; This is the old revision from which we want to
  ;; extract python-scipy.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
         (commit
          "02ddafef55"))))

(define-public python-scipy-inferior
   ;; An inferior representing the revision for python-scipy.
   (inferior-for-channels python-scipy-channel))

[-- Attachment #3: Type: text/plain, Size: 6476 bytes --]


>> [3]
>> #+begin_example scheme
>> In current input:
>>      12:3 12 (_)
>> In guix/store.scm:
>
> [...]
>
>>     619:8  4 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>    626:19  3 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>     155:9  2 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>    223:20  1 (proc #(#(#(#<directory (build-self) 1842aa0>) "/…" …) …))
>> In unknown file:
>>            0 (%resolve-variable (7 . %guix-register-program) #<direc…>)
>> #+end_example

>> But how would I know what a "good" guix inferior commit is for all of
>> these packages?
>
> Let’s fix ideas and let’s say you are interested by the package
> python-foo at the version 1.2.3.  This package depends on python-bar at
> version x.y.z available at commit range 12345-67890 and on python-baz at
> version m.n available at commit range ABCDE-FGHIJ.
>
> Then there is no commit where python-bar *and* python-baz are both at
> the correct versions (resp. x.y.z and m.n.).  In other words, the 2
> commit ranges 12345-67890 and ABCDE-FGHIJ do not overlap.
>
> Is it your use case?

I do not think so. I have three different scenarios I'm working with:
1. python-foo does not exist in guix :: So I use guix import to give me
a package definition and work from there (so far this is okay, but if it
weren't because of its dependencies I see where we might need to follow
your hypothetical).
2. python-foo does exist in guix, but the desired version is in the past
:: So I (attempt to) use a guix inferior from a channel where it was
defined (this is where I have issues, but I assume that if guix had a
definition for it, then python-bar and python-baz would both be
available in the entire commit range that it defines python-foo). 3.
python-foo does exist in guix, but I need a newer version :: I inherit
from the old version, changing the necessary fields (like 1. I see where
we would need to follow the hypothetical, but I haven't seen any missing
dependencies errors).

I successfully made the channel https://github.com/branjam4/cna-python
(with packages from 3. and 1.) when I last worked on this. Hopefully
having a successful channel build isn't deceiving me, but I thought
given the successful build that I got lucky with needing to figure out
dependencies. My primary block is this "unregisted program" error I get
when using guix inferiors that I'd like to add to my manifest.

> From my experience, one simple way to start is:
>
>   guix time-machine --commit=<old> \
>        -- build -L <path/to/local/packages> <your-package>
>
> where <old> is an old Guix commit providing bunch of dependencies used
> by <your-package> of interest defined in the folder
> <path/to/local/packages>.
> 
> A channels.scm file with the option ’--channels/-C’ seems even better.

Hmm, my end goal is to make /something/ that will be simple for
people with minimal assumed experience in guix (my professor, for
example) to run in a virtual machine with guix and direnv available, so
they can pull a git repository, then follow code examples from the book
without focusing on dependency building (there would be the initial cost
of waiting for things to build the first time, though). Assuming
time-machine works, is it straightforward to make a profile from it (or
something else enabling direnv to understand what I want when I go into
the cna-python directory)? That was something I missed while reading the
time-machine part of the manual.

>> Believing guix could more faithfully/cleanly reproduce the environment
>> from the book than my former approach using pip, I:
>> + imported/wrote definitions for packages not in guix
>
> Usually, this is done via the option “--load-path” or Custom Channels.
>
> <https://guix.gnu.org/manual/devel/en/guix.html#Creating-a-Channel>

True, I have older generations with my repo as a channel:
#+BEGIN_EXAMPLE bash
  cna-python 35290a1
    repository URL: https://github.com/branjam4/cna-python.git
    branch: master
    commit: 35290a12d09ad528babba6052d37bcd09bd4c9a4
  guix 0e1b095
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 0e1b0958bde5ccc34a4fed9a09cf949d5f9c9519
#+END_EXAMPLE

Though with recent guix pulls it fails at that same commit, saying that
syntax has changed. I'm not as concerned about that at the moment
compared with the guix inferior channels issue though.

>> + wrote inferiors for older versions of packages that were in guix.
>
> You might be interested by this Channel:
>
>   <https://gitlab.inria.fr/guix-hpc/guix-past>
>

I see. First looked at this channel a few days ago, but went straight to
its package definitions and only skimmed the readme (which at the
beginning mentioned using software from ten years ago, throwing me off
from accepting it for my use case). Under the "Hacking" section it says:

> It makes most sense to add software dating back to before 2019, around
> the time where time travel became possible.

The packages I'm trying to reproduce would have been current in mid
2017. Does that mean neither guix inferiors nor guix time-machine will
work reliably for commits in that year?

> BTW, a tool to find the range of commit where one specific version is
> available is the Data Service.  Give a look at:
>
> <https://data.guix.gnu.org/repository/1/branch/master/package/python-scipy/output-history>
>

Thanks, this definitely seems more robust than my git log grepping!

> I have tried to write a script for my personal needs running via “guix
> repl”.  It fetches the JSON file from the Data Service, which give the
> commit range for the required version per package.  Then I have not
> finished it (yet) but the idea was to be able to spot out the commit
> providing all the packages at the required versions, i.e., the
> intersection.  Pieces of the necessary material is in guix/git.scm.
>
> <https://git.savannah.gnu.org/cgit/guix.git/tree/guix/git.scm#n444>

Ah, definitely intriguing. While I was on my grepping expedition I
wondered whether I could use one or two old guix commits instead of
five, so I like this concept.

> Being on a foreign distro does not change.  The main difference between
> between foreign and native is only about services, not packages.

Figured, thank you for confirming.

--Brandon

Re: Reproducing a Python project environment (using guix inferiors)

[zimoun]

Hi,

On Mon, 30 Nov 2020 at 14:39, Brandon Ellington <branjam4@gmail.com> wrote:

>>> | package of interest | guix commit  | status |
>>> |---------------------+--------------+--------|
>>> | python-matplotlib   | "7e06086522" | bad    |
>>> | python-pandas       | ce2cfcabfc   | bad    |
>>> | python-networkx     | 269f100330   | good   |
>>> | python-numpy        | 4d6ed794dd   | bad    |
>>> | python-scipy        | 02ddafef55   | good   |

Which version of <package of interest> do you need?


>> What do you mean by “bad”?
>
> For <package of interest>, guix has a definition for it at commit
> <guix commit>. When I declare an inferior channel at <guix commit> and
> try to build that channel, I note in the <status> column whether it
> builds ("good") or I got the [3] message following a backtrace ("bad").
> If it were to build, then I would use (first (lookup-inferior-packages
> (<<package-of-interest>>-inferior))) in a manifest for the
> package itself.

Because, I think that “bad” commits 7e06086522 (Jan 2018), ce2cfcabfc
(Feb 2018) and 4d6ed794dd (Jan 2018) simply pre-date the introduction of
«Inferior».  Therefore, they are not reachable by this mechanism.
That’s one of the motivations for the channel “guix-past”. :-)

Basically, the Backtrace should say something as:

--8<---------------cut here---------------start------------->8---
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
error: %guix-register-program: unbound variable
--8<---------------cut here---------------end--------------->8---


>> What did you run?  The command and the file(s)?
> I attached the file I used to define the channels and inferiors. As for
> the command, I just load these definitions piecemeal into the repl.

I see.

>>>     619:8  4 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>>    626:19  3 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>>     155:9  2 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>>    223:20  1 (proc #(#(#(#<directory (build-self) 1842aa0>) "/…" …) …))
>>> In unknown file:
>>>            0 (%resolve-variable (7 . %guix-register-program) #<direc…>)
>>> #+end_example

Ah, that’s the same backtrace message showed above. :-)


> I do not think so. I have three different scenarios I'm working with:
> 1. python-foo does not exist in guix :: So I use guix import to give me
> a package definition and work from there (so far this is okay, but if it
> weren't because of its dependencies I see where we might need to follow
> your hypothetical).
> 2. python-foo does exist in guix, but the desired version is in the past
> :: So I (attempt to) use a guix inferior from a channel where it was
> defined (this is where I have issues, but I assume that if guix had a
> definition for it, then python-bar and python-baz would both be
> available in the entire commit range that it defines python-foo).
> 3.
> python-foo does exist in guix, but I need a newer version :: I inherit
> from the old version, changing the necessary fields (like 1. I see where
> we would need to follow the hypothetical, but I haven't seen any missing
> dependencies errors).
>
> I successfully made the channel https://github.com/branjam4/cna-python
> (with packages from 3. and 1.) when I last worked on this. Hopefully
> having a successful channel build isn't deceiving me, but I thought
> given the successful build that I got lucky with needing to figure out
> dependencies. My primary block is this "unregisted program" error I get
> when using guix inferiors that I'd like to add to my manifest.

What I will do is: a) fix a Guix version b) create a channel containing
all the necessary variants (backporting dependencies if required, i.e.,
copy/pasting old package definition and fix them if they does not build
anymore) to build my short list of <packages of interest>.

This is more or less the strategy used to feed the channel “guix-past“.

And I will not use the inferior mechanism because it adds a lot of
complexity and will not solve your problem since old packages are not
reachable and/or your need to add old dependencies.


>> From my experience, one simple way to start is:
>>
>>   guix time-machine --commit=<old> \
>>        -- build -L <path/to/local/packages> <your-package>
>>
>> where <old> is an old Guix commit providing bunch of dependencies used
>> by <your-package> of interest defined in the folder
>> <path/to/local/packages>.
>> 
>> A channels.scm file with the option ’--channels/-C’ seems even better.
>
> Hmm, my end goal is to make /something/ that will be simple for
> people with minimal assumed experience in guix (my professor, for
> example) to run in a virtual machine with guix and direnv available, so
> they can pull a git repository, then follow code examples from the book
> without focusing on dependency building (there would be the initial cost
> of waiting for things to build the first time, though). Assuming
> time-machine works, is it straightforward to make a profile from it (or
> something else enabling direnv to understand what I want when I go into
> the cna-python directory)? That was something I missed while reading the
> time-machine part of the manual.

The time-machine is orthogonal with the way to distribute, IMHO.  It is
simply a easy CLI to fix the Guix version.

Usually, I do:

  guix describe -f channels > channels.scm

  edit my-manifest.scm # containing <package-of-interest>
  
  [.. hack my custom variants ..]
  [.. backport variant dependencies when I need for these variants ..]
  guix time-machine -C channels.scm
       -- build -L /path/to/variants/ -m my-manifest.scm
  [.. loop hack until it works ..]

and then I create what my audience expects, e.g., Docker image:

  guix time-machine -C channels.scm
       -- pack -f docker -L /path/to/variants/ -m my-manifest.scm

or relocatable tarball (pack -RR) or ‘system docker-image’ or whatever.


The /path/to/variants is a Git repo and I add the files channels.scm and
manifest.scm.  And now, it is easy to rebuild everything in the future.


> Though with recent guix pulls it fails at that same commit, saying that
> syntax has changed. I'm not as concerned about that at the moment
> compared with the guix inferior channels issue though.

The fail is about your channel.  Well, I better understand your initial
questions:

        From what I can gather, inferiors are the recommended way to declare
        packages defined in older revisions of guix.

From my understanding, the answer is: it depends.

For commits after the complete introduction of Inferiors mechanism
(~July 2018, please read [1]), it is ok.  Be aware that each inferior
has a cost and if your package foo depends on X packages coming from
inferiors, then it means you are running under the hood X time: ‘guix
pull’ then ’guix build <dependency>’ which could require to also build
other dependencies, therefore installing foo could be long.

Explicitly, imagine that python-foo depends on python, python-bar and
python-baz.  Imagine that you use 2 inferiors, one for python-bar and
one for python-baz.  Now imagine that python-bar and python-baz depends
both on the package python.  Then, there is a high probability to build
3 times tiny variants of the package python.  And maybe the 3 packages
python-foo, python-bar and python-baz perfectly works with the same
variant of python.  And it is worse because that applies to all the
implicit dependencies (compilers and so on).

For commits before ~July 2018, the strategy is the one of ‘guix-past’
channel; which I roughly described above.

1: https://guix.gnu.org/en/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/


        But how would I know what a "good" guix inferior commit is for
        all of these packages?  Would using the definitions instead of
        asking guix to reproduce five different worlds for one package
        each be an anti-pattern?

Now, these questions and the other ones about ’guix-past’ are answered,
right?



All the best,
simon

Re: Reproducing a Python project environment (using guix inferiors)

[Brandon Ellington]

Thanks again for the reply simon!

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Mon, 30 Nov 2020 at 14:39, Brandon Ellington <branjam4@gmail.com> wrote:
>
>>>> | package of interest | guix commit  | status |
>>>> |---------------------+--------------+--------|
>>>> | python-matplotlib   | "7e06086522" | bad    |
>>>> | python-pandas       | ce2cfcabfc   | bad    |
>>>> | python-networkx     | 269f100330   | good   |
>>>> | python-numpy        | 4d6ed794dd   | bad    |
>>>> | python-scipy        | 02ddafef55   | good   |
>
> Which version of <package of interest> do you need?
You can kind of get it from [2] in my initial email, but here is a
cleaner table:

| package of interest | version (rec. minimum) |
|---------------------+------------------------|
| python              |                  3.6.5 | do not rebuild
|---------------------+------------------------|
| python-matplotlib   |                  2.0.2 | used to exist in guix
| python-pandas       |                 0.22.0 |
| python-networkx     |                    2.1 |
| python-numpy        |                 1.13.3 |
| python-scipy        |                  1.0.1 |
|---------------------+------------------------|
| python-pygraphviz   |                    1.3 | bump from current guix
| python-nltk         |                  3.2.5 |
|---------------------+------------------------|
| python-wikipedia    |                  1.4.0 | importing
| python-toposort     |                    1.5 |
| python-community    |                   0.10 |
| python-louvain      |                   0.14 |
|---------------------+------------------------|
| (author's own)      |       (not applicable) |

> Because, I think that “bad” commits 7e06086522 (Jan 2018), ce2cfcabfc
> (Feb 2018) and 4d6ed794dd (Jan 2018) simply pre-date the introduction of
> «Inferior».  Therefore, they are not reachable by this mechanism.
> That’s one of the motivations for the channel “guix-past”. :-)
> 
> [...]
> 
> For commits before ~July 2018, the strategy is the one of ‘guix-past’
> channel; which I roughly described above.
>
> 1: https://guix.gnu.org/en/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/

Ah, lesson learned! I'll direct my attention toward guix-past then :)

> For commits after the complete introduction of Inferiors mechanism
> (~July 2018, please read [1]), it is ok.  Be aware that each inferior
> has a cost and if your package foo depends on X packages coming from
> inferiors, then it means you are running under the hood X time: ‘guix
> pull’ then ’guix build <dependency>’ which could require to also build
> other dependencies, therefore installing foo could be long.

Indeed I noticed this. Building the two good channels was not too bad,
but I have not tried building networkx or scipy yet. As for your
hypothetical...

> Explicitly, imagine that python-foo depends on python, python-bar and
> python-baz.  Imagine that you use 2 inferiors, one for python-bar and
> one for python-baz.  Now imagine that python-bar and python-baz depends
> both on the package python.  Then, there is a high probability to build
> 3 times tiny variants of the package python.  And maybe the 3 packages
> python-foo, python-bar and python-baz perfectly works with the same
> variant of python.  And it is worse because that applies to all the
> implicit dependencies (compilers and so on).

...I can imagine it, and that would be terrible yes. I thought python
wouldn't be a dependency of python packages in part for that reason
though. Or maybe I read this for something else like emacs and
extrapolated to python. Maybe I'm dreaming :P. In any case, the overall
idea makes sense, especially the implicit dependencies thing. For me, if
I went for full reproducibility including the older version of python,
then all these packages would work on 3.6.5, and I'd still have a lot to
learn about the implicit dependencies.

> What I will do is: a) fix a Guix version b) create a channel containing
> all the necessary variants (backporting dependencies if required, i.e.,
> copy/pasting old package definition and fix them if they does not build
> anymore) to build my short list of <packages of interest>.
>
> This is more or less the strategy used to feed the channel “guix-past“.
>
> And I will not use the inferior mechanism because it adds a lot of
> complexity and will not solve your problem since old packages are not
> reachable and/or your need to add old dependencies.

Got it. The last part of the last sentence is surprising (because the
mechanism must have been the right choice to someone/for something) but
everything else adds up.

> The time-machine is orthogonal with the way to distribute, IMHO.  It is
> simply a easy CLI to fix the Guix version.
>
> Usually, I do:
>
>   guix describe -f channels > channels.scm
>
>   edit my-manifest.scm # containing <package-of-interest>
>   
>   [.. hack my custom variants ..]
>   [.. backport variant dependencies when I need for these variants ..]
>   guix time-machine -C channels.scm
>        -- build -L /path/to/variants/ -m my-manifest.scm
>   [.. loop hack until it works ..]
>
> and then I create what my audience expects, e.g., Docker image:
>
>   guix time-machine -C channels.scm
>        -- pack -f docker -L /path/to/variants/ -m my-manifest.scm
>
> or relocatable tarball (pack -RR) or ‘system docker-image’ or whatever.
>
>
> The /path/to/variants is a Git repo and I add the files channels.scm and
> manifest.scm.  And now, it is easy to rebuild everything in the future.

Okay, definitely worth a try!

> Now, these questions and the other ones about ’guix-past’ are answered,
> right?

Yes, I appreciate you helping me out here! Now I can go down the correct
rabbit hole to build my project :)

--Brandon