Finding versions of packages (was: [outreachy] Walk through the Git history)

Finding versions of packages (was: [outreachy] Walk through the Git history)

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 2511 bytes --]

Hi there! I've been following the "guix git" discourse with interest
because I know a lot of people who care about pinning packages to
specific versions and selecting specific versions of software to
install. This constituency currently relies heavily on systems like rvm,
nvm, and conda to manage the available versions of libraries and
runtimes they care about.

The proposal to add a "guix git" command to help find old versions of
packages seems unnecessarily indirect to me. Certainly you can learn
interesting things about the history of Guix packages using the git
log,  but that doesn't address the questions "which versions of this
software are available and how to I use them" except via an abstraction
that depends on the operator's tacit knowledge, including what "git" is
and how to read git commits. We can do better.

I propose a different approach: the "guix versions" subcommand provides
direct answers to practical user questions.
- What package versions are available?
- How do I use them?
- Which versions are known to be vulnerable?
- Which have available substitutes?

For example, the command "guix versions esbuild" can provide this
output:
name: esbuild version: 0.8.21 guix-hash: eee3af86c7 name: esbuild
version: 0.8.19 guix-hash: 6374a25357 name: esbuild version: 0.8.16
guix-hash: 8c3caf9c5d vulnerabilities: cve-2020-1337, cve-2019-1024
...and so on.

Then, to make the output directly actionable, we extend Guix to accept
recfile manifests following the same structure, such that given the
following file "packages.rec:"
name: esbuild version: 0.8.19 guix-hash: 6374a25357 name: python-
html5lib version: 1.1 guix-hash: 6374a25357
…we can use those exact packages using "guix environment -m
packages.rec", or find which of those packages have substitutes using
"guix weather -m packages.rec"

I make this proposal because I perceive the following advantages:
- It makes no the assumption that the operator knows what git is, cares
about the internal git structure of Guix, wants to read git commits, or
wants to master the details of the internal "inferiors" mechanism we use
to provide this functionality.
- It answers directly the operator's practical questions rather than
providing oblique advice which depend upon the operator's tacit
knowledge.
- The user can act immediately upon the output of the command to create
a manifest that represents the software they want to use.

[-- Attachment #2: Type: text/html, Size: 4925 bytes --]

Re: Finding versions of packages

[Ludovic Courtès]

Hi!

Ryan Prior <ryanprior@hey.com> skribis:

> I propose a different approach: the "guix versions" subcommand provides
> direct answers to practical user questions.
> - What package versions are available?
> - How do I use them?
> - Which versions are known to be vulnerable?
> - Which have available substitutes?
>
> For example, the command "guix versions esbuild" can provide this
> output:
> name: esbuild version: 0.8.21 guix-hash: eee3af86c7 name: esbuild
> version: 0.8.19 guix-hash: 6374a25357 name: esbuild version: 0.8.16
> guix-hash: 8c3caf9c5d vulnerabilities: cve-2020-1337, cve-2019-1024
> ...and so on.

Interesting.  I think it could grab information locally (from the
available ‘guix pull’ generations) and from the Guix Data Service.

However, it might give the false idea that users can pick
package versions independently (as in: I want esbuild X, GCC Y, and Go
Z), which is not really the case: packages are interrelated.

> I make this proposal because I perceive the following advantages:
> - It makes no the assumption that the operator knows what git is, cares
> about the internal git structure of Guix, wants to read git commits, or
> wants to master the details of the internal "inferiors" mechanism we use
> to provide this functionality.

Yeah, I think ‘guix git log’ can be very useful for developers, but is
not a great UI for browsing package history for the reasons you give.

> - It answers directly the operator's practical questions rather than
> providing oblique advice which depend upon the operator's tacit
> knowledge.
> - The user can act immediately upon the output of the command to create
> a manifest that represents the software they want to use.

Yes, that is a really nice property, as long as it doesn’t suggest that
one can do that with any number of packages at once.  (Whether ‘-m’
should accept rec files or not can be treated separately.)

Thanks,
Ludo’.

Re: Finding versions of packages

[zimoun]

Hi Ludo,

On Mon, 14 Dec 2020 at 11:38, Ludovic Courtès <ludo@gnu.org> wrote:

> However, it might give the false idea that users can pick
> package versions independently (as in: I want esbuild X, GCC Y, and Go
> Z), which is not really the case: packages are interrelated.

Someone tried that on help-guix.  The idea was to recreate a Python
environment.  It was an extensive use of inferiors: one inferior per
package.  We already discussed that when discussing how to recreate a
profile from a <profile>/manifest file.  

Just to point the manual about inferiors:

--8<---------------cut here---------------start------------->8---
   Sometimes you might need to mix packages from the revision of Guix
you’re currently running with packages available in a different revision
of Guix.  Guix “inferiors” allow you to achieve that by composing
different Guix revisions in arbitrary ways.
--8<---------------cut here---------------end--------------->8---

Therefore, I want esbuild X (compiled with GCC A and foo A), GCC Y (compiled with
foo B) and Go Z (compiled with foo C).  It is technically possible, right?

<https://yhetil.org/guix/875z5m5v0q.fsf@branjam-ASM100.i-did-not-set--mail-host-address--so-tickle-me>


Cheers,
simon

Re: Finding versions of packages

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Mon, 14 Dec 2020 at 11:38, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> However, it might give the false idea that users can pick
>> package versions independently (as in: I want esbuild X, GCC Y, and Go
>> Z), which is not really the case: packages are interrelated.
>
> Someone tried that on help-guix.  The idea was to recreate a Python
> environment.  It was an extensive use of inferiors: one inferior per
> package.  We already discussed that when discussing how to recreate a
> profile from a <profile>/manifest file.  
>
> Just to point the manual about inferiors:
>
>    Sometimes you might need to mix packages from the revision of Guix
> you’re currently running with packages available in a different revision
> of Guix.  Guix “inferiors” allow you to achieve that by composing
> different Guix revisions in arbitrary ways.
>
> Therefore, I want esbuild X (compiled with GCC A and foo A), GCC Y (compiled with
> foo B) and Go Z (compiled with foo C).  It is technically possible, right?

It is possible, and it’s nice to have.

However, doing such composition on a per-package basis and as the
default way of composing packages is inefficient and, more importantly,
the resulting compositions may not work.  A package written for Python 2
may not work with Python 3, and so on.

Ludo’.

Re: Finding versions of packages

[zimoun]

Hi,

On Tue, 15 Dec 2020 at 12:16, Ludovic Courtès <ludo@gnu.org> wrote:

> However, doing such composition on a per-package basis and as the
> default way of composing packages is inefficient and, more importantly,
> the resulting compositions may not work.  A package written for Python 2
> may not work with Python 3, and so on.

About inefficiency, I agree.  The number of inferiors should be
minimized, IMHO.

By “the resulting composition may not work”, you mean that what is
propagated may be incompatible, right?  For example, bytecode or ABI
incompatibilities.

Cheers,
simon

Re: Finding versions of packages

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Tue, 15 Dec 2020 at 12:16, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> However, doing such composition on a per-package basis and as the
>> default way of composing packages is inefficient and, more importantly,
>> the resulting compositions may not work.  A package written for Python 2
>> may not work with Python 3, and so on.
>
> About inefficiency, I agree.  The number of inferiors should be
> minimized, IMHO.
>
> By “the resulting composition may not work”, you mean that what is
> propagated may be incompatible, right?  For example, bytecode or ABI
> incompatibilities.

Yes, all sort of incompatibilities could arise.

The great thing with having all the package definitions in a single tree
is that we know they were tested to work together well (hopefully).  If
you start composing packages taken from different revisions, you lose
that guarantee.

Ludo’.

Re: Finding versions of packages (was: [outreachy] Walk through the Git history)

[zimoun]

Hi,

Thanks for your inputs.


On Sat, 12 Dec 2020 at 22:08, Ryan Prior <ryanprior@hey.com> wrote:

> I propose a different approach: the "guix versions" subcommand provides
> direct answers to practical user questions.
> - What package versions are available?
> - How do I use them?
> - Which versions are known to be vulnerable?
> - Which have available substitutes?

Roughly speaking, it is already possible with the Data Service.  Missing
glue code to use JSONs. :-)

<https://data.guix.gnu.org/repository/1/branch/master/package/gmsh/output-history.json>

For the last Hacktathon about “time travel”, I tried to implement
something fetching these JSONs, collecting the hash, find intersection
about 2 commit ranges, etc.  But 1) I do not want to depend on external
services, especially when it starts late on 2019 and 2) custom channels
will be always missing.

<https://hpc.guix.info/blog/2020/07/reproducible-research-hackathon-experience-report>


(Aside the “bug” with the accuracy of in/out hash commits, see
<https://yhetil.org/guix/868sazm4tu.fsf@gmail.com>.)


> For example, the command "guix versions esbuild" can provide this
> output:
> name: esbuild version: 0.8.21 guix-hash: eee3af86c7 name: esbuild
> version: 0.8.19 guix-hash: 6374a25357 name: esbuild version: 0.8.16
> guix-hash: 8c3caf9c5d vulnerabilities: cve-2020-1337, cve-2019-1024
> ...and so on.

Your proposal is the final aim when I proposed “guix git log”.  It comes
from this old discussion:

<https://yhetil.org/guix/CAOOOohSzUezKvm=RO0bXRGH3m0eo2x0cOTvd--vARxWoqtceaQ@mail.gmail.com>

which clearly shows that something is missing; especially when all the
information is somehow already there.

In my views, “guix git log” is the first actionable step in the
direction of what you are proposing.  Your description is what I want
too at the end, so I proposed the “guix git log” starting point with
then potential incremental improvements.

Since ~/.cache/guix/checkouts/ provides all the information required to
implement this, what is missing is somehow “glue code” and all is
already there.  Even if it is not straightforward to get the accurate
commit range: does Gmsh at commit c5372cd488 is the same version as Gmsh
at commit 44e65a75, even if both are the code source from upstream
4.6.0?  These 2 commits do not have the same derivation because one of
the dependency changed.  Well, “guix versions” is more complicated than
it appears at first (parse the commit message).

Therefore, “guix git log” is only the first step with a concrete
outcome:

  guix git log | grep esbuild

This path will provide information about performance, what is missing on
the Guile-Git side and how to deal with multichannels, etc.  Then, yes
the next idea–out of the Outreachy scope–is to create a local
cache–probably substituable–with all the versions and the commit hash
in/out per version, and probably also add bit of graph theory to find
commit range with multi-packages multi-version; in order to reduce the
number of inferiors to create.

Since it is piece of work, the idea is to implement one step after the
other. :-)

> Then, to make the output directly actionable, we extend Guix to accept
> recfile manifests following the same structure, such that given the
> following file "packages.rec:"
> name: esbuild version: 0.8.19 guix-hash: 6374a25357 name: python-
> html5lib version: 1.1 guix-hash: 6374a25357
> …we can use those exact packages using "guix environment -m
> packages.rec", or find which of those packages have substitutes using
> "guix weather -m packages.rec"

Ah interesting.  Personally, I am taking the other direction: adding
other “formatters“ than ’package->recutils’, for instance
’package->bibtex’ or ’package->custom’ or ’package->name-it’, because it
should be possible to display the information as I want.  For example:

  guix search emacs --format=oneline
  emacs                 27.1            The extensible, customizable, self-documenting text editor  
  emacs-with-editor     2.9.4-1.c4768f5 Emacs library for using Emacsclient as EDITOR  
  emacs-restart-emacs   0.1.1-1.9aa90d3 Restart Emacs from within Emacs  
  emacs-spacemacs-theme 0-1.f79c40f     Light and dark theme for spacemacs that supports GUI and terminal 
  […]

Therefore, adding somehow a formatter compatible with the ’--manifest’
option.


All the best,
simon