How to get the package metadata as structured data?

How to get the package metadata as structured data?

[Philippe Ombredanne]

Is there a way to get all the package metadata as pure
data beside the html page [1] or the Guile sources [2]?
Like some JSON of sorts or similar?

That would be somewhat needed to audit the licenses
asserted in the metadata and compare them with actually
detected licenses.

 [1] https://www.gnu.org/software/guix/packages/
 [2] http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/linux.scm

-- 
Cordially
Philippe Ombredanne

Re: How to get the package metadata as structured data?

[David Craven]

Hi Philippe,

Thank you for your interest in guix =)

All package metadata can be accessed easily through scheme. There is
no way to export this data currently, although it probably wouldn't be
too hard to implement.

The approach I was planning on taking is the following:
Take a package, remove all build-phases except unpack, add a phase to
run scancode and add the output json to the store.

This way guix already takes care of fetching and unpacking the package
and caching the result. Unless a build input changes, for example
scancode (an input to the scancode phase) is updated, or a scanned
package is updated, running a tool that I'll call `guix license` for
now will work of the json file that was already generated. This means
that improving the reporting or postprocessing the results doesn't
require rescanning a dependency.

Maybe someone else has other/better ;-) thoughts on this?

Cheers
David

Re: How to get the package metadata as structured data?

[Ludovic Courtès]

Hi!

David Craven <david@craven.ch> skribis:

> All package metadata can be accessed easily through scheme. There is
> no way to export this data currently, although it probably wouldn't be
> too hard to implement.

One way is:

  guix package -s .

which returns a recutils database¹, which is essentially plain text.

Alternately we could write a snippet to export it as JSON if you prefer.

Ludo’.

¹ https://gnu.org/s/recutils/manual/html_node/The-Rec-Format.html

RE: How to get the package metadata as structured data?

[Cook, Malcolm]

Hi,

Could the guix project please make available the results of `guix package -s .` that would list all available packages in recutils format for those who do not (yet) have fully functioning guix install?

Thanks?


 > -----Original Message-----
 > From: Guix-devel [mailto:guix-devel-bounces+mec=stowers.org@gnu.org]
 > On Behalf Of Ludovic Courtès
 > Sent: Thursday, August 04, 2016 4:09 PM
 > To: Philippe Ombredanne <pombredanne@nexb.com>; David Craven
 > <david@craven.ch>
 > Cc: guix-devel <guix-devel@gnu.org>
 > Subject: Re: How to get the package metadata as structured data?
 > 
 > Hi!
 > 
 > David Craven <david@craven.ch> skribis:
 > 
 > > All package metadata can be accessed easily through scheme. There is
 > > no way to export this data currently, although it probably wouldn't be
 > > too hard to implement.
 > 
 > One way is:
 > 
 >   guix package -s .
 > 
 > which returns a recutils database¹, which is essentially plain text.
 > 
 > Alternately we could write a snippet to export it as JSON if you prefer.
 > 
 > Ludo’.
 > 
 > ¹ https://gnu.org/s/recutils/manual/html_node/The-Rec-Format.html

Re: How to get the package metadata as structured data?

[David Craven]

This should works, but it's pretty slow...

guix package --list-available | grep -o --regex "^[a-zA-Z0-9-]*" |
xargs -n1 guix package -s 1> guix-metadata.txt

time sh -c "guix package -s guix &> /dev/null"
sh -c "guix package -s guix &> /dev/null"  2.42s user 0.07s system
113% cpu 2.203 total

Re: How to get the package metadata as structured data?

[David Craven]

I aborted, since I realised that guix package -s doesn't include the
source url and hash, which would be important for a testsuite...

Some interesting measurements:
guix package --list-available | grep -o --regex "^[a-zA-Z0-9-]*" | wc -l
3797

cat guix-metadata.txt | grep --regex "^name: .*$" | wc -l
13949

looks like we have many package variants?

and an example of what guix package -s guix exports

guix package -s guix
name: guix
version: 0.10.0
outputs: out
systems: x86_64-linux i686-linux armhf-linux mips64el-linux
dependencies: bzip2-1.0.6 emacs-magit-popup-2.7.0 emacs-minimal-24.5
geiser-0.8.1
+ gnutls-3.5.2 guile-2.0.11 guile-json-0.5.0 gzip-1.8 libgcrypt-1.7.0
pkg-config-0.29
+ sqlite-3.12.2 zlib-1.2.8
location: gnu/packages/package-management.scm:71:2
homepage: http://www.gnu.org/software/guix
license: GPL 3+
synopsis: Functional package manager for installed software packages
and versions
description: GNU Guix is a functional package manager for the GNU
system, and is also
+ a distribution thereof.  It includes a virtual machine image.
Besides the usual
+ package management features, it also supports transactional upgrades
and roll-backs,
+ per-user profiles, and much more.  It is based on the Nix package manager.

name: guix
version: 0.10.0-1.97c8
outputs: out
systems: x86_64-linux i686-linux armhf-linux mips64el-linux
dependencies: autoconf-wrapper-2.69 automake-1.15 bzip2-1.0.6
emacs-magit-popup-2.7.0
+ emacs-minimal-24.5 geiser-0.8.1 gettext-0.19.8 gnutls-3.5.2 graphviz-2.38.0
+ guile-2.0.11 guile-json-0.5.0 gzip-1.8 help2man-1.47.4 libgcrypt-1.7.0
+ pkg-config-0.29 sqlite-3.12.2 texinfo-6.1 zlib-1.2.8
location: gnu/packages/package-management.scm:216:4
homepage: http://www.gnu.org/software/guix
license: GPL 3+
synopsis: Functional package manager for installed software packages
and versions
description: GNU Guix is a functional package manager for the GNU
system, and is also
+ a distribution thereof.  It includes a virtual machine image.
Besides the usual
+ package management features, it also supports transactional upgrades
and roll-backs,
+ per-user profiles, and much more.  It is based on the Nix package manager.

Re: How to get the package metadata as structured data?

[Aljosha Papsch]

[-- Attachment #1: Type: text/plain, Size: 2479 bytes --]

guix package prints its output in rec format. You can use recutils to process the output.

Am 5. August 2016 13:04:49 MESZ, schrieb David Craven <david@craven.ch>:
>I aborted, since I realised that guix package -s doesn't include the
>source url and hash, which would be important for a testsuite...
>
>Some interesting measurements:
>guix package --list-available | grep -o --regex "^[a-zA-Z0-9-]*" | wc
>-l
>3797
>
>cat guix-metadata.txt | grep --regex "^name: .*$" | wc -l
>13949
>
>looks like we have many package variants?
>
>and an example of what guix package -s guix exports
>
>guix package -s guix
>name: guix
>version: 0.10.0
>outputs: out
>systems: x86_64-linux i686-linux armhf-linux mips64el-linux
>dependencies: bzip2-1.0.6 emacs-magit-popup-2.7.0 emacs-minimal-24.5
>geiser-0.8.1
>+ gnutls-3.5.2 guile-2.0.11 guile-json-0.5.0 gzip-1.8 libgcrypt-1.7.0
>pkg-config-0.29
>+ sqlite-3.12.2 zlib-1.2.8
>location: gnu/packages/package-management.scm:71:2
>homepage: http://www.gnu.org/software/guix
>license: GPL 3+
>synopsis: Functional package manager for installed software packages
>and versions
>description: GNU Guix is a functional package manager for the GNU
>system, and is also
>+ a distribution thereof.  It includes a virtual machine image.
>Besides the usual
>+ package management features, it also supports transactional upgrades
>and roll-backs,
>+ per-user profiles, and much more.  It is based on the Nix package
>manager.
>
>name: guix
>version: 0.10.0-1.97c8
>outputs: out
>systems: x86_64-linux i686-linux armhf-linux mips64el-linux
>dependencies: autoconf-wrapper-2.69 automake-1.15 bzip2-1.0.6
>emacs-magit-popup-2.7.0
>+ emacs-minimal-24.5 geiser-0.8.1 gettext-0.19.8 gnutls-3.5.2
>graphviz-2.38.0
>+ guile-2.0.11 guile-json-0.5.0 gzip-1.8 help2man-1.47.4
>libgcrypt-1.7.0
>+ pkg-config-0.29 sqlite-3.12.2 texinfo-6.1 zlib-1.2.8
>location: gnu/packages/package-management.scm:216:4
>homepage: http://www.gnu.org/software/guix
>license: GPL 3+
>synopsis: Functional package manager for installed software packages
>and versions
>description: GNU Guix is a functional package manager for the GNU
>system, and is also
>+ a distribution thereof.  It includes a virtual machine image.
>Besides the usual
>+ package management features, it also supports transactional upgrades
>and roll-backs,
>+ per-user profiles, and much more.  It is based on the Nix package
>manager.

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[-- Attachment #2: Type: text/html, Size: 3068 bytes --]

Re: How to get the package metadata as structured data?

[Philippe Ombredanne]

David Craven <david@craven.ch> wrote:
> I aborted, since I realised that guix package -s doesn't include the
> source url and hash, which would be important for a testsuite...

IMHO, if the rec data is the only way to get to the packages data, the
source url would be rather essential to get in.

-- 
Cordially
Philippe Ombredanne

Re: How to get the package metadata as structured data?

[Ricardo Wurmus]

Philippe Ombredanne <pombredanne@nexb.com> writes:

> David Craven <david@craven.ch> wrote:
>> I aborted, since I realised that guix package -s doesn't include the
>> source url and hash, which would be important for a testsuite...
>
> IMHO, if the rec data is the only way to get to the packages data, the
> source url would be rather essential to get in.

The recutils output is not the *only* way to access the package data.
All packages in Guix are just Scheme variables.  Package data are
available as S-expressions and can be read by Scheme programs or parsed
by external applications.  The recutils output is just an additional
format used when interacting with Guix on the command line.

(Personally, I’m not enthusiastic about adding a serialised form of the
source field to the recutils output.)

~~ Ricardo

Re: How to get the package metadata as structured data?

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

2016-08-10 13:46 GMT+02:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Philippe Ombredanne <pombredanne@nexb.com> writes:
>
> > David Craven <david@craven.ch> wrote:
> >> I aborted, since I realised that guix package -s doesn't include the
> >> source url and hash, which would be important for a testsuite...
> >
> > IMHO, if the rec data is the only way to get to the packages data, the
> > source url would be rather essential to get in.
>
> The recutils output is not the *only* way to access the package data.
> All packages in Guix are just Scheme variables.  Package data are
> available as S-expressions and can be read by Scheme programs or parsed
> by external applications.  The recutils output is just an additional
> format used when interacting with Guix on the command line.
>
> (Personally, I’m not enthusiastic about adding a serialised form of the
> source field to the recutils output.)
>
> ~~ Ricardo
>
>
>
Not so long ago, someone posted a script that produced a web page with the
results of linting all the packages

That's an example

One could produce an xml file to be imported in Gephi, just to make another
example.

Or a SQL text file to be imported in some relational db, or a different
format to be imported in some not relational db...

[-- Attachment #2: Type: text/html, Size: 1978 bytes --]

Re: How to get the package metadata as structured data?

[Ricardo Wurmus]

Catonano <catonano@gmail.com> writes:

> 2016-08-10 13:46 GMT+02:00 Ricardo Wurmus <rekado@elephly.net>:
>
>>
>> Philippe Ombredanne <pombredanne@nexb.com> writes:
>>
>> > David Craven <david@craven.ch> wrote:
>> >> I aborted, since I realised that guix package -s doesn't include the
>> >> source url and hash, which would be important for a testsuite...
>> >
>> > IMHO, if the rec data is the only way to get to the packages data, the
>> > source url would be rather essential to get in.
>>
>> The recutils output is not the *only* way to access the package data.
>> All packages in Guix are just Scheme variables.  Package data are
>> available as S-expressions and can be read by Scheme programs or parsed
>> by external applications.  The recutils output is just an additional
>> format used when interacting with Guix on the command line.
>>
>> (Personally, I’m not enthusiastic about adding a serialised form of the
>> source field to the recutils output.)
>>
>> ~~ Ricardo
>>
>>
>>
> Not so long ago, someone posted a script that produced a web page with the
> results of linting all the packages
>
> That's an example
>
> One could produce an xml file to be imported in Gephi, just to make another
> example.
>
> Or a SQL text file to be imported in some relational db, or a different
> format to be imported in some not relational db...

I understand that it may be useful.  I just think that the
representation as a Guile Scheme expression/value is *already* much more
useful.  That’s what made “guix web” possible, a web interface like this
one:

    http://guix.mdc-berlin.de

Producing other formats (such as XML files) is also quite simple.  The
package expression itself is *executable*.  By overriding the meaning of
the executable parts one could easily evaluate the expression such that
it generates some other format.

~~ Ricardo

Re: How to get the package metadata as structured data?

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

2016-08-10 14:52 GMT+02:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Catonano <catonano@gmail.com> writes:
>
> > 2016-08-10 13:46 GMT+02:00 Ricardo Wurmus <rekado@elephly.net>:
> >
> >>
> >> Philippe Ombredanne <pombredanne@nexb.com> writes:
> >>
> >> > David Craven <david@craven.ch> wrote:
> >> >> I aborted, since I realised that guix package -s doesn't include the
> >> >> source url and hash, which would be important for a testsuite...
> >> >
> >> > IMHO, if the rec data is the only way to get to the packages data, the
> >> > source url would be rather essential to get in.
> >>
> >> The recutils output is not the *only* way to access the package data.
> >> All packages in Guix are just Scheme variables.  Package data are
> >> available as S-expressions and can be read by Scheme programs or parsed
> >> by external applications.  The recutils output is just an additional
> >> format used when interacting with Guix on the command line.
> >>
> >> (Personally, I’m not enthusiastic about adding a serialised form of the
> >> source field to the recutils output.)
> >>
> >> ~~ Ricardo
> >>
> >>
> >>
> > Not so long ago, someone posted a script that produced a web page with
> the
> > results of linting all the packages
> >
> > That's an example
> >
> > One could produce an xml file to be imported in Gephi, just to make
> another
> > example.
> >
> > Or a SQL text file to be imported in some relational db, or a different
> > format to be imported in some not relational db...
>
> I understand that it may be useful.  I just think that the
> representation as a Guile Scheme expression/value is *already* much more
> useful.  That’s what made “guix web” possible, a web interface like this
> one:
>
>     http://guix.mdc-berlin.de



Yes, Ricardo, I was not disagreeing with you. I was just trying to offer
Philippe some more perspective.

In fact I believe that the approach you envisioned (a cycle through the
packages values, leaving only the decompress phase and adding a phase that
calls Philippe's tool) has its merits.

Only, if I was Philippe, I would regret storing the output in the store

I'd prefer to be able to store the output in the regular file system

In this regard, I think that the threads about pipelines (for reproducible
science), some time ago, were quite interesting

[-- Attachment #2: Type: text/html, Size: 3353 bytes --]