New ‘guix graph’ command

New ‘guix graph’ command

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 938 bytes --]

Hi!

Here’s a long overdue ‘guix graph’ command (documentation below.)
Comments welcome!

Ludo’.


5.9 Invoking ‘guix graph’
=========================

Packages and their dependencies form a “graph”, specifically a directed
acyclic graph (DAG). It can quickly become difficult to have a mental
model of the package DAG, so the ‘guix graph’ command is here to provide
a visual representation of the DAG. ‘guix graph’ emits a DAG
representation in the input format of Graphviz
(http://www.graphviz.org/), so its output can be passed directly to
Graphviz’s ‘dot’ command, for instance.  The general syntax is:

     guix graph OPTIONS PACKAGE…

   For example, the following command generates a PDF file representing
the package DAG for the GNU Core Utilities, showing its build-time
dependencies:

     guix graph coreutils | dot -Tpdf > dag.pdf

   The output looks like this:


[-- Attachment #2: DAG --]
[-- Type: image/png, Size: 13378 bytes --]

[-- Attachment #3: Type: text/plain, Size: 828 bytes --]


   Nice little graph, no?

   But there’s more than one graph!  The one above is concise: it’s the
graph of package objects, omitting implicit inputs such as GCC, libc,
grep, etc.  It’s often useful to have such a concise graph, but
sometimes you want to see more details.  ‘guix graph’ supports several
types of graphs, allowing you to choose the level of details:

‘package’
     This is the default type, the one we used above.  It shows the DAG
     of package objects, excluding implicit dependencies.  It is
     concise, but filters out many details.

‘bag-emerged’
     This is the package DAG, _including_ implicit inputs.

     For instance, the following command:

          guix graph --type=bag-emerged coreutils | dot -Tpdf > dag.pdf

     ...  yields this bigger graph:


[-- Attachment #4: big DAG --]
[-- Type: image/png, Size: 621727 bytes --]

[-- Attachment #5: Type: text/plain, Size: 1414 bytes --]


     At the bottom of the graph, we see all the implicit inputs of
     GNU-BUILD-SYSTEM (*note ‘gnu-build-system’: Build Systems.).

     Now, note that the dependencies of those implicit inputs—that is,
     the “bootstrap dependencies” (*note Bootstrapping::)—are not shown
     here, for conciseness.

‘bag’
     Similar to ‘bag-emerged’, but this time including all the bootstrap
     dependencies.

‘derivations’
     This is the most detailed representation: It shows the DAG of
     derivations (*note Derivations::) and plain store items.  Compared
     to the above representation, many additional nodes are visible,
     including builds scripts, patches, Guile modules, etc.

   All the above types correspond to _build-time dependencies_.  The
following graph type represents the _run-time dependencies_:

‘references’
     This is the graph of “references” of a package output, as returned
     by ‘guix gc --references’ (*note Invoking guix gc::).

     If the given package output is not available in the store, ‘guix
     graph’ attempts to obtain dependency information from substitutes.

   The available options are the following:

‘--type=TYPE’
‘-t TYPE’
     Produce a graph output of TYPE, where TYPE must be one of the
     values listed above.

‘--list-types’
     List the supported graph types.

Re: New ‘guix graph’ command

[Pjotr Prins]

Very useful!

On Thu, Aug 27, 2015 at 12:53:03AM +0200, Ludovic Courtès wrote:
> Hi!
> 
> Here’s a long overdue ‘guix graph’ command (documentation below.)
> Comments welcome!
> 
> Ludo’.

Re: New ‘guix graph’ command

[Mathieu Lirzin]

Nice job, It is really convenient to have such utility.

Does it sound feasible to produce different edge colors depending of the type
of inputs?

ludo@gnu.org (Ludovic Courtès) writes:

> ‘bag-emerged’
>      This is the package DAG, _including_ implicit inputs.
>
>      For instance, the following command:
>
>           guix graph --type=bag-emerged coreutils | dot -Tpdf > dag.pdf
>
>      ...  yields this bigger graph:
>
>
>
>      At the bottom of the graph, we see all the implicit inputs of
>      GNU-BUILD-SYSTEM (*note ‘gnu-build-system’: Build Systems.).
>
>      Now, note that the dependencies of those implicit inputs—that is,
>      the “bootstrap dependencies” (*note Bootstrapping::)—are not shown
>      here, for conciseness.
>
> ‘bag’
>      Similar to ‘bag-emerged’, but this time including all the bootstrap
>      dependencies.

Not really important, but IMO it would be clearer to define bag-emerged
in terms of what bag is doing with something like this.

--8<---------------cut here---------------start------------->8---
‘bag’                                                           
     This is the package DAG, _including_ implicit inputs.              
                                                                        
‘bag-emerged’                                                                   
     Similar to ‘bag’, but this time without all the bootstrap
     dependencies.

     For instance, the following command:                               
                                                                        
          guix graph --type=bag-emerged coreutils | dot -Tpdf > dag.pdf 
                                                                        
     [...]
--8<---------------cut here---------------end--------------->8---

What do you think?

--
Mathieu Lirzin

Re: New ‘guix graph’ command

[Ludovic Courtès]

Mathieu Lirzin <mthl@openmailbox.org> skribis:

> Nice job, It is really convenient to have such utility.
>
> Does it sound feasible to produce different edge colors depending of the type
> of inputs?

Of course!  But this is left as an exercise to the reader.  :-)

Similarly, I thought we could use different color boxes based on the
size of a store item, for the ‘references’ DAGs.

It “just needs” adding an attribute argument to the ‘emit-node’ and
‘emit-edges’ hooks of <graph-backend>.  I don’t plan to work on it in
the near future but would definitely welcome patches.

> Not really important, but IMO it would be clearer to define bag-emerged
> in terms of what bag is doing with something like this.
>
> ‘bag’                                                           
>      This is the package DAG, _including_ implicit inputs.              
>                                                                         
> ‘bag-emerged’                                                                   
>      Similar to ‘bag’, but this time without all the bootstrap
>      dependencies.
>
>      For instance, the following command:                               
>                                                                         
>           guix graph --type=bag-emerged coreutils | dot -Tpdf > dag.pdf 
>                                                                         
>      [...]
>
> What do you think?

Perhaps this isn’t very clear but I wanted to list the graph types in
order of increasing complexity, which is why ‘bag’ comes after
‘bag-emerged’.  Dunno if something needs to be changed; thoughts?

Thanks,
Ludo’.

Re: New ‘guix graph’ command

[Alex Kost]

Ludovic Courtès (2015-08-27 01:53 +0300) wrote:

> Here’s a long overdue ‘guix graph’ command (documentation below.)
> Comments welcome!

This is really cool!  Thank you!

-- 
Alex

Re: New ‘guix graph’ command

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 3605 bytes --]

Very nice! This will be much easier than my ersatz solution of making
filesystem trees by hand and then running `tree` on them.

Another tool for the visual thinkers...


On Wed, Aug 26, 2015, at 18:53, Ludovic Courtès wrote:
> Hi!
>
> Here’s a long overdue ‘guix graph’ command (documentation below.)
> Comments welcome!
>
> Ludo’.
>
>
> 5.9 Invoking ‘guix graph’
> =========================
>
> Packages and their dependencies form a “graph”, specifically a
> directed acyclic graph (DAG). It can quickly become difficult to have
> a mental model of the package DAG, so the ‘guix graph’ command is here
> to provide a visual representation of the DAG. ‘guix graph’ emits a
> DAG representation in the input format of Graphviz
> (http://www.graphviz.org/), so its output can be passed directly to
> Graphviz’s ‘dot’ command, for instance.  The general syntax is:
>
> guix graph OPTIONS PACKAGE…
>
> For example, the following command generates a PDF file representing
> the package DAG for the GNU Core Utilities, showing its build-time
> dependencies:
>
> guix graph coreutils | dot -Tpdf > dag.pdf
>
> The output looks like this:
>
>
>
> Nice little graph, no?
>
> But there’s more than one graph!  The one above is concise: it’s the
> graph of package objects, omitting implicit inputs such as GCC, libc,
> grep, etc.  It’s often useful to have such a concise graph, but
> sometimes you want to see more details.  ‘guix graph’ supports several
> types of graphs, allowing you to choose the level of details:
>
> ‘package’    This is the default type, the one we used above.  It
> shows the DAG    of package objects, excluding implicit dependencies.
> It is    concise, but filters out many details.
>
> ‘bag-emerged’    This is the package DAG, __including__
> implicit inputs.
>
> For instance, the following command:
>
> guix graph --type=bag-emerged coreutils | dot -Tpdf > dag.pdf
>
> ...  yields this bigger graph:
>
>
>
> At the bottom of the graph, we see all the implicit inputs of    GNU-BUILD-
> SYSTEM (*note ‘gnu-build-system’: Build Systems.).
>
> Now, note that the dependencies of those implicit inputs—that is,
> the “bootstrap dependencies” (*note Bootstrapping::)—are not shown
> here, for conciseness.
>
> ‘bag’    Similar to ‘bag-emerged’, but this time including all the
> bootstrap    dependencies.
>
> ‘derivations’    This is the most detailed representation: It shows
> the DAG of    derivations (*note Derivations::) and plain store items.
> Compared    to the above representation, many additional nodes are
> visible,    including builds scripts, patches, Guile modules, etc.
>
> All the above types correspond to _build-time dependencies_.  The
> following graph type represents the _run-time dependencies_:
>
> ‘references’    This is the graph of “references” of a package output,
> as returned    by ‘guix gc --references’ (*note Invoking guix gc::).
>
> If the given package output is not available in the store, ‘guix
> graph’ attempts to obtain dependency information from substitutes.
>
> The available options are the following:
>
> ‘--type=TYPE’ ‘-t TYPE’    Produce a graph output of TYPE, where TYPE
> must be one of the    values listed above.
>
> ‘--list-types’    List the supported graph types.
>
> Email had 2 attachments:


>  * coreutils-graph.png  18k (image/png)
>  * coreutils-bag-graph.png  831k (image/png)

[-- Attachment #2.1: Type: text/html, Size: 5941 bytes --]

[-- Attachment #2.2: coreutils-graph.png --]
[-- Type: image/png, Size: 13378 bytes --]

[-- Attachment #2.3: coreutils-bag-graph.png --]
[-- Type: image/png, Size: 621727 bytes --]

Re: New ‘guix graph’ command

[Thompson, David]

On Wed, Aug 26, 2015 at 6:53 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi!
>
> Here’s a long overdue ‘guix graph’ command (documentation below.)
> Comments welcome!

This is awesome!  I think it will prove to be a really helpful tool
for understanding the relationships between packages and discovering
issues like packages that have references to build-time dependencies.

Just one question: I see that when using the 'references' type, each
node in the graph has an arrow pointing to itself.  Is that
intentional?

Thanks for the awesome new tool!

- Dave

Re: New ‘guix graph’ command

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> Just one question: I see that when using the 'references' type, each
> node in the graph has an arrow pointing to itself.  Is that
> intentional?

It’s not really “each node” but rather “most nodes.”  :-)

The daemon keeps track of all the references of each item, including
self-references.  And it turns out that most packages record their
$prefix in their binaries, for instance, hence the self-reference.  This
can be seen with ‘guix gc --references’ as well.

Ludo’.

Re: New ‘guix graph’ command

[Thompson, David]

On Thu, Aug 27, 2015 at 4:55 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> Just one question: I see that when using the 'references' type, each
>> node in the graph has an arrow pointing to itself.  Is that
>> intentional?
>
> It’s not really “each node” but rather “most nodes.”  :-)
>
> The daemon keeps track of all the references of each item, including
> self-references.  And it turns out that most packages record their
> $prefix in their binaries, for instance, hence the self-reference.  This
> can be seen with ‘guix gc --references’ as well.

That makes sense. Thanks for the explanation!

- Dave

Re: New ‘guix graph’ command

[Andreas Enge]

Hello!

On Thu, Aug 27, 2015 at 12:53:03AM +0200, Ludovic Courtès wrote:
> Here’s a long overdue ‘guix graph’ command (documentation below.)
> Comments welcome!

Very neat indeed! A tiny bit of nitpicking: Would it not be preferable if
the arrows were reversed, from the inputs to the dependent packages?

Andreas

Re: New ‘guix graph’ command

[Ludovic Courtès]

Hey!

Andreas Enge <andreas@enge.fr> skribis:

> Very neat indeed! A tiny bit of nitpicking: Would it not be preferable if
> the arrows were reversed, from the inputs to the dependent packages?

Dunno, maybe the graph people would choose the other direction but I
like it this way, looks more “intuitive” to me.  What do people think?

Ludo’.

Re: New ‘guix graph’ command

[Thompson, David]

On Thu, Sep 3, 2015 at 4:46 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hey!
>
> Andreas Enge <andreas@enge.fr> skribis:
>
>> Very neat indeed! A tiny bit of nitpicking: Would it not be preferable if
>> the arrows were reversed, from the inputs to the dependent packages?
>
> Dunno, maybe the graph people would choose the other direction but I
> like it this way, looks more “intuitive” to me.  What do people think?

I prefer it the way that it is because it reflects the actual data
structure.  It's intuitive because I often ask "what does this
software depend on?", not "what depends on this software?"

- Dave

RE: New ‘guix graph’ command

[Cook, Malcolm]

Hi,

I like the request.

I think the dependent packages should be pointed to.

In this way, each node in the graph and its outgoing arrows reflect the contents of a single package.

~Malco

 > -----Original Message-----
 > From: guix-devel-bounces+mec=stowers.org@gnu.org [mailto:guix-devel-
 > bounces+mec=stowers.org@gnu.org] On Behalf Of Ludovic Courtès
 > Sent: Thursday, September 03, 2015 3:47 PM
 > To: Andreas Enge <andreas@enge.fr>
 > Cc: Guix-devel <Guix-devel@gnu.org>
 > Subject: Re: New ‘guix graph’ command
 > 
 > Hey!
 > 
 > Andreas Enge <andreas@enge.fr> skribis:
 > 
 > > Very neat indeed! A tiny bit of nitpicking: Would it not be preferable
 > > if the arrows were reversed, from the inputs to the dependent packages?
 > 
 > Dunno, maybe the graph people would choose the other direction but I like it
 > this way, looks more “intuitive” to me.  What do people think?
 > 
 > Ludo’.

Re: New ‘guix graph’ command

[Pjotr Prins]

Maybe we should have both. I like the current version because it reads
logically from a *users* perspective. E.g.

  http://dthompson.us/images/guix/ruby-pg-graph.png

ruby-pg depends on postgresql 9.3.8, depends on zlib 1.2.7. Logical.

Pj.

On Thu, Sep 03, 2015 at 08:55:46PM +0000, Cook, Malcolm wrote:
> Hi,
> 
> I like the request.
> 
> I think the dependent packages should be pointed to.
> 
> In this way, each node in the graph and its outgoing arrows reflect the contents of a single package.
> 
> ~Malco
> 
>  > -----Original Message-----
>  > From: guix-devel-bounces+mec=stowers.org@gnu.org [mailto:guix-devel-
>  > bounces+mec=stowers.org@gnu.org] On Behalf Of Ludovic Courtès
>  > Sent: Thursday, September 03, 2015 3:47 PM
>  > To: Andreas Enge <andreas@enge.fr>
>  > Cc: Guix-devel <Guix-devel@gnu.org>
>  > Subject: Re: New ‘guix graph’ command
>  > 
>  > Hey!
>  > 
>  > Andreas Enge <andreas@enge.fr> skribis:
>  > 
>  > > Very neat indeed! A tiny bit of nitpicking: Would it not be preferable
>  > > if the arrows were reversed, from the inputs to the dependent packages?
>  > 
>  > Dunno, maybe the graph people would choose the other direction but I like it
>  > this way, looks more “intuitive” to me.  What do people think?
>  > 
>  > Ludo’.
> 

-- 

Re: New ‘guix graph’ command

[Andreas Enge]

On Fri, Sep 04, 2015 at 10:13:01AM +0200, Pjotr Prins wrote:
> Maybe we should have both.

Well, there is no real need for two versions; if most people like things
as they are, they should stay as they are.

Andreas

Re: New ‘guix graph’ command

[Cyril Roelandt]

On 08/27/2015 12:53 AM, Ludovic Courtès wrote:
> Here’s a long overdue ‘guix graph’ command (documentation below.)
> Comments welcome!

Would it be hard to do the opposite of what guix graph does ? I'd like
to be able to find the list of packages which depend on a given package.
This would be useful when updating a package, to make sure that the
update does not break anything.

WDYT?


Cyril.

Re: New ‘guix graph’ command

[Leo Famulari]

I think that is what `guix gc --referrers /gnu/store/...package` does.

http://www.gnu.org/software/guix/manual/html_node/Invoking-guix-gc.html

On Thu, Sep 10, 2015, at 20:02, Cyril Roelandt wrote:
> On 08/27/2015 12:53 AM, Ludovic Courtès wrote:
> > Here’s a long overdue ‘guix graph’ command (documentation below.)
> > Comments welcome!
> 
> Would it be hard to do the opposite of what guix graph does ? I'd like
> to be able to find the list of packages which depend on a given package.
> This would be useful when updating a package, to make sure that the
> update does not break anything.
> 
> WDYT?
> 
> 
> Cyril.
> 

Re: New ‘guix graph’ command

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 08/27/2015 12:53 AM, Ludovic Courtès wrote:
>> Here’s a long overdue ‘guix graph’ command (documentation below.)
>> Comments welcome!
>
> Would it be hard to do the opposite of what guix graph does ? I'd like
> to be able to find the list of packages which depend on a given package.
> This would be useful when updating a package, to make sure that the
> update does not break anything.

It shouldn’t be too hard, using something akin to the code of ‘guix
refresh --list-dependent’.

Ludo’.