Debugging Guix packages?

Debugging Guix packages?

[Jookia]

Hi there,

I've been hitting some bugs in Guix packages which I'm up for fixing, though
currently I'm finding it difficult to follow along the build process then
improvise with my own inspection. There's only two ways I know of to do this:

- Have the package fail and inspect it afterwards using guix environment.

This could work, but it isn't helpful if the package doesn't fail and I want to
try adding my own patches without doing a full rebuild that could fail.

- Read the package and roughly convert it to Bash commands.

I'm still not good at this. For instance, if I want to build coreutils with
some tweaks in the build process without modifying the package I'd have to
figure out how to patch shebangs from the command line and figure out how to do
this if I'm cross compiling.

It'd be a nice feature to have where I could step through the build process in
an environment close to the actual build and run build commands myself like
'patch' or 'make' or 'configure', but builders can also include Guile code.

Debugging the builder using Guile seems to be somewhat useful but I haven't
figured out how to do that as I'm not versed in Guile and this may not be the
right level of abstraction.

I apologize if my thoughts aren't clear, currently it's annoying to have to
decipher packages to hope I'm following along with the build. The alternative is
to iterate through full builds hoping I get the right output or manually failing
after a certain phase (I'm not sure how to do this.)

Cheers,
Jookia.

Re: Debugging Guix packages?

[Ricardo Wurmus]

Jookia <166291@gmail.com> writes:

> It'd be a nice feature to have where I could step through the build process in
> an environment close to the actual build and run build commands myself like
> 'patch' or 'make' or 'configure', but builders can also include Guile code.

We already have “guix environment --{pure,container} pkg” which spawns a
shell where all declared inputs are available and all environment
variables are set, but it does lack a method to run build phases.  It
would be very nice if we had a tool to selectively run build phases as
defined in the arguments field.

> Debugging the builder using Guile seems to be somewhat useful but I haven't
> figured out how to do that as I'm not versed in Guile and this may not be the
> right level of abstraction.

I have been packaging many applications for Guix and have only felt the
need for something more advanced than “guix environment” when building
really large stuff like the icedtea or GCC packages.  These packages
also have complicated build phases that patch the sources and set
additional environment variables, and it would have been helpful to have
a tool to run selected build phases in the current directory.

> I apologize if my thoughts aren't clear, currently it's annoying to have to
> decipher packages to hope I'm following along with the build. The alternative is
> to iterate through full builds hoping I get the right output or manually failing
> after a certain phase (I'm not sure how to do this.)

You can make packages fail by adding a failing build phase.  To fail
after “configure”, for example, you could do something like this:

    (arguments
      `(modify-phases %standard-phases
          (add-after 'configure 'bleh
            (const #f))))

~~ Ricardo

Re: Debugging Guix packages?

[Christopher Allan Webber]

Ricardo Wurmus writes:

> Jookia <166291@gmail.com> writes:
>
>> It'd be a nice feature to have where I could step through the build process in
>> an environment close to the actual build and run build commands myself like
>> 'patch' or 'make' or 'configure', but builders can also include Guile code.
>
> We already have “guix environment --{pure,container} pkg” which spawns a
> shell where all declared inputs are available and all environment
> variables are set, but it does lack a method to run build phases.  It
> would be very nice if we had a tool to selectively run build phases as
> defined in the arguments field.

I hadn't thought othat, and yes it would be nice to run the build phases
as such.

>> Debugging the builder using Guile seems to be somewhat useful but I haven't
>> figured out how to do that as I'm not versed in Guile and this may not be the
>> right level of abstraction.
>
> I have been packaging many applications for Guix and have only felt the
> need for something more advanced than “guix environment” when building
> really large stuff like the icedtea or GCC packages.  These packages
> also have complicated build phases that patch the sources and set
> additional environment variables, and it would have been helpful to have
> a tool to run selected build phases in the current directory.

I've packaged many fewer packages than you have Ricardo, but maybe even
because of that (I'm a lot more inexperienced!) it's sometimes hard for
me to figure out what's going on.

But anyway your above suggestion to start a container or shell and run
certain build phases up to a point would help me a lot.

Re: Debugging Guix packages?

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Jookia <166291@gmail.com> writes:
>
>> It'd be a nice feature to have where I could step through the build process in
>> an environment close to the actual build and run build commands myself like
>> 'patch' or 'make' or 'configure', but builders can also include Guile code.
>
> We already have “guix environment --{pure,container} pkg” which spawns a
> shell where all declared inputs are available and all environment
> variables are set, but it does lack a method to run build phases.  It
> would be very nice if we had a tool to selectively run build phases as
> defined in the arguments field.

Yes, I agree that this would be nice.

>> Debugging the builder using Guile seems to be somewhat useful but I haven't
>> figured out how to do that as I'm not versed in Guile and this may not be the
>> right level of abstraction.
>
> I have been packaging many applications for Guix and have only felt the
> need for something more advanced than “guix environment” when building
> really large stuff like the icedtea or GCC packages.  These packages
> also have complicated build phases that patch the sources and set
> additional environment variables, and it would have been helpful to have
> a tool to run selected build phases in the current directory.

OTOH, for things like GCC, once you start fiddling with the build tree,
you quickly lose track of what state you’re in.

My workflow has been:

  guix build foo -K
  # build fails
  cd /tmp/guix-build*
  source environment-variables
  # Fiddle with the build tree to get additional info about the problem
  # and a possible fix.
  # Write a phase that hopefully fixes the issue.
  # Try again.

Since the ‘environment-variables’ file always contains the value of
environment variables at the time where the build failed (rather than
their initial value), it usually works quite well.

My 2¢,
Ludo’.

Re: Debugging Guix packages?

[Jookia]

On Wed, Jan 20, 2016 at 11:26:14PM +0100, Ludovic Courtès wrote:
> Yes, I agree that this would be nice.

I wonder how this could be implemented, any ideas?

> OTOH, for things like GCC, once you start fiddling with the build tree,
> you quickly lose track of what state you’re in.
>
> My workflow has been:
>
>   guix build foo -K
>   # build fails
>   cd /tmp/guix-build*
>   source environment-variables
>   # Fiddle with the build tree to get additional info about the problem
>   # and a possible fix.
>   # Write a phase that hopefully fixes the issue.
>   # Try again.
>
> Since the ‘environment-variables’ file always contains the value of
> environment variables at the time where the build failed (rather than
> their initial value), it usually works quite well.

This could be good enough for now but again this doesn't work with this like
builds that don't fail. My concrete example is wanting to patch software but not
knowing how to get to the pre-patch state so I can then work from that in
testing which patches apply and which don't, or make my own.

> My 2¢,
> Ludo’.

Cheers,
Jookia.

Re: Debugging Guix packages?

[Leo Famulari]

On Thu, Jan 21, 2016 at 11:21:34AM +1100, Jookia wrote:
> On Wed, Jan 20, 2016 at 11:26:14PM +0100, Ludovic Courtès wrote:
> > Yes, I agree that this would be nice.
> 
> I wonder how this could be implemented, any ideas?
> 
> > OTOH, for things like GCC, once you start fiddling with the build tree,
> > you quickly lose track of what state you’re in.
> >
> > My workflow has been:
> >
> >   guix build foo -K
> >   # build fails
> >   cd /tmp/guix-build*
> >   source environment-variables
> >   # Fiddle with the build tree to get additional info about the problem
> >   # and a possible fix.
> >   # Write a phase that hopefully fixes the issue.
> >   # Try again.
> >
> > Since the ‘environment-variables’ file always contains the value of
> > environment variables at the time where the build failed (rather than
> > their initial value), it usually works quite well.
> 
> This could be good enough for now but again this doesn't work with this like
> builds that don't fail. My concrete example is wanting to patch software but not
> knowing how to get to the pre-patch state so I can then work from that in
> testing which patches apply and which don't, or make my own.

My workflow for creating patches:
$ tar xf $(guix build --source foo)
# or, if the package is in your local source tree:
$ tar xf $(./pre-inst-env guix build --source foo)
$ cd foo && git init && git add -A && git commit -m "Initial commit"
# make edits and commit them. Then:
$ git format-patch

It's not necessary to use git but it helps me keep track of my changes.

> 
> > My 2¢,
> > Ludo’.
> 
> Cheers,
> Jookia.
> 

Re: Debugging Guix packages?

[Ben Woodcroft]

Hi there,

On 21/01/16 10:21, Jookia wrote:
> On Wed, Jan 20, 2016 at 11:26:14PM +0100, Ludovic Courtès wrote:
>> OTOH, for things like GCC, once you start fiddling with the build tree,
>> you quickly lose track of what state you’re in.
>>
>> My workflow has been:
>>
>>    guix build foo -K
>>    # build fails
>>    cd /tmp/guix-build*
>>    source environment-variables
>>    # Fiddle with the build tree to get additional info about the problem
>>    # and a possible fix.
>>    # Write a phase that hopefully fixes the issue.
>>    # Try again.
>>
>> Since the ‘environment-variables’ file always contains the value of
>> environment variables at the time where the build failed (rather than
>> their initial value), it usually works quite well.
> This could be good enough for now but again this doesn't work with this like
> builds that don't fail. My concrete example is wanting to patch software but not
> knowing how to get to the pre-patch state so I can then work from that in
> testing which patches apply and which don't, or make my own.
Can I ask, why Ricardo's suggestion does not address this concern? This 
approach makes builds fail, so then you can inspect. I often use this.

My 2c here, would it be possible to echo any system* commands used in 
the build to the log? This would help make things concrete and put flags 
in the build log sand.

Thanks,
ben

Re: Debugging Guix packages?

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> On Wed, Jan 20, 2016 at 11:26:14PM +0100, Ludovic Courtès wrote:
>> Yes, I agree that this would be nice.
>
> I wonder how this could be implemented, any ideas?

Vague ideas.  :-)

Roughly, we could generate a Scheme file containing the list of phases
(the generated build script contains them, but it immediately passes
them to ‘gnu-build’ or similar), which could be loaded into Guile.  From
there we could explicitly manipulate the list of phases and run them.

There are complications though, such as the fact that build systems can
lack the notion of phases altogether (this is the case of
‘trivial-build-system’, but it may be the only one.)  I wonder if we
should move phases from the build side to the host side at some point…

Ludo’.

Re: Debugging Guix packages?

[Ludovic Courtès]

Jookia <166291@gmail.com> skribis:

> This could be good enough for now but again this doesn't work with this like
> builds that don't fail. My concrete example is wanting to patch software but not
> knowing how to get to the pre-patch state so I can then work from that in
> testing which patches apply and which don't, or make my own.

To get the unpatched source, you can always comment out the ‘patches’
field, and then run, say:

  tar xf $(guix build -S my-package)

Or do I misunderstand?

Ludo’.

Re: Debugging Guix packages?

[Jookia]

On Thu, Jan 21, 2016 at 09:58:38PM +0100, Ludovic Courtès wrote:
> Jookia <166291@gmail.com> skribis:
> 
> > This could be good enough for now but again this doesn't work with this like
> > builds that don't fail. My concrete example is wanting to patch software but not
> > knowing how to get to the pre-patch state so I can then work from that in
> > testing which patches apply and which don't, or make my own.
> 
> To get the unpatched source, you can always comment out the ‘patches’
> field, and then run, say:
> 
>   tar xf $(guix build -S my-package)
> 
> Or do I misunderstand?


One difficulty with the approach of modifying the package is that in a lot of
cases Guix is read-only in a store. So I guess the right way to do this
patching is to modify packages and have Guix use that, though this solution
applies to everything. Maybe a debugger isn't needed? I find it hard to justify
outside 'it makes it easier to try new things quickly'.

> Ludo’.

Cheers,
Jookia.

Re: Debugging Guix packages?

[Christopher Allan Webber]

To ignore specifics, there are a lot of current methods people are using
to debug, as well as a few proposed methods.

Regardless, maybe it would be nice to copy useful techniques into the
manual.  It seems like good information/history to capture.

 - Chris

Re: Debugging Guix packages?

[Pjotr Prins]

On Thu, Jan 21, 2016 at 09:55:32PM +0100, Ludovic Courtès wrote:
> > I wonder how this could be implemented, any ideas?
> 
> Roughly, we could generate a Scheme file containing the list of phases
> (the generated build script contains them, but it immediately passes
> them to ‘gnu-build’ or similar), which could be loaded into Guile.  From
> there we could explicitly manipulate the list of phases and run them.
> 
> There are complications though, such as the fact that build systems can
> lack the notion of phases altogether (this is the case of
> ‘trivial-build-system’, but it may be the only one.)  I wonder if we
> should move phases from the build side to the host side at some point…

How about extending the -K switch with a build phase option where it
should stop? 

I think the container thing is nice, but not really suitable for
fixing stuff (as Chris notes, it ends up read-only in the store). I am
pretty happy with -K.

Pj.
-- 

Re: Debugging Guix packages?

[Jookia]

On Fri, Jan 22, 2016 at 05:13:02AM +0100, Pjotr Prins wrote:
> On Thu, Jan 21, 2016 at 09:55:32PM +0100, Ludovic Courtès wrote:
> > > I wonder how this could be implemented, any ideas?
> > 
> > Roughly, we could generate a Scheme file containing the list of phases
> > (the generated build script contains them, but it immediately passes
> > them to ‘gnu-build’ or similar), which could be loaded into Guile.  From
> > there we could explicitly manipulate the list of phases and run them.
> > 
> > There are complications though, such as the fact that build systems can
> > lack the notion of phases altogether (this is the case of
> > ‘trivial-build-system’, but it may be the only one.)  I wonder if we
> > should move phases from the build side to the host side at some point…
> 
> How about extending the -K switch with a build phase option where it
> should stop? 
> 
> I think the container thing is nice, but not really suitable for
> fixing stuff (as Chris notes, it ends up read-only in the store). I am
> pretty happy with -K.

This might be good enough, but if you change options in the configure phase
you'll probably want to see what's happening in the check phase.

I like the idea of refactoring build scripts in to multiple functions that can
be run in a Guile prompt and have Guix or someone in a Guile prompt use those.

> Pj.
> -- 

Cheers,
Jookia.

Re: Debugging Guix packages?

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Thu, Jan 21, 2016 at 09:55:32PM +0100, Ludovic Courtès wrote:
>> > I wonder how this could be implemented, any ideas?
>> 
>> Roughly, we could generate a Scheme file containing the list of phases
>> (the generated build script contains them, but it immediately passes
>> them to ‘gnu-build’ or similar), which could be loaded into Guile.  From
>> there we could explicitly manipulate the list of phases and run them.
>> 
>> There are complications though, such as the fact that build systems can
>> lack the notion of phases altogether (this is the case of
>> ‘trivial-build-system’, but it may be the only one.)  I wonder if we
>> should move phases from the build side to the host side at some point…
>
> How about extending the -K switch with a build phase option where it
> should stop? 

I’m really skeptical about the usefulness of such a feature.  As Ricardo
suggested earlier, stopping at a specific phase is really easy to do by
hand, on the rare occasions where you actually want that.  (Not to
mention that one can also simply hit C-c while the thing is building.)

Ludo’.