make packagers life easier

make packagers life easier

[Tomáš Čech]

Hello Guix,

I'd like to share some thoughts about packaging to find I'm doing it
wrong or show that there is room for improvements. I don't think that
the proposed improvements are hard to implement (from algorithmic POV)
and I'm aware that all the things can be solved on my side, but that
is not the point.

Stop after/before phase
=======================

What it can help with?

- download sources of package without constructing URL
- prepare sources for creating or applying patch
- when preparing configure parameters you can stop right after
  configuration phase

How to implement?
- inject always failing phase into phases list

Another level of this feature could be even continuing of the build.


Ommit developed package from generations
========================================

How it can help?

When I work on packages I have to take care of handling generations
even when I don't care. It complicates removing old package builds
using `guix gc -d' and it is unwanted burden.

How to implement?
I'm not exactly sure, but not creating new generation with the package
could be sufficient.



Improve output of the build
===========================

What I miss?

Mostly one particular and obvious thing - after successful build print
the path of newly created store in well visible, easy-to-mouse and
easy-to-parse way.

example:

New store created: /gnu/store/s0meb0r1ngandl000ngh4shlead1ngt0-package-1.0/

Right now I need to dig that from any build message above.



And in future it would be nice to add some colors to output. It helps
with orientation speed a lot.

Thanks for reading,

S_W

Re: make packagers life easier

[Ludovic Courtès]

Hi!

Tomáš Čech <sleep_walker@gnu.org> skribis:

> Stop after/before phase
> =======================
>
> What it can help with?
>
> - download sources of package without constructing URL
> - prepare sources for creating or applying patch

I think ‘guix build -S foo’ does exactly that, doesn’t it?

> - when preparing configure parameters you can stop right after
>  configuration phase
>
> How to implement?
> - inject always failing phase into phases list

I’d like to read what others think because I’m probably biased, but my
first reaction is that (1) I’ve rarely if ever felt the need for this,
(2) adding a failing phase is easy to do by hand, and (3) it would be
hard to implement this generically currently (phases are not exposed.)

> Ommit developed package from generations
> ========================================

Hmm, what do you mean?  For a package to end up in your profile, you
have to explicitly install it.  So I’m really not sure what you mean
here.

> When I work on packages I have to take care of handling generations
> even when I don't care. It complicates removing old package builds
> using `guix gc -d' and it is unwanted burden.

I rarely do ‘guix gc -d’; I rather let the GC collect garbage by itself,
once it a while.  :-)

> Improve output of the build
> ===========================
>
> What I miss?
>
> Mostly one particular and obvious thing - after successful build print
> the path of newly created store in well visible, easy-to-mouse and
> easy-to-parse way.

Successful ‘guix build’ invocations always print the resulting file
name(s) as the last line(s) to stdout, such that one can always do
things like:

  ls $(guix build foo)

Is it what you want, or did you mean something else?

Thanks for the suggestions!

Ludo’.

Re: make packagers life easier

[Andreas Enge]

On Mon, Mar 16, 2015 at 06:01:15PM +0100, Ludovic Courtès wrote:
> > How to implement?
> > - inject always failing phase into phases list
> I’d like to read what others think because I’m probably biased, but my
> first reaction is that (1) I’ve rarely if ever felt the need for this,
> (2) adding a failing phase is easy to do by hand, and (3) it would be
> hard to implement this generically currently (phases are not exposed.)

Agreed. One thing that works when one adds a customised phase is to let
it finish with #f. Of course, if everything is standard, one needs to
add such a phase, which is a bit of typing. Maybe with Taylan's new syntax,
it has become easier.

Andreas

Re: make packagers life easier

[Sree Harsha Totakura]

On 03/15/2015 06:00 PM, Tomáš Čech wrote:
> 
> How to implement?
> - inject always failing phase into phases list
> 
> Another level of this feature could be even continuing of the build.

I have a similar feature request.  Long time ago when I was packaging
gnunet and curl, I had to spend a lot of time trying to understand why
`make check` phase fails in the build environment.  More often than not,
I had to extensively study the test cases and the testing mechanism
being used to understand and then resolve the failures, much to my
chagrin, by trail-and-error as I did not have access to the build
environment.

Sure, `guix build -K` helped to some extent but there were some
hard-to-debug cases which only failed in the build environment due to
separate network and filesystem namespaces.

I discussed this with Christian and his suggestion was to implement a
debug shell which the guix builder will bail out to when a phase fails.
 The packager can then examine the environment right at the moment where
the failure happens.

I know that since the build happens in a separate namespace this can be
hard to implement, but one idea is to configure the build environments
network namespace to accept connections from localhost and spawn sshd
when the build fails.

WDYT?

Regards,
Sree

Re: make packagers life easier

[Ludovic Courtès]

Sree Harsha Totakura <sreeharsha@totakura.in> skribis:

> I have a similar feature request.  Long time ago when I was packaging
> gnunet and curl, I had to spend a lot of time trying to understand why
> `make check` phase fails in the build environment.  More often than not,
> I had to extensively study the test cases and the testing mechanism
> being used to understand and then resolve the failures, much to my
> chagrin, by trail-and-error as I did not have access to the build
> environment.
>
> Sure, `guix build -K` helped to some extent but there were some
> hard-to-debug cases which only failed in the build environment due to
> separate network and filesystem namespaces.

On GuixSD, there are few differences between the build environment and
the “real” environment, which makes ‘guix build -K’ probably more
useful.  Notable differences are that the real environment has network
access and /bin/sh.

The /bin/sh thing is usually easily fixed.  Problems related to separate
network name spaces can be more difficult as you noticed, although it’s
also always possible to run ‘strace’ in the build process or some Guile
snippet to get a better understanding of what’s going on.

However, when using Guix on another distro, then of course, in addition
to /bin/sh and networking, you get an FHS tree, which makes a
significant difference compared to the build environment.

> I discussed this with Christian and his suggestion was to implement a
> debug shell which the guix builder will bail out to when a phase fails.
>  The packager can then examine the environment right at the moment where
> the failure happens.
>
> I know that since the build happens in a separate namespace this can be
> hard to implement, but one idea is to configure the build environments
> network namespace to accept connections from localhost and spawn sshd
> when the build fails.
>
> WDYT?

I think it’s a nice idea, but it’s easier said than done.  :-)

An option would be to make do without the daemon’s help.  But then, do
we add lsh/OpenSSH as a dependency of every build?  And more
importantly, can we actually open connections available from the
outside?

The other option is to handle it within the daemon, which would do all
the container-with-ssh-daemon setup by itself when needed.  That seems a
bit heavyweight to do there, and perhaps more investment than desirable
in this C++ code base.

If container support was librarified, akin to what David suggested for
GSoC¹, it would be easy for users to start a container that is really
comparable to that created by guix-daemon.  Then that would make this
sort of debugging much easier, among other benefits.

My 2¢,
Ludo’.

¹ http://www.gnu.org/software/soc-projects/ideas-2015.html#guix