Developing libraries for the GNU system with Guix

Developing libraries for the GNU system with Guix

[sbaugh]

Hi guix-devel,

When I am hacking on some library Z, I continuously want to test the
effects that my changes to Z have on packages A/B/C which depend on
Z. The same applies, in general, when hacking on any package Z which
other packages A/B/C depend on: While developing, I want to be able to
rapidly mutate Z and see how this affects A/B/C.

I am not sure how to best achieve this. Here are some solutions:

- When you change Z, rebuild A/B/C.
  This is much too slow at present.

- Use grafts to update A/B/C through binary patching.
  This is also too slow, and AFAIK it can't really be sped up.

- Use LD_LIBRARY_PATH to cause A/B/C to search for Z in a mutable place.
  This works for C libraries, but not generically; there are equivalent
  variables for some other languages but it's not a full solution.

- Before starting to hack on Z, build a new version of Z which includes
  a random hash and which A/B/C depend on; then bind-mount a mutable
  directory on top of that. (suggested by mark_weaver on IRC)
  This is the most palatable hack, but still a hack. The inclusion of a
  random hash prevents collision with other packages and the use of
  bind-mounting means we aren't actually mutating the store. This
  unfortunately also requires privileges: it's not usable by
  unprivileged users.

Here are some not currently available possibilities:

- Create some kind of GUIX_LIBRARY_PATH in which packages are looked up
  first before looking at the compiled-in hash.
  This would be an attempt to make a generic equivalent of
  LD_LIBRARY_PATH. This could theoretically be implemented with variant
  symlinks, if they were available: https://lwn.net/Articles/680705/

- Currently every dependency is located at a well known globally unique
  and globally meaningful path; add some kind of "variant package"
  construct which specifies a package which is "passed in" to the
  environment (maybe by bind-mounting it in a filesystem namespace;
  implementation specifics aren't important). To put this in a
  programming language sense, one of these packages being present would
  turn a Guix distribution from a value into a function.

- Massively speed up rebuilding A/B/C by performing incremental
  builds. Not sure how exactly this could work, it's just a thought.

What do you think, guix-devel?

By the way, I think this is a pretty important feature overall (though
it's described in fairly abstract terms in this email). I think this is
one of the last missing features to make the Guix approach a clear and
uncontroversial improvement over existing widespread practices.

Re: Developing libraries for the GNU system with Guix

[Ludovic Courtès]

Hello!

sbaugh@catern.com skribis:

> When I am hacking on some library Z, I continuously want to test the
> effects that my changes to Z have on packages A/B/C which depend on
> Z. The same applies, in general, when hacking on any package Z which
> other packages A/B/C depend on: While developing, I want to be able to
> rapidly mutate Z and see how this affects A/B/C.

A very common use case.  Others have asked about it.

> I am not sure how to best achieve this. Here are some solutions:
>
> - When you change Z, rebuild A/B/C.
>   This is much too slow at present.

Right, but note that it’s the only way to be confident that the change
in Z doesn’t break A/B/C.  That said, I do understand that sometimes,
you want an “I know what I’m doing” (i.e., the ABI of Z hasn’t changed)
option to bypass that and test the run-time behavior of A/B/C.

For cases where you do want to rebuild anyway, I was thinking of making
the ‘--with-source’ option “recursive”, such that:

  guix build --with-source=./guile-2.0.14rc1.tar.gz guile-json

would replace the source of Guile and build Guile-JSON against that.

> - Use grafts to update A/B/C through binary patching.
>   This is also too slow, and AFAIK it can't really be sped up.

A/B/C have to be really big for this to be too slow!  I think grafting
processes several MiB/s on my SSD laptop.

Again to make this more convenient, I thought we could have a
--with-graft option, which would work like --with-input except that it
would graft the new Z onto A/B/C instead of rebuilding them.

> - Use LD_LIBRARY_PATH to cause A/B/C to search for Z in a mutable place.
>   This works for C libraries, but not generically; there are equivalent
>   variables for some other languages but it's not a full solution.

Yeah, not great.

> - Before starting to hack on Z, build a new version of Z which includes
>   a random hash and which A/B/C depend on; then bind-mount a mutable
>   directory on top of that. (suggested by mark_weaver on IRC)
>   This is the most palatable hack, but still a hack. The inclusion of a
>   random hash prevents collision with other packages and the use of
>   bind-mounting means we aren't actually mutating the store. This
>   unfortunately also requires privileges: it's not usable by
>   unprivileged users.

It is usable by unprivileged users when user namespaces are available.
Under these conditions, you could use ‘call-with-container’ and
bind-mount anything anywhere.  Doesn’t sound too nice to me though.

> Here are some not currently available possibilities:
>
> - Create some kind of GUIX_LIBRARY_PATH in which packages are looked up
>   first before looking at the compiled-in hash.
>   This would be an attempt to make a generic equivalent of
>   LD_LIBRARY_PATH. This could theoretically be implemented with variant
>   symlinks, if they were available: https://lwn.net/Articles/680705/

At first sight this sounds very hacky, very much against the whole idea
of functional package management.

> - Currently every dependency is located at a well known globally unique
>   and globally meaningful path; add some kind of "variant package"
>   construct which specifies a package which is "passed in" to the
>   environment (maybe by bind-mounting it in a filesystem namespace;
>   implementation specifics aren't important). To put this in a
>   programming language sense, one of these packages being present would
>   turn a Guix distribution from a value into a function.

Not sure I understand.  What do you mean by “passed in to the
environment”?

> - Massively speed up rebuilding A/B/C by performing incremental
>   builds. Not sure how exactly this could work, it's just a thought.

No idea how this could work either.

Thanks for raising the issue!

Ludo’.

Re: Developing libraries for the GNU system with Guix

[sbaugh]

ludo@gnu.org (Ludovic Courtès) writes:
> sbaugh@catern.com skribis:
>> - Currently every dependency is located at a well known globally unique
>>   and globally meaningful path; add some kind of "variant package"
>>   construct which specifies a package which is "passed in" to the
>>   environment (maybe by bind-mounting it in a filesystem namespace;
>>   implementation specifics aren't important). To put this in a
>>   programming language sense, one of these packages being present would
>>   turn a Guix distribution from a value into a function.
>
> Not sure I understand.  What do you mean by “passed in to the
> environment”?

I just mean that this would be a path that is not necessarily pointing
to a directory containing the files desired by other packages depending
on it. To make a complete functioning system, the path would need to be
pointed at something containing the right files. (something of the right
type) It could be pointed at different directories in different
filesystem namespaces, and that "pointing" would happen outside the
filesystem namespace when the namespace is created.

Re: Developing libraries for the GNU system with Guix

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Again to make this more convenient, I thought we could have a
> --with-graft option, which would work like --with-input except that it
> would graft the new Z onto A/B/C instead of rebuilding them.

This is a good idea!

~~ Ricardo

Re: Developing libraries for the GNU system with Guix

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Again to make this more convenient, I thought we could have a
>> --with-graft option, which would work like --with-input except that it
>> would graft the new Z onto A/B/C instead of rebuilding them.
>
> This is a good idea!

Done in 645b9df858683dc05ffa04c9eb2fdc45ccef4a65.  Feedback welcome!

Ludo’.

Re: Developing libraries for the GNU system with Guix

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Again to make this more convenient, I thought we could have a
>>> --with-graft option, which would work like --with-input except that it
>>> would graft the new Z onto A/B/C instead of rebuilding them.
>>
>> This is a good idea!
>
> Done in 645b9df858683dc05ffa04c9eb2fdc45ccef4a65.  Feedback welcome!

Thanks!  This looks great.  The only change I’d make is in the
documentation:

   “instead of rebuilding all the dependency chain, …”

“all the … chain” sounds odd.  How about “the whole dependency chain”
instead?

~~ Ricardo

Re: Developing libraries for the GNU system with Guix

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>
>>> Ludovic Courtès <ludo@gnu.org> writes:
>>>
>>>> Again to make this more convenient, I thought we could have a
>>>> --with-graft option, which would work like --with-input except that it
>>>> would graft the new Z onto A/B/C instead of rebuilding them.
>>>
>>> This is a good idea!
>>
>> Done in 645b9df858683dc05ffa04c9eb2fdc45ccef4a65.  Feedback welcome!
>
> Thanks!  This looks great.  The only change I’d make is in the
> documentation:
>
>    “instead of rebuilding all the dependency chain, …”
>
> “all the … chain” sounds odd.  How about “the whole dependency chain”
> instead?

Yes, please go ahead.  :-)

Ludo’.

Re: Developing libraries for the GNU system with Guix

[sbaugh]

ludo@gnu.org (Ludovic Courtès) writes:
> Done in 645b9df858683dc05ffa04c9eb2fdc45ccef4a65.  Feedback welcome!
>
> Ludo’.

Excellent! I will try to use this in the future, and see how well it
works for my use case.