Re: Developing libraries for the GNU system with Guix

[ludo@gnu.org (Ludovic Courtès)]

Hello!

sbaugh@catern.com skribis:

> When I am hacking on some library Z, I continuously want to test the
> effects that my changes to Z have on packages A/B/C which depend on
> Z. The same applies, in general, when hacking on any package Z which
> other packages A/B/C depend on: While developing, I want to be able to
> rapidly mutate Z and see how this affects A/B/C.

A very common use case.  Others have asked about it.

> I am not sure how to best achieve this. Here are some solutions:
>
> - When you change Z, rebuild A/B/C.
>   This is much too slow at present.

Right, but note that it’s the only way to be confident that the change
in Z doesn’t break A/B/C.  That said, I do understand that sometimes,
you want an “I know what I’m doing” (i.e., the ABI of Z hasn’t changed)
option to bypass that and test the run-time behavior of A/B/C.

For cases where you do want to rebuild anyway, I was thinking of making
the ‘--with-source’ option “recursive”, such that:

  guix build --with-source=./guile-2.0.14rc1.tar.gz guile-json

would replace the source of Guile and build Guile-JSON against that.

> - Use grafts to update A/B/C through binary patching.
>   This is also too slow, and AFAIK it can't really be sped up.

A/B/C have to be really big for this to be too slow!  I think grafting
processes several MiB/s on my SSD laptop.

Again to make this more convenient, I thought we could have a
--with-graft option, which would work like --with-input except that it
would graft the new Z onto A/B/C instead of rebuilding them.

> - Use LD_LIBRARY_PATH to cause A/B/C to search for Z in a mutable place.
>   This works for C libraries, but not generically; there are equivalent
>   variables for some other languages but it's not a full solution.

Yeah, not great.

> - Before starting to hack on Z, build a new version of Z which includes
>   a random hash and which A/B/C depend on; then bind-mount a mutable
>   directory on top of that. (suggested by mark_weaver on IRC)
>   This is the most palatable hack, but still a hack. The inclusion of a
>   random hash prevents collision with other packages and the use of
>   bind-mounting means we aren't actually mutating the store. This
>   unfortunately also requires privileges: it's not usable by
>   unprivileged users.

It is usable by unprivileged users when user namespaces are available.
Under these conditions, you could use ‘call-with-container’ and
bind-mount anything anywhere.  Doesn’t sound too nice to me though.

> Here are some not currently available possibilities:
>
> - Create some kind of GUIX_LIBRARY_PATH in which packages are looked up
>   first before looking at the compiled-in hash.
>   This would be an attempt to make a generic equivalent of
>   LD_LIBRARY_PATH. This could theoretically be implemented with variant
>   symlinks, if they were available: https://lwn.net/Articles/680705/

At first sight this sounds very hacky, very much against the whole idea
of functional package management.

> - Currently every dependency is located at a well known globally unique
>   and globally meaningful path; add some kind of "variant package"
>   construct which specifies a package which is "passed in" to the
>   environment (maybe by bind-mounting it in a filesystem namespace;
>   implementation specifics aren't important). To put this in a
>   programming language sense, one of these packages being present would
>   turn a Guix distribution from a value into a function.

Not sure I understand.  What do you mean by “passed in to the
environment”?

> - Massively speed up rebuilding A/B/C by performing incremental
>   builds. Not sure how exactly this could work, it's just a thought.

No idea how this could work either.

Thanks for raising the issue!

Ludo’.