A single reference to installed non-binaries

A single reference to installed non-binaries

[Phil Beadling]

Hi all,

I have some platform independent files I have created a package for using
copy-build-system.  This works great but I've come across situation I don't
know how to handle.

Let's call the package I've made package-x, and let's say that package-y
(which is for arguments sake is a python build system) lists package-x as a
propagated-input.

package-x is installing a directory, let's say "bar" from the source into
"share/foo/"

'(#:install-plan '(("bar" "share/foo/")))

Now whenever we install package-y, I should expect it's "share" directory
to contain foo/bar - and I find it to be the case.

No suprises so far.

The problem comes when I want to reference a file under "bar" in the source
code of package-y.

Depending on whether I install package-y via "guix install package-y -p
/path/to/profile" or via "guix environment --ad-hoc package-y" there is no
single reference to the bar directory that covers every use-case.

When I "install" the package - I can reference it using $GUIX_PROFILE
But as for example a developer when I'm coding package-y I would reference
it using $GUIX_ENVIRONMENT

This means that any source that references it must presumably attempt to
read it from $GUIX_ENVIRONMENT, and then on failure fallback to
$GUIX_PROFILE.

This feels a bit brittle to me, and I'm hoping I've missed a trick, and
there's a better way to singluarly reference the location of a share
directory from any GUIX profile or environment?

Note the problem doesn't happen with binaries as the order of precedence in
the PATH variable avoids the issue.

Any ideas?

Thanks,
Phil.

Re: A single reference to installed non-binaries

[Edouard Klein]

Hi !

I side-step this kind of things by adding a stage in package-y that will
find and replace all references to "bar" with the complete path to the
installation path of package-x, or that will set the needed environment
variable to the full path of the dependency.

See e.g.
https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/packages/scheme-xyz.scm#L68

Here, xlsxio and tzdir will be expanded to their full install path in
the store.

It has the advantage of not needing to integrate any guix-realted stuff
in package-y, which I would consider an abstraction leak.

I hope this helps :)

Cheers,

Edouard.

Phil Beadling writes:

> Hi all,
>
> I have some platform independent files I have created a package for using
> copy-build-system.  This works great but I've come across situation I don't
> know how to handle.
>
> Let's call the package I've made package-x, and let's say that package-y
> (which is for arguments sake is a python build system) lists package-x as a
> propagated-input.
>
> package-x is installing a directory, let's say "bar" from the source into
> "share/foo/"
>
> '(#:install-plan '(("bar" "share/foo/")))
>
> Now whenever we install package-y, I should expect it's "share" directory
> to contain foo/bar - and I find it to be the case.
>
> No suprises so far.
>
> The problem comes when I want to reference a file under "bar" in the source
> code of package-y.
>
> Depending on whether I install package-y via "guix install package-y -p
> /path/to/profile" or via "guix environment --ad-hoc package-y" there is no
> single reference to the bar directory that covers every use-case.
>
> When I "install" the package - I can reference it using $GUIX_PROFILE
> But as for example a developer when I'm coding package-y I would reference
> it using $GUIX_ENVIRONMENT
>
> This means that any source that references it must presumably attempt to
> read it from $GUIX_ENVIRONMENT, and then on failure fallback to
> $GUIX_PROFILE.
>
> This feels a bit brittle to me, and I'm hoping I've missed a trick, and
> there's a better way to singluarly reference the location of a share
> directory from any GUIX profile or environment?
>
> Note the problem doesn't happen with binaries as the order of precedence in
> the PATH variable avoids the issue.
>
> Any ideas?
>
> Thanks,
> Phil.

Re: A single reference to installed non-binaries

[Phil]

Thanks for comments Edouard!  Responses inline.


Edouard Klein writes:


> See e.g.
> https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/packages/scheme-xyz.scm#L68

Ahh so wrap-program creates a script that sets the two env vars
LD_LIBRARY_PATH and TZDIR before calling the original script?

> It has the advantage of not needing to integrate any guix-realted stuff
> in package-y, which I would consider an abstraction leak.

Yes it's nicer for the underlying program not to have to know about
Guix.

There is one small wrinkle with this in my particular case.  Whilst I
can wrap the main entry point to the program easily enough, there is a
second entry point via the unit tester (pytest), which is obviously an
external tool so not so easy to wrap.

I could patch the source code of the unit test itself as an alternative.

Another slightly more leftfield idea is to change (or add) package-x to be a
python package which holds the original data files and a very thin API
client that can serve those files up to Python.

Then package-x's python module will be self-aware of its location, relative
to the text files and we can serve up either a file path or file
object to package-y just by importing package-x, and calling a function.

The disadvantage of this would be if we had non-python clients
too, but we could keep two different build systems for the same source -
one copies the files, and the other installs a python module.

I'll give this a whirl.

Re: A single reference to installed non-binaries

[Edouard Klein]

Phil writes:

> Thanks for comments Edouard!  Responses inline.

:)

>
>
> Edouard Klein writes:
>
>
>> See e.g.
>> https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/packages/scheme-xyz.scm#L68
>
> Ahh so wrap-program creates a script that sets the two env vars
> LD_LIBRARY_PATH and TZDIR before calling the original script?

Yes, it is exactly what it does, with the added cleverness that if the
original script is already a wrap, it edits the wrap instead of re-wrapping.
>
>> It has the advantage of not needing to integrate any guix-realted stuff
>> in package-y, which I would consider an abstraction leak.
>
> Yes it's nicer for the underlying program not to have to know about
> Guix.
>
> There is one small wrinkle with this in my particular case.  Whilst I
> can wrap the main entry point to the program easily enough, there is a
> second entry point via the unit tester (pytest), which is obviously an
> external tool so not so easy to wrap.
>
> I could patch the source code of the unit test itself as an alternative.
>

As you can see a bit below where I linked before, you can redefine the test
stage in order to set the correct environment variables before calling
your test command. It will be a bit redundant with the wrapping phase,
but it would allow you not to edit your application code.


> Another slightly more leftfield idea is to change (or add) package-x to be a
> python package which holds the original data files and a very thin API
> client that can serve those files up to Python.
>
> Then package-x's python module will be self-aware of its location, relative
> to the text files and we can serve up either a file path or file
> object to package-y just by importing package-x, and calling a function.
>

I don't know the particulars of your code, but package-x and package-y
seem to be quite coupled together, so I would question the assumption
that they belong in separate packages.

Maybe they do, nobody is better placed than you to judge. But putting
them both in the same package would make your problems disappear.


> The disadvantage of this would be if we had non-python clients
> too, but we could keep two different build systems for the same source -
> one copies the files, and the other installs a python module.
>
> I'll give this a whirl.


Good luck !

Cheers,

Edouard.

Re: A single reference to installed non-binaries

[Phil]

Edouard Klein writes:

> As you can see a bit below where I linked before, you can redefine the test
> stage in order to set the correct environment variables before calling
> your test command. It will be a bit redundant with the wrapping phase,
> but it would allow you not to edit your application code.

This makes sense - thanks.

> I don't know the particulars of your code, but package-x and package-y
> seem to be quite coupled together, so I would question the assumption
> that they belong in separate packages.

package-x is need by other packages not just package-y, otherwise I
agree the easiest solution would be to move pacakge-x inside package-y.

Re: A single reference to installed non-binaries

[Leo Famulari]

On Tue, Aug 17, 2021 at 04:01:15PM +0200, Edouard Klein wrote:
> See e.g.
> https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/packages/scheme-xyz.scm#L68
> 
> Here, xlsxio and tzdir will be expanded to their full install path in
> the store.

I'm going off-topic, but you should not patch TZDIR like this.

The time zone database should be found dynamically at run-time via an
environment variable set by the system. Otherwise, your built package
will eventually "go stale" as the time zone database is updated, which
happens several times per year.

Does that make sense?

Setting TZDIR (was Re: A single reference to installed non-binaries)

[Edouard Klein]

Leo Famulari writes:

> On Tue, Aug 17, 2021 at 04:01:15PM +0200, Edouard Klein wrote:
>> See e.g.
>> https://gitlab.com/edouardklein/guix/-/blob/beaverlabs/beaver/packages/scheme-xyz.scm#L68
>> 
>> Here, xlsxio and tzdir will be expanded to their full install path in
>> the store.
>
> I'm going off-topic, but you should not patch TZDIR like this.

I appreciate you going off-topic to warn me about a mistake ! Thanks :)

>
> The time zone database should be found dynamically at run-time via an
> environment variable set by the system. Otherwise, your built package
> will eventually "go stale" as the time zone database is updated, which
> happens several times per year.
>
> Does that make sense?

It does, thanks, but I'm in a bit of a pickle there because this
packagge needs, for reasons outside of my control, to be deployable on
a multitude of host (non-GuixSD) linux distributions, and as a docker
container. In those cases, I can't expect the timezone data to be up to
date or to be there at all.

Even if I list tzdata as a dependency, the host system's TZDIR will not
point to it.

I understand that if I regularly guix pull and guix package -u, then the
tzdata package will be kept up to date. Is that correct ?

If it is, then I need to rebuild the .tar.gz and the docker image every
so often, and this problem would be solved for the foreing distros as well.

Thanks for alerting me about this !

Cheers,

Edouard.

Re: Setting TZDIR (was Re: A single reference to installed non-binaries)

[Leo Famulari]

On Tue, Aug 17, 2021 at 08:49:27PM +0200, Edouard Klein wrote:
> It does, thanks, but I'm in a bit of a pickle there because this
> packagge needs, for reasons outside of my control, to be deployable on
> a multitude of host (non-GuixSD) linux distributions, and as a docker
> container. In those cases, I can't expect the timezone data to be up to
> date or to be there at all.
> 
> Even if I list tzdata as a dependency, the host system's TZDIR will not
> point to it.

You wouldn't list tzdata as a dependency at all. It's the responsibility
of the host operating system to provide the correct and current time
zone database, and applications should look it up at run-time via TZDIR.

If you are deploying to systems without a time zone database, then you
could indeed consider "baking" the reference into your package while
building, or consider if you need to care about having correct local
time at all.

> I understand that if I regularly guix pull and guix package -u, then the
> tzdata package will be kept up to date. Is that correct ?

Yes. But, installed packages should not need to be rebuilt to display
the correct time. To elaborate, if you hardcode a store reference to
tzdata in your package definitions, then old generations of installed
packages will eventually display incorrect local time, because time
zones are frequently changed by local governments. And that failure mode
is not okay for packages included in GNU Guix.

This is one of the rare cases in Guix packaging where dynamic lookup of
dependencies is preferred.