Extra files in build container

Extra files in build container

[Gregor Giesen]

Hello Guix,

I'm trying to pack unbound, however its unittests require /etc/protocols
and /etc/services. Now, I have found them in the net-base package 
which I included in native-inputs, but that is not sufficient for
glibc's getprotoent and getservent to find them. Can I symlink the
net-base ones to the build container's /etc? Adding an extra phase 
before 'check does not allow me to write /etc in the container.

Cheers,
Gregor

Re: Extra files in build container

[Ludovic Courtès]

Hi Gregor,

Gregor Giesen <giesen@zaehlwerk.net> skribis:

> I'm trying to pack unbound, however its unittests require /etc/protocols
> and /etc/services. Now, I have found them in the net-base package 
> which I included in native-inputs, but that is not sufficient for
> glibc's getprotoent and getservent to find them. Can I symlink the
> net-base ones to the build container's /etc? Adding an extra phase 
> before 'check does not allow me to write /etc in the container.

Indeed, /etc is read-only in the build container, and it contains only
the bare minimum:

  https://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html

The code that populates it can be found at:

  https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc#n1852

In cases like the one you describe, we usually end up modifying tests to
use the numerical values for services and protocols rather than their
names.

It would probably make sense to allow derivations to populate /etc
though.

HTH,
Ludo’.

Re: Extra files in build container

[Gregor Giesen]

Dear Ludo’,

thanks for the help!

On Fri, Jun 16, 2017 at 09:58:55AM +0200, Ludovic Courtès wrote:
> In cases like the one you describe, we usually end up modifying tests to
> use the numerical values for services and protocols rather than their
> names.
Unfortunately, this turns out to be quite cumbersome since in my case
(unittests for unbound) there is a lot of test data to be modified and
in many cases not only plain text but also encrypted records (DNSSEC 
tests). On the other hand the values to be looked up are mostly “udp”
and “tcp” in /etc/protocols and “domain” in /etc/services, so I decided 
that using a preload library for these few glibc calls just in case of 
the unittest should do the trick rather than no checks at all. However, 
it is an ugly hack and bloats the package definition.

> It would probably make sense to allow derivations to populate /etc
> though.
That might help!

Many thanks to all contributers for this very nice peace of software!

Best,
Gregor

Re: Extra files in build container

[Ludovic Courtès]

Hi Gregor,

Gregor Giesen <giesen@zaehlwerk.net> skribis:

> On Fri, Jun 16, 2017 at 09:58:55AM +0200, Ludovic Courtès wrote:
>> In cases like the one you describe, we usually end up modifying tests to
>> use the numerical values for services and protocols rather than their
>> names.
> Unfortunately, this turns out to be quite cumbersome since in my case
> (unittests for unbound) there is a lot of test data to be modified and
> in many cases not only plain text but also encrypted records (DNSSEC 
> tests). On the other hand the values to be looked up are mostly “udp”
> and “tcp” in /etc/protocols and “domain” in /etc/services, so I decided 
> that using a preload library for these few glibc calls just in case of 
> the unittest should do the trick rather than no checks at all.

I think it would be easier to just use ‘substitute*’ to replace all the
occurrences of “tcp”, etc., wouldn’t it?

> However, it is an ugly hack and bloats the package definition.

I agree, but it’s hard to improve on it without compromising
reproducibility.

Thanks,
Ludo’.

Re: Extra files in build container

[Gregor Giesen]

Dear Ludo’,

On Mon, Jun 19, 2017 at 13:26AM +0200 Ludovic Courtès wrote:
>> On Fri, Jun 16, 2017 at 09:58:55AM +0200, Ludovic Courtès wrote:
>>> In cases like the one you describe, we usually end up modifying tests to
>>> use the numerical values for services and protocols rather than their
>>> names.
>> Unfortunately, this turns out to be quite cumbersome since in my case
>> (unittests for unbound) there is a lot of test data to be modified and
>> in many cases not only plain text but also encrypted records (DNSSEC
>> tests). On the other hand the values to be looked up are mostly “udp”
>> and “tcp” in /etc/protocols and “domain” in /etc/services, so I decided
>> that using a preload library for these few glibc calls just in case of
>> the unittest should do the trick rather than no checks at all.
> 
> I think it would be easier to just use ‘substitute*’ to replace all the
> occurrences of “tcp”, etc., wouldn’t it?
alas many occurences of "tcp", "udp", etc. are hidden in 
encrypted/hashed records, so simple plain text substitutions would break 
the signatures.

>> However, it is an ugly hack and bloats the package definition.
> I agree, but it’s hard to improve on it without compromising
> reproducibility.
It's a good thing that the hack only affects the tests rather than any 
installation files.

I have submitted the package in a patch yesterday (27419). I wonder 
whether one might want to put the source code of the preload library in 
an extra file rather than the package definition?

Cheers,
Gregor

Re: Extra files in build container

[Ludovic Courtès]

Hello,

Gregor Giesen <giesen@zaehlwerk.net> skribis:

> On Mon, Jun 19, 2017 at 13:26AM +0200 Ludovic Courtès wrote:
>>> On Fri, Jun 16, 2017 at 09:58:55AM +0200, Ludovic Courtès wrote:
>>>> In cases like the one you describe, we usually end up modifying tests to
>>>> use the numerical values for services and protocols rather than their
>>>> names.
>>> Unfortunately, this turns out to be quite cumbersome since in my case
>>> (unittests for unbound) there is a lot of test data to be modified and
>>> in many cases not only plain text but also encrypted records (DNSSEC
>>> tests). On the other hand the values to be looked up are mostly “udp”
>>> and “tcp” in /etc/protocols and “domain” in /etc/services, so I decided
>>> that using a preload library for these few glibc calls just in case of
>>> the unittest should do the trick rather than no checks at all.
>>
>> I think it would be easier to just use ‘substitute*’ to replace all the
>> occurrences of “tcp”, etc., wouldn’t it?
> alas many occurences of "tcp", "udp", etc. are hidden in
> encrypted/hashed records, so simple plain text substitutions would
> break the signatures.

Yes, I can imagine.  But maybe (maybe not!), by selecting the right set
of files to modify, and by adjusting the regexp (let’s say “\<tcp\>” or
“"tcp"”) to match only what matters, it could work.

If that’s really not workable, then the LD_PRELOAD approach is fine.

>>> However, it is an ugly hack and bloats the package definition.
>> I agree, but it’s hard to improve on it without compromising
>> reproducibility.
> It's a good thing that the hack only affects the tests rather than any
> installation files.
>
> I have submitted the package in a patch yesterday (27419). I wonder
> whether one might want to put the source code of the preload library
> in an extra file rather than the package definition?

Since it’s small I think it’s OK as inline code.

Thanks,
Ludo’.

Re: Extra files in build container

[Gregor Giesen]

Dear Ludo’,

On Mon, Jun 19, 2017 at 04:49:43PM +0200, Ludovic Courtès wrote:
> >> I think it would be easier to just use ‘substitute*’ to replace all the
> >> occurrences of “tcp”, etc., wouldn’t it?
> > alas many occurences of "tcp", "udp", etc. are hidden in
> > encrypted/hashed records, so simple plain text substitutions would
> > break the signatures.
> 
> Yes, I can imagine.  But maybe (maybe not!), by selecting the right set
> of files to modify, and by adjusting the regexp (let’s say “\<tcp\>” or
> “"tcp"”) to match only what matters, it could work.
here is a sample record pair:

wks02.types-signed.wb.sidnlabs.nl.      60      IN      WKS     10.0.0.1 udp 0 1 2 domain
wks02.types-signed.wb.sidnlabs.nl.      60      IN      RRSIG   WKS 8 5 60 20140201000000 20130930114324 62298 types-signed.wb.sidnlabs.nl. P+p1gcQbO4I/sBQUZktrz4Q1osWIFIFGlK8o+SzIXMjxmdNy36vjwW6Sfy97CycbLRFIQ2grPv/cPaXtoMb+usHCoDtl5sSvLTJFmg9hpQ+xm12GvunZvAYAGx9fZic+QvLahSg+cjqX1M0oR9B68gcx+duMdLzawlUcIqX8gmA=

So changing the first line requires to recreating the signature record
in the second line which I find much more difficult then the LD_PRELOAD.

> If that’s really not workable, then the LD_PRELOAD approach is fine.
> Since it’s small I think it’s OK as inline code.
Thanks, I just submitted a revised patch including your remarks and
some additional comments as explanation.

Best,
Gregor

Re: Extra files in build container

[Ludovic Courtès]

Hi Gregor,

Gregor Giesen <giesen@zaehlwerk.net> skribis:

> On Mon, Jun 19, 2017 at 04:49:43PM +0200, Ludovic Courtès wrote:
>> >> I think it would be easier to just use ‘substitute*’ to replace all the
>> >> occurrences of “tcp”, etc., wouldn’t it?
>> > alas many occurences of "tcp", "udp", etc. are hidden in
>> > encrypted/hashed records, so simple plain text substitutions would
>> > break the signatures.
>> 
>> Yes, I can imagine.  But maybe (maybe not!), by selecting the right set
>> of files to modify, and by adjusting the regexp (let’s say “\<tcp\>” or
>> “"tcp"”) to match only what matters, it could work.
> here is a sample record pair:
>
> wks02.types-signed.wb.sidnlabs.nl.      60      IN      WKS     10.0.0.1 udp 0 1 2 domain
> wks02.types-signed.wb.sidnlabs.nl.      60      IN      RRSIG   WKS 8 5 60 20140201000000 20130930114324 62298 types-signed.wb.sidnlabs.nl. P+p1gcQbO4I/sBQUZktrz4Q1osWIFIFGlK8o+SzIXMjxmdNy36vjwW6Sfy97CycbLRFIQ2grPv/cPaXtoMb+usHCoDtl5sSvLTJFmg9hpQ+xm12GvunZvAYAGx9fZic+QvLahSg+cjqX1M0oR9B68gcx+duMdLzawlUcIqX8gmA=
>
> So changing the first line requires to recreating the signature record
> in the second line which I find much more difficult then the LD_PRELOAD.

Indeed, I hadn’t realized that test data included such things.  All
right then!

Ludo’.