Is it necessary to download the native inputs while installing packages?

Is it necessary to download the native inputs while installing packages?

[Fis Trivial]

The network connections to hydra is bad at day time from where I am, the
download speed is about 10kB/s, I can only work with guix at night.
And sometimes I use guix --fallback option(like texlive), guix will redirect to
the upstream download link. Chances are the upstream has even worse network
bandwidth, then I have to manually find a mirror of that particular upstream
and use `guix download`.

In the case of *texlive-texmf*, it serves mostly as a build dependence. To make
things even less convenient, most of the mirror site only store its ISO
package, not the tar ball used in guix package.

So, I gotta wonder, can we just download the normal inputs and propagated
inputs from hydra while installing packages? Would that break something like
the functional property of guix?

Re: Is it necessary to download the native inputs while installing packages?

[Tobias Geerinckx-Rice]

Hullo,

There are quite a few unrelated points mixed together here. I'll try to
address all of them, but I'm not entirely sure what your main question
is. I hope to answer it by accident.

Fis Trivial wrote on 18/01/18 at 16:19:
> The network connections to hydra is bad at day time from where I am,
>  the download speed is about 10kB/s, I can only work with guix at 
> night. And sometimes I use guix --fallback option(like texlive), guix
> will redirect to the upstream download link.

I get your meaning, but ‘redirect’ isn't the right word to use here.
Redirection is something servers do, and that's not the case here.

When installing a package, Guix queries all known substitute servers for
a pre-built binary substitute. What ‘--fallback’ does is tell Guix to
not give up if all servers answer negatively (404), but to build the
package from source locally. For that it needs to download the source
from upstream servers, and of course the native inputs required to
perform the build. There's no way around that.

If any of the substitute servers *can* provide a binary, no
native-inputs will be downloaded at all.

Why are you adding ‘--fallback’?

> Chances are the upstream has even worse network bandwidth, then I 
> have to manually find a mirror of that particular upstream and use 
> `guix download`.

This is an unrelated but valid point. I've previously mused that it
should be possible to let users choose (a set of) their favourite
mirror(s), instead of always downloading sources from the first mirror
queried.

It could be as simple as adding ISO codes to each mirror URI.

> In the case of *texlive-texmf*, it serves mostly as a build 
> dependence. To make things even less convenient, most of the mirror 
> site only store its ISO package, not the tar ball used in guix 
> package.

Oh. That's unfortunate. I'm not familiar with the Guix texlive packages,
and all the ones I happened to look at now have a direct ftp:// link.

Is upstream the one doing the — heh — redirection here? If so, it should
arguably doing a better job. We could still help matters by hard-coding
a shortlist of mirrors known to host the files we need.

> So, I gotta wonder, can we just download the normal inputs and 
> propagated inputs from hydra while installing packages? Would that 
> break something like the functional property of guix?

‘Installing’ is too ambiguous to answer.

- When *substituting* (installing a binary), native-inputs are not
required and should never be downloaded.
- When *building* (from source), they are.

Kind regards,

T G-R

PS: Oh. Right. Sigh... To complicate matters, texlive-texmf is a special
case that's never substituted, because doing so is thought to be worse
than just building it from source[1].

That's the reason you're downloading TexLive from upstream, but it's
*not* the reason you're pulling in TexLive to begin with and doesn't
change the native-inputs situation. I think.

[1]:
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/hydra.gnu.org-locations.conf#n23

Re: Is it necessary to download the native inputs while installing packages?

[Leo Famulari]

Sorry for the top-posting.

I need to clarify what --fallback does.

Guix will always build locally if the substitute servers say they lack something Guix needs.

--fallback works when the substitute servers say they have something, and then the substitution fails unexpectedly. For example, the server may become overloaded and drop the connection.


-------- Original Message --------
From: Tobias Geerinckx-Rice <me@tobias.gr>
Sent: January 18, 2018 9:36:50 AM PST
To: ybbs.daans@hotmail.com, guix-devel@gnu.org
Subject: Re: Is it necessary to download the native inputs while installing	packages?

Hullo,

There are quite a few unrelated points mixed together here. I'll try to
address all of them, but I'm not entirely sure what your main question
is. I hope to answer it by accident.

Fis Trivial wrote on 18/01/18 at 16:19:
> The network connections to hydra is bad at day time from where I am,
>  the download speed is about 10kB/s, I can only work with guix at 
> night. And sometimes I use guix --fallback option(like texlive), guix
> will redirect to the upstream download link.

I get your meaning, but ‘redirect’ isn't the right word to use here.
Redirection is something servers do, and that's not the case here.

When installing a package, Guix queries all known substitute servers for
a pre-built binary substitute. What ‘--fallback’ does is tell Guix to
not give up if all servers answer negatively (404), but to build the
package from source locally. For that it needs to download the source
from upstream servers, and of course the native inputs required to
perform the build. There's no way around that.

If any of the substitute servers *can* provide a binary, no
native-inputs will be downloaded at all.

Why are you adding ‘--fallback’?

> Chances are the upstream has even worse network bandwidth, then I 
> have to manually find a mirror of that particular upstream and use 
> `guix download`.

This is an unrelated but valid point. I've previously mused that it
should be possible to let users choose (a set of) their favourite
mirror(s), instead of always downloading sources from the first mirror
queried.

It could be as simple as adding ISO codes to each mirror URI.

> In the case of *texlive-texmf*, it serves mostly as a build 
> dependence. To make things even less convenient, most of the mirror 
> site only store its ISO package, not the tar ball used in guix 
> package.

Oh. That's unfortunate. I'm not familiar with the Guix texlive packages,
and all the ones I happened to look at now have a direct ftp:// link.

Is upstream the one doing the — heh — redirection here? If so, it should
arguably doing a better job. We could still help matters by hard-coding
a shortlist of mirrors known to host the files we need.

> So, I gotta wonder, can we just download the normal inputs and 
> propagated inputs from hydra while installing packages? Would that 
> break something like the functional property of guix?

‘Installing’ is too ambiguous to answer.

- When *substituting* (installing a binary), native-inputs are not
required and should never be downloaded.
- When *building* (from source), they are.

Kind regards,

T G-R

PS: Oh. Right. Sigh... To complicate matters, texlive-texmf is a special
case that's never substituted, because doing so is thought to be worse
than just building it from source[1].

That's the reason you're downloading TexLive from upstream, but it's
*not* the reason you're pulling in TexLive to begin with and doesn't
change the native-inputs situation. I think.

[1]:
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/hydra.gnu.org-locations.conf#n23

Re: Is it necessary to download the native inputs while installing packages?

[Tobias Geerinckx-Rice]

Leo,

Leo Famulari wrote on 18/01/18 at 20:56:
> --fallback works when the substitute servers say they have something,
> and then the substitution fails unexpectedly. For example, the server
> may become overloaded and drop the connection.

You are, of course, completely right & I must have been smoking drugs.
'specially since I was thinking that's probably what's happening on an
unreliable connection.

Thanks,

T G-R

Re: Is it necessary to download the native inputs while installing packages?

[Fis Trivial]

Sorry about the mess in the original question.

>
> ‘Installing’ is too ambiguous to answer.
>
> - When *substituting* (installing a binary), native-inputs are not
> required and should never be downloaded.

This answers my question in mind, thanks. :)

>
> If any of the substitute servers *can* provide a binary, no
> native-inputs will be downloaded at all.
>
> Why are you adding ‘--fallback’?
>

As you explained, the substituting process failed.

Thanks for explaining things clearly. I will try to look into the source of
guix next time.

Re: Is it necessary to download the native inputs while installing packages?

[Ludovic Courtès]

Fis Trivial <ybbs.daans@hotmail.com> skribis:

>> If any of the substitute servers *can* provide a binary, no
>> native-inputs will be downloaded at all.
>>
>> Why are you adding ‘--fallback’?
>>
>
> As you explained, the substituting process failed.

This particular issue should now be fixed: <https://bugs.gnu.org/30157>.

Ludo’.

Re: Is it necessary to download the native inputs while installing packages?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On Thu, Jan 18, 2018 at 08:54:32PM +0000, Fis Trivial wrote:
> Sorry about the mess in the original question.
> 
> >
> > ‘Installing’ is too ambiguous to answer.
> >
> > - When *substituting* (installing a binary), native-inputs are not
> > required and should never be downloaded.
> 
> This answers my question in mind, thanks. :)

Although native-inputs are typically things that are only required while
building [0], there's nothing that prevents a built package from keeping
references to native-inputs.

[0] The "native" part is related to cross-compilation.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Prevent native-inputs references ending up in the final binary

[Danny Milosavljevic]

Hi Leo,

> Although native-inputs are typically things that are only required while
> building [0], there's nothing that prevents a built package from keeping
> references to native-inputs.

We should change that in core-updates-next, if possible.
 
I think that native-inputs shouldn't end up in the final binary as a
reference, especially when cross-compiling
(but we don't do cross-compilation much in Guix - usually, we let
qemu-arm emulate the ARM CPU on x86_64 and just call the target tool :) ).

If there are indeed parts of the same package, one a native part and one a
runtime dependency part, I actually write the same package reference twice,
once in the inputs, once in the native-inputs, in my custom package
definitions.

In a "previous life" I did a lot of Linux cellphone development and,
there, it was kinda important that a x86_64 toolchain doesn't end
up being referenced in an ARM binary, so the habit stuck - and I
think it's important to distinguish the mold used to form a product
from an integral part of that product.

Re: Prevent native-inputs references ending up in the final binary

[Tobias Geerinckx-Rice]

Danny,

Danny Milosavljevic wrote on 20/01/18 at 11:40:
> We should change that in core-updates-next, if possible.
>  
> I think that native-inputs shouldn't end up in the final binary as a
> reference [...]
This has been discussed before, and I agree. (I started a branch to do
so but it breaks quite a few things and it got tedious. I think I'm
ready for more now.)

Kind regards,

T G-R

Re: Prevent native-inputs references ending up in the final binary

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

On Sat, Jan 20, 2018 at 04:47:14PM +0100, Tobias Geerinckx-Rice wrote:
> Danny,
> 
> Danny Milosavljevic wrote on 20/01/18 at 11:40:
> > We should change that in core-updates-next, if possible.
> >  
> > I think that native-inputs shouldn't end up in the final binary as a
> > reference [...]
> This has been discussed before, and I agree. (I started a branch to do
> so but it breaks quite a few things and it got tedious. I think I'm
> ready for more now.)

Yeah, I think a lot of things will break.

Something to consider is that not every reference has an effect on the
built software. Sometimes the reference is a nonfunctional string that
ends up in documentation, or for error messages, or many other places.
It may be a lot of work to remove them all.

I'd rather we do it somewhere else than core-updates.

It's already very difficult to complete the core-updates cycles. We
should limit core-updates to updates of core packages, and handle big
changes to Guix itself on their own branches.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Prevent native-inputs references ending up in the final binary

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 1027 bytes --]

Leo,

Leo Famulari wrote on 21/01/18 at 23:37:
> On Sat, Jan 20, 2018 at 04:47:14PM +0100, Tobias Geerinckx-Rice wrote:
>> Danny,
>>
>> Danny Milosavljevic wrote on 20/01/18 at 11:40:
>>> We should change that in core-updates-next, if possible.
>>>  
>>> I think that native-inputs shouldn't end up in the final binary as a
>>> reference [...]
>> This has been discussed before, and I agree. (I started a branch to do
>> so but it breaks quite a few things and it got tedious. I think I'm
>> ready for more now.)

[...]

> I'd rather we do it somewhere else than core-updates.
> 
> It's already very difficult to complete the core-updates cycles. We
> should limit core-updates to updates of core packages, and handle big
> changes to Guix itself on their own branches.

Er, yeah. Definitely.

Unfortunate quoting on my part if it seemed like I was implying
otherwise. This is the right thing to do, but doing it on core-updates
is going to hurt rather than help anyone.

Kind regards,

T G-R


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

Re: Prevent native-inputs references ending up in the final binary

[Ludovic Courtès]

Hi,

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> Leo Famulari wrote on 21/01/18 at 23:37:
>> On Sat, Jan 20, 2018 at 04:47:14PM +0100, Tobias Geerinckx-Rice wrote:
>>> Danny,
>>>
>>> Danny Milosavljevic wrote on 20/01/18 at 11:40:
>>>> We should change that in core-updates-next, if possible.
>>>>  
>>>> I think that native-inputs shouldn't end up in the final binary as a
>>>> reference [...]
>>> This has been discussed before, and I agree. (I started a branch to do
>>> so but it breaks quite a few things and it got tedious. I think I'm
>>> ready for more now.)
>
> [...]
>
>> I'd rather we do it somewhere else than core-updates.
>> 
>> It's already very difficult to complete the core-updates cycles. We
>> should limit core-updates to updates of core packages, and handle big
>> changes to Guix itself on their own branches.
>
> Er, yeah. Definitely.

+1

Anyway, I think it’s worth experimenting this on a branch.  We’ll have
to expect lots of breakage as Leo writes, so we’ll need to refine things
progressively.

Once such a branch exist, we can get it built on berlin or hydra.

Ludo’.

Re: Prevent native-inputs references ending up in the final binary

[Fis Trivial]

On 01/20/2018 06:40 PM, Danny Milosavljevic wrote:
> Hi Leo,
> 
>> Although native-inputs are typically things that are only required while
>> building [0], there's nothing that prevents a built package from keeping
>> references to native-inputs.
> 
> We should change that in core-updates-next, if possible.
>  
> I think that native-inputs shouldn't end up in the final binary as a
> reference, especially when cross-compiling
> (but we don't do cross-compilation much in Guix - usually, we let
> qemu-arm emulate the ARM CPU on x86_64 and just call the target tool :) ).
> 
> If there are indeed parts of the same package, one a native part and one a
> runtime dependency part, I actually write the same package reference twice,
> once in the inputs, once in the native-inputs, in my custom package
> definitions.
> 
> In a "previous life" I did a lot of Linux cellphone development and,
> there, it was kinda important that a x86_64 toolchain doesn't end
> up being referenced in an ARM binary, so the habit stuck - and I
> think it's important to distinguish the mold used to form a product
> from an integral part of that product.
> 

I'm no expert, but can this little utility from nix help?
https://nixos.org/patchelf.html

Re: Prevent native-inputs references ending up in the final binary

[Tobias Geerinckx-Rice]

Hullo,

On Fri, 2018-01-26 at 00:56 +0000, Fis Trivial wrote:
> On 01/20/2018 06:40 PM, Danny Milosavljevic wrote:
> > I think that native-inputs shouldn't end up in the final binary as
> > a
> > reference, especially when cross-compiling
> > (but we don't do cross-compilation much in Guix - usually, we let
> > qemu-arm emulate the ARM CPU on x86_64 and just call the target
> > tool :) ).
> > 
> > If there are indeed parts of the same package, one a native part
> > and one a
> > runtime dependency part, I actually write the same package
> > reference twice,
> > once in the inputs, once in the native-inputs, in my custom package
> > definitions.
> > 
> > In a "previous life" I did a lot of Linux cellphone development
> > and,
> > there, it was kinda important that a x86_64 toolchain doesn't end
> > up being referenced in an ARM binary, so the habit stuck - and I
> > think it's important to distinguish the mold used to form a product
> > from an integral part of that product.
> > 
> 
> I'm no expert, but can this little utility from nix help?
> https://nixos.org/patchelf.html

In what way?

Patchelf re-writes library and/or loader paths in compiled binaries.
It's cool, but I don't immediately see the connexion...

Kind regards,

T G-R