Avoid sending sources to offload servers

Avoid sending sources to offload servers

[Simen Endsjø]

I notice package sources are sent to offload servers. I have a build that has
1.9GB sources, and results in a 4.7GB store item. I would have thought only the
definitions were sent to the build machine, and it would download the 1.9GB
file, but it seems this file is sent from my machine. I cannot see a way around
this, and it makes it difficult to combine this package with packages I wish to
offload as I cannot state certain packages shouldn't be offloaded.

Not sure what I'm asking here, just wanted to point out an issue which causes
problems.

Re: Avoid sending sources to offload servers

[Hilton Chain]

Hi Simen,

On Thu, 07 Nov 2024 16:57:26 +0800,
Simen Endsjø wrote:
>
> I notice package sources are sent to offload servers. I have a build that has
> 1.9GB sources, and results in a 4.7GB store item. I would have thought only the
> definitions were sent to the build machine, and it would download the 1.9GB
> file, but it seems this file is sent from my machine. I cannot see a way around
> this, and it makes it difficult to combine this package with packages I wish to
> offload as I cannot state certain packages shouldn't be offloaded.
>
> Not sure what I'm asking here, just wanted to point out an issue which causes
> problems.

Guix commands that can spawn builds have a ‘--no-offload’ option, you can see
more these options here[1].


Thanks
---
[1]: https://guix.gnu.org/manual/devel/en/html_node/Common-Build-Options.html

Re: Avoid sending sources to offload servers

[Simen Endsjø]

Yes, that's what I'm using, but it's not granular. The problem is things like `guix shell a b c`.
- a has a small source and long build time
- b has a large source and short build time
- c has a large output

a is no problem to offload, but b will cause a lot of unnecessary traffic, and I might want to run c locally to to avoid passing a lot of data.

I cannot tell --no-offload skip b and c, so If I have a single problematic package in my system, home or in a guix shell, I have to disable offloading for all packages.

On Thu, Nov 7, 2024, at 10:14, Hilton Chain wrote:
> Hi Simen,
> 
> On Thu, 07 Nov 2024 16:57:26 +0800,
> Simen Endsjø wrote:
> >
> > I notice package sources are sent to offload servers. I have a build that has
> > 1.9GB sources, and results in a 4.7GB store item. I would have thought only the
> > definitions were sent to the build machine, and it would download the 1.9GB
> > file, but it seems this file is sent from my machine. I cannot see a way around
> > this, and it makes it difficult to combine this package with packages I wish to
> > offload as I cannot state certain packages shouldn't be offloaded.
> >
> > Not sure what I'm asking here, just wanted to point out an issue which causes
> > problems.
> 
> Guix commands that can spawn builds have a ‘--no-offload’ option, you can see
> more these options here[1].
> 
> 
> Thanks
> ---
> [1]: https://guix.gnu.org/manual/devel/en/html_node/Common-Build-Options.html
> 

Re: Avoid sending sources to offload servers

[Edouard Klein]

I may be mistaken, but you I think you can run guix build a with
offloading, and guix build b c without, then when you run guix shell, a,
b, and c will all be in your store and won't need to be rebuilt.


Simen Endsjø <contact@simendsjo.me> writes:

> Yes, that's what I'm using, but it's not granular. The problem is things like `guix shell a b c`.
> - a has a small source and long build time
> - b has a large source and short build time
> - c has a large output
>
> a is no problem to offload, but b will cause a lot of unnecessary traffic, and I might want to run c locally to to avoid passing a lot of data.
>
> I cannot tell --no-offload skip b and c, so If I have a single problematic package in my system, home or in a guix shell, I have to disable offloading for all packages.
>
> On Thu, Nov 7, 2024, at 10:14, Hilton Chain wrote:
>> Hi Simen,
>>
>> On Thu, 07 Nov 2024 16:57:26 +0800,
>> Simen Endsjø wrote:
>> >
>> > I notice package sources are sent to offload servers. I have a build that has
>> > 1.9GB sources, and results in a 4.7GB store item. I would have thought only the
>> > definitions were sent to the build machine, and it would download the 1.9GB
>> > file, but it seems this file is sent from my machine. I cannot see a way around
>> > this, and it makes it difficult to combine this package with packages I wish to
>> > offload as I cannot state certain packages shouldn't be offloaded.
>> >
>> > Not sure what I'm asking here, just wanted to point out an issue which causes
>> > problems.
>>
>> Guix commands that can spawn builds have a ‘--no-offload’ option, you can see
>> more these options here[1].
>>
>>
>> Thanks
>> ---
>> [1]: https://guix.gnu.org/manual/devel/en/html_node/Common-Build-Options.html
>>

Re: Avoid sending sources to offload servers

[Simen Endsjø]

Yes, but it is very cumbersome. I can no longer just "reconfigure" my home or system, but have to manually to a build for all packages with the correct parameters first.
Same for guix shell; I'd have to first split them into "with offload" and "without offload".

I guess what I would like is at least that the sources for the builds to be downloaded by the build server instead of transferred by my system. But this might be a side effect of sources being built as store items? So it's a dependency for the build, and passed to the build server so it doesn't have to compute it? Maybe having a size limit for passing to the build server could be a solution. If an item is greater than the threshold, the build server will compute it.

On Thu, Nov 7, 2024, at 12:00, Edouard Klein wrote:
> I may be mistaken, but you I think you can run guix build a with
> offloading, and guix build b c without, then when you run guix shell, a,
> b, and c will all be in your store and won't need to be rebuilt.
> 
> 
> Simen Endsjø <contact@simendsjo.me> writes:
> 
> > Yes, that's what I'm using, but it's not granular. The problem is things like `guix shell a b c`.
> > - a has a small source and long build time
> > - b has a large source and short build time
> > - c has a large output
> >
> > a is no problem to offload, but b will cause a lot of unnecessary traffic, and I might want to run c locally to to avoid passing a lot of data.
> >
> > I cannot tell --no-offload skip b and c, so If I have a single problematic package in my system, home or in a guix shell, I have to disable offloading for all packages.
> >
> > On Thu, Nov 7, 2024, at 10:14, Hilton Chain wrote:
> >> Hi Simen,
> >>
> >> On Thu, 07 Nov 2024 16:57:26 +0800,
> >> Simen Endsjø wrote:
> >> >
> >> > I notice package sources are sent to offload servers. I have a build that has
> >> > 1.9GB sources, and results in a 4.7GB store item. I would have thought only the
> >> > definitions were sent to the build machine, and it would download the 1.9GB
> >> > file, but it seems this file is sent from my machine. I cannot see a way around
> >> > this, and it makes it difficult to combine this package with packages I wish to
> >> > offload as I cannot state certain packages shouldn't be offloaded.
> >> >
> >> > Not sure what I'm asking here, just wanted to point out an issue which causes
> >> > problems.
> >>
> >> Guix commands that can spawn builds have a ‘--no-offload’ option, you can see
> >> more these options here[1].
> >>
> >>
> >> Thanks
> >> ---
> >> [1]: https://guix.gnu.org/manual/devel/en/html_node/Common-Build-Options.html
> >>
> 

Re: Avoid sending sources to offload servers

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

Simen Endsjø <contact@simendsjo.me> writes:

> I guess what I would like is at least that the sources for the builds
> to be downloaded by the build server instead of transferred by my
> system.

I guess one limitation here is that the build server is not always able
to download the sources.  Some of my packages have source field like
this:

--8<---------------cut here---------------start------------->8---
(source (local-file "/some/path/some/tar-0.0.0.tar.gz"))
--8<---------------cut here---------------end--------------->8---

I can imagine having packages with sources that are for example
available on LAN exposed HTTP server, and the build server does not have
to be on the same network segment.  I do not have use case for that yet,
but I image other people might have.

All of that is probably solvable, but I just wanted to point out that
there are some edge cases in this that would require deeper thought.

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: Avoid sending sources to offload servers

[Simen Endsjø]

Tomas Volf <~@wolfsden.cz> writes:

> Simen Endsjø <contact@simendsjo.me> writes:
>
>> I guess what I would like is at least that the sources for the builds
>> to be downloaded by the build server instead of transferred by my
>> system.
>
> I guess one limitation here is that the build server is not always able
> to download the sources.  Some of my packages have source field like
> this:
>
> --8<---------------cut here---------------start------------->8---
> (source (local-file "/some/path/some/tar-0.0.0.tar.gz"))
> --8<---------------cut here---------------end--------------->8---
>

Good point. The packages in question doesn't have any local sources, but
I see your point.

It would also be impossible to know per-package what would be the most
optimal build plan. There's the input source size, the build time and
the output size to factor in. At least there's no magic going on here so
I can tweak things manually when needed.

> I can imagine having packages with sources that are for example
> available on LAN exposed HTTP server, and the build server does not have
> to be on the same network segment.  I do not have use case for that yet,
> but I image other people might have.
>
> All of that is probably solvable, but I just wanted to point out that
> there are some edge cases in this that would require deeper thought.
>
> Tomas