guix package build: can distcc be used?

guix package build: can distcc be used?

[Andy Tai]

curious if distcc can be used for building packages in Guix?

If yes that can reduce the time spent in building significantly.

Re: guix package build: can distcc be used?

[Adam Faiz]

> curious if distcc can be used for building packages in Guix?
> 
> If yes that can reduce the time spent in building significantly.
Currently, guix build doesn't support distcc for distributing individual 
source code files for compilation.
However, guix offload can be used to offload the whole build jobs to 
another trusted machine(s).

It's unknown whether implementing usage of distcc and enabling it in 
`guix build` will affect bit-for-bit reproducibility of packages.

The distcc feature request and adding distcc to guix's dependency 
closure and should be discussed further on guix-devel before implementing.

Re: guix package build: can distcc be used?

[zimoun]

Hi,

On Sun, 25 Dec 2022 at 18:21, Adam Faiz <adam.faiz@disroot.org> wrote:

> It's unknown whether implementing usage of distcc and enabling it in 
> `guix build` will affect bit-for-bit reproducibility of packages.

The question is how isolated the distributed computational environments
are.

> The distcc feature request and adding distcc to guix's dependency 
> closure and should be discussed further on guix-devel before implementing.

Personally, I am missing what distcc brings compared to the current
Guix offload mechanism.  Could you (distcc user) elaborate?


Cheers,
simon

Re: guix package build: can distcc be used?

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

On 2023-01-06, zimoun wrote:
> On Sun, 25 Dec 2022 at 18:21, Adam Faiz <adam.faiz@disroot.org> wrote:
>
>> It's unknown whether implementing usage of distcc and enabling it in 
>> `guix build` will affect bit-for-bit reproducibility of packages.
>
> The question is how isolated the distributed computational environments
> are.
>
>> The distcc feature request and adding distcc to guix's dependency 
>> closure and should be discussed further on guix-devel before implementing.
>
> Personally, I am missing what distcc brings compared to the current
> Guix offload mechanism.  Could you (distcc user) elaborate?

I believe distcc would allow sending the workload of individual parts of
a single package to multiple machines, so a bit finer grained than guix
offload, which can only send a single package (or more accurately,
derivation?) to each of the offloading machines, if I understand
correctly.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: guix package build: can distcc be used?

[Simon Tournier]

Hi,

On ven., 06 janv. 2023 at 10:48, Vagrant Cascadian <vagrant@debian.org> wrote:

> I believe distcc would allow sending the workload of individual parts of
> a single package to multiple machines, so a bit finer grained than guix
> offload, which can only send a single package (or more accurately,
> derivation?) to each of the offloading machines, if I understand
> correctly.

Well, I would be interested to see charts comparing the overall
compilation time versus the number of files vs the number of remote
workers.  Especially for various network connection links between the
client and the volunteer machines versus the performance of the hardware
storage. :-)

Speaking about improving some compilation performances, it could be nice
to have a kind of optional ccache [1].  For instance,

  guix build foo --cache
  edit foo-recipe
  guix build foo --cache

would avoid to rebuild again and again the same parts.


Cheers,
simon

1: <https://ccache.dev/>