substitute derivation: also substitute grafts?

substitute derivation: also substitute grafts?

[Ricardo Wurmus]

Hi Guix,

I really like the fact that Guix can substitute derivations.  It’s a
great way to transfer a profile from one powerful machine to a weak
machine without having to set up SSH for “guix copy” first.

I just build a profile on machine A, and then on the weak machine B I do

  guix build --substitute-urls="http://A:8000" /gnu/store/…-profile.drv

and all items for that profile are copied to B.

Did I say *all items*?  Well, … grafts are not included, because graft
derivations are marked as not substitutable.

Can we change that conditionally?  I would really like to avoid having
to build grafts on B when they have already been built on A.

-- 
Ricardo

PS: please keep me in Cc as I’m not subscribed to guix-devel at the moment.

Re: substitute derivation: also substitute grafts?

[Csepp]

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Guix,
>
> I really like the fact that Guix can substitute derivations.  It’s a
> great way to transfer a profile from one powerful machine to a weak
> machine without having to set up SSH for “guix copy” first.
>
> I just build a profile on machine A, and then on the weak machine B I
> do
>
>   guix build --substitute-urls="http://A:8000"
> /gnu/store/…-profile.drv
>
> and all items for that profile are copied to B.
>
> Did I say *all items*? Well, … grafts are not included, because graft
> derivations are marked as not substitutable.
>
> Can we change that conditionally? I would really like to avoid having
> to build grafts on B when they have already been built on A.

I would love this too, because IO can be incredibly slow on HDDs and
large packages.  My netbook would be thankful.

Re: substitute derivation: also substitute grafts?

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 1524 bytes --]



On 15-09-2022 16:46, Csepp wrote:
> 
> Ricardo Wurmus <rekado@elephly.net> writes:
> 
>> [...]
>> Did I say *all items*? Well, … grafts are not included, because graft
>> derivations are marked as not substitutable.
>>
>> Can we change that conditionally? I would really like to avoid having
>> to build grafts on B when they have already been built on A.
> 
> I would love this too, because IO can be incredibly slow on HDDs and
> large packages.  My netbook would be thankful.
>

There are some some opportunities for optimizations in the grafting code 
before substituting more -- for example, to avoid seek times, it would 
be possible to rewrite multiple files concurrently (maybe using 
'par-for-each' to process each file in a directory in parallel).

This is already done in (guix build grafts), except that:

  * 'find-files' itself is not parallelised, even though parallelising it
     could potentially reduce the time spent seeking (*)
  * it uses (parallel-job-count), which is (IIUC) 1 by default because
    --cores=1 by default.  As grafts are more IO intensive than CPU
    intensive, maybe it would be reasonable to impose a _minimum_ amount
    of parallelism, I don't, know, 2 or 4 or so?

(I'm assuming the main problem here is seek times.)

Also combinable with the proposal of having substitutes for grafts.

Greetings,
Maxime.

(*) IIUC, the time for N parallel seeks should, in theory, ≃ 1 seek, for 
small values of N, because of elevator algorithms.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: substitute derivation: also substitute grafts?

[Csepp]

Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:Undecided]]
>
>
> On 15-09-2022 16:46, Csepp wrote:
>> Ricardo Wurmus <rekado@elephly.net> writes:
>> 
>>> [...]
>>> Did I say *all items*? Well, … grafts are not included, because
>>> graft
>>> derivations are marked as not substitutable.
>>>
>>> Can we change that conditionally? I would really like to avoid
>>> having
>>> to build grafts on B when they have already been built on A.
>> I would love this too, because IO can be incredibly slow on HDDs and
>> large packages.  My netbook would be thankful.
>>
>
> There are some some opportunities for optimizations in the grafting
> code before substituting more -- for example, to avoid seek times, it
> would be possible to rewrite multiple files concurrently (maybe using
> 'par-for-each' to process each file in a directory in parallel).
>
> This is already done in (guix build grafts), except that:
>
>  * 'find-files' itself is not parallelised, even though parallelising it
>     could potentially reduce the time spent seeking (*)
>  * it uses (parallel-job-count), which is (IIUC) 1 by default because
>    --cores=1 by default.  As grafts are more IO intensive than CPU
>    intensive, maybe it would be reasonable to impose a _minimum_ amount
>    of parallelism, I don't, know, 2 or 4 or so?
>
> (I'm assuming the main problem here is seek times.)
>
> Also combinable with the proposal of having substitutes for grafts.
>
> Greetings,
> Maxime.
>
> (*) IIUC, the time for N parallel seeks should, in theory, ≃ 1 seek,
> for small values of N, because of elevator algorithms.
>
> [2. OpenPGP public key --- application/pgp-keys; OpenPGP_0x49E3EE22191725EE.asc]...
>
> [[End of PGP Signed Part]]

Could we store the offsets of references somewhere at build time?

Re: substitute derivation: also substitute grafts?

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 749 bytes --]



On 15-09-2022 19:43, Csepp wrote:
> Could we store the offsets of references somewhere at build time?

I now remember that idea, I forgot about that one.

My answer: I don't see why not, maybe by adding a phase to 
%standard-phases (at the very end, to avoid it becoming invalid) that 
saves it in, say, OUTPUT/.graft-offsets?

That would avoid grafting files that don't even have any references, 
where a copy-file (maybe using copy_file_range or such when available)
would suffice.

Fallbacks might be necessary (not every store item is constructed from a 
package), but it all sounds doable and efficient.  Also the union could 
needs to be modified to ignore the .graft-offsets of the union'ed things.

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: substitute derivation: also substitute grafts?

[Josselin Poiret]

Hi everyone,

Maxime Devos <maximedevos@telenet.be> writes:
> Fallbacks might be necessary (not every store item is constructed from a 
> package), but it all sounds doable and efficient.  Also the union could 
> needs to be modified to ignore the .graft-offsets of the union'ed things.

If I understand the whole thing properly, the daemon already scans for
references and could register their location in the database, avoiding
the need for a potentially brittle in-store format as you highlighted,
although that would require a non-trivial change to both the daemon and
the database format.

In any case, I don't think it's necessary to treat packages in a special
way: any derivation can hold derivations and thus be grafted.

Best,
-- 
Josselin Poiret

Re: substitute derivation: also substitute grafts?

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 2429 bytes --]

On 19-09-2022 18:26, Josselin Poiret wrote:
> Hi everyone,
> 
> Maxime Devos <maximedevos@telenet.be> writes:
>> Fallbacks might be necessary (not every store item is constructed from a
>> package), but it all sounds doable and efficient.  Also the union could
>> needs to be modified to ignore the .graft-offsets of the union'ed things.
> 
> If I understand the whole thing properly, the daemon already scans for
> references and could register their location in the database, avoiding
> the need for a potentially brittle in-store format as you highlighted,
> although that would require a non-trivial change to both the daemon and
> the database format.

If you do this, fallbacks are still required for old daemons.

Daemon database changes sound much more brittle to me than the simple
.graft-offsets -- the only brittleness I see is the unioning code, but 
that's easy to address.

OTOH, in the ".graft-offsets" proposal, the build process has to do 
reference scanning (even though the daemon will do so as ell later), 
which adds some inefficiency -- so perhaps performance-wise, the 
additional testing for the fallbacks and brittleness might be worth it.

> In any case, I don't think it's necessary to treat packages in a special
> way: any derivation can hold references and thus be grafted.

(Corrected a likely typo derivations->references.)

They indeed don't need to be treated specially -- in my proposal, the 
grafting code wouldn't care whether a store item is of a package or 
something else, the only thing it cares about, is the presence of a 
.graft-offsets file.

However (unless we go for daemon changes), this .graft-offsets still 
needs to be made somewhere -- my proposal was to do so this, in case of 
packages, in an additional phase (which would call a 
'make-graft-offsets' procedure '(make-graft-offsets #$output ...)'.

Then, I suppose this could be extended to non-packages as well (e.g. 
modify gexp->derivation and sexp->derivation to do the 
make-graft-offsets automatically).  However, packages should cover 
almost everything (profiles are mostly a bunch of symlinks and thus 
don't benefit from grafting optimisations much IIUC), and then 
gexp->derivation would insert an additional file without it being asked 
for, which would sound rather surprising to me, so I think that covering 
packages would be sufficient.

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: substitute derivation: also substitute grafts?

[Ricardo Wurmus]

Maxime Devos <maximedevos@telenet.be> writes:

> On 15-09-2022 16:46, Csepp wrote:
>> Ricardo Wurmus <rekado@elephly.net> writes:
>> 
>>> [...]
>>> Did I say *all items*? Well, … grafts are not included, because graft
>>> derivations are marked as not substitutable.
>>>
>>> Can we change that conditionally? I would really like to avoid having
>>> to build grafts on B when they have already been built on A.
>> I would love this too, because IO can be incredibly slow on HDDs and
>> large packages.  My netbook would be thankful.
>>
>
> There are some some opportunities for optimizations in the grafting
> code before substituting more -- for example, to avoid seek times, it
> would be possible to rewrite multiple files concurrently (maybe using
> 'par-for-each' to process each file in a directory in parallel).

While this is also an interesting development for grafts, my question is
much narrowed.  I’d be very happy if we could have a little switch that
made my daemon(s) ignore the “substitutable?” property and just
substitute everything.

I’m not looking for performance improvements in grafting, but in an
option to totally avoid doing the work twice when substituting a
derivation from a server in the same network.

-- 
Ricardo

Re: substitute derivation: also substitute grafts?

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

> While this is also an interesting development for grafts, my question is
> much narrowed.  I’d be very happy if we could have a little switch that
> made my daemon(s) ignore the “substitutable?” property and just
> substitute everything.
>
> I’m not looking for performance improvements in grafting, but in an
> option to totally avoid doing the work twice when substituting a
> derivation from a server in the same network.

The premise was that computing a graft is usually quicker than
transferring it, which is why they have #:substitutable? #f.

I don’t think it would be reasonable to have a switch to toggle
#:substitutable? because that would make .drv depend on external
factors.

However, when copying things around, you can still do: ‘guix copy
/gnu/store/…-xyz’ and that’ll work fine, whether or not it’s a graft.
But maybe that’s not a satisfying answer for your situation?

Ludo’.

Re: substitute derivation: also substitute grafts?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> The premise was that computing a graft is usually quicker than
> transferring it, which is why they have #:substitutable? #f.
>
> I don’t think it would be reasonable to have a switch to toggle
> #:substitutable? because that would make .drv depend on external
> factors.

If substitutable was merely a hint that a daemon could be free to ignore
it wouldn’t be any worse than downloading a substitute vs building
something locally.

> However, when copying things around, you can still do: ‘guix copy
> /gnu/store/…-xyz’ and that’ll work fine, whether or not it’s a graft.
> But maybe that’s not a satisfying answer for your situation?

“guix copy” only works over SSH, which requires more configuration
(sshd, key exchange, etc) and firewall exceptions.  The beauty of
derivation substitution is that it just works without any prior
configuration.

-- 
Ricardo

Re: substitute derivation: also substitute grafts?

[Tobias Geerinckx-Rice]

Quick note:

On 1 October 2022 17:29:03 UTC, Ricardo Wurmus <rekado@elephly.net> wrote:
>If substitutable was merely a hint that a daemon could be free to ignore
>it wouldn’t be any worse than downloading a substitute vs building
>something locally.

Substitutability has legal implications in Guix, whether or not it shouldn't.  Any 'suggestion' would have to be a third state.  Just something to factor into the decision.



Kind regards,

T G-R

Sent on the go.  Excuse or enjoy my brevity.

Re: substitute derivation: also substitute grafts?

[Ricardo Wurmus]

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Quick note:
>
> On 1 October 2022 17:29:03 UTC, Ricardo Wurmus <rekado@elephly.net> wrote:
>>If substitutable was merely a hint that a daemon could be free to ignore
>>it wouldn’t be any worse than downloading a substitute vs building
>>something locally.
>
> Substitutability has legal implications in Guix, whether or not it
> shouldn't.  Any 'suggestion' would have to be a third state.  Just
> something to factor into the decision.

Yes, I was about to mention that.

In my scenario I have two machines, one building stuff the other only
substituting.  The serving machine would be the one deciding whether to
enforce substitutability or not.

-- 
Ricardo

Re: substitute derivation: also substitute grafts?

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> In my scenario I have two machines, one building stuff the other only
> substituting.  The serving machine would be the one deciding whether to
> enforce substitutability or not.

An is it too expensive for that machine to build grafts locally?

Ludo’.

Re: substitute derivation: also substitute grafts?

[zimoun]

Hi,

On Wed, 05 Oct 2022 at 12:03, Ludovic Courtès <ludo@gnu.org> wrote:
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> In my scenario I have two machines, one building stuff the other only
>> substituting.  The serving machine would be the one deciding whether to
>> enforce substitutability or not.
>
> An is it too expensive for that machine to build grafts locally?

Using my old desktop from work, yes the experience is really poor;
especially for some packages where the number of grafts is sometimes
something, e.g., see [1] where grafting were longer than building.

1: <https://yhetil.org/guix/86wnud6db3.fsf@gmail.com>


Cheers,
simon

Re: substitute derivation: also substitute grafts?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> In my scenario I have two machines, one building stuff the other only
>> substituting.  The serving machine would be the one deciding whether to
>> enforce substitutability or not.
>
> An is it too expensive for that machine to build grafts locally?

It’s a matter of time.  Having prepared the environment previously on
another fast machine I’d like to avoid repeating the same work again on
a slow machine.

We needn’t restrict ourselves to substitutions — if “guix copy” could
work without first having to set up SSH that would be an equally valid
solution to this problem.

-- 
Ricardo

Re: substitute derivation: also substitute grafts?

[Mark H Weaver]

zimoun <zimon.toutoune@gmail.com> writes:

> On Wed, 05 Oct 2022 at 12:03, Ludovic Courtès <ludo@gnu.org> wrote:
>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>
>>> In my scenario I have two machines, one building stuff the other only
>>> substituting.  The serving machine would be the one deciding whether to
>>> enforce substitutability or not.
>>
>> An is it too expensive for that machine to build grafts locally?
>
> Using my old desktop from work, yes the experience is really poor;
> especially for some packages where the number of grafts is sometimes
> something, e.g., see [1] where grafting were longer than building.
>
> 1: <https://yhetil.org/guix/86wnud6db3.fsf@gmail.com>

I wonder how fast grafting could be if it were rewritten in C.

       Mark

-- 
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>.

Re: substitute derivation: also substitute grafts?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Wed, 05 Oct 2022 at 12:03, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> An is it too expensive for that machine to build grafts locally?
>
> Using my old desktop from work, yes the experience is really poor;
> especially for some packages where the number of grafts is sometimes
> something, e.g., see [1] where grafting were longer than building.
>
> 1: <https://yhetil.org/guix/86wnud6db3.fsf@gmail.com>

This has to be compared with the cost of a rebuild/redownload of the
same set of packages.  Even in the worst case, grafts are faster than
that.  Now, the difference is that those grafts need to be recomputed
regularly, where the rebuild/redownload would be relatively rare.

It’s a tradeoff.  We could have machinery to help determine when to
ungraft and perhaps even automate ungrafting.

Mark H Weaver <mhw@netris.org> skribis:

> I wonder how fast grafting could be if it were rewritten in C.

I’d expect it to be I/O-bound, so maybe not that much?

Thanks,
Ludo’.

Re: substitute derivation: also substitute grafts?

[zimoun]

Hi Ludo,

On Mon, 10 Oct 2022 at 17:32, Ludovic Courtès <ludo@gnu.org> wrote:

> This has to be compared with the cost of a rebuild/redownload of the
> same set of packages.  Even in the worst case, grafts are faster than
> that.  Now, the difference is that those grafts need to be recomputed
> regularly, where the rebuild/redownload would be relatively rare.

I have not done some stats, indeed. :-)

The build/download or rebuild/redownload is really faster (at least
for some R packages) than the graft part for some machines with
spinning disks.  Well, I have been enough annoyed for some packages on
that machine to end by systematically run the '--no-grafts' option for
all packages. :-/

Cheers,
simon

Re: substitute derivation: also substitute grafts?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Ludo,
>
> On Mon, 10 Oct 2022 at 17:32, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> This has to be compared with the cost of a rebuild/redownload of the
>> same set of packages.  Even in the worst case, grafts are faster than
>> that.  Now, the difference is that those grafts need to be recomputed
>> regularly, where the rebuild/redownload would be relatively rare.
>
> I have not done some stats, indeed. :-)
>
> The build/download or rebuild/redownload is really faster (at least
> for some R packages) than the graft part for some machines with
> spinning disks.  Well, I have been enough annoyed for some packages on
> that machine to end by systematically run the '--no-grafts' option for
> all packages. :-/

I’m not worried about grafts as such; I only think it’s a pity to have
grafts recomputed on machines on the very same network.  When I have a
direct link from one machine to another it is very fast to just copy
things over instead of performing the graft locally.

This is not something that can be decided generally.  When substituting
from a remote server it’s probably the right thing to compute the graft
locally, but there are a number of cases where that’s not a good idea
(e.g. machines on the same network, underpowered aarch64 machines
downloading substitutes from a faster machine, etc).

-- 
Ricardo