How/When will guix pull stop to compile the 616 files?

How/When will guix pull stop to compile the 616 files?

[Paul Dufresne]

I have seen it too many times already.

The first times, it was stopping because my machine hangs in the
desktop environment
if I let it without inputing stuff once in a while. My machine have
broken ACPI tables...
not your fault I suppose...

Next, after I learns to start it in tty terminal (Ctrl-Alt 2), it had
stopped at 96% because
of gc out of memory. I learn that 2 GB of memory was just too small
and had to activate
my swap partition for it to finished.

Oh, I also forgot that I learned to add the 2 exports line in my
.bashrc file because without
it it would complain guile-git is not installed, even if it is.

Well, I thought after first success, it would stop to compile it ...
but it seems to compile
it every time... or maybe if one of the 616 files changed, it have to
recompile them all?

I wonder, why it cannot recompile only the files that have changed.

Re: How/When will guix pull stop to compile the 616 files?

[Paul Dufresne]

Ok, I was able to make a guix pull without compiling the 616 files.

But I just done it again, and it seems to compile the full 616 files
each time a commit is done...
I think.

Re: How/When will guix pull stop to compile the 616 files?

[Alex Sassmannshausen]

Hi Paul

Paul Dufresne writes:

> Ok, I was able to make a guix pull without compiling the 616 files.
>
> But I just done it again, and it seems to compile the full 616 files
> each time a commit is done...
> I think.

This is right — guix pull will download (now via git) the latest
available snapshot.  If that snapshot is identical to your current
snapshot, it will not compile anything, otherwise it will do a full
installation, which involves compiling all project files.

New snapshots are indeed created by people commiting to master.

Now that guix pull uses a git backend, it might be possible to move
towards a model whereby pull only recompiles files affected by the
missing commits (that would be leaves that have been changed, or the
file thas has been changed and all its dependents).  But that's me
speculating, not volunteering unfortunately…

HTH,

Alex

Re: How/When will guix pull stop to compile the 616 files?

[Paul Dufresne]

I try to get to the way/vocabulary you use.

It takes 45 mins. on my relatively old dual core to guix-pull (would
say 95% of the time
compiling).

The way I begin to see it, the problem seems that guix-system itself seems to be
considered as latest-drv derivation. I guess I would have prefer that
they would become
616 scheme (all?) derivations.

I believe they would be  independant in the sense that compiling
source code to virtual machine does not seems to have dependencies
between the files.

Sadly, I am a lot more of a thinker than a doer.

Re: How/When will guix pull stop to compile the 616 files?

[Ricardo Wurmus]

Paul Dufresne <dufresnep@gmail.com> writes:

> It takes 45 mins. on my relatively old dual core to guix-pull

Yeah, this isn’t great.  Since 0.13.0 compilation is slower and requires
a whole lot more memory.  That’s a known problem.

There are some ideas to reduce the amount of compilation that has to
happen locally, but it isn’t quite as simple as a first look may
suggest.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: How/When will guix pull stop to compile the 616 files?

[Jack Hill]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 684 bytes --]

On Tue, 8 Aug 2017, Ricardo Wurmus wrote:

>
> Paul Dufresne <dufresnep@gmail.com> writes:
>
>> It takes 45 mins. on my relatively old dual core to guix-pull
>
> Yeah, this isn’t great.  Since 0.13.0 compilation is slower and requires
> a whole lot more memory.  That’s a known problem.
>
> There are some ideas to reduce the amount of compilation that has to
> happen locally, but it isn’t quite as simple as a first look may
> suggest.

Could building be avoided entirely with substitutes? Are substitutes not 
appropriate for some reason, or is it just that substitutes are not 
produced on Hydra for every guix commit?

Thanks for clearing this up for a Guix beginner.

Jack

Re: How/When will guix pull stop to compile the 616 files?

[Paul Dufresne]

Jack asked:
> Could building be avoided entirely with substitutes? Are substitutes not
> appropriate for some reason, or is it just that substitutes are not
> produced on Hydra for every guix commit?
Now, I feel dumb for not having thought to ask that question!

Re: How/When will guix pull stop to compile the 616 files?

[Ricardo Wurmus]

Jack Hill <jackhill@jackhill.us> writes:

> On Tue, 8 Aug 2017, Ricardo Wurmus wrote:
>
>>
>> Paul Dufresne <dufresnep@gmail.com> writes:
>>
>>> It takes 45 mins. on my relatively old dual core to guix-pull
>>
>> Yeah, this isn’t great.  Since 0.13.0 compilation is slower and requires
>> a whole lot more memory.  That’s a known problem.
>>
>> There are some ideas to reduce the amount of compilation that has to
>> happen locally, but it isn’t quite as simple as a first look may
>> suggest.
>
> Could building be avoided entirely with substitutes? Are substitutes not
> appropriate for some reason, or is it just that substitutes are not
> produced on Hydra for every guix commit?

Before a substitute can be requested, Guix will have to compute a
derivation locally.  In the case of Guix itself this is rather
expensive.

Once we have the derivation we can ask substitute servers if they have a
binary substitute for performing the work the derivation describes.

So, local *computation* cannot be avoided, but local building should be
avoidable for most packages — here the problem is that hydra isn’t fast
enough yet.  I’ve been preparing an alternative build farm at the
institute where I work, which will hopefully soon be powerful enough to
build packages more quickly than our current Hydra does.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: How/When will guix pull stop to compile the 616 files?

[Jack Hill]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 702 bytes --]

On Wed, 9 Aug 2017, Ricardo Wurmus wrote:

> Before a substitute can be requested, Guix will have to compute a
> derivation locally.  In the case of Guix itself this is rather
> expensive.
>
> Once we have the derivation we can ask substitute servers if they have a
> binary substitute for performing the work the derivation describes.
>
> So, local *computation* cannot be avoided, but local building should be
> avoidable for most packages — here the problem is that hydra isn’t fast
> enough yet.  I’ve been preparing an alternative build farm at the
> institute where I work, which will hopefully soon be powerful enough to
> build packages more quickly than our current Hydra does.

Thanks!!

Re: How/When will guix pull stop to compile the 616 files?

[Quiliro Ordonez Baca]

Paul Dufresne <dufresnep@gmail.com> writes:

> It takes 45 mins. on my relatively old dual core to guix-pull (would
> say 95% of the time
> compiling).

I have the same machine you do. I have opted to stop xorg and enter
tty1. Then, when the download is finished, I stop networking in order to
make use of the least resources. I have a lightweight-desktop.scm
configuration, so I probably have a minimum of services started
also. These are the steps I take to stop those services I said before:
sudo herd stop xorg-service
sudo herd stop networking
If I later want to restart them, I execute the same commands but
replacing "stop" with "restart".

I hope this helps complete compilation when needed. :-)