Re: Guix on aarch64

Re: Guix on aarch64

[Mark H Weaver]

Hi Ludovic and Efraim,

I think there may be a serious problem with substitutes on Aarch64.  See
below, where Benjamin Slade reports that substitutes aren't working for
him on Aarch64, although he reports having authorized berlin's key.

If I'm not mistaken, I believe I have confirmed with the test below that
a substitute for binutils from early commencement on aarch64 is not
available on berlin.

I ran "guix build -n -s aarch64-linux hello" on my x86_64 system, to
generate the derivations and see if the hashes matched what's in
Benjamin's transcript.  The derivation which failed below,

  /gnu/store/8bkfchbibkfflp72vvdf39hdbv362vm7-binutils-2.30.tar.xz.drv,

matches what I generated on my system.  I looked in the derivation file
to find the output path, and found this:

  /gnu/store/4ka60wq7j51ywy7pmpwy7fqqnzk12ay6-binutils-2.30.tar.xz

I tried to fetch this manually from berlin.guixsd.org, and it returned
404:

--8<---------------cut here---------------start------------->8---
mhw@jojen ~$ wget https://berlin.guixsd.org/guix/nar/gzip/4ka60wq7j51ywy7pmpwy7fqqnzk12ay6-binutils-2.30.tar.xz
--2018-08-23 00:26:01--  https://berlin.guixsd.org/guix/nar/gzip/4ka60wq7j51ywy7pmpwy7fqqnzk12ay6-binutils-2.30.tar.xz
Resolving berlin.guixsd.org (berlin.guixsd.org)... 141.80.181.40
Connecting to berlin.guixsd.org (berlin.guixsd.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-08-23 00:26:02 ERROR 404: Not Found.

mhw@jojen ~$ wget https://berlin.guixsd.org/guix/nar/4ka60wq7j51ywy7pmpwy7fqqnzk12ay6-binutils-2.30.tar.xz
--2018-08-23 00:26:27--  https://berlin.guixsd.org/guix/nar/4ka60wq7j51ywy7pmpwy7fqqnzk12ay6-binutils-2.30.tar.xz
Resolving berlin.guixsd.org (berlin.guixsd.org)... 141.80.181.40
Connecting to berlin.guixsd.org (berlin.guixsd.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-08-23 00:26:27 ERROR 404: Not Found.
--8<---------------cut here---------------end--------------->8---

It occurs to me that on Hydra, I have implemented a system to ensure
that *all* derivations from a certain set of _evaluations_ (the most
recent release and the last two weeks of 'master' evaluations) are
permanently kept as GC roots, regardless of how long ago they were
built.  Among other things, this ensures that even the early
commencement derivations are kept even if they were built a long time
ago.

Berlin.guixsd.org may not have any similar mechanism in place.  It may
still be using the old policy, where old GC roots are deleted based
solely on their timestamps in the filesystem, which indicate when the GC
root symlinks were first installed, long ago during the last core
updates build-out.  This could result in the early commencement
derivations and corresponding outputs being deleted prematurely.

Since Aarch64 is the only architecture that's on berlin but not on
Hydra, it's the only architecture where this problem would arise.  For
other architectures, even if berlin deletes these old derivations, Hydra
still always has a copy because of the evaluation-based GC policy I
implemented there.

What do you think?

      Mark


Benjamin Slade <beoram@gmail.com> writes:

> Though, unfortunately, when I try `guix package -i hello` (or any other
> package), I get the following error:
>
> ````
> ............
> /gnu/store/qzdabl7b02cp4xsfkz4znxpy482gsgh8-bootstrap-binaries-0/bin/tar: /gnu/store/4ka60wq7j51ywy7pmpwy7fqqnzk12ay6-binutils-2.30.tar.xz: Wrote only 4096 of 10240 bytes
> /gnu/store/qzdabl7b02cp4xsfkz4znxpy482gsgh8-bootstrap-binaries-0/bin/tar: Child died with signal 9
> /gnu/store/qzdabl7b02cp4xsfkz4znxpy482gsgh8-bootstrap-binaries-0/bin/tar: Error is not recoverable: exiting now
> source is under 'binutils-2.30'
> applying '/gnu/store/r7imyzhcnqbyzrbxygg60iwqp1jmwgrq-binutils-loongson-workaround.patch'...
> Backtrace:
> In ice-9/boot-9.scm:
>  160: 9 [catch #t #<catch-closure 9d13c0> ...]
> In unknown file:
>    ?: 8 [apply-smob/1 #<catch-closure 9d13c0>]
> In ice-9/boot-9.scm:
>   66: 7 [call-with-prompt prompt0 ...]
> In ice-9/eval.scm:
>  432: 6 [eval # #]
> In ice-9/boot-9.scm:
> 2412: 5 [save-module-excursion #<procedure a40780 at ice-9/boot-9.scm:4084:3 ()>]
> 4089: 4 [#<procedure a40780 at ice-9/boot-9.scm:4084:3 ()>]
> 1734: 3 [%start-stack load-stack #<procedure a52560 at ice-9/boot-9.scm:4080:10 ()>]
> 1739: 2 [#<procedure a54d20 ()>]
> In unknown file:
>    ?: 1 [primitive-load "/gnu/store/976r9wpcnsi9lhsg2yr8hdzkqnilyq9n-binutils-2.30.tar.xz-builder"]
> In ./guix/build/utils.scm:
>  616: 0 [invoke "/gnu/store/qzdabl7b02cp4xsfkz4znxpy482gsgh8-bootstrap-binaries-0/bin/tar" ...]
>
> ./guix/build/utils.scm:616:6: In procedure invoke:
> ./guix/build/utils.scm:616:6: Throw to key `srfi-34' with args `(#<condition &invoke-error [program: "/gnu/store/qzdabl7b02cp4xsfkz4znxpy482gsgh8-bootstrap-binaries-0/bin/tar" arguments: ("cvf" "/gnu/store/4ka60wq7j51ywy7pmpwy7fqqnzk12ay6-binutils-2.30.tar.xz" "--use-compress-program=/gnu/store/qzdabl7b02cp4xsfkz4znxpy482gsgh8-bootstrap-binaries-0/bin/xz --threads=0" "--mtime=@0" "--owner=root:0" "--group=root:0" "--sort=name" "binutils-2.30") exit-status: 2 term-signal: #f stop-signal: #f] d7ff00>)'.
> builder for `/gnu/store/8bkfchbibkfflp72vvdf39hdbv362vm7-binutils-2.30.tar.xz.drv' failed with exit code 1
> cannot build derivation `/gnu/store/6mbhl1b7kdmysilwsxk59vkws2ca9ajh-binutils-2.30.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/20q3yb3yri1g7nqng06j9kwc07ypxh1g-binutils-cross-boot0-2.30.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/sghh265k30fn0cj24rxc9p6xvn6yv3qs-perl-5.26.1.drv': 1 dependencies couldn't be built
> guix package: error: build failed: build of `/gnu/store/sghh265k30fn0cj24rxc9p6xvn6yv3qs-perl-5.26.1.drv' failed
> ````
>
>  —Ben

Re: Guix on aarch64

[Benjamin Slade]

On 2018-08-22T22:58:10-0600, Mark H Weaver <mhw@netris.org> wrote:

 > Hi Ludovic and Efraim,

 > I think there may be a serious problem with substitutes on Aarch64.  See
 > below, where Benjamin Slade reports that substitutes aren't working for
 > him on Aarch64, although he reports having authorized berlin's key.

....
 > HTTP request sent, awaiting response... 404 Not Found
 > 2018-08-23 00:26:27 ERROR 404: Not Found.
 > --8<---------------cut here---------------end--------------->8---

I also recall seeing a 404 error at some point of trying to install.

-- 
Benjamin Slade - https://babbagefiles.xyz
  `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
    '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
       `(Choose Linux ,(Choose Freedom) . https://linux.com )

Re: Guix on aarch64

[Ludovic Courtès]

Hello Mark,

Mark H Weaver <mhw@netris.org> skribis:

> If I'm not mistaken, I believe I have confirmed with the test below that
> a substitute for binutils from early commencement on aarch64 is not
> available on berlin.

[...]

> It occurs to me that on Hydra, I have implemented a system to ensure
> that *all* derivations from a certain set of _evaluations_ (the most
> recent release and the last two weeks of 'master' evaluations) are
> permanently kept as GC roots, regardless of how long ago they were
> built.  Among other things, this ensures that even the early
> commencement derivations are kept even if they were built a long time
> ago.
>
> Berlin.guixsd.org may not have any similar mechanism in place.  It may
> still be using the old policy, where old GC roots are deleted based
> solely on their timestamps in the filesystem, which indicate when the GC
> root symlinks were first installed, long ago during the last core
> updates build-out.  This could result in the early commencement
> derivations and corresponding outputs being deleted prematurely.

Correct.  Berlin uses ‘guix publish’ with a TTL of 45 days: if a nar is
not requested within 45 days, and if its corresponding store item was
GC’d in the meantime, it disappears.

I think there are two problems that made that happen: our two aarch64
build machines have been offline for several weeks so nothing new gets
built, and aarch64 substitutes are probably unpopular and thus more
likely to disappear given the above policy.

We’ll probably need a mechanism similar to what Hydra and you are doing
on hydra.gnu.org, to explicitly retain derivations from certain
evaluations; we can implement that in Cuirass.

In addition, Ricardo has a plan to throw more storage at berlin (we
currently have 1TB for the store).  That would allow us to increase the
TTL and generally worry less, though it’s no substitute for the GC root
mechanism above.

(That said, the error Benjamin reported appears to be a build failure,
which is surely worth investigating in its own right.)

Thanks,
Ludo’.

Re: Guix on aarch64

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> In addition, Ricardo has a plan to throw more storage at berlin (we
> currently have 1TB for the store).  That would allow us to increase the
> TTL and generally worry less, though it’s no substitute for the GC root
> mechanism above.

Right.  Unfortunately, this depends on me writing a multipathd service
for GuixSD so that we can use the external storage redundantly from the
head node of berlin.guixsd.org.

I haven’t been able to reserve enough time to implement this, but it’s
near the top of my list.

--
Ricardo

Re: Guix on aarch64

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> If I'm not mistaken, I believe I have confirmed with the test below that
>> a substitute for binutils from early commencement on aarch64 is not
>> available on berlin.
>
> [...]
>
>> It occurs to me that on Hydra, I have implemented a system to ensure
>> that *all* derivations from a certain set of _evaluations_ (the most
>> recent release and the last two weeks of 'master' evaluations) are
>> permanently kept as GC roots, regardless of how long ago they were
>> built.  Among other things, this ensures that even the early
>> commencement derivations are kept even if they were built a long time
>> ago.
>>
>> Berlin.guixsd.org may not have any similar mechanism in place.  It may
>> still be using the old policy, where old GC roots are deleted based
>> solely on their timestamps in the filesystem, which indicate when the GC
>> root symlinks were first installed, long ago during the last core
>> updates build-out.  This could result in the early commencement
>> derivations and corresponding outputs being deleted prematurely.
>
> Correct.  Berlin uses ‘guix publish’ with a TTL of 45 days: if a nar is
> not requested within 45 days, and if its corresponding store item was
> GC’d in the meantime, it disappears.

The 'guix publish' TTL is a secondary issue, because, as you say, the
NARs are only deleted if the corresponding store item has been GC'd.
The more important question is: what is the policy for deleting GC roots
on Berlin?

Regardless, I think we should seriously consider moving the Aarch64
build slave(s) to Hydra for now, until Cuirass is more mature.

That Aarch64 is only supported on Berlin causes several problems:

* We're unable to effectively monitor when packages become broken on
  Aarch64, due to the lack of a mature web interface.  I'm glad to see
  that progress is being made there, but it's still very far from
  sufficient for our purposes.

* When builds fail, they cannot be restarted on Berlin.  It is quite
  common for faulty test suites or other problems to cause spurious
  failures, sometimes leading to a many important dependency failures.
  This leads to missing substitutes.

* Our small number of Aarch64 build slaves makes it crucial to
  prioritize the most important builds.  On Hydra, I regularly inspect
  the build queue and prioritize builds that I deem to be important.  I
  also cancel builds when appropriate, e.g. for wip branches such as
  staging or core-updates, when a new evaluation is made, I cancel the
  outdated builds.  These interventions are quite important in practice,
  especially for the slower architectures, or architectures with
  insufficient build capacity, because we often create new builds faster
  than our build farm is able to sustain.

* As we've just discovered, substitutes for early commencement packages,
  and possibly other important packages, are missing on Aarch64 due to
  an inadequate GC policy on Berlin, which makes Aarch64 effectively
  unusable.

What do you think?

      Mark

Re: Guix on aarch64

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 646 bytes --]

On Sun, Aug 26, 2018 at 12:13:44PM -0400, Mark H Weaver wrote:
> Regardless, I think we should seriously consider moving the Aarch64
> build slave(s) to Hydra for now, until Cuirass is more mature.

I agree with your points about why berlin.guixsd.org makes it harder to
maintain the aarch64 port. I use Berlin substitutes alongside those of
Hydra but I don't actually do any build debugging with Berlin yet
because I don't know how to use the interface effectively.

Would it make any sense to split the two aarch64 machines up; one for
Hydra, one for Berlin?

It would be really great to get another pair of machines with similar
capabilities.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix on aarch64

[Ricardo Wurmus]

Hi Leo,

> On Sun, Aug 26, 2018 at 12:13:44PM -0400, Mark H Weaver wrote:
>> Regardless, I think we should seriously consider moving the Aarch64
>> build slave(s) to Hydra for now, until Cuirass is more mature.
>
> I agree with your points about why berlin.guixsd.org makes it harder to
> maintain the aarch64 port. I use Berlin substitutes alongside those of
> Hydra but I don't actually do any build debugging with Berlin yet
> because I don't know how to use the interface effectively.

I think our use of Hydra is not sustainable.  It requires regular manual
intervention by Mark, careful tuning of SQL queries, conscientious
clean up of old substitutes, and we have not a single person familiar
with the Perl code.

We do have people working on Cuirass, though.  Let’s add important
missing features to Cuirass instead of making efforts to keep Hydra on
life support.

> It would be really great to get another pair of machines with similar
> capabilities.

I agree.  We need volunteers to pick a particular configuration (my
suggestion is to start with an Overdrive 3000 rack server[1]) and find a
place to host these servers.  We have not been successful in delegating
this to other people and unfortunately the maintainers cannot take care
of this themselves.

[1]: https://shop.softiron.com/product/overdrive-3000/

--
Ricardo

Re: Guix on aarch64

[Mark H Weaver]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

>> On Sun, Aug 26, 2018 at 12:13:44PM -0400, Mark H Weaver wrote:
>>> Regardless, I think we should seriously consider moving the Aarch64
>>> build slave(s) to Hydra for now, until Cuirass is more mature.
>>
>> I agree with your points about why berlin.guixsd.org makes it harder to
>> maintain the aarch64 port. I use Berlin substitutes alongside those of
>> Hydra but I don't actually do any build debugging with Berlin yet
>> because I don't know how to use the interface effectively.
>
> I think our use of Hydra is not sustainable.  It requires regular manual
> intervention by Mark, careful tuning of SQL queries, conscientious
> clean up of old substitutes, and we have not a single person familiar
> with the Perl code.
>
> We do have people working on Cuirass, though.  Let’s add important
> missing features to Cuirass instead of making efforts to keep Hydra on
> life support.

I fail to see how moving the Aarch64 build slaves to Hydra in the short
term would constitute an "effort to keep Hydra on life support", nor how
this would hinder efforts to improve Cuirass.  I agree that we should
retire Hydra when Cuirass matures.

My proposal is not about helping Hydra, but rather about *using* Hydra,
our only sufficiently useful CI system right now, to make our Aarch64
port usable.  It currently isn't.

If anyone believes that I'm mistaken about our Aarch64 port being
unusable, then please help Benjamin Slade install Guix on his Aarch64
system.  I'd be glad to be proved wrong.

  https://lists.gnu.org/archive/html/help-guix/2018-08/msg00085.html

       Mark

Re: Guix on aarch64

[Mark H Weaver]

Hi again,

Ricardo Wurmus <rekado@elephly.net> writes:
> We do have people working on Cuirass, though.  Let’s add important
> missing features to Cuirass instead of making efforts to keep Hydra on
> life support.

Perhaps the idea is to use the fact that Aarch64 is unusable to motivate
people to work on Cuirass?

If that's the idea, then we could go further.  We could bring Hydra
offline right now, and subject all Guix users to the same problems that
Aarch64 users are experiencing today.  That ought to help motivate
people to work on Cuirass with even greater urgency.

If that's not the idea, then I'm not sure what advantage you see in
leaving Aarch64 in a bad state, until some unknown date in the future
when Cuirass becomes sufficiently mature.

       Mark

Re: Guix on aarch64

[Ricardo Wurmus]

Hi Mark,

I think you are overreacting and I really don’t think the sarcastic
response is justified.  This kind of communication is very demotivating
to me.

If this is an urgent problem (and your hostility indicates that it is)
then by all means go ahead and add one of the aarch64 build nodes to
hydra and remove the same from berlin.  Please note that I cannot take
care of this myself right now.

My statement about Hydra was merely to clarify our short-term strategy.
That’s all I have to say about this.

--
Ricardo

Re: Guix on aarch64

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

On Mon, Aug 27, 2018 at 10:04:12AM +0200, Ricardo Wurmus wrote:
> I think our use of Hydra is not sustainable.  It requires regular manual
> intervention by Mark, careful tuning of SQL queries, conscientious
> clean up of old substitutes, and we have not a single person familiar
> with the Perl code.

True, but in the meantime the Hydra web interface makes it relatively
easy to judge when a branch is ready to merge into master. Can you
recommend a workflow for doing this with Cuirass on berlin?

> I agree.  We need volunteers to pick a particular configuration (my
> suggestion is to start with an Overdrive 3000 rack server[1]) and find a
> place to host these servers.  We have not been successful in delegating
> this to other people and unfortunately the maintainers cannot take care
> of this themselves.

In my home I can host small-ish systems that don't require serious
climate control. So, not the Overdrive 3000, but perhaps the Overdrive
1000.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix on aarch64

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

>> I agree.  We need volunteers to pick a particular configuration (my
>> suggestion is to start with an Overdrive 3000 rack server[1]) and find a
>> place to host these servers.  We have not been successful in delegating
>> this to other people and unfortunately the maintainers cannot take care
>> of this themselves.
>
> In my home I can host small-ish systems that don't require serious
> climate control. So, not the Overdrive 3000, but perhaps the Overdrive
> 1000.

Thank you for the offer, Leo.  Shall we discuss the details on
guix-sysadmin?

--
Ricardo

Re: Guix on aarch64

[Mark H Weaver]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:
> I think you are overreacting and I really don’t think the sarcastic
> response is justified.  This kind of communication is very demotivating
> to me.

Believe it or not, I wasn't conscious of the fact that I was being
sarcastic or hostile.  Sorry about that.  I'm grasping at straws trying
to understand your reasoning.  In general, I _do_ think that sometimes
it's better to motivate people to work on a proper solution, by refusing
temporary workarounds.

> If this is an urgent problem (and your hostility indicates that it is)
> then by all means go ahead and add one of the aarch64 build nodes to
> hydra and remove the same from berlin.

I lack access to both Berlin and to the Aarch64 build slaves, so I can't
do this.  Note that I'm not asking for access; I actively do not want
it.

Also, Ludovic has stated multiple times that he prefers for the Aarch64
build slaves to be connected to Berlin, not Hydra, and I would not
implement a contrary policy without his consent.

       Mark

Re: Guix on aarch64

[Mark H Weaver]

Hi again,

> Ricardo Wurmus <rekado@elephly.net> writes:
>> I think you are overreacting and I really don’t think the sarcastic
>> response is justified.  This kind of communication is very demotivating
>> to me.
>
> Believe it or not, I wasn't conscious of the fact that I was being
> sarcastic or hostile.

I should clarify.  I wasn't trying to be sarcastic when I wrote this:

> Perhaps the idea is to use the fact that Aarch64 is unusable to
> motivate people to work on Cuirass?

because, as I said, I think it's sometimes justified to block a
temporary workaround to motivate implementation of a proper fix.

My suggestion that perhaps we should take Hydra offline was an attempt
to argue against the idea by reductio ad absurdum, but I can see how it
came off as sarcastic and hostile.

Anyway, I guess I'm under too much stress and hostility is leaking out.
Sorry.  I should probably take a break from communicating for a while.

    Regards,
      Mark

Re: Guix on aarch64

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>> I think you are overreacting and I really don’t think the sarcastic
>> response is justified.  This kind of communication is very demotivating
>> to me.
>
> Believe it or not, I wasn't conscious of the fact that I was being
> sarcastic or hostile.

Oh, who am I kidding?  No one but myself, I expect.

You're right.  My message was sarcastic and hostile.  I'm embarrassed to
have been in denial about that fact for any length of time.

Please accept my sincere apologies.  Thank you for your consistently
level-headed communications, even in the face of hostility.

This is not an urgent matter.  Do as you think is best.  I trust your
judgment more than my own at this point.

       Mark

Re: Guix on aarch64

[Ricardo Wurmus]

Hi Mark,

> Ricardo Wurmus <rekado@elephly.net> writes:
>> I think you are overreacting and I really don’t think the sarcastic
>> response is justified.  This kind of communication is very demotivating
>> to me.
>
> Believe it or not, I wasn't conscious of the fact that I was being
> sarcastic or hostile.  Sorry about that.  I'm grasping at straws trying
> to understand your reasoning.

When in doubt my reasoning is probably just lacking information.  I’m
spreading myself a little too thin in trying to read everything on the
mailing lists, writing responses, and trying to fix problems or recruit
people to help in fixing them.  My apologies for the confusion.

I commented specifically on Leo’s statement about build debugging on
Cuirass:

    “I don't actually do any build debugging with Berlin yet because I
     don't know how to use the interface effectively.”

This did not sound like a severe problem with Cuirass to me, but rather
a problem that could be fixed by adding minor features to Cuirass like
the display of build logs.  Leo, could you please tell us what missing
feature in the Cuirass web interface is the most urgent for you to use
the interface effectively?

Another problem that was mentioned is the garbage collection policy on
berlin.guixsd.org, which regularly removes too many built store items.
To address this I could move the store to a 37TB storage device, but I
have been having problems with the lack of multipath support at boot
time.  Obviously, I want berlin.guixsd.org to be reliable, so I’d prefer
to only attach the storage once it is available via multipath, and not
just attach it over a single HBA link.  This is a little tricky as
multipath support needs to be available early during boot.  If
/gnu/store is on a multipath device, then all packages required for
multipath support need to be available in the initrd as statically
linked executables.  I haven’t been able to make this work yet.

> I lack access to both Berlin and to the Aarch64 build slaves, so I can't
> do this.

Oh.  I can remove one of the build machines from Berlin.  But we need
input from Ludovic or Efraim (who currently both host one of the
machines) before we can verify the relocation of one of the machines.

@Ludo: could we remove the aarch64 machine at your place from
berlin.guixsd.org today and add it to Hydra?  I’m available during most
of the day to help test things.

--
Ricardo

Re: Guix on aarch64

[Andreas Enge]

Hello,

On Tue, Aug 28, 2018 at 09:57:49AM +0200, Ricardo Wurmus wrote:
> I can remove one of the build machines from Berlin.

would this not aggravate the problem with the availability of substitutes?
As I understand it, then we would have two separate machines, each of
which would build the same packages; and if two machines are not enough
to keep up with the pace of development, with only one things should become
even worse. So if we move machines, maybe we should move both of them for a
little while, until the proposed changes to cuirass and the berlin storage
architecture are implemented?

Or would it make sense to instead disable automatic evaluation of the aarch64
derivations, and to only start them manually for "important" commits?

Andreas

Re: Guix on aarch64

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> Hello,
>
> On Tue, Aug 28, 2018 at 09:57:49AM +0200, Ricardo Wurmus wrote:
>> I can remove one of the build machines from Berlin.
>
> would this not aggravate the problem with the availability of substitutes?
> As I understand it, then we would have two separate machines, each of
> which would build the same packages; and if two machines are not enough
> to keep up with the pace of development, with only one things should become
> even worse. So if we move machines, maybe we should move both of them for a
> little while, until the proposed changes to cuirass and the berlin storage
> architecture are implemented?

I agree.  I think it would be very bad to split the two machines up.  It
would effectively halve the already unsufficient build capacity on that
architecture.  I'd rather keep them both on Berlin than to split them
up.

      Mark

Re: Guix on aarch64

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1775 bytes --]

On Tue, Aug 28, 2018 at 09:57:49AM +0200, Ricardo Wurmus wrote:
> I commented specifically on Leo’s statement about build debugging on
> Cuirass:
> 
>     “I don't actually do any build debugging with Berlin yet because I
>      don't know how to use the interface effectively.”
> 
> This did not sound like a severe problem with Cuirass to me, but rather
> a problem that could be fixed by adding minor features to Cuirass like
> the display of build logs.  Leo, could you please tell us what missing
> feature in the Cuirass web interface is the most urgent for you to use
> the interface effectively?

My apologies in advance if these features already exist in the Cuirass
web interface! I didn't find them yet.

I mostly use Hydra for two things:

1) I use it to manage large branch re-builds (e.g. core-updates) by
looking at the lists of failed builds for the lastest evaluation of that
particular "jobset" (i.e. Git branch).

Hydra gives a list of tabs like this that are helpful:

Aborted jobs (2)
Newly failing jobs (2)
Newly succeeding jobs (1)
New jobs (29)
Removed jobs (23)
Still failing jobs (2425)
Still succeeding jobs (20470)
Queued jobs (675)
Inputs

I might also compare the evaluation to another evaluation of the same
jobset, or against another jobset (usually master).

The concept of "dependency failures" and their reporting is also really
helpful (i.e "foo was not built because its dependency bar failed").

Also, I use the Hydra web interface to start jobset evaluations and
restart particular package builds, and to restart all builds that were
skipped due to dependency failures.

2) Checking build logs of a particular package to see if a failure or
success is reproducible on the build farm.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guix on aarch64

[Mark H Weaver]

Leo Famulari <leo@famulari.name> writes:

> On Tue, Aug 28, 2018 at 09:57:49AM +0200, Ricardo Wurmus wrote:
>> I commented specifically on Leo’s statement about build debugging on
>> Cuirass:
>> 
>>     “I don't actually do any build debugging with Berlin yet because I
>>      don't know how to use the interface effectively.”
>> 
>> This did not sound like a severe problem with Cuirass to me, but rather
>> a problem that could be fixed by adding minor features to Cuirass like
>> the display of build logs.  Leo, could you please tell us what missing
>> feature in the Cuirass web interface is the most urgent for you to use
>> the interface effectively?
>
> My apologies in advance if these features already exist in the Cuirass
> web interface! I didn't find them yet.
>
> I mostly use Hydra for two things:
>
> 1) I use it to manage large branch re-builds (e.g. core-updates) by
> looking at the lists of failed builds for the lastest evaluation of that
> particular "jobset" (i.e. Git branch).
>
> Hydra gives a list of tabs like this that are helpful:
>
> Aborted jobs (2)
> Newly failing jobs (2)
> Newly succeeding jobs (1)
> New jobs (29)
> Removed jobs (23)
> Still failing jobs (2425)
> Still succeeding jobs (20470)
> Queued jobs (675)
> Inputs
>
> I might also compare the evaluation to another evaluation of the same
> jobset, or against another jobset (usually master).
>
> The concept of "dependency failures" and their reporting is also really
> helpful (i.e "foo was not built because its dependency bar failed").

Right, these are all indispensible tools.

> Also, I use the Hydra web interface to start jobset evaluations and
> restart particular package builds, and to restart all builds that were
> skipped due to dependency failures.

These are too, although "restart all dependency failures", which I
hastily added, is too crude of a tool in my opinion.  I've since cooked
up a PostgreSQL transaction which does something better: restart all
dependency failures that were caused by a particular failed derivation.
That should be what we aim for in Cuirass, I think.

> 2) Checking build logs of a particular package to see if a failure or
> success is reproducible on the build farm.

Yes, this too.

       Mark

Re: Guix on aarch64

[Joshua Branson]

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Leo,
>
>> On Sun, Aug 26, 2018 at 12:13:44PM -0400, Mark H Weaver wrote:
>>> Regardless, I think we should seriously consider moving the Aarch64
>>> build slave(s) to Hydra for now, until Cuirass is more mature.
>>
>> I agree with your points about why berlin.guixsd.org makes it harder to
>> maintain the aarch64 port. I use Berlin substitutes alongside those of
>> Hydra but I don't actually do any build debugging with Berlin yet
>> because I don't know how to use the interface effectively.
>
> I think our use of Hydra is not sustainable.  It requires regular manual
> intervention by Mark, careful tuning of SQL queries, conscientious
> clean up of old substitutes, and we have not a single person familiar
> with the Perl code.
>
> We do have people working on Cuirass, though.  Let’s add important
> missing features to Cuirass instead of making efforts to keep Hydra on
> life support.
>
>> It would be really great to get another pair of machines with similar
>> capabilities.
>
> I agree.  We need volunteers to pick a particular configuration (my
> suggestion is to start with an Overdrive 3000 rack server[1]) and find a
> place to host these servers.  We have not been successful in delegating
> this to other people and unfortunately the maintainers cannot take care
> of this themselves.
>
> [1]: https://shop.softiron.com/product/overdrive-3000/

Just out of curiosity, would that server run libreboot?

>
> --
> Ricardo

Re: Guix on aarch64

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> The 'guix publish' TTL is a secondary issue, because, as you say, the
> NARs are only deleted if the corresponding store item has been GC'd.
> The more important question is: what is the policy for deleting GC roots
> on Berlin?

As I wrote, there’s no policy other than “make sure there’s at least N
GiB free.”

I think the ‘guix publish’ TTL is not a secondary issue if the question
is “what does it take for substitutes to remain available?”  GC roots
allow us to retain substitutes regardless of their popularity (so it’s
definitely useful for releases), while the TTL allow us to retain
popular substitutes.

I think I already wrote that I acknowledge the issues you raised and
that we should fix Cuirass to create GC roots for evaluations when we
ask it to.

Ricardo Wurmus <rekado@elephly.net> skribis:

> I think our use of Hydra is not sustainable.  It requires regular manual
> intervention by Mark, careful tuning of SQL queries, conscientious
> clean up of old substitutes, and we have not a single person familiar
> with the Perl code.
>
> We do have people working on Cuirass, though.  Let’s add important
> missing features to Cuirass instead of making efforts to keep Hydra on
> life support.

+1

To be clear, I want us to switch to berlin as the main official
substitute server in the coming weeks.  Emacs-build-farm and the web
interface are becoming pretty nice.  There _are_ missing features and
rough edges, but I’d like us to focus on addressing those.

The Cuirass web interface code is nice and pleasant to work with, so I
encourage everyone to take a look and add whatever they deem important!

  https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/cuirass/http.scm
  https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/cuirass/templates.scm

I also think it’s important to be able to react calmly and
constructively to issues like the one Benjamin reported.  It’s not a
minor issue, but if we want to be address it in good conditions, we have
to keep cool and focus on concrete incremental steps we can take to
address it, while also keeping in mind our longer-term goals.

Thanks,
Ludo’.

Re: Guix on aarch64

[Ricardo Wurmus]

Hi Joshua,

>> I agree.  We need volunteers to pick a particular configuration (my
>> suggestion is to start with an Overdrive 3000 rack server[1]) and find a
>> place to host these servers.  We have not been successful in delegating
>> this to other people and unfortunately the maintainers cannot take care
>> of this themselves.
>>
>> [1]: https://shop.softiron.com/product/overdrive-3000/
>
> Just out of curiosity, would that server run libreboot?

No, this is an ARM server.

-- 
Ricardo