hydra.gnu.org off-line for maintenance

hydra.gnu.org off-line for maintenance

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

Hello Guix,

The hydra.gnu.org build farm has been off-line since yesterday ~4PM UTC
and will remain off-line roughly until the end of the week.  The FSF
sysadmins (it’s hosted at the FSF’s data center) are performing a major
storage upgrade that happens to need more time than expected.

In the meantime, if you use mirror.hydra.gnu.org, you’ll still get
substitutes, though not the latest one.

If you use berlin.guixsd.org, everything is fine; if you don’t, now is
the time to add it to your substitute URLs!  See
<https://gnu.org/software/guix/manual/en/html_node/Substitutes.html>.

Apologies for the inconvenience!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hydra.gnu.org off-line for maintenance

[Luther Thompson]

On Wed, 24 Oct 2018 11:57:15 +0200
ludo@gnu.org (Ludovic Courtès) wrote:
> If you use berlin.guixsd.org, everything is fine; if you don’t, now is
> the time to add it to your substitute URLs!  See
> <https://gnu.org/software/guix/manual/en/html_node/Substitutes.html>.

How do we do that? I followed the link, but I can't find the info to
pass to `guix archive --authorize` anywhere.

Luther

Re: hydra.gnu.org off-line for maintenance

[Jonathan Brielmaier]

On 10/26/18 12:41 AM, Luther Thompson wrote:
> On Wed, 24 Oct 2018 11:57:15 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>> If you use berlin.guixsd.org, everything is fine; if you don’t, now is
>> the time to add it to your substitute URLs!  See
>> <https://gnu.org/software/guix/manual/en/html_node/Substitutes.html>.
> 
> How do we do that? I followed the link, but I can't find the info to
> pass to `guix archive --authorize` anywhere.

It's described here:
https://www.gnu.org/software/guix/manual/en/guix.html#Substitute-Server-Authorization

So basically `guix archive --authorize < berlin.guixsd.org.pub`

and then run your commands with:
`--substitute-urls="https://berlin.guixsd.org`

I hope that helps :)

Re: hydra.gnu.org off-line for maintenance

[Thomas Danckaert]

From: Jonathan Brielmaier <jonathan.brielmaier@web.de>
Subject: Re: hydra.gnu.org off-line for maintenance
Date: Fri, 26 Oct 2018 08:01:11 +0200

> It's described here:
> https://www.gnu.org/software/guix/manual/en/guix.html#Substitute-Server-Authorization
>
> So basically `guix archive --authorize < berlin.guixsd.org.pub`
>
> and then run your commands with:
> `--substitute-urls="https://berlin.guixsd.org`

Also, you may want to add this setting globally and permanently, so 
you don't have to specify it every time you run guix.  You can do 
this by passing the option to the guix daemon

  - when using guix on another distribution with systemd, you can 
modify the the ExecStart line in 
/etc/systemd/system/guix-daemon.service as follows:

ExecStart=/var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon 
--build-users-group=guixbuild 
"--substitute-urls=https://mirror.hydra.gnu.org 
https://berlin.guixsd.org"

  - when using GuixSD, modify guix-service in your system 
configuration, by specifying the substitute-urls in the 
configuration. (see guix-configuration in 
https://www.gnu.org/software/guix/manual/en/html_node/Base-Services.html 
)

By the way: when revisiting this issue, I found the manual a bit 
confusing.  You'll find all the information you need when you read 
the entire section on substitutes from front to back, but when you 
just try to find specific information using the headlines, it's a bit 
confusing.  For example, to learn about --substitute-urls and the 
daemon settings for other substitute servers, you'll need to read 
"Official Substitute Server", which you might not do when looking to 
add another server.  The subtitle for "Substitute Server 
Authorization" says "How to enable or disable substitutes", but 
doesn't give all the information you need to enable another 
substitute server.

If people here agree, I'd be happy to send a documentation patch.

Sincerely,

Thomas

Re: hydra.gnu.org off-line for maintenance

[Thorsten Wilms]

On 26/10/2018 11.28, Thomas Danckaert wrote:

> Also, you may want to add this setting globally and permanently, so you 
> don't have to specify it every time you run guix.  You can do this by 
> passing the option to the guix daemon
> 
>   - when using guix on another distribution with systemd, you can modify 
> the the ExecStart line in /etc/systemd/system/guix-daemon.service as 
> follows:
> 
> ExecStart=/var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon 
> --build-users-group=guixbuild 
> "--substitute-urls=https://mirror.hydra.gnu.org https://berlin.guixsd.org"

Shouldn't that be:
---
ExecStart=/var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon 
--build-users-group=guixbuild 
--substitute-urls="https://mirror.hydra.gnu.org https://hydra.gnu.org 
https://berlin.guixsd.org"
---
?

> By the way: when revisiting this issue, I found the manual a bit 
> confusing.  You'll find all the information you need when you read the 
> entire section on substitutes from front to back, but when you just try 
> to find specific information using the headlines, it's a bit confusing.  
> For example, to learn about --substitute-urls and the daemon settings 
> for other substitute servers, you'll need to read "Official Substitute 
> Server", which you might not do when looking to add another server.  The 
> subtitle for "Substitute Server Authorization" says "How to enable or 
> disable substitutes", but doesn't give all the information you need to 
> enable another substitute server.
> 
> If people here agree, I'd be happy to send a documentation patch.

For whatever it may be worth, when I first read about 
https://berlin.guixsd.org, I wondered if it could/should be used 
additionally, looked at the documentation, found it confusing and then 
forgot about the issue.


-- 
Thorsten Wilms

thorwil's design for free software:
http://thorwil.wordpress.com/

Re: hydra.gnu.org off-line for maintenance

[Clément Lassieur]

Hi,

Thomas Danckaert <post@thomasdanckaert.be> writes:

> Also, you may want to add this setting globally and permanently, so you don't
> have to specify it every time you run guix.  You can do this by passing the
> option to the guix daemon
>
>  - when using guix on another distribution with systemd, you can modify the
> the ExecStart line in /etc/systemd/system/guix-daemon.service as follows:
>
> ExecStart=/var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon
> --build-users-group=guixbuild "--substitute-urls=https://mirror.hydra.gnu.org
> https://berlin.guixsd.org"

Note that https://mirror.hydra.gnu.org doesn't work well either[1].  You
might be better off with just https://berlin.guixsd.org.

But I noticed today that https://berlin.guixsd.org doesn't evaluate
anything anymore[2], so you will only get old evaluations from it.

At the moment I can't find any working solution.

[1]: https://bugs.gnu.org/33151
[2]: https://berlin.guixsd.org/jobset/guix-master

Cheers,
Clément

Re: hydra.gnu.org off-line for maintenance

[Tobias Geerinckx-Rice]

Thorsten,

Thorsten Wilms wrote:
> On 26/10/2018 11.28, Thomas Danckaert wrote:
>> ExecStart=/var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon
>> --build-users-group=guixbuild
>> "--substitute-urls=https://mirror.hydra.gnu.org
>> https://berlin.guixsd.org"
>
> Shouldn't that be:
> ---
> ExecStart=/var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon
> --build-users-group=guixbuild
> --substitute-urls="https://mirror.hydra.gnu.org 
> https://hydra.gnu.org
> https://berlin.guixsd.org"
> ---
> ?

Slow ol' hydra.gnu.org shouldn't be used directly like this, even 
though there's currently nothing stopping you from doing so and I 
suspect many do.

The mirror should suffice.

Kind regards,

T G-R

Re: hydra.gnu.org off-line for maintenance

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

Hello,

For myself, I use the following option:

  --substitute-urls='https://berlin.guixsd.org https://mirror.guixsd.org https://mirror.hydra.gnu.org https://hydra.gnu.org'

I choose 'berlin.guixsd.org' to be the default because it seems to build
substitutes faster than 'hydra.gnu.org'.

Cheers,
Alex

Clément Lassieur <clement@lassieur.org> writes:

> Hi,
>
> Thomas Danckaert <post@thomasdanckaert.be> writes:
>
>> Also, you may want to add this setting globally and permanently, so you don't
>> have to specify it every time you run guix.  You can do this by passing the
>> option to the guix daemon
>>
>>  - when using guix on another distribution with systemd, you can modify the
>> the ExecStart line in /etc/systemd/system/guix-daemon.service as follows:
>>
>> ExecStart=/var/guix/profiles/per-user/root/guix-profile/bin/guix-daemon
>> --build-users-group=guixbuild "--substitute-urls=https://mirror.hydra.gnu.org
>> https://berlin.guixsd.org"
>
> Note that https://mirror.hydra.gnu.org doesn't work well either[1].  You
> might be better off with just https://berlin.guixsd.org.
>
> But I noticed today that https://berlin.guixsd.org doesn't evaluate
> anything anymore[2], so you will only get old evaluations from it.
>
> At the moment I can't find any working solution.
>
> [1]: https://bugs.gnu.org/33151
> [2]: https://berlin.guixsd.org/jobset/guix-master
>
> Cheers,
> Clément

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Evaluations on berlin.guixsd.org

[Ludovic Courtès]

Hello,

Clément Lassieur <clement@lassieur.org> skribis:

> But I noticed today that https://berlin.guixsd.org doesn't evaluate
> anything anymore[2], so you will only get old evaluations from it.

I believe this is fixed by fdf02484289e155de56e7320f7e1dd1e57431f4b.

Everyone: please make sure to run ‘guix lint’ before pushing, especially
when pushing patch series where it’s easy to overlook something.

The issue wasn’t easy to find in the middle of the Cuirass build log,
and it’s also problematic that Cuirass doesn’t distinguish between
failed evaluations and evaluations in progress.  We should look into
that sometime.  :-)

Thanks for the heads-up, Clément!

Ludo’.

Re: hydra.gnu.org off-line for maintenance

[Tobias Geerinckx-Rice]

Alex,

Alex Vong wrote:
> For myself, I use the following option:
>
>   --substitute-urls='https://berlin.guixsd.org 
>   https://mirror.guixsd.org https://mirror.hydra.gnu.org 
>   https://hydra.gnu.org'

I suggest keeping only one of mirror.guixsd.org and 
mirror.hydra.gnu.org.

I don't know the story, or which one's preferable (I forgot m.g.o 
existed), but both point to the exact same server. Hitting (and 
caching?) the same box twice is probably bad news for both of you.

> I choose 'berlin.guixsd.org' to be the default because it seems 
> to build
> substitutes faster than 'hydra.gnu.org'.

I share that impression.

Kind regards,

T G-R

Re: hydra.gnu.org off-line for maintenance

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Alex,
>
> Alex Vong wrote:
>> For myself, I use the following option:
>>
>>   --substitute-urls='https://berlin.guixsd.org
>> https://mirror.guixsd.org https://mirror.hydra.gnu.org
>> https://hydra.gnu.org'
>
> I suggest keeping only one of mirror.guixsd.org and
> mirror.hydra.gnu.org.
>
> I don't know the story, or which one's preferable (I forgot m.g.o
> existed), but both point to the exact same server. Hitting (and
> caching?) the same box twice is probably bad news for both of you.
>
Thanks for the catch! 'mirror.guixsd.org' was mentioned in one of the
threads. (of course, I forget which thread it was...) I don't know the 2
urls resolve same server.

>> I choose 'berlin.guixsd.org' to be the default because it seems to
>> build
>> substitutes faster than 'hydra.gnu.org'.
>
> I share that impression.
>
> Kind regards,
>
> T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: hydra.gnu.org off-line for maintenance

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

Hello Guix!

ludo@gnu.org (Ludovic Courtès) skribis:

> The hydra.gnu.org build farm has been off-line since yesterday ~4PM UTC
> and will remain off-line roughly until the end of the week.  The FSF
> sysadmins (it’s hosted at the FSF’s data center) are performing a major
> storage upgrade that happens to need more time than expected.
>
> In the meantime, if you use mirror.hydra.gnu.org, you’ll still get
> substitutes, though not the latest one.
>
> If you use berlin.guixsd.org, everything is fine; if you don’t, now is
> the time to add it to your substitute URLs!  See
> <https://gnu.org/software/guix/manual/en/html_node/Substitutes.html>.

For the record, the maintenance of hydra.gnu.org is still on-going.
From what the FSF sysadmin told us, it could last until Tuesday,
Nov. 6th.

I strongly recommend switching to berlin.guixsd.org as your primary
source of substitutes.

Apologies for the inconvenience!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: hydra.gnu.org off-line for maintenance

[Pjotr Prins]

On Thu, Nov 01, 2018 at 04:07:56PM +0100, Ludovic Courtès wrote:
> For the record, the maintenance of hydra.gnu.org is still on-going.
> From what the FSF sysadmin told us, it could last until Tuesday,
> Nov. 6th.
> 
> I strongly recommend switching to berlin.guixsd.org as your primary
> source of substitutes.

Can't we build a standard fallback into the default install?

Pj.

Re: hydra.gnu.org off-line for maintenance

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

Hello,

I’m happy to announce that hydra.gnu.org is now back up, thanks to
Mark H Weaver who made sure all the services were up and running
yesterday.

For the record, the outage was initiated by the FSF sysadmins to change
disks in the machine’s disk array (hydra.gnu.org is hosted by the FSF).
They initially thought this would take much less time.

Apologies for the inconvenience!

From there on we must see on our side what we can do to improve the
resilience of our main substitute servers, should the head node of the
build farm fail.

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]