bug#41708: "guix weather" : 504 error

bug#41708: "guix weather" : 504 error

[zimoun]

Dear,

By default, "guix weather" returns:

--8<---------------cut here---------------start------------->8---
'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
--8<---------------cut here---------------end--------------->8---

which is not fatal but annoying.  Especially, it takes time.

As discussed on IRC [1], it seems that it is a bug server side.

In addition, something appears similar with Bayfront, well the 4
substitutes servers that I know returns:

--8<---------------cut here---------------start------------->8---
'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
'https://bayfront.guix.gnu.org/api/queue?nr=1000' returned 504
("Gateway Time-out")
(continuous integration information unavailable)
'https://guix.tobias.gr/api/queue?nr=1000' returned 410 ("Gone")
--8<---------------cut here---------------end--------------->8---


All the best,
simon

[1] http://logs.guix.gnu.org/guix/2020-06-03.log#201511

bug#41708: "guix weather" : 504 error

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2203 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> By default, "guix weather" returns:
>
> --8<---------------cut here---------------start------------->8---
> 'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
> --8<---------------cut here---------------end--------------->8---
>
> which is not fatal but annoying.  Especially, it takes time.
>
> As discussed on IRC [1], it seems that it is a bug server side.
>
> In addition, something appears similar with Bayfront, well the 4
> substitutes servers that I know returns:
>
> --8<---------------cut here---------------start------------->8---
> 'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
> 'https://bayfront.guix.gnu.org/api/queue?nr=1000' returned 504
> ("Gateway Time-out")
> (continuous integration information unavailable)
> 'https://guix.tobias.gr/api/queue?nr=1000' returned 410 ("Gone")
> --8<---------------cut here---------------end--------------->8---

Hey,

So I think I've got a patch [1] to Cuirass to "fix" this.

1: https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00117.html

I expected this was due to the complicated part of the db-get-builds
query, but I think it's actually down to the simple part that fetches
all the outputs for all the builds. Fetching the outputs for a build is
slow, because at the moment, there's no index on the Outputs field, so
looking up the outputs requires reading through the whole table, and the
Outputs table can be quite large.

The performance should improve with the additional query. To see why,
you can use EXPLAIN QUERY PLAN to see what SQLite plans to do when
processing the query:

sqlite> EXPLAIN QUERY PLAN SELECT name, path FROM Outputs WHERE derivation = 'foo';
QUERY PLAN
`--SCAN TABLE Outputs

sqlite> CREATE INDEX Outputs_derivation_index ON Outputs (derivation);

sqlite> EXPLAIN QUERY PLAN SELECT name, path FROM Outputs WHERE derivation = 'foo';
QUERY PLAN
`--SEARCH TABLE Outputs USING INDEX Outputs_derivation_index (derivation=?)


I believe that searching the table using an index is going to be faster
than scanning the table, and testing locally and on bayfront suggests
this will resolve the performance issue.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

bug#41708: "guix weather" : 504 error

[zimoun]

Hi Chris,

On Tue, 09 Jun 2020 at 22:12, Christopher Baines <mail@cbaines.net> wrote:

> So I think I've got a patch [1] to Cuirass to "fix" this.
>
> 1: https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00117.html

[...]

> I believe that searching the table using an index is going to be faster
> than scanning the table, and testing locally and on bayfront suggests
> this will resolve the performance issue.

Is the patch installed on Bayfront?
If yes, I still see 'returned 504 ("Gateway Time-out")' when quering
Bayfront as '--substitute-urls'.
If no, is it possible to install it? for testing it in "real life".


Cheers,
simon

ps:
It is an old request because I complained [1] about it one year ago.
Maybe something related to the summer coming. ;-)

1: https://lists.gnu.org/archive/html/help-guix/2019-05/msg00355.html

bug#41708: "guix weather" : 504 error

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> On Tue, 09 Jun 2020 at 22:12, Christopher Baines <mail@cbaines.net> wrote:
>
>> So I think I've got a patch [1] to Cuirass to "fix" this.
>>
>> 1: https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00117.html
>
> [...]
>
>> I believe that searching the table using an index is going to be faster
>> than scanning the table, and testing locally and on bayfront suggests
>> this will resolve the performance issue.
>
> Is the patch installed on Bayfront?
> If yes, I still see 'returned 504 ("Gateway Time-out")' when quering
> Bayfront as '--substitute-urls'.
> If no, is it possible to install it? for testing it in "real life".
>
>
> Cheers,
> simon
>
> ps:
> It is an old request because I complained [1] about it one year ago.
> Maybe something related to the summer coming. ;-)
>
> 1: https://lists.gnu.org/archive/html/help-guix/2019-05/msg00355.html

I've now deployed the change to Bayfront, let me know if you see a
difference? :)

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

bug#41708: "guix weather" : 504 error

[zimoun]

Hi Chris,

On Sat, 13 Jun 2020 at 17:23, Christopher Baines <mail@cbaines.net> wrote:

> I've now deployed the change to Bayfront, let me know if you see a
> difference? :)

Yes! :-)

--8<---------------cut here---------------start------------->8---
$ guix weather guix --substitute-urls='https://ci.guix.gnu.org https://bayfront.guix.gnu.org'
computing 1 package derivations for x86_64-linux...

looking for 1 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  100.0% substitutes available (1 out of 1)
  at least 106.4 MiB of nars (compressed)
  278.7 MiB on disk (uncompressed)
  0.002 seconds per request (0.0 seconds in total)
  543.8 requests per second
  'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")

looking for 1 store items on https://bayfront.guix.gnu.org...
https://bayfront.guix.gnu.org
  100.0% substitutes available (1 out of 1)
  at least 106.3 MiB of nars (compressed)
  278.6 MiB on disk (uncompressed)
  0.001 seconds per request (0.0 seconds in total)
  791.8 requests per second

  at least 1,000 queued builds
      x86_64-linux: 1000 (100.0%)
  build rate: 9.22 builds per hour
      x86_64-linux: 9.22 builds per hour
--8<---------------cut here---------------end--------------->8---

I will try a couple more of queries.  But I propose to push the patch on
Berlin too.  WDYT?


Cheers,
simon

bug#41708: "guix weather" : 504 error

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1581 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> On Sat, 13 Jun 2020 at 17:23, Christopher Baines <mail@cbaines.net> wrote:
>
>> I've now deployed the change to Bayfront, let me know if you see a
>> difference? :)
>
> Yes! :-)
>
> --8<---------------cut here---------------start------------->8---
> $ guix weather guix --substitute-urls='https://ci.guix.gnu.org https://bayfront.guix.gnu.org'
> computing 1 package derivations for x86_64-linux...
>
> looking for 1 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   100.0% substitutes available (1 out of 1)
>   at least 106.4 MiB of nars (compressed)
>   278.7 MiB on disk (uncompressed)
>   0.002 seconds per request (0.0 seconds in total)
>   543.8 requests per second
>   'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
>
> looking for 1 store items on https://bayfront.guix.gnu.org...
> https://bayfront.guix.gnu.org
>   100.0% substitutes available (1 out of 1)
>   at least 106.3 MiB of nars (compressed)
>   278.6 MiB on disk (uncompressed)
>   0.001 seconds per request (0.0 seconds in total)
>   791.8 requests per second
>
>   at least 1,000 queued builds
>       x86_64-linux: 1000 (100.0%)
>   build rate: 9.22 builds per hour
>       x86_64-linux: 9.22 builds per hour
> --8<---------------cut here---------------end--------------->8---
>
> I will try a couple more of queries.  But I propose to push the patch on
> Berlin too.  WDYT?

I believe it's been deployed now. It was working earlier today at least,
however all of Cuirass seems to be stuck now.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

bug#41708: "guix weather" : 504 error

[zimoun]

On Mon, 15 Jun 2020 at 19:43, Christopher Baines <mail@cbaines.net> wrote:

> I believe it's been deployed now. It was working earlier today at least,
> however all of Cuirass seems to be stuck now.

Well, it seems working.  Feel free to close the bug.

--8<---------------cut here---------------start------------->8---
$ guix weather 
computing 13,814 package derivations for x86_64-linux...
looking for 14,354 store items on https://ci.guix.gnu.org...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
https://ci.guix.gnu.org
  88.2% substitutes available (12,661 out of 14,354)
  at least 72,495.6 MiB of nars (compressed)
  118,865.0 MiB on disk (uncompressed)
  0.021 seconds per request (303.4 seconds in total)
  47.3 requests per second

  0.0% (0 out of 1,693) of the missing items are queued
  at least 1,000 queued builds
      i686-linux: 310 (31.0%)
      aarch64-linux: 230 (23.0%)
      armhf-linux: 313 (31.3%)
      x86_64-linux: 147 (14.7%)
  build rate: 39.53 builds per hour
      x86_64-linux: 9.53 builds per hour
      aarch64-linux: 9.57 builds per hour
      armhf-linux: 8.54 builds per hour
      i686-linux: 11.90 builds per hour
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

bug#41708: "guix weather" : 504 error

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> On Mon, 15 Jun 2020 at 19:43, Christopher Baines <mail@cbaines.net> wrote:
>
>> I believe it's been deployed now. It was working earlier today at least,
>> however all of Cuirass seems to be stuck now.
>
> Well, it seems working.  Feel free to close the bug.

Great, closing :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

bug#41708: "guix weather" : 504 error

[zimoun]

Dear Chris,

On Mon, 15 Jun 2020 at 21:23, Christopher Baines <mail@cbaines.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:
>> On Mon, 15 Jun 2020 at 19:43, Christopher Baines <mail@cbaines.net> wrote:

>>> I believe it's been deployed now. It was working earlier today at least,
>>> however all of Cuirass seems to be stuck now.
>>
>> Well, it seems working.  Feel free to close the bug.
>
> Great, closing :)

I do not know if it is only this evening, but something appears wrong:

--8<---------------cut here---------------start------------->8---
$ guix weather
computing 13,814 package derivations for x86_64-linux...
looking for 14,354 store items on https://ci.guix.gnu.org...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
https://ci.guix.gnu.org
  88.3% substitutes available (12,672 out of 14,354)
  at least 72,508.1 MiB of nars (compressed)
  118,880.3 MiB on disk (uncompressed)
  0.001 seconds per request (12.8 seconds in total)
  1,123.2 requests per second
  'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
--8<---------------cut here---------------end--------------->8---

For the record:

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 36	Jun 08 2020 00:18:47	(current)
  guix e782756
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: e78275608065ef073775fabb9f1a757da65851f2
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

bug#41708: "guix weather" : 504 error

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1348 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Dear Chris,
>
> On Mon, 15 Jun 2020 at 21:23, Christopher Baines <mail@cbaines.net> wrote:
>> zimoun <zimon.toutoune@gmail.com> writes:
>>> On Mon, 15 Jun 2020 at 19:43, Christopher Baines <mail@cbaines.net> wrote:
>
>>>> I believe it's been deployed now. It was working earlier today at least,
>>>> however all of Cuirass seems to be stuck now.
>>>
>>> Well, it seems working.  Feel free to close the bug.
>>
>> Great, closing :)
>
> I do not know if it is only this evening, but something appears wrong:
>
> --8<---------------cut here---------------start------------->8---
> $ guix weather
> computing 13,814 package derivations for x86_64-linux...
> looking for 14,354 store items on https://ci.guix.gnu.org...
> updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> https://ci.guix.gnu.org
>   88.3% substitutes available (12,672 out of 14,354)
>   at least 72,508.1 MiB of nars (compressed)
>   118,880.3 MiB on disk (uncompressed)
>   0.001 seconds per request (12.8 seconds in total)
>   1,123.2 requests per second
>   'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
> --8<---------------cut here---------------end--------------->8---

Did you check if any pages loaded for Cuirass, or was it just this API
endpoint that was timing out?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

bug#41708: "guix weather" : 504 error

[zimoun]

Hi Chris,

On Sat, 20 Jun 2020 at 10:22, Christopher Baines <mail@cbaines.net> wrote:

> > --8<---------------cut here---------------start------------->8---
> > $ guix weather
> > computing 13,814 package derivations for x86_64-linux...
> > looking for 14,354 store items on https://ci.guix.gnu.org...
> > updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> > https://ci.guix.gnu.org
> >   88.3% substitutes available (12,672 out of 14,354)
> >   at least 72,508.1 MiB of nars (compressed)
> >   118,880.3 MiB on disk (uncompressed)
> >   0.001 seconds per request (12.8 seconds in total)
> >   1,123.2 requests per second
> >   'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
> > --8<---------------cut here---------------end--------------->8---
>
> Did you check if any pages loaded for Cuirass, or was it just this API
> endpoint that was timing out?

I am not sure to understand your both questions.
What I see as an end-user is again "returned 504 ("Gateway
Time-out")".  And I do not know if something has changed on
ci.guix.gnu.org or if there is another problem.
Tell me if I can do something for helping.

Cheers,
simon

bug#41708: "guix weather" : 504 error

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> On Sat, 20 Jun 2020 at 10:22, Christopher Baines <mail@cbaines.net> wrote:
>
>> > --8<---------------cut here---------------start------------->8---
>> > $ guix weather
>> > computing 13,814 package derivations for x86_64-linux...
>> > looking for 14,354 store items on https://ci.guix.gnu.org...
>> > updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> > https://ci.guix.gnu.org
>> >   88.3% substitutes available (12,672 out of 14,354)
>> >   at least 72,508.1 MiB of nars (compressed)
>> >   118,880.3 MiB on disk (uncompressed)
>> >   0.001 seconds per request (12.8 seconds in total)
>> >   1,123.2 requests per second
>> >   'https://ci.guix.gnu.org/api/queue?nr=1000' returned 504 ("Gateway Time-out")
>> > --8<---------------cut here---------------end--------------->8---
>>
>> Did you check if any pages loaded for Cuirass, or was it just this API
>> endpoint that was timing out?
>
> I am not sure to understand your both questions.
> What I see as an end-user is again "returned 504 ("Gateway
> Time-out")".  And I do not know if something has changed on
> ci.guix.gnu.org or if there is another problem.
> Tell me if I can do something for helping.

Right now I see a response when I visit that URL.
But recently the web interface has been a bit flakey and I had to
restart it not long ago.

bug#41708: "guix weather" : 504 error

[zimoun]

Hi Ricardo,

On Wed, 24 Jun 2020 at 14:58, Ricardo Wurmus <rekado@elephly.net> wrote:

> Right now I see a response when I visit that URL.
> But recently the web interface has been a bit flakey and I had to
> restart it not long ago.

Thank you for the notification.
Now it works again as expected for me.  I have read on #guix that Ludo
restarted the Web Interface at 14:32 and just after the 504
disappeared.

Let say that now this bug is close. :-)

Cheers,
simon

bug#41708: "guix weather" : 504 error

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> On Mon, 15 Jun 2020 at 19:43, Christopher Baines <mail@cbaines.net> wrote:
>
>> I believe it's been deployed now. It was working earlier today at least,
>> however all of Cuirass seems to be stuck now.
>
> Well, it seems working.  Feel free to close the bug.
>
> $ guix weather 
> computing 13,814 package derivations for x86_64-linux...
> looking for 14,354 store items on https://ci.guix.gnu.org...
> updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> https://ci.guix.gnu.org
>   88.2% substitutes available (12,661 out of 14,354)
>   at least 72,495.6 MiB of nars (compressed)
>   118,865.0 MiB on disk (uncompressed)
>   0.021 seconds per request (303.4 seconds in total)
>   47.3 requests per second
>
>   0.0% (0 out of 1,693) of the missing items are queued
>   at least 1,000 queued builds
>       i686-linux: 310 (31.0%)
>       aarch64-linux: 230 (23.0%)
>       armhf-linux: 313 (31.3%)
>       x86_64-linux: 147 (14.7%)
>   build rate: 39.53 builds per hour
>       x86_64-linux: 9.53 builds per hour
>       aarch64-linux: 9.57 builds per hour
>       armhf-linux: 8.54 builds per hour
>       i686-linux: 11.90 builds per hour

Awesome, thank you Chris!

Ludo’.