all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / Atom feed
* bug#30290: guix-daemon slows to a crawl when a substitute server is offline
@ 2018-01-30  3:07 Maxim Cournoyer
  2020-12-03  0:20 ` zimoun
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Maxim Cournoyer @ 2018-01-30  3:07 UTC (permalink / raw)
  To: 30290

When a substitute server used by guix-daemon is offline, the daemon will
keep attempting to connect to it, even when it shouldn't need any data
(ran 'sudo guix system reconfigure my-config.scm' multiple times in a
row.

With the disconnected server (bayfront in my case), that command would
take close to 8 minutes, with many system calls like:

--8<---------------cut here---------------start------------->8---
connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
--8<---------------cut here---------------end--------------->8---

which wasted 5 seconds each time.

After removing this server from my substitute servers list, the same
operation (system reconfigure) is 8 times faster (1 minute).

Suggestion: the daemon should stop trying to use the offline substitute
server after trying for X times, and print a warning about it.

Maxim

^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#30290: guix-daemon slows to a crawl when a substitute server is offline
  2018-01-30  3:07 bug#30290: guix-daemon slows to a crawl when a substitute server is offline Maxim Cournoyer
@ 2020-12-03  0:20 ` zimoun
  2020-12-19  3:04   ` Maxim Cournoyer
  2020-12-22 15:16 ` Ludovic Courtès
  2021-03-24 21:55 ` zimoun
  2 siblings, 1 reply; 7+ messages in thread
From: zimoun @ 2020-12-03  0:20 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: 30290

Hi Maxim,

On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
> When a substitute server used by guix-daemon is offline, the daemon will
> keep attempting to connect to it, even when it shouldn't need any data
> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
> row.
>
> With the disconnected server (bayfront in my case), that command would
> take close to 8 minutes, with many system calls like:
>
> connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>
> which wasted 5 seconds each time.
>
> After removing this server from my substitute servers list, the same
> operation (system reconfigure) is 8 times faster (1 minute).
>
> Suggestion: the daemon should stop trying to use the offline substitute
> server after trying for X times, and print a warning about it.

This looks like as a wishlist, right?  Do it make sense to include such
feature to the recent discussions about the revamp of offloading,
Cuirass, publish, etc.


All the best,
simon




^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#30290: guix-daemon slows to a crawl when a substitute server is offline
  2020-12-03  0:20 ` zimoun
@ 2020-12-19  3:04   ` Maxim Cournoyer
  0 siblings, 0 replies; 7+ messages in thread
From: Maxim Cournoyer @ 2020-12-19  3:04 UTC (permalink / raw)
  To: zimoun; +Cc: 30290

Hi!

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>> When a substitute server used by guix-daemon is offline, the daemon will
>> keep attempting to connect to it, even when it shouldn't need any data
>> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
>> row.
>>
>> With the disconnected server (bayfront in my case), that command would
>> take close to 8 minutes, with many system calls like:
>>
>> connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>>
>> which wasted 5 seconds each time.
>>
>> After removing this server from my substitute servers list, the same
>> operation (system reconfigure) is 8 times faster (1 minute).
>>
>> Suggestion: the daemon should stop trying to use the offline substitute
>> server after trying for X times, and print a warning about it.
>
> This looks like as a wishlist, right?  Do it make sense to include such
> feature to the recent discussions about the revamp of offloading,
> Cuirass, publish, etc.

To me it's an issue more than a feature request, especially in a build
farm setting; having a substitute machine down shouldn't cause a slow
down for as long as it's down!

I'm not sure if the recent offloading work that Mathieu did touched that
topic.  I'd need to test the scenario.  Perhaps a system test would be
useful.

Maxim




^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#30290: guix-daemon slows to a crawl when a substitute server is offline
  2018-01-30  3:07 bug#30290: guix-daemon slows to a crawl when a substitute server is offline Maxim Cournoyer
  2020-12-03  0:20 ` zimoun
@ 2020-12-22 15:16 ` Ludovic Courtès
  2020-12-28 12:19   ` Efraim Flashner
  2021-03-24 21:55 ` zimoun
  2 siblings, 1 reply; 7+ messages in thread
From: Ludovic Courtès @ 2020-12-22 15:16 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: 30290

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> When a substitute server used by guix-daemon is offline, the daemon will
> keep attempting to connect to it, even when it shouldn't need any data
> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
> row.
>
> With the disconnected server (bayfront in my case), that command would
> take close to 8 minutes, with many system calls like:
>
> connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>
> which wasted 5 seconds each time.

Is it still a problem?  Commit 4f5234be0378368e6af25925db46612838d25e58
(Nov. 2019) added a table of unreachable hosts.  That way, a ‘guix
substitute --query’ process won’t retry connections to an unreachable
host.

Ludo’.




^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#30290: guix-daemon slows to a crawl when a substitute server is offline
  2020-12-22 15:16 ` Ludovic Courtès
@ 2020-12-28 12:19   ` Efraim Flashner
  0 siblings, 0 replies; 7+ messages in thread
From: Efraim Flashner @ 2020-12-28 12:19 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 30290, Maxim Cournoyer

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On Tue, Dec 22, 2020 at 04:16:08PM +0100, Ludovic Courtès wrote:
> Hi,
> 
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
> 
> > When a substitute server used by guix-daemon is offline, the daemon will
> > keep attempting to connect to it, even when it shouldn't need any data
> > (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
> > row.
> >
> > With the disconnected server (bayfront in my case), that command would
> > take close to 8 minutes, with many system calls like:
> >
> > connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
> >
> > which wasted 5 seconds each time.
> 
> Is it still a problem?  Commit 4f5234be0378368e6af25925db46612838d25e58
> (Nov. 2019) added a table of unreachable hosts.  That way, a ‘guix
> substitute --query’ process won’t retry connections to an unreachable
> host.
> 
> Ludo’.
> 

Occasionally my internet drops itself, and I find I'm left forever
waiting for a timeout to see what sources I have cached locally.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#30290: guix-daemon slows to a crawl when a substitute server is offline
  2018-01-30  3:07 bug#30290: guix-daemon slows to a crawl when a substitute server is offline Maxim Cournoyer
  2020-12-03  0:20 ` zimoun
  2020-12-22 15:16 ` Ludovic Courtès
@ 2021-03-24 21:55 ` zimoun
  2021-06-09 21:34   ` zimoun
  2 siblings, 1 reply; 7+ messages in thread
From: zimoun @ 2021-03-24 21:55 UTC (permalink / raw)
  To: Maxim Cournoyer, Mathieu Othacehe; +Cc: 30290

Hi,

On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
> When a substitute server used by guix-daemon is offline, the daemon will
> keep attempting to connect to it, even when it shouldn't need any data
> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
> row.
>
> With the disconnected server (bayfront in my case), that command would
> take close to 8 minutes, with many system calls like:
>
> connect(14, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>
> which wasted 5 seconds each time.
>
> After removing this server from my substitute servers list, the same
> operation (system reconfigure) is 8 times faster (1 minute).
>
> Suggestion: the daemon should stop trying to use the offline substitute
> server after trying for X times, and print a warning about it.

What is the status of this bug?  Especially with the recent additions in
Cuirass?

Is it still an issue?  Is some timeout still happening?

Well, in summary, the 3 relevant messages are:

-------------------- Start of forwarded message --------------------
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 18 Dec 2020 22:04:04 -0500

I'm not sure if the recent offloading work that Mathieu did touched that
topic.  I'd need to test the scenario.  Perhaps a system test would be
useful.
-------------------- End of forwarded message --------------------

-------------------- Start of forwarded message --------------------
From: Ludovic Courtès <ludo@gnu.org>
Date: Tue, 22 Dec 2020 16:16:08 +0100

Is it still a problem?  Commit 4f5234be0378368e6af25925db46612838d25e58
(Nov. 2019) added a table of unreachable hosts.  That way, a ‘guix
substitute --query’ process won’t retry connections to an unreachable
host.
-------------------- End of forwarded message --------------------

-------------------- Start of forwarded message --------------------
Date: Mon, 28 Dec 2020 14:19:02 +0200
From: Efraim Flashner <efraim@flashner.co.il>

Occasionally my internet drops itself, and I find I'm left forever
waiting for a timeout to see what sources I have cached locally.
-------------------- End of forwarded message --------------------


Cheers,
simon




^ permalink raw reply	[flat|nested] 7+ messages in thread

* bug#30290: guix-daemon slows to a crawl when a substitute server is offline
  2021-03-24 21:55 ` zimoun
@ 2021-06-09 21:34   ` zimoun
  0 siblings, 0 replies; 7+ messages in thread
From: zimoun @ 2021-06-09 21:34 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: Mathieu Othacehe, 30290

Hi,


On Wed, 24 Mar 2021 at 22:55, zimoun <zimon.toutoune@gmail.com> wrote:
> On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>> When a substitute server used by guix-daemon is offline, the daemon will
>> keep attempting to connect to it, even when it shouldn't need any data
>> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
>> row.
>>
>> With the disconnected server (bayfront in my case), that command would
>> take close to 8 minutes, with many system calls like:
>>
>> connect(14, {sa_family=AF_INET, sin_port=htons(443),
>> sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>>
>> which wasted 5 seconds each time.
>>
>> After removing this server from my substitute servers list, the same
>> operation (system reconfigure) is 8 times faster (1 minute).
>>
>> Suggestion: the daemon should stop trying to use the offline substitute
>> server after trying for X times, and print a warning about it.

[...]

> From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> Date: Fri, 18 Dec 2020 22:04:04 -0500 (24 weeks, 4 days, 18 hours ago)
>
> I'm not sure if the recent offloading work that Mathieu did touched that
> topic.  I'd need to test the scenario.  Perhaps a system test would be
> useful.
> ----------
>
> From: Ludovic Courtès <ludo@gnu.org>
> Date: Tue, 22 Dec 2020 16:16:08 +0100
> Date: Tue, 22 Dec 2020 16:16:08 +0100 (24 weeks, 1 day, 6 hours ago)
>
> Is it still a problem?  Commit 4f5234be0378368e6af25925db46612838d25e58
> (Nov. 2019) added a table of unreachable hosts.  That way, a ‘guix
> substitute --query’ process won’t retry connections to an unreachable
> host.
> ----------
>
> From: Efraim Flashner <efraim@flashner.co.il>
> Date: Mon, 28 Dec 2020 14:19:02 +0200
> Date: Mon, 28 Dec 2020 14:19:02 +0200 (23 weeks, 2 days, 9 hours ago)
>
> Occasionally my internet drops itself, and I find I'm left forever
> waiting for a timeout to see what sources I have cached locally.
> ----------

What is the current stats of this bug?   Is it still happening with the
recent improvements of Cuirass?

Cheers,
simon




^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-06-09 21:44 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-30  3:07 bug#30290: guix-daemon slows to a crawl when a substitute server is offline Maxim Cournoyer
2020-12-03  0:20 ` zimoun
2020-12-19  3:04   ` Maxim Cournoyer
2020-12-22 15:16 ` Ludovic Courtès
2020-12-28 12:19   ` Efraim Flashner
2021-03-24 21:55 ` zimoun
2021-06-09 21:34   ` zimoun

all messages for Guix-related lists mirrored at yhetil.org

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix

Example config snippet for mirrors.


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git