From: "Ludovic Courtès" <ludo@gnu.org>
To: Mathieu Othacehe <othacehe@gnu.org>
Cc: 48468@debbugs.gnu.org
Subject: bug#48468: substitute server connection timeout
Date: Thu, 08 Dec 2022 11:26:28 +0100 [thread overview]
Message-ID: <87wn72fay3.fsf@gnu.org> (raw)
In-Reply-To: <878rjj8eun.fsf@gnu.org> (Mathieu Othacehe's message of "Wed, 07 Dec 2022 15:31:44 +0100")
Hi,
Mathieu Othacehe <othacehe@gnu.org> skribis:
>> /gnu/store/qmzr030rzgikdxv3g9msqv0l8qp5j6y2-btrfs-raid-root-os.drv,
>> which was marked as failed earlier today due to missing .drv. It’s a
>> 4KiB file, and the cache-bypass-threshold is ‘guix publish’ is typically
>> set to something much higher than that. So ‘guix publish’ won’t return
>> 404 in that case.
>
> Yes but that derivation also depends on other derivations, for instance
> qemu-minimal and if I try:
>
> mathieu@berlin ~$ guix build qemu-minimal
> ...
> /gnu/store/lwv2pl0m6dkf6bkzip755w5p71g5akq4-qemu-minimal-7.1.0
>
>
> and then, from my machine.
>
> curl https://ci.guix.gnu.org/lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo
> We're baking it: /lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo
Yes, but derivations (.drv) are depend only on “sources” (like
*-guile-builder, *.patch, *.scm) and on other derivations, all of which
are typically less than 1 MiB.
Derivations don’t depend on derivation outputs like that of qemu-minimal
above.
> So any build that requires a heavy substitute, heavier than the cache
> bypass threshold at least, will fail on the workers, as it would fail
> locally.
The main ‘guix publish’ instance on berlin has cache-bypass-threshold
set to 100 MiB; I don’t think we have any source or .drv that is this
big, or even a tenth of it. :-)
The ‘guix publish’ instance spawned in (cuirass remote) doesn’t use
caching at all, so there’s no bypass threshold.
The only clue we have so far is a long chain of successful .drv
substitutions followed by a failing one:
--8<---------------cut here---------------start------------->8---
Downloading http://141.80.167.131/nar/zstd/9nm35401i8j42559iadi3iz3kmhmj7pr-guix-system-tests.drv...
[K guix-system-tests.drv 1KiB 0B/s 00:00 [ ] 0.0%
[K guix-system-tests.drv 1KiB 1.3MiB/s 00:00 [##################] 100.0%
[K guix-system-tests.drv 1KiB 408KiB/s 00:00 [##################] 100.0%
@ substituter-succeeded /gnu/store/9nm35401i8j42559iadi3iz3kmhmj7pr-guix-system-tests.drv
fetching path `/gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-tests-modules.drv'...
@ substituter-started /gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-tests-modules.drv substitute
Downloading http://141.80.167.131/nar/zstd/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-tests-modules.drv...
[K guix-system-tests-modules.drv 543B 0B/s 00:00 [ ] 0.0%
[K guix-system-tests-modules.drv 543B 851KiB/s 00:00 [##################] 100.0%
[K guix-system-tests-modules.drv 543B 484KiB/s 00:00 [##################] 100.0%
@ substituter-succeeded /gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-tests-modules.drv
cannot build missing derivation ?/gnu/store/zq7idl0j51fdzqhhqm9ql90d0f2326k7-btrfs-root-on-subvolume-os.drv?
--8<---------------cut here---------------end--------------->8---
(From <https://ci.guix.gnu.org/build/224849/log/raw>.)
Plausible explanations that come to mind:
1. ‘guix publish’ returning 404, but not due to baking. Instead the
.drv is simply not in store, hence 404.
2. Client timeout (‘guix publish’ fails to reply on time).
3. Cached 404 in nginx (though we’re not supposed to cache those I
think?), or timeout in nginx (again due to ‘guix publish’ being too
slow and ‘proxy_read_timeout’ is reached, currently 10s).
We should see if we can get useful info from nginx/publish logs.
Thanks,
Ludo’.
next prev parent reply other threads:[~2022-12-08 10:27 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-16 17:57 bug#48468: substitute server connection timeout Mathieu Othacehe
2021-05-16 18:26 ` Christopher Baines
2021-05-17 14:49 ` Mathieu Othacehe
2021-05-18 14:35 ` Mathieu Othacehe
2021-05-21 13:30 ` Mathieu Othacehe
2021-05-29 21:44 ` Ludovic Courtès
2021-06-18 12:33 ` Ludovic Courtès
2021-06-29 16:49 ` Mathieu Othacehe
2022-12-05 13:21 ` Ludovic Courtès
2022-12-07 9:44 ` Mathieu Othacehe
2022-12-07 13:38 ` Ludovic Courtès
2022-12-07 14:31 ` Mathieu Othacehe
2022-12-08 10:26 ` Ludovic Courtès [this message]
2022-12-10 10:55 ` Ludovic Courtès
2022-12-27 9:52 ` Mathieu Othacehe
2022-12-28 14:23 ` Mathieu Othacehe
2023-01-06 22:42 ` Ludovic Courtès
2023-01-07 13:40 ` Mathieu Othacehe
2023-01-09 9:39 ` Ludovic Courtès
2023-01-10 8:10 ` Mathieu Othacehe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87wn72fay3.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=48468@debbugs.gnu.org \
--cc=othacehe@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).