From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id 8PYjDgLbIGcopwAAe85BDQ:P1 (envelope-from ) for ; Tue, 29 Oct 2024 12:54:26 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id 8PYjDgLbIGcopwAAe85BDQ (envelope-from ) for ; Tue, 29 Oct 2024 13:54:26 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1730206466; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=ZGlZMjgVdgGMOuD5RrjWeuV/d9ve2SeYh6GqwYXS8BE=; b=GcQZ5jDYQwQ9/2sBoS5wJm0vg9eyZLnbWI0NwXyLb2g/+SuH+yLhtaq4/DqI82tCXya/CW soPQb7iP8E+1lqMQF5vlVyYPh5VJTt4sNjdv0Nzep/gw3N9rhISj53Of4D1Il8eGNVo2V5 Tl669O7wUCEeJf8lyh3Dhmfldpwn7OfKp0bKsNqZBx+gdpe4nsf67O1WheTCPWmnXk9jt5 ABy1eqTjxXKaQgVdnRRPVTPAlOaTlk3hMTyquCiN57SMQ7umgjnYueB43YYFfxzvnKn1Oi yxA/6wb15hfDF+a/EVGYPyaW8MwBeAdKX5ifVy5ZQemTriBIjKXB5mjLeq63RQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1730206466; a=rsa-sha256; cv=none; b=EIsqx4kzQG/cxrgoNcd7kO0b/amLNjRbC89LF4/R2VdcLvx3Btzjib0gU0D0xFxC4OrFzE B+9iD8IjTGfSMMwciaUXQdq7fG4iq6Y3kdPtvhZDMcostWV12KAnchfRfBMBDGcXgfGIBb 1++oKcRaPKKLGC9hvBJ4rSePdjVX3Y9ulW78nparkLlFDt+rKYi8WVzrfJgKAJQQ3OVfwU k6BYDnKK8eh+6cKwPjVbCOjHjkJnIli7m6KVpH1TMoWKM33mX9O363aEMR6NN6J0YQ6iPS 9Syf3MgHCCxoxEQVapZFJ6OU3r27uKyHMSzuUddNn4acaFOjohPkUJ6DZ3E+yQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CEA626384 for ; Tue, 29 Oct 2024 13:54:25 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t5ljU-0002Uv-RZ; Tue, 29 Oct 2024 08:54:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t5ljP-0002UV-VP for guix-devel@gnu.org; Tue, 29 Oct 2024 08:53:55 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t5ljN-0003gp-9o for guix-devel@gnu.org; Tue, 29 Oct 2024 08:53:55 -0400 Received: from localhost (unknown [212.132.255.81]) by mira.cbaines.net (Postfix) with ESMTPSA id 9C49D27BBE2; Tue, 29 Oct 2024 12:53:47 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id bec34d49; Tue, 29 Oct 2024 12:53:46 +0000 (UTC) From: Christopher Baines To: Andreas Enge Cc: Nicolas Graves , guix-devel@gnu.org Subject: Re: Default postgresql version In-Reply-To: (Andreas Enge's message of "Tue, 29 Oct 2024 13:07:06 +0100") References: <874j4wtuny.fsf@ngraves.fr> <87sesgscgx.fsf@ngraves.fr> <874j4vnov9.fsf@ngraves.fr> Date: Tue, 29 Oct 2024 12:53:44 +0000 Message-ID: <87plnjkr3b.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -4.81 X-Spam-Score: -4.81 X-Migadu-Queue-Id: CEA626384 X-Migadu-Scanner: mx13.migadu.com X-TUID: Q6ALdb/iPRzE --=-=-= Content-Type: text/plain Andreas Enge writes: > Am Tue, Oct 29, 2024 at 12:42:32PM +0100 schrieb Andreas Enge: >> I have restarted nginx and nar-herder (just an educated guess), but >> to no avail. > > I think the problem is further upstream, in the place where the nars are > actually stored (hatysa). When I download a file that is in the cache, > such as > wget https://bordeaux.guix.gnu.org/nar/zstd/000z8mclk0p7jmrijj6l54bpnam29bqj-r-minfi-1.48.0 > it works. Hmm, yeah the nar-herder/nginx on hatysa seems to be stuck, it looks like quite a low level problem since I see btrfs related errors in /var/log/messages Oct 29 10:14:07 localhost vmunix: [2910798.796657] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Oct 29 10:14:07 localhost vmunix: [2910798.802941] rcu: Tasks blocked on level-0 rcu_node (CPUs 0-15): P30086 Oct 29 10:14:07 localhost vmunix: [2910798.809730] rcu: (detected by 11, t=5252 jiffies, g=253094433, q=9317 ncpus=16) Oct 29 10:14:07 localhost vmunix: [2910798.817292] task:nginx state:R running task stack:0 pid:30086 tgid:30086 ppid:30085 flags:0x0000000c Oct 29 10:14:07 localhost vmunix: [2910798.828334] Call trace: Oct 29 10:14:07 localhost vmunix: [2910798.830945] dump_backtrace+0x90/0xe8 Oct 29 10:14:07 localhost vmunix: [2910798.834789] show_stack+0x18/0x24 Oct 29 10:14:07 localhost vmunix: [2910798.838273] sched_show_task+0x12c/0x18c Oct 29 10:14:07 localhost vmunix: [2910798.842364] rcu_sched_clock_irq+0xd00/0xdcc Oct 29 10:14:07 localhost vmunix: [2910798.846804] update_process_times+0x68/0xac Oct 29 10:14:07 localhost vmunix: [2910798.851155] tick_sched_handle+0x34/0x58 Oct 29 10:14:07 localhost vmunix: [2910798.855245] tick_nohz_highres_handler+0x50/0xa8 Oct 29 10:14:07 localhost vmunix: [2910798.860030] __hrtimer_run_queues+0x138/0x1b0 Oct 29 10:14:07 localhost vmunix: [2910798.864556] hrtimer_interrupt+0xe8/0x244 Oct 29 10:14:07 localhost vmunix: [2910798.868736] arch_timer_handler_phys+0x34/0x44 Oct 29 10:14:07 localhost vmunix: [2910798.873349] handle_percpu_devid_irq+0x84/0x130 Oct 29 10:14:07 localhost vmunix: [2910798.878048] generic_handle_domain_irq+0x2c/0x44 Oct 29 10:14:07 localhost vmunix: [2910798.882834] gic_handle_irq+0x4c/0x110 Oct 29 10:14:07 localhost vmunix: [2910798.886748] call_on_irq_stack+0x24/0x4c Oct 29 10:14:07 localhost vmunix: [2910798.890839] do_interrupt_handler+0x80/0x84 Oct 29 10:14:07 localhost vmunix: [2910798.895189] el1_interrupt+0x34/0x68 Oct 29 10:14:07 localhost vmunix: [2910798.898934] el1h_64_irq_handler+0x18/0x24 Oct 29 10:14:07 localhost vmunix: [2910798.903195] el1h_64_irq+0x64/0x68 Oct 29 10:14:07 localhost vmunix: [2910798.906763] xas_descend+0x20/0x8c Oct 29 10:14:07 localhost vmunix: [2910798.910332] filemap_get_entry+0x5c/0x1ac Oct 29 10:14:07 localhost vmunix: [2910798.914513] __filemap_get_folio+0x4c/0x394 Oct 29 10:14:07 localhost vmunix: [2910798.918864] alloc_extent_buffer+0x210/0x860 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.924037] read_tree_block+0x20/0x16c [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.928766] btrfs_release_path+0x2ac/0x3b4 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.933843] btrfs_search_slot+0x340/0xb24 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.938835] btrfs_delete_one_dir_name+0x194/0x4c8 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.944516] btrfs_lookup_bio_sums+0x1e4/0x40c [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.949854] __btrfs_panic+0xb1c/0xec4 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.954498] btrfs_submit_bio+0x28/0x40 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.959224] ordered_data_exit+0x638/0x2e28 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.964300] extent_readahead+0x34c/0x384 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.969203] btrfs_transaction_exit+0x1a3c/0x3f08 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.974796] read_pages+0x80/0x2e0 Oct 29 10:14:07 localhost vmunix: [2910798.978374] page_cache_ra_unbounded+0x160/0x1e4 Oct 29 10:14:07 localhost vmunix: [2910798.983159] page_cache_ra_order+0x8c/0x2f8 Oct 29 10:14:07 localhost vmunix: [2910798.987510] ondemand_readahead+0x170/0x30c Oct 29 10:14:07 localhost vmunix: [2910798.991859] page_cache_sync_ra+0x74/0xa4 Oct 29 10:14:07 localhost vmunix: [2910798.996036] filemap_get_pages+0xc4/0x66c Oct 29 10:14:07 localhost vmunix: [2910799.000215] filemap_read+0xbc/0x398 Oct 29 10:14:07 localhost vmunix: [2910799.003959] btrfs_release_file+0xc84/0x13d8 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910799.009128] vfs_read+0x24c/0x2ec Oct 29 10:14:07 localhost vmunix: [2910799.012617] ksys_pread64+0x78/0xbc Oct 29 10:14:07 localhost vmunix: [2910799.016271] __arm64_sys_pread64+0x20/0x2c Oct 29 10:14:07 localhost vmunix: [2910799.020533] invoke_syscall+0x48/0x118 Oct 29 10:14:07 localhost vmunix: [2910799.024451] el0_svc_common.constprop.0+0x40/0xe8 Oct 29 10:14:07 localhost vmunix: [2910799.029322] do_el0_svc+0x20/0x2c Oct 29 10:14:07 localhost vmunix: [2910799.032803] el0_svc+0x34/0xb8 Oct 29 10:14:07 localhost vmunix: [2910799.036025] el0t_64_sync_handler+0x13c/0x158 Oct 29 10:14:07 localhost vmunix: [2910799.040547] el0t_64_sync+0x190/0x194 > As far as I know, only Chris has access to this machine. We should be able to get other people SSH access, I'll try restarting the machine. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmcg2thfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfPhg//cBNwvxxWNhrpWEiZ1nmxop2+1OD9O1VK IUwYM6hGmgoDulq8ixvqSzQ7SVEOAuECdT/qWOs2ulu8N7E3Bju00AHySh+En1s4 ak6nHhl6Wp0eGtOKW8SbAogOapRk9IohE1W6of8FVpMy7LzkwHddk4rVJfjC/CUF V0DTLF+xfGnSx5k1R9q91g3PHJwtXCsivmnLCabtm+sKKZ93FrMwunnmQbjmKUF/ e46Bg9A9mVfCqypVn8Ukx7FjfoW7QRciDLXfQJr058JH7IGYQHx/OWqpLxUfK/5/ JuHnmYfR451a35Vdp16Bku3D+mf56jIA97+CSQHhXUVwd87lKRPwyshKX68x5L9f +jud7JeGRlv1ZUd3MjPIR0J5CpIr37o1ElQrelfY9UTl+utiBBdvaeYRO+iH3jSu EymTtPktjaOufP4DK5g+ps0MM48t7Zgrnq2yTdRaJnzqd/+x2E/90h0bqag6THNI v6xe0dZ+dh6o+PHBlD3jn0EkDjSbSa6FDbFLf5HqoYWJ2ZCpybi+dZVrqqe9WENz YDYqRlPruqk3kGivon/hDfOsDo7ybMAVyZV4UGm03m4Imq2RBlJkwdZMDuPTROU8 uCdxA5hlpjCnqAaHjl+/tPkWyZO+IPj54U3qYDUS3K7BFEHZa6/NEmtdaxNUC3F0 +CbkT3+zZeQ= =s28K -----END PGP SIGNATURE----- --=-=-=--