* bug#72166: Shepherd periodically goes unresponsive on one of my machines
@ 2024-07-18 0:43 Jonathan Frederickson
2024-07-19 15:35 ` Ludovic Courtès
0 siblings, 1 reply; 11+ messages in thread
From: Jonathan Frederickson @ 2024-07-18 0:43 UTC (permalink / raw)
To: 72166
I've been running into an issue with Shepherd on one of my machines. Every so often (and I haven't figured out what conditions trigger it), my Shepherd instances (both home and PID 1) will go unresponsive. I thought I had tracked it down to a misbehaving home service that I had configured, but it's just happened again without that service running.
'herd status' hangs indefinitely:
jfred@terracard ~$ sudo herd status
Password:
<never returns>
...on both instances:
jfred@terracard ~$ herd status
<never returns>
The PID 1 shepherd instance isn't reaping defunct processes:
jfred@terracard ~$ ps aux | grep -i lock
jfred 541 0.0 0.0 3700 2304 ? S 18:30 0:00 swayidle -w timeout 300 swaylock -f -i ~/.wallpapers/user-manual.jpg timeout 10 if pgrep swaylock; then swaymsg "output * dpms off"; fi resume swaymsg "output * dpms on" before-sleep swaylock -f -i ~/.wallpapers/user-manual.jpg
jfred 3111 0.0 0.0 0 0 ? Z 18:53 0:00 [swaylock] <defunct>
jfred 3112 0.0 0.0 0 0 ? Zs 18:53 0:00 [swaylock] <defunct>
Some further troubleshooting... strace indicates that it's waiting on a read() on its fd 9:
jfred@terracard ~ [env]$ sudo strace -fp 1
Password:
strace: Process 1 attached with 5 threads
[pid 144] read(9, <unfinished ...>
[pid 142] futex(0x7fa43892abe8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 141] futex(0x7fa43892abe8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 140] futex(0x7fa43892abe8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY^
...which seems to be:
jfred@terracard ~ [env]$ sudo ls -l /proc/1/fd/9
lr-x------ 1 root root 64 Jul 17 20:39 /proc/1/fd/9 -> 'pipe:[4015]'
jfred@terracard ~ [env]$ sudo lsof -n | grep 4015
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
Output information may be incomplete.
shepherd 1 root 9r FIFO 0,15 0t0 4015 pipe
shepherd 1 root 11w FIFO 0,15 0t0 4015 pipe
shepherd 1 140 GC-marker root 9r FIFO 0,15 0t0 4015 pipe
shepherd 1 140 GC-marker root 11w FIFO 0,15 0t0 4015 pipe
shepherd 1 141 GC-marker root 9r FIFO 0,15 0t0 4015 pipe
shepherd 1 141 GC-marker root 11w FIFO 0,15 0t0 4015 pipe
shepherd 1 142 GC-marker root 9r FIFO 0,15 0t0 4015 pipe
shepherd 1 142 GC-marker root 11w FIFO 0,15 0t0 4015 pipe
shepherd 1 144 shepherd root 9r FIFO 0,15 0t0 4015 pipe
shepherd 1 144 shepherd root 11w FIFO 0,15 0t0 4015 pipe
My system configuration for this machine can be found here, and I last ran a 'guix pull' on June 21: https://github.com/jfrederickson/dotfiles/blob/master/guix/guix/system/machines/terracard/config.scm
Has anyone else run into this?
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-07-18 0:43 bug#72166: Shepherd periodically goes unresponsive on one of my machines Jonathan Frederickson
@ 2024-07-19 15:35 ` Ludovic Courtès
2024-07-19 16:25 ` Jonathan Frederickson
0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2024-07-19 15:35 UTC (permalink / raw)
To: Jonathan Frederickson; +Cc: 72166
Hi Jonathan,
"Jonathan Frederickson" <jonathan@terracrypt.net> skribis:
> I've been running into an issue with Shepherd on one of my machines. Every so often (and I haven't figured out what conditions trigger it), my Shepherd instances (both home and PID 1) will go unresponsive. I thought I had tracked it down to a misbehaving home service that I had configured, but it's just happened again without that service running.
>
> 'herd status' hangs indefinitely:
>
> jfred@terracard ~$ sudo herd status
> Password:
> <never returns>
>
> ...on both instances:
>
> jfred@terracard ~$ herd status
> <never returns>
Ouch. What version of shepherd is running? (You can view it with
“cat /proc/1/cmdline | xargs -0”.)
> The PID 1 shepherd instance isn't reaping defunct processes:
>
> jfred@terracard ~$ ps aux | grep -i lock
> jfred 541 0.0 0.0 3700 2304 ? S 18:30 0:00 swayidle -w timeout 300 swaylock -f -i ~/.wallpapers/user-manual.jpg timeout 10 if pgrep swaylock; then swaymsg "output * dpms off"; fi resume swaymsg "output * dpms on" before-sleep swaylock -f -i ~/.wallpapers/user-manual.jpg
> jfred 3111 0.0 0.0 0 0 ? Z 18:53 0:00 [swaylock] <defunct>
> jfred 3112 0.0 0.0 0 0 ? Zs 18:53 0:00 [swaylock] <defunct>
>
> Some further troubleshooting... strace indicates that it's waiting on a read() on its fd 9:
Interesting. There were bugs in earlier 0.10.x version that could cause
this sort of thing; let’s see what version you have, first.
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-07-19 15:35 ` Ludovic Courtès
@ 2024-07-19 16:25 ` Jonathan Frederickson
2024-07-22 7:14 ` Ludovic Courtès
0 siblings, 1 reply; 11+ messages in thread
From: Jonathan Frederickson @ 2024-07-19 16:25 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 72166
On Fri, Jul 19, 2024, at 11:35 AM, Ludovic Courtès wrote:
> Hi Jonathan,
>
> "Jonathan Frederickson" <jonathan@terracrypt.net> skribis:
>
> > I've been running into an issue with Shepherd on one of my machines. Every so often (and I haven't figured out what conditions trigger it), my Shepherd instances (both home and PID 1) will go unresponsive. I thought I had tracked it down to a misbehaving home service that I had configured, but it's just happened again without that service running.
> >
> > 'herd status' hangs indefinitely:
> >
> > jfred@terracard ~$ sudo herd status
> > Password:
> > <never returns>
> >
> > ...on both instances:
> >
> > jfred@terracard ~$ herd status
> > <never returns>
>
> Ouch. What version of shepherd is running? (You can view it with
> “cat /proc/1/cmdline | xargs -0”.)
>
> > The PID 1 shepherd instance isn't reaping defunct processes:
> >
> > jfred@terracard ~$ ps aux | grep -i lock
> > jfred 541 0.0 0.0 3700 2304 ? S 18:30 0:00 swayidle -w timeout 300 swaylock -f -i ~/.wallpapers/user-manual.jpg timeout 10 if pgrep swaylock; then swaymsg "output * dpms off"; fi resume swaymsg "output * dpms on" before-sleep swaylock -f -i ~/.wallpapers/user-manual.jpg
> > jfred 3111 0.0 0.0 0 0 ? Z 18:53 0:00 [swaylock] <defunct>
> > jfred 3112 0.0 0.0 0 0 ? Zs 18:53 0:00 [swaylock] <defunct>
> >
> > Some further troubleshooting... strace indicates that it's waiting on a read() on its fd 9:
>
> Interesting. There were bugs in earlier 0.10.x version that could cause
> this sort of thing; let’s see what version you have, first.
>
> Ludo’.
>
Hi Ludo, thanks for the troubleshooting help. Looks like I'm running 0.10.4:
jfred@terracard ~$ cat /proc/1/cmdline | xargs -0
/gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile --no-auto-compile /gnu/store/39li5qpiaj1lx89xgahlbgvfnjhpcpwg-shepherd-0.10.4/bin/shepherd --config /gnu/store/hfyri6ygfdjq4w3nkha2ypa2k98hhfxj-shepherd.conf
I see now that 0.10.5 was released a few weeks ago, does that have a fix that could be related?
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-07-19 16:25 ` Jonathan Frederickson
@ 2024-07-22 7:14 ` Ludovic Courtès
2024-07-25 0:08 ` Jonathan Frederickson
0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2024-07-22 7:14 UTC (permalink / raw)
To: Jonathan Frederickson; +Cc: 72166
Hi,
"Jonathan Frederickson" <jonathan@terracrypt.net> skribis:
> Hi Ludo, thanks for the troubleshooting help. Looks like I'm running 0.10.4:
>
> jfred@terracard ~$ cat /proc/1/cmdline | xargs -0
> /gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile --no-auto-compile /gnu/store/39li5qpiaj1lx89xgahlbgvfnjhpcpwg-shepherd-0.10.4/bin/shepherd --config /gnu/store/hfyri6ygfdjq4w3nkha2ypa2k98hhfxj-shepherd.conf
>
> I see now that 0.10.5 was released a few weeks ago, does that have a fix that could be related?
Yes, it could be related. Per the ‘NEWS’ file of Shepherd:
** ‘herd unload root SERVICE’ no longer hands when there’s a replacement
(<https://issues.guix.gnu.org/71478>)
It used to be that, for a running service S that has a replacement registered,
‘herd unload root S’ would hang shepherd, making it totally unresponsive—‘herd
status’, ‘halt’, etc. would hang forever, and inetd-style services would no
longer start, etc. This is now fixed.
Depending on previous ‘guix system reconfigure’ invocations on these
machines, it’s possible that you ended up in this state.
Would be great if you could upgrade and see if the problem still occurs.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-07-22 7:14 ` Ludovic Courtès
@ 2024-07-25 0:08 ` Jonathan Frederickson
2024-08-16 16:12 ` Ludovic Courtès
0 siblings, 1 reply; 11+ messages in thread
From: Jonathan Frederickson @ 2024-07-25 0:08 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 72166
On Mon, Jul 22, 2024, at 3:14 AM, Ludovic Courtès wrote:
> Hi,
>
> "Jonathan Frederickson" <jonathan@terracrypt.net> skribis:
>
> > Hi Ludo, thanks for the troubleshooting help. Looks like I'm running 0.10.4:
> >
> > jfred@terracard ~$ cat /proc/1/cmdline | xargs -0
> > /gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile --no-auto-compile /gnu/store/39li5qpiaj1lx89xgahlbgvfnjhpcpwg-shepherd-0.10.4/bin/shepherd --config /gnu/store/hfyri6ygfdjq4w3nkha2ypa2k98hhfxj-shepherd.conf
> >
> > I see now that 0.10.5 was released a few weeks ago, does that have a fix that could be related?
>
> Yes, it could be related. Per the ‘NEWS’ file of Shepherd:
>
> ** ‘herd unload root SERVICE’ no longer hands when there’s a replacement
> (<https://issues.guix.gnu.org/71478>)
>
> It used to be that, for a running service S that has a replacement registered,
> ‘herd unload root S’ would hang shepherd, making it totally unresponsive—‘herd
> status’, ‘halt’, etc. would hang forever, and inetd-style services would no
> longer start, etc. This is now fixed.
>
> Depending on previous ‘guix system reconfigure’ invocations on these
> machines, it’s possible that you ended up in this state.
>
> Would be great if you could upgrade and see if the problem still occurs.
>
> Thanks,
> Ludo’.
I've gotten this machine upgraded to 0.10.5 and just experienced the same thing again:
jfred@terracard ~$ ps aux | grep swaylo
jfred 544 0.0 0.0 3700 2432 ? S 19:02 0:00 swayidle -w timeout 300 swaylock -f -i ~/.wallpapers/user-manual.jpg timeout 10 if pgrep swaylock; then swaymsg "output * dpms off"; fi resume swaymsg "output * dpms on" before-sleep swaylock -f -i ~/.wallpapers/user-manual.jpg
jfred 1956 0.0 0.0 0 0 ? Z 19:22 0:00 [swaylock] <defunct>
jfred 1957 0.0 0.0 0 0 ? Zs 19:22 0:00 [swaylock] <defunct>
jfred 2162 0.0 0.0 0 0 ? Z 19:38 0:00 [swaylock] <defunct>
jfred 2163 0.0 0.0 0 0 ? Zs 19:38 0:00 [swaylock] <defunct>
jfred 2604 0.0 0.0 6116 2432 pts/2 S+ 20:04 0:00 grep --color=auto swaylo
jfred@terracard ~$ cat /proc/1/cmdline
/gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile--no-auto-compile/gnu/store/wrmyav254ydjn9cad3q169fxg7x6p80b-shepherd-0.10.5/bin/shepherd--config/gnu/store/sfjww12mipyx4zxa6i9x8nxxfyb7h3y4-shepherd.conf
Of note, I haven't run 'guix system reconfigure' or any manual 'herd' commands on this machine since boot.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-07-25 0:08 ` Jonathan Frederickson
@ 2024-08-16 16:12 ` Ludovic Courtès
2024-08-18 22:54 ` Jonathan Frederickson
0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2024-08-16 16:12 UTC (permalink / raw)
To: Jonathan Frederickson; +Cc: 72166
Hi,
"Jonathan Frederickson" <jonathan@terracrypt.net> skribis:
> I've gotten this machine upgraded to 0.10.5 and just experienced the same thing again:
Argh.
> jfred@terracard ~$ ps aux | grep swaylo
> jfred 544 0.0 0.0 3700 2432 ? S 19:02 0:00 swayidle -w timeout 300 swaylock -f -i ~/.wallpapers/user-manual.jpg timeout 10 if pgrep swaylock; then swaymsg "output * dpms off"; fi resume swaymsg "output * dpms on" before-sleep swaylock -f -i ~/.wallpapers/user-manual.jpg
> jfred 1956 0.0 0.0 0 0 ? Z 19:22 0:00 [swaylock] <defunct>
> jfred 1957 0.0 0.0 0 0 ? Zs 19:22 0:00 [swaylock] <defunct>
> jfred 2162 0.0 0.0 0 0 ? Z 19:38 0:00 [swaylock] <defunct>
> jfred 2163 0.0 0.0 0 0 ? Zs 19:38 0:00 [swaylock] <defunct>
> jfred 2604 0.0 0.0 6116 2432 pts/2 S+ 20:04 0:00 grep --color=auto swaylo
> jfred@terracard ~$ cat /proc/1/cmdline
> /gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile--no-auto-compile/gnu/store/wrmyav254ydjn9cad3q169fxg7x6p80b-shepherd-0.10.5/bin/shepherd--config/gnu/store/sfjww12mipyx4zxa6i9x8nxxfyb7h3y4-shepherd.conf
>
> Of note, I haven't run 'guix system reconfigure' or any manual 'herd' commands on this machine since boot.
Could you share (maybe privately) the relevant excerpt of
/var/log/messages?
Could you also share (ideally) a minimum Guix System config and a
sequence of commands to reproduce it?
Are you able to reproduce it in ‘guix system vm’?
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-08-16 16:12 ` Ludovic Courtès
@ 2024-08-18 22:54 ` Jonathan Frederickson
2024-08-19 20:01 ` Jonathan Frederickson
[not found] ` <eec39e18-8a2b-440f-ad97-4779e56362af@app.fastmail.com>
0 siblings, 2 replies; 11+ messages in thread
From: Jonathan Frederickson @ 2024-08-18 22:54 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 72166
[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]
On Fri, Aug 16, 2024, at 12:12 PM, Ludovic Courtès wrote:
> Could you share (maybe privately) the relevant excerpt of
> /var/log/messages?
>
> Could you also share (ideally) a minimum Guix System config and a
> sequence of commands to reproduce it?
>
> Are you able to reproduce it in ‘guix system vm’?
I'll send some logs to you privately; I don't actually see any logs that look relevant in /var/log/messages but maybe you'll spot something I haven't.
I'll try to come up with a reproduction and a minimal system config to reproduce the issue, though I will say that I haven't yet tracked down the exact situation that triggers it.
(Perhaps notably, both this machine running Guix System and another machine running Guix Home on a foreign distro where I've experienced the same issue are running Sway, and I think swaylock may be related somehow.)
I'll also try reproducing it with 'guix system vm' if I can, though that may be challenging as this machine only has 4 GB of RAM.
[-- Attachment #2: Type: text/html, Size: 1391 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-08-18 22:54 ` Jonathan Frederickson
@ 2024-08-19 20:01 ` Jonathan Frederickson
[not found] ` <eec39e18-8a2b-440f-ad97-4779e56362af@app.fastmail.com>
1 sibling, 0 replies; 11+ messages in thread
From: Jonathan Frederickson @ 2024-08-19 20:01 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 72166
No news yet on reproduction, but I did spot something else interesting when running 'guix home switch-generation' on one of my other machines that's been having similar issues:
> SLoading /gnu/store/b7sbff937mm32lwv1a0dbh4mqhwkvpfn-shepherd.conf.
> herd: error: exception caught while executing 'load' on service 'root':
> In procedure fport_write: Input/output error
> Comparing /gnu/store/69shsgavrzkb408hvrfi4yjs26z7n112-home/profile/share/fonts and
> /gnu/store/dn7kk3d6hhmgjm89y2p3mrlwz0dgkzpy-home/profile/share/fonts... done (same)
> Evaluating on-change gexps.
>
> On-change gexps evaluation finished.
This is another machine whose user shepherd instance is currently in this unresponsive state.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
[not found] ` <b86349b4-4c6f-4fef-b29b-95db86065a85@terracrypt.net>
@ 2024-08-22 9:35 ` Ludovic Courtès
2024-08-22 17:52 ` Jonathan Frederickson
0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2024-08-22 9:35 UTC (permalink / raw)
To: Jonathan Frederickson; +Cc: 72166
Hi,
You wrote:
> The machine in question does not have a battery-backed RTC, so it
> loses time when it loses power, but notice the time changes once ntpd
> starts up.
So you’re hitting the Fibers bug described here:
https://issues.guix.gnu.org/70848
There is still no fix upstream unfortunately, but I hope we’ll get there
in time for Shepherd 1.0.
The only workaround so far is to build shepherd against
‘guile-fibers-1.1’, which is what is done in Guix on AArch64 and RISC-V
since these platforms are likely to be used with single-board computers
lacking a battery-backed RTC.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-08-22 9:35 ` Ludovic Courtès
@ 2024-08-22 17:52 ` Jonathan Frederickson
2024-09-04 9:24 ` Ludovic Courtès
0 siblings, 1 reply; 11+ messages in thread
From: Jonathan Frederickson @ 2024-08-22 17:52 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 72166
[-- Attachment #1: Type: text/plain, Size: 1196 bytes --]
Aug 22, 2024 05:35:47 Ludovic Courtès <ludo@gnu.org>:
> Hi,
>
> You wrote:
>
>> The machine in question does not have a battery-backed RTC, so it
>> loses time when it loses power, but notice the time changes once ntpd
>> starts up.
>
> So you’re hitting the Fibers bug described here:
>
> https://issues.guix.gnu.org/70848
>
> There is still no fix upstream unfortunately, but I hope we’ll get there
> in time for Shepherd 1.0.
>
> The only workaround so far is to build shepherd against
> ‘guile-fibers-1.1’, which is what is done in Guix on AArch64 and RISC-V
> since these platforms are likely to be used with single-board computers
> lacking a battery-backed RTC.
>
> Thanks,
> Ludo’.
Ah! Yes, that seems likely.
I've also run into what looks like the same thing on a laptop that does have an RTC, but it also has an NTP daemon running on it. (I have also noticed that this is most common on that laptop after a suspend/resume cycle, so maybe that's triggering the bug as well?)
I'll check to see if I've recently had an NTP sync the next time this happens, because that would be a definitive confirmation that I'm running into that bug.
[-- Attachment #2: Type: text/html, Size: 2147 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#72166: Shepherd periodically goes unresponsive on one of my machines
2024-08-22 17:52 ` Jonathan Frederickson
@ 2024-09-04 9:24 ` Ludovic Courtès
0 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2024-09-04 9:24 UTC (permalink / raw)
To: Jonathan Frederickson; +Cc: 72166
Hi,
Jonathan Frederickson <jonathan@terracrypt.net> skribis:
> I've also run into what looks like the same thing on a laptop that does have an RTC, but it also has an NTP daemon running on it. (I have also noticed that this is most common on that laptop after a suspend/resume cycle, so maybe that's triggering the bug as well?)
Suspend/resume does not trigger the bug no (fortunately), but a clock
jump due to NTP sync can cause it (specifically, shepherd will consume
CPU time proportional to the drift).
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-09-04 9:28 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-07-18 0:43 bug#72166: Shepherd periodically goes unresponsive on one of my machines Jonathan Frederickson
2024-07-19 15:35 ` Ludovic Courtès
2024-07-19 16:25 ` Jonathan Frederickson
2024-07-22 7:14 ` Ludovic Courtès
2024-07-25 0:08 ` Jonathan Frederickson
2024-08-16 16:12 ` Ludovic Courtès
2024-08-18 22:54 ` Jonathan Frederickson
2024-08-19 20:01 ` Jonathan Frederickson
[not found] ` <eec39e18-8a2b-440f-ad97-4779e56362af@app.fastmail.com>
[not found] ` <8734mztl9u.fsf@gnu.org>
[not found] ` <b86349b4-4c6f-4fef-b29b-95db86065a85@terracrypt.net>
2024-08-22 9:35 ` Ludovic Courtès
2024-08-22 17:52 ` Jonathan Frederickson
2024-09-04 9:24 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).