guix pull/guix upgrade often fails over VPN with TLS error message

guix pull/guix upgrade often fails over VPN with TLS error message

[Thom R Harmon]

tldr; guix substitute: error: TLS error in procedure 
'write_to_session_record_port': Error in the push function.

The full error message:

 > substitute: updating substitutes from 
'https://bordeaux.guix.gnu.org'...   0.0%guix substitute: error: TLS 
error in procedure 'write_to_session_record_port': Error in the push 
function.
guix upgrade: error: 
`/gnu/store/8wp75vw27zm2c8cfkpxqg73glslqvmgn-guix-command substitute' 
died unexpectedly

There's at least one thread about this message in the archives but it's 
from before I subscribed and I don't know how/if I can tag into that 
thread so I'll just start a new one.

I've been seeing the above error message on `guix pull` and `guix 
upgrade` off-and-on for about 2 years. It's not 100% correlated but 
seems to be more prevalent when I'm doing the guix operation over a VPN. 
It seems to come in bunches meaning I won't see the error for a few 
weeks and then it will persist for a couple of weeks to the point where 
I basically cannot do a pull/upgrade.

Anyone have any thoughts as to what might happening? Or, even better, 
ideas as to how I might troubleshoot? Previous attempts to capture it 
with wireshark/tshark have not been indicative of the root cause.

thx

Re: guix pull/guix upgrade often fails over VPN with TLS error message

[Thom R Harmon]

Well, that was fun. Sorry, folks. I have no idea why that came through 
so many times. One heck of an intro to the list though. ;)

Re: guix pull/guix upgrade often fails over VPN with TLS error message

[Thom R Harmon]

I've still not come up with a decent way to troubleshoot this issue but 
it persists. It seems to be happening quite frequently now even when I'm 
not accessing the substitute servers over a VPN connection. The box 
exhibiting this behavior consistently is a Debian 12 box with guix 
installed via binary install. I have another machine running an Ubuntu 
release and, as far as I can tell, it does not have this problem.

Still happy to get advice as to how I might be able to debug.

Re: guix pull/guix upgrade often fails over VPN with TLS error message

[Thom R Harmon]

It's probably worth mentioning that I earlier today, on the theory that 
perhaps the required connections were timing out, I tested both `guix 
pull` and `guix upgrade` with `--timeout=0`. Unfortunately that did not 
fix the issue.

Re: guix pull/guix upgrade often fails over VPN with TLS error message

[Thom R Harmon via]

So this is interesting.... I did a pkt capture with tshark while a 'guix pull' was running and captured RST packets for the TLS connection:

1 0.000000000    <redacted>           185.233.100.56        SSL      2804   Continuation Data

2 0.000047880    <redacted>           185.233.100.56        SSL      2804   Continuation Data

3 0.355735909    185.233.100.56        <redacted>           TCP      62     443 → 53526 [RST] Seq=1 Win=0 Len=0

4 0.355891353    185.233.100.56        <redacted>           TCP      62     443 → 53526 [RST] Seq=1 Win=0 Len=0

5 0.355891393    185.233.100.56        <redacted>           TCP      62     443 → 53526 [RST] Seq=1 Win=0 Len=0

6 0.355939644    185.233.100.56        <redacted>           TCP      62     443 → 53526 [RST] Seq=1 Win=0 Len=0

7 0.356476147    185.233.100.56        <redacted>           TCP      62     443 → 53526 [RST] Seq=1 Win=0 Len=0

8 0.356476197    185.233.100.56        <redacted>           TCP      62     443 → 53526 [RST] Seq=1 Win=0 Len=0

Now, is that RST coming from an intermediate device (ex: my firewall) or directly from the sub server? Not sure but I will inspect firewall logs and its interesting that its only the one host exhibiting this behavior. FWIW, no host-based firewall or IPS/IDS in play here.

Re: guix pull/guix upgrade often fails over VPN with TLS error message

[Thom R Harmon]

Just closing the loop on this...

So, nobody had any advice as to how to troubleshoot this and I stopped 
looking for root cause and started looking for a fix of any sort. Turns 
out that was to make sure nothing was using /gnu/store and then `rm -rf 
/gnu/* /var/guix` and re-install guix. All of the nodes which were 
exhibiting this behavior stopped doing so after a re-install.

My only theory is that there is something I am doing when managing the 
guix binary install that will occasionally result in the systems getting 
into this state. Perhaps something to do with the TLS libs.

Re: guix pull/guix upgrade often fails over VPN with TLS error message

[Richard Sent]

Thom R Harmon <trharmon@proton.me> writes:

> Just closing the loop on this...
>
> So, nobody had any advice as to how to troubleshoot this and I stopped 
> looking for root cause and started looking for a fix of any sort. Turns 
> out that was to make sure nothing was using /gnu/store and then `rm -rf 
> /gnu/* /var/guix` and re-install guix. All of the nodes which were 
> exhibiting this behavior stopped doing so after a re-install.
>
> My only theory is that there is something I am doing when managing the 
> guix binary install that will occasionally result in the systems getting 
> into this state. Perhaps something to do with the TLS libs.

I missed this email earlier, but FYI your issue sounds similar to
https://issues.guix.gnu.org/71238. A root cause hasn't been identified
there either.

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.