bug#53806: remaining connection slots are reserved for non-replication superuser connections

bug#53806: remaining connection slots are reserved for non-replication superuser connections

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1816 bytes --]

Hi guix,

Looking at <https://ci.guix.gnu.org/>, I noticed the 'core-updates'
jobset failed. Looking at the log 
https://ci.guix.gnu.org/eval/66725/log/raw>, I see


squee.scm:257:10: In procedure connect-to-postgres-paramstring:
Throw to key `psql-connect-error' with args `(connection-bad "FATAL:  remaining connection slots are reserved for non-replication superuser connections\n")'.

with bactkrace:

Backtrace:
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          12 (apply-smob/0 #<thunk 7f25bcf37020>)
In ice-9/boot-9.scm:
    724:2 11 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8 10 (_ #(#(#<directory (guile-user) 7f25bcf40c80>)))
In cuirass/ui.scm:
   104:10  9 (run-cuirass-command _ . _)
In ice-9/boot-9.scm:
    152:2  8 (with-fluid* _ _ _)
In ice-9/eval.scm:
   293:34  7 (_ #(#(#(#(#<directory (cuirass scripts evalu?> ?)) ?) ?))
   191:35  6 (_ #(#(#(#(#<directory (cuirass scripts evalu?> ?)) ?) ?))
In cuirass/utils.scm:
    123:6  5 (make-worker-thread-channel _ #:parallelism _)
In srfi/srfi-1.scm:
    634:9  4 (for-each #<procedure 7f25b365a450 at cuirass/utils.sc?> ?)
In cuirass/utils.scm:
   125:21  3 (_ . _)
In ice-9/eval.scm:
    155:9  2 (_ #(#(#<directory (cuirass scripts evaluate) 7f25bcf?>)))
In cuirass/database.scm:
   375:12  1 (db-open #:database _)
In squee.scm:
   257:10  0 (connect-to-postgres-paramstring _)

I don't know what that means precisely but apparently something went wrong.
Possibly relevant:
<https://www.postgresql.org/message-id/CAJNY3ivfgBy5V%2B5po1HZwP7d-%2Bgp2UqV%3DR-N8XFcbxBq3rZ%2Bgw%40mail.gmail.com>.
It might be possible to use persistent connections and/or raise the connection
limit.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#53806: remaining connection slots are reserved for non-replication superuser connections

[Mathieu Othacehe]

Hello Maxime,

Thanks for the report. There are 300 postgresql connection slots on
berlin which is enough under nominal CI activity. However, this issue:
http://issues.guix.gnu.org/53463 causes most of the Guix jobset
evaluations to hang, hence the connection slots starvation.

Now the 53463 issue is probably caused by two underlying issues:

- Offloading is never timing out
- Guix on aarch64 can not be evaluated correctly.

Many things to fix then.

Mathieu