Cuirass instability on aarch64 systems

Cuirass instability on aarch64 systems

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 2165 bytes --]


Hi Guix,

I'm running Cuirass for one of my channels on an aarch64 VM on the Hetzner
Cloud. When I visit the Cuirass web interface it sometimes serves the page,
but most of the time it just serves a blank page.

When looking at the logs I see the following error messages:

```
2024-01-21 14:54:27 GET /
2024-01-21 14:54:27 In cuirass/http.scm:
2024-01-21 14:54:27   1008:33  1 (url-handler #<input-output: socket 43> #<<request> me?> ?)
2024-01-21 14:54:27 In ice-9/boot-9.scm:
2024-01-21 14:54:27   1685:16  0 (raise-exception _ #:continuable? _)
2024-01-21 14:54:27 Throw to key `psql-query-error' with args `(#f #f "another command is already in2024-01-21 14:54:27 Uncaught exception in task:
2024-01-21 14:54:27 In fibers.scm:
2024-01-21 14:54:27     172:8  1 (_)
2024-01-21 14:54:27 In cuirass/http.scm:
2024-01-21 14:54:27 In ice-9/boot-9.scm:
2024-01-21 14:54:27   1685:16  0 (raise-exception _ #:continuable? _)
2024-01-21 14:54:27 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2024-01-21 14:54:27 In procedure port-auxiliary-write-buffer: Wrong type argument in position 1 (exp2024-01-21 14:54:27   1010:23  1 (url-handler _ #<<request> method: GET uri: #<<uri> sc?> ?)
2024-01-21 14:54:27 In ice-9/boot-9.scm:
2024-01-21 14:54:27   1685:16  0 (raise-exception _ #:continuable? _)
2024-01-21 14:54:27 Throw to key `psql-query-error' with args `(#f #f "another command is already in
```

Does anyone have an idea what is going on here? It looks to me like there is
some deadlock with PostgreSQL, but I'm not sure how to further debug this.

Recently I submitted this patch [2] to guile-fibers to get my system
updated. There were no substitutes available and I had to build the package on
my server. Unfortuntly that build never completed it just hang in the tests.

Now I'm starting to wonder if this is related. Are there any known issues with
guile-fibers on aarch64 systems? Any ideas on how to fix/debug this issue?

My server configuration can be found here [1].

Thanks, Roman.

[1] https://github.com/asahi-guix/channel/blob/main/src/asahi/guix/system/server.scm
[2] https://lists.gnu.org/archive/html/guix-patches/2024-01/msg01116.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]

Re: Cuirass instability on aarch64 systems

[guix]

[-- Attachment #1: Type: text/plain, Size: 1977 bytes --]

Hi,

TL;DR: To mitigate this, add `GUILE_JIT_THRESHOLD=-1' as an environment
variable for Cuirass. If you use the cuirass package from the guix
channel you should be able to use `package-with-patches' and the
attached patch.

On 2024-01-21T16:16:57+0100, Roman Scherer wrote:
> 
> Hi Guix,
> 
> I'm running Cuirass for one of my channels on an aarch64 VM on the Hetzner
> Cloud. When I visit the Cuirass web interface it sometimes serves the page,
> but most of the time it just serves a blank page.

I have been tracking down the exact same error last week.

My debbuging showed that a related issue has been reported to
guile-fibers in [0]. There, Andy Wingo suggested that it might be bad
codegen for atomics[1]. I tried the test case in [0] for guile commits
33e07fc56c18d4f3ffbfd53c4a8300bb2b5e49ba which should be the first
commit to be using the atomic instructions and it failed while it
succeeded with the previous commit. I also tried applying only part of
33e07fc56c18d4f3ffbfd53c4a8300bb2b5e49ba and found that applying the
changes for `compile_atomic_scm_swap_immediate' also makes the test
fail. Unfortunately, I'm not proficient in assembly, atomics and JIT and
don't see anything wrong with it. Even worse, I definitely should be
doing other things right now :D and haven't had time to report my
findings. Feel free to do so :). Else, I might find some time next week.

> [...]
> Recently I submitted this patch [2] to guile-fibers to get my system
> updated. There were no substitutes available and I had to build the package on
> my server. Unfortuntly that build never completed it just hang in the tests.

I too had that problem and was able to work around it by building it on
my x86_64 laptop using transparent emulation. I don't know if this is
related and haven't tested it with `GUILE_JIT_THRESHOLD=-1'.

I hope this helps

vicvbcun

[0]: https://github.com/wingo/fibers/issues/83
[1]: https://github.com/wingo/fibers/issues/83#issuecomment-1538427808

[-- Attachment #2: cuirass-disable-jit.patch --]
[-- Type: text/plain, Size: 487 bytes --]

diff --git a/bin/cuirass.in b/bin/cuirass.in
index 207eecf..5514466 100644
--- a/bin/cuirass.in
+++ b/bin/cuirass.in
@@ -3,6 +3,7 @@
 # @configure_input@
 #GUILE_LOAD_PATH="@PACKAGE_LOAD_PATH@${GUILE_LOAD_PATH:+:}$GUILE_LOAD_PATH"
 #GUILE_LOAD_COMPILED_PATH="@PACKAGE_LOAD_COMPILED_PATH@${GUILE_LOAD_COMPILED_PATH:+:}$GUILE_LOAD_COMPILED_PATH"
+export GUILE_JIT_THRESHOLD=-1
 exec ${GUILE:-@GUILE@} --no-auto-compile -e main -s "$0" "$@"
 !#
 ;;;; cuirass -- continuous integration tool

Re: Cuirass instability on aarch64 systems

[Roman Scherer]

Hi,

thank you for your answer. I will try out your suggestion soon.

Roman

On Sun, Jan 21, 2024, 17:47 <guix@ikherbers.com> wrote:

> Hi,
>
> TL;DR: To mitigate this, add `GUILE_JIT_THRESHOLD=-1' as an environment
> variable for Cuirass. If you use the cuirass package from the guix
> channel you should be able to use `package-with-patches' and the
> attached patch.
>
> On 2024-01-21T16:16:57+0100, Roman Scherer wrote:
> >
> > Hi Guix,
> >
> > I'm running Cuirass for one of my channels on an aarch64 VM on the
> Hetzner
> > Cloud. When I visit the Cuirass web interface it sometimes serves the
> page,
> > but most of the time it just serves a blank page.
>
> I have been tracking down the exact same error last week.
>
> My debbuging showed that a related issue has been reported to
> guile-fibers in [0]. There, Andy Wingo suggested that it might be bad
> codegen for atomics[1]. I tried the test case in [0] for guile commits
> 33e07fc56c18d4f3ffbfd53c4a8300bb2b5e49ba which should be the first
> commit to be using the atomic instructions and it failed while it
> succeeded with the previous commit. I also tried applying only part of
> 33e07fc56c18d4f3ffbfd53c4a8300bb2b5e49ba and found that applying the
> changes for `compile_atomic_scm_swap_immediate' also makes the test
> fail. Unfortunately, I'm not proficient in assembly, atomics and JIT and
> don't see anything wrong with it. Even worse, I definitely should be
> doing other things right now :D and haven't had time to report my
> findings. Feel free to do so :). Else, I might find some time next week.
>
> > [...]
> > Recently I submitted this patch [2] to guile-fibers to get my system
> > updated. There were no substitutes available and I had to build the
> package on
> > my server. Unfortuntly that build never completed it just hang in the
> tests.
>
> I too had that problem and was able to work around it by building it on
> my x86_64 laptop using transparent emulation. I don't know if this is
> related and haven't tested it with `GUILE_JIT_THRESHOLD=-1'.
>
> I hope this helps
>
> vicvbcun
>
> [0]: https://github.com/wingo/fibers/issues/83
> [1]: https://github.com/wingo/fibers/issues/83#issuecomment-1538427808
>