I'm building Guile 3.0.9 on iOS 14.5.1 (Darwin 20.4.0) which is suffering some system API differences within pthread_jit* stuff. cat alist.doc array-handle.doc array-map.doc arrays.doc async.doc atomic.doc backtrace.doc boolean.doc bitvectors.doc bytevectors.doc chars.doc control.doc continuations.doc debug.doc deprecated.doc deprecation.doc dynl.doc dynwind.doc eq.doc error.doc eval.doc evalext.doc exceptions.doc expand.doc extensions.doc fdes-finalizers.doc feature.doc filesys.doc fluids.doc foreign.doc fports.doc gc-malloc.doc gc.doc gettext.doc generalized-vectors.doc goops.doc gsubr.doc guardians.doc hash.doc hashtab.doc hooks.doc i18n.doc init.doc ioext.doc keywords.doc list.doc load.doc macros.doc mallocs.doc memoize.doc modules.doc numbers.doc objprop.doc options.doc pairs.doc ports.doc print.doc procprop.doc procs.doc promises.doc r6rs-ports.doc random.doc rdelim.doc read.doc rw.doc scmsigs.doc script.doc simpos.doc smob.doc sort.doc srcprop.doc srfi-1.doc srfi-4.doc srfi-13.doc srfi-14.doc srfi-60.doc stackchk.doc stacks.doc stime.doc strings.doc strorder.doc strports.doc struct.doc symbols.doc syntax.doc threads.doc throw.doc unicode.doc uniform.doc values.doc variable.doc vectors.doc version.doc vports.doc weak-set.doc weak-table.doc weak-vector.doc dynl.doc posix.doc net_db.doc socket.doc regex-posix.doc | GUILE_AUTO_COMPILE=0 ../meta/build-env guild snarf-check-and-output-texi > guile-procedures.texi || { rm guile-procedures.texi; false; } cat: write error: Broken pipe After inspecting the crash logs, it might be some unexpected freeing of mmap cache, the librecompat.0.dylib includes a sys_icache_invalidate wrapper for __clear_cache function (as the same implementation with macOS one since they are all come from LLVM) which has not been compiled to system libcompiler_rt.dylib for iOS. Exception Type: EXC_BAD_ACCESS (SIGBUS) Exception Subtype: KERN_PROTECTION_FAILURE at 0x0000000104ccb9d8 VM Region Info: 0x104ccb9d8 is in 0x104cc4000-0x104ccc000; bytes after start: 31192 bytes before end: 1575 REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL __LINKEDIT 104bc0000-104cc4000 [ 1040K] r--/rw- SM=COW ...e-3.0.1.dylib ---> __TEXT 104cc4000-104ccc000 [ 32K] r--/rw- SM=COW ...ompat.0.dylib __DATA_CONST 104ccc000-104cd0000 [ 16K] rw-/rw- SM=COW ...ompat.0.dylib Termination Signal: Bus error: 10 Termination Reason: Namespace SIGNAL, Code 0xa Terminating Process: exc handler [9957] Triggered by Thread: 0 Thread 0 name: Dispatch queue: com.apple.main-thread Thread 0 Crashed: 0 + librecompat.0.dylib 0x104ccb9d8 0x104cc4000 + 0x79d8 // __clear_cache + 0x0 1 libguile-3.0.1.dylib 0x104b4b728 0x104a44000 + 0x107728 // /buildroot/guile-3.0.9/libguile/./lightening/lightening/aarch64.c:217 2 libguile-3.0.1.dylib 0x104b4b3b0 0x104a44000 + 0x1073b0 // /buildroot/guile-3.0.9/libguile/./lightening/lightening/lightening.c:241 3 libguile-3.0.1.dylib 0x104aa12e0 0x104a44000 + 0x5d2e0 // /buildroot/guile-3.0.9/libguile/jit.c:1424 4 libguile-3.0.1.dylib 0x104aa0c38 0x104a44000 + 0x5cc38 // /buildroot/guile-3.0.9/libguile/jit.c:5895 5 libsystem_pthread.dylib 0x1def01f60 0x1deeff000 + 0x2f60 // __pthread_once_handler + 0x50 6 libsystem_platform.dylib 0x1deefb964 0x1deef7000 + 0x4964 // _os_once_callout + 0x20 7 libsystem_pthread.dylib 0x1def01ef4 0x1deeff000 + 0x2ef4 // pthread_once + 0x64 8 libguile-3.0.1.dylib 0x104aa0558 0x104a44000 + 0x5c558 // /buildroot/guile-3.0.9/libguile/jit.c:0 9 libguile-3.0.1.dylib 0x104aa0410 0x104a44000 + 0x5c410 // /buildroot/guile-3.0.9/libguile/jit.c:6031 10 libguile-3.0.1.dylib 0x104b35774 0x104a44000 + 0xf1774 // /buildroot/guile-3.0.9/libguile/./vm-engine.c:370 11 libguile-3.0.1.dylib 0x104b323d0 0x104a44000 + 0xee3d0 // /buildroot/guile-3.0.9/libguile/vm.c:1615 12 libguile-3.0.1.dylib 0x104a6776c 0x104a44000 + 0x2376c // /buildroot/guile-3.0.9/libguile/eval.c:496 13 libguile-3.0.1.dylib 0x104af9e20 0x104a44000 + 0xb5e20 // /buildroot/guile-3.0.9/libguile/read.c:1553 14 libguile-3.0.1.dylib 0x104abb158 0x104a44000 + 0x77158 // /buildroot/guile-3.0.9/libguile/load.c:124 15 libguile-3.0.1.dylib 0x104abc3d4 0x104a44000 + 0x783d4 // /buildroot/guile-3.0.9/libguile/load.c:1267 16 libguile-3.0.1.dylib 0x104abcb28 0x104a44000 + 0x78b28 // /buildroot/guile-3.0.9/libguile/load.c:1275 17 libguile-3.0.1.dylib 0x104a8e0ec 0x104a44000 + 0x4a0ec // /buildroot/guile-3.0.9/libguile/init.c:230 18 libguile-3.0.1.dylib 0x104a8e458 0x104a44000 + 0x4a458 // /buildroot/guile-3.0.9/libguile/init.c:510 19 libguile-3.0.1.dylib 0x104b2a89c 0x104a44000 + 0xe689c // /buildroot/guile-3.0.9/libguile/threads.c:578 20 libguile-3.0.1.dylib 0x104b2d47c 0x104a44000 + 0xe947c // /buildroot/guile-3.0.9/libguile/threads.c:642 21 + libgc.1.dylib 0x1053688e4 0x105354000 + 0x148e4 // /buildroot/gc-8.2.2/misc.c:2173 22 libguile-3.0.1.dylib 0x104b2a97c 0x104a44000 + 0xe697c // /buildroot/guile-3.0.9/libguile/threads.c:692 23 libguile-3.0.1.dylib 0x104b2a930 0x104a44000 + 0xe6930 // /buildroot/guile-3.0.9/libguile/threads.c:698 24 libguile-3.0.1.dylib 0x104a8e190 0x104a44000 + 0x4a190 // /buildroot/guile-3.0.9/libguile/init.c:295 25 guile (*) 0x104a37d90 0x104a30000 + 0x7d90 // /buildroot/guile-3.0.9/libguile/guile.c:94 26 libdyld.dylib 0x193c92cf8 0x193c91000 + 0x1cf8 // start + 0x4 But according to the JIT allocation problems which has been met by other projects: https://patchwork.kernel.org/project/qemu-devel/patch/20201108232425.1705-7-j@getutm.app/#23841029 I attempted to patch libguile/jit.c like that: --- libguile/jit.c 1676274604.835054871 +++ libguile/jit.c.old 1676204975.415324076 @@ -47,8 +47,17 @@ #include #endif -#if defined __APPLE__ && HAVE_PTHREAD_JIT_WRITE_PROTECT_NP +#if defined __APPLE__ #include +#include "tcg-apple-jit.h" + +static void tb_exec_change(bool locked) +{ + if (jit_write_protect_supported()) { + jit_write_protect(locked); + } +} + #endif #include "jit.h" @@ -1414,8 +1435,9 @@ uint8_t *ret = jit_address (j->jit); -#if defined __APPLE__ && HAVE_PTHREAD_JIT_WRITE_PROTECT_NP - pthread_jit_write_protect_np(0); +#if defined __APPLE__ +// pthread_jit_write_protect_np(0); + tb_exec_change(0); #endif emit (j); @@ -1423,10 +1445,11 @@ size_t size; if (!jit_has_overflow (j->jit) && jit_end (j->jit, &size)) { -#if defined __APPLE__ && HAVE_PTHREAD_JIT_WRITE_PROTECT_NP +#if defined __APPLE__ /* protect previous code arena. leave unprotected after emit() since jit_end() also writes to code arena. */ - pthread_jit_write_protect_np(1); +// pthread_jit_write_protect_np(1); + tb_exec_change(1); sys_icache_invalidate(arena->base, arena->size); #endif ASSERT (size <= (arena->size - arena->used)); @@ -1442,9 +1465,10 @@ } else { -#if defined __APPLE__ && HAVE_PTHREAD_JIT_WRITE_PROTECT_NP +#if defined __APPLE__ /* protect previous code arena */ - pthread_jit_write_protect_np(1); +// pthread_jit_write_protect_np(1); + tb_exec_change(1); sys_icache_invalidate(arena->base, arena->size); #endif jit_reset (j->jit); Which replaces the previous `pthread_jit_write_protect_np` with another iOS capable implementation that provided in the link above: /* * Apple Silicon functions for JIT handling * * Copyright (c) 2020 osy * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. * * This library is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * You should have received a copy of the GNU Lesser General Public * License along with this library; if not, see . */ #ifndef TCG_APPLE_JIT_H #define TCG_APPLE_JIT_H /* * APRR handling * Credits to: https://siguza.github.io/APRR/ * Reversed from /usr/lib/system/libsystem_pthread.dylib */ #if defined(__aarch64__) && defined(__APPLE__) #define _COMM_PAGE_START_ADDRESS (0x0000000FFFFFC000ULL) /* In TTBR0 */ #define _COMM_PAGE_APRR_SUPPORT (_COMM_PAGE_START_ADDRESS + 0x10C) #define _COMM_PAGE_APPR_WRITE_ENABLE (_COMM_PAGE_START_ADDRESS + 0x110) #define _COMM_PAGE_APRR_WRITE_DISABLE (_COMM_PAGE_START_ADDRESS + 0x118) static __attribute__((__always_inline__)) bool jit_write_protect_supported(void) { /* Access shared kernel page at fixed memory location. */ uint8_t aprr_support = *(volatile uint8_t *)_COMM_PAGE_APRR_SUPPORT; return aprr_support > 0; } /* write protect enable = write disable */ static __attribute__((__always_inline__)) void jit_write_protect(int enabled) { /* Access shared kernel page at fixed memory location. */ uint8_t aprr_support = *(volatile uint8_t *)_COMM_PAGE_APRR_SUPPORT; if (aprr_support == 0 || aprr_support > 3) { return; } else if (aprr_support == 1) { __asm__ __volatile__ ( "mov x0, %0\n" "ldr x0, [x0]\n" "msr S3_4_c15_c2_7, x0\n" "isb sy\n" :: "r" (enabled ? _COMM_PAGE_APRR_WRITE_DISABLE : _COMM_PAGE_APPR_WRITE_ENABLE) : "memory", "x0" ); } else { __asm__ __volatile__ ( "mov x0, %0\n" "ldr x0, [x0]\n" "msr S3_6_c15_c1_5, x0\n" "isb sy\n" :: "r" (enabled ? _COMM_PAGE_APRR_WRITE_DISABLE : _COMM_PAGE_APPR_WRITE_ENABLE) : "memory", "x0" ); } } #else /* defined(__aarch64__) && defined(__APPLE__) */ static __attribute__((__always_inline__)) bool jit_write_protect_supported(void) { return false; } static __attribute__((__always_inline__)) void jit_write_protect(int enabled) { } #endif #endif /* define TCG_APPLE_JIT_H */ Then we have a successful jit allocation in the first process, but while it comes to stage 0, every allocation fails like that: make[3]: Entering directory '/buildroot/guile-3.0.9/libguile' cat alist.doc array-handle.doc array-map.doc arrays.doc async.doc atomic.doc backtrace.doc boolean.doc bitvectors.doc bytevectors.doc chars.doc control.doc continuations.doc debug.doc deprecated.doc deprecation.doc dynl.doc dynwind.doc eq.doc error.doc eval.doc evalext.doc exceptions.doc expand.doc extensions.doc fdes-finalizers.doc feature.doc filesys.doc fluids.doc foreign.doc fports.doc gc-malloc.doc gc.doc gettext.doc generalized-vectors.doc goops.doc gsubr.doc guardians.doc hash.doc hashtab.doc hooks.doc i18n.doc init.doc ioext.doc keywords.doc list.doc load.doc macros.doc mallocs.doc memoize.doc modules.doc numbers.doc objprop.doc options.doc pairs.doc ports.doc print.doc procprop.doc procs.doc promises.doc r6rs-ports.doc random.doc rdelim.doc read.doc rw.doc scmsigs.doc script.doc simpos.doc smob.doc sort.doc srcprop.doc srfi-1.doc srfi-4.doc srfi-13.doc srfi-14.doc srfi-60.doc stackchk.doc stacks.doc stime.doc strings.doc strorder.doc strports.doc struct.doc symbols.doc syntax.doc threads.doc throw.doc unicode.doc uniform.doc values.doc variable.doc vectors.doc version.doc vports.doc weak-set.doc weak-table.doc weak-vector.doc dynl.doc posix.doc net_db.doc socket.doc regex-posix.doc | GUILE_AUTO_COMPILE=0 ../meta/build-env guild snarf-check-and-output-texi > guile-procedures.texi || { rm guile-procedures.texi; false; } make[3]: Leaving directory '/buildroot/guile-3.0.9/libguile' make[2]: Leaving directory '/buildroot/guile-3.0.9/libguile' Making all in module make[2]: Entering directory '/buildroot/guile-3.0.9/module' make[2]: Nothing to be done for 'all'. make[2]: Leaving directory '/buildroot/guile-3.0.9/module' Making all in stage0 make[2]: Entering directory '/buildroot/guile-3.0.9/stage0' GUILE_BOOTSTRAP_STAGE=stage0 ../meta/build-env guild compile --target="aarch64-apple-darwin20.4.0" -W0 -O1 -L "/buildroot/guile-3.0.9/module" -o "ice-9/eval.go" "../module/ice-9/eval.scm" allocating JIT code buffer failed: Invalid argument JIT failed due to resource exhaustion disabling automatic JIT compilation The issue #47429 mentions a similar issue related with JIT allocation under Darwin20+, but alongside to the implementation of `allocate_code_arena` method in libguile/jit.c in 3.0.5 we don't have any processes regarding to the Darwin20+ JIT write protections, the culprit might different. Now what I can confirm is that - MAP_JIT is working on both macOS and iOS - If not providing an alternative method of pthread_write_protect_np, we then should properly handle the flags that passing to mmaped pointers to make sure RWX does not applied same time when doing W/X operations. - Current Guile does not have a working JIT on arm64 iOS if there's no HAVE_PTHREAD_JIT_WRITE_PROTECT_NP defined