* bug#61476: mmap() with RWX cannot correctly invalidate cache
@ 2023-02-13 9:09 Torrekie
0 siblings, 0 replies; only message in thread
From: Torrekie @ 2023-02-13 9:09 UTC (permalink / raw)
To: 61476
[-- Attachment #1: Type: text/plain, Size: 13980 bytes --]
I'm building Guile 3.0.9 on iOS 14.5.1 (Darwin 20.4.0) which is suffering some system API differences within pthread_jit* stuff.
cat alist.doc array-handle.doc array-map.doc arrays.doc async.doc atomic.doc backtrace.doc boolean.doc bitvectors.doc bytevectors.doc chars.doc control.doc continuations.doc debug.doc deprecated.doc deprecation.doc dynl.doc dynwind.doc eq.doc error.doc eval.doc evalext.doc exceptions.doc expand.doc extensions.doc fdes-finalizers.doc feature.doc filesys.doc fluids.doc foreign.doc fports.doc gc-malloc.doc gc.doc gettext.doc generalized-vectors.doc goops.doc gsubr.doc guardians.doc hash.doc hashtab.doc hooks.doc i18n.doc init.doc ioext.doc keywords.doc list.doc load.doc macros.doc mallocs.doc memoize.doc modules.doc numbers.doc objprop.doc options.doc pairs.doc ports.doc print.doc procprop.doc procs.doc promises.doc r6rs-ports.doc random.doc rdelim.doc read.doc rw.doc scmsigs.doc script.doc simpos.doc smob.doc sort.doc srcprop.doc srfi-1.doc srfi-4.doc srfi-13.doc srfi-14.doc srfi-60.doc stackchk.doc stacks.doc stime.doc strings.doc strorder.doc strports.doc struct.doc symbols.doc syntax.doc threads.doc throw.doc unicode.doc uniform.doc values.doc variable.doc vectors.doc version.doc vports.doc weak-set.doc weak-table.doc weak-vector.doc dynl.doc posix.doc net_db.doc socket.doc regex-posix.doc | GUILE_AUTO_COMPILE=0 ../meta/build-env guild snarf-check-and-output-texi > guile-procedures.texi || { rm guile-procedures.texi; false; }
cat: write error: Broken pipe
After inspecting the crash logs, it might be some unexpected freeing of mmap cache, the librecompat.0.dylib includes a sys_icache_invalidate wrapper for __clear_cache function (as the same implementation with macOS one since they are all come from LLVM) which has not been compiled to system libcompiler_rt.dylib for iOS.
Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Subtype: KERN_PROTECTION_FAILURE at 0x0000000104ccb9d8
VM Region Info: 0x104ccb9d8 is in 0x104cc4000-0x104ccc000; bytes after start: 31192 bytes before end: 1575
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
__LINKEDIT 104bc0000-104cc4000 [ 1040K] r--/rw- SM=COW ...e-3.0.1.dylib
---> __TEXT 104cc4000-104ccc000 [ 32K] r--/rw- SM=COW ...ompat.0.dylib
__DATA_CONST 104ccc000-104cd0000 [ 16K] rw-/rw- SM=COW ...ompat.0.dylib
Termination Signal: Bus error: 10
Termination Reason: Namespace SIGNAL, Code 0xa
Terminating Process: exc handler [9957]
Triggered by Thread: 0
Thread 0 name: Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0 + librecompat.0.dylib 0x104ccb9d8 0x104cc4000 + 0x79d8 // __clear_cache + 0x0
1 libguile-3.0.1.dylib 0x104b4b728 0x104a44000 + 0x107728 // /buildroot/guile-3.0.9/libguile/./lightening/lightening/aarch64.c:217
2 libguile-3.0.1.dylib 0x104b4b3b0 0x104a44000 + 0x1073b0 // /buildroot/guile-3.0.9/libguile/./lightening/lightening/lightening.c:241
3 libguile-3.0.1.dylib 0x104aa12e0 0x104a44000 + 0x5d2e0 // /buildroot/guile-3.0.9/libguile/jit.c:1424
4 libguile-3.0.1.dylib 0x104aa0c38 0x104a44000 + 0x5cc38 // /buildroot/guile-3.0.9/libguile/jit.c:5895
5 libsystem_pthread.dylib 0x1def01f60 0x1deeff000 + 0x2f60 // __pthread_once_handler + 0x50
6 libsystem_platform.dylib 0x1deefb964 0x1deef7000 + 0x4964 // _os_once_callout + 0x20
7 libsystem_pthread.dylib 0x1def01ef4 0x1deeff000 + 0x2ef4 // pthread_once + 0x64
8 libguile-3.0.1.dylib 0x104aa0558 0x104a44000 + 0x5c558 // /buildroot/guile-3.0.9/libguile/jit.c:0
9 libguile-3.0.1.dylib 0x104aa0410 0x104a44000 + 0x5c410 // /buildroot/guile-3.0.9/libguile/jit.c:6031
10 libguile-3.0.1.dylib 0x104b35774 0x104a44000 + 0xf1774 // /buildroot/guile-3.0.9/libguile/./vm-engine.c:370
11 libguile-3.0.1.dylib 0x104b323d0 0x104a44000 + 0xee3d0 // /buildroot/guile-3.0.9/libguile/vm.c:1615
12 libguile-3.0.1.dylib 0x104a6776c 0x104a44000 + 0x2376c // /buildroot/guile-3.0.9/libguile/eval.c:496
13 libguile-3.0.1.dylib 0x104af9e20 0x104a44000 + 0xb5e20 // /buildroot/guile-3.0.9/libguile/read.c:1553
14 libguile-3.0.1.dylib 0x104abb158 0x104a44000 + 0x77158 // /buildroot/guile-3.0.9/libguile/load.c:124
15 libguile-3.0.1.dylib 0x104abc3d4 0x104a44000 + 0x783d4 // /buildroot/guile-3.0.9/libguile/load.c:1267
16 libguile-3.0.1.dylib 0x104abcb28 0x104a44000 + 0x78b28 // /buildroot/guile-3.0.9/libguile/load.c:1275
17 libguile-3.0.1.dylib 0x104a8e0ec 0x104a44000 + 0x4a0ec // /buildroot/guile-3.0.9/libguile/init.c:230
18 libguile-3.0.1.dylib 0x104a8e458 0x104a44000 + 0x4a458 // /buildroot/guile-3.0.9/libguile/init.c:510
19 libguile-3.0.1.dylib 0x104b2a89c 0x104a44000 + 0xe689c // /buildroot/guile-3.0.9/libguile/threads.c:578
20 libguile-3.0.1.dylib 0x104b2d47c 0x104a44000 + 0xe947c // /buildroot/guile-3.0.9/libguile/threads.c:642
21 + libgc.1.dylib 0x1053688e4 0x105354000 + 0x148e4 // /buildroot/gc-8.2.2/misc.c:2173
22 libguile-3.0.1.dylib 0x104b2a97c 0x104a44000 + 0xe697c // /buildroot/guile-3.0.9/libguile/threads.c:692
23 libguile-3.0.1.dylib 0x104b2a930 0x104a44000 + 0xe6930 // /buildroot/guile-3.0.9/libguile/threads.c:698
24 libguile-3.0.1.dylib 0x104a8e190 0x104a44000 + 0x4a190 // /buildroot/guile-3.0.9/libguile/init.c:295
25 guile (*) 0x104a37d90 0x104a30000 + 0x7d90 // /buildroot/guile-3.0.9/libguile/guile.c:94
26 libdyld.dylib 0x193c92cf8 0x193c91000 + 0x1cf8 // start + 0x4
But according to the JIT allocation problems which has been met by other projects:
https://patchwork.kernel.org/project/qemu-devel/patch/20201108232425.1705-7-j@getutm.app/#23841029 <https://patchwork.kernel.org/project/qemu-devel/patch/20201108232425.1705-7-j@getutm.app/#23841029>
I attempted to patch libguile/jit.c like that:
--- libguile/jit.c 1676274604.835054871
+++ libguile/jit.c.old 1676204975.415324076
@@ -47,8 +47,17 @@
#include <sys/mman.h>
#endif
-#if defined __APPLE__ && HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
+#if defined __APPLE__
#include <libkern/OSCacheControl.h>
+#include "tcg-apple-jit.h"
+
+static void tb_exec_change(bool locked)
+{
+ if (jit_write_protect_supported()) {
+ jit_write_protect(locked);
+ }
+}
+
#endif
#include "jit.h"
@@ -1414,8 +1435,9 @@
uint8_t *ret = jit_address (j->jit);
-#if defined __APPLE__ && HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
- pthread_jit_write_protect_np(0);
+#if defined __APPLE__
+// pthread_jit_write_protect_np(0);
+ tb_exec_change(0);
#endif
emit (j);
@@ -1423,10 +1445,11 @@
size_t size;
if (!jit_has_overflow (j->jit) && jit_end (j->jit, &size))
{
-#if defined __APPLE__ && HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
+#if defined __APPLE__
/* protect previous code arena. leave unprotected after emit()
since jit_end() also writes to code arena. */
- pthread_jit_write_protect_np(1);
+// pthread_jit_write_protect_np(1);
+ tb_exec_change(1);
sys_icache_invalidate(arena->base, arena->size);
#endif
ASSERT (size <= (arena->size - arena->used));
@@ -1442,9 +1465,10 @@
}
else
{
-#if defined __APPLE__ && HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
+#if defined __APPLE__
/* protect previous code arena */
- pthread_jit_write_protect_np(1);
+// pthread_jit_write_protect_np(1);
+ tb_exec_change(1);
sys_icache_invalidate(arena->base, arena->size);
#endif
jit_reset (j->jit);
Which replaces the previous `pthread_jit_write_protect_np` with another iOS capable implementation that provided in the link above:
/*
* Apple Silicon functions for JIT handling
*
* Copyright (c) 2020 osy
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, see <http://www.gnu.org/licenses/>.
*/
#ifndef TCG_APPLE_JIT_H
#define TCG_APPLE_JIT_H
/*
* APRR handling
* Credits to: https://siguza.github.io/APRR/
* Reversed from /usr/lib/system/libsystem_pthread.dylib
*/
#if defined(__aarch64__) && defined(__APPLE__)
#define _COMM_PAGE_START_ADDRESS (0x0000000FFFFFC000ULL) /* In TTBR0 */
#define _COMM_PAGE_APRR_SUPPORT (_COMM_PAGE_START_ADDRESS + 0x10C)
#define _COMM_PAGE_APPR_WRITE_ENABLE (_COMM_PAGE_START_ADDRESS + 0x110)
#define _COMM_PAGE_APRR_WRITE_DISABLE (_COMM_PAGE_START_ADDRESS + 0x118)
static __attribute__((__always_inline__)) bool jit_write_protect_supported(void)
{
/* Access shared kernel page at fixed memory location. */
uint8_t aprr_support = *(volatile uint8_t *)_COMM_PAGE_APRR_SUPPORT;
return aprr_support > 0;
}
/* write protect enable = write disable */
static __attribute__((__always_inline__)) void jit_write_protect(int enabled)
{
/* Access shared kernel page at fixed memory location. */
uint8_t aprr_support = *(volatile uint8_t *)_COMM_PAGE_APRR_SUPPORT;
if (aprr_support == 0 || aprr_support > 3) {
return;
} else if (aprr_support == 1) {
__asm__ __volatile__ (
"mov x0, %0\n"
"ldr x0, [x0]\n"
"msr S3_4_c15_c2_7, x0\n"
"isb sy\n"
:: "r" (enabled ? _COMM_PAGE_APRR_WRITE_DISABLE
: _COMM_PAGE_APPR_WRITE_ENABLE)
: "memory", "x0"
);
} else {
__asm__ __volatile__ (
"mov x0, %0\n"
"ldr x0, [x0]\n"
"msr S3_6_c15_c1_5, x0\n"
"isb sy\n"
:: "r" (enabled ? _COMM_PAGE_APRR_WRITE_DISABLE
: _COMM_PAGE_APPR_WRITE_ENABLE)
: "memory", "x0"
);
}
}
#else /* defined(__aarch64__) && defined(__APPLE__) */
static __attribute__((__always_inline__)) bool jit_write_protect_supported(void)
{
return false;
}
static __attribute__((__always_inline__)) void jit_write_protect(int enabled)
{
}
#endif
#endif /* define TCG_APPLE_JIT_H */
Then we have a successful jit allocation in the first process, but while it comes to stage 0, every allocation fails like that:
make[3]: Entering directory '/buildroot/guile-3.0.9/libguile'
cat alist.doc array-handle.doc array-map.doc arrays.doc async.doc atomic.doc backtrace.doc boolean.doc bitvectors.doc bytevectors.doc chars.doc control.doc continuations.doc debug.doc deprecated.doc deprecation.doc dynl.doc dynwind.doc eq.doc error.doc eval.doc evalext.doc exceptions.doc expand.doc extensions.doc fdes-finalizers.doc feature.doc filesys.doc fluids.doc foreign.doc fports.doc gc-malloc.doc gc.doc gettext.doc generalized-vectors.doc goops.doc gsubr.doc guardians.doc hash.doc hashtab.doc hooks.doc i18n.doc init.doc ioext.doc keywords.doc list.doc load.doc macros.doc mallocs.doc memoize.doc modules.doc numbers.doc objprop.doc options.doc pairs.doc ports.doc print.doc procprop.doc procs.doc promises.doc r6rs-ports.doc random.doc rdelim.doc read.doc rw.doc scmsigs.doc script.doc simpos.doc smob.doc sort.doc srcprop.doc srfi-1.doc srfi-4.doc srfi-13.doc srfi-14.doc srfi-60.doc stackchk.doc stacks.doc stime.doc strings.doc strorder.doc strports.doc struct.doc symbols.doc syntax.doc threads.doc throw.doc unicode.doc uniform.doc values.doc variable.doc vectors.doc version.doc vports.doc weak-set.doc weak-table.doc weak-vector.doc dynl.doc posix.doc net_db.doc socket.doc regex-posix.doc | GUILE_AUTO_COMPILE=0 ../meta/build-env guild snarf-check-and-output-texi > guile-procedures.texi || { rm guile-procedures.texi; false; }
make[3]: Leaving directory '/buildroot/guile-3.0.9/libguile'
make[2]: Leaving directory '/buildroot/guile-3.0.9/libguile'
Making all in module
make[2]: Entering directory '/buildroot/guile-3.0.9/module'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/buildroot/guile-3.0.9/module'
Making all in stage0
make[2]: Entering directory '/buildroot/guile-3.0.9/stage0'
GUILE_BOOTSTRAP_STAGE=stage0 ../meta/build-env guild compile --target="aarch64-apple-darwin20.4.0" -W0 -O1 -L "/buildroot/guile-3.0.9/module" -o "ice-9/eval.go" "../module/ice-9/eval.scm"
allocating JIT code buffer failed: Invalid argument
JIT failed due to resource exhaustion
disabling automatic JIT compilation
The issue #47429 mentions a similar issue related with JIT allocation under Darwin20+, but alongside to the implementation of `allocate_code_arena` method in libguile/jit.c in 3.0.5 we don't have any processes regarding to the Darwin20+ JIT write protections, the culprit might different.
Now what I can confirm is that
- MAP_JIT is working on both macOS and iOS
- If not providing an alternative method of pthread_write_protect_np, we then should properly handle the flags that passing to mmaped pointers to make sure RWX does not applied same time when doing W/X operations.
- Current Guile does not have a working JIT on arm64 iOS if there's no HAVE_PTHREAD_JIT_WRITE_PROTECT_NP defined
[-- Attachment #2: Type: text/html, Size: 16304 bytes --]
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2023-02-13 9:09 UTC | newest]
Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-13 9:09 bug#61476: mmap() with RWX cannot correctly invalidate cache Torrekie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).