From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Helmut Eller Newsgroups: gmane.emacs.devel Subject: Re: MPS: Forwording symbols Date: Sun, 16 Jun 2024 21:27:07 +0200 Message-ID: <87jziod6yc.fsf@gmail.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="30688"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: Emacs Devel , Eli Zaretskii To: Gerd =?utf-8?Q?M=C3=B6llmann?= Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Jun 16 21:27:54 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1sIvXe-0007pJ-Bf for ged-emacs-devel@m.gmane-mx.org; Sun, 16 Jun 2024 21:27:54 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sIvX2-0001N0-Th; Sun, 16 Jun 2024 15:27:16 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sIvX0-0001Mp-PL for emacs-devel@gnu.org; Sun, 16 Jun 2024 15:27:14 -0400 Original-Received: from mail-ed1-x532.google.com ([2a00:1450:4864:20::532]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sIvWx-0004JC-Em; Sun, 16 Jun 2024 15:27:14 -0400 Original-Received: by mail-ed1-x532.google.com with SMTP id 4fb4d7f45d1cf-57c76497cefso4327741a12.1; Sun, 16 Jun 2024 12:27:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718566029; x=1719170829; darn=gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=YiqL5B1Z7DS97ifjCBmQKFIQe/KJk+tAuE0gfAIvpyg=; b=ctQ/CCSrJn/Ye2ggp7BiDr77Jk3UZjkpqifVT0wQTT6kh9sKIxQdITlRrPZeptjWvh SmMh/mrkX6/wNbtGR+7q2t08EKXa7+Q01ThZVKR3qDLrqrDXyHiZak4qi2u3kA4Qd8T+ 2mTzIXnr1Ss5JhF2dlbuGVgZoMAqlIefkKjVhWSZL00OhEb830F3STzQ7UtQflf6Fut7 Cddsdxd1dddajX/lcRQsJ7ELv+TL81RTl9BAq1/8/jesadgyxqku0nImpR2eTHUWfF+N TB7ES0JstJ07gqrYKFRFqckB+xvuABOMWcjgNI3kKdVdGsI0TPra99BbAD/V/deknp7M tc6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718566029; x=1719170829; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YiqL5B1Z7DS97ifjCBmQKFIQe/KJk+tAuE0gfAIvpyg=; b=Ls5azGoG/2WQ/0BGvvmB3LFrYMgvz5P55AU71ic21UAxjYPljBNLvgwJ2fQ2BUZj9D lpc3cy7z1w6X11Y/mo4IKnqPAWX1y1B5J08D2M8z/YU4rGdr/wZiSvY5hVShi8bnNI3e 0BMUuuDBJbDT0tQ7/J04BsK5PoIyHRpTe5uO0YsqrPRJdbfRZ3Tx5Y1jxP+G9DVdH6FW Qj1BJLLCJNlspcok9TS74px3gBxkeN0Ve3Nm36vYXzj+D5rkyEPRGHscAafGYLWGunJY zWDhafELoW/PBawOwLoAix2L076M+8ccQFvPGDJTL51CPsWNZk0vCJnb/UuUb+uTFxf2 McJg== X-Forwarded-Encrypted: i=1; AJvYcCUqPNeblr9eDzICJCvhiQsJyxtEc6Rpx2CrvayMb9hJWRbtgzsYflkxuC0lW+hIxSSIR7yai/7pFhB1B7k= X-Gm-Message-State: AOJu0YzBAoMW5AZZ82zwB3eYgVz6heLwq1gjj06RuXKCgnNUDQ5Q4FiP DmalAqTqO66dG6Aw2rT4U/CiNAJ1w6QC0XUbOt1NECLfaqzaFc96re/BeQ== X-Google-Smtp-Source: AGHT+IFlR19GWkH+EYqDKdsnDGNW8mmoFFnvJe2/fshLa73EBwDU5NkYbxQjhvOZAMnkow8kfsAL0A== X-Received: by 2002:a50:8e59:0:b0:57c:6234:e95b with SMTP id 4fb4d7f45d1cf-57cbd649713mr5322444a12.5.1718566029292; Sun, 16 Jun 2024 12:27:09 -0700 (PDT) Original-Received: from caladan (dial-189091.pool.broadband44.net. [212.46.189.91]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-57cb72e96b4sm5483595a12.50.2024.06.16.12.27.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Jun 2024 12:27:08 -0700 (PDT) In-Reply-To: ("Gerd =?utf-8?Q?M=C3=B6llman?= =?utf-8?Q?n=22's?= message of "Sun, 16 Jun 2024 11:43:11 +0200") Received-SPF: pass client-ip=2a00:1450:4864:20::532; envelope-from=eller.helmut@gmail.com; helo=mail-ed1-x532.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:320195 Archived-At: --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Sun, Jun 16 2024, Gerd M=C3=B6llmann wrote: > If we don't dump_fwd, we would have to make sure though not to overwrite > the existing values for symbols in lispsym, which happens in > dump_do_dump_relocation > > case RELOC_DUMP_TO_EMACS_PTR_RAW: > { > uintptr_t value =3D dump_read_word_from_dump (dump_base, reloc_of= fset); > eassert (dump_reloc_size (reloc) =3D=3D sizeof (value)); > value +=3D emacs_basis (); > dump_write_word_to_dump (dump_base, reloc_offset, value); > break; > } > > The name dump_write_word_to_dump is misleading. It does a memcpy to > Emacs' data segment. Hm, why do you think it writes to the data segment? Confusing names aside, it seems to read a value, which presumably is a pointer, adjusts it to the new basis and then writes it back. > Maybe one could introduce a new RELOC_xyz type to signify that a > forwarding symbol is patched and save the fwd value around the memcpy if > needed. > > Or something like that? Yes, looks like a better reloc kind is needed. Below is what I currently have. Tracing those IGC_OBJ_DUMPED_FWD objects seems problematic, especially while the binding of the symbol is temporarily swapped out. --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-WIP-Allocate-the-dump-as-a-single-MPS-block.patch >From 3e106ed1fb15a8c41f4d7296a4ec72d4736c6c55 Mon Sep 17 00:00:00 2001 From: Helmut Eller Date: Sun, 16 Jun 2024 20:18:01 +0200 Subject: [PATCH 1/3] WIP Allocate the dump as a single MPS block * src/igc.c (dump_mmap_contiguous_mps): New variant of dump_mmap_contiguous. (dump_mmap_release_mps): New. (pdumper_load): Call igc_on_pdump_loaded with more details. (dump_header): Record some important positions. (dump_fwd_int, dump_fwd_bool, dump_fwd_obj, dump_fwd_buffer_obj) (dump_fwd_kboard_obj): Add IGC_OBJ_DUMPED_FWD header. (dump_charset_table): Add IGC_OBJ_DUMPED_CHARSET_TABLE header. (dump_cold_charsets): New. Add IGC_OBJ_DUMPED_CODE_SPACE_MASKS header. (dump_cold_buffer): Add IGC_OBJ_DUMPED_BUFFER_TEXT header. (Fdump_emacs_portable): Record positions for cold_user_data_start and heap_end. * src/igc.h (igc_obj_type): Add some tags to mark object in the dump. (igc_on_pdump_loaded): Add some arguments for various regions. (igc_alloc_dump): New. * src/igc.c (igc_alloc_dump): Implement it. (igc_on_pdump_loaded): Insert some IGC_OBJ_PAD headers, pin some stuff, and prepare the text of the buffers. (builtin_obj_type_and_hash): Allow some of the new object types. --- src/igc.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++- src/igc.h | 12 ++- src/pdumper.c | 203 +++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 372 insertions(+), 39 deletions(-) diff --git a/src/igc.c b/src/igc.c index d9fb5a783f7..50fec431529 100644 --- a/src/igc.c +++ b/src/igc.c @@ -205,6 +205,11 @@ #define IGC_DEFINE_LIST(data) \ "IGC_OBJ_BUILTIN_SYMBOL", "IGC_OBJ_BUILTIN_THREAD", "IGC_OBJ_BUILTIN_SUBR", + "IGC_OBJ_DUMPED_FWD", + "IGC_OBJ_DUMPED_CHARSET_TABLE", + "IGC_OBJ_DUMPED_CODE_SPACE_MASKS", + "IGC_OBJ_DUMPED_BUFFER_TEXT", + "IGC_OBJ_DUMPED_BYTES", }; igc_static_assert (ARRAYELTS (obj_type_names) == IGC_OBJ_NUM_TYPES); @@ -1280,6 +1285,22 @@ fix_handler (mps_ss_t ss, struct handler *h) return MPS_RES_OK; } +static mps_res_t +fix_charset_table (mps_ss_t ss, struct charset *table, size_t nbytes) +{ + igc_assert (table == charset_table); + igc_assert (nbytes + == (charset_table_size * sizeof (struct charset) + + sizeof (struct igc_header))); + MPS_SCAN_BEGIN (ss) + { + for (size_t i = 0, len = nbytes / sizeof (struct charset); i < len; i++) + IGC_FIX12_OBJ (ss, &table[i].attributes); + } + MPS_SCAN_END (ss); + return MPS_RES_OK; +} + static mps_res_t fix_vector (mps_ss_t ss, struct Lisp_Vector *v); static mps_res_t @@ -1341,6 +1362,9 @@ dflt_scan_obj (mps_ss_t ss, mps_addr_t base_start, mps_addr_t base_limit, case IGC_OBJ_STRING_DATA: case IGC_OBJ_FLOAT: case IGC_OBJ_BYTES: + case IGC_OBJ_DUMPED_CODE_SPACE_MASKS: + case IGC_OBJ_DUMPED_BUFFER_TEXT: + case IGC_OBJ_DUMPED_BYTES: /* Can occur in the dump. */ break; @@ -1392,6 +1416,15 @@ dflt_scan_obj (mps_ss_t ss, mps_addr_t base_start, mps_addr_t base_limit, IGC_FIX_CALL_FN (ss, struct Lisp_Buffer_Local_Value, client, fix_blv); break; + + case IGC_OBJ_DUMPED_FWD: + IGC_FIX_CALL (ss, fix_fwd (ss, (lispfwd){ client })); + break; + + case IGC_OBJ_DUMPED_CHARSET_TABLE: + IGC_FIX_CALL (ss, fix_charset_table (ss, (struct charset *)client, + obj_size (header))); + break; } } MPS_SCAN_END (ss); @@ -2646,6 +2679,11 @@ finalize (struct igc *gc, mps_addr_t base) case IGC_OBJ_BUILTIN_SYMBOL: case IGC_OBJ_BUILTIN_THREAD: case IGC_OBJ_BUILTIN_SUBR: + case IGC_OBJ_DUMPED_FWD: + case IGC_OBJ_DUMPED_CHARSET_TABLE: + case IGC_OBJ_DUMPED_CODE_SPACE_MASKS: + case IGC_OBJ_DUMPED_BUFFER_TEXT: + case IGC_OBJ_DUMPED_BYTES: case IGC_OBJ_BYTES: case IGC_OBJ_NUM_TYPES: emacs_abort (); @@ -2813,6 +2851,11 @@ thread_ap (enum igc_obj_type type) case IGC_OBJ_BUILTIN_SYMBOL: case IGC_OBJ_BUILTIN_THREAD: case IGC_OBJ_BUILTIN_SUBR: + case IGC_OBJ_DUMPED_FWD: + case IGC_OBJ_DUMPED_CHARSET_TABLE: + case IGC_OBJ_DUMPED_CODE_SPACE_MASKS: + case IGC_OBJ_DUMPED_BUFFER_TEXT: + case IGC_OBJ_DUMPED_BYTES: case IGC_OBJ_NUM_TYPES: emacs_abort (); @@ -3600,6 +3643,17 @@ builtin_obj_type_and_hash (size_t *hash, enum igc_obj_type type, void *client) return IGC_OBJ_BUILTIN_SUBR; } + if (type == IGC_OBJ_DUMPED_FWD + || type == IGC_OBJ_DUMPED_CHARSET_TABLE + || type == IGC_OBJ_DUMPED_CODE_SPACE_MASKS + || type == IGC_OBJ_DUMPED_BUFFER_TEXT + || type == IGC_OBJ_DUMPED_BYTES + ) + { + *hash = 0; + return type; + } + emacs_abort (); } @@ -3860,6 +3914,12 @@ copy_dump (struct igc_mirror *m) case IGC_OBJ_BUILTIN_THREAD: case IGC_OBJ_BUILTIN_SUBR: break; + case IGC_OBJ_DUMPED_FWD: + case IGC_OBJ_DUMPED_CHARSET_TABLE: + case IGC_OBJ_DUMPED_CODE_SPACE_MASKS: + case IGC_OBJ_DUMPED_BUFFER_TEXT: + case IGC_OBJ_DUMPED_BYTES: + emacs_abort (); } } record_time (m, "Copy objects to MPS"); @@ -4415,6 +4475,11 @@ mirror (struct igc_mirror *m, void *org_base, void *copy_base) case IGC_OBJ_PAD: case IGC_OBJ_FWD: case IGC_OBJ_INVALID: + case IGC_OBJ_DUMPED_FWD: + case IGC_OBJ_DUMPED_CHARSET_TABLE: + case IGC_OBJ_DUMPED_CODE_SPACE_MASKS: + case IGC_OBJ_DUMPED_BUFFER_TEXT: + case IGC_OBJ_DUMPED_BYTES: case IGC_OBJ_NUM_TYPES: emacs_abort (); @@ -4549,19 +4614,144 @@ mirror_dump (void *start, void *end) redirect_roots (&m); if (getenv ("IGC_MIRROR_STATS")) - print_mirror_stats (&m); + print_mirror_stats (&m); } } +static mps_addr_t pinned_objects_in_dump[3]; + /* Called from pdumper_load. [START, END) is the hot section of the dump. Copy objects from the dump to MPS, and discard the dump. */ void -igc_on_pdump_loaded (void *start, void *end) +igc_on_pdump_loaded (void *dump_base, void *hot_start, void *hot_end, + void *cold_start, void *cold_end, + void *cold_user_data_start, void *heap_end) +{ + // mirror_dump (start, end); + igc_assert (global_igc->park_count > 0); + eassert (base_to_client (hot_start) == charset_table); + eassert (((struct igc_header *)cold_start)->obj_type + == IGC_OBJ_DUMPED_CODE_SPACE_MASKS); + eassert (((struct igc_header *)cold_user_data_start)->obj_type + == IGC_OBJ_DUMPED_BYTES); + eassert (((struct igc_header *)heap_end)->obj_type == IGC_OBJ_DUMPED_BYTES); + + size_t discardable_size = (uint8_t *)cold_start - (uint8_t *)hot_end; + size_t dump_size = (uint8_t *)cold_end - (uint8_t *)dump_base; + // size_t cold_size = (uint8_t *)cold_end - (uint8_t *)cold_start; + size_t dump_header_size = (uint8_t *)hot_start - (uint8_t *)dump_base; + size_t relocs_size = (uint8_t *)cold_end - (uint8_t *)heap_end; + struct igc_header *h = client_to_base (dump_base); + igc_assert (h->obj_type == IGC_OBJ_INVALID); + igc_assert (obj_size (h) == sizeof *h + dump_size); + igc_assert (discardable_size > 2 * sizeof *h); + /* Ignore dump_header */ + set_header (h, IGC_OBJ_PAD, sizeof *h + dump_header_size, 0); + /* Ignore discardable section */ + set_header (hot_end, IGC_OBJ_PAD, discardable_size, 0); + /* Ignore relocs */ + set_header (heap_end, IGC_OBJ_PAD, relocs_size, 0); + + /* Pin some stuff in the dump */ + mps_addr_t pinned_roots[] = { + charset_table, + base_to_client (cold_start), /* code_space_masks */ + base_to_client (cold_user_data_start), + }; + igc_static_assert (sizeof pinned_roots == sizeof pinned_objects_in_dump); + memcpy (pinned_objects_in_dump, pinned_roots, sizeof pinned_roots); + igc_root_create_ambig (pinned_objects_in_dump, + (uint8_t *)pinned_objects_in_dump + + sizeof pinned_objects_in_dump); + + /* Copy to buffer text to out of pdump */ + for (Lisp_Object l = Vbuffer_alist; !NILP (l); l = XCDR (l)) + { + struct buffer *buf = XBUFFER (XCDR (XCAR (l))); + eassert (pdumper_object_p (buf->text->beg)); + enlarge_buffer_text (buf, 0); + eassert (!pdumper_object_p (buf->text->beg)); + } +} + +void * +igc_alloc_dump (size_t nbytes) { - mirror_dump (start, end); + igc_assert (global_igc->park_count > 0); + mps_ap_t ap = thread_ap (IGC_OBJ_CONS); + mps_addr_t block; + do + { + mps_res_t res = mps_reserve (&block, ap, nbytes); + if (res != MPS_RES_OK) + memory_full (0); + } + while (!mps_commit (ap, block, nbytes)); + set_header (block, IGC_OBJ_INVALID, nbytes, 0); + return base_to_client (block); } +struct gap_it +{ + void *last_end; + struct pdumper_object_it it; +}; + +static void * +gap_it_next (void **gap_end, struct gap_it *it) +{ + if (!it->last_end) + { + it->last_end = (void *)dump_public.start; + return gap_it_next (gap_end, it); + } + else + { + for (;;) + { + struct igc_header *h = pdumper_next_object (&it->it); + if (!h) + { + if (it->last_end == (void *)dump_public.end) + return NULL; + else + { + void *gap_start = it->last_end; + it->last_end = *gap_end = (void *)dump_public.end; + return gap_start; + } + } + void *end = dflt_skip (h); + if (it->last_end == h) + { + it->last_end = end; + continue; + } + void *gap_start = it->last_end; + *gap_end = h; + it->last_end = end; + return gap_start; + } + } +} + +void +print_gaps (size_t start, size_t len) +{ + struct gap_it it = { 0 }; + for (size_t i = start; i < start + len; i++) + { + void *gap_end; + void *gap_start = gap_it_next (&gap_end, &it); + if (!gap_start) + return; + size_t offset = (size_t)gap_start - dump_public.start; + size_t len = (size_t)gap_end - (size_t)gap_start; + fprintf (stderr, "%p: %p %p %zu\n", (void *)offset, gap_start, gap_end, + len); + } +} /*********************************************************************** diff --git a/src/igc.h b/src/igc.h index 559b95bd7e8..e78ec659b8d 100644 --- a/src/igc.h +++ b/src/igc.h @@ -51,6 +51,11 @@ #define EMACS_IGC_H IGC_OBJ_BUILTIN_SYMBOL, IGC_OBJ_BUILTIN_THREAD, IGC_OBJ_BUILTIN_SUBR, + IGC_OBJ_DUMPED_FWD, + IGC_OBJ_DUMPED_CHARSET_TABLE, + IGC_OBJ_DUMPED_CODE_SPACE_MASKS, + IGC_OBJ_DUMPED_BUFFER_TEXT, + IGC_OBJ_DUMPED_BYTES, IGC_OBJ_NUM_TYPES }; @@ -65,7 +70,11 @@ #define EMACS_IGC_H void *igc_thread_add (struct thread_state *ts); void igc_thread_remove (void *info); void igc_on_idle (void); -void igc_on_pdump_loaded (void *start, void *end); +void igc_on_pdump_loaded (void *dump_base, + void *hot_start, void *hot_end, + void *cold_start, void *cold_end, + void *cold_user_data_start, + void *heap_end); void igc_on_face_cache_change (void *face_cache); void igc_process_messages (void); @@ -146,6 +155,7 @@ #define EMACS_IGC_H size_t igc_header_size (void); char *igc_dump_finish_obj (void *client, enum igc_obj_type type, char *base, char *end); +void *igc_alloc_dump (size_t nbytes); # define eassert_not_mps() eassert (false) #else diff --git a/src/pdumper.c b/src/pdumper.c index 269bc2d2050..c824da06b5e 100644 --- a/src/pdumper.c +++ b/src/pdumper.c @@ -373,6 +373,9 @@ dump_fingerprint (FILE *output, char const *label, # ifdef HAVE_MPS struct dump_table_locator igc_object_starts; + dump_off code_space_masks; + dump_off cold_user_data_start; + dump_off heap_end; # endif /* Relocation table for Emacs; each entry is a struct @@ -2360,71 +2363,82 @@ dump_fwd_int (struct dump_context *ctx, const struct Lisp_Intfwd *intfwd) #endif dump_emacs_reloc_immediate_intmax_t (ctx, intfwd->intvar, *intfwd->intvar); struct Lisp_Intfwd out; - dump_object_start_1 (ctx, &out, sizeof (out)); + //dump_object_start_1 (ctx, &out, sizeof (out)); + dump_object_start (ctx, intfwd, IGC_OBJ_DUMPED_FWD, &out, sizeof (out)); DUMP_FIELD_COPY (&out, intfwd, type); dump_field_emacs_ptr (ctx, &out, intfwd, &intfwd->intvar); - return dump_object_finish_1 (ctx, &out, sizeof (out)); + //return dump_object_finish_1 (ctx, &out, sizeof (out)); + return dump_object_finish (ctx, &out, sizeof (out)); } static dump_off dump_fwd_bool (struct dump_context *ctx, const struct Lisp_Boolfwd *boolfwd) { -#if CHECK_STRUCTS && !defined (HASH_Lisp_Boolfwd_0EA1C7ADCC) -# error "Lisp_Boolfwd changed. See CHECK_STRUCTS comment in config.h." +#if CHECK_STRUCTS && !defined(HASH_Lisp_Boolfwd_0EA1C7ADCC) +#error "Lisp_Boolfwd changed. See CHECK_STRUCTS comment in config.h." #endif dump_emacs_reloc_immediate_bool (ctx, boolfwd->boolvar, *boolfwd->boolvar); struct Lisp_Boolfwd out; - dump_object_start_1 (ctx, &out, sizeof (out)); + // dump_object_start_1 (ctx, &out, sizeof (out)); + dump_object_start (ctx, boolfwd, IGC_OBJ_DUMPED_FWD, &out, sizeof (out)); DUMP_FIELD_COPY (&out, boolfwd, type); dump_field_emacs_ptr (ctx, &out, boolfwd, &boolfwd->boolvar); - return dump_object_finish_1 (ctx, &out, sizeof (out)); + // return dump_object_finish_1 (ctx, &out, sizeof (out)); + return dump_object_finish (ctx, &out, sizeof (out)); } static dump_off dump_fwd_obj (struct dump_context *ctx, const struct Lisp_Objfwd *objfwd) { -#if CHECK_STRUCTS && !defined (HASH_Lisp_Objfwd_45D3E513DC) -# error "Lisp_Objfwd changed. See CHECK_STRUCTS comment in config.h." +#if CHECK_STRUCTS && !defined(HASH_Lisp_Objfwd_45D3E513DC) +#error "Lisp_Objfwd changed. See CHECK_STRUCTS comment in config.h." #endif if (NILP (Fgethash (dump_off_to_lisp (emacs_offset (objfwd->objvar)), - ctx->staticpro_table, - Qnil))) + ctx->staticpro_table, Qnil))) dump_emacs_reloc_to_lv (ctx, objfwd->objvar, *objfwd->objvar); struct Lisp_Objfwd out; - dump_object_start_1 (ctx, &out, sizeof (out)); + // dump_object_start_1 (ctx, &out, sizeof (out)); + dump_object_start (ctx, objfwd, IGC_OBJ_DUMPED_FWD, &out, sizeof (out)); DUMP_FIELD_COPY (&out, objfwd, type); dump_field_emacs_ptr (ctx, &out, objfwd, &objfwd->objvar); - return dump_object_finish_1 (ctx, &out, sizeof (out)); + // return dump_object_finish_1 (ctx, &out, sizeof (out)); + return dump_object_finish (ctx, &out, sizeof (out)); } static dump_off dump_fwd_buffer_obj (struct dump_context *ctx, - const struct Lisp_Buffer_Objfwd *buffer_objfwd) + const struct Lisp_Buffer_Objfwd *buffer_objfwd) { -#if CHECK_STRUCTS && !defined (HASH_Lisp_Buffer_Objfwd_611EBD13FF) -# error "Lisp_Buffer_Objfwd changed. See CHECK_STRUCTS comment in config.h." +#if CHECK_STRUCTS && !defined(HASH_Lisp_Buffer_Objfwd_611EBD13FF) +#error "Lisp_Buffer_Objfwd changed. See CHECK_STRUCTS comment in config.h." #endif struct Lisp_Buffer_Objfwd out; - dump_object_start_1 (ctx, &out, sizeof (out)); + // dump_object_start_1 (ctx, &out, sizeof (out)); + dump_object_start (ctx, buffer_objfwd, IGC_OBJ_DUMPED_FWD, &out, + sizeof (out)); DUMP_FIELD_COPY (&out, buffer_objfwd, type); DUMP_FIELD_COPY (&out, buffer_objfwd, offset); dump_field_lv (ctx, &out, buffer_objfwd, &buffer_objfwd->predicate, - WEIGHT_NORMAL); - return dump_object_finish_1 (ctx, &out, sizeof (out)); + WEIGHT_NORMAL); + //return dump_object_finish_1 (ctx, &out, sizeof (out)); + return dump_object_finish (ctx, &out, sizeof (out)); } static dump_off dump_fwd_kboard_obj (struct dump_context *ctx, - const struct Lisp_Kboard_Objfwd *kboard_objfwd) + const struct Lisp_Kboard_Objfwd *kboard_objfwd) { -#if CHECK_STRUCTS && !defined (HASH_Lisp_Kboard_Objfwd_CAA7E71069) -# error "Lisp_Intfwd changed. See CHECK_STRUCTS comment in config.h." +#if CHECK_STRUCTS && !defined(HASH_Lisp_Kboard_Objfwd_CAA7E71069) +#error "Lisp_Intfwd changed. See CHECK_STRUCTS comment in config.h." #endif struct Lisp_Kboard_Objfwd out; - dump_object_start_1 (ctx, &out, sizeof (out)); + // dump_object_start_1 (ctx, &out, sizeof (out)); + dump_object_start (ctx, kboard_objfwd, IGC_OBJ_DUMPED_FWD, &out, + sizeof (out)); DUMP_FIELD_COPY (&out, kboard_objfwd, type); DUMP_FIELD_COPY (&out, kboard_objfwd, offset); - return dump_object_finish_1 (ctx, &out, sizeof (out)); + // return dump_object_finish_1 (ctx, &out, sizeof (out)); + return dump_object_finish (ctx, &out, sizeof (out)); } static dump_off @@ -3425,6 +3439,9 @@ dump_charset_table (struct dump_context *ctx) struct dump_flags old_flags = ctx->flags; ctx->flags.pack_objects = true; dump_align_output (ctx, alignof (struct charset)); +# ifdef HAVE_MPS + dump_igc_start_obj (ctx, IGC_OBJ_DUMPED_CHARSET_TABLE, charset_table); +# endif dump_off offset = ctx->offset; if (dump_set_referrer (ctx)) ctx->current_referrer = build_string ("charset_table"); @@ -3437,6 +3454,9 @@ dump_charset_table (struct dump_context *ctx) dump_clear_referrer (ctx); dump_emacs_reloc_to_dump_ptr_raw (ctx, &charset_table, offset); ctx->flags = old_flags; +# ifdef HAVE_MPS + dump_igc_finish_obj (ctx); +# endif return offset; } @@ -3591,6 +3611,32 @@ dump_cold_charset (struct dump_context *ctx, Lisp_Object data) dump_write (ctx, cs->code_space_mask, 256); } +#ifdef HAVE_MPS +/* The charsets come all from the charset_table. Combine them to + a single IGC_OBJ_DUMPED_CODE_SPACE_MASKS object. + */ +static void +dump_cold_charsets (struct dump_context *ctx, Lisp_Object *cold_queue, + Lisp_Object data) +{ + dump_align_output (ctx, DUMP_ALIGNMENT); + dump_igc_start_obj (ctx, IGC_OBJ_DUMPED_CODE_SPACE_MASKS, charset_table); + eassert (!ctx->header.code_space_masks); + ctx->header.code_space_masks = ctx->offset; + for (;;) + { + dump_cold_charset (ctx, data); + Lisp_Object next = XCAR (*cold_queue); + enum cold_op op = (enum cold_op)XFIXNUM (XCAR (next)); + if (op != COLD_OP_CHARSET) + break; + data = XCDR (next); + *cold_queue = XCDR (*cold_queue); + } + dump_igc_finish_obj (ctx); +} +#endif + static void dump_cold_buffer (struct dump_context *ctx, Lisp_Object data) { @@ -3609,11 +3655,17 @@ dump_cold_buffer (struct dump_context *ctx, Lisp_Object data) + 1; if (nbytes > DUMP_OFF_MAX) error ("buffer too large"); +# ifdef HAVE_MPS + dump_igc_start_obj (ctx, IGC_OBJ_DUMPED_BUFFER_TEXT, b->own_text.beg); +# endif dump_remember_fixup_ptr_raw (ctx, buffer_offset + dump_offsetof (struct buffer, own_text.beg), ctx->offset); dump_write (ctx, b->own_text.beg, ptrdiff_t_to_dump_off (nbytes)); +# ifdef HAVE_MPS + dump_igc_finish_obj (ctx); +# endif } static void @@ -3685,7 +3737,11 @@ dump_drain_cold_data (struct dump_context *ctx) dump_cold_string (ctx, data); break; case COLD_OP_CHARSET: +#ifdef HAVE_MPS + dump_cold_charsets (ctx, &cold_queue, data); +#else dump_cold_charset (ctx, data); +#endif break; case COLD_OP_BUFFER: dump_cold_buffer (ctx, data); @@ -4479,6 +4535,13 @@ DEFUN ("dump-emacs-portable", never change and so can be direct-mapped from the dump without special processing. */ dump_drain_cold_data (ctx); +# ifdef HAVE_MPS + dump_align_output (ctx, DUMP_ALIGNMENT); + fprintf (stderr, "cold user data: %x\n", (unsigned)ctx->offset); + ctx->header.cold_user_data_start = ctx->offset; + dump_igc_start_obj (ctx, IGC_OBJ_DUMPED_BYTES, &discardable_end); +# endif + /* dump_drain_user_remembered_data_cold needs to be after dump_drain_cold_data in case dump_drain_cold_data dumps a lisp object to which C code points. @@ -4486,10 +4549,20 @@ DEFUN ("dump-emacs-portable", objects have been dumped. */ dump_drain_user_remembered_data_cold (ctx); +# ifdef HAVE_MPS + dump_align_output (ctx, DUMP_ALIGNMENT); + dump_igc_finish_obj (ctx); + fprintf (stderr, "heap end: %x\n", (unsigned)ctx->offset); +# endif /* After this point, the dump file contains no data that can be part of the Lisp heap. */ ctx->end_heap = ctx->offset; +# ifdef HAVE_MPS + ctx->header.heap_end = ctx->offset; + dump_igc_start_obj (ctx, IGC_OBJ_DUMPED_BYTES, &discardable_end); +# endif + /* Make remembered modifications to the dump file itself. */ dump_do_fixups (ctx); @@ -4520,6 +4593,10 @@ DEFUN ("dump-emacs-portable", drain_reloc_list (ctx, dump_emit_emacs_reloc, dump_merge_emacs_relocs, &ctx->emacs_relocs, &ctx->header.emacs_relocs); +# ifdef HAVE_MPS + fprintf (stderr, "cold end: %x\n", (unsigned)ctx->offset); + dump_igc_finish_obj (ctx); +# endif const dump_off cold_end = ctx->offset; eassert (dump_queue_empty_p (&ctx->dump_queue)); @@ -5072,6 +5149,49 @@ dump_mmap_contiguous_heap (struct dump_memory_map *maps, int nr_maps, return ret; } +#ifdef HAVE_MPS + +static void +dump_mmap_release_mps (struct dump_memory_map *map) +{ + emacs_abort (); +} + +/* Implement dump_mmap using mps_reserve and read. */ +static bool +dump_mmap_contiguous_mps (struct dump_memory_map *maps, int nr_maps, + size_t total_size) +{ + uint8_t *p = igc_alloc_dump (igc_header_size () + total_size); + for (size_t i = 0; i < nr_maps; ++i) + { + struct dump_memory_map *map = &maps[i]; + const struct dump_memory_map_spec spec = map->spec; + if (!spec.size) + continue; + map->mapping = p; + map->release = dump_mmap_release_mps; + map->private = NULL; + if (spec.fd < 0) + memset (map->mapping, 0, spec.size); + else + { + if (lseek (spec.fd, spec.offset, SEEK_SET) < 0) + return false; + ssize_t nb = dump_read_all (spec.fd, map->mapping, spec.size); + if (nb != spec.size) + { + if (nb >= 0) + errno = EIO; + return false; + } + } + p += spec.size; + } + return true; +} +#endif + static void dump_mmap_release_vm (struct dump_memory_map *map) { @@ -5206,12 +5326,17 @@ dump_mmap_contiguous (struct dump_memory_map *maps, int nr_maps) eassert (maps[i].release == NULL); eassert (maps[i].private == NULL); if (i != nr_maps - 1) - eassert (maps[i].spec.size % worst_case_page_size == 0); + eassert (maps[i].spec.size % worst_case_page_size == 0); total_size += maps[i].spec.size; } - return (VM_SUPPORTED ? dump_mmap_contiguous_vm : dump_mmap_contiguous_heap) - (maps, nr_maps, total_size); +#ifdef HAVE_MPS + return dump_mmap_contiguous_mps (maps, nr_maps, total_size); +#else + return (VM_SUPPORTED + ? dump_mmap_contiguous_vm + : dump_mmap_contiguous_heap) (maps, nr_maps, total_size); +#endif } typedef uint_fast32_t dump_bitset_word; @@ -5953,14 +6078,6 @@ pdumper_load (const char *dump_filename, char *argv0) dump_public.start = dump_base; dump_public.end = dump_public.start + dump_size; -#ifdef HAVE_MPS - size_t aligned_header_size - = ((sizeof (struct dump_header) + DUMP_ALIGNMENT - 1) - & ~(DUMP_ALIGNMENT - 1)); - void *hot_start = (void *) (dump_base + aligned_header_size); - void *hot_end = (void *) (dump_base + header->discardable_start); -#endif - dump_do_all_dump_reloc_for_phase (header, dump_base, EARLY_RELOCS); dump_do_all_emacs_relocations (header, dump_base); @@ -5991,8 +6108,24 @@ pdumper_load (const char *dump_filename, char *argv0) dump_do_all_dump_reloc_for_phase (header, dump_base, LATE_RELOCS); dump_do_all_dump_reloc_for_phase (header, dump_base, VERY_LATE_RELOCS); +#ifdef HAVE_MPS + size_t aligned_header_size + = ((sizeof (struct dump_header) + DUMP_ALIGNMENT - 1) + & ~(DUMP_ALIGNMENT - 1)); + void *hot_start = (void *) (dump_base + aligned_header_size); + void *hot_end = (void *) (dump_base + header->discardable_start); + void *cold_start = (void *) (dump_base + header->cold_start); + void *cold_end = (void *) (dump_base + dump_size); + void *cold_user_data_start = (void *) (dump_base + + header->cold_user_data_start); + void *heap_end = (void *) (dump_base + header->heap_end); +#endif + # ifdef HAVE_MPS - igc_on_pdump_loaded (hot_start, hot_end); + igc_on_pdump_loaded ((void *)dump_base, + hot_start, hot_end, + cold_start, cold_end, + cold_user_data_start, heap_end); # endif /* Run the functions Emacs registered for doing post-dump-load -- 2.39.2 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0002-WIP-Trace-the-FWD-objects-in-the-dump.patch >From b8bd9c8e7586c0a71281711ded92816c6d42a0aa Mon Sep 17 00:00:00 2001 From: Helmut Eller Date: Sun, 16 Jun 2024 20:33:41 +0200 Subject: [PATCH 2/3] WIP Trace the FWD objects in the dump. * src/igc.c (fix_symbol): --- src/igc.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/igc.c b/src/igc.c index 50fec431529..56154d0a538 100644 --- a/src/igc.c +++ b/src/igc.c @@ -787,6 +787,7 @@ fix_symbol (mps_ss_t ss, struct Lisp_Symbol *sym) break; case SYMBOL_FORWARDED: + IGC_FIX12_RAW (ss, &sym->u.s.val.fwd); /* for IGC_OBJ_DUMPED_FWD */ IGC_FIX_CALL (ss, fix_fwd (ss, sym->u.s.val.fwd)); break; } -- 2.39.2 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0003-WIP-Remove-uses-of-pdumper_object_p.patch >From e287a4c8b3264eccf7a40dce105342edf7c96090 Mon Sep 17 00:00:00 2001 From: Helmut Eller Date: Sun, 16 Jun 2024 20:35:05 +0200 Subject: [PATCH 3/3] WIP Remove uses of pdumper_object_p * src/igc.c (is_mps, igc_xfree, igc_hash): After the first collection, pdumper_object_p doesn't make much sense any more. --- src/igc.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/src/igc.c b/src/igc.c index 56154d0a538..044c24d4ac8 100644 --- a/src/igc.c +++ b/src/igc.c @@ -126,7 +126,8 @@ is_pure (const mps_addr_t addr) static bool is_mps (const mps_addr_t addr) { - return addr >= min_addr && addr < max_addr && !pdumper_object_p (addr) + return addr >= min_addr && addr < max_addr + //&& !pdumper_object_p (addr) && !c_symbol_p (addr) && !is_pure (addr); } @@ -2385,7 +2386,9 @@ igc_realloc_ambig (void *block, size_t size) void igc_xfree (void *p) { - if (p == NULL || pdumper_object_p (p)) + if (p == NULL + // || pdumper_object_p (p) + ) return; struct igc_root_list *r = root_find (p); igc_assert (r != NULL); @@ -2956,7 +2959,9 @@ igc_hash (Lisp_Object key) } /* Objects in the The dump have igc_headers, too. */ - if (is_mps (client) || pdumper_object_p (client)) + if (is_mps (client) + //|| pdumper_object_p (client) + ) { // The following assertion is very expensive. // igc_assert (mps_arena_has_addr (global_igc->arena, client)); -- 2.39.2 --=-=-=--