From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?utf-8?Q?Gerd_M=C3=B6llmann?= Newsgroups: gmane.emacs.devel Subject: Re: MPS: Loaded pdump Date: Thu, 09 May 2024 13:20:15 +0200 Message-ID: References: <86wmo35jxb.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="37702"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: eller.helmut@gmail.com, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu May 09 13:21:12 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1s51po-0009Zv-6Z for ged-emacs-devel@m.gmane-mx.org; Thu, 09 May 2024 13:21:12 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s51p5-0002Hz-AY; Thu, 09 May 2024 07:20:27 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s51p3-0002HR-IR for emacs-devel@gnu.org; Thu, 09 May 2024 07:20:25 -0400 Original-Received: from mail-ej1-x633.google.com ([2a00:1450:4864:20::633]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s51oy-0006xv-QS; Thu, 09 May 2024 07:20:25 -0400 Original-Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-a59ce1e8609so327033266b.0; Thu, 09 May 2024 04:20:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715253617; x=1715858417; darn=gnu.org; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d7M2LQaOmD1pkmFjIcnMhZkG2mO8pi75wmi2OtnZjno=; b=cYoWp9VfamKJ9k257NDFoqd7ieaW8oWqZILvHpk55pyAOe3dOwBc/OJD95s7oMQVnY kb1Z2Da4Kz4ENz3eXbL6OPiL8p9H0/xtkziQ7HhbGxrLMe9KGDJv3D6cgYV4DNNa65Ce BvhXKUsLm7KtWIjgMbvjYgxqIn1ZbzBL0II/WQBc2JthuhVnhHnlTtWLFcZSsRBSeTGZ bxlH5U8V3ERYR9Bm/B3+SGReBqEwGb/WPZTFmEmJqJ5ruEXaIhTwlEtM/OjCYnfzr49Y kLG1uiMMTdNdJuzSgxrrcw24kmOjKPf2UxdlVheuZEQ7kUAG2wX7NBQ+FnZlbVIj/sNR jf+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715253617; x=1715858417; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=d7M2LQaOmD1pkmFjIcnMhZkG2mO8pi75wmi2OtnZjno=; b=g6FQeyACPiPL8spm2OmqzQb+QTVTiJz+K4KnzcPC72tfkpFza2m8+nYEmCpBqWEynh SKo6azDTqWyidsTdL9uiwmGT66nqpYId2IGyqGMDruxn+jqdazeFFBVRrJ8zy1P/hkkg hhaOMp78oy08vCvKRahVtNTEBeCKdellkZoDvF9HhbuDhm1wt4ujZuHm0qVWWVmiL2zD KhJSvphw9WvqWctw+UdEJlp937zuzu1SkcU6B0E8Bum7E9j/7TdgwvEBgvqSuaK6/yDg t62zaM9QQUdpF6jLaxjIrxPOvx3xANAyGRTBDPUxQjpUQvG7/agUiBiK7botHmo0AKhT 8o0g== X-Forwarded-Encrypted: i=1; AJvYcCVF8GEcQH7JCe3GmiojFVh69HdjgjDmlYKG9fYdwjazA54kKKHWwYQrAhuF6PUnZ53rOK0j1d/ol5EHn2ZcEo8gn1Gh X-Gm-Message-State: AOJu0YwRs7fME8Rv4J1EHFJGXyD5DKwdY3W6LR7le2yGEz0zA7kh7I2O /AD/IWTQtdZBYZF/XoM28qxPtCECKVzUeDFkbT7JgdNJP6+1R621aGUJrA== X-Google-Smtp-Source: AGHT+IFej6vOFMRlGBnYyM0bFkFBJCctYkoJX3F0f8LYwDKDKgNGIoQIx3LLIdm6+cWQfGKO4qX7FA== X-Received: by 2002:a17:906:fc04:b0:a55:adf5:e698 with SMTP id a640c23a62f3a-a5a11845831mr173612166b.28.1715253616615; Thu, 09 May 2024 04:20:16 -0700 (PDT) Original-Received: from pro2.fritz.box (pd9e36021.dip0.t-ipconnect.de. [217.227.96.33]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5733bea6586sm601445a12.14.2024.05.09.04.20.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 May 2024 04:20:16 -0700 (PDT) In-Reply-To: <86wmo35jxb.fsf@gnu.org> (Eli Zaretskii's message of "Thu, 09 May 2024 14:00:48 +0300") Received-SPF: pass client-ip=2a00:1450:4864:20::633; envelope-from=gerd.moellmann@gmail.com; helo=mail-ej1-x633.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:319070 Archived-At: Eli Zaretskii writes: >> From: Gerd M=C3=B6llmann >> Cc: Emacs Devel >> Date: Thu, 09 May 2024 12:52:03 +0200 >>=20 >> I feel more and more that the handling of the loaded pdump with MPS as >> it is now is not sustainable, and would like ask you for your ideas. >>=20 >> What we do now is make the hot part of the dump an ambig root. > > Is this specific to native-compilation, or is it not? It's not specific to native comp. > If it isn't, why does it come up only now? We've been running > MPS-built Emacs for a week at least; if that produces problems, can > you tell what problems and how can that be reproduced? As I said, I thought we could get away with the root, and that finding an alternative would be an optimization to be done later. (I think admin/igc.org has some ideas regarding the loaded pdump from some time ago.) What I wrote are fundamental problems, so they can't be reproduced. > IOW, I'd like to understand better why we need to make the hot part of > the dump an ambiguous root before we consider solutions. I think this mail I sent has the answer: Helmut Eller writes: >> @Helmut: Did we already talk about what the problem with the frame in >> the loaded pdump could be? Sorry that I don't remember. > > I never heard of that before. As the famous philosopher Manuel Manousakis says: Katastrophe! =F0=9F=99=82 Okay, I think one can understand this best when I try to describe what the pdumper does. Let's start with generating a pdump. I'll try to leave out as much details as a can. When we create a pdump, we start by allocating 3 big memory blocks which I'll call H (hot), C (cold), and R (relocs). We then traverse the graph of live objects like the old GC, starting from known roots. Each newly encountered object is copyied to H or C in binary form. C is used for leaf objects like strings and floats, H for the rest. The copying of objects is done by invoking type-specific functions, example dump_float, dump_vector, etc. We cannot use memcpy for the copying because we need more information when the pdump is loaded, namely relocation information, which goes to R. Relocation is necessary because both Emacs' DATA segment as well as H/C may end up at different addresses in a new process. Relocation info is recorded in S, and tells us where in the copied objects Lisp_Objects or pointers are that need patching when loaded. At the end, we write H, C, S to one big file. Good. Now let's load that file. We mmap the whole file and now have H', C', R' in the new process. C' and R'are good to go (In C are leaf objects). H' is patches according to the reloc info that is in S'. At the end of the relocation H' is ready to use. Some additional setup and initalizations, and we are good to go. I won't describe these. Thing is that H' now contains real Lisp objects of basically all types. Lisp objects contain references, so I make H' an ambig root. So far so good, but some Lisp objects contain not only references to other Lisp objects but also pointers to malloc'd memory. Not initially, in the dump, but during their lifetime. And finally we have reached face_cache. If initial_frame is an object in H', fix_frame won't be called for it. It cannot because the dump is not part of the MPS memory, and is instead traced ambigously as part of the big blob H'. Does that make any sense?