From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: Loading tramp for dump goes into infinite regress Date: Tue, 9 Aug 2022 08:29:13 -0400 Message-ID: References: <8735erhrlg.fsf@gmx.de> <83wnc2g0n8.fsf@gnu.org> <83sfmqfxcb.fsf@gnu.org> <83a68yfp6j.fsf@gnu.org> <83r129e1op.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="27650"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org To: Eli Zaretskii , Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Tue Aug 09 14:32:04 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oLOOy-000743-3X for ged-emacs-devel@m.gmane-mx.org; Tue, 09 Aug 2022 14:32:04 +0200 Original-Received: from localhost ([::1]:56984 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oLOOw-0007YS-BT for ged-emacs-devel@m.gmane-mx.org; Tue, 09 Aug 2022 08:32:02 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:48148) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oLOMV-0006cx-Ed for emacs-devel@gnu.org; Tue, 09 Aug 2022 08:29:31 -0400 Original-Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]:38805) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oLOMT-0006JV-OY; Tue, 09 Aug 2022 08:29:31 -0400 Original-Received: by mail-pf1-x42c.google.com with SMTP id d20so10646190pfq.5; Tue, 09 Aug 2022 05:29:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=dBIX8qOkdwGhPM7T6DznMrUs0/pR+3F6ORtGKrUEqFQ=; b=eZa+ahjfJYq8yKkHfcYIye4Hz6yoaNRzPRAHF6keeMGxwWsSsWFWN2VcWkxvYec3yp GGTZxUr4a9H+Od8BBFwTd9mXoVL7HsdXCFD6SC6b4i+RaECatLJMfOmz9Uk+6kX/Axnw fKscAw2KJZwt9y68bly3ahIM+pHedelXy91nBDAS73dHO7dNARdjn7RUEQawltINnVtJ zrDJ8mmRIG8fPnKuPOUdX7hGadt/MQ+vy/00xC/PdcRA7/lPNTxn1UjQcTvdVAIS8hQ9 0ySME/4QEMEdZCX11nAd0n06IXNEOPNHiFyt8iGREzNEL0j/8Tn9Z1WH7CJvzksn5eUT AvsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=dBIX8qOkdwGhPM7T6DznMrUs0/pR+3F6ORtGKrUEqFQ=; b=5JV/es+S5dhgb0ogQuvXjrkAZ4rJUJbpzA3Uv1LjZ28bmG7GyCuGc4W08QjWMsPlKM w9ffacR8W7VHUEfiCGM/hsXVHSNq4dGqEuJHOo8b46xiMV0KVi0APmSyXYjod1WmL5CR QLoFpF8pMIO4IlXbp2sl2WEx7SDQdhVSFPGDpbwk1nwgRm35RhtfbUkLa8FODe6+vWiV EYWW86gOL34FsmX/j13darV/rstPA1b25rK+2zfUkJjSiY3bWvjxbzk2tVaoPSnsBZeh 4C735h47A8Kp4efFgWsfRy4JPj9WHhF93acGy8xN2DLpsveM50DMf5B32Kz1mHJZf+ld qQnw== X-Gm-Message-State: ACgBeo1VoJyqaP+HvsP2rX0zNU8UogLRp+itPYGUguSJSW7swbdnzCie ddbNqecKoavKp9yu0ZGcqSu7mfjUYGBpgERxEcQdFDJW X-Google-Smtp-Source: AA6agR5VsFLmC1Xg/jcSoCNts7zz+s3JPy0JGFH5wfds7iQoGdjGExp6sChlAFB4H4LSESQTSxl/FYtv6QzJ5UtZC6k= X-Received: by 2002:a63:2d1:0:b0:41d:9a9e:f2ae with SMTP id 200-20020a6302d1000000b0041d9a9ef2aemr5899287pgc.488.1660048167051; Tue, 09 Aug 2022 05:29:27 -0700 (PDT) In-Reply-To: <83r129e1op.fsf@gnu.org> Received-SPF: pass client-ip=2607:f8b0:4864:20::42c; envelope-from=owinebar@gmail.com; helo=mail-pf1-x42c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:293304 Archived-At: I wanted to circle back and answer this now that I have a working "mega-dump". On Mon, Jul 25, 2022 at 9:56 AM Eli Zaretskii wrote: > > Another benefit I expect from native-compilation, dumped or not, is more efficient memory use when running > > multiple emacs processes. With dumping, I would expect (or hope for) better garbage collector behavior > > since the amount of allocation required for the loaded modules should be pre-determined (whether byte- or > > native-compiled). If the image is 300MB (including the shared libraries), so be it, as long as the memory is > > shared between multiple processes. > > I don't think I understand this expectation, and I don't think > natively-compiled code has any advantages wrt GC over the > byte-compiled code. There should be at least one advantage - code representing function calls to other byte code (even in the same compilation unit) are represented by references in the heap which must be traced by the GC. Take this with a block of salt, as I have not verified by inspection of the natively compiled code, but many of those heap references should be translated to control transfers to other code addresses, which are not in the heap allocated or traced by the GC. But that isn't what I was referring to above. The first statement was intended with regard to the efficiency of using shared libraries with code in read-only pages between multiple processes versus byte code being loaded into modifiable memory regions along with mutable data, increasing the probability that the memory will either be not shared from the start or become unshareable due to a CoW fault. The second statement was comparing dumped versus undumped performance, whether native- or byte- compiled. This isn't a novel observation, but the dump is basically a one-shot mark-compact GC collection, and the mmap'ed pdump files would ideally be treated like the old generation of a generational collector, and so not traced in most (or all) collection cycles. This is (or should be) one benefit of pure space, although the requirement that it be read-only could be dropped as long as the write-barrier remains and is used to record any inter-generational pointers to a root set traced by the collector. > > I'd also like a baseline expectation of performance with native-compiled libraries in the dumped image. > > What kind of performance? Primarily usability. And I can report that going from undumped to dumped native-compiled does massively improve performance. I have not measured byte-compiled versus native-compiled when both are dumped. Even with 2100+ eln files being loaded, the startup is very fast even loading my .init settings (that turn on all the Semantic minor modes, for example). When I turned on the profiler for a while, it reported about 45% of the cpu time was in gc, which I found surprising. It was a little laggy in some places, but not "stop-the-world collection of a 200MB heap" laggy. I'm tempted to try a quickish hack to see if I can turn pure space into a never-collected elder generation by making the size a static variable instead of a define and pointing pure to the start of the dump image, then modifying the write barrier as described above, just for my local 28.1 variant. I don't mind wasting the memory if it keeps the heap traced by the collector to a reasonable size focused on shorter-lived constructs than the ones that survive the dump collection. I'd be surprised if it worked that easily, though. > > I'm saying that your job could be easier if you did the first step > with byte-compiled files. And it was - thanks. Let me know if there are any specific metrics that would be useful. I can't promise I'll be able to get back with details quickly (I may have to create novel implementations of the algorithms on my personal machines), but I will see what I can do. The emacs dev team and contributors have really made tremendous improvements since I last considered using the emacs source for playing around with some language implementation ideas somewhere in the 2004-2007 period. I thought the massive code base of dynamic scoping and the implementation of closure with "funvecs" (as they were called then) was just too big to contemplate, but you guys have done it. "Closing the loop" by enabling the dumping of additional natively compiled libraries is a minor tweak in the code, but it makes a huge difference if you're trying to make use of all the additional functionality provided by the available add-on packages. Lynn