From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Working fully native-compiled "mega dump" (was Re: Loading tramp for dump goes into infinite regress) Date: Sat, 6 Aug 2022 16:23:30 -0400 Message-ID: References: <8735erhrlg.fsf@gmx.de> <83wnc2g0n8.fsf@gnu.org> <83sfmqfxcb.fsf@gnu.org> <83a68yfp6j.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000ef08e305e5985b41" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="19947"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Eli Zaretskii , Michael Albinus , emacs-devel To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Aug 06 22:24:39 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oKQLf-00051n-9e for ged-emacs-devel@m.gmane-mx.org; Sat, 06 Aug 2022 22:24:39 +0200 Original-Received: from localhost ([::1]:47276 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oKQLd-0001Nr-SR for ged-emacs-devel@m.gmane-mx.org; Sat, 06 Aug 2022 16:24:37 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:35984) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oKQL2-0000iu-NX for emacs-devel@gnu.org; Sat, 06 Aug 2022 16:24:00 -0400 Original-Received: from mail-pj1-x102f.google.com ([2607:f8b0:4864:20::102f]:33746) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oKQKn-0007JX-QD; Sat, 06 Aug 2022 16:23:58 -0400 Original-Received: by mail-pj1-x102f.google.com with SMTP id t22-20020a17090a449600b001f617f2bf3eso2529107pjg.0; Sat, 06 Aug 2022 13:23:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=cPwrK4XtMEyv2Fzu+NxH9Mj4ZvBKGnuydvKt0GwpGrA=; b=BJimgVm6LtpnXJ2Q7eu3eHPQpt/XJwJ/mUJM2istDf49Rln8+LahWA5AIjP+EbQfaZ u6fPwRkr7sYUM1cBghy7NiaDlNWgVVE6n0JaGSb2HuSVzMQTi6zYN2CiFqZHtiz3uIni E/oJCMO4tyaem3teCJ5oB+yfS2oAFCvkqnd5Vz1laEoK7WckXIFCwSoIooV+sf21P+pd vjG8r8x7dNdLiUmp13GDycJv5tXEca/bocUdQc0zuf0R4wXDZnWwkfX2NGCndmJZezop JA/r788rHF/0NoJbbWGoPfwQmELmjDT4xDvfKTuKFS9xllr+nE2ytln59Vip9/1KSEm4 QG5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=cPwrK4XtMEyv2Fzu+NxH9Mj4ZvBKGnuydvKt0GwpGrA=; b=Al3tfp41WW0OFEfflqe4NerDo+TDPgYo+bO9wKr6ELxoaH9XJtceSv+o/aSgFO+AZm SrbOKZWYYuHisOyd305+oXK4JjluqFdKEKk0rzIEhM26D6s7oWtsGwrvCfTxQiEKW/Du z8T0A9ZmkWUTWogEHKHyIJYcIQtyS786CkFdEfBErnOWZ0KgYeWxImujhRUlhHk+QlRU E/qr6v2xTNRN1SHUDDlRzoU81wGk5qdExotSE+Ag3oIU/hC8PBkAQb03iCL5cBh164Z0 ode3KaaGEtZschg0gi4Wdg8yqjnc4dPOEjB6Vhi07ST0A6HTRmC8CrKOqSI0m8jU/eSo c9xw== X-Gm-Message-State: ACgBeo0oKtUyRVhar6oS1w4/7v/ac2ffOHWQHheSkPB71SBMXNM61L0Q w92p5LOjDMQ7nzwx5CmMOIQ6vXljlgtPwNV4dDw= X-Google-Smtp-Source: AA6agR6bRG2PCco7EmlRT1dAHxlll8/OrLV9N7n4yUKDZ6XSuQuMr+Gdh+MMkkUAjDY9X4tlvycs6Ab80J5Y8H2VwIc= X-Received: by 2002:a17:90a:2948:b0:1f3:1b42:a810 with SMTP id x8-20020a17090a294800b001f31b42a810mr14014725pjf.203.1659817423733; Sat, 06 Aug 2022 13:23:43 -0700 (PDT) In-Reply-To: Received-SPF: pass client-ip=2607:f8b0:4864:20::102f; envelope-from=owinebar@gmail.com; helo=mail-pj1-x102f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, T_SPF_HELO_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:293155 Archived-At: --000000000000ef08e305e5985b41 Content-Type: text/plain; charset="UTF-8" On Sat, Aug 6, 2022, 11:39 AM Lynn Winebarger wrote: > On Sat, Aug 6, 2022, 8:57 AM Lynn Winebarger wrote: > >> On Sat, Aug 6, 2022, 2:07 AM Lynn Winebarger wrote: >> >>> I've added another flag variable that allows me to turn off the ban on >>> implicit loading in dump mode. >>> >> >> The dump with all ~2500 ELN libraries loaded completed last night. >> Unfortunately during startup (before any frame opens) one of them is >> evaluating an expression with an argument that should never produce nil but >> somehow is nil. >> Fortunately I can now redump with just the offending library in >> site-load. The error prints fairly quickly when run normally, but shared >> libraries evidently open a lot slower under GDB. >> Hopefully I'll get the same error with the smaller dump and be able to >> determine if there's an issue in the dump file, the dump loading, or >> something else. >> > > It started up fine when I only explicitly added the one library with the > failing expression, and then again when I dumped after loading the original > list up to that library. > The good news is that emacs starts in a couple of seconds with 1198 ELN > files in the dump. I haven't been able to compile with profiling enabled > on this system, but I can tell you that is a couple of orders of magnitude > faster than just loading them in .init. > The bug seems to be from a couple of older (2010-2012) library files that are dynamically scoped and use copy-face with the "frame" parameter. All the other combinations I've tried start up without a problem, so I'm just going to leave these packages out of the dump. Also excluded are most lsp-mode libraries, because they seem to slow down the startup process (like 4 seconds instead of 2 or less}, vterm and zmq (dynamic modules can't be dumped), and a few modes that insist on attempting to download things on startup. What's included: All of CEDET All files in emacs-lisp Most if not all files in tramp/mail/net Everything in progmode that doesn't depend on an external utility (namely moz) being installed Any TeX or reference/citation functionality (reftex, bibtex, bibliography, citar, citeproc) Icon/emoji/image support (e.g alltheicons, major-mode-icons) All completion and widening/narrowing libraries - helm, ivy, counsel, company, auto-complete, Perspective, window-purpose, and similar Evil packages Treemacs Many other miscellaneous packages I'm forgetting. All in the dump file is about 184 MB for an older Linux on an x86_64 architecture. That would include all the lsp-mode packages from Melpa, or at least the ones with more than a nominal number of downloads. There are 2000-2500 ELN files that will be loaded. When I loaded these from my init file I could make a cup of coffee, maybe do a crossword before it would be ready to use. One of the packages with the dynamic scoping/face issues is tabbar-ruler, which I only started using during this exercise and think is fantastic. Hopefully it will work when required from the init file, as it does currently. At any rate it's fantastic at startup. The customization interface is very responsive without the usual lag from lazy construction of the customization groups. I'll have to see how it behaves after it's been running a few days. Lynn --000000000000ef08e305e5985b41 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sat, Aug 6, 2022, 11:39 AM Lynn Winebarger <owinebar@gmail.com> wrote:
On Sat, Aug 6, 2022, = 8:57 AM Lynn Winebarger <owinebar@gmail.com> wrote:
On Sat, Aug 6, 2022, 2:07 A= M Lynn Winebarger <owinebar@= gmail.com> wrote:
I've added another flag variable that allows me to turn off the ba= n on implicit loading in dump mode.=C2=A0

The dump with al= l ~2500 ELN libraries loaded completed last night.=C2=A0 Unfortunately duri= ng startup (before any frame opens) one of them is evaluating an expression= with an argument that should never produce nil but somehow is nil.
Fortunately I can now redump with just the offending librar= y in site-load.=C2=A0 The error prints fairly quickly when run normally, bu= t shared libraries evidently open a lot slower under GDB.
Hopefully I'll get the same error with the smaller dump and be ab= le to determine if there's an issue in the dump file, the dump loading,= or something else.

It started up fine when I only explicitly added the on= e library with the failing expression, and then again when I dumped after l= oading the original list up to that library.=C2=A0=C2=A0
The good news is that emacs starts in a couple of seconds with 11= 98 ELN files in the dump.=C2=A0 I haven't been able to compile with pro= filing enabled on this system, but I can tell you that is a couple of order= s of magnitude faster than just loading them in .init.=C2=A0

The bug seems= to be from a couple of older (2010-2012) library files that are dynamicall= y scoped and use copy-face with the "frame" parameter.=C2=A0 All = the other combinations I've tried start up without a problem, so I'= m just going to leave these packages out of the dump.=C2=A0 Also excluded a= re most lsp-mode libraries, because they seem to slow down the startup proc= ess (like 4 seconds instead of 2 or less}, vterm and zmq (dynamic modules c= an't be dumped), and a few modes that insist on attempting to download = things on startup.
What's included:
All of CEDET
All files in emacs-lisp
Most if not all files in tramp/mail/net
Everything in progmode that doesn't depend on an external utilit= y (namely moz) being installed
Any TeX or reference/= citation functionality (reftex, bibtex, bibliography, citar, citeproc)
Icon/emoji/image support (e.g alltheicons, major-mode-ic= ons)
All completion and widening/narrowing libraries= - helm, ivy, counsel, company, auto-complete,=C2=A0
Perspective, window-purpose, and similar
Evil packa= ges
Treemacs
Many other misce= llaneous packages I'm forgetting.

All in the dump file is about 184 MB for an older Linux on an= x86_64 architecture.=C2=A0 That would include all the lsp-mode packages fr= om Melpa, or at least the ones with more than a nominal number of downloads= .=C2=A0 There are 2000-2500 ELN files that will be loaded.=C2=A0 When I loa= ded these from my init file I could make a cup of coffee, maybe do a crossw= ord before it would be ready to use.=C2=A0=C2=A0
One of the packages with the dynamic scoping/face = issues is tabbar-ruler, which I only started using during this exercise and= think is fantastic.=C2=A0 Hopefully it will work when required from the in= it file, as it does currently.

At any rate it's fantastic at startup.=C2=A0 The customization i= nterface is very responsive without the usual lag from lazy construction of= the customization groups.=C2=A0 I'll have to see how it behaves after = it's been running a few days.

Lynn


--000000000000ef08e305e5985b41--