From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Wrap-up of customize native-compiled dump Date: Sun, 21 Aug 2022 13:31:42 -0400 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="3894"; mail-complaints-to="usenet@ciao.gmane.io" To: emacs-devel Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Aug 21 19:33:17 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oPop3-0000sQ-7P for ged-emacs-devel@m.gmane-mx.org; Sun, 21 Aug 2022 19:33:17 +0200 Original-Received: from localhost ([::1]:39182 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oPop1-0004qg-Mx for ged-emacs-devel@m.gmane-mx.org; Sun, 21 Aug 2022 13:33:15 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:40186) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oPono-00049A-CY for emacs-devel@gnu.org; Sun, 21 Aug 2022 13:32:01 -0400 Original-Received: from mail-pj1-x102a.google.com ([2607:f8b0:4864:20::102a]:42793) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oPonm-0002lK-3Y for emacs-devel@gnu.org; Sun, 21 Aug 2022 13:32:00 -0400 Original-Received: by mail-pj1-x102a.google.com with SMTP id s3-20020a17090a2f0300b001facfc6fdbcso8528937pjd.1 for ; Sun, 21 Aug 2022 10:31:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc; bh=o7vRm0rxDSSNEMFd76Fc4bH/QNpMiZbhC3N+53NKRBU=; b=bgmfvESjbtAq7NJRBzy/3NebGXfGgM0wdF8ARI0/wlzniz1hGTAybmP9G38He+n8QF yTXKb3fE70qA9qKeKTrLJspwzSBnhn4zowM/xi57+zjg8cVrTEHXRxc5ty4CJTiWkt2K VYFSBwIdXdWHfXd3auPzA1vPUIERmb+Lhx8LetQc8KAmzqwE0zDNDNf8+PuAWfKB6ojk VqRHl5eyyPV09/8L6CgfUG4w0rUlMJyKxsE7yMinOuflian546L4nmce2NKdWvlmBeja JotjGKbIz2HiZ5tkcMr6Ebq//XjNG30gRj7UZrq5Rt4/kaXUE3FE5FsUSjrb+C3t4onF adbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc; bh=o7vRm0rxDSSNEMFd76Fc4bH/QNpMiZbhC3N+53NKRBU=; b=kk/lVPIGOwQ+52tnYVGhV+ChF+JhAwa612j84IO2oVk5ZhbWBATBs6RyQtTttAXkqw PrKRo98xRsk3RhRxx39u4OCePm8UC/OVnKKZ8FdtvslVXuVEzQwG7c81975xJKwkGwjs aEAU3fXGncXuFIKlx/CBDwzomKmknF5C+HdhamkER4P+3VFdUxrV5fRnmVqdYj2/B07j A/5zRA3q9JWXul6wRY/On4cwhnYfyP5ZmgU3Agz8UXcbPw7ZUo1Q6tbdhndTJL8x1xFM vB+hdqUGDzKynSo21r7aextkLF5MCwrII+nS8VVEh6ztySCR3mB4+vSsoLiwKlrzhWg1 RRrg== X-Gm-Message-State: ACgBeo23kY2mvZf0+ryRf8n5XByAB5+idDpu0fA5xf+6Gts12sMHcxw3 46NN7Wn2YoSa71z0kbD77QNO54Uf2E7loF0eOWF/io1OhPY= X-Google-Smtp-Source: AA6agR4HJVJ7MYxI02IUFZBQEz0Qc2Eht8ZV6YDOmTyIE8TlDORjaaVCTtLJuzQNOklE3IWYpn0JNtOGLRTasBJ4e4o= X-Received: by 2002:a17:902:ce11:b0:172:6f2c:a910 with SMTP id k17-20020a170902ce1100b001726f2ca910mr16922975plg.156.1661103114912; Sun, 21 Aug 2022 10:31:54 -0700 (PDT) Received-SPF: pass client-ip=2607:f8b0:4864:20::102a; envelope-from=owinebar@gmail.com; helo=mail-pj1-x102a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:293756 Archived-At: A while ago I was asking for assistance in making a "mega-dump" with a large sample of native-compiled packages preloaded using the emacs 28.1 source tarball. I've had the dump working for a week or so on a Linux system based on a 3.10 kernel. I was asked if I could share the results of my effort, and while I can't share the patches required, I can describe the results if it makes any difference for current developers considering the value of this feature. My "final" build has a little over 2400 native compilation units in the dump. The dump file is ~180MB when I left out lsp-mode libraries, and ~190MB with lsp-mode included First, the positives: * It's very fast to startup and use. I have Semantic/EDE turned on and mostly do not have any noticeable effect on responsiveness. Without lsp-mode, startup takes a couple of seconds or less,with lsp-mode (for most supported languages) it takes a few more seconds. * Garbage collection is surprisingly fast for a heap size starting at ~190MB. I haven't run any benchmark for this, but I did have one library that was constantly calling "eval" to determine the tabs for the header-line when semantic took over the header line. That generated a lot of garbage, so it was pausing frequently. However, the pauses themselves seemed very short - much shorter than I associate with having an emacs session that has built-up a 190+MB heap from the standard dump. I can only speculate that having everything native-compiled means there are relatively few byte-code vectors that have to be traversed in the heap, and that most of the garbage from the loading and initialization is effectively collected by the portable dump process. Once I fixed that setting that called "eval" in the problem library, which was just constructing a call to a thunk, to just call the thunk directly, the pauses became much less frequent, and still not very noticeable. * At the end of site-load, I have a call to load the full customization edit system to force the calculation of customization groups ahead of time instead of calculating them lazily. For a dump with over 1000 non-core packages, that lazy computation after startup is very slow and each group seems to only be computed when it is first opened - I'm assuming autoload is involved. Having those groups precalculated in the dump appears to work well, for the most part, customize buffers open quickly, and the ordering of the groups and setting is at least consistent every time instead of depending on the order of library autoloading. * Icons etc are included in the dump. * Whatever the process limits opening shared libraries is, I did not encounter them. IOW, the infrastructure for native-compiled files in 28.1 already provides very efficient inclusion of those files in dumps, and there appears to be a virtuous-cycle effect with respect to garbage collection. I wouldn't mind having my experience verified (or disproven) , though. The cons - basically the work required to get the system dumped: * I submitted a list of issues I had to deal with to be able to perform any customized dump using site-load.el at https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57035 . This list excludes issues related to including any particular library in that dump, even if they are in the core * I massaged about 1200 ELPA/MELPA packages so that the library code was under site-lisp in the source directory, and any data files were put in the /site-data/. Most of that process was automated, but there were a nontrivial number of cases that required special attention to ensure any package variable set to reference a data directory relative to the file being loaded or byte-compiled was redirected. * A lot of packages - including some core libraries - have issues with variables that expect to be initialized when loaded rather than at system initialization. Most of these can be dealt with using :initialize #'custom-initialize-delay and :set-after '(). I'm not sure if the ":set-after" option would have been sufficient when the dependencies had the delayed initialization setting, but it usually worked that way * A few variables from the core libraries still caused problems for customizations initialized based on them. I haven't reported them as bugs because custom dumping with native-compiled libraries isn't really supported as far as I know, but they'll cause issues when it is (list from memory): ** user-emacs-directory (issue in starting the customization system, I believe) ** user-mail-address (regular variable from startup.el, so perhaps out of the late initialization framework?) ** image-load-path (the issue is from ezimage generating images at load time rather than initialization, so the late-initialization framework for customization variables doesn't apply) * There may well be some libraries that aren't working and I don't know it. One point of the preloading is to make features of these packages more "discoverable", but it's difficult to tell if there are conflicting libraries in the system a priori. * There's no great way of updating pre-loaded packages one at a time. I did arrange all the pre-compiled packages to be the "preloaded" subdirectory, so putting newer versions in the main native-lisp system cache might be a viable way to install updates. * The dump process is *slow* (no benefit from having many cores) and any getting errors toward the end of a list of 1000-2000 libraries to load is a long wait - like 20+ minutes on the system I did this on. * Normal process loading is fast even with 2400+ eln files, but loading that dump under gdb will be a long wait. I don't know how long, as I gave up after letting it run for an hour or two and built a dump with just the autoload files and the problem modules. I find the result worth my effort in putting in "the last mile"of infrastructure. Lynn