From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: native compilation units Date: Sun, 05 Jun 2022 10:20:13 -0400 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="12327"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) Cc: Andrea Corallo , emacs-devel@gnu.org To: Lynn Winebarger Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Jun 05 16:22:38 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nxr9K-00032G-C8 for ged-emacs-devel@m.gmane-mx.org; Sun, 05 Jun 2022 16:22:38 +0200 Original-Received: from localhost ([::1]:34164 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nxr9I-0003Kg-R4 for ged-emacs-devel@m.gmane-mx.org; Sun, 05 Jun 2022 10:22:36 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:60292) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nxr7J-0001ga-0Z for emacs-devel@gnu.org; Sun, 05 Jun 2022 10:20:34 -0400 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:45036) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nxr7E-0001wk-4Q for emacs-devel@gnu.org; Sun, 05 Jun 2022 10:20:31 -0400 Original-Received: from pmg1.iro.umontreal.ca (localhost.localdomain [127.0.0.1]) by pmg1.iro.umontreal.ca (Proxmox) with ESMTP id 87A3B100796; Sun, 5 Jun 2022 10:20:26 -0400 (EDT) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg1.iro.umontreal.ca (Proxmox) with ESMTP id 6E793100171; Sun, 5 Jun 2022 10:20:20 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1654438820; bh=k7MBocU2tYQQ5iC5TEvEQQ5frhM2XwqhSSEFlLAq8Uw=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=HhS8x6BRrOqVBOBq6JmQPIphQC49KrX/X0faBPcXInZan8u14gL4lmurWPypliS3f y0W94miqj3PXNGvVlgN8lNP3rGdYT9pxubkjpGh54XX86TTxPHNNtPGlpWkvGoQnJW 3Lw/6TNZiMq6d1I8G0jtow+p4AyrtRDxiFY6B3wTrW6S9u3oT8ew2LR6VbBgHIy33n rR4u34Otl/W3IJY4Rf0tmvv+Mbaov2iiJkGAS9FOY+Ij+vW3gZppp6RGrX8VSNMGA1 YVeRFDLsLm/YzySzv16wYWNeIRIllNvNInbvdFnJutaV7QIywD6tUvfm2Kzd60xKJF ZhbRasJjyOFfQ== Original-Received: from pastel (unknown [45.72.221.51]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 37A93120193; Sun, 5 Jun 2022 10:20:20 -0400 (EDT) In-Reply-To: (Lynn Winebarger's message of "Sun, 5 Jun 2022 08:16:25 -0400") Received-SPF: pass client-ip=132.204.25.50; envelope-from=monnier@iro.umontreal.ca; helo=mailscanner.iro.umontreal.ca X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:290716 Archived-At: > Unfortunately sometimes we have to cope with environment we use. And for > all I know some of the performance penalties may be inherent in the > (security related) infrastructure requirements in a highly regulated > industry. What we learned at the end of last century is exactly that there aren't any such *inherent* performance penalties. It may take extra coding work in the file-system to make it fast with 10k entries. It may take yet more work to make it fast with 10G entries. But it can be done (and has been done), and compared to the overall complexity of current kernels, it's a drop in the bucket. So nowadays if it's slow with 10k entries you should treat it as a bug (could be a configuration problem, or some crap software (anti-virus?) getting in the way, or ...). > Not that that should be a primary concern for the development team, but it > is something a local packager might be stuck with. Indeed. Especially if it only affects a few rare Emacs users which don't have much leverage with the MS-certified sysadmins. >> >> [ But that doesn't mean we shouldn't try to compile several ELisp files >> >> into a single ELN file, especially since the size of ELN files seems >> >> to be proportionally larger for small ELisp files than for large >> >> ones. ] >> > >> > Since I learned of the native compiler in 28.1, I decided to try it out >> and >> > also "throw the spaghetti at the wall" with a bunch of packages that >> > provide features similar to those found in more "modern" IDEs. In terms >> of >> > startup time, the normal package system does not deal well with hundreds >> of >> > directories on the load path, regardless of AOR native compilation, so >> I'm >> > tranforming the packages to install in the version-specific load path, >> and >> > compiling that ahead of time. At least for the ones amenable to such >> > treatment. >> >> There are two load-paths at play (`load-path` and >> `native-comp-eln-load-path`) and I'm not sure which one you're taking >> about. OT1H `native-comp-eln-load-path` should not grow with the number >> of packages so it typically contains exactly 2 entries, and definitely >> not hundreds. OTOH `load-path` is unrelated to native compilation. >> > > Not entirely - as I understand it, the load system first finds the source > file and computers a hash before determining if there is an ELN file > corresponding to it. `load-path` is used for native-compiled files, yes. But it's used in exactly the same way (and should hence cost the same) for: - No native compilation - AOT native compilation - lazy native compilation Which is what I meant by "unrelated to native compilation". > Although I do wonder if there is some optimization for ELN files in the > system directory as opposed to the user's cache. I have one build where I > native compiled (but not byte compiled) all the el files in the lisp > directory, IIUC current code only loads an ELN file if there is a corresponding ELC file, so natively compiling a file without also byte-compiling it is definitely not part of the expected situation. Buyer beware. >> I also don't understand what you mean by "version-specific load path". > In the usual unix installation, there will be a "site-lisp" one directory > above the version specific installation directory, and another site-lisp in > the version-specific installation directory. I'm referring to installing > the source (ultimately) in ..../emacs/28.1/site-lisp. During the build > it's just in the site-lisp subdirectory of the source root path. I'm not following you. Are you talking about compiling third-party packages during the compilation of Emacs itself by placing them into a `site-lisp` subdirectory inside Emacs's own source code tree, and then moving the resulting `.el` and `.elc` files to the `../NN.MM/site-lisp` subdirectory in Emacs's installation target directory? And you're saying that whether you place them in `../NN.MM/site-lisp` rather than in `../site-lisp` makes a significant performance difference? >> Also, what kind of startup time are you talking about? >> E.g., are you using `package-quickstart`? > That was the first alternative I tried. With 1250 packages, it did not > work. Please `M-x report-emacs-bug` (and put me in `X-Debbugs-Cc`). > First, the file consisted of a series of "let" forms corresponding > to the package directories, and apparently the autoload forms are ignored > if they appear anywhere below top-level. At least I got a number of > warnings to that effect. > The other problem was that I got a "bytecode overflow error". I only got > the first error after chopping off the file approximately after the first > 10k lines. Oddly enough, when I put all the files in the site-lisp > directory, and collect all the autoloads for that directory in a single > file, it has no problem with the 80k line file that results. We need to fix those problems. Please try and give as much detail as possible in your bug report so we can try and reproduce it on our end (both for the warnings about non-top-level forms and for the bytecode overflow). > I'm pretty sure the load-path is an issue with 1250 packages, even if half > of them consist of single files. I'm afraid so, indeed. > One issue with this approach is that the package selection mechanism > doesn't recognize the modules as being installed, or provide any assistance > in selectively activating modules. Indeed, since the selective activation relies crucially on the `load-path` for that. > Other places where there is a noticeable slowdown with large numbers of > packages: > * Browsing customization groups - just unfolding a single group can take > minutes (this is on fast server hardware with a lot of free memory) Hmm... can't think of why that would be. You might want to make a separate bug-report for that. > * Browsing custom themes with many theme packages installed > I haven't gotten to the point that I can test the same situation by > explicitly loading the same modules from the site-lisp directory that had > been activated as packages. Installing the themes in the system directory > does skip the "suspicious files" check that occurs when loading them from > the user configuration. Same here. I'm not very familiar with the custom-theme code, but it does seem "unrelated" in the sense that I don't think fixing some of the other problems you've encountered will fix this one. >> I think you're right here, but I'd expect the effect to be fairly small >> except when the .elc/.eln files are themselves small. > There are a lot of packages that have fairly small source files, just > because they've factored their code the same way it would be in languages > where the shared libraries are not in 1-1 correspondence with source files. Oh, indeed, small source files are quite common. > I would expect this would apply to most top-level defuns in elisp > packages/modules. From my cursory review, it looks like the ability to > redefine these defuns is mostly useful when developing the packages > themselves, and "sealing" them for use would be appropriate. Advice are not used very often, but it's very hard to predict on which function(s) they may end up being needed, and sealing would make advice ineffective. I would personally recommend to just stay away from the level 3 of the native compiler's optimization. Or at least, only use it in targeted ways, i.e. only at the very rare few spots where you've clearly found it to have a noticeable performance benefit. In lower levels of optimization, those same calls are still optimized but just less aggressively, which basically means they turn into: if (; else ; Stefan