From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: Suppressing native compilation (short and long term) Date: Thu, 13 Oct 2022 23:22:25 +0300 Message-ID: <834jw7a2ym.fsf@gnu.org> References: <87ill8paw7.fsf@trouble.defaultvalue.org> <83o7uzivey.fsf@gnu.org> <3ac9d2b9632f75018327a1bcde0c373f152c404a.camel@gmail.com> <835ygob7ja.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="13082"; mail-complaints-to="usenet@ciao.gmane.io" Cc: rlb@defaultvalue.org, emacs-devel@gnu.org To: Liliana Marie Prikler Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Oct 13 22:25:56 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oj4mB-0003EK-BG for ged-emacs-devel@m.gmane-mx.org; Thu, 13 Oct 2022 22:25:55 +0200 Original-Received: from localhost ([::1]:47946 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oj4mA-0002Fu-DU for ged-emacs-devel@m.gmane-mx.org; Thu, 13 Oct 2022 16:25:54 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:42822) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oj4j0-0007aA-Bg for emacs-devel@gnu.org; Thu, 13 Oct 2022 16:22:38 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]:42238) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oj4iz-0000SM-TB; Thu, 13 Oct 2022 16:22:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:References:Subject:In-Reply-To:To:From: Date; bh=aX0X2zcSOuu2AU1DISd8Hl73R1ELDa/7VA2yNO/DGC8=; b=WhFatak+L1+KglGnmKAt A5DufXDhlS5viM49Jaidgw99IF+TImrclxkuCm4ZyoHFFjCD72mfoOUb1O7BJKwRNOSpqKtHhgIwI yVaOn7PTZQU2VURGLuPsLqApnUj5hpOM3lvh6UAz1Eqp8jIauMRLth/rT+99UAbr6uc2vWRKzkZ0o Fzf2jETcw8UgwNI21dyIv/bEmQerfozZjuhJ1BmXljREL5B+CBqVq0fzbLwINgKWZc2eCxpLWnIDj lkv6dUJjxKLkQonxLvLPjF6/dsaN4DzUz/EsGeSyfghaSuJsyGWB/lAMpQmS/VMgtRm1Avw5qTldc XuIR7x4gFHlFjw==; Original-Received: from [87.69.77.57] (port=3308 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oj4iz-0004hd-Bc; Thu, 13 Oct 2022 16:22:37 -0400 In-Reply-To: (message from Liliana Marie Prikler on Thu, 13 Oct 2022 21:20:13 +0200) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:297696 Archived-At: > From: Liliana Marie Prikler > Cc: rlb@defaultvalue.org, emacs-devel@gnu.org > Date: Thu, 13 Oct 2022 21:20:13 +0200 > > > When you encounter bugs in native compilation, please report them to > > us, so we could fix them.  As of now, we are not aware of any such > > bugs that were reported and haven't been fixed.  So if you still have > > such problem, please report them ASAP. > Of course, that's the intention, but this fix will only make it into > the next Emacs release. Thus, if you're between releases, you still > need a workaround. If the fix is urgent, why can't you patch the sources when you prepare your distribution? > A particular candidate known to cause issues with the currently > packaged 28.1 is [1]. Where's the description of the actual problem with natively compiling that package? And would you please submit a bug report with the details, if you know them? > > Why isn't it sufficient to use no-native-compile?  It just means that > > on some architectures the corresponding file will be loaded as > > byte-compiled, and thus will be slightly slower (how much slower > > depends on the code, so if you are worried, my recommendation is > > first to measure the difference -- you might be surprised). > Because it'd require a distro-wide fix to address something that e.g. > only happens on some AMD CPUs. I'm asking why doing so is a problem? Did you measure the effect on performance and found it to be unacceptable in some cases? > > > In GNU Guix, we default to not compiling packages ahead-of-time, > > > instead using a minimal emacs that can only do byte compilation for > > > most packages.  Users can however pretty easily switch to ahead-of- > > > time compilation by swapping out the emacs package (using what Guix > > > calls package transformations).  This is also important because > > > apart from the current Emacs we typically provide an emacs-next > > > package for upcoming versions, as well as some other variants that > > > the user might want to use instead.  Again, we assume that users > > > who want to opt in to native compilation do so via a > > > transformation. > > > > Sorry, I'm unfamiliar with this terminology.  When you say > > "ahead-of-time compilation", do you mean native compilation of all > > the Lisp files before they are loaded for the first time?  Or do you > > mean something else? > Exactly, it's more or less the same as ahead-of-time compilation via > package.el, which Emacs supports out of the box. > > > And what is "swapping out the emacs package", and what is "package > > transformation"? > Guix users can decide between an Emacs package that only does byte- > compilation (emacs-minimal, the default) or native compilation (emacs). > They can easily write this by using the command-line switch --with- > input=emacs-minimal=emacs while building their Emacs package (e.g. > emacs-dash for dash.el). OK, so why is this relevant to the issue of disabling? Those who choose ahead-of-time compilation will never see async JIT compilation, and those who selected not to do ahead-of-time will naturally see JIT compilation, as they've chosen. What is the problem here? > > > In this context, the default of compiling everything that has > > > hitherto only been byte-compiled is an ill-advised choice.  Now, > > > there is a chance that the user meant to do this anyway, but I > > > think they rather a) use whatever the distro default is without > > > caring either way or b) actually didn't want this package natively > > > compiled.  Due to some packages breaking, we had a lot of b) in our > > > mailing lists in the past few days. > > > > If a package is a single file or a small number of files, those users > > can add the no-native-compile cookies in those files. > This is not trivial in the case where the Elisp code is placed in > system-managed storage and thus requires elevated privileges to modify > (as is the default in most package managers, I assume). Of course, you > can copy the file to your $HOME, but editing it with a broken Emacs is > rather painful. Using broken packages is always painful, and native compilation doesn't change that. Packages provided by a distribution and installed into directories where users cannot easily write should be well tested by the distributor. IOW, I don't understand why the upstream project should be held responsible for packages distributed by downstream distributions without testing them well enough to let users use them without pain, and why should the upstream project introduce options to support such "broken" distros. It isn't fair to expect us to solve such problems, because they should be solved elsewhere. > > And again, disabling native compilation is a method that doesn't > > allow them fine-tuning anyway, so I fail to see how it could help > > them here.  If the problem is real (and I don't yet see it is), we > > should perhaps discuss its details and invent some new method of > > disabling compilation with finer granularity. > The granularity here is disabling compilation of anything that isn't > already compiled – this makes it so that people can still use their > Emacs for byte compilation by invoking it with "emacs -Q", they just > won't compile anything that their package manager hasn't compiled for > them. That's quite a blunt weapon. Why not tell them to stay with Emacs 27, until the problems are solved, or install Emacs 28 without native compilation? They are similarly drastic solutions, but they are already available and will definitely work. > > I don't think you can set native-comp-eln-load-path to nil.  The last > > entry, the one which points to where the preloaded *.eln files live, > > must be there, or else Emacs will refuse to start.  And at least one > > other directory should be there as well, because if and when some > > package advises some primitive, Emacs will need to generate and > > compile a trampoline .eln file.  But yes, if users want to prevent > > loading from a certain directory, they can remove it from > > native-comp-eln-load-path, provided that the two necessary entries > > are still left in the list. > I already found this annoying while implementing native compilation as > part of our emacs-build-system (i.e. the build system used to compile > Emacs packages), and I find it extra questionable that users on > traditional distros, where they don't usually get to choose their > Emacs, have no means of disabling this loading. You mean, you find the loading of preloaded *.eln files at startup annoying? Then you should know that this is the best solution we found for dumping Emacs with natively-compiled preloaded code. If you know of a better solution that doesn't suffer from any fatal issues we found with the alternatives, please suggest such solutions, and we will definitely consider them. And again, if Emacs with natively-compilation is annoying, by all means offer your users an Emacs built without natively-compilation. This is supported and will continue to be supported for the observable future. > Calling back to the earlier point on measuring performance, an easy > way to measure performance between bytecode and native code (in a > benchmark) would be to simply disable native code loading in one > process. But here, it requires two separate builds of Emacs. As I told earlier, disabling loading of native code made no sense to us while Emacs 28 was in development; it still doesn't. Either one wants native-compilation, or one doesn't. Making Emacs code more complicated and harder to maintain due to features that make no sense to us is a non-starter. I see no problem with having to use a separate build, since building a release tarball takes a minute or so on a modern system. And distros should definitely have a build without native-compilation on offer, for a variety of valid reasons. > I agree for the most part that JIT compilation is harmless. However, > there are some cases in which it isn't. For one, on slower hardware, > this can take unreasonable times at startup. System that are that slow should not install Emacs with native-compilation, plain and simple. My suggestion is to clearly explain this in your README files, so that users could decide which build is for them, exactly like they decide whether they want a GTK build or a Lucid build (or any other of the configurations you offer). > While bytecode performance on such machines might too be slow (but > perhaps tolerable for the task), ahead-of-time compilation, perhaps > with offloading, is preferable. I recommend against this, because it is impossible to rely on AOT installations to never compile at run time. Users cannot rely on that, and should be advised accordingly. > For another, it can cause bugs like [2]. That bug by itself (the cause of massive launching of async subprocesses) was never explored or described in that thread? It seems like the discussion switched to looking for ways of disabling native-compilation right away, without a good understand of what was happening. Or did I mis something? Async compilation by default never launches more subprocesses than half the execution units of the CPU, so what is described there should be carefully investigated and the findings described. The other problem in that discussion, with warnings during async JIT compilation is well-known, was reported several times, and the culprit is always in the 3rd-part packages being compiled, which should be fixed. In any case, those are just warnings in almost all cases, so their only adverse effect is annoyance (that can be suppressed by clicking the button in the message). Again, I see no reason to blame the upstream project for these issues. They should be solved by the offending 3rd-party packages, and the distro should ideally uncover and fix them before they get to users (I presume that you build and compile the add-on packages you offer?). > > Thanks for the explanations.  I still think the reasons for disabling > > native compilation are rather weak at best, and the users' requests > > to allow that based on either bugs that need to be solved or surprise > > and fears that have no real basis.  Moreover, disabling native > > compilation is a very blunt instrument that cannot be applied better > > than the existing ones, like no-native-compile (and a few others that > > we didn't mention; see the defcustom's in comp.el). > Which defcustom? Begin with those described in the ELisp manual, in the "Native-Compilation Variables" node. And my recommendation is to review _all_ of the defcustoms in comp.el > I fear that for all of its customizability, Emacs is > currently lacking a good way of disabling native compilation outside of > package management. Yes, because, as mentioned, this makes no sense. And disabling native-compilation completely is currently technically impossible (for te same reason: Emacs wasn't designed to support that because it didn't and still doesn't seem needed). > I also tried setting no-native-compile globally, but it seems to > only have an effect as a file-local variable. Yes, as designed. This variable is the equivalent of no-byte-compile, and works very similarly. To summarize: native compilation in a build which supports it is ubiquitous, and is not designed to be disabled except by no-native-compile on a file by file level. If a more general disabling is needed for some reason, users should simply use a build without native-compilation. It's the same as various toolkit builds: if the toolkit is broken or doesn't fit the user's needs, those users should install a build with a different toolkit.