From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Liliana Marie Prikler Newsgroups: gmane.emacs.devel Subject: Re: Suppressing native compilation (short and long term) Date: Fri, 14 Oct 2022 21:46:09 +0200 Message-ID: References: <87ill8paw7.fsf@trouble.defaultvalue.org> <83o7uzivey.fsf@gnu.org> <3ac9d2b9632f75018327a1bcde0c373f152c404a.camel@gmail.com> <835ygob7ja.fsf@gnu.org> <834jw7a2ym.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="23029"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Evolution 3.46.0 Cc: rlb@defaultvalue.org, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Oct 15 08:09:54 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ojaMr-0005r9-V5 for ged-emacs-devel@m.gmane-mx.org; Sat, 15 Oct 2022 08:09:54 +0200 Original-Received: from localhost ([::1]:40870 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ojaMq-0005jS-OP for ged-emacs-devel@m.gmane-mx.org; Sat, 15 Oct 2022 02:09:52 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:40708) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ojQda-0007V7-Jc for emacs-devel@gnu.org; Fri, 14 Oct 2022 15:46:31 -0400 Original-Received: from mail-ed1-x541.google.com ([2a00:1450:4864:20::541]:37614) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ojQdU-0001Sx-DI; Fri, 14 Oct 2022 15:46:28 -0400 Original-Received: by mail-ed1-x541.google.com with SMTP id m16so8230595edc.4; Fri, 14 Oct 2022 12:46:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:from:to:cc:subject :date:message-id:reply-to; bh=7Kpx62Pxop82/Xu4G2WXi8D11XBHy1HzhVh02+qJ3E4=; b=BPePu7n3tbmnJyAScoe06p1rCIcnhz1SPYgWvVtIsEE543vi7rCSAFdaUDHAhi5gxa wAhqVVWZ3mq0vKgv7i/Zsi7NSbERU3dLh8aiymDL1vKAkU8cizUla8GUVc1UreYqqqSM vk3lwZVoB/61f9mAuXpBTeDqv4P9X2Ra5tTceRIfDNAGfV8KNZa8oB09sO2cRHpZsOxc dMMJfP+794Yn5OOADe+/YLIrF4myfZo/27pwNx7YglhINgiFjmPX5De8TuYMkTw4kR9Z dfIcI7N2FzrNxbLxMl60SRJl1yO8UKm/dX0zXjxkUaobHl+4UHENoqegeAPRbDPrjJzx aUTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=7Kpx62Pxop82/Xu4G2WXi8D11XBHy1HzhVh02+qJ3E4=; b=3JIt4VQgNqx/vDHoOYRluiiVZzWeJv4feaoTI2hZ78TKLYC5URIHh15/VUObivNBU8 7ZtMd1asjWx2ioqwHLbtB+W+fYqBxpw92a2FSE+7OZUQLI5BDynSAbQnVueLgvRSStU2 yvYG52dsQEXRiGBA4XJaYuHwueZRnWQY3hrd1pgdnXqAVCsqdivZdmzZUDDmyCE/B2jx q4W1gu17ByhGFgNCGb5oVCTvBcIKqLfrymifBWwRyd3dxob4LWkHGKvVU8OidS6gIFiT 3PiincUjNN0UKacjQDOenvE/gJ95I1WscTO9DA7LNQDEjsZ6auig/PapUqKOqPtEIDLW Hi+A== X-Gm-Message-State: ACrzQf0Wx+mEYlx5nRVMRDeuzJDL3Ai9Ykh4XuafMrhan3rHMvtFATag DNv8Gg3yyb8w03BrNA1EBY3Vy+zLtBY= X-Google-Smtp-Source: AMsMyM4z1zFyceT8acimbs78uuAPQwHnvHFxH9On8sJaM28fDma1Iv3vLtDu8xAIsmIJJ/Oyem2TsA== X-Received: by 2002:a05:6402:26c2:b0:45c:1fef:ee1d with SMTP id x2-20020a05640226c200b0045c1fefee1dmr5517073edd.13.1665776771485; Fri, 14 Oct 2022 12:46:11 -0700 (PDT) Original-Received: from lumine.fritz.box (85-127-52-93.dsl.dynamic.surfer.at. [85.127.52.93]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b00730bfe6adc4sm2021665ejb.37.2022.10.14.12.46.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 12:46:10 -0700 (PDT) In-Reply-To: <834jw7a2ym.fsf@gnu.org> Received-SPF: pass client-ip=2a00:1450:4864:20::541; envelope-from=liliana.prikler@gmail.com; helo=mail-ed1-x541.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Sat, 15 Oct 2022 02:08:12 -0400 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:297752 Archived-At: Am Donnerstag, dem 13.10.2022 um 23:22 +0300 schrieb Eli Zaretskii: > > From: Liliana Marie Prikler > > Cc: rlb@defaultvalue.org, emacs-devel@gnu.org > > Date: Thu, 13 Oct 2022 21:20:13 +0200 > >=20 > > > When you encounter bugs in native compilation, please report them > > > to us, so we could fix them.=C2=A0 As of now, we are not aware of any > > > such bugs that were reported and haven't been fixed.=C2=A0 So if you > > > still have such problem, please report them ASAP. > > Of course, that's the intention, but this fix will only make it > > into the next Emacs release.=C2=A0 Thus, if you're between releases, yo= u > > still need a workaround. >=20 > If the fix is urgent, why can't you patch the sources when you > prepare your distribution? Guix prides itself in being a package manager that can work around many failures (even as the proper workaround to bugs is discussed in mailing lists). The fact, that the solutions to this issue is "compile 28.1 without native-comp" or "use Emacs 27" does not reflect that particularly well. > > A particular candidate known to cause issues with the currently > > packaged 28.1 is [1]. >=20 > Where's the description of the actual problem with natively compiling > that package?=C2=A0 And would you please submit a bug report with the > details, if you know them? I am not personally affected, so I can't. I could direct people to the Emacs mailing lists, but it seems people in other threads have already started debugging. Do you still wish me to do so?=20 > > > Why isn't it sufficient to use no-native-compile?=C2=A0 It just means > > > that on some architectures the corresponding file will be loaded > > > as byte-compiled, and thus will be slightly slower (how much > > > slower depends on the code, so if you are worried, my > > > recommendation is first to measure the difference -- you might be > > > surprised). > > Because it'd require a distro-wide fix to address something that > > e.g. only happens on some AMD CPUs. >=20 > I'm asking why doing so is a problem?=C2=A0 Did you measure the effect on > performance and found it to be unacceptable in some cases? Isn't performance one of the main reasons to use native compilation?=20 Note that I am talking in hypotheticals here when mentioning the AMD thing, i.e. we could very well imagine a performance-critical Emacs package having a native-compilation bug (I imagine those to be particularly likely for those trailing unreleased Emacs versions, though thankfully I don't think we've encountered one so far.) > > > > In GNU Guix, we default to not compiling packages ahead-of- > > > > time, instead using a minimal emacs that can only do byte > > > > compilation for most packages.=C2=A0 Users can however pretty easil= y > > > > switch to ahead-of-time compilation by swapping out the emacs > > > > package (using what Guix calls package transformations).=C2=A0 This > > > > is also important because apart from the current Emacs we > > > > typically provide an emacs-next package for upcoming versions, > > > > as well as some other variants that the user might want to use > > > > instead.=C2=A0 Again, we assume that users who want to opt in to > > > > native compilation do so via a transformation. > > >=20 > > > Sorry, I'm unfamiliar with this terminology.=C2=A0 When you say > > > "ahead-of-time compilation", do you mean native compilation of > > > all the Lisp files before they are loaded for the first time?=C2=A0 O= r > > > do you mean something else? > > Exactly, it's more or less the same as ahead-of-time compilation > > via package.el, which Emacs supports out of the box. > >=20 > > > And what is "swapping out the emacs package", and what is > > > "package transformation"? > > Guix users can decide between an Emacs package that only does byte- > > compilation (emacs-minimal, the default) or native compilation > > (emacs). > > They can easily write this by using the command-line switch --with- > > input=3Demacs-minimal=3Demacs while building their Emacs package (e.g. > > emacs-dash for dash.el). >=20 > OK, so why is this relevant to the issue of disabling?=C2=A0 Those who > choose ahead-of-time compilation will never see async JIT > compilation, and those who selected not to do ahead-of-time will > naturally see JIT compilation, as they've chosen.=C2=A0 What is the > problem here? The problem is that I can't meaningfully choose the "I don't want JIT for stuff I haven't AOT'd" option, especially not combined with "but I do want to load what I have AOT'd". > > > > In this context, the default of compiling everything that has > > > > hitherto only been byte-compiled is an ill-advised choice.=C2=A0 > > > > Now, there is a chance that the user meant to do this anyway, > > > > but I think they rather a) use whatever the distro default is > > > > without caring either way or b) actually didn't want this > > > > package natively compiled.=C2=A0 Due to some packages breaking, we > > > > had a lot of b) in our mailing lists in the past few days. > > >=20 > > > If a package is a single file or a small number of files, those > > > users can add the no-native-compile cookies in those files. > > This is not trivial in the case where the Elisp code is placed in > > system-managed storage and thus requires elevated privileges to > > modify (as is the default in most package managers, I assume).=C2=A0 Of > > course, you can copy the file to your $HOME, but editing it with a > > broken Emacs is rather painful. >=20 > Using broken packages is always painful, and native compilation > doesn't change that. Using broken packages normally doesn't result in the OOM killer firing off. > Packages provided by a distribution and installed into directories > where users cannot easily write should be well tested by the > distributor.=C2=A0=C2=A0 I think you're underestimating the number of breakages that can happen in a rolling release model. Not every distro is as stable as Debian, but the joke's still on you because despite Debian's hard requirements, they still ended up encountering this bug. > IOW, I don't understand why the upstream project should be held > responsible for packages distributed by downstream distributions > without testing them well enough to let users use them without pain, > and why should the upstream project introduce options to support such > "broken" distros.=C2=A0 It isn't fair to expect us to solve such problems= , > because they should be solved elsewhere. There are limits to testing. When I added native comp to Guix, I gave folks in the mailing lists a heads up to try their own configuration and report bugs. But people don't always read the mailing lists and therefore aren't aware of upcoming changes that may break their system, and I personally can't test every Emacs configuration in existence. > > > And again, disabling native compilation is a method that doesn't > > > allow them fine-tuning anyway, so I fail to see how it could help > > > them here.=C2=A0 If the problem is real (and I don't yet see it is), > > > we should perhaps discuss its details and invent some new method > > > of disabling compilation with finer granularity. > > The granularity here is disabling compilation of anything that > > isn't already compiled =E2=80=93 this makes it so that people can still= use > > their Emacs for byte compilation by invoking it with "emacs -Q", > > they just won't compile anything that their package manager hasn't > > compiled for them. >=20 > That's quite a blunt weapon.=C2=A0 Why not tell them to stay with Emacs > 27, until the problems are solved, or install Emacs 28 without native > compilation? That's what they're currently resorting to. Guix already supports this use case. > They are similarly drastic solutions, but they are already available > and will definitely work. Yet they are suboptimal, particularly on traditional distributions, that don't support this use-case well. > > > I don't think you can set native-comp-eln-load-path to nil.=C2=A0 The > > > last entry, the one which points to where the preloaded *.eln > > > files live, must be there, or else Emacs will refuse to start.=C2=A0 > > > And at least one other directory should be there as well, because > > > if and when some package advises some primitive, Emacs will need > > > to generate and compile a trampoline .eln file.=C2=A0 But yes, if > > > users want to prevent loading from a certain directory, they can > > > remove it from native-comp-eln-load-path, provided that the two > > > necessary entries are still left in the list. > > I already found this annoying while implementing native compilation > > as part of our emacs-build-system (i.e. the build system used to > > compile Emacs packages), and I find it extra questionable that > > users on traditional distros, where they don't usually get to > > choose their Emacs, have no means of disabling this loading. >=20 > You mean, you find the loading of preloaded *.eln files at startup > annoying?=C2=A0 Then you should know that this is the best solution we > found for dumping Emacs with natively-compiled preloaded code. No, I find it annoying that Emacs supposes it has a writable eln-cache always. This is not the case in typical package manager scenarios and it also isn't the case when users choose to make (parts of) their $HOME read-only, which is a supported configuration in Nix and Guix. I can't think of a good reason why one would want to assume this invariant. > If you know of a better solution that doesn't suffer from any fatal > issues we found with the alternatives, please suggest such solutions, > and we will definitely consider them. I haven't read the discussions around the alternatives, but couldn't you just generate one trampoline per function which you use as soon as it's advised? Also, how come advice isn't breaking byte-compilation in exactly the same manner? > And again, if Emacs with natively-compilation is annoying, by all > means offer your users an Emacs built without natively-compilation. > This is supported and will continue to be supported for the > observable future. >=20 > > Calling back to the earlier point on measuring performance, an easy > > way to measure performance between bytecode and native code (in a > > benchmark) would be to simply disable native code loading in one > > process.=C2=A0 But here, it requires two separate builds of Emacs. >=20 > As I told earlier, disabling loading of native code made no sense to > us while Emacs 28 was in development; it still doesn't.=C2=A0 Either one > wants native-compilation, or one doesn't.=C2=A0 Making Emacs code more > complicated and harder to maintain due to features that make no sense > to us is a non-starter.=C2=A0 I see no problem with having to use a > separate build, since building a release tarball takes a minute or so > on a modern system.=C2=A0 And distros should definitely have a build > without native-compilation on offer, for a variety of valid reasons. I don't think that asking distros to package every Emacs variant twice is a great idea. At Guix, we prefer to offer the most complete version of a package, so we ship with native compilation enabled. If a user wants a UI, but no native compilation (i.e. neither emacs nor emacs- minimal), they have to roll their own package description. >=20 > > While bytecode performance on such machines might too be slow (but > > perhaps tolerable for the task), ahead-of-time compilation, perhaps > > with offloading, is preferable. >=20 > I recommend against this, because it is impossible to rely on AOT > installations to never compile at run time.=C2=A0 Users cannot rely on > that, and should be advised accordingly. But why can't they? > > For another, it can cause bugs like [2]. >=20 > That bug by itself (the cause of massive launching of async > subprocesses) was never explored or described in that thread?=C2=A0 It > seems like the discussion switched to looking for ways of disabling > native-compilation right away, without a good understand of what was > happening.=C2=A0 Or did I mis something?=C2=A0 Async compilation by defau= lt > never launches more subprocesses than half the execution units of the > CPU, so what is described there should be carefully investigated and > the findings described. It'd be weird if someone found a counterexample to the above statement. > The other problem in that discussion, with warnings during async JIT > compilation is well-known, was reported several times, and the > culprit is always in the 3rd-part packages being compiled, which > should be fixed.=C2=A0 In any case, those are just warnings in almost all > cases, so their only adverse effect is annoyance (that can be > suppressed by clicking the button in the message). I read no such problem in that discussion. Do we read the same thread? > Again, I see no reason to blame the upstream project for these > issues. They should be solved by the offending 3rd-party packages, > and the distro should ideally uncover and fix them before they get to > users (I presume that you build and compile the add-on packages you > offer?). I'd like to tap at the "rolling release distro is not Debian" sign, but again, stable distros like Debian are experiencing issues with native compilation. > > > Thanks for the explanations.=C2=A0 I still think the reasons for > > > disabling native compilation are rather weak at best, and the > > > users' requests to allow that based on either bugs that need to > > > be solved or surprise and fears that have no real basis.=C2=A0 > > > Moreover, disabling native compilation is a very blunt instrument > > > that cannot be applied better than the existing ones, like no- > > > native-compile (and a few others that we didn't mention; see the > > > defcustom's in comp.el). > > Which defcustom? >=20 > Begin with those described in the ELisp manual, in the > "Native-Compilation Variables" node.=C2=A0 And my recommendation is to > review _all_ of the defcustoms in comp.el The only one I found is setting native-comp-speed to -1. Is that the solution? It doesn't appear to be. > > I fear that for all of its customizability, Emacs is > > currently lacking a good way of disabling native compilation > > outside of package management. >=20 > Yes, because, as mentioned, this makes no sense. I think you'll find it does. > And disabling native-compilation completely is currently technically > impossible (for the same reason: Emacs wasn't designed to support > that because it didn't and still doesn't seem needed). >=20 > > I also tried setting no-native-compile globally, but it seems to > > only have an effect as a file-local variable. >=20 > Yes, as designed.=C2=A0 This variable is the equivalent of no-byte- > compile, and works very similarly. >=20 > To summarize: native compilation in a build which supports it is > ubiquitous, and is not designed to be disabled except by > no-native-compile on a file by file level.=C2=A0 If a more general > disabling is needed for some reason, users should simply use a build > without native-compilation.=C2=A0 It's the same as various toolkit builds= : > if the toolkit is broken or doesn't fit the user's needs, those users > should install a build with a different toolkit. Pardon my French, but that thinking in and of itself is broken. Native compilation is not a choice in which you pick the one that most suits your fancy from a range of options =E2=80=93 it could be that if you allowe= d the user to choose between libgccjit, clang and some other compilers that shall not be named, not that I recommend you implement this. As such, I think users who do want to use native compilation should get some more say in when, where, and what to compile. Cheers