From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Newsgroups: gmane.emacs.bugs Subject: bug#55305: 28.0.50: With async nativecomp, package manager fails to load hyperbole-autoloads.el before compilation Date: Sun, 15 May 2022 16:12:48 -0400 Message-ID: References: <83czgekby6.fsf@gnu.org> <83bkvykbpx.fsf@gnu.org> <83a6bik9vi.fsf@gnu.org> Reply-To: Stefan Monnier Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33692"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) Cc: rsw@gnu.org, rswgnu@gmail.com, 55305@debbugs.gnu.org, akrl@sdf.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun May 15 22:13:14 2022 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nqKc5-0008X5-A3 for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 15 May 2022 22:13:13 +0200 Original-Received: from localhost ([::1]:48872 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nqKc4-00058m-80 for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 15 May 2022 16:13:12 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:58360) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nqKbu-00058V-Gj for bug-gnu-emacs@gnu.org; Sun, 15 May 2022 16:13:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:56918) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nqKbu-0003Ol-7w for bug-gnu-emacs@gnu.org; Sun, 15 May 2022 16:13:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nqKbt-0002j6-Sy for bug-gnu-emacs@gnu.org; Sun, 15 May 2022 16:13:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 15 May 2022 20:13:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55305 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: moreinfo Original-Received: via spool by 55305-submit@debbugs.gnu.org id=B55305.165264557910472 (code B ref 55305); Sun, 15 May 2022 20:13:01 +0000 Original-Received: (at 55305) by debbugs.gnu.org; 15 May 2022 20:12:59 +0000 Original-Received: from localhost ([127.0.0.1]:50815 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nqKbq-0002io-Nj for submit@debbugs.gnu.org; Sun, 15 May 2022 16:12:59 -0400 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:31189) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nqKbp-0002ia-Bg for 55305@debbugs.gnu.org; Sun, 15 May 2022 16:12:57 -0400 Original-Received: from pmg1.iro.umontreal.ca (localhost.localdomain [127.0.0.1]) by pmg1.iro.umontreal.ca (Proxmox) with ESMTP id F314B100163; Sun, 15 May 2022 16:12:51 -0400 (EDT) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg1.iro.umontreal.ca (Proxmox) with ESMTP id 4DC7F10000A; Sun, 15 May 2022 16:12:50 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1652645570; bh=VOSjhnjVuMZTJedgaalZ7/ijMuZ8/9zsTNHQP2BZL8M=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=bZ/Y6gRH3awQb4/xJRh2HYQyFutXC6At+AAoMvolA33jWjIqdzZjUOgPsJ+pOc+T4 GF/Fe4o+kd1C2yWdHpOlfqjgR0MGEOoi0Loot1Cayx494yXoUaJgMgQ0tMZrt6bi+4 HJmeMEFzwRlrgyAYFz0ntWVeWi2PnpVxR83sJvW1EA5vhIWsJjS4Q9PJMmKJg6gnCR 23tjsZOymT85MHluWNMMq7eCv3OcTpAujAT4z5VMJym7pJcktJJrhxRxQdeFMOcAOK J/UUMp6q9A9Shj+2H9G3jCfYTUo5m/yXD5KBTyuE7OI9TWFk/hSLm4bHGFRtnHnMwL mA25K7R0oB94w== Original-Received: from pastel (unknown [45.72.221.51]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id F0920120298; Sun, 15 May 2022 16:12:49 -0400 (EDT) In-Reply-To: <83a6bik9vi.fsf@gnu.org> (Eli Zaretskii's message of "Sun, 15 May 2022 20:01:53 +0300") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:232342 Archived-At: >> So in order for the compilation to happen correctly, our async workers >> need to mimic to some extent the currently running Emacs session. > That was never the way byte-compilation worked in Emacs. It has never been officially documented, you're right. But all the files bundled in Emacs are byte-compiled in an Emacs session where `lisp/loaddefs.el` has already been loaded and many packages rely on that to minimize the amount of explicit `require` they use (and to break some cyclic dependencies). In ELPA packages, that translates into an assumption that the package's `-autoloads.el` has already been loaded (which is indeed the case during `package-install` and is also the case when compiled via the code in GNU ELPA's `elpa-admin.el`). > We have all those 'require' and 'eval-when-compile' things precisely > so a file can tell the compiler what is needed for the compilation. > And we _need_ a way to make the compilation be completely independent > of any local customizations or installed packages. When a package uses some other package's macro, it necessarily depends on the locally installed packages to be compiled correctly. Until now `comp.el` limits the support for "local customizations or installed packages" to the act of propagating the current session's `load-path`. In theory, it could be sufficient. But this is not the same as what has been provided for the last ten years when compiling ELPA packages, so it will inevitably bump into packages for which it breaks compilation (such as Hyperbole). I'm not claiming that calling `package-activate-all` is right for reasons of principle. We sadly never clearly defined what it is that a package can count on. In practice ELPA packages have been able to count on the fact that their autoloads (and their dependencies's autoloads) have all been loaded (which also implies that all those packages have been added to the `load-path`). > I'm firmly against doing this. OK. >> AFAIK the only way to make async compilation work reliably is to make it >> generate the `.eln` file without using the `.el` file (i.e. using the >> `.elc` file instead, which can be compiled without having to load any >> user-installed file, expand any macro, or run any user-installed >> function). That will also save us from mis-compiling file and from >> (re)emitting compilation warnings. > This is unrelated, I firmly disagree with this, but I agree that it's a side discussion because we haven't yet found anyone motivated to tackle this problem in our native-comp pipeline. So in the mean time we have to paper over the consequences. Stefan