From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Newsgroups: gmane.emacs.bugs Subject: bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code Date: Thu, 04 May 2023 17:53:34 -0400 Message-ID: References: <17b74a48-94e1-9106-cc79-d31972313910@gutov.dev> <837cujaqzq.fsf@gnu.org> <87wn2ilwed.fsf@localhost> <83zg7ealrd.fsf@gnu.org> <87pm8a8dx4.fsf@localhost> <87pm7vt0mx.fsf@localhost> <87cz3k8i27.fsf@localhost> <87sfcfdldt.fsf@localhost> <87bkj1g10g.fsf@localhost> <1c5d0ff0-5bae-1123-d2f7-64d9013fbc0f@gmail.com> Reply-To: Stefan Monnier Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="12741"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: Ihor Radchenko , 62762@debbugs.gnu.org, bzg@gnu.org, dmitry@gutov.dev, Alan Mackenzie , Eli Zaretskii To: Max Nikulin Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu May 04 23:54:25 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pugu8-000323-DP for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 04 May 2023 23:54:24 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pugto-00017s-Po; Thu, 04 May 2023 17:54:04 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pugtm-00017i-8O for bug-gnu-emacs@gnu.org; Thu, 04 May 2023 17:54:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pugtl-00045T-Vt for bug-gnu-emacs@gnu.org; Thu, 04 May 2023 17:54:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pugtl-0002sN-Po for bug-gnu-emacs@gnu.org; Thu, 04 May 2023 17:54:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 04 May 2023 21:54:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62762 X-GNU-PR-Package: emacs Original-Received: via spool by 62762-submit@debbugs.gnu.org id=B62762.168323722611030 (code B ref 62762); Thu, 04 May 2023 21:54:01 +0000 Original-Received: (at 62762) by debbugs.gnu.org; 4 May 2023 21:53:46 +0000 Original-Received: from localhost ([127.0.0.1]:52230 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pugtV-0002rp-CQ for submit@debbugs.gnu.org; Thu, 04 May 2023 17:53:45 -0400 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:60910) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pugtT-0002rY-NR for 62762@debbugs.gnu.org; Thu, 04 May 2023 17:53:44 -0400 Original-Received: from pmg3.iro.umontreal.ca (localhost [127.0.0.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id 0E914442A84; Thu, 4 May 2023 17:53:38 -0400 (EDT) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id C953E442A80; Thu, 4 May 2023 17:53:35 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1683237215; bh=fGRYnX8OrjKs9YaYMiC2Tfue1nqLWaxKRUNoMwEJIa8=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=g9BGrfhMdh3yb4EsZp5gZ9lnKWwSOtVDI2NuokS3+8Mb4RQtgwpoPHZDFPqu9jo/Y 5QFsA4Z5vuzqZu0Bh0J8W981rSZrO4K0d5SsHpX7Gl7BMhC6uuOMlP0qENj5I3M3rg 3kaomz463U4fBfGudmmNnInSnAgrKPc4QAY9qMt910T7tFQDOK9DVF0YzRkvl9PomB PaoLAwDcj/pc9na7/1NapTP4x6pTIDNkFiFRdnUDhWpbG6B6NNKfOLigOmF/+CFXNU 0g5OW5n43GOcqBbu+zn01euXTscbNBkRuYZqKRxGgKzHRrfm0962UzP1Mf6nSpQD5H 96f6o3Oaqxvsg== Original-Received: from lechazo (lechon.iro.umontreal.ca [132.204.27.242]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 68AD2120074; Thu, 4 May 2023 17:53:35 -0400 (EDT) In-Reply-To: <1c5d0ff0-5bae-1123-d2f7-64d9013fbc0f@gmail.com> (Max Nikulin's message of "Thu, 4 May 2023 22:31:43 +0700") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:261034 Archived-At: > First of all, I think, incremental builds are broken in Emacs. Indeed, but in practice they usually work fine. So in the absence of a better solution, we stay with this hack and tell people to try `make bootstrap` when there's a problem. Also, the use of `load-prefer-newer` in lisp/Makefile eliminates most of the brokenness we used to have in our incremental builds. > `org-assert-version' is just the most apparent manifestation. AFAIK `org-assert-version` tries to solve a different problem. Indeed, we recently introduced `org--inhibit-version-check` specifically so as not to use `org-assert-version` for Emacs's broken incremental builds (i.e. we decided we preferred that brokenness there). > gcc supports generation of dependency files as a side effect of > preprocessing for decades, see [...] > Perhaps a similar trick may be done in elisp by advicing e.g. `load'. I encourage people to try that out, yes. Having been there (many years ago), I can already warn them that an important problem will be the presence of cyclic dependencies. > Instead of dependency tracking Makefile in Emacs uses much more limited > approach with the main-first target. Hmm... no, `main-first` doesn't have much to do with dependencies (and things like deciding when to *re*compile a file), it's concerned with compile-time performance (typically for the first compilation, when the `.d` dependencies wouldn't be available yet). > That is why incremental build may easily result in mixed-version > compilation. Actually, now that we set `load-prefer-newer`, it is rarely a problem. [ Of course, it can still be a problem, but only in cases such as when a macro is changed such that the old expansion is not compatible with code using the new definitions. This happens (e.g. when we changed structs to use `record` instead of `vector`), but not fairly rarely. ] > By the way, generated dependency map might help to properly reload a package > update without restarting of Emacs. BTW, I was reminded recently that `load-history` keeps track of `require`s so we already have most of that info at hand. > I have heard that some users complain concerning Emacs startup time. Likely > a hashmap will be required in addition to the `load-history' list to avoid > performance degradation. I am unsure if the following idea have some > benefits: check shadowing when `load-path' is modified, not for each file > loaded from the newly added directory. The idea was to check the shadowing only for one specific file, so as to try and keep the added cost in check. Checking shadowings when `load-path` is modified is harder (because `load-history` doesn't remember which name was used to load a file, so it could be confused and think a file like `lisp/progmodes/compile.el` shadows a previously loaded `lisp/cedet/srecode/compile.el`) and could be more costly (because it has to check for "all" files). > I am unsure that path comparison is able to detect a problem when a user > reloads Org after git pull and compiling new version. [ AFAICT, there are different situations with related yet different problems that manifest slightly differently and that may be solved differently as well, so please be specific when you mention problematic scenarios. ] If you mean `git pull; make` in Emacs's source code, then AFAIK this should almost never be a problem (thanks to `load-prefer-newer`), and if it is, it's a general problem that's not specific to Org, and "we" changed `org-assert-version` so as not to bother trying to solve this case. If you mean `git pull; make` in Org's source repository, then I must admit that I don't have much experience with it, but make Org's make file could set `load-prefer-newer` (or use some hack to autogenerate `.d` dependency files :-). `my-require-with-shadow-check` is instead aimed at the case where the users try to load a mix of two different Org versions in the same Emacs sessions, either because of things like: - they're byte-compiling Org-2 in an Emacs that has Org-1 already loaded. - they inadvertently end up loading part of Org-1 early in their init file before they set their `load-path` to point to Org-2. I understand Org people have had lots of problems with mixed-versions in the past, but I personally don't know which circumstances have been more often at the source of those problems, so maybe my suggestion addresses a problem that's not very important, indeed. I my own experience `git pull; make` in Emacs's source code has virtually never caused me trouble with Org files, and it's never cause any trouble in my checkout of Org's source code either, which is why I'm suggesting something like `my-require-with-shadow-check`, which targets other scenarios. > In respect to incremental builds, `org-assert-version' is a disaster. > Any update requires full recompiling. Indeed, I used to disable it locally (before `org--inhibit-version-check`). > It is a reason why I always considered it as a kludge. Unfortunately > I do not have a better idea. Maybe a clear set of examples of the kinds of problems that `org-assert-version` aims to catch would be a good start. For me it's still not really clear. > The issue is not limited to package.el or replacing built-in package. > Various ways to load packages are used, so a couple of versions may > appear in load-path even for non built-in package. So far I haven't seen enough of the diversity of situations are (source of) problems to have a clear idea of what a good solution should look like, admittedly. `my-require-with-shadow-check` is only based on my intuition that it might catch the most common issues where two different Org versions are used in the same Emacs session, which seems to cover most of the problems (except for those due to `git pull; make`, which seem to be qualitatively of a different kind). Stefan