From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Max Nikulin Newsgroups: gmane.emacs.bugs Subject: bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code Date: Fri, 5 May 2023 11:18:17 +0700 Message-ID: <6070e598-7dee-1b7a-7f97-26a90618cb7a@gmail.com> References: <17b74a48-94e1-9106-cc79-d31972313910@gutov.dev> <837cujaqzq.fsf@gnu.org> <87wn2ilwed.fsf@localhost> <83zg7ealrd.fsf@gnu.org> <87pm8a8dx4.fsf@localhost> <87pm7vt0mx.fsf@localhost> <87cz3k8i27.fsf@localhost> <87sfcfdldt.fsf@localhost> <87bkj1g10g.fsf@localhost> <1c5d0ff0-5bae-1123-d2f7-64d9013fbc0f@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="21210"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Cc: Ihor Radchenko , 62762@debbugs.gnu.org, bzg@gnu.org, dmitry@gutov.dev, Alan Mackenzie , Eli Zaretskii To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Fri May 05 06:19:22 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pumuf-0005FC-O0 for geb-bug-gnu-emacs@m.gmane-mx.org; Fri, 05 May 2023 06:19:21 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pumuO-0000og-HG; Fri, 05 May 2023 00:19:04 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pumuM-0000oJ-Vf for bug-gnu-emacs@gnu.org; Fri, 05 May 2023 00:19:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pumuM-00068R-NO for bug-gnu-emacs@gnu.org; Fri, 05 May 2023 00:19:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pumuM-0000sk-Ed for bug-gnu-emacs@gnu.org; Fri, 05 May 2023 00:19:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Max Nikulin Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 05 May 2023 04:19:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62762 X-GNU-PR-Package: emacs Original-Received: via spool by 62762-submit@debbugs.gnu.org id=B62762.16832603083341 (code B ref 62762); Fri, 05 May 2023 04:19:02 +0000 Original-Received: (at 62762) by debbugs.gnu.org; 5 May 2023 04:18:28 +0000 Original-Received: from localhost ([127.0.0.1]:52781 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pumtn-0000rn-TB for submit@debbugs.gnu.org; Fri, 05 May 2023 00:18:28 -0400 Original-Received: from mail-lf1-f46.google.com ([209.85.167.46]:56723) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pumtl-0000rS-Rv for 62762@debbugs.gnu.org; Fri, 05 May 2023 00:18:26 -0400 Original-Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-4edc114c716so1489856e87.1 for <62762@debbugs.gnu.org>; Thu, 04 May 2023 21:18:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683260300; x=1685852300; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=SwR8kN6WtvVBfiJkKbaL0hFXbRWEHcFKsGp3opUIi+U=; b=joYQXpHxkUu4DJB5G0a/kHnrW6knSmBqxuDzP16iOChAra6JHP5QqQU+xdFxH3aAK4 eJh1apH+wSxb5sa6g4Fs04TOsphLbYgm3mkbdy+SMf8GaUjTdTs7W2PoO2NRuls8kzyY AhAf54Br3uKRH54Gio4Ujk6vTLbHYpnH81ud6DF9exImLNUHN4QxerNZK+Yw1mhco2Ce 9k2pDP5nkYtRGTrVnW7lgP9ALKJ/b5t4rNBFWs1HxxjxGcWzCmfJWK4Kpw1Qza+pR6Z/ DgCHQWzkCR/UKK3hrpmueYecVULMUQZRDLq6JWrSqJ66CVyzQyXCciXlem30ZyyZZoTY H/ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683260300; x=1685852300; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SwR8kN6WtvVBfiJkKbaL0hFXbRWEHcFKsGp3opUIi+U=; b=HtsxEQVtFVVqGLetN15NRR4ta6Di740efL6oaKeqm3Nu1Y2R5uzGUDadFZgFhndZB1 noqxVo3I9wNIo80e0RpumuFHyMdgDZ3fmiBzoa+qly+7gE5MFZA5XfLlYjh5BPYwjBG/ oYj+UPVmiTSr0vixjIVd0uD90N+zOVzeML+GsbWgqAhe/ioT1I/H3xRxpxHGO4qVE65l xpWXUfNNQ96iAl1Vp9aRfk/CzGlLdYXZ7wPq4EYzftR2Gww8pjas+tPitBH0Q56dV787 L3f3CLVa7RTSi/j5o6I0Tsf0RYjfTUZdnRyrKbRk4bKCSvyg0x+5gKI5c9dTw0dGu/md m1pg== X-Gm-Message-State: AC+VfDyNi4+Bm9cUA3eFyOnNQzmvRmM5IQVK1sjyXwq00MAY2o/QKlSw //g3nJG7oOK5RD134M287mY= X-Google-Smtp-Source: ACHHUZ4NqIz5GYwwzTjkKPjQyegAZKJnk+aDKO+XV2n1410pHRtJ6tiWporylXfsBaZnHkOHuKi9Ng== X-Received: by 2002:ac2:4902:0:b0:4b5:9b8f:cc82 with SMTP id n2-20020ac24902000000b004b59b8fcc82mr148250lfi.0.1683260299364; Thu, 04 May 2023 21:18:19 -0700 (PDT) Original-Received: from [192.168.0.101] (nat-0-0.nsk.sibset.net. [5.44.169.188]) by smtp.googlemail.com with ESMTPSA id d24-20020ac24c98000000b004eafabb4dc1sm127604lfl.250.2023.05.04.21.18.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 04 May 2023 21:18:18 -0700 (PDT) Content-Language: en-US In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:261046 Archived-At: On 05/05/2023 04:53, Stefan Monnier wrote: >> First of all, I think, incremental builds are broken in Emacs. > > Indeed, but in practice they usually work fine. ... > Also, the use of `load-prefer-newer` in lisp/Makefile eliminates most of > the brokenness we used to have in our incremental builds. `load-prefer-newer' is a kludge as well. E.g. Python people migrated from comparison of timestamps to inscribing of .py file hash into byte compiled .pyc files. So if the hash in .pyc does not match .py content then .pyc file is recompiled or just ignored. By the way, `load-prefer-newer' make things even worse in the case of `org-assert-version' when full dependency tree is not available. I have tried to add dependency for all lisp/org/*.elc files on lisp/org/org-macs.elc and lisp/org/org-version.el, but it is not enough. During recompilation `require' prefers .elc files (.el files not changed, so they are older) with inscribed stale `org-version' causing compilation error. If .el files were loaded, recompiling would be slower, but with up to date macro definitions. So in general, due to `load-prefer-newer' the chance to get stale macro during incremental build is higher than .el files are preferred. I admit, it is not common case when definition of a macro depends on another macro loaded from another file. I am unsure if it is possible to express in make that all lisp/org/*.elc files must be removed on change of lisp/org/{org-version.el,org-macs.el}. It might alleviate the issue till implementation of properly dependency tracking. >> `org-assert-version' is just the most apparent manifestation. > > AFAIK `org-assert-version` tries to solve a different problem. Of course `org-assert-version' was introduced to solve another problem, but it highlighted issues with dependency handling during incremental builds. > Indeed, > we recently introduced `org--inhibit-version-check` specifically so as > not to use `org-assert-version` for Emacs's broken incremental builds > (i.e. we decided we preferred that brokenness there). `org--inhibit-version-check' is even uglier kludge that partially deactivated `org-assert-version' and just hides the issue with dependencies for incremental builds. >> gcc supports generation of dependency files as a side effect of >> preprocessing for decades, see > [...] >> Perhaps a similar trick may be done in elisp by advicing e.g. `load'. Another idea: while single file is compiled per emacs invocation `load-history' may be compared before and after byte compilation. > I can already warn them that an > important problem will be the presence of cyclic dependencies. In the C and C++ world the solution for cyclic dependencies is forward declarations. Some kind of such approach I see in Org as well. lisp/org/ol.el and lisp/org/org-element.el are mutually dependent. org-element.el requires 'ol, while the latter just declares functions from 'org-element. When dependency files are generated, it is possible to create a report on cyclic dependency and to disentangle them. >> Instead of dependency tracking Makefile in Emacs uses much more limited >> approach with the main-first target. > > Hmm... no, `main-first` doesn't have much to do with dependencies (and > things like deciding when to *re*compile a file), it's concerned with > compile-time performance (typically for the first compilation, when the > `.d` dependencies wouldn't be available yet). My impression is that main-first defines order of compilation. In this sense it servers for the same purpose as dependencies. >> By the way, generated dependency map might help to properly reload a package >> update without restarting of Emacs. > > BTW, I was reminded recently that `load-history` keeps track of > `require`s so we already have most of that info at hand. I agree that heuristics based on `load-history' mostly works, but I suspect corner cases might exist. I have not inspected `org-reload' closely and what Emacs core offers instead. > The idea was to check the shadowing only for one specific file, so as to > try and keep the added cost in check. That specific file must be checked for each file loaded from the updated directory. So hash O(1) should be better than scanning the list. >> I am unsure that path comparison is able to detect a problem when a user >> reloads Org after git pull and compiling new version. > > If you mean `git pull; make` in Emacs's source code, I mean reloading of updated Org from a clone of the org-mode repository to emacs installed as a system package (or as an independent custom built). A recent case where origin of the problem remained obscure: Colin Baxter to emacs-orgmode. Why am I being told to use "straight.el"? Fri, 21 Apr 2023 10:42:27 +0100. https://list.orgmode.org/87ildpbmgs.fsf@yandex.com Notice that mixed version loading may happen without changing of `load-path'. New org is loaded from the same directory. > If you mean `git pull; make` in Org's source repository, then I must > admit that I don't have much experience with it, but make Org's make > file could set `load-prefer-newer` Not enough, see beginning of the message. > `my-require-with-shadow-check` is instead aimed at the case where the > users try to load a mix of two different Org versions in the same Emacs > sessions, either because of things like: > - they're byte-compiling Org-2 in an Emacs that has Org-1 already loaded. Does in help in the following case? 1. Base Org part is loaded on opening of some .org file. 2. Org in that directory is updated and recompiled. 3. New Org feature is loaded (autoloaded or by explicit call of e.g. (require ob-shell)) > I my own experience `git pull; make` in Emacs's source code has > virtually never caused me trouble with Org files, I recall issues after introducing of `org-encode-time' and finally Ihor replaced original macro to a less efficient function. However I am not sure what was the real reason: - Some issue with conditional macro definition and native compilation - Incomplete incremental rebuilds - Macro was not properly defined so it was disappearing from byte compiled code. However "undefined" complains might be similar to what I saw for `org-assert-version'. >> In respect to incremental builds, `org-assert-version' is a disaster. >> Any update requires full recompiling. > > Indeed, I used to disable it locally (before `org--inhibit-version-check`). By disaster I mean increased build time. `org--inhibit-version-check' allows mixed version compilation for macros unrelated to `org-assert-version', it is another disaster. > Maybe a clear set of examples of the kinds of problems that > `org-assert-version` aims to catch would be a good start. > For me it's still not really clear. - You are almost certainly going to ignore package.el issues in released Emacs versions including 28. - Besides package.el there are other package managers. I have heard of weird issues with straight.el and org, but I have no links describing details. - Some users just clone org-mode repository and add that directory to `load-path'. Directory remains the same after updates. - Users tend to load some part of org before adding new version to load-path: explicit require, through dependencies, unsure if org file as startup screen may cause it as well - User wish to update Org without restarting of Emacs. Recompiling may be called from Emacs (e.g. package.el) or by make or other scripts. The goals: - prevent mixed version compiling - prevent mixed version loading - error message should be clear for users explaining what actions they should perform for recovery from broken state. I have not isolated an issue in Emacs-28 with package.el and difference with emacs binary is running from source tree or from install tree (.el+.elc vs. el.gz+.elc files in the directory with built-in Org and preference of .el vs. elc in require), so I can not tell if it affects other package managers in newer Emacs. It is the case when `org-assert-version' prevents loading of mixed compilation result, but user experience is terrible since simple recompilation does not help.