From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?utf-8?Q?Przemys=C5=82aw_Kryger?= Newsgroups: gmane.emacs.devel Subject: Issues when upgrading built in packages Date: Wed, 27 Nov 2024 15:44:15 +0000 Message-ID: <82F95411-5215-43C9-95C9-80ECE8EE8AFE@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="11906"; mail-complaints-to="usenet@ciao.gmane.io" To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Wed Nov 27 16:45:16 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1tGKE8-0002wx-8U for ged-emacs-devel@m.gmane-mx.org; Wed, 27 Nov 2024 16:45:16 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tGKDR-0003uk-TY; Wed, 27 Nov 2024 10:44:33 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tGKDP-0003u9-Hk for emacs-devel@gnu.org; Wed, 27 Nov 2024 10:44:31 -0500 Original-Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tGKDN-0006fb-G3 for emacs-devel@gnu.org; Wed, 27 Nov 2024 10:44:31 -0500 Original-Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-434ab114753so6204905e9.0 for ; Wed, 27 Nov 2024 07:44:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732722267; x=1733327067; darn=gnu.org; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:from:to:cc:subject:date:message-id:reply-to; bh=3DhVs8rq2ezkB9ux2Kx+tJJSAWnxAvtvcUPMbgRPkcE=; b=E3XrqGxtjK286Ywatz7Wzq5ThofrgsqpCl+XQ2mQxb+rpY98kbYiXBdxn3OqDMdavI rACt4hcn8poeXcA7/XbrURfLrJV3C3inUt4xQJ15tGimTPQdNntOKxQ3RhwfKL2vT7wz xm5pITDVmyCtyIFpQlJLXYP/AkF7yBIQX8AQCc9tJ5QcTyhmn+xpLwaNtwHQbDqAvcLB aRn9taM8Qh55vGZ/UrKGpl/6FQNnzXR7P5tWBoUym6Pl3s33jfuD05Cl9x9TPHIH0r4d QFr+FPz2tsN1WGnEbvo+Ohh1r4CC/fhnzhhR/9LX/DA1IxuanixM3JoTfK4aJ2QYs34X WeAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732722267; x=1733327067; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3DhVs8rq2ezkB9ux2Kx+tJJSAWnxAvtvcUPMbgRPkcE=; b=aP9xEOfg7Gw9U8xsRSwbaYyQKRmK5iSsW/Jda3vikKU9jsUKfzWNtweh4gO+05bkjn L+uhk4YBV1mo9RK2lcEldcULT6a0tPOcC14RBfUsdxBSZCxt5kfYRTQ2oAL5n7nF/pPQ ZVDHkuPAIey6fdQjjQp9t9m66QN5RMIF8EeFBue09skz9JZj+dBvh1QajPLCWUb8Loly o6XxOiG8rBCDz6Hkk3S/89ARUqkW483R5TmUNK3C/BbS3hYYzxPl+Ixrm9dqkaW7Gb2q bvnPfKMsriJEEX7DmwlYlojvHCGBTw6IhOi9idEeUfR/SpCq4kI1v4rh/rg3HP8IFR1T c7nA== X-Gm-Message-State: AOJu0Yy+pGdopL0q33wu1loBH0PhuMDPT8H+OrK67mRoJihAhFxSHbps /yGdlixVWqiRWEbpSFwvVy76FGQgbsKOvYvrrzXOAnEZTumvPanDQNeJqA== X-Gm-Gg: ASbGncuiHYjHJq2RY5cyBU9d1wGYkXXPtTrmxO6TjsheAVNwj+YI+3xeXF6sAefuS5P 2d4QhbTOTeYKw/tMhN9nPbhcysusOw5aLSBl8zZS+pGmvvTPQAVNReQD60MQ5FBDmidbRGWr9M4 yy/UsodbXJ2Ojfz68C+IwzwFelmjfUBPJeqDnag1itRgffMAzflI9AS6LzZTvWMR7H5/oy/qAAM 8b1YXpXPD9cha0+J1rIqyYM8vSP5R+Sug+ZD+yqbbyLOAD5IZqTSSD+8K6s0XnP+qecg7XLFWk= X-Google-Smtp-Source: AGHT+IFKsi7qlqcGM8rXruL9h8gyqFMbdyojYBy9dyiQFgtCB73xEvf67E8jCTsjU25yZ4FAg1MYUg== X-Received: by 2002:a05:6000:2a9:b0:382:5010:c8cd with SMTP id ffacd0b85a97d-385c6ed7749mr2837623f8f.44.1732722266699; Wed, 27 Nov 2024 07:44:26 -0800 (PST) Original-Received: from smtpclient.apple ([2a00:23c4:f513:d201:84c0:d6bf:589c:8dc3]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-434aa763c50sm24707465e9.12.2024.11.27.07.44.25 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Nov 2024 07:44:26 -0800 (PST) X-Mailer: Apple Mail (2.3826.200.121) Received-SPF: pass client-ip=2a00:1450:4864:20::335; envelope-from=pkryger@gmail.com; helo=mail-wm1-x335.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, FREEMAIL_REPLY=1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:325780 Archived-At: Hello, I have been doing some work involving upgrading built in packages from ELPA. Some of examples include: seq for older Emacs, org, transient, = eglot. I noticed that sometimes, these upgrades were failing because some files couldn't be found or macro definitions were not available. See at the = bottom of this message for concrete examples. I've done some investigation and if I understood correctly the issue is = caused because `load-path' is updated only after the package is reloaded for = the first time. The gist is that `package-install` calls `package-unpack` which = in turn does: (1) Calls `package-activate-1' with :reload = (https://github.com/emacs-mirror/emacs/blob/bef9eeb/lisp/emacs-lisp/packag= e.el#L1029). That causes a call to `package--reload-previously-loaded' be made before = loading autoloads. The latter seems to be updating the `load-path' to include = the newly installed package directory, see = https://github.com/emacs-mirror/emacs/blob/bef9eeb/lisp/emacs-lisp/package= .el#L907-L910. (2) Byte compiles the package. The `load-path` already contains the = newly installed package directory. (3) Calls `package--reload-previously-loaded', see = https://github.com/emacs-mirror/emacs/blob/bef9eeb/lisp/emacs-lisp/package= .el#L1038 While I cant find example that fails on the most recent snapshot, it = doesn't seem that code has changed that much since Emacs-29.1, and I verified = with Edebug that relatively recent snapshot may still be affected. At least `load-path` doesn't include newly installed package directory when first = reload happens. To my understanding this may also cause the very same macro issue that = the first package reloading is supposed to protect (or at least this is = suggested by this comment = https://github.com/emacs-mirror/emacs/blob/bef9eeb/lisp/emacs-lisp/package= .el#L1026-L1028). For an example, say there's a file foo.el that defines a macro = `foo-macro` that uses a macro `bar-macro` defined in file bar.el. Both files are not = byte compile-able. If both of these files exist in the old version of the = package, both defining the same macros, and it happens that file foo.el is loaded = first, then it will use an old version of `bar-macro`. Is that correct? When I added expected new package directory to `load-path' the issue = doesn't happen. I.e., I am able to `package-install` built-in packages from = ELPA. Thank you, PK Examples of failures: (1) On Emacs-29.1 the seq-2.24 was not able to find seq-25: (let ((debug-on-error t) (package-install-upgrade-built-in t) (desc (cadr (assq 'seq package-archive-contents)))) (package-install desc)) Debugger entered--Lisp error: (file-missing "Cannot open load file" "No = such file or directory" "seq-25") require(seq-25) (if (version< emacs-version "25") (require 'seq-24) (require 'seq-25)) eval-buffer(# nil = "/home/runner/.emacs.d/elpa-..." nil t) ; Reading at buffer position = 1779 load-with-code-conversion("/home/runner/.emacs.d/elpa-..." = "/home/runner/.emacs.d/elpa-..." nil t) load("/home/runner/.emacs.d/elpa-..." nil t) #f(compiled-function (c) #)(("/home/runner/.emacs.d/elpa-..." . 76)) mapc(#f(compiled-function (c) #) = (("/home/runner/.emacs.d/elpa-..." . 76))) package--reload-previously-loaded(#s(package-desc :name seq :version = (2 24) :summary "Sequence manipulation functions" :reqs nil :kind nil = :archive nil :dir "/home/runner/.emacs.d/elpa-..." :extras ((:keywords = "sequences") (:maintainer nil . "emacs-devel@gnu.org") (:authors = ("Nicolas Petton" . "nicolas@petton.fr")) (:url . = "https://elpa.gnu.org/packages/seq.html") (:commit . = "27a90793a13f149121180e864fa53d68b9eac0b3")) :signed nil)) package-activate-1(#s(package-desc :name seq :version (2 24) :summary = "Sequence manipulation functions" :reqs nil :kind nil :archive nil :dir = "/home/runner/.emacs.d/elpa-..." :extras ((:keywords "sequences") = (:maintainer nil . "emacs-devel@gnu.org") (:authors ("Nicolas Petton" . = "nicolas@petton.fr")) (:url . "https://elpa.gnu.org/packages/seq.html") = (:commit . "27a90793a13f149121180e864fa53d68b9eac0b3")) :signed nil) = :reload :deps) package-unpack(#s(package-desc :name seq :version (2 24) :summary = "Sequence manipulation functions" :reqs nil :kind tar :archive "gnu" = :dir nil :extras ((:keywords "sequences") (:maintainer nil . = "emacs-devel@gnu.org") (:authors ("Nicolas Petton" . = "nicolas@petton.fr")) (:url . "https://elpa.gnu.org/packages/seq.html") = (:commit . "27a90793a13f149121180e864fa53d68b9eac0b3")) :signed nil)) (2) On Emacs-29.4 the org-9.7.14 was not able to find org-element-ast. = This is not happening every time, I guess due to order in which files are = reloaded. (let ((debug-on-error t) (package-install-upgrade-built-in t) (desc (cadr (assq 'org package-archive-contents)))) (package-install desc)) Debugger entered--Lisp error: (error "Eager macro-expansion failure: = (file-missing \"Cann...") signal(error ("Eager macro-expansion failure: (file-missing = \"Cann...")) error("Eager macro-expansion failure: %S" (file-missing "Cannot open = load file" "No such file or directory" "org-element-ast")) internal-macroexpand-for-load((defalias 'org-fold-core-region-folded-p = #'(lambda (beg end &optional spec-or-alias) "Non-nil if the region = between BEG and END is folde..." (org-with-point-at beg (catch :visible = (while (< ... end) (unless ... ...) (goto-char ...)) t)))) t) eval-buffer(# nil = "/home/runner/.emacs.d/elpa-..." nil t) ; Reading at buffer position = 38825 load-with-code-conversion("/home/runner/.emacs.d/elpa-..." = "/home/runner/.emacs.d/elpa-..." nil t) load("/home/runner/.emacs.d/elpa-..." nil t) #f(compiled-function (c) #)(("/home/runner/.emacs.d/elpa-..." . 278)) mapc(#f(compiled-function (c) #) = (("/home/runner/.emacs.d/elpa-..." . 273) = ("/home/runner/.emacs.d/elpa-..." . 274) = ("/home/runner/.emacs.d/elpa-..." . 277) = ("/home/runner/.emacs.d/elpa-..." . 278) = ("/home/runner/.emacs.d/elpa-..." . 279) = ("/home/runner/.emacs.d/elpa-..." . 280) = ("/home/runner/.emacs.d/elpa-..." . 285) = ("/home/runner/.emacs.d/elpa-..." . 286) = ("/home/runner/.emacs.d/elpa-..." . 287) = ("/home/runner/.emacs.d/elpa-..." . 288) = ("/home/runner/.emacs.d/elpa-..." . 289) = ("/home/runner/.emacs.d/elpa-..." . 290) = ("/home/runner/.emacs.d/elpa-..." . 291) = ("/home/runner/.emacs.d/elpa-..." . 292) = ("/home/runner/.emacs.d/elpa-..." . 294) = ("/home/runner/.emacs.d/elpa-..." . 295) = ("/home/runner/.emacs.d/elpa-..." . 296) = ("/home/runner/.emacs.d/elpa-..." . 297) = ("/home/runner/.emacs.d/elpa-..." . 299) = ("/home/runner/.emacs.d/elpa-..." . 300) = ("/home/runner/.emacs.d/elpa-..." . 301) = ("/home/runner/.emacs.d/elpa-..." . 302) = ("/home/runner/.emacs.d/elpa-..." . 303) = ("/home/runner/.emacs.d/elpa-..." . 304) = ("/home/runner/.emacs.d/elpa-..." . 305) = ("/home/runner/.emacs.d/elpa-..." . 306) = ("/home/runner/.emacs.d/elpa-..." . 307) = ("/home/runner/.emacs.d/elpa-..." . 308) = ("/home/runner/.emacs.d/elpa-..." . 309) = ("/home/runner/.emacs.d/elpa-..." . 310) = ("/home/runner/.emacs.d/elpa-..." . 311) = ("/home/runner/.emacs.d/elpa-..." . 313) = ("/home/runner/.emacs.d/elpa-..." . 314))) package--reload-previously-loaded(#s(package-desc :name org :version = (9 7 14) :summary "Outline-based notes management and organizer" :reqs = ((emacs (26 1))) :kind nil :archive nil :dir = "/home/runner/.emacs.d/elpa-..." :extras ((:url . "https://orgmode.org") = (:keywords "outlines" "hypermedia" "calendar" "text") (:maintainer = "Bastien Guerry" . "bzg@gnu.org") (:authors ("Carsten Dominik" . = "carsten.dominik@gmail.com")) (:commit . = "49e0c4a74ad6c22b98702cfbcefff92a95af1146")) :signed nil)) package-activate-1(#s(package-desc :name org :version (9 7 14) = :summary "Outline-based notes management and organizer" :reqs ((emacs = (26 1))) :kind nil :archive nil :dir "/home/runner/.emacs.d/elpa-..." = :extras ((:url . "https://orgmode.org") (:keywords "outlines" = "hypermedia" "calendar" "text") (:maintainer "Bastien Guerry" . = "bzg@gnu.org") (:authors ("Carsten Dominik" . = "carsten.dominik@gmail.com")) (:commit . = "49e0c4a74ad6c22b98702cfbcefff92a95af1146")) :signed nil) :reload :deps) package-unpack(#s(package-desc :name org :version (9 7 14) :summary = "Outline-based notes management and organizer" :reqs ((emacs (26 1))) = :kind tar :archive "gnu" :dir nil :extras ((:url . = "https://orgmode.org") (:keywords "outlines" "hypermedia" "calendar" = "text") (:maintainer "Bastien Guerry" . "bzg@gnu.org") (:authors = ("Carsten Dominik" . "carsten.dominik@gmail.com")) (:commit . = "49e0c4a74ad6c22b98702cfbcefff92a95af1146")) :signed nil)) =20=