From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id KFoWIXS8BmeZhAAAqHPOHw:P1 (envelope-from ) for ; Wed, 09 Oct 2024 17:25:08 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id KFoWIXS8BmeZhAAAqHPOHw (envelope-from ) for ; Wed, 09 Oct 2024 19:25:08 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=qRD1kWxI; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=Bh8xdiCK; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1728494708; a=rsa-sha256; cv=none; b=lnxwg9eTcStzxU2QAW9Vzxkp8dI0bQ/l881IsQFusftVVzXa4IDKek+A3t8rPK+L/IVlj2 9KpXYwWJNAis9K2NFY+vg2MF05JzjjBpzOHVeZjLTDm+m/pxDKMdImdKYL8qi2YAGjFvkY EaDFfFnrplZ3LiB3lGqnXth49wLjC5H1PgTjvV45/pthnPGEOrgDG7sL/Lf5KdHGanDmTP vstBeNImWD0Ncs2kp2v5cQOSG5uzY7uctl7bjOe+UtUl2Uc/ia0gBfl5y4FBvAncIzWWgH ui1HsrM+vkQ9HgwgwiNZiVk7DElNzDDfSio4Crev9j1PR0RkyMBhua723smBgA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=qRD1kWxI; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=Bh8xdiCK; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1728494708; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=nGcM6ypkWCGVfuUex0/4qocmeUiDhMYxEtANOuKUQDs=; b=kW/G+kH7IsreFzpHY663367hoI9Cw7QWH9zTPI80CPGcGrf1gDgYdNSyy/hhc0iZ2UcOB6 rie2VGNA/fkrpK9yV+SBgy4h2u/+ku5/lBETaOHuJYvUd5AjrxPKSvykEYfkPDITh4Akqr SeADRhVZ9CHxvwUQe+VxY6YRTIXfKHVs8mkMfiTeidvJvvc4WKqXqVDDoAlnbI0nBhjVTI oMTssy7v3fn8yDXtJg1J/S0AGQRj/47A3H/f/VwiD5pwCvi/roYsXibw0YrsdKWQcKFIPS jCfwVmNCSmRh8oMoYsMZQlxbpxpO5Q3z6vtHAUj15kWsDCcbk5JaTCAs/hNBOQ== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1E9A819216 for ; Wed, 09 Oct 2024 19:25:08 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1syaQg-0002y5-OP; Wed, 09 Oct 2024 13:24:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1syaQe-0002xv-TZ for bug-guix@gnu.org; Wed, 09 Oct 2024 13:24:53 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1syaQe-00082p-Cw for bug-guix@gnu.org; Wed, 09 Oct 2024 13:24:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=nGcM6ypkWCGVfuUex0/4qocmeUiDhMYxEtANOuKUQDs=; b=qRD1kWxI5yFLcBP5+2B0zt36800baHIsHEihCXfuXSaO65I3cFH1+EnYNVcUoucIQAPKpTwZM8uUUfDcz8Tvw0hVYdWnWuMYdp1zhSnpr8eEPpegJAcHAKr8lsTn9ku+ZopXW8cgXWIbXpizRAc2w4sfkhqHUShtYvS36CbkLM5ps8VFVLF4RN3TilYY9DRwRVQQV0Y3x4HHfB38CYGpAbwsmdYuZWMJoibBVkQimnO4LUO/cEumFdWCQDIwjQ8D8SWi5Wuo7bZUoHMAgftyiPcaJqw7YLs0P1kfBZIljgA+bL88wj4QraexsiEfDfh89xbkAjlN7sbhE3cLwzCMiw==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1syaQn-0003ov-VQ for bug-guix@gnu.org; Wed, 09 Oct 2024 13:25:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#73681: Maybe partly undo the patch on Elisp comp-el-to-eln-filename Resent-From: Liliana Marie Prikler Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 09 Oct 2024 17:25:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 73681 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Martin =?UTF-8?Q?Edstr=C3=B6m?= Cc: 73681 <73681@debbugs.gnu.org> Received: via spool by 73681-submit@debbugs.gnu.org id=B73681.172849465914620 (code B ref 73681); Wed, 09 Oct 2024 17:25:01 +0000 Received: (at 73681) by debbugs.gnu.org; 9 Oct 2024 17:24:19 +0000 Received: from localhost ([127.0.0.1]:57521 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1syaQ6-0003nh-0d for submit@debbugs.gnu.org; Wed, 09 Oct 2024 13:24:18 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:53556) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1syaQ3-0003nS-8u for 73681@debbugs.gnu.org; Wed, 09 Oct 2024 13:24:16 -0400 Received: by mail-wm1-f65.google.com with SMTP id 5b1f17b1804b1-430ee5c9570so36595e9.3 for <73681@debbugs.gnu.org>; Wed, 09 Oct 2024 10:24:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728494579; x=1729099379; darn=debbugs.gnu.org; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:from:to:cc:subject :date:message-id:reply-to; bh=nGcM6ypkWCGVfuUex0/4qocmeUiDhMYxEtANOuKUQDs=; b=Bh8xdiCKkAB5HJ/H2LXqWWAfFdtIyXAW6LcZlE68DrTAuXdvKGUQU4DHIeOdhPcdZS jTAfIWN+22grKSSX+bFOwlY2dF/kv1gSZCQTKXLAVU+gWhZYmLc3YfDTxS0TZBw0dLAQ ZmJSmdFdPQPB7vul21TNCK4EEu2UUMUtdE/bFt368v1Ib63u//Y6gsSHjQH3Gsby/N2S 5dpdylABvnFjxS2WHE68bJ+ndJHwI7EeMD6NPqTLFpq3MyELAHALiJEV+smIekA6z/bG 87XJKqV0VoiSeJVER54EDn0O/B5y9HqAolrlmkTtzJtU7jNJoWEKXqPYPTJiCyXFpfms pgig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728494579; x=1729099379; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=nGcM6ypkWCGVfuUex0/4qocmeUiDhMYxEtANOuKUQDs=; b=ves4imMF8i+67EgWZp6D/tMpOkNcDQS/H6LffCz2KTqKMrOLd/g064wFbzIeCQ0u1L iPUYLLbFfWrWnBciGSYe5xeC17ASxkg+fk4ap+K5OFVBSlH1WEFyH+IoywG7Nov46cpE 3FdbOR1UvHbQo/SvJJYtmyCvRILyFbo+m5tYqwy5yScWyTHWSC7/G3bxy1VGkLkh+37z KuaqvuvTqEQTBGWsp2cuRCd8KEHo835LNdJzTHd1JfqgCWiEYheBrB+LmMepDCu2sQLk Zm4ImcgHlpIXQmpF+PiXXwCWriGU1x73hYUyZJC9jKG0WQa6HKa2xMERLcuGUREUtG4e cATA== X-Gm-Message-State: AOJu0YyKU1wBa+GCldfokNl2bOYHtigoy7roKAv5vLeYU9OCpR21Qwxn R8YNTkdyCRpYRNwd3BVjWAv5dt2zkn0nGbvJ5UBOFUR0JMR9Rjvf X-Google-Smtp-Source: AGHT+IFYLJjd+P6Y9gbyMn35KlQw821v0noLmkhHGgQX50np4pkNCwk4fkY/cqOuIwHF+T5u+6YKKg== X-Received: by 2002:a05:600c:1c0b:b0:42b:ac3d:3abc with SMTP id 5b1f17b1804b1-430d70b3cb1mr36105745e9.24.1728494578780; Wed, 09 Oct 2024 10:22:58 -0700 (PDT) Received: from lumine.fritz.box (85-127-114-32.dsl.dynamic.surfer.at. [85.127.114.32]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-430ccf60953sm26175385e9.29.2024.10.09.10.22.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Oct 2024 10:22:58 -0700 (PDT) Message-ID: <518988807953a1b77acb5f9833992fa9ced883c1.camel@gmail.com> From: Liliana Marie Prikler Date: Wed, 09 Oct 2024 19:22:55 +0200 In-Reply-To: References: <58598114857dce8a25e3b4d0477d212467a0173f.camel@gmail.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.48.4 MIME-Version: 1.0 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -1.85 X-Spam-Score: -1.85 X-Migadu-Queue-Id: 1E9A819216 X-Migadu-Scanner: mx12.migadu.com X-TUID: Bb7wVSj5N2F0 Am Mittwoch, dem 09.10.2024 um 17:15 +0200 schrieb Martin Edstr=C3=B6m: > Hi, thanks for info! This email will be a bit long because I'm quite > confused and thinking aloud, so no need to reply to all of it.=C2=A0 But = I > appreciate any attempt to shed clarity! >=20 >=20 > On Tue, 08 Oct 2024 19:33:06 +0200, Liliana Marie Prikler > wrote: >=20 > > > It comes as part of the package. I don't want to assume that it > > > has been compiled, since it's fairly performance-sensitive. > > > That's why I'll either use a previously existing compiled object > > > or make a new one. > > Could you leave that decision to the user? >=20 > I'm considering it, but ran into essentially the same problem.=C2=A0 I > need a way to tell which one was loaded, of the .elc, .eln or .el. >=20 > This takes the thread a bit off-topic but ... Do you know how? >=20 > Currently I'm leaning towards an algorithm like >=20 > #+begin_src elisp > (defun guess-loaded-lib (basename) > =C2=A0 (let* ((el (locate-library (concat basename ".el"))) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (elc (locate-library (co= ncat basename ".elc"))) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (eln (comp-el-to-eln-fil= ename el))) > =C2=A0=C2=A0=C2=A0 (if load-prefer-newer > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (car (sort (delq 'nil (list el= elc eln)) #'file-newer-than- > file-p)) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (if (and (file-newer-than-file-p elc el) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 (file-newer-than-file-p eln el)) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ;; If elc is newer= than el, assume it was compiled from el, > and > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ;; therefore is sa= fe to replace with eln > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eln > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 elc)))) > #+end_src >=20 > I don't know how to "leave it to the user" any better than that.=C2=A0= =20 The user can just use the output of the help function, it will tell them whether a function is natively-compiled, byte-compiled, or just interpreted. Emacs 30 even gives you `native-comp-function-p'. See `gnu/packages/aux-files/emacs/comp-integrity[-next].el' for how we assert that our Emacs loads natively-compiled libraries. > On Guix, even in the future when we re-enable JIT compilation, this > algorithm could never return an .eln, since =3Dfile-newer-than-file-p=3D > returns nil when both paths have identical timestamps from 1970. >=20 > I found a pretty neat built-in:=20 >=20 > =C2=A0=C2=A0=C2=A0=C2=A0 (symbol-file 'SOME-FUNCTION-FROM-THE-LIB nil t) >=20 > but its internals also use =3Dfile-newer-than-file-p=3D and =3Dcomp-el-to= - > eln-rel-filename=3D, so not sure it is any more reliable than what I > have above. I'm not sure these things work the way you think. IIUC, `load-prefer- newer' should guard against the case where the .el is newer than the .el[cn], not the other way round, so it should still load the compiled/native-compiled variant if they have the same stamp. > > There is a separate load path for natively compiled files, called > > `native-comp-eln-load-path'. >=20 > Good to know! I don't know how Emacs consults it though.=C2=A0 I can't > e.g. locate an .eln of a given library by calling something like >=20 > #+begin_src elisp > (locate-eln-file "org-node-parser") > #+end_src >=20 > so it seems I must simply do: >=20 > #+begin_src elisp > (let ((el (locate-library "org-node-parser.el"))) > =C2=A0 (when el > =C2=A0=C2=A0=C2=A0 (locate-eln-file (comp-el-to-eln-rel-filename el)))) > #+end_src >=20 > which is functionally equivalent to: >=20 > #+begin_src elisp > (let ((eln (comp-el-to-eln-filename (locate-library "org-node- > parser.el")))) > =C2=A0 (when (file-exists-p eln) > =C2=A0=C2=A0=C2=A0 eln)) > #+end_src I mean, the C code is there to read (along with the patch we've made), but I can't fault you for not going that deep. =20 > > > I infer that Emacs starts with finding a library in load-path, > > > then converts the path with `comp-el-to-eln-filename`, and checks > > > if that file exists, then loads it. > > >=20 > > > And crucially, it is not just about the filepath, the function > > > hashes the file contents as well. That ensures that the output > > > path is always different if the source file changes. > > I think relying on such implementation details is perhaps permitted > > if it's inside of Emacs itself, but even then it clashes with our > > expectation that Emacs be graftable. >=20 > OK, so if passing "file.el" to =3Dcomp-el-to-eln-filename=3D produces a > path with a hash of the content, like: >=20 > =C2=A0=C2=A0=C2=A0 .../eln-cache/29.4-HASH/module/file-HASH-BASED-ON-CONT= ENT.eln >=20 > this would make Emacs non-graftable.=C2=A0 Because the hash would change > after file.el is upgraded.=C2=A0 I suppose that makes sense, thanks! >=20 > Maybe, as a hack, the graft process could symlink the pre-graft > filename to the post-graft filename... so we don't have to alter the > behavior of =3Dcomp-el-to-eln-filename=3D. The grafting process doesn't look that deep into file names. On paper, that could work, since the length of the name is preserved, but it'd also make the grafting process take longer and not really add all that much to its reliability. > But yea... it's less dev effort to just not pre-compile any .eln > files. >=20 > Out of curiosity: if Guix does no JIT compilation at all anyway, why > does it not let Emacs do it the usual way into ~/.emacs.d/eln-cache, > post-installation?=C2=A0 (Aside from the fact it would necessitate > reverting the behavior of =3Dcomp-el-to-eln-filename=3D.) There is only one `comp-el-to-eln-filename' to call, you can't (without a much more ugly hack) have two strategies, and even if you did have them, it'd be even more error-prone. > Actually... (with reservation that I may be wasting your time) I'm > starting to wonder why the filename needs to stay the same for Emacs > to be graftable?=C2=A0 Realistically, if all Emacs packages get upgraded, > and if =3Dcomp-el-to-eln-filename=3D works like upstream, we have some > paths like >=20 > =C2=A0=C2=A0=C2=A0 /gnu/store/emacs-29.4/bin/emacs > =C2=A0=C2=A0=C2=A0 /gnu/store/emacs-magit-OLD/lisp/magit.el > =C2=A0=C2=A0=C2=A0 /gnu/store/emacs-magit-NEW/lisp/magit.el > =C2=A0=C2=A0=C2=A0 .../wherever/magit-OLDHASH.eln > =C2=A0=C2=A0=C2=A0 .../wherever/magit-NEWHASH.eln >=20 > Then we start Emacs, and run its =3Dcomp-el-to-eln-filename=3D on the new > magit.el, it would correctly return .../wherever/magit-NEWHASH.eln. > No need for static filenames to swap out. I haven't linked at the shared objects themselves, but I have strong hunch that they use dynamic linking between the libraries. Since we only update the /gnu/store/emacs-whatever-HERE-WE-HAVE-A-HASH portion and nothing post that, we get file names that don't exist :) If you dig into the patch series, it references the bug it fixes. > [=E2=80=A6] >=20 > No, grafting is about avoiding a world re-compile, right?=C2=A0 So like i= f > package-A was built with an old dependency package-B, it does not get > re-built when package-B is upgraded. But does grafting actually ever > change anything in Emacs?=C2=A0 When you start emacs and it loads package= - > A which in turn loads package-B because there's a =3Drequire=3D statement > for package-B, Emacs will actually load the fresh package-B from > /gnu/store/emacs-package-B, won't it?=C2=A0 So grafting would seem to be > meaningless. For example, GTK is a package deep in the world, that gets grafted a lot and causes Emacs to be grafted as well. > Not sure how to feel if it's like this instead: a compiled function > from package A, containing a call to some function "package-B- > frobnicate", will actually call some bytecode originating from > /gnu/store/emacs-package-A/share/emacs/site-lisp/package-B.elc? > That's... not what happens, right?=C2=A0 OK, probably yes if it was a > defsubst or defmacro. >=20 > So we can have package A using a macro that was defined in an old > version of package-B.=C2=A0 I like that not, but the reason I ask is I'm > trying to find out where it is that the file path would influence the > graft. I haven't looked in the ABI stability of Emacs Lisp packages, but most of them would actually not need a graft. It's Emacs itself that needs the grafting :)=20 > > The way our load paths are set up, it is actually the opposite > > (which still is a bug, just not the one reported).=C2=A0 While `guix > > upgrade` or a command to the similar effect will swap out the .eln > > under the hood, the `.el` and `.elc` files stay stable =E2=80=93 rememb= er > > what I wrote in the previous message about that having caused > > issues with byte compilation? >=20 > I confess I'm puzzled.=C2=A0 Does this just apply to a case like the > following? >=20 > - Emacs itself is upgraded > - A package emacs-magit is NOT upgraded >=20 > so that >=20 > - magit.eln is rebuilt > - magit.el and magit.elc stay the same >=20 > That would make sense.=C2=A0 Do you mean it's a bug because .elc files ar= e > also not portable between different Emacsen? We had issues, where a major upgrade between Emacsen would cause invalid bytecode to be loaded, yes. The issue in Guix is not so much that a package is not upgrade, but Emacs being upgraded while Emacs is running. You'd get a functioning Emacs once you exit the process and start a new one, but that feels very dirty. (Imagine exiting Emacs, because your package manager tells you so, ugh!) > Just checking: is it correct to expect that if you actually upgrade > the emacs-magit package, then its .el, .elc and .eln are all > upgraded? Yes, the .el, .elc, and .eln files are built as *one* package. Any package manager you use with Emacs should provide you with that invariant. Cheers