From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id YBY/LNbD5l8EeQAA0tVLHw (envelope-from ) for ; Sat, 26 Dec 2020 05:02:14 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 2O4GKNbD5l9UGQAA1q6Kng (envelope-from ) for ; Sat, 26 Dec 2020 05:02:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 84E10940366 for ; Sat, 26 Dec 2020 05:02:13 +0000 (UTC) Received: from localhost ([::1]:55766 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kt1iU-00042J-Oq for larch@yhetil.org; Sat, 26 Dec 2020 00:02:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42466) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kt1iM-000429-RM for bug-guix@gnu.org; Sat, 26 Dec 2020 00:02:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:47064) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kt1iM-0006Zn-K1 for bug-guix@gnu.org; Sat, 26 Dec 2020 00:02:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kt1iM-0004sb-92 for bug-guix@gnu.org; Sat, 26 Dec 2020 00:02:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 26 Dec 2020 05:02:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45316 X-GNU-PR-Package: guix X-GNU-PR-Keywords: patch To: Leo Prikler Received: via spool by 45316-submit@debbugs.gnu.org id=B45316.160895890018726 (code B ref 45316); Sat, 26 Dec 2020 05:02:02 +0000 Received: (at 45316) by debbugs.gnu.org; 26 Dec 2020 05:01:40 +0000 Received: from localhost ([127.0.0.1]:58610 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kt1hz-0004rx-8p for submit@debbugs.gnu.org; Sat, 26 Dec 2020 00:01:40 -0500 Received: from mail-qt1-f174.google.com ([209.85.160.174]:46124) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kt1hx-0004rl-CF for 45316@debbugs.gnu.org; Sat, 26 Dec 2020 00:01:38 -0500 Received: by mail-qt1-f174.google.com with SMTP id h19so3826214qtq.13 for <45316@debbugs.gnu.org>; Fri, 25 Dec 2020 21:01:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=W1Pd6zmZLOK7SbF2HUibkfkZaHOGI0bvWvOYngYSb24=; b=G+JJWbLwhxI5oNC5wt0XVnBCn7XZ0RoEi1Z4Gbgat8vmxiaooVAIpKNVdjj+/hOdok IFCi99iloxA8IqgDhYrhhlstBsc8wPl0riDOP+9qqsxU7/II3UzQbIUYqtKBx6w7rSCK FEcUnBctC7WwWncSXDguS9jT2AhVS3vzVtM686JNQSSkibkCUYpElovHeQCRk9ivy+/z 4AtsWWq5wYKf46mf9UfDR3m8+d13nFfZ1Okazo43kfngMHozFK3tguYos5mHhkeQcXP/ byb+WbROqGiac9FqsxKmqoG9SxkbrNpa+Oq++3XDVzr9FZYxO6ZbeCRjw0RrKc4F4UzL K1wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=W1Pd6zmZLOK7SbF2HUibkfkZaHOGI0bvWvOYngYSb24=; b=iU6EB2HtxG819wI7lfNmXeJe5xXz9uhm+0KG2ltw2fYMNzyMmTgO9zPljym1RiuVri QaWvBPDo0sISZ4mIT5vVoSFf8MF/e55gmYi5RkReLz8+eNI0XagUIGhFysDbWNJZi7vc rdFdgIflJzhiDhuwfsK/2sIitdgTahdwx5/J5x4JnNfGMbMQ7eEYnxfMhoLYQJTzLGEP Ixd08VxJuMkIAKRIBhGo1DEOb1qQe2P7N6gOf5R7aE4J7jrjekbjjxagydWkJ980ej5P LA78vFmobz3h44yRxkPUi7amidmtCZ70ZXkuguc4aQcpgZdG/bu/qRMCmN1LjRxkhJ3C sCYw== X-Gm-Message-State: AOAM532yMGvpzLKkm648Y+l1mNeWSC5qU7FHamti1BCkAkFI8uUCFnEa f4aPBhnSoZMEURgLJomDG1Wf768r47w= X-Google-Smtp-Source: ABdhPJw3wQunBYfgwMAIn94G4jpLDwRiJKieLwGkdL2PcxPBYHmG7LAYR0Gvg8OAelcuF8rAG0V3yQ== X-Received: by 2002:ac8:128c:: with SMTP id y12mr36023843qti.127.1608958891217; Fri, 25 Dec 2020 21:01:31 -0800 (PST) Received: from hurd (dsl-148-101.b2b2c.ca. [66.158.148.101]) by smtp.gmail.com with ESMTPSA id 5sm19046580qtp.55.2020.12.25.21.01.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Dec 2020 21:01:30 -0800 (PST) From: Maxim Cournoyer References: <87ft3ysen7.fsf@gmail.com> <878dbbdf4883d509865a1049c11fbe9ae792cace.camel@student.tugraz.at> <875z4tyapq.fsf@gmail.com> <4f40e3fb175201b21e40718022e93eb27efab3fb.camel@student.tugraz.at> Date: Sat, 26 Dec 2020 00:01:29 -0500 In-Reply-To: <4f40e3fb175201b21e40718022e93eb27efab3fb.camel@student.tugraz.at> (Leo Prikler's message of "Tue, 22 Dec 2020 20:10:40 +0100") Message-ID: <87pn2xw47q.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 45316@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.22 Authentication-Results: aspmx1.migadu.com; dkim=fail (headers rsa verify failed) header.d=gmail.com header.s=20161025 header.b=G+JJWbLw; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: 84E10940366 X-Spam-Score: -1.22 X-Migadu-Scanner: scn0.migadu.com X-TUID: j5k/znUOQ1uQ Hi Leo! Leo Prikler writes: > Hi Maxim, > > Am Dienstag, den 22.12.2020, 13:09 -0500 schrieb Maxim Cournoyer: >> Hi Leo, >> >> Leo Prikler writes: >> >> > Hello Maxim, >> > >> > As someone, who lobbied for the current status quo, I have some >> > thoughts to share. >> >> I'm happy you're commenting :-). >> >> > Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer: >> > > The Emacs packages built with the Emacs built system used to be >> > > installed in a sub-directory under the share/emacs/guix.d/ >> > > directory, >> > > but this was changed in commit >> > > 65a7dd2950ca13a8b942b2836260a2192351b271 >> > > shortly after having accommodated the site-start.el machinery to >> > > enable >> > > loading packages from any profile (via the EMACSLOADPATH search >> > > path >> > > specification). >> > Won't this reintroduce then? >> >> This bug was caused by the EMACSLOADPATH environment variable being >> too >> long. This was because the original search path specification was >> recursively adding any directories under site-lisp/ so that package >> installed under a guix.d sub-directory could were directly added to >> the >> load path. >> >> In the current situation, the EMACSLOADPATH contains only the site- >> lisp/ >> directory of any profile, plus the versioned lisp/ directory of the >> installed Emacs package. This patch series doesn't change that, so >> no, >> that bug wouldn't be reintroduced. > Which is also the reason why our startup code changes and people have > to accommodate to that, right? Yes. The startup code is now responsible to activate packages found not directly in the EMACSLOADPATH (share/emacs/site-lisp), but in sub-directories under share/emacs/site-lisp/guix. >> > > While this change allowed to expose simply and directly the >> > > packages >> > > found in EMACSLOADPATH, it does introduce the risk of file name >> > > collisions when multiple Emacs packages are joined in the same >> > > profile, >> > > especially with Emacs packages increasing in complexity (e.g., >> > > using >> > > more than a single .el file!) and expecting to have both their >> > > sources >> > > and resources extracted under their own nested directory rather >> > > than >> > > as >> > > a flat collection (ELPA, MELPA). >> > > One recent example I stumbled on was attempting to use the >> > > emacs-yasnippet-snippets package along with emacs-elpy; both >> > > wanted >> > > to >> > > install a 'snippets' directory to share/emacs/site-lisp/snippets, >> > > collided and resulted in problems that prove difficult to >> > > understand. >> > I believe that to be a problem in those packages. Data should not >> > be >> > installed into share/emacs/site-lisp, but share/emacs/etc. See for >> > instance also emacs-telega, which =E2=80=93 while not quite adhering t= o the >> > above =E2=80=93 installs its data in share/emacs/{telega-contrib,teleg= a- >> > data}. >> > >> > Regardless of what you intend to do with site-lisp otherwise, data >> > files should *not*, I repeat *not* be installed there. I do >> > believe >> > standardizing share/emacs/etc is the way to go, however. >> >> While I agree that it would be more elegant the way you propose, >> that's >> not the way Emacs packages have been standardized. So going against >> the >> sin--8<---------------cut here---------------start------------->8--- --8<---------------cut here---------------start------------->8--- gle "content directory" (c.f. info "(elisp) Packaging Basics") >> would >> break many Elisp library assumptions about where there files are, >> causing more friction (thus work) to adapt them in Guix. The content >> directory of an Elisp package can contain any kind of files (code, >> images, etc.), according to info "(elisp) Multi-file Packages". > > I suppose said manual is perhaps slightly outdated. The doc/package.texi file of the Emacs git repository is actively maintained, although cited contents above hasn't changed much for the last 8 years. In any case if you look into package.el, you'll see that package contents are fetched as a single archived, and its content installed to a single directory. >> A package is either a =E2=80=9Csimple package=E2=80=9D or a =E2=80=9Cmul= ti-file package=E2=80=9D. A >> simple package is stored in a package archive as a single Emacs Lisp >> file, while a multi-file package is stored as a tar file (containing >> multiple Lisp files, and possibly non-Lisp files such as a manual). > When was the last time, you've installed a tar to site-lisp? I also > feel as though the expected contents are limited very much to README, > COPYING and the like: > > $ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \ > -not -name '*.el' \ > -and -not -name '*.elc' \ > -and -not -name '*.el.gz' > ~/.guix-profile/share/emacs/27.1/lisp/term/README > ~/.guix-profile/share/emacs/27.1/lisp/COPYING > ~/.guix-profile/share/emacs/27.1/lisp/README The lisp/ directory is for "the standard Lisp files that come with Emacs" (from info "(elisp)Library Search"). I don't think we can draw much comparison between these and user-installable packages from ELPA or MELPA. Also, just to be clear, the tar is not installed as-is; it is extracted and its contents are installed in the "content directory". Citing again '(elisp) Multi-file Packages': Prior to installation, a multi-file package is stored in a package archive as a tar file. The tar file must be named =E2=80=98NAME-VERSIO= N.tar=E2=80=99, where NAME is the package name and VERSION is the version number. Its contents, once extracted, must all appear in a directory named =E2=80=98NAME-VERSION=E2=80=99, the =E2=80=9Ccontent directory=E2=80=9D= (*note Packaging Basics::). Files may also extract into subdirectories of the content directory. One of the files in the content directory must be named =E2=80=98NAME-pkg.el=E2=80=99. It must contain a single Lisp form, con= sisting of a call to the function =E2=80=98define-package=E2=80=99, described below. Thi= s defines the package=E2=80=99s attributes: version, brief description, and requireme= nts. For example, if we distribute version 1.3 of the superfrobnicator as a multi-file package, the tar file would be =E2=80=98superfrobnicator-1= .3.tar=E2=80=99. Its contents would extract into the directory =E2=80=98superfrobnicator= -1.3=E2=80=99, and one of these would be the file =E2=80=98superfrobnicator-pkg.el=E2= =80=99. If you want to have a clearer idea of how packages from ELPA and the likes are installed, you can have a peek into the 'package.el' file shipped with Emacs (spoiler: it's basically just extracting the contents of the package archive to a single directory -- see the `package-unpack' procedure). Here's an experiment I've done with a profile containing the whole of our current emacs-build-system based packages, with this current change, that checks for collision at the top level of a package installation directory: --8<---------------cut here---------------end--------------->8--- $ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-manifest= .scm $ find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2 -mindepth 2 -not -regex '.*\.elc?$' \ | awk -F '/' '{ if ($9 in packages) \ {printf("%s directory of %s package collides with that of packa= ge %s\n", $9, $8, packages[$9])} \ else {packages[$9] =3D $8} }' doc directory of modus-operandi-theme-1.0.2 package collides with that of p= ackage racket-mode-0.0.2-6.5eb31a2 tools directory of unidecode-0.2-1.5502ada package collides with that of pa= ckage company-cabal-0.3.0-1.62112a7 snippets directory of minitest-0.8.0-1.1aadb78 package collides with that o= f package elpy-1.35.0 test directory of realgud-1.5.1 package collides with that of package flych= eck-haskell-0.8-2.32ddff8 snippets directory of yasnippet-snippets-0.23 package collides with that of= package elpy-1.35.0 snippets directory of haskell-snippets-0.1.0-0.07b0f46 package collides wit= h that of package elpy-1.35.0 snippets directory of rspec-1.11-1.66ea7cc package collides with that of pa= ckage elpy-1.35.0 doc directory of modus-themes-1.0.2 package collides with that of package r= acket-mode-0.0.2-6.5eb31a2 data directory of emojify-1.2 package collides with that of package unideco= de-0.2-1.5502ada test directory of systemd-mode-1.6 package collides with that of package fl= ycheck-haskell-0.8-2.32ddff8 lib directory of slime-2.26.1 package collides with that of package robe-0.= 8.2 contrib directory of sly-1.0.0-7.68561f1 package collides with that of pack= age slime-2.26.1 lib directory of sly-1.0.0-7.68561f1 package collides with that of package = robe-0.8.2 doc directory of modus-vivendi-theme-1.0.2 package collides with that of pa= ckage racket-mode-0.0.2-6.5eb31a2 doc directory of evil-1.14.0 package collides with that of package racket-m= ode-0.0.2-6.5eb31a2 data directory of all-the-icons-4.0.1 package collides with that of package= unidecode-0.2-1.5502ada snippets directory of feature-mode-20190801-1.11ae167 package collides with= that of package elpy-1.35.0 --8<---------------cut here---------------end--------------->8--- So 17 Emacs packages in Guix currently conflict, and Guix seems to be silent about it when building a profile with them via 'guix package -p' (a bug?). > Perhaps we should ask Emacs folks how they feel about separating code > and data, but bugs have already been caused here and elsewhere by > stuffing them together. (I believe yasnippet was once already the > culprit at some point before it got reorganized.) In the meantime, > yes, doing so might cause extra work, but > 1) it only affects a subset of packages, and > 2) of this subset, we can prioritize those, that do exhibit this > behaviour. The above does output does show that's it's a subset of the packages that are affected. [...] >> We have two schemes to accommodate for our Emacs packages: >> >> 1. Those installed via their own mean, e.g. make install, using the >> gnu-build-system for example. These would still typically install >> their >> packages directly under site-lisp, possibly multiple files (that >> could >> still collide). > Why? Seems inconsistent, does it not? They can be adapted in time; until then it's easy to accommodate both. >> 2. Those installed via the emacs-build-system. With the proposed >> changes, those now go to site-lisp/guix/. The 'guix' sub-directory >> makes it unambiguous that anything found under is to be loaded by >> package.el; the `package-directory-list' variable can be pointed to >> it >> to have the Emacs' package library discover these self-contained >> packages. > I think you've lost me here. Basically, instead of being able to > `(require 'my-package)` you first need to load my-package through > package.el and then can `(require 'my-package)`? I am not sure, what > the benefit of that would be, if any. Only if you've specified --no-site-start, which would disable user-installed package discovery. This is the drawback of the trade-off, not the benefit :-). >> Currently if you use -Q, the Elisp libraries are in the load-path, >> but >> not loaded (you need manually do M-x load-library before you can use >> it), or call M-x guix-emacs-autoload-packages to load their >> autoloads. >> For programs, this mean (require 'some-library) works, but M-x >> some-library-autoloaded-function doesn't. >> >> After this change, if you use -Q the new style Emacs packages are not >> visible at all until you activate them with 'M-x load-library >> guix-emacs' then 'M-x guix-package-initialize', or (require 'guix- >> emacs) >> then (guix-package-initialize), programmatically. > I do think, that there are legitimate reasons to not require a full > initialization here, particularly for the use in batch scripts, but I'm > not quite sure how much work we currently put into making sure that > everything is available. That's a fair point, although I'm not concerned about the overhead of loading compiled autoloads files. >> I don't see this as a problem. -Q is simply an alias for >> "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources" >> "--no-site-file". > But not "--no-load-path" "--no-nothing" ;) One could argue that the spirit of -Q is to give an Emacs as minimal as possible (that's what the Emacs manual has to say about it, and the implicit --no-site-lisp has the meaning of "no user installed packages"). > Being able to load the same libraries as without is vital for debugging > and scripts. (Granted, some distros probably break with --no-site- > lisp, but that's nothing to strive for imo.) I think you meant --no-site-file here. There's not much magic; if you allow it to run, user installed packages are added to the load path and autoloaded; if you don't you don't then you only get Emacs standard libraries. Nothing else happens in the site-start.el file in Guix proposed here. Scripting works well enough that our 900'ish packages can be rebuilt with the change, many of which come with a test suite. Thanks for having me think harder about if this is really desirable/needed. For me the main plus is that we'd be adhering to the current standards used for Emacs packaging: everything is unpacked into a single directory owned by that package, so as long as we include any needed resources in the #:include argument of the emacs-build-system, it works as upstream intended it, with no need to patch variables or anything. I expect this situation to worsen as Emacs packages tend to get more complicated, and I don't feel strongly enough about it to go arguing about Emacs packaging standards on the Emacs mailing list :-). On the minus side it makes the startup slightly more expensive in terms of pure scripting, and requires the users to be mindful that site-start.el evaluation is needed for the user installed packages to be activated. Further thoughts? Thanks again, Maxim