bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Leo Prikler <leo.prikler@student.tugraz.at>]

Hi Maxim,

Am Samstag, den 26.12.2020, 00:01 -0500 schrieb Maxim Cournoyer:
> > > > > While this change allowed to expose simply and directly the
> > > > > packages
> > > > > found in EMACSLOADPATH, it does introduce the risk of file
> > > > > name
> > > > > collisions when multiple Emacs packages are joined in the
> > > > > same
> > > > > profile,
> > > > > especially with Emacs packages increasing in complexity
> > > > > (e.g.,
> > > > > using
> > > > > more than a single .el file!) and expecting to have both
> > > > > their
> > > > > sources
> > > > > and resources extracted under their own nested directory
> > > > > rather
> > > > > than
> > > > > as
> > > > > a flat collection (ELPA, MELPA).
> > > > > One recent example I stumbled on was attempting to use the
> > > > > emacs-yasnippet-snippets package along with emacs-elpy; both
> > > > > wanted
> > > > > to
> > > > > install a 'snippets' directory to share/emacs/site-
> > > > > lisp/snippets,
> > > > > collided and resulted in problems that prove difficult to
> > > > > understand.
> > > > I believe that to be a problem in those packages.  Data should
> > > > not
> > > > be
> > > > installed into share/emacs/site-lisp, but share/emacs/etc.  See
> > > > for
> > > > instance also emacs-telega, which – while not quite adhering to
> > > > the
> > > > above – installs its data in share/emacs/{telega-
> > > > contrib,telega-
> > > > data}.
> > > > 
> > > > Regardless of what you intend to do with site-lisp otherwise,
> > > > data
> > > > files should *not*, I repeat *not* be installed there.  I do
> > > > believe
> > > > standardizing share/emacs/etc is the way to go, however.
> > > 
> > > While I agree that it would be more elegant the way you propose,
> > > that's
> > > not the way Emacs packages have been standardized.  So going
> > > against
> > > the
> > > sin--8<---------------cut here---------------start-------------
> > > >8---
> --8<---------------cut here---------------start------------->8---
> gle "content directory" (c.f. info "(elisp) Packaging Basics")
> > > would
> > > break many Elisp library assumptions about where there files are,
> > > causing more friction (thus work) to adapt them in Guix.  The
> > > content
> > > directory of an Elisp package can contain any kind of files
> > > (code,
> > > images, etc.), according to info "(elisp) Multi-file Packages".
> > 
> > I suppose said manual is perhaps slightly outdated.
> 
> The doc/package.texi file of the Emacs git repository is actively
> maintained, although cited contents above hasn't changed much for the
> last 8 years.  In any case if you look into package.el, you'll see
> that
> package contents are fetched as a single archived, and its content
> installed to a single directory.
> 
> > > A package is either a “simple package” or a “multi-file
> > > package”.  A
> > > simple package is stored in a package archive as a single Emacs
> > > Lisp
> > > file, while a multi-file package is stored as a tar file
> > > (containing
> > > multiple Lisp files, and possibly non-Lisp files such as a
> > > manual).
> > When was the last time, you've installed a tar to site-lisp?  I
> > also
> > feel as though the expected contents are limited very much to
> > README,
> > COPYING and the like:
> > 
> > $ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \
> >   -not -name '*.el' \
> >   -and -not -name '*.elc' \
> >   -and -not -name '*.el.gz'
> > ~/.guix-profile/share/emacs/27.1/lisp/term/README
> > ~/.guix-profile/share/emacs/27.1/lisp/COPYING
> > ~/.guix-profile/share/emacs/27.1/lisp/README
> 
> The lisp/ directory is for "the standard Lisp files that come with
> Emacs" (from info "(elisp)Library Search").  I don't think we can
> draw
> much comparison between these and user-installable packages from ELPA
> or
> MELPA.
Fair enough, perhaps I read too much into that.

> Also, just to be clear, the tar is not installed as-is; it is
> extracted
> and its contents are installed in the "content directory".  Citing
> again
> '(elisp) Multi-file Packages':
> 
>     Prior to installation, a multi-file package is stored in a
> package
>     archive as a tar file.  The tar file must be named ‘NAME-
> VERSION.tar’,
>     where NAME is the package name and VERSION is the version
> number.  Its
>     contents, once extracted, must all appear in a directory named
>     ‘NAME-VERSION’, the “content directory” (*note Packaging
> Basics::).
>     Files may also extract into subdirectories of the content
> directory.
> 
>        One of the files in the content directory must be named
>     ‘NAME-pkg.el’.  It must contain a single Lisp form, consisting of
> a call
>     to the function ‘define-package’, described below.  This defines
> the
>     package’s attributes: version, brief description, and
> requirements.
> 
>        For example, if we distribute version 1.3 of the
> superfrobnicator as
>     a multi-file package, the tar file would be ‘superfrobnicator-
> 1.3.tar’.
>     Its contents would extract into the directory ‘superfrobnicator-
> 1.3’,
>     and one of these would be the file ‘superfrobnicator-pkg.el’.
> 
> If you want to have a clearer idea of how packages from ELPA and the
> likes are installed, you can have a peek into the 'package.el' file
> shipped with Emacs (spoiler: it's basically just extracting the
> contents
> of the package archive to a single directory -- see the `package-
> unpack'
> procedure).
If we follow that to the letter, should we not install those packages
directly to site-lisp/$PACKAGE-$VERSION?  Why the guix directory?

Also, if we do go that route, I think we should install a subdirs.el
through a profile hook, that adds all these package directories to the
load path.  See also '(elisp) Startup Summary'.  Alternatively, we can
let subdirs.el defer to guix-emacs.el, but that'd be very cheeky. 
Either way, we should definitely make -Q work.

> Here's an experiment I've done with a profile containing the whole of
> our current emacs-build-system based packages, with this current
> change,
> that checks for collision at the top level of a package installation
> directory:
> 
> --8<---------------cut here---------------end--------------->8---
> $ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-
> manifest.scm
> 
> $  find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2
> -mindepth 2 -not -regex '.*\.elc?$' \
>           | awk -F '/' '{ if ($9 in packages) \
>             {printf("%s directory of %s package collides with that of
> package %s\n", $9, $8, packages[$9])} \
>             else {packages[$9] = $8} }'
> 
> doc directory of modus-operandi-theme-1.0.2 package collides with
> that of package racket-mode-0.0.2-6.5eb31a2
> tools directory of unidecode-0.2-1.5502ada package collides with that
> of package company-cabal-0.3.0-1.62112a7
> snippets directory of minitest-0.8.0-1.1aadb78 package collides with
> that of package elpy-1.35.0
> test directory of realgud-1.5.1 package collides with that of package
> flycheck-haskell-0.8-2.32ddff8
> snippets directory of yasnippet-snippets-0.23 package collides with
> that of package elpy-1.35.0
> snippets directory of haskell-snippets-0.1.0-0.07b0f46 package
> collides with that of package elpy-1.35.0
> snippets directory of rspec-1.11-1.66ea7cc package collides with that
> of package elpy-1.35.0
> doc directory of modus-themes-1.0.2 package collides with that of
> package racket-mode-0.0.2-6.5eb31a2
> data directory of emojify-1.2 package collides with that of package
> unidecode-0.2-1.5502ada
> test directory of systemd-mode-1.6 package collides with that of
> package flycheck-haskell-0.8-2.32ddff8
> lib directory of slime-2.26.1 package collides with that of package
> robe-0.8.2
> contrib directory of sly-1.0.0-7.68561f1 package collides with that
> of package slime-2.26.1
> lib directory of sly-1.0.0-7.68561f1 package collides with that of
> package robe-0.8.2
> doc directory of modus-vivendi-theme-1.0.2 package collides with that
> of package racket-mode-0.0.2-6.5eb31a2
> doc directory of evil-1.14.0 package collides with that of package
> racket-mode-0.0.2-6.5eb31a2
> data directory of all-the-icons-4.0.1 package collides with that of
> package unidecode-0.2-1.5502ada
> snippets directory of feature-mode-20190801-1.11ae167 package
> collides with that of package elpy-1.35.0
> --8<---------------cut here---------------end--------------->8---
> 
> So 17 Emacs packages in Guix currently conflict, and Guix seems to be
> silent about it when building a profile with them via 'guix package
> -p'
> (a bug?).
It's a bug in those packages; not in Guix.  The whole idea of Guix is,
that such directories are merged.  Now collisions within files in those
directories are a different topic – only one can be chosen there and
IIRC there should also be a way to make them errors.

> > Perhaps we should ask Emacs folks how they feel about separating
> > code
> > and data, but bugs have already been caused here and elsewhere by
> > stuffing them together.  (I believe yasnippet was once already the
> > culprit at some point before it got reorganized.)  In the meantime,
> > yes, doing so might cause extra work, but
> > 1) it only affects a subset of packages, and
> > 2) of this subset, we can prioritize those, that do exhibit this
> > behaviour.
> 
> The above does output does show that's it's a subset of the packages
> that are affected.
> 
> [...]
> 
> > > We have two schemes to accommodate for our Emacs packages:
> > > 
> > > 1. Those installed via their own mean, e.g. make install, using
> > > the
> > > gnu-build-system for example.  These would still typically
> > > install
> > > their
> > > packages directly under site-lisp, possibly multiple files (that
> > > could
> > > still collide).
> > Why?  Seems inconsistent, does it not?
> 
> They can be adapted in time; until then it's easy to accommodate
> both.
I think we should aim for consistency here, but let's adapt them in
follow-up commits.

> > > 2. Those installed via the emacs-build-system.  With the proposed
> > > changes, those now go to site-lisp/guix/.  The 'guix' sub-
> > > directory
> > > makes it unambiguous that anything found under is to be loaded by
> > > package.el; the `package-directory-list' variable can be pointed
> > > to
> > > it
> > > to have the Emacs' package library discover these self-contained
> > > packages.
> > I think you've lost me here.  Basically, instead of being able to
> > `(require 'my-package)` you first need to load my-package through
> > package.el and then can `(require 'my-package)`?  I am not sure,
> > what
> > the benefit of that would be, if any.
> 
> Only if you've specified --no-site-start, which would disable
> user-installed package discovery.  This is the drawback of the
> trade-off, not the benefit :-).
It wasn't worded like one.  The word unambiguous typically has a
positive connotation in my mind.  And package discovery should function
regardless of site-start.

> > > Currently if you use -Q, the Elisp libraries are in the load-
> > > path,
> > > but
> > > not loaded (you need manually do M-x load-library before you can
> > > use
> > > it), or call M-x guix-emacs-autoload-packages to load their
> > > autoloads.
> > > For programs, this mean (require 'some-library) works, but M-x
> > > some-library-autoloaded-function doesn't.
> > > 
> > > After this change, if you use -Q the new style Emacs packages are
> > > not
> > > visible at all until you activate them with 'M-x load-library
> > > guix-emacs' then 'M-x guix-package-initialize', or (require
> > > 'guix-
> > > emacs)
> > > then (guix-package-initialize), programmatically.
> > I do think, that there are legitimate reasons to not require a full
> > initialization here, particularly for the use in batch scripts, but
> > I'm
> > not quite sure how much work we currently put into making sure that
> > everything is available.
> 
> That's a fair point, although I'm not concerned about the overhead of
> loading compiled autoloads files.
> 
> > > I don't see this as a problem.  -Q is simply an alias for
> > > "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-
> > > resources"
> > > "--no-site-file".
> > But not "--no-load-path" "--no-nothing" ;)
> 
> One could argue that the spirit of -Q is to give an Emacs as minimal
> as
> possible (that's what the Emacs manual has to say about it, and the
> implicit --no-site-lisp has the meaning of "no user installed
> packages").
Well then the spirit of -Q is lost.

> > Being able to load the same libraries as without is vital for
> > debugging
> > and scripts.  (Granted, some distros probably break with --no-site-
> > lisp, but that's nothing to strive for imo.)
> 
> [...]
> 
> Thanks for having me think harder about if this is really
> desirable/needed.  For me the main plus is that we'd be adhering to
> the
> current standards used for Emacs packaging: everything is unpacked
> into
> a single directory owned by that package, so as long as we include
> any
> needed resources in the #:include argument of the emacs-build-system, 
> it
> works as upstream intended it, with no need to patch variables or
> anything.  I expect this situation to worsen as Emacs packages tend
> to
> get more complicated, and I don't feel strongly enough about it to go
> arguing about Emacs packaging standards on the Emacs mailing list :-
> ).
I agree, that adhering to the standards sounds nice, but this patch
makes it seem as if we're still cooking our own soup.  I believe we'd
need to adhere to them harder than that in order for it to be an
upgrade of the status quo.

> On the minus side it makes the startup slightly more expensive in
> terms
> of pure scripting, and requires the users to be mindful that
> site-start.el evaluation is needed for the user installed packages to
> be
> activated.
> 
> Further thoughts?
I think I've already laid that out above, but we really ought to have
site-lisp/$PACKAGE-$VERSION and a working subdirs.el.  If we do it like
that, I don't think the multi-directory layout should cause us too many
troubles.

As far as the build system is concerned, we would probably need to set
up an environment similar to what will be found in the end.
Imagine this:

- environment-variables
- $PACKAGE-$VERSION/
- $PACKAGE-$VERSION-inputs/
  - subdirs.el
  - $INPUT1/
  - $INPUT2/
  - ...

Regards,
Leo