bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

Hi Leo!

Leo Prikler <leo.prikler@student.tugraz.at> writes:

> Hi Maxim,
>
> Am Dienstag, den 22.12.2020, 13:09 -0500 schrieb Maxim Cournoyer:
>> Hi Leo,
>>
>> Leo Prikler <leo.prikler@student.tugraz.at> writes:
>>
>> > Hello Maxim,
>> >
>> > As someone, who lobbied for the current status quo, I have some
>> > thoughts to share.
>>
>> I'm happy you're commenting :-).
>>
>> > Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer:
>> > > The Emacs packages built with the Emacs built system used to be
>> > > installed in a sub-directory under the share/emacs/guix.d/
>> > > directory,
>> > > but this was changed in commit
>> > > 65a7dd2950ca13a8b942b2836260a2192351b271
>> > > shortly after having accommodated the site-start.el machinery to
>> > > enable
>> > > loading packages from any profile (via the EMACSLOADPATH search
>> > > path
>> > > specification).
>> > Won't this reintroduce <https://bugs.gnu.org/38309> then?
>>
>> This bug was caused by the EMACSLOADPATH environment variable being
>> too
>> long.  This was because the original search path specification was
>> recursively adding any directories under site-lisp/ so that package
>> installed under a guix.d sub-directory could were directly added to
>> the
>> load path.
>>
>> In the current situation, the EMACSLOADPATH contains only the site-
>> lisp/
>> directory of any profile, plus the versioned lisp/ directory of the
>> installed Emacs package.  This patch series doesn't change that, so
>> no,
>> that bug wouldn't be reintroduced.

> Which is also the reason why our startup code changes and people have
> to accommodate to that, right?

Yes. The startup code is now responsible to activate packages found not
directly in the EMACSLOADPATH (share/emacs/site-lisp), but in
sub-directories under share/emacs/site-lisp/guix.

>> > > While this change allowed to expose simply and directly the
>> > > packages
>> > > found in EMACSLOADPATH, it does introduce the risk of file name
>> > > collisions when multiple Emacs packages are joined in the same
>> > > profile,
>> > > especially with Emacs packages increasing in complexity (e.g.,
>> > > using
>> > > more than a single .el file!) and expecting to have both their
>> > > sources
>> > > and resources extracted under their own nested directory rather
>> > > than
>> > > as
>> > > a flat collection (ELPA, MELPA).
>> > > One recent example I stumbled on was attempting to use the
>> > > emacs-yasnippet-snippets package along with emacs-elpy; both
>> > > wanted
>> > > to
>> > > install a 'snippets' directory to share/emacs/site-lisp/snippets,
>> > > collided and resulted in problems that prove difficult to
>> > > understand.
>> > I believe that to be a problem in those packages.  Data should not
>> > be
>> > installed into share/emacs/site-lisp, but share/emacs/etc.  See for
>> > instance also emacs-telega, which – while not quite adhering to the
>> > above – installs its data in share/emacs/{telega-contrib,telega-
>> > data}.
>> >
>> > Regardless of what you intend to do with site-lisp otherwise, data
>> > files should *not*, I repeat *not* be installed there.  I do
>> > believe
>> > standardizing share/emacs/etc is the way to go, however.
>>
>> While I agree that it would be more elegant the way you propose,
>> that's
>> not the way Emacs packages have been standardized.  So going against
>> the
>> sin--8<---------------cut here---------------start------------->8---
--8<---------------cut here---------------start------------->8---
gle "content directory" (c.f. info "(elisp) Packaging Basics")
>> would
>> break many Elisp library assumptions about where there files are,
>> causing more friction (thus work) to adapt them in Guix.  The content
>> directory of an Elisp package can contain any kind of files (code,
>> images, etc.), according to info "(elisp) Multi-file Packages".
>
> I suppose said manual is perhaps slightly outdated.

The doc/package.texi file of the Emacs git repository is actively
maintained, although cited contents above hasn't changed much for the
last 8 years.  In any case if you look into package.el, you'll see that
package contents are fetched as a single archived, and its content
installed to a single directory.

>> A package is either a “simple package” or a “multi-file package”.  A
>> simple package is stored in a package archive as a single Emacs Lisp
>> file, while a multi-file package is stored as a tar file (containing
>> multiple Lisp files, and possibly non-Lisp files such as a manual).
> When was the last time, you've installed a tar to site-lisp?  I also
> feel as though the expected contents are limited very much to README,
> COPYING and the like:
>
> $ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \
>   -not -name '*.el' \
>   -and -not -name '*.elc' \
>   -and -not -name '*.el.gz'
> ~/.guix-profile/share/emacs/27.1/lisp/term/README
> ~/.guix-profile/share/emacs/27.1/lisp/COPYING
> ~/.guix-profile/share/emacs/27.1/lisp/README

The lisp/ directory is for "the standard Lisp files that come with
Emacs" (from info "(elisp)Library Search").  I don't think we can draw
much comparison between these and user-installable packages from ELPA or
MELPA.

Also, just to be clear, the tar is not installed as-is; it is extracted
and its contents are installed in the "content directory".  Citing again
'(elisp) Multi-file Packages':

    Prior to installation, a multi-file package is stored in a package
    archive as a tar file.  The tar file must be named ‘NAME-VERSION.tar’,
    where NAME is the package name and VERSION is the version number.  Its
    contents, once extracted, must all appear in a directory named
    ‘NAME-VERSION’, the “content directory” (*note Packaging Basics::).
    Files may also extract into subdirectories of the content directory.

       One of the files in the content directory must be named
    ‘NAME-pkg.el’.  It must contain a single Lisp form, consisting of a call
    to the function ‘define-package’, described below.  This defines the
    package’s attributes: version, brief description, and requirements.

       For example, if we distribute version 1.3 of the superfrobnicator as
    a multi-file package, the tar file would be ‘superfrobnicator-1.3.tar’.
    Its contents would extract into the directory ‘superfrobnicator-1.3’,
    and one of these would be the file ‘superfrobnicator-pkg.el’.

If you want to have a clearer idea of how packages from ELPA and the
likes are installed, you can have a peek into the 'package.el' file
shipped with Emacs (spoiler: it's basically just extracting the contents
of the package archive to a single directory -- see the `package-unpack'
procedure).

Here's an experiment I've done with a profile containing the whole of
our current emacs-build-system based packages, with this current change,
that checks for collision at the top level of a package installation
directory:

--8<---------------cut here---------------end--------------->8---
$ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-manifest.scm

$  find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2
-mindepth 2 -not -regex '.*\.elc?$' \
          | awk -F '/' '{ if ($9 in packages) \
            {printf("%s directory of %s package collides with that of package %s\n", $9, $8, packages[$9])} \
            else {packages[$9] = $8} }'

doc directory of modus-operandi-theme-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
tools directory of unidecode-0.2-1.5502ada package collides with that of package company-cabal-0.3.0-1.62112a7
snippets directory of minitest-0.8.0-1.1aadb78 package collides with that of package elpy-1.35.0
test directory of realgud-1.5.1 package collides with that of package flycheck-haskell-0.8-2.32ddff8
snippets directory of yasnippet-snippets-0.23 package collides with that of package elpy-1.35.0
snippets directory of haskell-snippets-0.1.0-0.07b0f46 package collides with that of package elpy-1.35.0
snippets directory of rspec-1.11-1.66ea7cc package collides with that of package elpy-1.35.0
doc directory of modus-themes-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
data directory of emojify-1.2 package collides with that of package unidecode-0.2-1.5502ada
test directory of systemd-mode-1.6 package collides with that of package flycheck-haskell-0.8-2.32ddff8
lib directory of slime-2.26.1 package collides with that of package robe-0.8.2
contrib directory of sly-1.0.0-7.68561f1 package collides with that of package slime-2.26.1
lib directory of sly-1.0.0-7.68561f1 package collides with that of package robe-0.8.2
doc directory of modus-vivendi-theme-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
doc directory of evil-1.14.0 package collides with that of package racket-mode-0.0.2-6.5eb31a2
data directory of all-the-icons-4.0.1 package collides with that of package unidecode-0.2-1.5502ada
snippets directory of feature-mode-20190801-1.11ae167 package collides with that of package elpy-1.35.0
--8<---------------cut here---------------end--------------->8---

So 17 Emacs packages in Guix currently conflict, and Guix seems to be
silent about it when building a profile with them via 'guix package -p'
(a bug?).

> Perhaps we should ask Emacs folks how they feel about separating code
> and data, but bugs have already been caused here and elsewhere by
> stuffing them together.  (I believe yasnippet was once already the
> culprit at some point before it got reorganized.)  In the meantime,
> yes, doing so might cause extra work, but
> 1) it only affects a subset of packages, and
> 2) of this subset, we can prioritize those, that do exhibit this
> behaviour.

The above does output does show that's it's a subset of the packages
that are affected.

[...]

>> We have two schemes to accommodate for our Emacs packages:
>>
>> 1. Those installed via their own mean, e.g. make install, using the
>> gnu-build-system for example.  These would still typically install
>> their
>> packages directly under site-lisp, possibly multiple files (that
>> could
>> still collide).
> Why?  Seems inconsistent, does it not?

They can be adapted in time; until then it's easy to accommodate both.

>> 2. Those installed via the emacs-build-system.  With the proposed
>> changes, those now go to site-lisp/guix/.  The 'guix' sub-directory
>> makes it unambiguous that anything found under is to be loaded by
>> package.el; the `package-directory-list' variable can be pointed to
>> it
>> to have the Emacs' package library discover these self-contained
>> packages.
> I think you've lost me here.  Basically, instead of being able to
> `(require 'my-package)` you first need to load my-package through
> package.el and then can `(require 'my-package)`?  I am not sure, what
> the benefit of that would be, if any.

Only if you've specified --no-site-start, which would disable
user-installed package discovery.  This is the drawback of the
trade-off, not the benefit :-).

>> Currently if you use -Q, the Elisp libraries are in the load-path,
>> but
>> not loaded (you need manually do M-x load-library before you can use
>> it), or call M-x guix-emacs-autoload-packages to load their
>> autoloads.
>> For programs, this mean (require 'some-library) works, but M-x
>> some-library-autoloaded-function doesn't.
>>
>> After this change, if you use -Q the new style Emacs packages are not
>> visible at all until you activate them with 'M-x load-library
>> guix-emacs' then 'M-x guix-package-initialize', or (require 'guix-
>> emacs)
>> then (guix-package-initialize), programmatically.

> I do think, that there are legitimate reasons to not require a full
> initialization here, particularly for the use in batch scripts, but I'm
> not quite sure how much work we currently put into making sure that
> everything is available.

That's a fair point, although I'm not concerned about the overhead of
loading compiled autoloads files.

>> I don't see this as a problem.  -Q is simply an alias for
>> "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources"
>> "--no-site-file".
> But not "--no-load-path" "--no-nothing" ;)

One could argue that the spirit of -Q is to give an Emacs as minimal as
possible (that's what the Emacs manual has to say about it, and the
implicit --no-site-lisp has the meaning of "no user installed
packages").

> Being able to load the same libraries as without is vital for debugging
> and scripts.  (Granted, some distros probably break with --no-site-
> lisp, but that's nothing to strive for imo.)

I think you meant --no-site-file here.  There's not much magic; if you
allow it to run, user installed packages are added to the load path and
autoloaded; if you don't you don't then you only get Emacs standard
libraries.  Nothing else happens in the site-start.el file in Guix
proposed here.  Scripting works well enough that our 900'ish packages
can be rebuilt with the change, many of which come with a test suite.

Thanks for having me think harder about if this is really
desirable/needed.  For me the main plus is that we'd be adhering to the
current standards used for Emacs packaging: everything is unpacked into
a single directory owned by that package, so as long as we include any
needed resources in the #:include argument of the emacs-build-system, it
works as upstream intended it, with no need to patch variables or
anything.  I expect this situation to worsen as Emacs packages tend to
get more complicated, and I don't feel strongly enough about it to go
arguing about Emacs packaging standards on the Emacs mailing list :-).

On the minus side it makes the startup slightly more expensive in terms
of pure scripting, and requires the users to be mindful that
site-start.el evaluation is needed for the user installed packages to be
activated.

Further thoughts?

Thanks again,

Maxim