bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Leo Prikler <leo.prikler@student.tugraz.at>]

Hi Maxim,

Am Dienstag, den 22.12.2020, 13:09 -0500 schrieb Maxim Cournoyer:
> Hi Leo,
> 
> Leo Prikler <leo.prikler@student.tugraz.at> writes:
> 
> > Hello Maxim,
> > 
> > As someone, who lobbied for the current status quo, I have some
> > thoughts to share.
> 
> I'm happy you're commenting :-).
> 
> > Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer:
> > > The Emacs packages built with the Emacs built system used to be
> > > installed in a sub-directory under the share/emacs/guix.d/
> > > directory,
> > > but this was changed in commit
> > > 65a7dd2950ca13a8b942b2836260a2192351b271
> > > shortly after having accommodated the site-start.el machinery to
> > > enable
> > > loading packages from any profile (via the EMACSLOADPATH search
> > > path
> > > specification).
> > Won't this reintroduce <https://bugs.gnu.org/38309> then?
> 
> This bug was caused by the EMACSLOADPATH environment variable being
> too
> long.  This was because the original search path specification was
> recursively adding any directories under site-lisp/ so that package
> installed under a guix.d sub-directory could were directly added to
> the
> load path.
> 
> In the current situation, the EMACSLOADPATH contains only the site-
> lisp/
> directory of any profile, plus the versioned lisp/ directory of the
> installed Emacs package.  This patch series doesn't change that, so
> no,
> that bug wouldn't be reintroduced.
Which is also the reason why our startup code changes and people have
to accommodate to that, right?

> > > While this change allowed to expose simply and directly the
> > > packages
> > > found in EMACSLOADPATH, it does introduce the risk of file name
> > > collisions when multiple Emacs packages are joined in the same
> > > profile,
> > > especially with Emacs packages increasing in complexity (e.g.,
> > > using
> > > more than a single .el file!) and expecting to have both their
> > > sources
> > > and resources extracted under their own nested directory rather
> > > than
> > > as
> > > a flat collection (ELPA, MELPA).
> > > One recent example I stumbled on was attempting to use the
> > > emacs-yasnippet-snippets package along with emacs-elpy; both
> > > wanted
> > > to
> > > install a 'snippets' directory to share/emacs/site-lisp/snippets,
> > > collided and resulted in problems that prove difficult to
> > > understand.
> > I believe that to be a problem in those packages.  Data should not
> > be
> > installed into share/emacs/site-lisp, but share/emacs/etc.  See for
> > instance also emacs-telega, which – while not quite adhering to the
> > above – installs its data in share/emacs/{telega-contrib,telega-
> > data}.
> > 
> > Regardless of what you intend to do with site-lisp otherwise, data
> > files should *not*, I repeat *not* be installed there.  I do
> > believe
> > standardizing share/emacs/etc is the way to go, however.
> 
> While I agree that it would be more elegant the way you propose,
> that's
> not the way Emacs packages have been standardized.  So going against
> the
> single "content directory" (c.f. info "(elisp) Packaging Basics")
> would
> break many Elisp library assumptions about where there files are,
> causing more friction (thus work) to adapt them in Guix.  The content
> directory of an Elisp package can contain any kind of files (code,
> images, etc.), according to info "(elisp) Multi-file Packages".

I suppose said manual is perhaps slightly outdated.
> A package is either a “simple package” or a “multi-file package”.  A
> simple package is stored in a package archive as a single Emacs Lisp
> file, while a multi-file package is stored as a tar file (containing
> multiple Lisp files, and possibly non-Lisp files such as a manual).
When was the last time, you've installed a tar to site-lisp?  I also
feel as though the expected contents are limited very much to README,
COPYING and the like:

--8<---------------cut here--------------begin-------------->8---
$ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \
  -not -name '*.el' \
  -and -not -name '*.elc' \
  -and -not -name '*.el.gz'
~/.guix-profile/share/emacs/27.1/lisp/term/README
~/.guix-profile/share/emacs/27.1/lisp/COPYING
~/.guix-profile/share/emacs/27.1/lisp/README
--8<---------------cut here---------------end--------------->8---

Perhaps we should ask Emacs folks how they feel about separating code
and data, but bugs have already been caused here and elsewhere by
stuffing them together.  (I believe yasnippet was once already the
culprit at some point before it got reorganized.)  In the meantime,
yes, doing so might cause extra work, but
1) it only affects a subset of packages, and
2) of this subset, we can prioritize those, that do exhibit this
behaviour.

> > > This is what motivated this patch series, where the site-start.el
> > > auxiliary code used for package discovery is extended to support
> > > packages installed in their own directory under a
> > > 'share/emacs/guix'
> > > installation prefix, via Emacs' own package library!
> > > 
> > > The emacs-build-system is updated for this new installation
> > > prefix,
> > > as
> > > well as existing packages and documentation.  Parting with a
> > > directly
> > > usable EMACSLOADPATH means that site-start.el *must* run for
> > > packages
> > > to
> > > appear in the load-path; that means for running a test suite, the
> > > -Q
> > > or
> > > --quick Emacs options cannot be used, since it implies --no-site-
> > > file.
> > Having no usable EMACSLOADPATH sounds like a dealbreaker to
> > me.  Could
> > one be restored by using direct subdirectories to share/emacs/site-
> > lisp
> > or would that merely be a cosmetic change?
> 
> We have two schemes to accommodate for our Emacs packages:
> 
> 1. Those installed via their own mean, e.g. make install, using the
> gnu-build-system for example.  These would still typically install
> their
> packages directly under site-lisp, possibly multiple files (that
> could
> still collide).
Why?  Seems inconsistent, does it not?

> 2. Those installed via the emacs-build-system.  With the proposed
> changes, those now go to site-lisp/guix/.  The 'guix' sub-directory
> makes it unambiguous that anything found under is to be loaded by
> package.el; the `package-directory-list' variable can be pointed to
> it
> to have the Emacs' package library discover these self-contained
> packages.
I think you've lost me here.  Basically, instead of being able to
`(require 'my-package)` you first need to load my-package through
package.el and then can `(require 'my-package)`?  I am not sure, what
the benefit of that would be, if any. 

> Currently if you use -Q, the Elisp libraries are in the load-path,
> but
> not loaded (you need manually do M-x load-library before you can use
> it), or call M-x guix-emacs-autoload-packages to load their
> autoloads.
> For programs, this mean (require 'some-library) works, but M-x
> some-library-autoloaded-function doesn't.
> 
> After this change, if you use -Q the new style Emacs packages are not
> visible at all until you activate them with 'M-x load-library
> guix-emacs' then 'M-x guix-package-initialize', or (require 'guix-
> emacs)
> then (guix-package-initialize), programmatically.
I do think, that there are legitimate reasons to not require a full
initialization here, particularly for the use in batch scripts, but I'm
not quite sure how much work we currently put into making sure that
everything is available.

> I don't see this as a problem.  -Q is simply an alias for
> "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources"
> "--no-site-file".
But not "--no-load-path" "--no-nothing" ;)
Being able to load the same libraries as without is vital for debugging
and scripts.  (Granted, some distros probably break with --no-site-
lisp, but that's nothing to strive for imo.)

> [...]
> When put in perspective, that's 3 seconds to refresh about 900
> packages.  I'm guessing most users must have less than 100 Emacs
> packages in their collections (I have about 35). I think the current
> performance is good enough to not worry optimizing it at this stage.
Point taken, I don't have that many either.  I was merely interested,
given that I rarely dive that deep into runtime statistics of Elisp
code.

Regards,
Leo