bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 2631 bytes --]

Hello Guix!

tl;dr: The Emacs build system and site-start.el loader are modified so
that Emacs packages are installed in their own distinct installation
directory.

The Emacs packages built with the Emacs built system used to be
installed in a sub-directory under the share/emacs/guix.d/ directory,
but this was changed in commit 65a7dd2950ca13a8b942b2836260a2192351b271
shortly after having accommodated the site-start.el machinery to enable
loading packages from any profile (via the EMACSLOADPATH search path
specification).

While this change allowed to expose simply and directly the packages
found in EMACSLOADPATH, it does introduce the risk of file name
collisions when multiple Emacs packages are joined in the same profile,
especially with Emacs packages increasing in complexity (e.g., using
more than a single .el file!) and expecting to have both their sources
and resources extracted under their own nested directory rather than as
a flat collection (ELPA, MELPA).

One recent example I stumbled on was attempting to use the
emacs-yasnippet-snippets package along with emacs-elpy; both wanted to
install a 'snippets' directory to share/emacs/site-lisp/snippets,
collided and resulted in problems that prove difficult to understand.

This is what motivated this patch series, where the site-start.el
auxiliary code used for package discovery is extended to support
packages installed in their own directory under a 'share/emacs/guix'
installation prefix, via Emacs' own package library!

The emacs-build-system is updated for this new installation prefix, as
well as existing packages and documentation.  Parting with a directly
usable EMACSLOADPATH means that site-start.el *must* run for packages to
appear in the load-path; that means for running a test suite, the -Q or
--quick Emacs options cannot be used, since it implies --no-site-file.

Benefits of using this approach:

+ Avoid inter-package file name collisions.

+ Better integration with user installed packages via M-x
package-install.  The Guix-installed packages are listed in M-x
package-list as 'external'.

Cons include:

- Slightly more complex loader (although much of it is offloaded to
  package.el), thus slightly slower (see the comparison below).

- Requires to ensure every package's test suite doesn't make use of -Q.

In my opinion the benefits outweighs the cons by a comfortable margin,
especially with the boring work of adapting the package collection
already done.

To test the performance of the new approach, the following manifest file
was used to test the rebuild of the ~900 Emacs packages making use of
the Emacs build system:


[-- Attachment #2: emacs-packages-manifest.scm --]
[-- Type: text/plain, Size: 806 bytes --]

(use-modules (gnu packages)
             (guix build-system)
             (guix packages)
             (srfi srfi-1))

(define %broken-emacs-packages
  (map specification->package
       '("emacs-picpocket"              ;tests fail
         "emacs-twittering-mode"        ;build fails

         ;; Broken only on current master, without new changes.
         "emacs-md4rd"
         "emacs-el-patch"
         "emacs-flymake-shellcheck"
         )))

(define %emacs-packages
  (fold-packages (lambda (package lst)
                   (if (eq? (build-system-name (package-build-system package))
                            'emacs)
                       (cons package lst)
                       lst))
                 '()))

(packages->manifest
 (lset-difference eqv? %emacs-packages %broken-emacs-packages))

[-- Attachment #3: Type: text/plain, Size: 2772 bytes --]


A simple benchmark testing the performance of the activation of the
hundreds of Emacs packages was then run using:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix environment --pure -m emacs-packages-manifest.scm \
                                  --ad-hoc emacs
[env]$ /run/setuid-programs/sudo /bin/sh -c 'echo 3 > /proc/sys/vm/drop_caches'
[env]$ emacs --batch --no-site-file \
    --eval="(progn (require 'guix-emacs) \
                   (require 'benchmark) \
                   (message \"(total gc-count gc-time) = %s\" \
                            (benchmark-run 1 (guix-emacs-autoload-packages))))"
--8<---------------cut here---------------end--------------->8---

On the master branch:

--8<---------------cut here---------------start------------->8---
[...]
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/zotxt-autoloads...
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/zoutline-autoloads...
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/ztree-autoloads...
(total gc-count gc-time) = (25.242400751 13 0.189669369)
--8<---------------cut here---------------end--------------->8---

Or about 0.65 s on a warm cache.

On a branch with these changes:

--8<---------------cut here---------------start------------->8---
Error loading autoloads: (file-missing Cannot open load file No such file or directory kotl/kotl-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory helm-easymenu)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/flycheck-cpplint-0.1-1.1d8a090/flycheck-cpplint-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/evil-anzu-0.03/evil-anzu-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/erc-image-0-3.82fb387/erc-image-autoloads)
ad-handle-definition: `ido-completing-read' got redefined
Error loading autoloads: (file-missing Cannot open load file No such file or directory tex-site)
(total gc-count gc-time) = (26.175704339 47 0.783184412)
--8<---------------cut here---------------end--------------->8---

Or about 3 seconds on a warm cache.

There a 6 errors that would need to be looked into, but I these look
like actual packaging problems rather than new issues.  The previously
used way to load the autoloads, '(load f 'noerror)' would have masked
them.

Thanks,

Maxim

bug#45316: [PATCH 01/26] build-systems/emacs: Install packages in their own directory.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 24722 bytes --]

This avoids the risk of file name collisions between Emacs packages installed
to a profile.  One example of such a clash is the 'snippets' directory
installed by both the emacs-elpy and emacs-yasnippet-snippets packages.

* guix/build/emacs-build-system.scm (%install-prefix): Append '/guix' to the
existing %install-dir prefix.
(version->elpa-version): New procedure.
(build, patch-el-files, install, move-doc, make-autoloads): Adjust phases.
(create-pkg.el): New phase.
(outputs->elpa-install-dir): New procedure.
(%standard-phases): Register the new 'create-pkg.el phase.
* guix/build/emacs-utils.scm (%emacs-quick-arguments): New variable.
(emacs-batch-eval): Remove the "--quick" option, and use
%emacs-quick-arguments instead.
(emacs-batch-edit-file): Likewise.
* gnu/packages/aux-files/emacs/guix-emacs.el: Update various comments.
(guix-emacs-package-initialize-ran-p): New variable.
(guix-emacs-package-initialize): New procedure, to replace
guix-emacs-autoload-packages.  Use 'package-initialize' from the 'package'
library of Emacs to do the initialization (load path configuration and
autoloads loading) of the Emacs packages.  Use the same autoload invocation as
'package-initialize', to harmonize the output.
(guix-package-initialize): New alias.
(guix-emacs-autoload-packages): New deprecated alias.
* gnu/packages/emacs.scm (emacs)[phases]{install-site-start}: Adjust for the
auxiliary file name change and new procedure name.  Streamline the require
directive: the 'guix-emacs library is shipped with our Emacs package; it
cannot be missing.  Fix typo in a comment.  Do not use the "--quick" option
when byte compiling.
* doc/guix.texi: Update doc.
---
 doc/contributing.texi                      |   7 +-
 doc/guix.texi                              |  37 ++++---
 gnu/packages/aux-files/emacs/guix-emacs.el |  79 +++++++++++----
 gnu/packages/emacs.scm                     |  13 ++-
 guix/build/emacs-build-system.scm          | 109 +++++++++++++++------
 guix/build/emacs-utils.scm                 |  20 ++--
 6 files changed, 186 insertions(+), 79 deletions(-)

diff --git a/doc/contributing.texi b/doc/contributing.texi
index 4195cb4105..1081ddc6d6 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -652,7 +652,12 @@ enabled by setting the @code{#:tests?} argument to @code{#true}.  By
 default, the command to run the test is @command{make check}, but any
 command can be specified via the @code{#:test-command} argument.  The
 @code{#:test-command} argument expects a list containing a command and
-its arguments, to be invoked during the @code{check} phase.
+its arguments, to be invoked during the @code{check} phase.  The Elisp
+libraries discovery mechanism used in Guix relies on the site file being
+run.  For this reason, scripts or Makefile recipes must not invoke Emacs
+with the @option{--no-site-file}, @option{--quick} or @option{-Q}
+options as this would cause the Elisp dependencies added as inputs to
+appear unavailable.
 
 The Elisp dependencies of Emacs packages are typically provided as
 @code{propagated-inputs} when required at run time.  As for other
diff --git a/doc/guix.texi b/doc/guix.texi
index 392baf5910..74713b1331 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -1887,19 +1887,27 @@ information.
 @subsection Emacs Packages
 
 @cindex @code{emacs}
-When you install Emacs packages with Guix, the Elisp files are placed
-under the @file{share/emacs/site-lisp/} directory of the profile in
-which they are installed.  The Elisp libraries are made available to
-Emacs through the @env{EMACSLOADPATH} environment variable, which is
-set when installing Emacs itself.
-
-Additionally, autoload definitions are automatically evaluated at the
-initialization of Emacs, by the Guix-specific
-@code{guix-emacs-autoload-packages} procedure.  If, for some reason, you
-want to avoid auto-loading the Emacs packages installed with Guix, you
-can do so by running Emacs with the @option{--no-site-file} option
-(@pxref{Init File,,, emacs, The GNU Emacs Manual}).
-
+When you install Emacs packages with Guix, the Elisp files are placed in
+their own directory under the @file{share/emacs/site-lisp/guix}
+installation prefix of the profile in which they are installed.  The
+Elisp libraries are made available to Emacs through the
+@env{EMACSLOADPATH} environment variable, which is set when installing
+Emacs itself.
+
+@cindex package activation, emacs
+@cindex guix-package-initialize
+Additionally, the packages are ``activated'', that is, their autoload
+definitions are evaluated and their specific installation directory
+added to the load path at the initialization of Emacs, by the
+Guix-specific @code{guix-package-initialize} procedure.  This procedure
+can also be called interactively in Emacs to refresh the list of
+available packages, for example, after installing new Emacs packages to
+the user profile.
+
+If, for some reason, you want to avoid auto-loading the Emacs packages
+installed with Guix, you can do so by running Emacs with the
+@option{--no-site-file} option (@pxref{Init File,,, emacs, The GNU Emacs
+Manual}).
 
 @node Upgrading Guix
 @section Upgrading Guix
@@ -7940,7 +7948,8 @@ It first creates the @code{@code{package}-autoloads.el} file, then it
 byte compiles all Emacs Lisp files.  Differently from the Emacs
 packaging system, the Info documentation files are moved to the standard
 documentation directory and the @file{dir} file is deleted.  The Elisp
-package files are installed directly under @file{share/emacs/site-lisp}.
+package files are installed in their own directory under the
+@file{share/site-lisp/guix} installation prefix.
 @end defvr
 
 @defvr {Scheme Variable} font-build-system
diff --git a/gnu/packages/aux-files/emacs/guix-emacs.el b/gnu/packages/aux-files/emacs/guix-emacs.el
index ca9146c535..a3c0eaa6fc 100644
--- a/gnu/packages/aux-files/emacs/guix-emacs.el
+++ b/gnu/packages/aux-files/emacs/guix-emacs.el
@@ -1,8 +1,8 @@
-;;; guix-emacs.el --- Emacs packages installed with Guix
+;;; guix-emacs.el --- Autoload Emacs packages installed with Guix
 
 ;; Copyright © 2014, 2015, 2016, 2017 Alex Kost <alezost@gmail.com>
 ;; Copyright © 2017 Kyle Meyer <kyle@kyleam.com>
-;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;; Copyright © 2019, 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 
 ;; This file is part of GNU Guix.
 
@@ -21,43 +21,84 @@
 
 ;;; Commentary:
 
-;; This file provides auxiliary code to autoload Emacs packages
+;; This file provides auxiliary code to initialize the Emacs packages
 ;; installed with Guix.
 
 ;;; Code:
+(require 'package)
 (require 'seq)
 
 (defvar guix-emacs-autoloads-regexp
   (rx (* any) "-autoloads.el" (zero-or-one "c") string-end)
-  "Regexp to match Emacs 'autoloads' file.")
+  "Regexp to match 'autoloads' file.")
 
 (defun guix-emacs-find-autoloads (directory)
-  "Return a list of Emacs 'autoloads' files in DIRECTORY.
+  "Return a list of 'autoloads' files in DIRECTORY.
 The files in the list do not have extensions (.el, .elc)."
   ;; `directory-files' doesn't honor group in regexp.
   (delete-dups (mapcar #'file-name-sans-extension
                        (directory-files directory 'full-name
                                         guix-emacs-autoloads-regexp))))
 
+(defvar guix-emacs-package-initialize-ran-p nil
+  "Non-nil if `guix-emacs-package-initialize' has run.")
+
 ;;;###autoload
-(defun guix-emacs-autoload-packages ()
-  "Autoload Emacs packages found in EMACSLOADPATH.
+(defun guix-emacs-package-initialize ()
+  "Initialize the Emacs packages.
 
-'Autoload' means to load the 'autoloads' files matching
-`guix-emacs-autoloads-regexp'."
+Load the autoloads files of the packages, and add their directory
+to the Emacs load path if they aren't already."
   (interactive)
-  (let* ((emacs-non-core-load-path-directories
-          ;; Filter out core Elisp directories, which are already autoloaded
-          ;; by Emacs.
-          (seq-filter (lambda (dir)
-                        (string-match-p "/share/emacs/site-lisp" dir))
-                      load-path))
-         (autoloads (mapcan #'guix-emacs-find-autoloads
-                            emacs-non-core-load-path-directories)))
+  (let* (;; Filter out core Elisp directories, which have already been
+         ;; autoloaded by Emacs.
+         (load-path* (seq-filter (lambda (f)
+                                   (string-match-p "/share/emacs/site-lisp" f))
+                                 load-path))
+         (autoloads (mapcan #'guix-emacs-find-autoloads load-path*))
+         (package-directory-list* (mapcar (lambda (f)
+                                            (expand-file-name "guix" f))
+                                          load-path*))
+         (user-emacs-directory-warning nil) ;squelch extraneous warnings
+         (package-user-dir ""))      ;do not activate package.el user packages
+
+    ;; Autoload packages installed without using 'emacs-build-system'; these
+    ;; are found directly under the site-lisp directory (flat hierarchy).
     (mapc (lambda (f)
-            (load f 'noerror))
-          autoloads)))
+            (with-demoted-errors "Error loading autoloads: %s"
+              (load f nil t)))
+          autoloads)
+
+    ;; Set the package-directory-list variable to a meaningful default value
+    ;; on Guix, which for 'package.el' is akin to system packages.  It's
+    ;; important that this variable contains the correct value to see the
+    ;; Guix packages listed (as 'external') in M-x package-list.
+    (setq-default package-directory-list package-directory-list*)
+
+    ;; Initialize the Guix-installed Emacs packages.
+    (package-initialize)
+
+    (unless guix-emacs-package-initialize-ran-p
+      ;; If this is running the first time, hide the fact that
+      ;; `'package-initialization' has run in order to leave the default
+      ;; behavior of package.el unchanged for its users (that is, to run
+      ;; automatically after the early-init.el file but before the user init
+      ;; file -- this can be useful if they configured their `'package-user-dir'
+      ;; variable).
+      (setq package--initialized nil)
+      (setq package--activated nil))
+
+    (setq guix-emacs-package-initialize-ran-p t)))
+
+;;;###autoload
+(defalias 'guix-package-initialize #'guix-emacs-package-initialize)
+
+;;;###autoload
+(define-obsolete-function-alias
+  'guix-emacs-autoload-packages 'guix-package-initialize "20201218")
 
+;;; The symbol name is suffixed by -emacs to avoid a name clash with
+;;; Emacs-Guix's guix.el.
 (provide 'guix-emacs)
 
 ;;; guix-emacs.el ends here
diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
index ca14584ada..98f886278a 100644
--- a/gnu/packages/emacs.scm
+++ b/gnu/packages/emacs.scm
@@ -145,8 +145,8 @@
                 "pwd"))
              #t))
          (add-after 'install 'install-site-start
-           ;; Use 'guix-emacs' in "site-start.el", which is used autoload the
-           ;; Elisp packages found in EMACSLOADPATH.
+           ;; Use 'guix-emacs' in "site-start.el", which is used to activate
+           ;; the Elisp packages found in EMACSLOADPATH.
            (lambda* (#:key inputs outputs #:allow-other-keys)
              (let* ((out      (assoc-ref outputs "out"))
                     (lisp-dir (string-append out "/share/emacs/site-lisp"))
@@ -158,18 +158,17 @@
                                (setq byte-compile-debug t)
                                (byte-recompile-directory
                                 (file-name-as-directory ,dir) 0 1))))
-                   (invoke emacs "--quick" "--batch"
-                           (format #f "--eval=~s" expr))))
+                   (invoke emacs "--batch" (format #f "--eval=~s" expr))))
 
                (copy-file (assoc-ref inputs "guix-emacs.el")
                           (string-append lisp-dir "/guix-emacs.el"))
                (with-output-to-file (string-append lisp-dir "/site-start.el")
                  (lambda ()
                    (display
-                    (string-append "(when (require 'guix-emacs nil t)\n"
-                                   "  (guix-emacs-autoload-packages))\n"))))
+                    (string-append "(require 'guix-emacs)\n"
+                                   "(guix-package-initialize)\n"))))
                ;; Remove the extraneous subdirs.el file, as it causes Emacs to
-               ;; add recursively all the the sub-directories of a profile's
+               ;; add recursively all the sub-directories of a profile's
                ;; share/emacs/site-lisp union when added to EMACSLOADPATH,
                ;; which leads to conflicts.
                (delete-file (string-append lisp-dir "/subdirs.el"))
diff --git a/guix/build/emacs-build-system.scm b/guix/build/emacs-build-system.scm
index 26ea59bc25..2846926298 100644
--- a/guix/build/emacs-build-system.scm
+++ b/guix/build/emacs-build-system.scm
@@ -2,7 +2,7 @@
 ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch>
 ;;; Copyright © 2016 David Thompson <davet@gnu.org>
 ;;; Copyright © 2016 Alex Kost <alezost@gmail.com>
-;;; Copyright © 2018, 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2018, 2019, 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -32,7 +32,8 @@
   #:export (%standard-phases
             %default-include
             %default-exclude
-            emacs-build))
+            emacs-build
+            outputs->elpa-install-dir))
 
 ;; Commentary:
 ;;
@@ -40,10 +41,13 @@
 ;;
 ;; Code:
 
-;;; All the packages are installed directly under site-lisp, which means that
-;;; having that directory in the EMACSLOADPATH is enough to have them found by
-;;; Emacs.
-(define %install-dir "/share/emacs/site-lisp")
+;;; All the packages are installed under their own directory under
+;;; site-lisp/guix, to prevent file name collisions in profiles.  This means
+;;; that some machinery is needed to activate the packages, since they aren't
+;;; directly on the load path.  This is taken care of by the
+;;; 'guix-package-initialize' procedure called from the site-start.el file
+;;; shipped with Emacs from Guix.
+(define %install-prefix "/share/emacs/site-lisp/guix")
 
 ;; These are the default inclusion/exclusion regexps for the install phase.
 (define %default-include '("^[^/]*\\.el$" "^[^/]*\\.info$" "^doc/.*\\.info$"))
@@ -62,6 +66,26 @@ name that has been stripped of the hash and version number."
                     (strip-store-file-name file) suffix))))
       (string-append name suffix))))
 
+(define (version->elpa-version version)
+  "Sanitize VERSION into an ELPA-compatible version.
+
+package.el uses the 'version-to-list' procedure from the 'subr' Emacs library,
+which supports a very limited set of version snapshot annotations, defined in
+the 'version-regexp-alist' variable of the same library.
+
+Strip the least significant components of a version string, so that a Guix
+version such as \"0.4.1-1.a41d5cc\" becomes the valid ELPA version \"0.4.1\".
+In case a valid ELPA version cannot be derived from VERSION, \"0.0.0\" is
+returned."
+  (let ((m (string-match "[0-9]+(\\.[0-9]+)*" version)))
+    (if m
+        (match:substring m)
+        (begin
+          (format (current-error-port)
+                  "warning: failed producing an ELPA-compatible \
+version string from ~s; using \"0.0.0\"~%" version)
+          "0.0.0"))))
+
 (define* (unpack #:key source #:allow-other-keys)
   "Unpack SOURCE into the build directory.  SOURCE may be a compressed
 archive, a directory, or an Emacs Lisp file."
@@ -93,13 +117,12 @@ archive, a directory, or an Emacs Lisp file."
 environment variable\n" source-directory)))
 
 (define* (build #:key outputs inputs #:allow-other-keys)
-  "Compile .el files."
-  (let* ((emacs (string-append (assoc-ref inputs "emacs") "/bin/emacs"))
-         (out (assoc-ref outputs "out"))
-         (site-lisp (string-append out %install-dir)))
+  "Byte compile .el files."
+  (let ((emacs (string-append (assoc-ref inputs "emacs") "/bin/emacs"))
+        (install-dir (outputs->elpa-install-dir outputs)))
     (setenv "SHELL" "sh")
     (parameterize ((%emacs emacs))
-      (emacs-byte-compile-directory site-lisp))))
+      (emacs-byte-compile-directory install-dir))))
 
 (define* (patch-el-files #:key outputs #:allow-other-keys)
   "Substitute the absolute \"/bin/\" directory with the right location in the
@@ -115,13 +138,12 @@ store in '.el' files."
            (else (loop (read-line in 'concat))))))
       #:binary #t))
 
-  (let* ((out (assoc-ref outputs "out"))
-         (site-lisp (string-append out %install-dir))
-         ;; (ice-9 regex) uses libc's regexp routines, which cannot deal with
-         ;; strings containing NULs.  Filter out such files.  TODO: Remove
-         ;; this workaround when <https://bugs.gnu.org/30116> is fixed.
-         (el-files (remove file-contains-nul-char?
-                           (find-files (getcwd) "\\.el$"))))
+  (let ((install-dir (outputs->elpa-install-dir outputs))
+        ;; (ice-9 regex) uses libc's regexp routines, which cannot deal with
+        ;; strings containing NULs.  Filter out such files.  TODO: Remove
+        ;; this workaround when <https://bugs.gnu.org/30116> is fixed.
+        (el-files (remove file-contains-nul-char?
+                          (find-files (getcwd) "\\.el$"))))
     (define (substitute-program-names)
       (substitute* el-files
         (("\"/bin/([^.]\\S*)\"" _ cmd-name)
@@ -130,7 +152,7 @@ store in '.el' files."
              (error "patch-el-files: unable to locate " cmd-name))
            (string-append "\"" cmd "\"")))))
 
-    (with-directory-excursion site-lisp
+    (with-directory-excursion install-dir
       ;; Some old '.el' files (e.g., tex-buf.el in AUCTeX) are still
       ;; ISO-8859-1-encoded.
       (unless (false-if-exception (substitute-program-names))
@@ -180,15 +202,14 @@ parallel. PARALLEL-TESTS? is ignored when using a non-make TEST-COMMAND."
       (and (any (cut match-stripped-file "included" <>) include)
            (not (any (cut match-stripped-file "excluded" <>) exclude)))))
 
-  (let* ((out (assoc-ref outputs "out"))
-         (site-lisp (string-append out %install-dir))
+  (let ((install-dir (outputs->elpa-install-dir outputs))
          (files-to-install (find-files source install-file?)))
     (cond
      ((not (null? files-to-install))
       (for-each
        (lambda (file)
          (let* ((stripped-file (string-drop file (string-length source)))
-                (target-file (string-append site-lisp stripped-file)))
+                (target-file (string-append install-dir stripped-file)))
            (format #t "`~a' -> `~a'~%" file target-file)
            (install-file file (dirname target-file))))
        files-to-install)
@@ -199,15 +220,31 @@ parallel. PARALLEL-TESTS? is ignored when using a non-make TEST-COMMAND."
                            (install-file? file stat #:verbose? #t)))
       #f))))
 
+(define* (create-pkg.el #:key outputs #:allow-other-keys)
+  "Generate the name-pkg.el file required by package.el."
+  (let*-values (((install-dir)   (outputs->elpa-install-dir outputs))
+                ((elpa-name-ver) (store-directory->elpa-name-version
+                                  (assoc-ref outputs "out")))
+                ((name version)  (package-name->name+version elpa-name-ver))
+                ((elpa-version)  (version->elpa-version version)))
+    (call-with-output-file (string-append install-dir "/" name "-pkg.el")
+      (lambda (port)
+        ;; Note: the file cannot be byte compiled, as the loader expects a
+        ;; single form present, which leaves no option to include (require
+        ;; 'package).
+        (display ";; Generated by Guix -*- no-byte-compile: t -*-\n" port)
+        (write `(define-package ,name ,elpa-version) port)
+        (display "\n" port)
+        #t))))
+
 (define* (move-doc #:key outputs #:allow-other-keys)
   "Move info files from the ELPA package directory to the info directory."
-  (let* ((out (assoc-ref outputs "out"))
-         (site-lisp (string-append out %install-dir))
-         (info-dir (string-append out "/share/info/"))
-         (info-files (find-files site-lisp "\\.info$")))
+  (let* ((install-dir (outputs->elpa-install-dir outputs))
+         (info-dir (string-append (assoc-ref outputs "out") "/share/info/"))
+         (info-files (find-files install-dir "\\.info$")))
     (unless (null? info-files)
       (mkdir-p info-dir)
-      (with-directory-excursion site-lisp
+      (with-directory-excursion install-dir
         (when (file-exists? "dir") (delete-file "dir"))
         (for-each (lambda (f)
                     (copy-file f (string-append info-dir "/" (basename f)))
@@ -218,12 +255,12 @@ parallel. PARALLEL-TESTS? is ignored when using a non-make TEST-COMMAND."
 (define* (make-autoloads #:key outputs inputs #:allow-other-keys)
   "Generate the autoloads file."
   (let* ((emacs (string-append (assoc-ref inputs "emacs") "/bin/emacs"))
-         (out (assoc-ref outputs "out"))
-         (site-lisp (string-append out %install-dir))
-         (elpa-name-ver (store-directory->elpa-name-version out))
+         (install-dir (outputs->elpa-install-dir outputs))
+         (elpa-name-ver (store-directory->elpa-name-version
+                         (assoc-ref outputs "out")))
          (elpa-name (package-name->name+version elpa-name-ver)))
     (parameterize ((%emacs emacs))
-      (emacs-generate-autoloads elpa-name site-lisp))))
+      (emacs-generate-autoloads elpa-name install-dir))))
 
 (define* (enable-autoloads-compilation #:key outputs #:allow-other-keys)
   "Remove the NO-BYTE-COMPILATION local variable embedded in the generated
@@ -258,6 +295,13 @@ second hyphen.  This corresponds to 'name-version' as used in ELPA packages."
             strip-store-file-name)
    store-dir))
 
+(define (outputs->elpa-install-dir outputs)
+  "Given OUTPUTS, a package outputs, return the ELPA-like Guix installation
+directory of the Emacs package."
+  (let ((out (assoc-ref outputs "out")))
+    (string-append out %install-prefix "/"
+                   (store-directory->elpa-name-version out))))
+
 (define %standard-phases
   (modify-phases gnu:%standard-phases
     (replace 'unpack unpack)
@@ -267,7 +311,8 @@ second hyphen.  This corresponds to 'name-version' as used in ELPA packages."
     (delete 'build)
     (replace 'check check)
     (replace 'install install)
-    (add-after 'install 'make-autoloads make-autoloads)
+    (add-after 'install 'create-pkg.el create-pkg.el)
+    (add-after 'create-pkg.el 'make-autoloads make-autoloads)
     (add-after 'make-autoloads 'enable-autoloads-compilation
       enable-autoloads-compilation)
     (add-after 'enable-autoloads-compilation 'patch-el-files patch-el-files)
diff --git a/guix/build/emacs-utils.scm b/guix/build/emacs-utils.scm
index 5f7ba71244..485218d040 100644
--- a/guix/build/emacs-utils.scm
+++ b/guix/build/emacs-utils.scm
@@ -23,6 +23,7 @@
   #:use-module (guix build utils)
   #:use-module (ice-9 format)
   #:export (%emacs
+            %emacs-quick-arguments
             emacs-batch-eval
             emacs-batch-edit-file
             emacs-batch-disable-compilation
@@ -42,6 +43,11 @@
   ;; The `emacs' command.
   (make-parameter "emacs"))
 
+(define %emacs-quick-arguments
+  ;; Like --quick (-Q), except the site-start.el file is loaded since it is
+  ;; required to load packages.
+  '("--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources"))
+
 (define (expr->string expr)
   "Converts EXPR, an expression, into a string."
   (if (string? expr)
@@ -51,15 +57,17 @@
 (define* (emacs-batch-eval expr #:key dynamic?)
   "Run Emacs in batch mode, and execute the Elisp code EXPR.  If DYNAMIC? is
 true, evaluate using dynamic scoping."
-  (invoke (%emacs) "--quick" "--batch"
-          (format #f "--eval=(eval '~a ~:[t~;nil~])"
-                  (expr->string expr) dynamic?)))
+  (apply invoke (%emacs) "--batch"
+         (format #f "--eval=(eval '~a ~:[t~;nil~])"
+                 (expr->string expr) dynamic?)
+         %emacs-quick-arguments))
 
 (define (emacs-batch-edit-file file expr)
   "Load FILE in Emacs using batch mode, and execute the elisp code EXPR."
-  (invoke (%emacs) "--quick" "--batch"
-          (string-append "--visit=" file)
-          (string-append "--eval=" (expr->string expr))))
+  (apply invoke (%emacs) "--batch"
+         (string-append "--visit=" file)
+         (string-append "--eval=" (expr->string expr))
+         %emacs-quick-arguments))
 
 (define (emacs-batch-disable-compilation file)
   (emacs-batch-edit-file file
-- 
2.29.2

bug#45316: [PATCH 02/26] gnu: emacs-ert-runner: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-ert-runner)[phases]{patch-launcher}: New
phase.
{install-executable}: Adjust.
---
 gnu/packages/emacs-xyz.scm | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index e242bcf5a8..6ae301aeaf 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -12367,6 +12367,14 @@ one if it fails.")
     (arguments
      `(#:phases
        (modify-phases %standard-phases
+         (add-after 'unpack 'patch-launcher
+           ;; Do not use -Q, as this implies not loading site-start.el, which
+           ;; is required to make Guix Elisp library packages discoverable.
+           (lambda _
+             (substitute* "bin/ert-runner"
+               (("-Q")
+                (string-join %emacs-quick-arguments)))
+             #t))
 	 (add-after 'install 'install-executable
 	   (lambda* (#:key inputs outputs #:allow-other-keys)
 	     (let ((out (assoc-ref outputs "out"))
@@ -12374,8 +12382,8 @@ one if it fails.")
 				      (getenv "TMPDIR") "/source")))
 	       (substitute* "bin/ert-runner"
 		 (("ERT_RUNNER=\"\\$\\(dirname \\$\\(dirname \\$0\\)\\)")
-		  (string-append "ERT_RUNNER=\"" out
-				 "/share/emacs/site-lisp")))
+		  (string-append "ERT_RUNNER=\""
+                                 (outputs->elpa-install-dir outputs))))
 	       (install-file "bin/ert-runner" (string-append out "/bin"))
 	       (wrap-program (string-append out "/bin/ert-runner")
 		 (list "EMACSLOADPATH" ":" 'prefix
-- 
2.29.2

bug#45316: [PATCH 03/26] gnu: emacs-xyz: Do not use --quick or -Q.

[Maxim Cournoyer]

The --quick options prevents loading the site-start.el, which is now required
for the initialization of the Guix Emacs packages.

* gnu/packages/emacs-xyz.scm (emacs-flycheck)
(emacs-perspective, emacs-nodejs-repl, emacs-rainbow-delimiters)
(emacs-emacsql-sqlite3, emacs-evil-magit, emacs-adoc-mode)
(emacs-cort, emacs-vdiff, emacs-org-generate, emacs-execline): Do not use the
--quick or -Q options.  Harmonize to use --batch instead of -batch.
---
 gnu/packages/emacs-xyz.scm | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 6ae301aeaf..01dab957aa 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -1176,7 +1176,7 @@ supports type hints, definition-jumping, completion, and more.")
                #t)))
          ;; TODO: many failing tests
          #:tests? #f
-         #:test-command '("emacs" "-Q" "--batch" "-L" "."
+         #:test-command '("emacs" "--batch" "-L" "."
                           "--load" "test/flycheck-test"
                           "--load" "test/run.el"
                           "-f" "flycheck-run-tests-main")))
@@ -7006,7 +7006,7 @@ regexp that matches all known keywords.")
     (build-system emacs-build-system)
     (arguments
      `(#:tests? #t
-       #:test-command '("emacs" "-Q" "-batch" "-L" "."
+       #:test-command '("emacs" "--batch" "-L" "."
                         "-l" "test/test-perspective.el"
                         "-f" "ert-run-tests-batch-and-exit")))
     (home-page "https://github.com/nex3/perspective-el")
@@ -8237,7 +8237,7 @@ strings, and code folding.")
                  ("nodejs-repl-command"
                   (string-append node "/bin/node")))))))
        #:tests? #t
-       #:test-command '("emacs" "-Q" "--batch"
+       #:test-command '("emacs" "--batch"
                         "-L" "."
                         "-l" "test/test.el"
                         "-f" "ert-run-tests-batch-and-exit")))
@@ -8817,7 +8817,7 @@ completion, interactive development and more.")
     (home-page "https://github.com/Fanael/rainbow-delimiters")
     (arguments
      `(#:tests? #t
-       #:test-command '("emacs" "-Q" "-batch"
+       #:test-command '("emacs" "--batch"
                         "-l" "rainbow-delimiters-test.el"
                         "-f" "ert-run-tests-batch-and-exit")))
     (synopsis "Highlight brackets according to their depth")
@@ -14638,7 +14638,7 @@ object @code{nil} corresponds 1:1 with @code{NULL} in the database.")
      `(("emacs-emacsql" ,emacs-emacsql)))
     (arguments
      `(#:tests? #t
-       #:test-command '("emacs" "-Q" "--batch" "-L" "."
+       #:test-command '("emacs" "--batch" "-L" "."
                         "--load" "emacsql-sqlite3-test.el"
                         "-f" "ert-run-tests-batch-and-exit")
        #:phases
@@ -17989,7 +17989,7 @@ Org-mode file, and citations of Zotero items in Pandoc Markdown files.")
          ("magit" ,emacs-magit)))
       (arguments
        `(#:tests? #t
-         #:test-command '("emacs" "-Q" "-batch"
+         #:test-command '("emacs" "--batch"
                           "-L" "."
                           "-l" "evil-magit-tests"
                           "-f" "ert-run-tests-batch-and-exit")))
@@ -19014,7 +19014,7 @@ and 'text viewing modes' respectively.")
                                "adoctest-test-tempo-quotes")))
              #t)))
        #:tests? #t
-       #:test-command '("emacs" "-Q" "-batch"
+       #:test-command '("emacs" "--batch"
                         "-l" "adoc-mode-test.el"
                         "-f" "ert-run-tests-batch-and-exit")))
     (propagated-inputs
@@ -19982,7 +19982,7 @@ functions to ensure they are called with the right arguments during testing.")
        `(("emacs-ansi" ,emacs-ansi)))
       (arguments
        `(#:tests? #t
-         #:test-command '("emacs" "--batch" "--quick"
+         #:test-command '("emacs" "--batch"
                           "--directory=." "--load=cort-tests.el"
                           "--funcall=cort-test-run")))
       (home-page "https://github.com/conao3/cort.el")
@@ -21724,7 +21724,7 @@ compatible with Emacs' shell modes.")
        `(("emacs-hydra" ,emacs-hydra)))
       (arguments
        `(#:tests? #t
-         #:test-command '("emacs" "-Q" "-batch" "-L" "."
+         #:test-command '("emacs" "--batch" "-L" "."
                           "-l" "vdiff-test.el"
                           "-f" "ert-run-tests-batch-and-exit")))
       (home-page "https://github.com/justbur/emacs-vdiff/")
@@ -25333,7 +25333,7 @@ personal wiki.")
      `(("emacs-mustache" ,emacs-mustache)))
     (arguments
      `(#:tests? #t
-       #:test-command '("emacs" "--batch" "--quick"
+       #:test-command '("emacs" "--batch"
                         "--directory=." "--load=org-generate-tests.el"
                         "--funcall=cort-test-run")))
     (build-system emacs-build-system)
@@ -25460,7 +25460,7 @@ or region and use of locally installed binaries.")
        `(("emacs-s" ,emacs-s)))
       (arguments
        `(#:tests? #t
-         #:test-command '("emacs" "-Q" "-batch" "-L" "."
+         #:test-command '("emacs" "--batch" "-L" "."
                           "-l" "t/indent.el"
                           "-f" "ert-run-tests-batch-and-exit")))
       (home-page "https://gitlab.com/KAction/emacs-execline/")
-- 
2.29.2

bug#45316: [PATCH 04/26] gnu: cedille: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 4073 bytes --]

Also enable byte compilation for most Elisp files.

* gnu/packages/cedille.scm (cedille)[arguments]: Add the cedille-mode and
se-mode to the #:include files list, which allows to...
[phases]{copy-cedille-mode}: ...remove this phase.
{extend-load-path}: New phase.
{build}: Restore phase.
{disable-byte-compilation-on-problematic-files}: New phase.
---
 gnu/packages/cedille.scm | 50 ++++++++++++++++++++++++----------------
 1 file changed, 30 insertions(+), 20 deletions(-)

diff --git a/gnu/packages/cedille.scm b/gnu/packages/cedille.scm
index d3dd576323..05eb42e9b2 100644
--- a/gnu/packages/cedille.scm
+++ b/gnu/packages/cedille.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2019 John Soo <jsoo1@asu.edu>
+;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -49,37 +50,46 @@
        ("ghc-happy" ,ghc-happy)))
     (build-system emacs-build-system)
     (arguments
-     `(#:phases
+     `(#:include (cons* "^cedille-mode/" "^se-mode/" %default-include)
+       #:phases
        (modify-phases %standard-phases
          (add-after 'unpack 'patch-cedille-paths
            (lambda* (#:key outputs #:allow-other-keys)
              (let ((out (assoc-ref outputs "out")))
                (substitute* "cedille-mode.el"
-                 (("/usr/share/emacs/site-lisp/cedille-mode")
-                  (string-append
-                   out "/share/emacs/site-lisp/cedille")))
+                 ;; As suggested at the top of cedille-mode.el
+                 ((".*defmacro cedille-platform-case.*" all)
+                  (string-append (format #f "(defconst cedille-path ~s)~%~%"
+                                         (outputs->elpa-install-dir outputs))
+                                 all)))
                (substitute* "cedille-mode/cedille-mode-info.el"
                  (("\\(concat cedille-path-el \"cedille-info-main.info\"\\)")
                   (string-append
                    "\"" out "/share/info/cedille-info-main.info.gz\"")))
                #t)))
-         (add-after 'patch-cedille-paths 'copy-cedille-mode
+         (add-before 'build 'extend-load-path
            (lambda* (#:key outputs #:allow-other-keys)
-             (let* ((out (assoc-ref outputs "out"))
-                    (lisp
-                     (string-append
-                      out "/share/emacs/site-lisp/cedille/")))
-               (mkdir-p (string-append lisp "cedille-mode"))
-               (copy-recursively
-                "cedille-mode"
-                (string-append lisp "cedille-mode"))
-               (mkdir-p (string-append lisp "se-mode"))
-               (copy-recursively
-                "se-mode"
-                (string-append lisp "se-mode"))
-               #t)))
-         ;; FIXME: Byte compilation fails
-         (delete 'build)
+             (let ((install-dir (outputs->elpa-install-dir outputs)))
+               (setenv "EMACSLOADPATH"
+                       (string-join
+                        (cons (getenv "EMACSLOADPATH")
+                              (map (lambda (f)
+                                     (string-append install-dir "/" f))
+                                   '("cedille-mode" "se-mode")))
+                        ":")))
+             #t))
+         (add-after 'unpack 'disable-byte-compilation-on-problematic-files
+           ;; Some files fail byte compilation (see:
+           ;; https://github.com/cedille/cedille/issues/160).
+           (lambda _
+             (let ((problematic-files '("cedille-mode.el"
+                                        "cedille-mode/cedille-mode-library.el"
+                                        "se-mode/se-hole.el")))
+               (for-each (lambda (f)
+                           (make-file-writable f)
+                           (emacs-batch-disable-compilation f))
+                         problematic-files))
+             #t))
          (replace 'check
            (lambda _
              (with-directory-excursion "cedille-tests"
-- 
2.29.2

bug#45316: [PATCH 05/26] gnu: emacs-libgit: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-libgit)[phases]
{set-libgit--module-file}: Adjust the installation prefix.
{create-pkg.el}: New phase.
---
 gnu/packages/emacs-xyz.scm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 01dab957aa..1a168075e3 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -457,7 +457,8 @@ on stdout instead of using a socket as the Emacsclient does.")
                  (make-file-writable "libgit.el")
                  (emacs-substitute-variables "libgit.el"
                    ("libgit--module-file"
-                    (string-append out "/share/emacs/site-lisp/libegit2.so")))
+                    (string-append (emacs:outputs->elpa-install-dir outputs)
+                                   "/libegit2.so")))
                  #t)))
            (add-before 'install 'prepare-for-install
              (lambda _
@@ -471,7 +472,9 @@ on stdout instead of using a socket as the Emacsclient does.")
                  (install #:outputs outputs
                           #:include (cons "\\.so$"
                                           emacs:%default-include)))))
-           (add-after 'install 'make-autoloads
+           (add-after 'install 'create-pkg.el
+             (assoc-ref emacs:%standard-phases 'create-pkg.el))
+           (add-after 'create-pkg.el 'make-autoloads
              (assoc-ref emacs:%standard-phases 'make-autoloads))
            (add-after 'make-autoloads 'enable-autoloads-compilation
              (assoc-ref emacs:%standard-phases 'enable-autoloads-compilation))
-- 
2.29.2

bug#45316: [PATCH 06/26] gnu: emacs-typit: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-typit)[arguments]: Include "dict".
[phases]{install-dictionaries}: Remove.
---
 gnu/packages/emacs-xyz.scm | 12 +-----------
 1 file changed, 1 insertion(+), 11 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 1a168075e3..2702738fc0 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -882,17 +882,7 @@ handful of functions that are not resource-specific.")
           (base32 "1savrxs7xl92ifyxpxkkzv2didr7lb405h0dwz1bs1wldr5fb53f"))))
       (build-system emacs-build-system)
       (arguments
-       `(#:phases
-         (modify-phases %standard-phases
-           (add-after 'install 'install-dictionaries
-             (lambda* (#:key outputs #:allow-other-keys)
-               (let* ((out (assoc-ref outputs "out"))
-                      (site-lisp
-                       (string-append
-                        out "/share/emacs/site-lisp/dict")))
-                 (mkdir-p site-lisp)
-                 (copy-recursively "dict" site-lisp)
-                 #t))))))
+       `(#:include (cons* "^dict/" %default-include)))
       (propagated-inputs
        `(("emacs-f" ,emacs-f)
          ("emacs-mmt" ,emacs-mmt)))
-- 
2.29.2

bug#45316: [PATCH 07/26] gnu: emacs-flycheck-grammalecte: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-flycheck-grammalecte)[arguments]{include}:
Extend rather than override %default-include.
{exclude}: Extend rather than override %default-exclude.
* gnu/packages/emacs-xyz.scm (emacs-flycheck-grammalecte): Adjust the
installation directory.
---
 gnu/packages/emacs-xyz.scm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 2702738fc0..202db888d4 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -4558,8 +4558,8 @@ for Flow files.")
         (base32 "1mzmzyik843r4j0ibpwqrxmb0g4xmirrf3lxr010bddkmmxf749a"))))
     (build-system emacs-build-system)
     (arguments
-     `(#:include '("\\.(el|py)$")
-       #:exclude '("^test-profile.el$")
+     `(#:include (cons "^[^/]*\\.py$" %default-include)
+       #:exclude (cons "^test-profile.el$" %default-exclude)
        #:emacs ,emacs                   ;need libxml support
        #:phases
        (modify-phases %standard-phases
@@ -4583,7 +4583,7 @@ for Flow files.")
                     (grammalecte (assoc-ref inputs "grammalecte"))
                     (version ,(version-major+minor (package-version python))))
                (with-directory-excursion
-                   (string-append out "/share/emacs/site-lisp")
+                   (outputs->elpa-install-dir outputs)
                  (symlink (string-append grammalecte "/lib/"
                                          "python" version "/site-packages/"
                                          "grammalecte")
-- 
2.29.2

bug#45316: [PATCH 08/26] gnu: emacs-scel: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-scel)[phases]{configure}: Adjust
installation prefix.
---
 gnu/packages/emacs-xyz.scm | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 202db888d4..cb58fc441a 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -5663,8 +5663,7 @@ to a key in your preferred mode.")
              (lambda* (#:key outputs #:allow-other-keys)
                (substitute* "el/CMakeLists.txt"
                  (("share/emacs/site-lisp/SuperCollider")
-                  (string-append
-                   "share/emacs/site-lisp")))
+                  (outputs->elpa-install-dir outputs)))
                ((assoc-ref cmake:%standard-phases 'configure)
                 #:outputs outputs
                 #:configure-flags '("-DSC_EL_BYTECOMPILE=OFF"))))
-- 
2.29.2

bug#45316: [PATCH 09/26] gnu: emacs-chess: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-chess)[arguments]{include}: New argument.
[phases]{install-pieces}: Remove.
---
 gnu/packages/emacs-xyz.scm | 11 +----------
 1 file changed, 1 insertion(+), 10 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index cb58fc441a..4e6fb922c9 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -6676,16 +6676,7 @@ board and goal value can be customized.")
         (base32 "1a4iwjdh6k348df6qywjws9z9f862d62m0b2sz57z4xhywiyxpr7"))))
     (build-system emacs-build-system)
     (arguments
-     `(#:phases
-       (modify-phases %standard-phases
-         (add-after 'install 'install-pieces
-           (lambda* (#:key outputs #:allow-other-keys)
-             (let* ((out (assoc-ref outputs "out"))
-                    (pieces
-                     (string-append out "/share/emacs/site-lisp/pieces")))
-               (mkdir-p pieces)
-               (copy-recursively "pieces" pieces)
-               #t))))))
+     `(#:include (cons "^pieces/" %default-include)))
     (home-page "https://elpa.gnu.org/packages/chess.html")
     (synopsis "Play chess in GNU Emacs")
     (description
-- 
2.29.2

bug#45316: [PATCH 10/26] gnu: emacs-org-contrib: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-org-contrib)[phases]{delete-org-files}:
Adjust installation prefix.
---
 gnu/packages/emacs-xyz.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 4e6fb922c9..0ff904af19 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -10308,7 +10308,7 @@ programming and reproducible research.")
                                                    contrib-files
                                                    org+contrib-files)))
                (with-directory-excursion
-                   (string-append out "/share/emacs/site-lisp")
+                   (outputs->elpa-install-dir outputs)
                  (for-each delete-file duplicates))
                #t))))))
     (propagated-inputs
-- 
2.29.2

bug#45316: [PATCH 11/26] gnu: emacs-edbi: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-edbi)[arguments]
{include}: Extend rather than override %default-include.
[phases]{patch-path}: Adjust installation prefix.
edb#	buffering.patch
---
 gnu/packages/emacs-xyz.scm | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 0ff904af19..3e4dae5314 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -20330,14 +20330,13 @@ asynchronous communications, the RPC response is fairly good.")
        `(("emacs-e2wm" ,emacs-e2wm)
          ("emacs-epc" ,emacs-epc)))
       (arguments
-       `(#:include '("\\.el$" "\\.pl$")
+       `(#:include (cons "^[^/]*\\.pl$" %default-include)
          #:phases
          (modify-phases %standard-phases
            (add-after 'install 'patch-path
              (lambda* (#:key inputs outputs #:allow-other-keys)
                (let ((perl (assoc-ref inputs "perl"))
-                     (dir (string-append  (assoc-ref outputs "out")
-                                          "/share/emacs/site-lisp")))
+                     (dir (outputs->elpa-install-dir outputs)))
                  (substitute* (string-append dir  "/edbi.el")
                    (("\"perl\"") (string-append "\"" perl "/bin/perl\"")))
                  (chmod (string-append dir "/edbi-bridge.pl") #o555)
-- 
2.29.2

bug#45316: [PATCH 12/26] gnu: emacs-telega: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-telega)[phases]{create-pkg.el}: Add phase.
---
 gnu/packages/emacs-xyz.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 3e4dae5314..1be2b36f04 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -23837,6 +23837,8 @@ other @code{helm-type-file} sources such as @code{helm-locate}.")
 	       #t))
            (add-after 'emacs-bytecomp-contrib 'emacs-install
              (assoc-ref emacs:%standard-phases 'install))
+           (add-after 'emacs-install 'create-pkg.el
+             (assoc-ref emacs:%standard-phases 'create-pkg.el))
            ;; This step installs subdir /etc, which contains images, sounds and
            ;; various other data, next to the site-lisp dir.
            (add-after 'emacs-install 'telega-install-data
-- 
2.29.2

bug#45316: [PATCH 13/26] gnu: emacs-rime: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-rime)[phases]
{install-emacs-module}: Adjust installation prefix.
---
 gnu/packages/emacs-xyz.scm | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 1be2b36f04..a027fdfa8f 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -24129,8 +24129,7 @@ REPL appropriate to the current major mode.")
          (add-after 'install 'install-emacs-module
            (lambda* (#:key outputs #:allow-other-keys)
              (install-file "librime-emacs.so"
-                           (string-append (assoc-ref outputs "out")
-                                          "/share/emacs/site-lisp"))
+                           (outputs->elpa-install-dir outputs))
              #t)))))
     (inputs
      `(("librime" ,librime)
-- 
2.29.2

bug#45316: [PATCH 14/26] gnu: emacs-haskell-snippets: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-haskell-snippets)[phases]
{install-snippets}: Remove.
[arguments]{include}: Add argument.
---
 gnu/packages/emacs-xyz.scm | 12 +-----------
 1 file changed, 1 insertion(+), 11 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index a027fdfa8f..17b643a287 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -25248,17 +25248,7 @@ conventions.")
           (base32 "0a7y3awi9hcyahggf0ghsdwvsmrhr9yq634wy9lkqjzrm2hqj0ci"))))
       (build-system emacs-build-system)
       (arguments
-       `(#:phases
-         (modify-phases %standard-phases
-           (add-after 'install 'install-snippets
-             (lambda* (#:key outputs #:allow-other-keys)
-               (let* ((out (assoc-ref outputs "out"))
-                      (snippets
-                       (string-append
-                        out "/share/emacs/site-lisp/snippets/haskell-mode")))
-                 (mkdir-p snippets)
-                 (copy-recursively "snippets/haskell-mode" snippets)
-                 #t))))))
+       `(#:include (cons* "^snippets/" %default-include)))
       (propagated-inputs
        `(("emacs-yasnippet" ,emacs-yasnippet)))
       (home-page "https://github.com/haskell/haskell-snippets")
-- 
2.29.2

bug#45316: [PATCH 15/26] gnu: guile-wisp: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 1885 bytes --]

* gnu/packages/guile-xyz.scm (guile-wisp)[phases]{compile-emacs-files}: Move
after 'make-autoloads.
{create-pkg.el}: New phase.
---
 gnu/packages/guile-xyz.scm | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index dfffbfc62f..349dce73d3 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -16,7 +16,7 @@
 ;;; Copyright © 2017 Theodoros Foradis <theodoros@foradis.org>
 ;;; Copyright © 2017 Nikita <nikita@n0.is>
 ;;; Copyright © 2017, 2018 Tobias Geerinckx-Rice <me@tobias.gr>
-;;; Copyright © 2018 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2018, 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2018, 2019, 2020 Arun Isaac <arunisaac@systemreboot.net>
 ;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
 ;;; Copyright © 2018 Eric Bavier <bavier@member.fsf.org>
@@ -1746,10 +1746,12 @@ The library is shipped with documentation in Info format and usage examples.")
                #t)))
          (add-after 'install 'install-emacs-files
            (assoc-ref emacs:%standard-phases 'install))
-         (add-after 'install-emacs-files 'compile-emacs-files
-           (assoc-ref emacs:%standard-phases 'build))
-         (add-after 'compile-emacs-files 'make-autoloads
-           (assoc-ref emacs:%standard-phases 'make-autoloads)))))
+         (add-after 'install-emacs-files 'create-pkg.el
+           (assoc-ref emacs:%standard-phases 'create-pkg.el))
+         (add-after 'create-pkg.el 'make-autoloads
+           (assoc-ref emacs:%standard-phases 'make-autoloads))
+         (add-after 'make-autoloads 'compile-emacs-files
+           (assoc-ref emacs:%standard-phases 'build)))))
     (home-page "https://www.draketo.de/english/wisp")
     (inputs
      `(("guile" ,guile-3.0)))
-- 
2.29.2

bug#45316: [PATCH 16/26] gnu: emacs-pdf-tools: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-pdf-tools)[phases]{emacs-build}: Move
after 'emacs-make-autoloads.
{emacs-create-pkg.el}: Add phase.
---
 gnu/packages/emacs-xyz.scm | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 17b643a287..a988e56c97 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -2729,10 +2729,12 @@ during idle time, while Emacs is doing nothing else.")
              (assoc-ref emacs:%standard-phases 'add-source-to-load-path))
            (add-after 'emacs-add-source-to-load-path 'emacs-install
              (assoc-ref emacs:%standard-phases 'install))
-           (add-after 'emacs-install 'emacs-build
-             (assoc-ref emacs:%standard-phases 'build))
-           (add-after 'emacs-install 'emacs-make-autoloads
-             (assoc-ref emacs:%standard-phases 'make-autoloads)))))
+           (add-after 'emacs-install 'emacs-create-pkg.el
+             (assoc-ref emacs:%standard-phases 'create-pkg.el))
+           (add-after 'emacs-create-pkg.el 'emacs-make-autoloads
+             (assoc-ref emacs:%standard-phases 'make-autoloads))
+           (add-after 'emacs-make-autoloads 'emacs-build
+             (assoc-ref emacs:%standard-phases 'build)))))
       (native-inputs
        `(("autoconf" ,autoconf)
          ("automake" ,automake)
-- 
2.29.2

bug#45316: [PATCH 17/26] gnu: emacs-emacsql: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-emacsql)[phases]{install}: Do not replace.
---
 gnu/packages/emacs-xyz.scm | 11 +----------
 1 file changed, 1 insertion(+), 10 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index a988e56c97..5663aff95d 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -14575,16 +14575,7 @@ object has been freed.")
                  ;; in the right place.
                  ("(defvar emacsql-sqlite-executable"
                   (string-append (assoc-ref outputs "out")
-                                 "/bin/emacsql-sqlite"))))))
-         (replace 'install
-           (lambda* (#:key outputs #:allow-other-keys)
-             (let* ((out (assoc-ref outputs "out")))
-               (install-file "sqlite/emacsql-sqlite"
-                             (string-append out "/bin"))
-               (for-each (cut install-file <>
-                              (string-append out "/share/emacs/site-lisp"))
-                         (find-files "." "\\.elc*$")))
-             #t)))))
+                                 "/bin/emacsql-sqlite")))))))))
     (inputs
      `(("emacs-minimal" ,emacs-minimal)
        ("mariadb" ,mariadb "lib")
-- 
2.29.2

bug#45316: [PATCH 18/26] gnu: emacs-racer: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-racer)[phases]{fix-makefile}: Do not use
the -Q option.
---
 gnu/packages/emacs-xyz.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 5663aff95d..70c9a08e90 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -19023,7 +19023,8 @@ files.  It focuses on highlighting the document to improve readability.")
          (add-before 'check 'fix-makefile
            (lambda _
              (substitute* "Makefile"
-               (("\\$\\{CASK\\} exec ") ""))
+               (("\\$\\{CASK\\} exec ") "")
+               (("-Q ") ""))
              #t))
          ;; Two tests are failing with Emacs 27, as reported here:
          ;; <https://github.com/racer-rust/emacs-racer/issues/136>.  Disable
-- 
2.29.2

bug#45316: [PATCH 19/26] gnu: emacs-magit: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-magit)[phases]{strip-Q-option}: New phase.
---
 gnu/packages/emacs-xyz.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 70c9a08e90..e4209fa43e 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -520,7 +520,12 @@ libgit2 bindings for Emacs, intended to boost the performance of Magit.")
          #:test-command '("make" "test")
          #:phases
          (modify-phases %standard-phases
-           (add-after 'unpack 'build-info-manual
+           (add-after 'unpack 'strip-Q-option
+             (lambda _
+               (substitute* "default.mk"
+                 (("-Q ") ""))
+               #t))
+           (add-after 'strip-Q-option 'build-info-manual
              (lambda _
                (invoke "make" "info")
                ;; Copy info files to the lisp directory, which acts as
-- 
2.29.2

bug#45316: [PATCH 20/26] gnu: emacs-org-super-agenda: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-org-super-agenda)
[phases]{prepare-for-tests.el}: Do not use -Q in test suite launch script.
---
 gnu/packages/emacs-xyz.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index e4209fa43e..cc6d0f03e7 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -14925,6 +14925,8 @@ as well as functions for navigating between these headings.")
              (substitute* "test/test.el"
                ((".*org-super-agenda-test--:auto-map.*" all)
                 (string-append all "  (skip-unless nil)\n")))
+             (substitute* "test/run"
+               (("-Q ") ""))            ;to load site-start.el
              #t)))))
     (native-inputs
      `(("emacs-f" ,emacs-f)
-- 
2.29.2

bug#45316: [PATCH 21/26] gnu: notmuch: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/mail.scm (notmuch)[phases]{configure}: Adjust install directory
prefix.
{create-pkg.el}: Add phase.
---
 gnu/packages/mail.scm | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/mail.scm b/gnu/packages/mail.scm
index f85713433e..862b6c939a 100644
--- a/gnu/packages/mail.scm
+++ b/gnu/packages/mail.scm
@@ -1280,8 +1280,7 @@ invoking @command{notifymuch} from the post-new hook.")
                       (setenv "CONFIG_SHELL" (which "sh"))
 
                       (let* ((out (assoc-ref outputs "out"))
-                             (elisp
-                              (string-append out "/share/emacs/site-lisp/")))
+                             (elisp (emacs:outputs->elpa-install-dir outputs)))
                         (invoke "./configure"
                                 (string-append "--prefix=" out)
                                 (string-append "--emacslispdir=" elisp)
@@ -1300,7 +1299,9 @@ invoking @command{notifymuch} from the post-new hook.")
                       (substitute* (find-files "test" "\\.sh$")
                         (("/bin/sh") (which "sh")))
                       #t))
-                  (add-after 'install 'make-autoloads
+                  (add-after 'install 'create-pkg.el
+                    (assoc-ref emacs:%standard-phases 'create-pkg.el))
+                  (add-after 'create-pkg.el 'make-autoloads
                     (assoc-ref emacs:%standard-phases 'make-autoloads)))))
     (native-inputs
      `(("bash-completion" ,bash-completion)
-- 
2.29.2

bug#45316: [PATCH 22/26] gnu: emacs-howm: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/mail.scm (emacs-howm)[configure-flags]: Adjust installation
prefix.
[phases]{create-pkg.el}: Add phase.
---
 gnu/packages/emacs-xyz.scm | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index cc6d0f03e7..446b25e502 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -2163,8 +2163,8 @@ a set of simplified face specifications and a user-supplied color palette")
      `(("emacs" ,emacs-minimal)))
     (arguments
      `(#:configure-flags
-       (list (string-append "--with-howmdir=" %output
-                            "/share/emacs/site-lisp/"))
+       (list (string-append "--with-howmdir="
+                            (emacs:outputs->elpa-install-dir %outputs)))
        #:modules ((guix build gnu-build-system)
                   ((guix build emacs-build-system) #:prefix emacs:)
                   (guix build utils))
@@ -2173,7 +2173,9 @@ a set of simplified face specifications and a user-supplied color palette")
                            (guix build emacs-utils))
        #:phases
        (modify-phases %standard-phases
-         (add-after 'install 'make-autoloads
+         (add-after 'install 'create-pkg.el
+           (assoc-ref emacs:%standard-phases 'create-pkg.el))
+         (add-after 'create-pkg.el 'make-autoloads
            (assoc-ref emacs:%standard-phases 'make-autoloads)))))
     (home-page "https://howm.osdn.jp/")
     (synopsis "Note-taking tool for Emacs")
-- 
2.29.2

bug#45316: [PATCH 23/26] gnu: emacs-rjsx-mode: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-rjsx-mode)
[phases]{strip-Q-option}: New phase.
---
 gnu/packages/emacs-xyz.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 446b25e502..aebe0e5bcf 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -21986,7 +21986,13 @@ definition-jumping and type-checking on demand.")
        `(("emacs-js2-mode" ,emacs-js2-mode)))
       (arguments
        `(#:tests? #t
-         #:test-command '("make" "test")))
+         #:test-command '("make" "test")
+         #:phases (modify-phases %standard-phases
+                    (add-after 'unpack 'strip-Q-option
+                      (lambda _
+                        (substitute* "Makefile"
+                          ((" -Q") ""))
+                        #t)))))
       (home-page "https://github.com/felipeochoa/rjsx-mode")
       (synopsis "Major mode for JSX files")
       (description "This package extends the parser of @code{js2-mode} to
-- 
2.29.2

bug#45316: [PATCH 24/26] gnu: emacs-sudo-edit: Update to commit 0e2c32b, fix tests.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-sudo-edit): Update to commit 0e2c32b.
[arguments]: Use the full blown emacs package, as one test fail otherwise.
[phases]{fix-makefile}: Do not use -Q.
---
 gnu/packages/emacs-xyz.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index aebe0e5bcf..a1e9d6dc9e 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -4833,7 +4833,8 @@ Gnus, e.g., for applying patches received by email.")
            (add-before 'check 'fix-makefile
              (lambda _
                (substitute* "Makefile"
-                 (("\\$\\(CASK\\) exec ") ""))
+                 (("\\$\\(CASK\\) exec ") "")
+                 (("-Q ") ""))
                #t)))
          #:tests? #t
          #:test-command '("make" "test")))
-- 
2.29.2

bug#45316: [PATCH 25/26] gnu: emacs-sly-named-readtables: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-sly-named-readtables)
[phases]{make-autoloads}: Delete phase.
---
 gnu/packages/emacs-xyz.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index a1e9d6dc9e..267b031be8 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -9534,6 +9534,8 @@ additional support for working with ASDF projects.")
       (arguments
        '(#:include (cons* "\\.lisp$" "\\.asd$" %default-include)
          #:phases (modify-phases %standard-phases
+                    ;; The package includes its own autoloads file.
+                    (delete 'make-autoloads)
                     ;; Byte compilation of the autoload file fails.
                     (delete 'enable-autoloads-compilation))))
       (synopsis "Named-readtables support for SLY")
-- 
2.29.2

bug#45316: [PATCH 26/26] gnu: emacs-realgud: Adjust following emacs-build-system changes.

[Maxim Cournoyer]

* gnu/packages/emacs-xyz.scm (emacs-realgud)
[phases]{strip-no-site-file-option}: New phase.
---
 gnu/packages/emacs-xyz.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 267b031be8..89914f54ab 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -7130,7 +7130,12 @@ after buffer changes.")
                                "(require-relative-list \
 '(\"../../common/run\") \"realgud:\")\n")))
              #t))
-         (add-after 'unpack 'fix-autogen-script
+         (add-after 'unpack 'strip-no-site-file-option
+           (lambda _
+             (substitute* '("configure.ac" "Makefile.am" "test/Makefile.am")
+               (("--no-site-file") ""))
+             #t))
+         (add-after 'strip-no-site-file-option 'fix-autogen-script
            (lambda _
              (substitute* "autogen.sh"
                (("./configure") "sh configure"))
-- 
2.29.2

bug#45316: [WIP PATCH 1/3] profiles: Add hook for Emacs subdirs.

[Leo Prikler]

* guix/profiles.scm (emacs-subdirs): New variable.
(%default-profile-hooks): Add it here.
* guix/status.scm (hook-message): Add a message for emacs-subdirs.
---
 guix/profiles.scm | 41 +++++++++++++++++++++++++++++++++++++++++
 guix/status.scm   |  2 ++
 2 files changed, 43 insertions(+)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index 67d90532c1..26fe266a61 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -1115,6 +1115,46 @@ MANIFEST.  Single-file bundles are required by programs such as Git and Lynx."
                     `((type . profile-hook)
                       (hook . ca-certificate-bundle))))
 
+(define (emacs-subdirs manifest)
+  (define build
+    (with-imported-modules (source-module-closure
+                            '((guix build profiles)
+                              (guix build utils)))
+      #~(begin
+          (use-modules (guix build utils)
+                       (guix build profiles)
+                       (ice-9 ftw) ; scandir
+                       (srfi srfi-1) ; append-map
+                       (srfi srfi-26))
+
+          (let ((destdir (string-append #$output "/share/emacs/site-lisp"))
+                (subdirs
+                 (append-map
+                  (lambda (dir)
+                    (filter
+                     file-is-directory?
+                     (map (cute string-append dir "/" <>)
+                          (scandir dir (negate (cute member <> '("." "..")))))))
+                  (filter file-exists?
+                          (map (cute string-append <> "/share/emacs/site-lisp")
+                               '#$(manifest-inputs manifest))))))
+            (mkdir-p destdir)
+            (with-directory-excursion destdir
+              (call-with-output-file "subdirs.el"
+                (lambda (port)
+                  (write
+                   `(normal-top-level-add-to-load-path
+                     (list ,@subdirs))
+                   port)
+                  (newline port)
+                  #t)))))))
+  (gexp->derivation "emacs-subdirs" build
+                    #:local-build? #t
+                    #:substitutable? #f
+                    #:properties
+                    `((type . profile-hook)
+                      (hook . emacs-subdirs))))
+
 (define (glib-schemas manifest)
   "Return a derivation that unions all schemas from manifest entries and
 creates the Glib 'gschemas.compiled' file."
@@ -1672,6 +1712,7 @@ MANIFEST."
         fonts-dir-file
         ghc-package-cache-file
         ca-certificate-bundle
+        emacs-subdirs
         glib-schemas
         gtk-icon-themes
         gtk-im-modules
diff --git a/guix/status.scm b/guix/status.scm
index d47bf1700c..ccde3d3063 100644
--- a/guix/status.scm
+++ b/guix/status.scm
@@ -379,6 +379,8 @@ the current build phase."
      (G_ "building GHC package cache..."))
     ('ca-certificate-bundle
      (G_ "building CA certificate bundle..."))
+    ('emacs-subdirs
+     (G_ "listing Emacs subdirs..."))
     ('glib-schemas
      (G_ "generating GLib schema cache..."))
     ('gtk-icon-themes
-- 
2.31.0

bug#45316: [WIP PATCH 2/3] build-system: emacs: Use subdirectories again.

[Leo Prikler]

With this, Emacs libraries are installed in the ELPA_NAME-VERSION subdirectory
of site-lisp and potential subdirectories should no longer collide.

* guix/build/emacs-build-system.scm (build, patch-el-files, make-autoloads):
Use ELPA name and version to construct subdirectories of %install-dir.
(install): Install in subdirectory.  Also create a subdirs.el, so that
emacs-build-system can find the library without needing to hack EMACSLOADPATH.
---
 guix/build/emacs-build-system.scm | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)

diff --git a/guix/build/emacs-build-system.scm b/guix/build/emacs-build-system.scm
index 26ea59bc25..adc4861f02 100644
--- a/guix/build/emacs-build-system.scm
+++ b/guix/build/emacs-build-system.scm
@@ -96,10 +96,11 @@ environment variable\n" source-directory)))
   "Compile .el files."
   (let* ((emacs (string-append (assoc-ref inputs "emacs") "/bin/emacs"))
          (out (assoc-ref outputs "out"))
-         (site-lisp (string-append out %install-dir)))
+         (elpa-name-ver (store-directory->elpa-name-version out))
+         (el-dir (string-append out %install-dir "/" elpa-name-ver)))
     (setenv "SHELL" "sh")
     (parameterize ((%emacs emacs))
-      (emacs-byte-compile-directory site-lisp))))
+      (emacs-byte-compile-directory el-dir))))
 
 (define* (patch-el-files #:key outputs #:allow-other-keys)
   "Substitute the absolute \"/bin/\" directory with the right location in the
@@ -116,7 +117,8 @@ store in '.el' files."
       #:binary #t))
 
   (let* ((out (assoc-ref outputs "out"))
-         (site-lisp (string-append out %install-dir))
+         (elpa-name-ver (store-directory->elpa-name-version out))
+         (el-dir (string-append out %install-dir "/" elpa-name-ver))
          ;; (ice-9 regex) uses libc's regexp routines, which cannot deal with
          ;; strings containing NULs.  Filter out such files.  TODO: Remove
          ;; this workaround when <https://bugs.gnu.org/30116> is fixed.
@@ -130,7 +132,7 @@ store in '.el' files."
              (error "patch-el-files: unable to locate " cmd-name))
            (string-append "\"" cmd "\"")))))
 
-    (with-directory-excursion site-lisp
+    (with-directory-excursion el-dir
       ;; Some old '.el' files (e.g., tex-buf.el in AUCTeX) are still
       ;; ISO-8859-1-encoded.
       (unless (false-if-exception (substitute-program-names))
@@ -182,16 +184,22 @@ parallel. PARALLEL-TESTS? is ignored when using a non-make TEST-COMMAND."
 
   (let* ((out (assoc-ref outputs "out"))
          (site-lisp (string-append out %install-dir))
+         (elpa-name-ver (store-directory->elpa-name-version out))
+         (el-dir (string-append site-lisp "/" elpa-name-ver))
          (files-to-install (find-files source install-file?)))
     (cond
      ((not (null? files-to-install))
       (for-each
        (lambda (file)
          (let* ((stripped-file (string-drop file (string-length source)))
-                (target-file (string-append site-lisp stripped-file)))
+                (target-file (string-append el-dir stripped-file)))
            (format #t "`~a' -> `~a'~%" file target-file)
            (install-file file (dirname target-file))))
        files-to-install)
+      (call-with-output-file (string-append site-lisp "/subdirs.el")
+        (lambda (port)
+          (write `(normal-top-level-add-to-load-path (list ,el-dir)) port)
+          (newline port)))
       #t)
      (else
       (format #t "error: No files found to install.\n")
@@ -219,11 +227,11 @@ parallel. PARALLEL-TESTS? is ignored when using a non-make TEST-COMMAND."
   "Generate the autoloads file."
   (let* ((emacs (string-append (assoc-ref inputs "emacs") "/bin/emacs"))
          (out (assoc-ref outputs "out"))
-         (site-lisp (string-append out %install-dir))
          (elpa-name-ver (store-directory->elpa-name-version out))
-         (elpa-name (package-name->name+version elpa-name-ver)))
+         (elpa-name (package-name->name+version elpa-name-ver))
+         (el-dir (string-append out %install-dir "/" elpa-name-ver)))
     (parameterize ((%emacs emacs))
-      (emacs-generate-autoloads elpa-name site-lisp))))
+      (emacs-generate-autoloads elpa-name el-dir))))
 
 (define* (enable-autoloads-compilation #:key outputs #:allow-other-keys)
   "Remove the NO-BYTE-COMPILATION local variable embedded in the generated
-- 
2.31.0

bug#45316: [WIP PATCH 3/3] gnu: emacs-libgit: Adjust to changes in emacs-build-system.

[Leo Prikler]

* gnu/packages/emacs-xyz.scm (emacs-libgit)[set-libgit--module-file]: Adjust
libgit--module-file path.
---
 gnu/packages/emacs-xyz.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index f45507c647..3712429651 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -500,7 +500,8 @@ on stdout instead of using a socket as the Emacsclient does.")
                  (make-file-writable "libgit.el")
                  (emacs-substitute-variables "libgit.el"
                    ("libgit--module-file"
-                    (string-append out "/share/emacs/site-lisp/libegit2.so")))
+                    (string-append out "/share/emacs/site-lisp/"
+                                   "libgit-" ,version "/libegit2.so")))
                  #t)))
            (add-before 'install 'prepare-for-install
              (lambda _
-- 
2.31.0

[bug#45359] [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

Hello,

I've goofed and sent this to guix-bugs, apparently, so I'm forwarding it
to guix-patches to make sure everybody possibly interested had a chance
to see it.

Thank you,

Maxim

-------------------- Start of forwarded message --------------------
Subject: bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.
To: 45316@debbugs.gnu.org
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 18 Dec 2020 17:00:10 -0500


[-- Attachment #2.1: Type: text/plain, Size: 2631 bytes --]

Hello Guix!

tl;dr: The Emacs build system and site-start.el loader are modified so
that Emacs packages are installed in their own distinct installation
directory.

The Emacs packages built with the Emacs built system used to be
installed in a sub-directory under the share/emacs/guix.d/ directory,
but this was changed in commit 65a7dd2950ca13a8b942b2836260a2192351b271
shortly after having accommodated the site-start.el machinery to enable
loading packages from any profile (via the EMACSLOADPATH search path
specification).

While this change allowed to expose simply and directly the packages
found in EMACSLOADPATH, it does introduce the risk of file name
collisions when multiple Emacs packages are joined in the same profile,
especially with Emacs packages increasing in complexity (e.g., using
more than a single .el file!) and expecting to have both their sources
and resources extracted under their own nested directory rather than as
a flat collection (ELPA, MELPA).

One recent example I stumbled on was attempting to use the
emacs-yasnippet-snippets package along with emacs-elpy; both wanted to
install a 'snippets' directory to share/emacs/site-lisp/snippets,
collided and resulted in problems that prove difficult to understand.

This is what motivated this patch series, where the site-start.el
auxiliary code used for package discovery is extended to support
packages installed in their own directory under a 'share/emacs/guix'
installation prefix, via Emacs' own package library!

The emacs-build-system is updated for this new installation prefix, as
well as existing packages and documentation.  Parting with a directly
usable EMACSLOADPATH means that site-start.el *must* run for packages to
appear in the load-path; that means for running a test suite, the -Q or
--quick Emacs options cannot be used, since it implies --no-site-file.

Benefits of using this approach:

+ Avoid inter-package file name collisions.

+ Better integration with user installed packages via M-x
package-install.  The Guix-installed packages are listed in M-x
package-list as 'external'.

Cons include:

- Slightly more complex loader (although much of it is offloaded to
  package.el), thus slightly slower (see the comparison below).

- Requires to ensure every package's test suite doesn't make use of -Q.

In my opinion the benefits outweighs the cons by a comfortable margin,
especially with the boring work of adapting the package collection
already done.

To test the performance of the new approach, the following manifest file
was used to test the rebuild of the ~900 Emacs packages making use of
the Emacs build system:


[-- Attachment #2.2: emacs-packages-manifest.scm --]
[-- Type: text/plain, Size: 806 bytes --]

(use-modules (gnu packages)
             (guix build-system)
             (guix packages)
             (srfi srfi-1))

(define %broken-emacs-packages
  (map specification->package
       '("emacs-picpocket"              ;tests fail
         "emacs-twittering-mode"        ;build fails

         ;; Broken only on current master, without new changes.
         "emacs-md4rd"
         "emacs-el-patch"
         "emacs-flymake-shellcheck"
         )))

(define %emacs-packages
  (fold-packages (lambda (package lst)
                   (if (eq? (build-system-name (package-build-system package))
                            'emacs)
                       (cons package lst)
                       lst))
                 '()))

(packages->manifest
 (lset-difference eqv? %emacs-packages %broken-emacs-packages))

[-- Attachment #2.3: Type: text/plain, Size: 2772 bytes --]


A simple benchmark testing the performance of the activation of the
hundreds of Emacs packages was then run using:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix environment --pure -m emacs-packages-manifest.scm \
                                  --ad-hoc emacs
[env]$ /run/setuid-programs/sudo /bin/sh -c 'echo 3 > /proc/sys/vm/drop_caches'
[env]$ emacs --batch --no-site-file \
    --eval="(progn (require 'guix-emacs) \
                   (require 'benchmark) \
                   (message \"(total gc-count gc-time) = %s\" \
                            (benchmark-run 1 (guix-emacs-autoload-packages))))"
--8<---------------cut here---------------end--------------->8---

On the master branch:

--8<---------------cut here---------------start------------->8---
[...]
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/zotxt-autoloads...
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/zoutline-autoloads...
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/ztree-autoloads...
(total gc-count gc-time) = (25.242400751 13 0.189669369)
--8<---------------cut here---------------end--------------->8---

Or about 0.65 s on a warm cache.

On a branch with these changes:

--8<---------------cut here---------------start------------->8---
Error loading autoloads: (file-missing Cannot open load file No such file or directory kotl/kotl-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory helm-easymenu)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/flycheck-cpplint-0.1-1.1d8a090/flycheck-cpplint-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/evil-anzu-0.03/evil-anzu-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/erc-image-0-3.82fb387/erc-image-autoloads)
ad-handle-definition: `ido-completing-read' got redefined
Error loading autoloads: (file-missing Cannot open load file No such file or directory tex-site)
(total gc-count gc-time) = (26.175704339 47 0.783184412)
--8<---------------cut here---------------end--------------->8---

Or about 3 seconds on a warm cache.

There a 6 errors that would need to be looked into, but I these look
like actual packaging problems rather than new issues.  The previously
used way to load the autoloads, '(load f 'noerror)' would have masked
them.

Thanks,

Maxim

[-- Attachment #3: Type: text/plain, Size: 67 bytes --]

-------------------- End of forwarded message --------------------

bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Leo Prikler]

Hello Maxim,

As someone, who lobbied for the current status quo, I have some
thoughts to share.

Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer:
> The Emacs packages built with the Emacs built system used to be
> installed in a sub-directory under the share/emacs/guix.d/ directory,
> but this was changed in commit
> 65a7dd2950ca13a8b942b2836260a2192351b271
> shortly after having accommodated the site-start.el machinery to
> enable
> loading packages from any profile (via the EMACSLOADPATH search path
> specification).
Won't this reintroduce <https://bugs.gnu.org/38309> then?

> While this change allowed to expose simply and directly the packages
> found in EMACSLOADPATH, it does introduce the risk of file name
> collisions when multiple Emacs packages are joined in the same
> profile,
> especially with Emacs packages increasing in complexity (e.g., using
> more than a single .el file!) and expecting to have both their
> sources
> and resources extracted under their own nested directory rather than
> as
> a flat collection (ELPA, MELPA).

> One recent example I stumbled on was attempting to use the
> emacs-yasnippet-snippets package along with emacs-elpy; both wanted
> to
> install a 'snippets' directory to share/emacs/site-lisp/snippets,
> collided and resulted in problems that prove difficult to understand.
I believe that to be a problem in those packages.  Data should not be
installed into share/emacs/site-lisp, but share/emacs/etc.  See for
instance also emacs-telega, which – while not quite adhering to the
above – installs its data in share/emacs/{telega-contrib,telega-data}.

Regardless of what you intend to do with site-lisp otherwise, data
files should *not*, I repeat *not* be installed there.  I do believe
standardizing share/emacs/etc is the way to go, however.

> This is what motivated this patch series, where the site-start.el
> auxiliary code used for package discovery is extended to support
> packages installed in their own directory under a 'share/emacs/guix'
> installation prefix, via Emacs' own package library!
> 
> The emacs-build-system is updated for this new installation prefix,
> as
> well as existing packages and documentation.  Parting with a directly
> usable EMACSLOADPATH means that site-start.el *must* run for packages
> to
> appear in the load-path; that means for running a test suite, the -Q
> or
> --quick Emacs options cannot be used, since it implies --no-site-
> file.
Having no usable EMACSLOADPATH sounds like a dealbreaker to me.  Could
one be restored by using direct subdirectories to share/emacs/site-lisp 
or would that merely be a cosmetic change?

> + Avoid inter-package file name collisions.
Note: This regards data, which should not be stored in site-lisp to
begin with.  You don't get any benefits doing this for code, because
Emacs has no strong module system.
> + Better integration with user installed packages via M-x
> package-install.  The Guix-installed packages are listed in M-x
> package-list as 'external'.
That does sounds like a genuine pro to me.
> - Slightly more complex loader (although much of it is offloaded to
>   package.el), thus slightly slower (see the comparison below).
That should be bearable.
> - Requires to ensure every package's test suite doesn't make use of
> -Q.
That not so much.  Note, that test cases are not the only use for -Q!

> In my opinion the benefits outweighs the cons by a comfortable
> margin,
> especially with the boring work of adapting the package collection
> already done.
> 
> To test the performance of the new approach, the following manifest
> file
> was used to test the rebuild of the ~900 Emacs packages making use of
> the Emacs build system:
> 
> A simple benchmark testing the performance of the activation of the
> hundreds of Emacs packages was then run using:
> 
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix environment --pure -m emacs-packages-
> manifest.scm \
>                                   --ad-hoc emacs
> [env]$ /run/setuid-programs/sudo /bin/sh -c 'echo 3 >
> /proc/sys/vm/drop_caches'
> [env]$ emacs --batch --no-site-file \
>     --eval="(progn (require 'guix-emacs) \
>                    (require 'benchmark) \
>                    (message \"(total gc-count gc-time) = %s\" \
>                             (benchmark-run 1 (guix-emacs-autoload-
> packages))))"
> --8<---------------cut here---------------end--------------->8---
> 
> On the master branch:
> 
> --8<---------------cut here---------------start------------->8---
> [...]
> Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-
> profile/share/emacs/site-lisp/zotxt-autoloads...
> Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-
> profile/share/emacs/site-lisp/zoutline-autoloads...
> Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-
> profile/share/emacs/site-lisp/ztree-autoloads...
> (total gc-count gc-time) = (25.242400751 13 0.189669369)
> --8<---------------cut here---------------end--------------->8---
> 
> Or about 0.65 s on a warm cache.
> 
> On a branch with these changes:
> 
> --8<---------------cut here---------------start------------->8---
> Error loading autoloads: (file-missing Cannot open load file No such
> file or directory kotl/kotl-autoloads)
> Error loading autoloads: (file-missing Cannot open load file No such
> file or directory helm-easymenu)
> Error loading autoloads: (file-missing Cannot open load file No such
> file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-
> profile/share/emacs/site-lisp/guix/flycheck-cpplint-0.1-
> 1.1d8a090/flycheck-cpplint-autoloads)
> Error loading autoloads: (file-missing Cannot open load file No such
> file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-
> profile/share/emacs/site-lisp/guix/evil-anzu-0.03/evil-anzu-
> autoloads)
> Error loading autoloads: (file-missing Cannot open load file No such
> file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-
> profile/share/emacs/site-lisp/guix/erc-image-0-3.82fb387/erc-image-
> autoloads)
> ad-handle-definition: `ido-completing-read' got redefined
> Error loading autoloads: (file-missing Cannot open load file No such
> file or directory tex-site)
> (total gc-count gc-time) = (26.175704339 47 0.783184412)
> --8<---------------cut here---------------end--------------->8---
> 
> Or about 3 seconds on a warm cache.
Looking at these, total does not seem to have changed much, but gc-
count and gc-time more than tripled.  Any idea on how to mitigate this
or do we have to live with that?

> There a 6 errors that would need to be looked into, but I these look
> like actual packaging problems rather than new issues.  The
> previously
> used way to load the autoloads, '(load f 'noerror)' would have masked
> them.
Regardless of how we handle site-lisp going forward, please do look
into those issues and perhaps file them against the individual packages
if they also cause (other) trouble using the current setup.

Regards,
Leo

bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Maxim Cournoyer]

Hi Leo,

Leo Prikler <leo.prikler@student.tugraz.at> writes:

> Hello Maxim,
>
> As someone, who lobbied for the current status quo, I have some
> thoughts to share.

I'm happy you're commenting :-).

> Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer:
>> The Emacs packages built with the Emacs built system used to be
>> installed in a sub-directory under the share/emacs/guix.d/ directory,
>> but this was changed in commit
>> 65a7dd2950ca13a8b942b2836260a2192351b271
>> shortly after having accommodated the site-start.el machinery to
>> enable
>> loading packages from any profile (via the EMACSLOADPATH search path
>> specification).
> Won't this reintroduce <https://bugs.gnu.org/38309> then?

This bug was caused by the EMACSLOADPATH environment variable being too
long.  This was because the original search path specification was
recursively adding any directories under site-lisp/ so that package
installed under a guix.d sub-directory could were directly added to the
load path.

In the current situation, the EMACSLOADPATH contains only the site-lisp/
directory of any profile, plus the versioned lisp/ directory of the
installed Emacs package.  This patch series doesn't change that, so no,
that bug wouldn't be reintroduced.

>> While this change allowed to expose simply and directly the packages
>> found in EMACSLOADPATH, it does introduce the risk of file name
>> collisions when multiple Emacs packages are joined in the same
>> profile,
>> especially with Emacs packages increasing in complexity (e.g., using
>> more than a single .el file!) and expecting to have both their
>> sources
>> and resources extracted under their own nested directory rather than
>> as
>> a flat collection (ELPA, MELPA).
>
>> One recent example I stumbled on was attempting to use the
>> emacs-yasnippet-snippets package along with emacs-elpy; both wanted
>> to
>> install a 'snippets' directory to share/emacs/site-lisp/snippets,
>> collided and resulted in problems that prove difficult to understand.

> I believe that to be a problem in those packages.  Data should not be
> installed into share/emacs/site-lisp, but share/emacs/etc.  See for
> instance also emacs-telega, which – while not quite adhering to the
> above – installs its data in share/emacs/{telega-contrib,telega-data}.
>
> Regardless of what you intend to do with site-lisp otherwise, data
> files should *not*, I repeat *not* be installed there.  I do believe
> standardizing share/emacs/etc is the way to go, however.

While I agree that it would be more elegant the way you propose, that's
not the way Emacs packages have been standardized.  So going against the
single "content directory" (c.f. info "(elisp) Packaging Basics") would
break many Elisp library assumptions about where there files are,
causing more friction (thus work) to adapt them in Guix.  The content
directory of an Elisp package can contain any kind of files (code,
images, etc.), according to info "(elisp) Multi-file Packages".

>> This is what motivated this patch series, where the site-start.el
>> auxiliary code used for package discovery is extended to support
>> packages installed in their own directory under a 'share/emacs/guix'
>> installation prefix, via Emacs' own package library!
>>
>> The emacs-build-system is updated for this new installation prefix,
>> as
>> well as existing packages and documentation.  Parting with a directly
>> usable EMACSLOADPATH means that site-start.el *must* run for packages
>> to
>> appear in the load-path; that means for running a test suite, the -Q
>> or
>> --quick Emacs options cannot be used, since it implies --no-site-
>> file.
> Having no usable EMACSLOADPATH sounds like a dealbreaker to me.  Could
> one be restored by using direct subdirectories to share/emacs/site-lisp
> or would that merely be a cosmetic change?

We have two schemes to accommodate for our Emacs packages:

1. Those installed via their own mean, e.g. make install, using the
gnu-build-system for example.  These would still typically install their
packages directly under site-lisp, possibly multiple files (that could
still collide).

2. Those installed via the emacs-build-system.  With the proposed
changes, those now go to site-lisp/guix/.  The 'guix' sub-directory
makes it unambiguous that anything found under is to be loaded by
package.el; the `package-directory-list' variable can be pointed to it
to have the Emacs' package library discover these self-contained
packages.

>> + Avoid inter-package file name collisions.
> Note: This regards data, which should not be stored in site-lisp to
> begin with.  You don't get any benefits doing this for code, because
> Emacs has no strong module system.

True.

>> + Better integration with user installed packages via M-x
>> package-install.  The Guix-installed packages are listed in M-x
>> package-list as 'external'.
> That does sounds like a genuine pro to me.
>> - Slightly more complex loader (although much of it is offloaded to
>>   package.el), thus slightly slower (see the comparison below).
> That should be bearable.
>> - Requires to ensure every package's test suite doesn't make use of
>> -Q.
> That not so much.  Note, that test cases are not the only use for -Q!

Currently if you use -Q, the Elisp libraries are in the load-path, but
not loaded (you need manually do M-x load-library before you can use
it), or call M-x guix-emacs-autoload-packages to load their autoloads.
For programs, this mean (require 'some-library) works, but M-x
some-library-autoloaded-function doesn't.

After this change, if you use -Q the new style Emacs packages are not
visible at all until you activate them with 'M-x load-library
guix-emacs' then 'M-x guix-package-initialize', or (require 'guix-emacs)
then (guix-package-initialize), programmatically.

I don't see this as a problem.  -Q is simply an alias for
"--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources"
"--no-site-file".

>> In my opinion the benefits outweighs the cons by a comfortable
>> margin,
>> especially with the boring work of adapting the package collection
>> already done.
>>
>> To test the performance of the new approach, the following manifest
>> file
>> was used to test the rebuild of the ~900 Emacs packages making use of
>> the Emacs build system:
>>
>> A simple benchmark testing the performance of the activation of the
>> hundreds of Emacs packages was then run using:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ ./pre-inst-env guix environment --pure -m emacs-packages-
>> manifest.scm \
>>                                   --ad-hoc emacs
>> [env]$ /run/setuid-programs/sudo /bin/sh -c 'echo 3 >
>> /proc/sys/vm/drop_caches'
>> [env]$ emacs --batch --no-site-file \
>>     --eval="(progn (require 'guix-emacs) \
>>                    (require 'benchmark) \
>>                    (message \"(total gc-count gc-time) = %s\" \
>>                             (benchmark-run 1 (guix-emacs-autoload-
>> packages))))"
>> --8<---------------cut here---------------end--------------->8---
>>
>> On the master branch:
>>
>> --8<---------------cut here---------------start------------->8---
>> [...]
>> Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-
>> profile/share/emacs/site-lisp/zotxt-autoloads...
>> Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-
>> profile/share/emacs/site-lisp/zoutline-autoloads...
>> Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-
>> profile/share/emacs/site-lisp/ztree-autoloads...
>> (total gc-count gc-time) = (25.242400751 13 0.189669369)
>> --8<---------------cut here---------------end--------------->8---
>>
>> Or about 0.65 s on a warm cache.
>>
>> On a branch with these changes:
>>
>> --8<---------------cut here---------------start------------->8---
>> Error loading autoloads: (file-missing Cannot open load file No such
>> file or directory kotl/kotl-autoloads)
>> Error loading autoloads: (file-missing Cannot open load file No such
>> file or directory helm-easymenu)
>> Error loading autoloads: (file-missing Cannot open load file No such
>> file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-
>> profile/share/emacs/site-lisp/guix/flycheck-cpplint-0.1-
>> 1.1d8a090/flycheck-cpplint-autoloads)
>> Error loading autoloads: (file-missing Cannot open load file No such
>> file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-
>> profile/share/emacs/site-lisp/guix/evil-anzu-0.03/evil-anzu-
>> autoloads)
>> Error loading autoloads: (file-missing Cannot open load file No such
>> file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-
>> profile/share/emacs/site-lisp/guix/erc-image-0-3.82fb387/erc-image-
>> autoloads)
>> ad-handle-definition: `ido-completing-read' got redefined
>> Error loading autoloads: (file-missing Cannot open load file No such
>> file or directory tex-site)
>> (total gc-count gc-time) = (26.175704339 47 0.783184412)
>> --8<---------------cut here---------------end--------------->8---
>>
>> Or about 3 seconds on a warm cache.
> Looking at these, total does not seem to have changed much, but gc-
> count and gc-time more than tripled.  Any idea on how to mitigate this
> or do we have to live with that?

I don't know enough about Elisp to offer a guess.  We'd have to profile
it.  When put in perspective, that's 3 seconds to refresh about 900
packages.  I'm guessing most users must have less than 100 Emacs
packages in their collections (I have about 35). I think the current
performance is good enough to not worry optimizing it at this stage.  In
any case, performance optimizations would need to be made to Emacs'
package.el.

>> There a 6 errors that would need to be looked into, but I these look
>> like actual packaging problems rather than new issues.  The
>> previously
>> used way to load the autoloads, '(load f 'noerror)' would have masked
>> them.

> Regardless of how we handle site-lisp going forward, please do look
> into those issues and perhaps file them against the individual packages
> if they also cause (other) trouble using the current setup.

I'll try having a look, thanks for the reminder.

Thank you for commenting!

Maxim

bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Leo Prikler]

Hi Maxim,

Am Dienstag, den 22.12.2020, 13:09 -0500 schrieb Maxim Cournoyer:
> Hi Leo,
> 
> Leo Prikler <leo.prikler@student.tugraz.at> writes:
> 
> > Hello Maxim,
> > 
> > As someone, who lobbied for the current status quo, I have some
> > thoughts to share.
> 
> I'm happy you're commenting :-).
> 
> > Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer:
> > > The Emacs packages built with the Emacs built system used to be
> > > installed in a sub-directory under the share/emacs/guix.d/
> > > directory,
> > > but this was changed in commit
> > > 65a7dd2950ca13a8b942b2836260a2192351b271
> > > shortly after having accommodated the site-start.el machinery to
> > > enable
> > > loading packages from any profile (via the EMACSLOADPATH search
> > > path
> > > specification).
> > Won't this reintroduce <https://bugs.gnu.org/38309> then?
> 
> This bug was caused by the EMACSLOADPATH environment variable being
> too
> long.  This was because the original search path specification was
> recursively adding any directories under site-lisp/ so that package
> installed under a guix.d sub-directory could were directly added to
> the
> load path.
> 
> In the current situation, the EMACSLOADPATH contains only the site-
> lisp/
> directory of any profile, plus the versioned lisp/ directory of the
> installed Emacs package.  This patch series doesn't change that, so
> no,
> that bug wouldn't be reintroduced.
Which is also the reason why our startup code changes and people have
to accommodate to that, right?

> > > While this change allowed to expose simply and directly the
> > > packages
> > > found in EMACSLOADPATH, it does introduce the risk of file name
> > > collisions when multiple Emacs packages are joined in the same
> > > profile,
> > > especially with Emacs packages increasing in complexity (e.g.,
> > > using
> > > more than a single .el file!) and expecting to have both their
> > > sources
> > > and resources extracted under their own nested directory rather
> > > than
> > > as
> > > a flat collection (ELPA, MELPA).
> > > One recent example I stumbled on was attempting to use the
> > > emacs-yasnippet-snippets package along with emacs-elpy; both
> > > wanted
> > > to
> > > install a 'snippets' directory to share/emacs/site-lisp/snippets,
> > > collided and resulted in problems that prove difficult to
> > > understand.
> > I believe that to be a problem in those packages.  Data should not
> > be
> > installed into share/emacs/site-lisp, but share/emacs/etc.  See for
> > instance also emacs-telega, which – while not quite adhering to the
> > above – installs its data in share/emacs/{telega-contrib,telega-
> > data}.
> > 
> > Regardless of what you intend to do with site-lisp otherwise, data
> > files should *not*, I repeat *not* be installed there.  I do
> > believe
> > standardizing share/emacs/etc is the way to go, however.
> 
> While I agree that it would be more elegant the way you propose,
> that's
> not the way Emacs packages have been standardized.  So going against
> the
> single "content directory" (c.f. info "(elisp) Packaging Basics")
> would
> break many Elisp library assumptions about where there files are,
> causing more friction (thus work) to adapt them in Guix.  The content
> directory of an Elisp package can contain any kind of files (code,
> images, etc.), according to info "(elisp) Multi-file Packages".

I suppose said manual is perhaps slightly outdated.
> A package is either a “simple package” or a “multi-file package”.  A
> simple package is stored in a package archive as a single Emacs Lisp
> file, while a multi-file package is stored as a tar file (containing
> multiple Lisp files, and possibly non-Lisp files such as a manual).
When was the last time, you've installed a tar to site-lisp?  I also
feel as though the expected contents are limited very much to README,
COPYING and the like:

--8<---------------cut here--------------begin-------------->8---
$ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \
  -not -name '*.el' \
  -and -not -name '*.elc' \
  -and -not -name '*.el.gz'
~/.guix-profile/share/emacs/27.1/lisp/term/README
~/.guix-profile/share/emacs/27.1/lisp/COPYING
~/.guix-profile/share/emacs/27.1/lisp/README
--8<---------------cut here---------------end--------------->8---

Perhaps we should ask Emacs folks how they feel about separating code
and data, but bugs have already been caused here and elsewhere by
stuffing them together.  (I believe yasnippet was once already the
culprit at some point before it got reorganized.)  In the meantime,
yes, doing so might cause extra work, but
1) it only affects a subset of packages, and
2) of this subset, we can prioritize those, that do exhibit this
behaviour.

> > > This is what motivated this patch series, where the site-start.el
> > > auxiliary code used for package discovery is extended to support
> > > packages installed in their own directory under a
> > > 'share/emacs/guix'
> > > installation prefix, via Emacs' own package library!
> > > 
> > > The emacs-build-system is updated for this new installation
> > > prefix,
> > > as
> > > well as existing packages and documentation.  Parting with a
> > > directly
> > > usable EMACSLOADPATH means that site-start.el *must* run for
> > > packages
> > > to
> > > appear in the load-path; that means for running a test suite, the
> > > -Q
> > > or
> > > --quick Emacs options cannot be used, since it implies --no-site-
> > > file.
> > Having no usable EMACSLOADPATH sounds like a dealbreaker to
> > me.  Could
> > one be restored by using direct subdirectories to share/emacs/site-
> > lisp
> > or would that merely be a cosmetic change?
> 
> We have two schemes to accommodate for our Emacs packages:
> 
> 1. Those installed via their own mean, e.g. make install, using the
> gnu-build-system for example.  These would still typically install
> their
> packages directly under site-lisp, possibly multiple files (that
> could
> still collide).
Why?  Seems inconsistent, does it not?

> 2. Those installed via the emacs-build-system.  With the proposed
> changes, those now go to site-lisp/guix/.  The 'guix' sub-directory
> makes it unambiguous that anything found under is to be loaded by
> package.el; the `package-directory-list' variable can be pointed to
> it
> to have the Emacs' package library discover these self-contained
> packages.
I think you've lost me here.  Basically, instead of being able to
`(require 'my-package)` you first need to load my-package through
package.el and then can `(require 'my-package)`?  I am not sure, what
the benefit of that would be, if any. 

> Currently if you use -Q, the Elisp libraries are in the load-path,
> but
> not loaded (you need manually do M-x load-library before you can use
> it), or call M-x guix-emacs-autoload-packages to load their
> autoloads.
> For programs, this mean (require 'some-library) works, but M-x
> some-library-autoloaded-function doesn't.
> 
> After this change, if you use -Q the new style Emacs packages are not
> visible at all until you activate them with 'M-x load-library
> guix-emacs' then 'M-x guix-package-initialize', or (require 'guix-
> emacs)
> then (guix-package-initialize), programmatically.
I do think, that there are legitimate reasons to not require a full
initialization here, particularly for the use in batch scripts, but I'm
not quite sure how much work we currently put into making sure that
everything is available.

> I don't see this as a problem.  -Q is simply an alias for
> "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources"
> "--no-site-file".
But not "--no-load-path" "--no-nothing" ;)
Being able to load the same libraries as without is vital for debugging
and scripts.  (Granted, some distros probably break with --no-site-
lisp, but that's nothing to strive for imo.)

> [...]
> When put in perspective, that's 3 seconds to refresh about 900
> packages.  I'm guessing most users must have less than 100 Emacs
> packages in their collections (I have about 35). I think the current
> performance is good enough to not worry optimizing it at this stage.
Point taken, I don't have that many either.  I was merely interested,
given that I rarely dive that deep into runtime statistics of Elisp
code.

Regards,
Leo

bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Maxim Cournoyer]

Hi Leo!

Leo Prikler <leo.prikler@student.tugraz.at> writes:

> Hi Maxim,
>
> Am Dienstag, den 22.12.2020, 13:09 -0500 schrieb Maxim Cournoyer:
>> Hi Leo,
>>
>> Leo Prikler <leo.prikler@student.tugraz.at> writes:
>>
>> > Hello Maxim,
>> >
>> > As someone, who lobbied for the current status quo, I have some
>> > thoughts to share.
>>
>> I'm happy you're commenting :-).
>>
>> > Am Montag, den 21.12.2020, 22:28 -0500 schrieb Maxim Cournoyer:
>> > > The Emacs packages built with the Emacs built system used to be
>> > > installed in a sub-directory under the share/emacs/guix.d/
>> > > directory,
>> > > but this was changed in commit
>> > > 65a7dd2950ca13a8b942b2836260a2192351b271
>> > > shortly after having accommodated the site-start.el machinery to
>> > > enable
>> > > loading packages from any profile (via the EMACSLOADPATH search
>> > > path
>> > > specification).
>> > Won't this reintroduce <https://bugs.gnu.org/38309> then?
>>
>> This bug was caused by the EMACSLOADPATH environment variable being
>> too
>> long.  This was because the original search path specification was
>> recursively adding any directories under site-lisp/ so that package
>> installed under a guix.d sub-directory could were directly added to
>> the
>> load path.
>>
>> In the current situation, the EMACSLOADPATH contains only the site-
>> lisp/
>> directory of any profile, plus the versioned lisp/ directory of the
>> installed Emacs package.  This patch series doesn't change that, so
>> no,
>> that bug wouldn't be reintroduced.

> Which is also the reason why our startup code changes and people have
> to accommodate to that, right?

Yes. The startup code is now responsible to activate packages found not
directly in the EMACSLOADPATH (share/emacs/site-lisp), but in
sub-directories under share/emacs/site-lisp/guix.

>> > > While this change allowed to expose simply and directly the
>> > > packages
>> > > found in EMACSLOADPATH, it does introduce the risk of file name
>> > > collisions when multiple Emacs packages are joined in the same
>> > > profile,
>> > > especially with Emacs packages increasing in complexity (e.g.,
>> > > using
>> > > more than a single .el file!) and expecting to have both their
>> > > sources
>> > > and resources extracted under their own nested directory rather
>> > > than
>> > > as
>> > > a flat collection (ELPA, MELPA).
>> > > One recent example I stumbled on was attempting to use the
>> > > emacs-yasnippet-snippets package along with emacs-elpy; both
>> > > wanted
>> > > to
>> > > install a 'snippets' directory to share/emacs/site-lisp/snippets,
>> > > collided and resulted in problems that prove difficult to
>> > > understand.
>> > I believe that to be a problem in those packages.  Data should not
>> > be
>> > installed into share/emacs/site-lisp, but share/emacs/etc.  See for
>> > instance also emacs-telega, which – while not quite adhering to the
>> > above – installs its data in share/emacs/{telega-contrib,telega-
>> > data}.
>> >
>> > Regardless of what you intend to do with site-lisp otherwise, data
>> > files should *not*, I repeat *not* be installed there.  I do
>> > believe
>> > standardizing share/emacs/etc is the way to go, however.
>>
>> While I agree that it would be more elegant the way you propose,
>> that's
>> not the way Emacs packages have been standardized.  So going against
>> the
>> sin--8<---------------cut here---------------start------------->8---
--8<---------------cut here---------------start------------->8---
gle "content directory" (c.f. info "(elisp) Packaging Basics")
>> would
>> break many Elisp library assumptions about where there files are,
>> causing more friction (thus work) to adapt them in Guix.  The content
>> directory of an Elisp package can contain any kind of files (code,
>> images, etc.), according to info "(elisp) Multi-file Packages".
>
> I suppose said manual is perhaps slightly outdated.

The doc/package.texi file of the Emacs git repository is actively
maintained, although cited contents above hasn't changed much for the
last 8 years.  In any case if you look into package.el, you'll see that
package contents are fetched as a single archived, and its content
installed to a single directory.

>> A package is either a “simple package” or a “multi-file package”.  A
>> simple package is stored in a package archive as a single Emacs Lisp
>> file, while a multi-file package is stored as a tar file (containing
>> multiple Lisp files, and possibly non-Lisp files such as a manual).
> When was the last time, you've installed a tar to site-lisp?  I also
> feel as though the expected contents are limited very much to README,
> COPYING and the like:
>
> $ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \
>   -not -name '*.el' \
>   -and -not -name '*.elc' \
>   -and -not -name '*.el.gz'
> ~/.guix-profile/share/emacs/27.1/lisp/term/README
> ~/.guix-profile/share/emacs/27.1/lisp/COPYING
> ~/.guix-profile/share/emacs/27.1/lisp/README

The lisp/ directory is for "the standard Lisp files that come with
Emacs" (from info "(elisp)Library Search").  I don't think we can draw
much comparison between these and user-installable packages from ELPA or
MELPA.

Also, just to be clear, the tar is not installed as-is; it is extracted
and its contents are installed in the "content directory".  Citing again
'(elisp) Multi-file Packages':

    Prior to installation, a multi-file package is stored in a package
    archive as a tar file.  The tar file must be named ‘NAME-VERSION.tar’,
    where NAME is the package name and VERSION is the version number.  Its
    contents, once extracted, must all appear in a directory named
    ‘NAME-VERSION’, the “content directory” (*note Packaging Basics::).
    Files may also extract into subdirectories of the content directory.

       One of the files in the content directory must be named
    ‘NAME-pkg.el’.  It must contain a single Lisp form, consisting of a call
    to the function ‘define-package’, described below.  This defines the
    package’s attributes: version, brief description, and requirements.

       For example, if we distribute version 1.3 of the superfrobnicator as
    a multi-file package, the tar file would be ‘superfrobnicator-1.3.tar’.
    Its contents would extract into the directory ‘superfrobnicator-1.3’,
    and one of these would be the file ‘superfrobnicator-pkg.el’.

If you want to have a clearer idea of how packages from ELPA and the
likes are installed, you can have a peek into the 'package.el' file
shipped with Emacs (spoiler: it's basically just extracting the contents
of the package archive to a single directory -- see the `package-unpack'
procedure).

Here's an experiment I've done with a profile containing the whole of
our current emacs-build-system based packages, with this current change,
that checks for collision at the top level of a package installation
directory:

--8<---------------cut here---------------end--------------->8---
$ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-manifest.scm

$  find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2
-mindepth 2 -not -regex '.*\.elc?$' \
          | awk -F '/' '{ if ($9 in packages) \
            {printf("%s directory of %s package collides with that of package %s\n", $9, $8, packages[$9])} \
            else {packages[$9] = $8} }'

doc directory of modus-operandi-theme-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
tools directory of unidecode-0.2-1.5502ada package collides with that of package company-cabal-0.3.0-1.62112a7
snippets directory of minitest-0.8.0-1.1aadb78 package collides with that of package elpy-1.35.0
test directory of realgud-1.5.1 package collides with that of package flycheck-haskell-0.8-2.32ddff8
snippets directory of yasnippet-snippets-0.23 package collides with that of package elpy-1.35.0
snippets directory of haskell-snippets-0.1.0-0.07b0f46 package collides with that of package elpy-1.35.0
snippets directory of rspec-1.11-1.66ea7cc package collides with that of package elpy-1.35.0
doc directory of modus-themes-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
data directory of emojify-1.2 package collides with that of package unidecode-0.2-1.5502ada
test directory of systemd-mode-1.6 package collides with that of package flycheck-haskell-0.8-2.32ddff8
lib directory of slime-2.26.1 package collides with that of package robe-0.8.2
contrib directory of sly-1.0.0-7.68561f1 package collides with that of package slime-2.26.1
lib directory of sly-1.0.0-7.68561f1 package collides with that of package robe-0.8.2
doc directory of modus-vivendi-theme-1.0.2 package collides with that of package racket-mode-0.0.2-6.5eb31a2
doc directory of evil-1.14.0 package collides with that of package racket-mode-0.0.2-6.5eb31a2
data directory of all-the-icons-4.0.1 package collides with that of package unidecode-0.2-1.5502ada
snippets directory of feature-mode-20190801-1.11ae167 package collides with that of package elpy-1.35.0
--8<---------------cut here---------------end--------------->8---

So 17 Emacs packages in Guix currently conflict, and Guix seems to be
silent about it when building a profile with them via 'guix package -p'
(a bug?).

> Perhaps we should ask Emacs folks how they feel about separating code
> and data, but bugs have already been caused here and elsewhere by
> stuffing them together.  (I believe yasnippet was once already the
> culprit at some point before it got reorganized.)  In the meantime,
> yes, doing so might cause extra work, but
> 1) it only affects a subset of packages, and
> 2) of this subset, we can prioritize those, that do exhibit this
> behaviour.

The above does output does show that's it's a subset of the packages
that are affected.

[...]

>> We have two schemes to accommodate for our Emacs packages:
>>
>> 1. Those installed via their own mean, e.g. make install, using the
>> gnu-build-system for example.  These would still typically install
>> their
>> packages directly under site-lisp, possibly multiple files (that
>> could
>> still collide).
> Why?  Seems inconsistent, does it not?

They can be adapted in time; until then it's easy to accommodate both.

>> 2. Those installed via the emacs-build-system.  With the proposed
>> changes, those now go to site-lisp/guix/.  The 'guix' sub-directory
>> makes it unambiguous that anything found under is to be loaded by
>> package.el; the `package-directory-list' variable can be pointed to
>> it
>> to have the Emacs' package library discover these self-contained
>> packages.
> I think you've lost me here.  Basically, instead of being able to
> `(require 'my-package)` you first need to load my-package through
> package.el and then can `(require 'my-package)`?  I am not sure, what
> the benefit of that would be, if any.

Only if you've specified --no-site-start, which would disable
user-installed package discovery.  This is the drawback of the
trade-off, not the benefit :-).

>> Currently if you use -Q, the Elisp libraries are in the load-path,
>> but
>> not loaded (you need manually do M-x load-library before you can use
>> it), or call M-x guix-emacs-autoload-packages to load their
>> autoloads.
>> For programs, this mean (require 'some-library) works, but M-x
>> some-library-autoloaded-function doesn't.
>>
>> After this change, if you use -Q the new style Emacs packages are not
>> visible at all until you activate them with 'M-x load-library
>> guix-emacs' then 'M-x guix-package-initialize', or (require 'guix-
>> emacs)
>> then (guix-package-initialize), programmatically.

> I do think, that there are legitimate reasons to not require a full
> initialization here, particularly for the use in batch scripts, but I'm
> not quite sure how much work we currently put into making sure that
> everything is available.

That's a fair point, although I'm not concerned about the overhead of
loading compiled autoloads files.

>> I don't see this as a problem.  -Q is simply an alias for
>> "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-resources"
>> "--no-site-file".
> But not "--no-load-path" "--no-nothing" ;)

One could argue that the spirit of -Q is to give an Emacs as minimal as
possible (that's what the Emacs manual has to say about it, and the
implicit --no-site-lisp has the meaning of "no user installed
packages").

> Being able to load the same libraries as without is vital for debugging
> and scripts.  (Granted, some distros probably break with --no-site-
> lisp, but that's nothing to strive for imo.)

I think you meant --no-site-file here.  There's not much magic; if you
allow it to run, user installed packages are added to the load path and
autoloaded; if you don't you don't then you only get Emacs standard
libraries.  Nothing else happens in the site-start.el file in Guix
proposed here.  Scripting works well enough that our 900'ish packages
can be rebuilt with the change, many of which come with a test suite.

Thanks for having me think harder about if this is really
desirable/needed.  For me the main plus is that we'd be adhering to the
current standards used for Emacs packaging: everything is unpacked into
a single directory owned by that package, so as long as we include any
needed resources in the #:include argument of the emacs-build-system, it
works as upstream intended it, with no need to patch variables or
anything.  I expect this situation to worsen as Emacs packages tend to
get more complicated, and I don't feel strongly enough about it to go
arguing about Emacs packaging standards on the Emacs mailing list :-).

On the minus side it makes the startup slightly more expensive in terms
of pure scripting, and requires the users to be mindful that
site-start.el evaluation is needed for the user installed packages to be
activated.

Further thoughts?

Thanks again,

Maxim

bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Leo Prikler]

Hi Maxim,

Am Samstag, den 26.12.2020, 00:01 -0500 schrieb Maxim Cournoyer:
> > > > > While this change allowed to expose simply and directly the
> > > > > packages
> > > > > found in EMACSLOADPATH, it does introduce the risk of file
> > > > > name
> > > > > collisions when multiple Emacs packages are joined in the
> > > > > same
> > > > > profile,
> > > > > especially with Emacs packages increasing in complexity
> > > > > (e.g.,
> > > > > using
> > > > > more than a single .el file!) and expecting to have both
> > > > > their
> > > > > sources
> > > > > and resources extracted under their own nested directory
> > > > > rather
> > > > > than
> > > > > as
> > > > > a flat collection (ELPA, MELPA).
> > > > > One recent example I stumbled on was attempting to use the
> > > > > emacs-yasnippet-snippets package along with emacs-elpy; both
> > > > > wanted
> > > > > to
> > > > > install a 'snippets' directory to share/emacs/site-
> > > > > lisp/snippets,
> > > > > collided and resulted in problems that prove difficult to
> > > > > understand.
> > > > I believe that to be a problem in those packages.  Data should
> > > > not
> > > > be
> > > > installed into share/emacs/site-lisp, but share/emacs/etc.  See
> > > > for
> > > > instance also emacs-telega, which – while not quite adhering to
> > > > the
> > > > above – installs its data in share/emacs/{telega-
> > > > contrib,telega-
> > > > data}.
> > > > 
> > > > Regardless of what you intend to do with site-lisp otherwise,
> > > > data
> > > > files should *not*, I repeat *not* be installed there.  I do
> > > > believe
> > > > standardizing share/emacs/etc is the way to go, however.
> > > 
> > > While I agree that it would be more elegant the way you propose,
> > > that's
> > > not the way Emacs packages have been standardized.  So going
> > > against
> > > the
> > > sin--8<---------------cut here---------------start-------------
> > > >8---
> --8<---------------cut here---------------start------------->8---
> gle "content directory" (c.f. info "(elisp) Packaging Basics")
> > > would
> > > break many Elisp library assumptions about where there files are,
> > > causing more friction (thus work) to adapt them in Guix.  The
> > > content
> > > directory of an Elisp package can contain any kind of files
> > > (code,
> > > images, etc.), according to info "(elisp) Multi-file Packages".
> > 
> > I suppose said manual is perhaps slightly outdated.
> 
> The doc/package.texi file of the Emacs git repository is actively
> maintained, although cited contents above hasn't changed much for the
> last 8 years.  In any case if you look into package.el, you'll see
> that
> package contents are fetched as a single archived, and its content
> installed to a single directory.
> 
> > > A package is either a “simple package” or a “multi-file
> > > package”.  A
> > > simple package is stored in a package archive as a single Emacs
> > > Lisp
> > > file, while a multi-file package is stored as a tar file
> > > (containing
> > > multiple Lisp files, and possibly non-Lisp files such as a
> > > manual).
> > When was the last time, you've installed a tar to site-lisp?  I
> > also
> > feel as though the expected contents are limited very much to
> > README,
> > COPYING and the like:
> > 
> > $ find ~/.guix-profile/share/emacs/27.1/lisp/ -type f \
> >   -not -name '*.el' \
> >   -and -not -name '*.elc' \
> >   -and -not -name '*.el.gz'
> > ~/.guix-profile/share/emacs/27.1/lisp/term/README
> > ~/.guix-profile/share/emacs/27.1/lisp/COPYING
> > ~/.guix-profile/share/emacs/27.1/lisp/README
> 
> The lisp/ directory is for "the standard Lisp files that come with
> Emacs" (from info "(elisp)Library Search").  I don't think we can
> draw
> much comparison between these and user-installable packages from ELPA
> or
> MELPA.
Fair enough, perhaps I read too much into that.

> Also, just to be clear, the tar is not installed as-is; it is
> extracted
> and its contents are installed in the "content directory".  Citing
> again
> '(elisp) Multi-file Packages':
> 
>     Prior to installation, a multi-file package is stored in a
> package
>     archive as a tar file.  The tar file must be named ‘NAME-
> VERSION.tar’,
>     where NAME is the package name and VERSION is the version
> number.  Its
>     contents, once extracted, must all appear in a directory named
>     ‘NAME-VERSION’, the “content directory” (*note Packaging
> Basics::).
>     Files may also extract into subdirectories of the content
> directory.
> 
>        One of the files in the content directory must be named
>     ‘NAME-pkg.el’.  It must contain a single Lisp form, consisting of
> a call
>     to the function ‘define-package’, described below.  This defines
> the
>     package’s attributes: version, brief description, and
> requirements.
> 
>        For example, if we distribute version 1.3 of the
> superfrobnicator as
>     a multi-file package, the tar file would be ‘superfrobnicator-
> 1.3.tar’.
>     Its contents would extract into the directory ‘superfrobnicator-
> 1.3’,
>     and one of these would be the file ‘superfrobnicator-pkg.el’.
> 
> If you want to have a clearer idea of how packages from ELPA and the
> likes are installed, you can have a peek into the 'package.el' file
> shipped with Emacs (spoiler: it's basically just extracting the
> contents
> of the package archive to a single directory -- see the `package-
> unpack'
> procedure).
If we follow that to the letter, should we not install those packages
directly to site-lisp/$PACKAGE-$VERSION?  Why the guix directory?

Also, if we do go that route, I think we should install a subdirs.el
through a profile hook, that adds all these package directories to the
load path.  See also '(elisp) Startup Summary'.  Alternatively, we can
let subdirs.el defer to guix-emacs.el, but that'd be very cheeky. 
Either way, we should definitely make -Q work.

> Here's an experiment I've done with a profile containing the whole of
> our current emacs-build-system based packages, with this current
> change,
> that checks for collision at the top level of a package installation
> directory:
> 
> --8<---------------cut here---------------end--------------->8---
> $ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-
> manifest.scm
> 
> $  find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2
> -mindepth 2 -not -regex '.*\.elc?$' \
>           | awk -F '/' '{ if ($9 in packages) \
>             {printf("%s directory of %s package collides with that of
> package %s\n", $9, $8, packages[$9])} \
>             else {packages[$9] = $8} }'
> 
> doc directory of modus-operandi-theme-1.0.2 package collides with
> that of package racket-mode-0.0.2-6.5eb31a2
> tools directory of unidecode-0.2-1.5502ada package collides with that
> of package company-cabal-0.3.0-1.62112a7
> snippets directory of minitest-0.8.0-1.1aadb78 package collides with
> that of package elpy-1.35.0
> test directory of realgud-1.5.1 package collides with that of package
> flycheck-haskell-0.8-2.32ddff8
> snippets directory of yasnippet-snippets-0.23 package collides with
> that of package elpy-1.35.0
> snippets directory of haskell-snippets-0.1.0-0.07b0f46 package
> collides with that of package elpy-1.35.0
> snippets directory of rspec-1.11-1.66ea7cc package collides with that
> of package elpy-1.35.0
> doc directory of modus-themes-1.0.2 package collides with that of
> package racket-mode-0.0.2-6.5eb31a2
> data directory of emojify-1.2 package collides with that of package
> unidecode-0.2-1.5502ada
> test directory of systemd-mode-1.6 package collides with that of
> package flycheck-haskell-0.8-2.32ddff8
> lib directory of slime-2.26.1 package collides with that of package
> robe-0.8.2
> contrib directory of sly-1.0.0-7.68561f1 package collides with that
> of package slime-2.26.1
> lib directory of sly-1.0.0-7.68561f1 package collides with that of
> package robe-0.8.2
> doc directory of modus-vivendi-theme-1.0.2 package collides with that
> of package racket-mode-0.0.2-6.5eb31a2
> doc directory of evil-1.14.0 package collides with that of package
> racket-mode-0.0.2-6.5eb31a2
> data directory of all-the-icons-4.0.1 package collides with that of
> package unidecode-0.2-1.5502ada
> snippets directory of feature-mode-20190801-1.11ae167 package
> collides with that of package elpy-1.35.0
> --8<---------------cut here---------------end--------------->8---
> 
> So 17 Emacs packages in Guix currently conflict, and Guix seems to be
> silent about it when building a profile with them via 'guix package
> -p'
> (a bug?).
It's a bug in those packages; not in Guix.  The whole idea of Guix is,
that such directories are merged.  Now collisions within files in those
directories are a different topic – only one can be chosen there and
IIRC there should also be a way to make them errors.

> > Perhaps we should ask Emacs folks how they feel about separating
> > code
> > and data, but bugs have already been caused here and elsewhere by
> > stuffing them together.  (I believe yasnippet was once already the
> > culprit at some point before it got reorganized.)  In the meantime,
> > yes, doing so might cause extra work, but
> > 1) it only affects a subset of packages, and
> > 2) of this subset, we can prioritize those, that do exhibit this
> > behaviour.
> 
> The above does output does show that's it's a subset of the packages
> that are affected.
> 
> [...]
> 
> > > We have two schemes to accommodate for our Emacs packages:
> > > 
> > > 1. Those installed via their own mean, e.g. make install, using
> > > the
> > > gnu-build-system for example.  These would still typically
> > > install
> > > their
> > > packages directly under site-lisp, possibly multiple files (that
> > > could
> > > still collide).
> > Why?  Seems inconsistent, does it not?
> 
> They can be adapted in time; until then it's easy to accommodate
> both.
I think we should aim for consistency here, but let's adapt them in
follow-up commits.

> > > 2. Those installed via the emacs-build-system.  With the proposed
> > > changes, those now go to site-lisp/guix/.  The 'guix' sub-
> > > directory
> > > makes it unambiguous that anything found under is to be loaded by
> > > package.el; the `package-directory-list' variable can be pointed
> > > to
> > > it
> > > to have the Emacs' package library discover these self-contained
> > > packages.
> > I think you've lost me here.  Basically, instead of being able to
> > `(require 'my-package)` you first need to load my-package through
> > package.el and then can `(require 'my-package)`?  I am not sure,
> > what
> > the benefit of that would be, if any.
> 
> Only if you've specified --no-site-start, which would disable
> user-installed package discovery.  This is the drawback of the
> trade-off, not the benefit :-).
It wasn't worded like one.  The word unambiguous typically has a
positive connotation in my mind.  And package discovery should function
regardless of site-start.

> > > Currently if you use -Q, the Elisp libraries are in the load-
> > > path,
> > > but
> > > not loaded (you need manually do M-x load-library before you can
> > > use
> > > it), or call M-x guix-emacs-autoload-packages to load their
> > > autoloads.
> > > For programs, this mean (require 'some-library) works, but M-x
> > > some-library-autoloaded-function doesn't.
> > > 
> > > After this change, if you use -Q the new style Emacs packages are
> > > not
> > > visible at all until you activate them with 'M-x load-library
> > > guix-emacs' then 'M-x guix-package-initialize', or (require
> > > 'guix-
> > > emacs)
> > > then (guix-package-initialize), programmatically.
> > I do think, that there are legitimate reasons to not require a full
> > initialization here, particularly for the use in batch scripts, but
> > I'm
> > not quite sure how much work we currently put into making sure that
> > everything is available.
> 
> That's a fair point, although I'm not concerned about the overhead of
> loading compiled autoloads files.
> 
> > > I don't see this as a problem.  -Q is simply an alias for
> > > "--no-init-file" "--no-site-lisp" "--no-splash" "--no-x-
> > > resources"
> > > "--no-site-file".
> > But not "--no-load-path" "--no-nothing" ;)
> 
> One could argue that the spirit of -Q is to give an Emacs as minimal
> as
> possible (that's what the Emacs manual has to say about it, and the
> implicit --no-site-lisp has the meaning of "no user installed
> packages").
Well then the spirit of -Q is lost.

> > Being able to load the same libraries as without is vital for
> > debugging
> > and scripts.  (Granted, some distros probably break with --no-site-
> > lisp, but that's nothing to strive for imo.)
> 
> [...]
> 
> Thanks for having me think harder about if this is really
> desirable/needed.  For me the main plus is that we'd be adhering to
> the
> current standards used for Emacs packaging: everything is unpacked
> into
> a single directory owned by that package, so as long as we include
> any
> needed resources in the #:include argument of the emacs-build-system, 
> it
> works as upstream intended it, with no need to patch variables or
> anything.  I expect this situation to worsen as Emacs packages tend
> to
> get more complicated, and I don't feel strongly enough about it to go
> arguing about Emacs packaging standards on the Emacs mailing list :-
> ).
I agree, that adhering to the standards sounds nice, but this patch
makes it seem as if we're still cooking our own soup.  I believe we'd
need to adhere to them harder than that in order for it to be an
upgrade of the status quo.

> On the minus side it makes the startup slightly more expensive in
> terms
> of pure scripting, and requires the users to be mindful that
> site-start.el evaluation is needed for the user installed packages to
> be
> activated.
> 
> Further thoughts?
I think I've already laid that out above, but we really ought to have
site-lisp/$PACKAGE-$VERSION and a working subdirs.el.  If we do it like
that, I don't think the multi-directory layout should cause us too many
troubles.

As far as the build system is concerned, we would probably need to set
up an environment similar to what will be found in the end.
Imagine this:

- environment-variables
- $PACKAGE-$VERSION/
- $PACKAGE-$VERSION-inputs/
  - subdirs.el
  - $INPUT1/
  - $INPUT2/
  - ...

Regards,
Leo

bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Maxim Cournoyer]

Hi Leo,

Leo Prikler <leo.prikler@student.tugraz.at> writes:

[...]

>> If you want to have a clearer idea of how packages from ELPA and the
>> likes are installed, you can have a peek into the 'package.el' file
>> shipped with Emacs (spoiler: it's basically just extracting the
>> contents
>> of the package archive to a single directory -- see the `package-
>> unpack'
>> procedure).
> If we follow that to the letter, should we not install those packages
> directly to site-lisp/$PACKAGE-$VERSION?  Why the guix directory?

I used to wonder the same, but after getting familiar with the 'package'
Emacs library, it defaults (per the default value of the
`package-directory-list' variable) to such a layout:

directory-on-load-path/prefix/name-version/

where 'prefix' defaults to 'elpa'.  It seems a good idea to have the
ELPA-style packages live by themselves, separated from the 'classic'
packages that are installed directly to the load path, to avoid the
package.el loader getting confused or scanning entries it wouldn't know
how to load anyway.

That said, it's shouldn't be strictly required, as old style packages or
resource directories would be lacking a *-pkg.el file and would be
disregarded, and it should be possible to install the package
directories directly to the load path if we wanted to.

> Also, if we do go that route, I think we should install a subdirs.el
> through a profile hook, that adds all these package directories to the
> load path.  See also '(elisp) Startup Summary'.  Alternatively, we can
> let subdirs.el defer to guix-emacs.el, but that'd be very cheeky.
> Either way, we should definitely make -Q work.

If we use a subdirs.el file, it could add the ELPA-style packages more
simply if they are under the guix sub-directory (guix/*).  Otherwise
we'd have to filter the resource directories that potentially find their
ways to the load path for old style packages.  Our custom subdirs.el
file could be shipped with our Emacs packages, installed in their output
at share/emacs/site-lisp/subdirs.el.  When a profile union would be
created this subdirs.el would automatically be made available.

>> Here's an experiment I've done with a profile containing the whole of
>> our current emacs-build-system based packages, with this current
>> change,
>> that checks for collision at the top level of a package installation
>> directory:
>>
>> --8<---------------cut here---------------end--------------->8---
>> $ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-packages-
>> manifest.scm
>>
>> $  find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2
>> -mindepth 2 -not -regex '.*\.elc?$' \
>>           | awk -F '/' '{ if ($9 in packages) \
>>             {printf("%s directory of %s package collides with that of
>> package %s\n", $9, $8, packages[$9])} \
>>             else {packages[$9] = $8} }'
>>
>> doc directory of modus-operandi-theme-1.0.2 package collides with
>> that of package racket-mode-0.0.2-6.5eb31a2
>> tools directory of unidecode-0.2-1.5502ada package collides with that
>> of package company-cabal-0.3.0-1.62112a7
>> snippets directory of minitest-0.8.0-1.1aadb78 package collides with
>> that of package elpy-1.35.0
>> test directory of realgud-1.5.1 package collides with that of package
>> flycheck-haskell-0.8-2.32ddff8
>> snippets directory of yasnippet-snippets-0.23 package collides with
>> that of package elpy-1.35.0
>> snippets directory of haskell-snippets-0.1.0-0.07b0f46 package
>> collides with that of package elpy-1.35.0
>> snippets directory of rspec-1.11-1.66ea7cc package collides with that
>> of package elpy-1.35.0
>> doc directory of modus-themes-1.0.2 package collides with that of
>> package racket-mode-0.0.2-6.5eb31a2
>> data directory of emojify-1.2 package collides with that of package
>> unidecode-0.2-1.5502ada
>> test directory of systemd-mode-1.6 package collides with that of
>> package flycheck-haskell-0.8-2.32ddff8
>> lib directory of slime-2.26.1 package collides with that of package
>> robe-0.8.2
>> contrib directory of sly-1.0.0-7.68561f1 package collides with that
>> of package slime-2.26.1
>> lib directory of sly-1.0.0-7.68561f1 package collides with that of
>> package robe-0.8.2
>> doc directory of modus-vivendi-theme-1.0.2 package collides with that
>> of package racket-mode-0.0.2-6.5eb31a2
>> doc directory of evil-1.14.0 package collides with that of package
>> racket-mode-0.0.2-6.5eb31a2
>> data directory of all-the-icons-4.0.1 package collides with that of
>> package unidecode-0.2-1.5502ada
>> snippets directory of feature-mode-20190801-1.11ae167 package
>> collides with that of package elpy-1.35.0
>> --8<---------------cut here---------------end--------------->8---
>>
>> So 17 Emacs packages in Guix currently conflict, and Guix seems to be
>> silent about it when building a profile with them via 'guix package
>> -p'
>> (a bug?).

> It's a bug in those packages; not in Guix.

I used to think the same, but after reading the Elisp reference manual
mentioning how Emacs packages to be installed with package.el are
expected to live in their own distinct directory with their own
resources; we can't really say that's it's a bug in the packages: they
just weren't designed to be merged in a flat directory with other Elisp
packages.  Users installing these packages manually can simply add their
top level directory to their load path, else use package.el to install
them.

> The whole idea of Guix is, that such directories are merged.  Now
> collisions within files in those directories are a different topic –
> only one can be chosen there and IIRC there should also be a way to
> make them errors.

I'll investigate this separately.

[...]

>> > > We have two schemes to accommodate for our Emacs packages:
>> > >
>> > > 1. Those installed via their own mean, e.g. make install, using
>> > > the
>> > > gnu-build-system for example.  These would still typically
>> > > install
>> > > their
>> > > packages directly under site-lisp, possibly multiple files (that
>> > > could
>> > > still collide).
>> > Why?  Seems inconsistent, does it not?
>>
>> They can be adapted in time; until then it's easy to accommodate
>> both.
> I think we should aim for consistency here, but let's adapt them in
> follow-up commits.

Agreed.

>> > > 2. Those installed via the emacs-build-system.  With the proposed
>> > > changes, those now go to site-lisp/guix/.  The 'guix' sub-
>> > > directory
>> > > makes it unambiguous that anything found under is to be loaded by
>> > > package.el; the `package-directory-list' variable can be pointed
>> > > to
>> > > it
>> > > to have the Emacs' package library discover these self-contained
>> > > packages.
>> > I think you've lost me here.  Basically, instead of being able to
>> > `(require 'my-package)` you first need to load my-package through
>> > package.el and then can `(require 'my-package)`?  I am not sure,
>> > what
>> > the benefit of that would be, if any.
>>
>> Only if you've specified --no-site-start, which would disable
>> user-installed package discovery.  This is the drawback of the
>> trade-off, not the benefit :-).
> It wasn't worded like one.  The word unambiguous typically has a
> positive connotation in my mind.  And package discovery should function
> regardless of site-start.

The word unambiguous was used to describe that using the sub-directory
guix/ would mean anything under it would be strictly packages to be
loaded with package.el and nothing else.

[...]

>> > Being able to load the same libraries as without is vital for
>> > debugging
>> > and scripts.  (Granted, some distros probably break with --no-site-
>> > lisp, but that's nothing to strive for imo.)
>>
>> [...]
>>
>> Thanks for having me think harder about if this is really
>> desirable/needed.  For me the main plus is that we'd be adhering to
>> the
>> current standards used for Emacs packaging: everything is unpacked
>> into
>> a single directory owned by that package, so as long as we include
>> any
>> needed resources in the #:include argument of the emacs-build-system,
>> it
>> works as upstream intended it, with no need to patch variables or
>> anything.  I expect this situation to worsen as Emacs packages tend
>> to
>> get more complicated, and I don't feel strongly enough about it to go
>> arguing about Emacs packaging standards on the Emacs mailing list :-
>> ).
> I agree, that adhering to the standards sounds nice, but this patch
> makes it seem as if we're still cooking our own soup.  I believe we'd
> need to adhere to them harder than that in order for it to be an
> upgrade of the status quo.

I understand this may feel a hack; because for all I know, it is ;-).
I'll try improving it with the subdirs.el discussed above.

[...]

> I think I've already laid that out above, but we really ought to have
> site-lisp/$PACKAGE-$VERSION and a working subdirs.el.  If we do it like
> that, I don't think the multi-directory layout should cause us too many
> troubles.
>
> As far as the build system is concerned, we would probably need to set
> up an environment similar to what will be found in the end.
> Imagine this:
>
> - environment-variables
> - $PACKAGE-$VERSION/
> - $PACKAGE-$VERSION-inputs/
>   - subdirs.el
>   - $INPUT1/
>   - $INPUT2/

In the build environment, there is no profile so each package add their
individual site-lisp/ to the load path (EMACSLOADPATH).  With my
proposed idea to add subdirs.el to Emacs itself, there'd be nothing to
do here.

Thanks again for this discussion; I'll work on a revised patch.  Until
then, happy new year!

Maxim

bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Leo Prikler]

Hi Maxim,
Am Samstag, den 26.12.2020, 23:44 -0500 schrieb Maxim Cournoyer:
> Hi Leo,
> 
> Leo Prikler <leo.prikler@student.tugraz.at> writes:
> 
> [...]
> 
> > > If you want to have a clearer idea of how packages from ELPA and
> > > the
> > > likes are installed, you can have a peek into the 'package.el'
> > > file
> > > shipped with Emacs (spoiler: it's basically just extracting the
> > > contents
> > > of the package archive to a single directory -- see the `package-
> > > unpack'
> > > procedure).
> > If we follow that to the letter, should we not install those
> > packages
> > directly to site-lisp/$PACKAGE-$VERSION?  Why the guix directory?
> 
> I used to wonder the same, but after getting familiar with the
> 'package'
> Emacs library, it defaults (per the default value of the
> `package-directory-list' variable) to such a layout:
> 
> directory-on-load-path/prefix/name-version/
> 
> where 'prefix' defaults to 'elpa'.  It seems a good idea to have the
> ELPA-style packages live by themselves, separated from the 'classic'
> packages that are installed directly to the load path, to avoid the
> package.el loader getting confused or scanning entries it wouldn't
> know
> how to load anyway.
> 
> That said, it's shouldn't be strictly required, as old style packages
> or
> resource directories would be lacking a *-pkg.el file and would be
> disregarded, and it should be possible to install the package
> directories directly to the load path if we wanted to.
I don't really know how to feel about faking elpa.  Also, I don't think
that 'classic' means 'not supported by package.el'.  There are other
reasons to use e.g. gnu-build-system over emacs-build-system and
packages might ship a *-pkg.el even if they do build using a Makefile. 
It's also trivial to provide one if they don't and Guix has the
facilities to do so.

> > Also, if we do go that route, I think we should install a
> > subdirs.el
> > through a profile hook, that adds all these package directories to
> > the
> > load path.  See also '(elisp) Startup Summary'.  Alternatively, we
> > can
> > let subdirs.el defer to guix-emacs.el, but that'd be very cheeky.
> > Either way, we should definitely make -Q work.
> 
> If we use a subdirs.el file, it could add the ELPA-style packages
> more
> simply if they are under the guix sub-directory (guix/*).  Otherwise
> we'd have to filter the resource directories that potentially find
> their
> ways to the load path for old style packages.  Our custom subdirs.el
> file could be shipped with our Emacs packages, installed in their
> output
> at share/emacs/site-lisp/subdirs.el.  When a profile union would be
> created this subdirs.el would automatically be made available.
IOW if we don't use guix/*, we would have to at once fix all our
packages, that don't adhere to this standard.  Sounds like a bit of
trouble, but also like it'd improve consistency.

> > > Here's an experiment I've done with a profile containing the
> > > whole of
> > > our current emacs-build-system based packages, with this current
> > > change,
> > > that checks for collision at the top level of a package
> > > installation
> > > directory:
> > > 
> > > --8<---------------cut here---------------end--------------->8---
> > > $ ./pre-inst-env guix package -p /tmp/nnew-emacs -m emacs-
> > > packages-
> > > manifest.scm
> > > 
> > > $  find -L /tmp/nnew-emacs/share/emacs/site-lisp/guix -maxdepth 2
> > > -mindepth 2 -not -regex '.*\.elc?$' \
> > >           | awk -F '/' '{ if ($9 in packages) \
> > >             {printf("%s directory of %s package collides with
> > > that of
> > > package %s\n", $9, $8, packages[$9])} \
> > >             else {packages[$9] = $8} }'
> > > 
> > > doc directory of modus-operandi-theme-1.0.2 package collides with
> > > that of package racket-mode-0.0.2-6.5eb31a2
> > > tools directory of unidecode-0.2-1.5502ada package collides with
> > > that
> > > of package company-cabal-0.3.0-1.62112a7
> > > snippets directory of minitest-0.8.0-1.1aadb78 package collides
> > > with
> > > that of package elpy-1.35.0
> > > test directory of realgud-1.5.1 package collides with that of
> > > package
> > > flycheck-haskell-0.8-2.32ddff8
> > > snippets directory of yasnippet-snippets-0.23 package collides
> > > with
> > > that of package elpy-1.35.0
> > > snippets directory of haskell-snippets-0.1.0-0.07b0f46 package
> > > collides with that of package elpy-1.35.0
> > > snippets directory of rspec-1.11-1.66ea7cc package collides with
> > > that
> > > of package elpy-1.35.0
> > > doc directory of modus-themes-1.0.2 package collides with that of
> > > package racket-mode-0.0.2-6.5eb31a2
> > > data directory of emojify-1.2 package collides with that of
> > > package
> > > unidecode-0.2-1.5502ada
> > > test directory of systemd-mode-1.6 package collides with that of
> > > package flycheck-haskell-0.8-2.32ddff8
> > > lib directory of slime-2.26.1 package collides with that of
> > > package
> > > robe-0.8.2
> > > contrib directory of sly-1.0.0-7.68561f1 package collides with
> > > that
> > > of package slime-2.26.1
> > > lib directory of sly-1.0.0-7.68561f1 package collides with that
> > > of
> > > package robe-0.8.2
> > > doc directory of modus-vivendi-theme-1.0.2 package collides with
> > > that
> > > of package racket-mode-0.0.2-6.5eb31a2
> > > doc directory of evil-1.14.0 package collides with that of
> > > package
> > > racket-mode-0.0.2-6.5eb31a2
> > > data directory of all-the-icons-4.0.1 package collides with that
> > > of
> > > package unidecode-0.2-1.5502ada
> > > snippets directory of feature-mode-20190801-1.11ae167 package
> > > collides with that of package elpy-1.35.0
> > > --8<---------------cut here---------------end--------------->8---
> > > 
> > > So 17 Emacs packages in Guix currently conflict, and Guix seems
> > > to be
> > > silent about it when building a profile with them via 'guix
> > > package
> > > -p'
> > > (a bug?).
> > It's a bug in those packages; not in Guix.
> 
> I used to think the same, but after reading the Elisp reference
> manual
> mentioning how Emacs packages to be installed with package.el are
> expected to live in their own distinct directory with their own
> resources; we can't really say that's it's a bug in the packages:
> they
> just weren't designed to be merged in a flat directory with other
> Elisp
> packages.  Users installing these packages manually can simply add
> their
> top level directory to their load path, else use package.el to
> install
> them.
Idk, feels like a mixed bag to me.  Especially yasnippet will actually
be broken by going to this new layout.  The user will be expected to
have $GUIX_PROFILE/share/emacs/site-lisp/yasnippet-snippets-
$VERSION/snippets in their yas-snippets-dir.  My really old setup
assumes, they'd be stored in $GUIX_PROFILE/share/emacs/yasnippet-
snippets/ instead, because I forgot it moved to site-lisp/snippets. 
Now what do you think happens on version upgrades?

I think, this is another argument to separate code and data even if we
do go the route of using subdirectories to store code.

> > I think I've already laid that out above, but we really ought to
> > have
> > site-lisp/$PACKAGE-$VERSION and a working subdirs.el.  If we do it
> > like
> > that, I don't think the multi-directory layout should cause us too
> > many
> > troubles.
> > 
> > As far as the build system is concerned, we would probably need to
> > set
> > up an environment similar to what will be found in the end.
> > Imagine this:
> > 
> > - environment-variables
> > - $PACKAGE-$VERSION/
> > - $PACKAGE-$VERSION-inputs/
> >   - subdirs.el
> >   - $INPUT1/
> >   - $INPUT2/
> 
> In the build environment, there is no profile so each package add
> their
> individual site-lisp/ to the load path (EMACSLOADPATH).  With my
> proposed idea to add subdirs.el to Emacs itself, there'd be nothing
> to
> do here.
Yes there would.  Exactly because there is no union-build, those
packages would not not have subdirs.el, so it's dubious as to whether
they'd be properly expanded.

> Thanks again for this discussion; I'll work on a revised
> patch.  Until
> then, happy new year!
Happy new year to you too and happy hacking 🙂

Leo