[bug#45359] [PATCH]: Re-introduce Emacs packages specific installation prefix.

[bug#45359] [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

Hello,

I've goofed and sent this to guix-bugs, apparently, so I'm forwarding it
to guix-patches to make sure everybody possibly interested had a chance
to see it.

Thank you,

Maxim

-------------------- Start of forwarded message --------------------
Subject: bug#45316: [PATCH]: Re-introduce Emacs packages specific installation prefix.
To: 45316@debbugs.gnu.org
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Fri, 18 Dec 2020 17:00:10 -0500


[-- Attachment #2.1: Type: text/plain, Size: 2631 bytes --]

Hello Guix!

tl;dr: The Emacs build system and site-start.el loader are modified so
that Emacs packages are installed in their own distinct installation
directory.

The Emacs packages built with the Emacs built system used to be
installed in a sub-directory under the share/emacs/guix.d/ directory,
but this was changed in commit 65a7dd2950ca13a8b942b2836260a2192351b271
shortly after having accommodated the site-start.el machinery to enable
loading packages from any profile (via the EMACSLOADPATH search path
specification).

While this change allowed to expose simply and directly the packages
found in EMACSLOADPATH, it does introduce the risk of file name
collisions when multiple Emacs packages are joined in the same profile,
especially with Emacs packages increasing in complexity (e.g., using
more than a single .el file!) and expecting to have both their sources
and resources extracted under their own nested directory rather than as
a flat collection (ELPA, MELPA).

One recent example I stumbled on was attempting to use the
emacs-yasnippet-snippets package along with emacs-elpy; both wanted to
install a 'snippets' directory to share/emacs/site-lisp/snippets,
collided and resulted in problems that prove difficult to understand.

This is what motivated this patch series, where the site-start.el
auxiliary code used for package discovery is extended to support
packages installed in their own directory under a 'share/emacs/guix'
installation prefix, via Emacs' own package library!

The emacs-build-system is updated for this new installation prefix, as
well as existing packages and documentation.  Parting with a directly
usable EMACSLOADPATH means that site-start.el *must* run for packages to
appear in the load-path; that means for running a test suite, the -Q or
--quick Emacs options cannot be used, since it implies --no-site-file.

Benefits of using this approach:

+ Avoid inter-package file name collisions.

+ Better integration with user installed packages via M-x
package-install.  The Guix-installed packages are listed in M-x
package-list as 'external'.

Cons include:

- Slightly more complex loader (although much of it is offloaded to
  package.el), thus slightly slower (see the comparison below).

- Requires to ensure every package's test suite doesn't make use of -Q.

In my opinion the benefits outweighs the cons by a comfortable margin,
especially with the boring work of adapting the package collection
already done.

To test the performance of the new approach, the following manifest file
was used to test the rebuild of the ~900 Emacs packages making use of
the Emacs build system:


[-- Attachment #2.2: emacs-packages-manifest.scm --]
[-- Type: text/plain, Size: 806 bytes --]

(use-modules (gnu packages)
             (guix build-system)
             (guix packages)
             (srfi srfi-1))

(define %broken-emacs-packages
  (map specification->package
       '("emacs-picpocket"              ;tests fail
         "emacs-twittering-mode"        ;build fails

         ;; Broken only on current master, without new changes.
         "emacs-md4rd"
         "emacs-el-patch"
         "emacs-flymake-shellcheck"
         )))

(define %emacs-packages
  (fold-packages (lambda (package lst)
                   (if (eq? (build-system-name (package-build-system package))
                            'emacs)
                       (cons package lst)
                       lst))
                 '()))

(packages->manifest
 (lset-difference eqv? %emacs-packages %broken-emacs-packages))

[-- Attachment #2.3: Type: text/plain, Size: 2772 bytes --]


A simple benchmark testing the performance of the activation of the
hundreds of Emacs packages was then run using:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix environment --pure -m emacs-packages-manifest.scm \
                                  --ad-hoc emacs
[env]$ /run/setuid-programs/sudo /bin/sh -c 'echo 3 > /proc/sys/vm/drop_caches'
[env]$ emacs --batch --no-site-file \
    --eval="(progn (require 'guix-emacs) \
                   (require 'benchmark) \
                   (message \"(total gc-count gc-time) = %s\" \
                            (benchmark-run 1 (guix-emacs-autoload-packages))))"
--8<---------------cut here---------------end--------------->8---

On the master branch:

--8<---------------cut here---------------start------------->8---
[...]
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/zotxt-autoloads...
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/zoutline-autoloads...
Loading /gnu/store/qajc70c7nqycs1301ram8s3x7k9ibg5f-profile/share/emacs/site-lisp/ztree-autoloads...
(total gc-count gc-time) = (25.242400751 13 0.189669369)
--8<---------------cut here---------------end--------------->8---

Or about 0.65 s on a warm cache.

On a branch with these changes:

--8<---------------cut here---------------start------------->8---
Error loading autoloads: (file-missing Cannot open load file No such file or directory kotl/kotl-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory helm-easymenu)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/flycheck-cpplint-0.1-1.1d8a090/flycheck-cpplint-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/evil-anzu-0.03/evil-anzu-autoloads)
Error loading autoloads: (file-missing Cannot open load file No such file or directory /gnu/store/ryh0rasi9frm98dkd3kbck6hya6hn2qr-profile/share/emacs/site-lisp/guix/erc-image-0-3.82fb387/erc-image-autoloads)
ad-handle-definition: `ido-completing-read' got redefined
Error loading autoloads: (file-missing Cannot open load file No such file or directory tex-site)
(total gc-count gc-time) = (26.175704339 47 0.783184412)
--8<---------------cut here---------------end--------------->8---

Or about 3 seconds on a warm cache.

There a 6 errors that would need to be looked into, but I these look
like actual packaging problems rather than new issues.  The previously
used way to load the autoloads, '(load f 'noerror)' would have masked
them.

Thanks,

Maxim

[-- Attachment #3: Type: text/plain, Size: 67 bytes --]

-------------------- End of forwarded message --------------------

[bug#45359] bug#47458: Terrible UX upgrading Emacs in Guix

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> Eventually, I realized what the problem was:
>
> (1) My existing emacs session started failing because
>     ~/.guix-profile/share/emacs/27.1 had disappeared out from under it.
>
> (2) My newly launched emacs sessions were failing because my
>     EMACSLOADPATH variable was still set to its old value, pointing at
>     /home/mhw/.guix-profile/share/emacs/27.1/lisp, which no longer
>     existed.
>
> I'm not sure why I've never run into this problem before.  I'm also not
> sure what can be done to make this better, but if anyone has ideas, that
> would be good.  If a 7+ year Guix veteran developer gets bitten badly by
> this, I doubt that less experienced users will be impressed.

Ouch.  “It used to be” (speaking like a veteran :-)) that Emacs in Guix
would not use EMACSLOADPATH.  Then we switched to EMACSLOADPATH, which
had some advantages, but necessarily has this drawback.

IIUC, <https://issues.guix.gnu.org/45359> is about possibly
backtracking.  Maxim, what’s the status of this one?

Thanks,
Ludo’.

[bug#45359] bug#47458: Terrible UX upgrading Emacs in Guix

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> Eventually, I realized what the problem was:
>
> (1) My existing emacs session started failing because
>     ~/.guix-profile/share/emacs/27.1 had disappeared out from under it.
>
> (2) My newly launched emacs sessions were failing because my
>     EMACSLOADPATH variable was still set to its old value, pointing at
>     /home/mhw/.guix-profile/share/emacs/27.1/lisp, which no longer
>     existed.
>
> I'm not sure why I've never run into this problem before.  I'm also not
> sure what can be done to make this better, but if anyone has ideas, that
> would be good.  If a 7+ year Guix veteran developer gets bitten badly by
> this, I doubt that less experienced users will be impressed.

Ouch.  “It used to be” (speaking like a veteran :-)) that Emacs in Guix
would not use EMACSLOADPATH.  Then we switched to EMACSLOADPATH, which
had some advantages, but necessarily has this drawback.

IIUC, <https://issues.guix.gnu.org/45359> is about possibly
backtracking.  Maxim, what’s the status of this one?

Thanks,
Ludo’.

bug#45359: bug#47458: Terrible UX upgrading Emacs in Guix

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Mark,
>
> Mark H Weaver <mhw@netris.org> skribis:
>
>> Eventually, I realized what the problem was:
>>
>> (1) My existing emacs session started failing because
>>     ~/.guix-profile/share/emacs/27.1 had disappeared out from under it.
>>
>> (2) My newly launched emacs sessions were failing because my
>>     EMACSLOADPATH variable was still set to its old value, pointing at
>>     /home/mhw/.guix-profile/share/emacs/27.1/lisp, which no longer
>>     existed.
>>
>> I'm not sure why I've never run into this problem before.  I'm also not
>> sure what can be done to make this better, but if anyone has ideas, that
>> would be good.  If a 7+ year Guix veteran developer gets bitten badly by
>> this, I doubt that less experienced users will be impressed.
>
> Ouch.  “It used to be” (speaking like a veteran :-)) that Emacs in Guix
> would not use EMACSLOADPATH.  Then we switched to EMACSLOADPATH, which
> had some advantages, but necessarily has this drawback.
>
> IIUC, <https://issues.guix.gnu.org/45359> is about possibly
> backtracking.  Maxim, what’s the status of this one?

It's abandoned, The MUMI tracker lacks the responses from Leo Prickler,
but they had good arguments maintaining the status quo rather than going
with the extra complexity.  It also wouldn't change the issue at hand;
it'd merely prevent conflicts of *resources* files of Emacs packages
(and somewhat integrate with the Emacs native package manager, while
making the autoloading a bit slower).  It seems the price to pay is too
high for such a small gain.  I'm closing it now.

On the other hand, this very problem was the motivation for this patch
series here: https://issues.guix.gnu.org/43627, which would solve the
issue ta hand.  You were skeptical of the benefits the last time you
took a look at it; perhaps it's time to take a new look at it :-).

Thanks,

Maxim

[bug#45359] closed (Re: bug#47458: Terrible UX upgrading Emacs in Guix)

[Maxim Cournoyer]

reopen 45359
thanks

Hi,

It seems I've closed that report by mistake.  Sorry!

Reopening.

Maxim

bug#45359: [PATCH]: Re-introduce Emacs packages specific installation prefix.

[Maxim Cournoyer]

Hi,

help-debbugs@gnu.org (GNU bug Tracking System) writes:

> Your bug report
>
> #45359: [PATCH]: Re-introduce Emacs packages specific installation prefix.
>
> which was filed against the guix-patches package, has been closed.
>
> The explanation is attached below, along with your original report.
> If you require more details, please reply to 45359@debbugs.gnu.org.

Seems the reply title got me confused; this *is* the issue I want to
close.  Doing so again, with the title fixed this time.

Maxim

[bug#45359] bug#47458: Terrible UX upgrading Emacs in Guix

[Ludovic Courtès]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> Ouch.  “It used to be” (speaking like a veteran :-)) that Emacs in Guix
>> would not use EMACSLOADPATH.  Then we switched to EMACSLOADPATH, which
>> had some advantages, but necessarily has this drawback.
>>
>> IIUC, <https://issues.guix.gnu.org/45359> is about possibly
>> backtracking.  Maxim, what’s the status of this one?
>
> It's abandoned, The MUMI tracker lacks the responses from Leo Prickler,
> but they had good arguments maintaining the status quo rather than going
> with the extra complexity.  It also wouldn't change the issue at hand;
> it'd merely prevent conflicts of *resources* files of Emacs packages
> (and somewhat integrate with the Emacs native package manager, while
> making the autoloading a bit slower).  It seems the price to pay is too
> high for such a small gain.  I'm closing it now.

I see, makes sense!

> On the other hand, this very problem was the motivation for this patch
> series here: https://issues.guix.gnu.org/43627, which would solve the
> issue ta hand.  You were skeptical of the benefits the last time you
> took a look at it; perhaps it's time to take a new look at it :-).

Ah!  Now I may have to revisit it, indeed.

Thanks for explaining!

Ludo’.