From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.bugs Subject: bug#30854: 27.0.50; Speeding up package.el startup Date: Mon, 19 Mar 2018 10:26:28 -0400 Message-ID: NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Trace: blaine.gmane.org 1521469573 12598 195.159.176.226 (19 Mar 2018 14:26:13 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 19 Mar 2018 14:26:13 +0000 (UTC) Cc: Stefan Monnier To: 30854@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Mon Mar 19 15:26:08 2018 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1exvjk-00038X-2x for geb-bug-gnu-emacs@m.gmane.org; Mon, 19 Mar 2018 15:26:08 +0100 Original-Received: from localhost ([::1]:42263 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1exvll-0005ee-DV for geb-bug-gnu-emacs@m.gmane.org; Mon, 19 Mar 2018 10:28:13 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:42766) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1exvkg-0004ym-Fz for bug-gnu-emacs@gnu.org; Mon, 19 Mar 2018 10:27:13 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1exvkd-0005Jy-8t for bug-gnu-emacs@gnu.org; Mon, 19 Mar 2018 10:27:06 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:34126) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1exvkd-0005Jd-38 for bug-gnu-emacs@gnu.org; Mon, 19 Mar 2018 10:27:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1exvkc-0002LN-LN; Mon, 19 Mar 2018 10:27:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Stefan Monnier Original-Sender: "Debbugs-submit" Resent-CC: monnier@iro.umontreal.ca, bug-gnu-emacs@gnu.org Resent-Date: Mon, 19 Mar 2018 14:27:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 30854 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: X-Debbugs-Original-To: bug-gnu-emacs@gnu.org X-Debbugs-Original-Xcc: Stefan Monnier Original-Received: via spool by submit@debbugs.gnu.org id=B.15214696128986 (code B ref -1); Mon, 19 Mar 2018 14:27:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 19 Mar 2018 14:26:52 +0000 Original-Received: from localhost ([127.0.0.1]:42023 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1exvkR-0002Kr-AQ for submit@debbugs.gnu.org; Mon, 19 Mar 2018 10:26:52 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:51722) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1exvkP-0002Ke-7Q for submit@debbugs.gnu.org; Mon, 19 Mar 2018 10:26:50 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1exvkH-000567-MP for submit@debbugs.gnu.org; Mon, 19 Mar 2018 10:26:44 -0400 Original-Received: from lists.gnu.org ([2001:4830:134:3::11]:56024) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1exvkH-00055x-IF for submit@debbugs.gnu.org; Mon, 19 Mar 2018 10:26:41 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:42660) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1exvkD-0004jx-1z for bug-gnu-emacs@gnu.org; Mon, 19 Mar 2018 10:26:41 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1exvk9-00050J-Qb for bug-gnu-emacs@gnu.org; Mon, 19 Mar 2018 10:26:37 -0400 Original-Received: from pruche.dit.umontreal.ca ([132.204.246.22]:36474) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1exvk9-0004xo-JD for bug-gnu-emacs@gnu.org; Mon, 19 Mar 2018 10:26:33 -0400 Original-Received: from ceviche.home (lechon.iro.umontreal.ca [132.204.27.242]) by pruche.dit.umontreal.ca (8.14.7/8.14.1) with ESMTP id w2JEQSiM015370 for ; Mon, 19 Mar 2018 10:26:28 -0400 Original-Received: by ceviche.home (Postfix, from userid 20848) id 40D6766461; Mon, 19 Mar 2018 10:26:28 -0400 (EDT) X-NAI-Spam-Flag: NO X-NAI-Spam-Level: * X-NAI-Spam-Threshold: 5 X-NAI-Spam-Score: 1.2 X-NAI-Spam-Rules: 5 Rules triggered BEC_TRC1_W_GEN_SPAM_FEATRE=0.7, BEC_TRC1=0.4, GEN_SPAM_FEATRE=0.1, EDT_SA_DN_PASS=0, RV6245=0 X-NAI-Spam-Version: 2.3.0.9418 : core <6245> : inlines <6502> : streams <1781655> : uri <2611081> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:144405 Archived-At: Package: Emacs Version: 27.0.50 There are many ways to make Emacs's startup slow. One of them is to have many packages installed. The growing popularity of ELPA means that it's now common for users to have hundreds of ELPA packages installed, which will easily cause Emacs's startup to take at least 1s even with an empty ~/.emacs. For users like me who (re)start their Emacs session only rarely, this is not an issue, but for others it can be an annoyance that's significant enough to try and circumvent it by not using package.el (e.g. installing all their packages by hand or using other packaging like DOOM). If there's no objection, I plan on installing the patch below which below lets users cut down package.el startup time by skipping package's initialization and instead loading a single precomputed file which is the concatenation of all the installed -autoloads.el. In my experience this speeds up activation of package.el by a factor 5. We could speed it up even further by byte-compiling this file, but this has bumped into some corner cases problems so I'm sticking to a non-compiled file for now. Stefan diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi index a69888cdbd..c3cfaabb8d 100644 --- a/doc/emacs/custom.texi +++ b/doc/emacs/custom.texi @@ -2602,17 +2602,16 @@ Early Init File @cindex early init file =20 Most customizations for Emacs can be put in the normal init file, -@file{.emacs} or @file{~/.emacs.d/init.el}. However, it is sometimes -desirable to have customizations that take effect during Emacs startup -earlier than the normal init file is processed. Such customizations -can be put in the early init file, @file{~/.emacs.d/early-init.el}. -This file is loaded before the package system is initialized, so in it -you can customize variables that affect the package initialization -process, such as @code{package-enable-at-startup}, -@code{package-load-list}, and @code{package-user-dir}. Note that -variables like @code{package-archives} which only affect the -installation of new packages, and not the process of making -already-installed packages available, may be customized in the regular +@file{.emacs} or @file{~/.emacs.d/init.el}. However, it is sometimes desi= rable +to have customizations that take effect during Emacs startup earlier than = the +normal init file is processed. Such customizations can be put in the early +init file, @file{~/.emacs.d/early-init.el}. This file is loaded before the +package system and GUI is initialized, so in it you can customize variables +that affect frame appearance as well as the package initialization process, +such as @code{package-enable-at-startup}, @code{package-load-list}, and +@code{package-user-dir}. Note that variables like @code{package-archives} +which only affect the installation of new packages, and not the process of +making already-installed packages available, may be customized in the regu= lar init file. @xref{Package Installation}. =20 For more information on the early init file, @pxref{Init File,,, diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi index 0e30ad519a..77ecb667f4 100644 --- a/doc/lispref/os.texi +++ b/doc/lispref/os.texi @@ -103,12 +103,12 @@ Startup Summary directory instead. =20 @item -It calls the function @code{package-initialize} to activate any +It calls the function @code{package-activate-all} to activate any optional Emacs Lisp package that has been installed. @xref{Packaging -Basics}. However, Emacs doesn't initialize packages when +Basics}. However, Emacs doesn't activate the packages when @code{package-enable-at-startup} is @code{nil} or when it's started with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}. To -initialize packages in the latter case, @code{package-initialize} +activate the packages in the latter case, @code{package-activate-all} should be called explicitly (e.g., via the @samp{--funcall} option). =20 @vindex initial-window-system@r{, and startup} diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi index 7e7a8cd9bc..37c1ee6697 100644 --- a/doc/lispref/package.texi +++ b/doc/lispref/package.texi @@ -105,16 +105,15 @@ Packaging Basics evaluates the autoload definitions in @file{@var{name}-autoloads.el}. =20 Whenever Emacs starts up, it automatically calls the function -@code{package-initialize} to make installed packages available to the +@code{package-activate-all} to make installed packages available to the current session. This is done after loading the early init file, but before loading the regular init file (@pxref{Startup Summary}). Packages are not automatically made available if the user option @code{package-enable-at-startup} is set to @code{nil} in the early init file. =20 -@deffn Command package-initialize &optional no-activate -This function initializes Emacs' internal record of which packages are -installed, and makes the packages available to the current session. +@defun package-activate-all +This function makes the packages available to the current session. The user option @code{package-load-list} specifies which packages to make available; by default, all installed packages are made available. If called during startup, this function also sets @@ -122,15 +121,20 @@ Packaging Basics evaluating package autoloads more than once. @xref{Package Installation,,, emacs, The GNU Emacs Manual}. =20 -The optional argument @var{no-activate}, if non-@code{nil}, causes -Emacs to update its record of installed packages without actually -making them available; it is for internal use only. - -In most cases, you should not need to call @code{package-initialize}, +In most cases, you should not need to call @code{package-activate-all}, as this is done automatically during startup. Simply make sure to put -any code that should run before @code{package-initialize} in the early +any code that should run before @code{package-activate-all} in the early init file, and any code that should run after it in the primary init file (@pxref{Init File,,, emacs, The GNU Emacs Manual}). +@end defun + +@deffn Command package-initialize &optional no-activate +This function initializes Emacs' internal record of which packages are +installed, and then calls @code{package-activate-all}. + +The optional argument @var{no-activate}, if non-@code{nil}, causes +Emacs to update its record of installed packages without actually +making them available. @end deffn =20 @node Simple Packages diff --git a/etc/NEWS b/etc/NEWS index 99f3f27486..a17791252f 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -59,14 +59,20 @@ package system is initialized given that initialization= now happens before loading the regular init file (see below). =20 +++ -** Emacs now calls 'package-initialize' before loading the init file. +** Installed packages are now activated *before* loading the init file. This is part of a change intended to eliminate the behavior of package.el inserting a call to 'package-initialize' into the init file, which was previously done when Emacs was started. As a result of this change, it is no longer necessary to call 'package-initialize' -in your init file. However, if your init file changes the values of -'package-load-list' or 'package-user-dir', then that code needs to be -moved to the early init file (see above). +in your init file. + +However, if your init file changes the values of 'package-load-list' or +'package-user-dir', or sets 'package-enable-at-startup' to nil then it won= 't +work right without some adjustment: +- you can move that code to the early init file (see above), so those sett= ings + apply before Emacs tries to activate the packages. +- you can use the new 'package-quickstart` so activation of packages does = not + need to pay attention to 'package-load-list' or 'package-user-dir' any m= ore. =20 * Changes in Emacs 27.1 @@ -149,6 +155,15 @@ for abbrevs that have them. It now treats the optional 2nd argument to mean that the URL should be shown in the currently selected window. =20 +** Package +*** New 'package-quickstart' feature +When 'package-quickstart' is non-nil, package.el precomputes a big autoloa= ds +file so that activation of packages can be done much faster, which can spe= ed up +your startup significantly. +It also causes variables like package-user-dir and package-load-list to be +consulted when 'package-quickstart-refresh' is run rather than at startup = so +you don't need to set them in your early init file. + ** Ecomplete *** The ecomplete sorting has changed to a decay-based algorithm. This can be controlled by the new `ecomplete-sort-predicate' variable. diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el index 1edc06d024..9faae54f7b 100644 --- a/lisp/emacs-lisp/package.el +++ b/lisp/emacs-lisp/package.el @@ -681,6 +681,9 @@ package--activate-autoloads-and-load-path (defvar Info-directory-list) (declare-function info-initialize "info" ()) =20 +(defvar package--quickstart-pkgs t + "If set to a list, we're computing the set of pkgs to activate.") + (defun package--load-files-for-activation (pkg-desc reload) "Load files for activating a package given by PKG-DESC. Load the autoloads file, and ensure `load-path' is setup. If @@ -723,7 +726,10 @@ package-activate-1 (message "Unable to activate package `%s'.\nRequired package `= %s-%s' is unavailable" name (car req) (package-version-join (cadr req))) (throw 'exit nil)))) - (package--load-files-for-activation pkg-desc reload) + (if (listp package--quickstart-pkgs) + ;; We're only collecting the set of packages to activate! + (push pkg-desc package--quickstart-pkgs) + (package--load-files-for-activation pkg-desc reload)) ;; Add info node. (when (file-exists-p (expand-file-name "dir" pkg-dir)) ;; FIXME: not the friendliest, but simple. @@ -1463,18 +1469,34 @@ package-initialize (setq package-enable-at-startup nil) (package-load-all-descriptors) (package-read-all-archive-contents) + (setq package--initialized t) (unless no-activate + (package-activate-all)) + ;; This uses `package--mapc' so it must be called after + ;; `package--initialized' is t. + (package--build-compatibility-table)) + +(defvar package-quickstart-file) + +;;;###autoload +(defun package-activate-all () + "Activate all installed packages. +The variable `package-load-list' controls which packages to load." + (setq package-enable-at-startup nil) + (if (file-readable-p package-quickstart-file) + ;; Skip load-source-file-function which would slow us down by a fact= or + ;; 2 (this assumes we were careful to save this file so it doesn't n= eed + ;; any decoding). + (let ((load-source-file-function nil)) + (load package-quickstart-file)) + (unless package--initialized + (package-initialize t)) (dolist (elt package-alist) (condition-case err (package-activate (car elt)) ;; Don't let failure of activation of a package arbitrarily stop ;; activation of further packages. - (error (message "%s" (error-message-string err)))))) - (setq package--initialized t) - ;; This uses `package--mapc' so it must be called after - ;; `package--initialized' is t. - (package--build-compatibility-table)) - + (error (message "%s" (error-message-string err))))))) ;;;; Populating `package-archive-contents' from archives ;; This subsection populates the variables listed above from the @@ -1856,18 +1878,26 @@ package-installed-p should be a version list. =20 If PACKAGE is a `package-desc' object, MIN-VERSION is ignored." - (unless package--initialized (error "package.el is not yet initialized!"= )) - (if (package-desc-p package) - (let ((dir (package-desc-dir package))) + (cond + ((package-desc-p package) + (let ((dir (package-desc-dir package))) (and (stringp dir) - (file-exists-p dir))) + (file-exists-p dir)))) + ((and (not package--initialized) + (null min-version) + package-activated-list) + ;; We used the quickstart: make it possible to use package-installed-p + ;; even before package is fully initialized. + (memq package package-activated-list)) + ((not package--initialized) (error "package.el is not yet initialized!"= )) + (t (or (let ((pkg-descs (cdr (assq package package-alist)))) (and pkg-descs (version-list-<=3D min-version (package-desc-version (car pkg-descs))))) ;; Also check built-in packages. - (package-built-in-p package min-version)))) + (package-built-in-p package min-version))))) =20 (defun package-download-transaction (packages) "Download and install all the packages in PACKAGES. @@ -1918,7 +1948,9 @@ package-install (package-compute-transaction (list pkg) (package-desc-reqs pkg))) (package-compute-transaction () (list (list pkg)))))) - (package-download-transaction transaction) + (progn + (package-download-transaction transaction) + (package--quickstart-maybe-refresh)) (message "`%s' is already installed" name)))) =20 (defun package-strip-rcs-id (str) @@ -2090,7 +2122,9 @@ package-delete (delete pkg-desc pkgs) (unless (cdr pkgs) (setq package-alist (delq pkgs package-alist)))) - (message "Package `%s' deleted." (package-desc-full-name pkg-de= sc)))))) + (package--quickstart-maybe-refresh) + (message "Package `%s' deleted." + (package-desc-full-name pkg-desc)))))) =20 ;;;###autoload (defun package-reinstall (pkg) @@ -3415,6 +3449,95 @@ package-list-packages-no-fetch (interactive) (list-packages t)) =20 +;;;; Quickstart: precompute activation actions for faster start up. + +;; Activating packages via `package-initialize' is costly: for N installed +;; packages, it needs to read all N -pkg.el files first to decide +;; which packages to activate, and then again N -autoloads.el files. +;; To speed this up, we precompute a mega-autoloads file which is the +;; concatenation of all those -autoloads.el, so we can activate +;; all packages by loading this one file (and hence without initializing +;; package.el). + +;; Other than speeding things up, this also offers a bootstrap feature: +;; it lets us activate packages according to package-load-list and +;; package-user-dir even before those vars are set. + +(defcustom package-quickstart nil + "Precompute activation actions to speed up startup. +This requires the use of `package-quickstart-refresh' every time the +activations need to be changed, such as when `package-load-list' is modifi= ed." + :type 'boolean) + +(defcustom package-quickstart-file + (locate-user-emacs-file "package-quickstart.el") + "Location of the file used to speed up activation of packages at startup= ." + :type 'file) + +(defun package--quickstart-maybe-refresh () + (if package-quickstart + ;; FIXME: Delay refresh in case we're installing/deleting + ;; several packages! + (package-quickstart-refresh) + (delete-file package-quickstart-file))) + +(defun package-quickstart-refresh () + "(Re)Generate the `package-quickstart-file'." + (interactive) + (package-initialize 'no-activate) + (require 'info) + (let ((package--quickstart-pkgs ()) + ;; Pretend we haven't activated anything yet! + (package-activated-list ()) + ;; Make sure we can load this file without load-source-file-functi= on. + (coding-system-for-write 'emacs-internal) + (Info-directory-list '(""))) + (dolist (elt package-alist) + (condition-case err + (package-activate (car elt)) + ;; Don't let failure of activation of a package arbitrarily stop + ;; activation of further packages. + (error (message "%s" (error-message-string err))))) + (setq package--quickstart-pkgs (nreverse package--quickstart-pkgs)) + (with-temp-file package-quickstart-file + (emacs-lisp-mode) ;For `syntax-ppss'. + (insert ";;; Quickstart file to activate all packages at startup -*= - lexical-binding:t -*-\n") + (insert ";; =A1=A1 This file is autogenerated by `package-quickstart= -refresh', DO NOT EDIT !!\n\n") + (dolist (pkg package--quickstart-pkgs) + (let* ((file + ;; Prefer uncompiled files (and don't accept .so files). + (let ((load-suffixes '(".el" ".elc"))) + (locate-library (package--autoloads-file-name pkg)))) + (pfile (prin1-to-string file))) + (insert "(let ((load-file-name " pfile "))\n") + (insert-file-contents file) + ;; Fixup the special #$ reader form and throw away comments. + (while (re-search-forward "#\\$\\|^;\\(.*\n\\)" nil 'move) + (unless (nth 8 (syntax-ppss)) + (replace-match (if (match-end 1) "" pfile) t t))) + (unless (bolp) (insert "\n")) + (insert ")\n"))) + (pp `(setq package-activated-list + (append ',(mapcar #'package-desc-name package--quickstart= -pkgs) + package-activated-list)) + (current-buffer)) + (let ((info-dirs (butlast Info-directory-list))) + (when info-dirs + (pp `(progn (require 'info) + (info-initialize) + (setq Info-directory-list + (append ',info-dirs Info-directory-list))) + (current-buffer)))) + ;; Use `\s' instead of a space character, so this code chunk is not + ;; mistaken for an actual file-local section of package.el. + (insert " +;; Local\sVariables: +;; version-control: never +;; no-byte-compile: t +;; no-update-autoloads: t +;; End: +")))) + (provide 'package) =20 ;;; package.el ends here diff --git a/lisp/files.el b/lisp/files.el index 8ec2bde588..1ead4a78da 100644 --- a/lisp/files.el +++ b/lisp/files.el @@ -3627,7 +3627,8 @@ hack-local-variables (push (cons (if (eq var 'eval) 'eval (indirect-variable var)) - val) result)))))) + val) + result)))))) (forward-line 1)))))))) ;; Now we've read all the local variables. ;; If HANDLE-MODE is t, return whether the mode was specified. diff --git a/lisp/startup.el b/lisp/startup.el index 2669342eda..1faeabf23b 100644 --- a/lisp/startup.el +++ b/lisp/startup.el @@ -1185,7 +1185,7 @@ command-line (package--description-file subdir) subdir)))) (throw 'package-dir-found t))))))) - (package-initialize)) + (package-activate-all)) =20 ;; Make sure window system's init file was loaded in loadup.el if ;; using a window system.