From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Radon Rosborough Newsgroups: gmane.emacs.devel Subject: Friendly discussion about (package-initialize) Date: Sun, 6 Aug 2017 17:37:55 -0700 Message-ID: NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Trace: blaine.gmane.org 1502066374 10122 195.159.176.226 (7 Aug 2017 00:39:34 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 7 Aug 2017 00:39:34 +0000 (UTC) Cc: pythonnut@gmail.com To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Aug 07 02:39:29 2017 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1deW4v-0002MZ-Ah for ged-emacs-devel@m.gmane.org; Mon, 07 Aug 2017 02:39:29 +0200 Original-Received: from localhost ([::1]:34790 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1deW51-0006X1-Ln for ged-emacs-devel@m.gmane.org; Sun, 06 Aug 2017 20:39:35 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:60238) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1deW4A-0006WU-Tq for emacs-devel@gnu.org; Sun, 06 Aug 2017 20:38:45 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1deW48-0003fV-4X for emacs-devel@gnu.org; Sun, 06 Aug 2017 20:38:42 -0400 Original-Received: from mail-lf0-x22b.google.com ([2a00:1450:4010:c07::22b]:36874) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1deW47-0003eg-Nt for emacs-devel@gnu.org; Sun, 06 Aug 2017 20:38:40 -0400 Original-Received: by mail-lf0-x22b.google.com with SMTP id m86so24029964lfi.4 for ; Sun, 06 Aug 2017 17:38:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=uc3Cz9VELqVJlSnWmygNWflet716qRBXzDvo+grwjiA=; b=uip0jxB5qPgfogd6Vv+8SQMPrY4aiGnUzAJrI1RPForLIBx/56scQASAg6gMBKPTs0 94rbr234ZjvM/GfYsVolcaVtYlpxCh7zXLm1ptSk5F/kPpv6oYAI3fDixIuxGrM1jZq3 T6TzMvuWEW2RfcFthfyaV/bqAx97eN45GHJK7dF8P/tbHKDpmgMKeGuFq73oKJEZEvXt 6EGcIm8HpL3CvxvLYVzRNtNtScseCpt9c5aKu9Sc6y1aLTUmQAO+pNrrZIWo4c6FcT4W pahRcNX12+GJRroae5ohU5SjjROlnrYY5C2V4Ao3XH50telpleqnvvTAG2OvhrJlM0R3 mbWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=uc3Cz9VELqVJlSnWmygNWflet716qRBXzDvo+grwjiA=; b=F/gsN5ppBMdONJqpdyO2v+75gRfWTpFCPka5X3wIwMsR45h82/Avd8hAuPjHNxrBi2 wBgAwt2ERM2IX1hD9GBAM5wrN97ALXHcN8K+QM9nTSRuajwBod09rIQXs4QvUMJ0XMzH BZbTIY4hUnqG6ouEkOBgx50sj2cmVxxRMsCRukbhk5OiZ1PwSP3iPWT7ndim+SFDztGa vs1s6VpE3BHbXk2HxJyIFqIBKtwIwahTg3xKuMgVZYhLc6gworlPgz1LMyU3KT5Ma5gt FgTWOPPEDmRcLo0PJOVhR/HRdy7RZ3SRdMumrh7JWpxhVbLrstxc5hb1Tb20PwEhLK1r 2cBg== X-Gm-Message-State: AHYfb5j3JKRLVHtpopU4coP++/bZ8725DZvgaObtlEmeIDSmfqmAWBU3 TUVNnEtIctz+MMugR0jir/q8wMzzKqQvwh8= X-Received: by 10.25.169.136 with SMTP id s130mr2493832lfe.173.1502066316290; Sun, 06 Aug 2017 17:38:36 -0700 (PDT) Original-Received: by 10.46.64.86 with HTTP; Sun, 6 Aug 2017 17:37:55 -0700 (PDT) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4010:c07::22b X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:217346 Archived-At: Hi all, I'd like to have a friendly discussion about (package-initialize). Specifically, about this comment that I'm sure we've all seen from time to time: ;; Added by Package.el. This must come before configurations of ;; installed packages. Don't delete this line. If you don't want it, ;; just comment it out by adding a semicolon to the start of the line. ;; You may delete these explanatory comments. (package-initialize) In this email, I'm going to explain why I think the current behavior of package.el is flawed, and propose some alternative behaviors that I believe would be better. If the community thinks that these alternatives are reasonable, I would be interested in contributing relevant patches. I know that this argument has the potential to be contentious, so I'm going to try to be as objective as possible, and stick strictly to the technical points. Since this is a rather long email, I've arranged it into sections. Feel free to skip or skim the parts that you already know. I've tried to be quite verbose so as to create a comprehensive and authoritative summary of the issue for people who may not be as familiar with Emacs packaging. ################################################################################ # WHAT DOES PACKAGE.EL DO? Since version 24.1 [1], Emacs has shipped with a built-in package manager called package.el [2]. Like other package managers such as Quelpa [3], Cask [4], el-get [5], Borg [6], and straight.el [7], the primary function of package.el is to acquire Elisp code, preprocess it (e.g. byte-compilation, autoload generation), and make it available to Emacs. The acquiry and preprocessing of Elisp code is not relevant to this discussion. What is important is how the code is made available to Emacs. The general process (called "activating" a package) is two-fold: * add the package's files to the load-path and Info-directory-list * evaluate the package's autoloads, so that running an autoloaded function loads the appropriate files from the package Importantly, these steps must be run every time Emacs is initialized. Installing Elisp code on the filesystem does nothing, unless it is somehow added to one of Emacs' default load-path entries. Before a package is activated, attempts to `require' its features will fail, and its autoloads will not be available. Different package managers have different opinions on how and when packages ought to be activated. For example, in straight.el [7], packages are activated one at a time as they are referenced in the user's init-file. However, the most common approach is for all packages to be activated at the same time. This is what package.el does; it provides a `package-initialize' function which fulfills this purpose. Therefore, absent any special treatment, one might expect a user's init-file to go something like this: ... code to configure package.el ... (package-initialize '(... some list of packages ...)) ... code to configure the packages that were initialized ... But users do not typically think in this way. They expect that installing a package is a permanent operation, and that no further action is required in order for that package to continue to be available. For this reason, package.el keeps track of which packages have been installed already, and it does this outside the init-file (in the state of the ~/.emacs.d/elpa directory). When `package-initialize' is called, all the installed packages are activated. You can request package.el to activate only a subset of the installed packages, but this is not the usual mode of operation. Thus, the user's init-file actually looks like this: ... code to configure package.el ... (package-initialize) ... code to configure the packages that were initialized ... Notice that the call to `package-initialize' must come *after* any code that configures package.el, but *before* any code that configures the packages that were initialized. This is important! If you try to configure package.el after `package-initialize', your configurations will not affect how package activation takes place. And if you try to configure packages before `package-initialize', then variables, functions, and so on are not going to be properly defined. (This might not be a problem, depending on how you handle your customizations. But it certainly does make things more complicated.) ################################################################################ #### WHAT IS THE PROBLEM? The problem is when users forget to initialize the package management system before configuring their packages. In other words, their init-file just looks like this: ... code to configure packages ... with no mention of `package-initialize' anywhere. Obviously this will not work, because the packages are never activated. The "correct" solution is to just call `package-initialize' in the init-file. But it is hard to teach users to do this, and the resulting errors have been a common problem for new Emacs users [8]. Thus, there was a discussion [8] on emacs-devel about how to arrange for things to work "correctly" in the absence of a user who knows what they are doing. The eventual outcome of that discussion is the subject of the next section. ################################################################################ #### WHAT IS THE CURRENT SOLUTION TO THE PROBLEM? The initial attempt was to have Emacs call `package-initialize' automatically during initialization. However, this is not so easy as it sounds. If Emacs calls `package-initialize' before loading the init-file, then users cannot customize package.el in their init-file anymore. And if Emacs calls `package-initialize' after loading the init-file, then users cannot customize their installed packages in their init-file anymore, unless they cause their customizations to be loaded after the end of initialization using `after-init-hook'. The only reasonable way for things to be set up is for `package-initialize' to be called in the user's init-file, but before package customizations are done. Thus, a patch [9] was suggested that causes Emacs to write a call to `package-initialize' into the user's init-file at startup, if it was not there to begin with. This was shortly merged and released in Emacs 25.1. Note that the behavior is inhibited in 'emacs -Q' mode. ################################################################################ #### JUSTIFICATIONS AND COUNTERARGUMENTS > So what are you saying is wrong with the current behavior? It modifies the user's init-file without asking. > What's wrong with modifying the user's init-file? After all, Custom > has been doing that for decades. The difference is that Custom only modifies your init-file when you ask it to save your customizations. In other words, it only modifies your init-file when you ask it to. On the other hand, package.el does it at Emacs startup without confirmation! > OK, so it's different than Custom. What's the problem, though? There are several issues: * Many users want to write their init-files by hand. If they want to initialize the package management system, they want to do it themselves, and they want to put in the code by hand so that they know it will run in the right place. * Many users have a modular Emacs configuration in which init.el loads their other customization files, one of which is usually deals with the package management system (including calling `package-initialize'). The result is that package.el sticks a superfluous, duplicate `package-initialize' into the root init-file. * It's confusing. New users have no idea why Emacs is modifying their init-file, and a proper explanation is not given. * It's extremely error-prone. Modifying the init-file is an irreversible operation, and it means that any bugs in this part of package.el are immediately 100x more frustrating. * Not everyone uses package.el. It's annoying to have to explicitly put in configuration to disable the existing package management system, if you want to use an alternative package manager. > Slow down there! Let's take this one point at a time. You said that > some users want to write their init-files by hand. But if they do > that, then can't they just put a call to `package-initialize' in > their init-files, and that will prevent the issue? No, because (1) they might be using a modular Emacs configuration that calls `package-initialize' somewhere other than init.el, and (2) they might be using a package manager other than package.el, and therefore have no need of calling `package-initialize' at all. > Well, what's wrong with them just putting a commented-out call to > `package-initialize' in their init-file, like the inserted comments > suggest? This is ugly and looks ridiculous. Is requiring a magic comment in the init-file to prevent unnecessary code from being inserted on Emacs startup really the best possible user experience? > Then what would be a better user experience? Let's wait until the ALTERNATIVES section for that :) > Wait, you said that it was a problem if `package-initialize' was put > somewhere other than in the init-file. But if it's called during > startup, then package.el doesn't run the logic to insert the a call > into the init-file. So what's the issue? The problem is that if there is an error while loading the init-file, then the call to `package-initialize' still gets inserted, if init hadn't gotten to the point where the actual call to `package-initialize' was supposed to happen. This is actually terrible, since it immediately makes debugging much more complicated: your init-file is being changed without your knowing it, while you're trying to debug initialization! > OK, but that only happens when there's an error during init. We > could make it so that the `package-initialize' happens only after a > successful init. That doesn't solve the problem, because there are all kinds of circumstances where you have a "successful" init even when the user's init-file was not loaded. For example, many modular Emacs configurations will sustain errors in one or more modules using `condition-case', and report the errors as warnings for an improved debugging experience. We're back to the beginning. Besides, often one wants to test something in a plain Emacs but not in 'emacs -Q'. In this case, the call to `package-initialize' will still get inserted. > How often does this really happen? At least 20 times in my use of Emacs, and I get progressively more annoyed every time. The existence of threads like [10] [11] indicates that there are other people who don't like this kind of behavior. Regardless of how we set up this system, it will never be possible to cover all the edge cases. Once in a while, there will be some sufficiently unexpected use case where an errant `package-initialize' gets inserted and screws something up. Since there are better solutions (see ALTERNATIVES), I think it's best to sidestep the whole problem by not taking this approach in the first place. > You said the system was confusing. Why? The comments that come with the inserted `package-initialize' tell the user "don't delete this line" without explaining the consequences if you do. The user inevitably will try deleting it, since they didn't ask for it to be added in the first place. But it will come back next time, prompting annoyance and consternation. The comments also advise "this must come before configurations of installed packages", without explaining why or anything about how package activation works. Also, the whole idea of "if you don't want this, comment it out, but don't remove it" is bizarre; I've never seen anything like it in any other program's configuration system. > Can't that be fixed by clearer comments? Yes, but the fact remains that a program inserting text into its own configuration file (except in the case where the program has a semantic understanding of the whole file, like for YAML) is highly nonstandard. I don't know of any other program that does something like this. As I said before, there are ALTERNATIVES that are more standard, and these will not violate the principle of least surprise as badly. > Why are you so concerned about bugs in package.el? Is that really > likely? I trust that the folks here on emacs-devel will do a good job in preventing bugs from creeping in. But nevertheless, I think putting in this kind of logic is simply a bad idea from a software engineering perspective. There will always be bugs, and we want to minimize the impact of the ones that inevitably do pop up. Having code that automatically modifies the init-file at startup is setting up the potential for a bug with a huge impact that will be extremely annoying and dangerous to work around. I think this is especially important to keep in mind since Emacs releases are extremely infrequent, and so we're going to have to deal with bugs in any release version for a long, long time. > What's this about alternative package managers? Shouldn't Emacs > include special support for the built-in one? After all, that's why > it's built in. Sure, I think it's reasonable for Emacs to provide special support for packages which are built in. But there's such a thing as going too far. And I personally think that you've gone too far in providing special support when that support actively makes it *more* difficult to swap out an alternative implementation. If you use an alternative package manager (and many people do), then you really don't want package.el to be automatically activating things when it's not asked to. You'll end up with duplicate load-path entries, conflicting versions, needlessly slow init times, and more: fun times (I've been there). And you *really* don't want package.el to be modifying your init-file that configures your alternative package manager so that it also activates package.el! Having such code inserted when it's actively harmful is a really frustrating experience. ################################################################################ #### ALTERNATIVES There is lots we can do to improve the situation. Some changes will be harmless and backwards compatible, while others would have to be rolled out carefully if we decide to follow them. * Don't have package.el insert `package-initialize' into the init-file under any circumstances. Instead, when Emacs is started up for the first time (when not in 'emacs -Q' mode), if no file exists at ~/.emacs.d/init.el, create that file with a basic template. This template would have a call to `package-initialize', and also some explanatory comments saying where you should put configuration of package.el versus configuration of packages, and why. This is my favorite option. It is a well-established (and, importantly, *unsurprising*) tradition for programs to come with skeleton configuration files that the user can customize as they desire. Besides, this would allow us to make a better experience for new users (remember all the complaints about "bad defaults"?) without making actual changes to the defaults and breaking backward compatibility. * Don't have package.el activate packages at startup. I would actually prefer this, but I understand that it's not likely to happen. There are two advantages: (1) users of alternative package managers don't have to explicitly disable package.el all the time (when testing without their configuration file, for example), and (2) the behavior is much less "magic". This would break backward compatibility, but mostly of expectations: only people who put their entire package configuration code into `after-init-hook' would have their configurations broken (and the fix would of course be trivial). If we implemented the skeleton init file, then the experience for new users would not change. * Improve the explanatory value of the comments that are inserted by package.el. I really, really think that the whole behavior of automatically inserting code into the init-file should be killed (with fire), but if the community really disagrees with me here, at least the comments should make an attempt at explaining what's going on and what the potential caveats are. This would be a trivial change with no downsides. * Make it trivial to disable package.el. Right now, the only foolproof way to prevent the init-file from being modified by package.el is to place multiple advices on internal functions, and to do this as early as possible during init just in case there is an error, and to also be really careful any time you load Emacs without your init-file, and to double-check for the inevitable times when you can't prevent package.el from doing stuff. ################################################################################ #### CONCLUSION AND NEXT STEPS I've argued strongly for removing the auto-insertion of `package-initialize' code into the init-file by package.el, in favor of providing a skeleton init-file. Although I've addressed a number of counterarguments, I'm sure there are other factors I haven't considered. Let me know what you think. Is this reasonable? If not, why not? If so, how can I help move it forward? Best regards, Radon Rosborough [1]: https://lists.gnu.org/archive/html/info-gnu-emacs/2012-06/msg00000.html [2]: https://www.gnu.org/software/emacs/manual/html_node/emacs/Packages.html [3]: https://github.com/quelpa/quelpa [4]: https://github.com/cask/cask [5]: https://github.com/dimitri/el-get [6]: https://github.com/emacscollective/borg [7]: https://github.com/raxod502/straight.el [8]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html [9]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01055.html [10]: https://www.reddit.com/r/emacs/comments/56fvgd/is_there_a_way_to_stop_emacs_from_adding_the/?ref=share&ref_source=link [11]: https://www.reddit.com/r/emacs/comments/4x655n/packageselectedpackages_always_appear_after/ [12]: https://github.com/raxod502/straight.el/issues/73