bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Joost Kremers]

The variable `package-selected-packages` should not be saved to the users init
file (or whatever `custom-file` points to). This was discussed before on
emacs-devel but wasn't resolved:

<https://lists.gnu.org/archive/html/emacs-devel/2016-02/msg00958.html> but 

The gist of it is that if you keep your configuration under version control,
having package-selected-packages written to your config file is inconvenient if
you share your config across multiple machines but do not want to have the exact
same set of packages installed on each.

Even if you do want to keep the exact same set of packages on each machine, you
still need to be careful about installing them, because the order of package
installation affects the order of package-selected-packages, and so does
deleting and reinstalling a package. As a result, I often experienced
meaningless VC conflicts that needed to be resolved.

IMHO the value of package-selected-packages should be written to a separate
file, so that it's possible to put it under version control or ignore it,
independently from the rest of one's configuration.


In GNU Emacs 28.1.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0)
 of 2022-08-03 built on Lenovo
Repository revision: f23d45603918721ec80bfcef883d22fab15cb9f1
Repository branch: emacs-28
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: elementary OS 6.1 Jólnir

Configured using:
 'configure --with-native-compilation'
Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE
XIM XPM GTK3 ZLIB
Important settings:
  value of $LC_MONETARY: en_IE.UTF-8
  value of $LC_NUMERIC: en_IE.UTF-8
  value of $LC_TIME: en_IE.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: mu4e:main

Minor modes in effect:
  flycheck-indicator-mode: t
  mu4e-search-minor-mode: t
  mu4e-update-minor-mode: t
  mu4e-context-minor-mode: t
  minions-mode: t
  doom-modeline-mode: t
  which-key-mode: t
  global-atomic-chrome-edit-mode: t
  marginalia-mode: t
  all-the-icons-completion-mode: t
  company-prescient-mode: t
  prescient-persist-mode: t
  vertico-multiform-mode: t
  vertico-mode: t
  projectile-mode: t
  global-diff-hl-mode: t
  global-git-commit-mode: t
  csv-field-index-mode: t
  shell-dirtrack-mode: t
  company-quickhelp-mode: t
  company-quickhelp-local-mode: t
  global-company-mode: t
  electric-pair-mode: t
  recentf-mode: t
  delete-selection-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-save-visited-mode: t
  overwrite-mode: overwrite-mode-binary

Load-path shadows:
~/.emacs.d/lisp/custom hides /usr/local/share/emacs/28.1.91/lisp/custom
/home/joost/.emacs.d/elpa/transient-20220806.2224/transient hides /usr/local/share/emacs/28.1.91/lisp/transient

Features:
(shadow emacsbug qp gnutls epa-file mailalias ace-window vc-mtn vc-hg vc-bzr
vc-src vc-sccs vc-cvs vc-rcs consult-vertico consult compat-28 vc-git
display-fill-column-indicator flycheck-indicator avy smiley gnus-cite mm-archive
mail-extr visual-fill-column display-line-numbers mu4e-settings gnus-dired mu4e
mu4e-org mu4e-view gnus-art mm-uu mml2015 mm-view mml-smime smime dig gnus-sum
gnus-group gnus-undo gnus-start gnus-dbus gnus-cloud nnimap nnmail mail-source
utf7 netrc nnoo gnus-spec gnus-int gnus-range gnus-win gnus nnheader mu4e-main
mu4e-headers mu4e-compose mu4e-draft mu4e-actions smtpmail mu4e-search
mu4e-lists mu4e-bookmarks mu4e-mark mu4e-message flow-fill mule-util
mu4e-contacts mu4e-update mu4e-folders mu4e-server mu4e-context mu4e-vars
mu4e-helpers mu4e-config jk-input-methods quail company-box company-box-doc
frame-local company-box-icons company-amelia amelia parse-csv dedicated wgrep-ag
wgrep flycheck-ledger vterm bookmark term disp-table ehelp vterm-module
term/xterm xterm ielm minions doom-modeline doom-modeline-segments
doom-modeline-env doom-modeline-core shrink-path which-key atomic-chrome iimage
image+ image-file image-converter marginalia all-the-icons-completion
company-prescient prescient orderless vertico-multiform vertico groovy-mode
dockerfile-mode sh-script smie executable jsonian impatient-mode htmlize tide
tide-lv typescript-mode indium indium-list-sources indium-scratch
indium-interaction indium-chrome indium-nodejs indium-repl indium-debugger
indium-debugger-litable js2-refactor js2r-paredit js2r-conveniences
js2r-conditionals js2r-wrapping js2r-functions js2r-vars
mc-hide-unmatched-lines-mode mc-mark-more mc-cycle-cursors multiple-cursors-core
rect js2r-iife js2r-formatting js2r-helpers indium-debugger-locals
indium-breakpoint indium-inspector indium-render indium-faces indium-seq-fix
indium-client indium-structs json-process-client js2-mode js cc-mode cc-fonts
cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs jupyter
python-pytest lsp-pyright dap-python paredit sesman clojure-mode align dap-mode
dap-tasks dap-launch lsp-docker dap-overlays eldoc-box jsonrpc ert debug
backtrace lsp-ui lsp-ui-flycheck lsp-ui-doc goto-addr lsp-ui-imenu lsp-ui-peek
lsp-ui-sideline flycheck-posframe flycheck lsp-mode network-stream markdown-mode
inline imenu f f-shortdoc shortdoc lsp-ui-util face-remap lsp-protocol ht ag
vc-svn find-dired s projectile lisp-mnt grep compile ibuf-ext ibuffer
ibuffer-loaddefs thingatpt diff-hl log-view vc-dir ewoc vc vc-dispatcher
diff-mode git-commit magit-git magit-base magit-section crm compat-27 compat-26
with-editor transient compat log-edit message rfc822 mml mml-sec epa derived epg
rfc6068 epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mailabbrev gmm-utils mailheader pcvs-util add-log autorevert filenotify
aggressive-indent sendmail mail-utils nswbuff yaml-mode yaml guess-language
flyspell ispell reftex reftex-loaddefs reftex-vars csv-mode sort tab-jump-out
yasnippet-snippets yasnippet company-org-block org-settings org-clock ob-jupyter
jupyter-tramp jupyter-server jupyter-server-ioloop jupyter-ioloop-comm
jupyter-ioloop zmq zmq-core jupyter-rest-api url-http url-auth mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr url-gw nsm rmc
jupyter-org-extensions jupyter-org-client jupyter-repl jupyter-kernel-manager
jupyter-channel jupyter-widget-client websocket bindat simple-httpd
jupyter-client jupyter-comm-layer jupyter-messages hmac-def jupyter-mime shr
kinsoku text-property-search puny svg jupyter-kernelspec jupyter-env
jupyter-base eieio-base ob-shell ob-scheme ob-clojure ob-python python tramp-sh
org-element avl-tree dom org ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-macro org-footnote org-src ob-comint org-pcomplete org-list org-faces
org-entities noutline outline org-version ob-emacs-lisp ob-core ob-eval
org-table oc-basic ol org-keys oc org-compat org-macs org-loaddefs find-func
cal-menu calendar cal-loaddefs denote xdg dired dired-loaddefs docker-tramp
tramp-cache tramp tramp-loaddefs trampver tramp-integration files-x tramp-compat
shell pcomplete comint ansi-color parse-time ls-lisp format-spec biblio
biblio-download biblio-dissemin biblio-ieee biblio-hal biblio-dblp
biblio-crossref biblio-arxiv timezone biblio-doi biblio-core let-alist url-queue
ido hl-line bibtex iso8601 time-date adaptive-wrap goggles comp comp-cstr
warnings pulse posframe bm company-quickhelp pos-tip all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons data-fileicons
data-faicons data-alltheicons company-keywords company-etags etags fileloop
generator xref project company-gtags company-dabbrev-code company-dabbrev
company-ipa company-files company-clang company-cmake company-semantic
company-template company-css company-capf company solarized-light-theme
solarized-theme solarized solarized-faces color paradox paradox-menu
paradox-commit-list hydra ring lv cus-edit pp cus-load paradox-execute
paradox-github paradox-core spinner dash cl-extra help-mode jk-functions advice
eshell esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg esh-module esh-groups
esh-util notifications dbus xml edmacro kmacro server elec-pair recentf
tree-widget wid-edit delsel use-package use-package-ensure use-package-delight
use-package-diminish use-package-bind-key bind-key use-package-core tex-site
easy-mmode rx realgud-recursive-autoloads info package browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util
mailcap url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib pcase iso-transl tooltip eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse
jit-lock font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 848943 78066)
 (symbols 48 64672 39)
 (strings 32 228119 24655)
 (string-bytes 1 8079504)
 (vectors 16 112770)
 (vector-slots 8 1998465 158112)
 (floats 8 1911 2096)
 (intervals 56 1567 885)
 (buffers 992 24))

-- 
Joost Kremers
Life has its moments

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Philip Kaludercic]

Joost Kremers <joostkremers@fastmail.fm> writes:

> The variable `package-selected-packages` should not be saved to the users init
> file (or whatever `custom-file` points to). This was discussed before on
> emacs-devel but wasn't resolved:
>
> <https://lists.gnu.org/archive/html/emacs-devel/2016-02/msg00958.html> but 
>
> The gist of it is that if you keep your configuration under version control,
> having package-selected-packages written to your config file is inconvenient if
> you share your config across multiple machines but do not want to have the exact
> same set of packages installed on each.
>
> Even if you do want to keep the exact same set of packages on each machine, you
> still need to be careful about installing them, because the order of package
> installation affects the order of package-selected-packages, and so does
> deleting and reinstalling a package. As a result, I often experienced
> meaningless VC conflicts that needed to be resolved.
>
> IMHO the value of package-selected-packages should be written to a separate
> file, so that it's possible to put it under version control or ignore it,
> independently from the rest of one's configuration.

How about writing it into a file like "~/.config/emacs/elpa/selected-packages"?

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

>>>>> On Fri, 02 Sep 2022 13:11:39 +0000, Philip Kaludercic <philipk@posteo.net> said:
    >> IMHO the value of package-selected-packages should be written to a separate
    >> file, so that it's possible to put it under version control or ignore it,
    >> independently from the rest of one's configuration.

    Philip> How about writing it into a file like "~/.config/emacs/elpa/selected-packages"?

Didnʼt Lars add `define-multisession-variable' exactly for this kind
of use case?

Robert
-- 

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> How about writing it into a file like
> "~/.config/emacs/elpa/selected-packages"?

The defcustom could just be changed to a define-multisession-variable.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

>>>>> On Fri, 02 Sep 2022 15:17:55 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Philip Kaludercic <philipk@posteo.net> writes:
    >> How about writing it into a file like
    >> "~/.config/emacs/elpa/selected-packages"?

    Lars> The defcustom could just be changed to a define-multisession-variable.

For values of 'just' that are a little bigger than changing that one
line 😀

roughʼnʼready patch attached. Seems to work fine. Iʼve not tested the
interaction with a .emacs that has package-selected-packages set, but
if I understand multisession variables correctly that will just be
ignored.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-package-selected-packages-a-multisession-variab.patch --]
[-- Type: text/x-diff, Size: 7539 bytes --]

From 1a84bfbc1b568da6bf63bf9679a317b29f78d0e9 Mon Sep 17 00:00:00 2001
From: Robert Pluim <rpluim@gmail.com>
Date: Fri, 2 Sep 2022 16:28:11 +0200
Subject: [PATCH] Make package-selected-packages a multisession variable
To: emacs-devel@gnu.org

---
 lisp/emacs-lisp/package.el | 63 ++++++++++++++++++--------------------
 1 file changed, 30 insertions(+), 33 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ed23ee5f22..cda55bd90e 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -384,7 +384,8 @@ package-unsigned-archives
   :risky t
   :version "24.4")
 
-(defcustom package-selected-packages nil
+(require 'multisession)
+(define-multisession-variable package-selected-packages nil
   "Store here packages installed explicitly by user.
 This variable is fed automatically by Emacs when installing a new package.
 This variable is used by `package-autoremove' to decide
@@ -395,8 +396,8 @@ package-selected-packages
 To check if a package is contained in this list here, use
 `package--user-selected-p', as it may populate the variable with
 a sane initial value."
-  :version "25.1"
-  :type '(repeat symbol))
+  :synchronized t
+  :package "package")
 
 (defcustom package-native-compile nil
   "Non-nil means to natively compile packages as part of their installation.
@@ -1907,22 +1908,15 @@ package--find-non-dependencies
              unless (memq name dep-list)
              collect name)))
 
-(defun package--save-selected-packages (&optional value)
-  "Set and save `package-selected-packages' to VALUE."
-  (when value
-    (setq package-selected-packages value))
-  (if after-init-time
-      (customize-save-variable 'package-selected-packages package-selected-packages)
-    (add-hook 'after-init-hook #'package--save-selected-packages)))
-
 (defun package--user-selected-p (pkg)
   "Return non-nil if PKG is a package was installed by the user.
 PKG is a package name.
 This looks into `package-selected-packages', populating it first
 if it is still empty."
-  (unless (consp package-selected-packages)
-    (package--save-selected-packages (package--find-non-dependencies)))
-  (memq pkg package-selected-packages))
+  (unless (consp (multisession-value package-selected-packages))
+    (setf (multisession-value package-selected-packages)
+          (package--find-non-dependencies)))
+  (memq pkg (multisession-value package-selected-packages)))
 
 (defun package--get-deps (pkgs)
   (let ((seen '()))
@@ -1950,7 +1944,7 @@ package--removable-packages
   "Return a list of names of packages no longer needed.
 These are packages which are neither contained in
 `package-selected-packages' nor a dependency of one that is."
-  (let ((needed (package--get-deps package-selected-packages)))
+  (let ((needed (package--get-deps (multisession-value package-selected-packages))))
     (cl-loop for p in (mapcar #'car package-alist)
              unless (or (memq p needed)
                         ;; Do not auto-remove external packages.
@@ -2151,8 +2145,8 @@ package-install
                   (package-desc-name pkg)
                 pkg)))
     (unless (or dont-select (package--user-selected-p name))
-      (package--save-selected-packages
-       (cons name package-selected-packages)))
+      (setf (multisession-value package-selected-packages)
+       (cons name (multisession-value package-selected-packages))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)
@@ -2284,8 +2278,8 @@ package-install-from-buffer
     ;; Install the package itself.
     (package-unpack pkg-desc)
     (unless (package--user-selected-p name)
-      (package--save-selected-packages
-       (cons name package-selected-packages)))
+      (setf (multisession-value package-selected-packages)
+       (cons name (multisession-value package-selected-packages))))
     (package--quickstart-maybe-refresh)
     pkg-desc))
 
@@ -2317,9 +2311,9 @@ package-install-selected-packages
   ;; We don't need to populate `package-selected-packages' before
   ;; using here, because the outcome is the same either way (nothing
   ;; gets installed).
-  (if (not package-selected-packages)
+  (if (not (multisession-value package-selected-packages))
       (message "`package-selected-packages' is empty, nothing to install")
-    (let* ((not-installed (seq-remove #'package-installed-p package-selected-packages))
+    (let* ((not-installed (seq-remove #'package-installed-p (multisession-value package-selected-packages)))
            (available (seq-filter (lambda (p) (assq p package-archive-contents)) not-installed))
            (difference (- (length not-installed) (length available))))
       (cond
@@ -2393,7 +2387,8 @@ package-delete
                ;; Don't deselect if this is an older version of an
                ;; upgraded package.
                (package--newest-p pkg-desc))
-      (package--save-selected-packages (remove name package-selected-packages)))
+      (setf (multisession-value package-selected-packages)
+            (remove name (multisession-value package-selected-packages))))
     (cond ((not (string-prefix-p (file-name-as-directory
                                   (expand-file-name package-user-dir))
                                  (expand-file-name dir)))
@@ -2485,7 +2480,7 @@ package-autoremove
   ;; If `package-selected-packages' is nil, it would make no sense to
   ;; try to populate it here, because then `package-autoremove' will
   ;; do absolutely nothing.
-  (when (or package-selected-packages
+  (when (or (multisession-value package-selected-packages)
             (yes-or-no-p
              (format-message
               "`package-selected-packages' is empty! Really remove ALL packages? ")))
@@ -3730,15 +3725,17 @@ package--update-selected-packages
 ADD and REMOVE must be disjoint lists of package names (or
 `package-desc' objects) to be added and removed to the selected
 packages list, respectively."
-  (dolist (p add)
-    (cl-pushnew (if (package-desc-p p) (package-desc-name p) p)
-                package-selected-packages))
-  (dolist (p remove)
-    (setq package-selected-packages
-          (remove (if (package-desc-p p) (package-desc-name p) p)
-                  package-selected-packages)))
-  (when (or add remove)
-    (package--save-selected-packages package-selected-packages)))
+  (let ((packages (multisession-value package-selected-packages)))
+    (dolist (p add)
+      (cl-pushnew (if (package-desc-p p) (package-desc-name p) p)
+                  packages))
+    (dolist (p remove)
+      (setq packages
+            (remove (if (package-desc-p p) (package-desc-name p) p)
+                    packages)))
+    (when (or add remove)
+      (setf (multisession-value package-selected-packages)
+            packages))))
 
 (defun package-menu-execute (&optional noquery)
   "Perform marked Package Menu actions.
@@ -3797,7 +3794,7 @@ package-menu-execute
           ;; Packages being upgraded are not marked as selected.
           (package--update-selected-packages .install .delete)
           (package-menu--perform-transaction install-list delete-list)
-          (when package-selected-packages
+          (when (multisession-value package-selected-packages)
             (if-let* ((removable (package--removable-packages)))
                 (message "Operation finished.  Packages that are no longer needed: %d.  Type `%s' to remove them"
                          (length removable)
-- 
2.37.2.382.g795ea8776b

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> For values of 'just' that are a little bigger than changing that one
> line 😀

😉

> roughʼnʼready patch attached. Seems to work fine. Iʼve not tested the
> interaction with a .emacs that has package-selected-packages set, but
> if I understand multisession variables correctly that will just be
> ignored.

Hm, yes -- if we do this, we'll end up with all users both having that
variable saved in their init file in addition to the multisession
storage.  Hm.  That sounds really confusing.

I can also see people running into difficulties when switching between
older and newer versions of Emacs, and Emacs becoming very confused.

Hm...  I don't immediately see any way to make this work seamlessly in a
way that's backwards/forwards compatible.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> For values of 'just' that are a little bigger than changing that one
>> line 😀
>
> 😉
>
>> roughʼnʼready patch attached. Seems to work fine. Iʼve not tested the
>> interaction with a .emacs that has package-selected-packages set, but
>> if I understand multisession variables correctly that will just be
>> ignored.
>
> Hm, yes -- if we do this, we'll end up with all users both having that
> variable saved in their init file in addition to the multisession
> storage.  Hm.  That sounds really confusing.
>
> I can also see people running into difficulties when switching between
> older and newer versions of Emacs, and Emacs becoming very confused.
>
> Hm...  I don't immediately see any way to make this work seamlessly in a
> way that's backwards/forwards compatible.

What about if we allowed setting `package-selected-packages' to a symbol
like 'external, in which case the list would be saved in a file?  That
way users would have to explicitly opt-in.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

>> Hm...  I don't immediately see any way to make this work seamlessly in a
>> way that's backwards/forwards compatible.
>
> What about if we allowed setting `package-selected-packages' to a symbol
> like 'external, in which case the list would be saved in a file?  That
> way users would have to explicitly opt-in.

But that would still break things for people who go back to using an
older Emacs version.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Hm...  I don't immediately see any way to make this work seamlessly in a
> way that's backwards/forwards compatible.

How about something like this:

- Keep the old defcustom, and add a new multisession variable.

- If the old variable exists, use it.  Otherwise the new one.

- Add a new command that "upgrades" the variable into a multisession
  variable.  Make it say that it will break things on Emacs < 29.  Don't
  run it automatically.

- In Emacs version N (e.g., 5 major versions from now?), upgrade any
  user automatically, without prompting, and make the defcustom
  obsolete.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The variable `package-selected-packages` should not be saved to the
> users init file (or whatever `custom-file` points to).

Yet, for some use cases it should.
So maybe we need to introduce a custom variable to choose whether it is
saved as a normal user config or not.


        Stefan

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

> - Keep the old defcustom, and add a new multisession variable.
>
> - If the old variable exists, use it.  Otherwise the new one.
>
> - Add a new command that "upgrades" the variable into a multisession
>   variable.  Make it say that it will break things on Emacs < 29.  Don't
>   run it automatically.
>
> - In Emacs version N (e.g., 5 major versions from now?), upgrade any
>   user automatically, without prompting, and make the defcustom
>   obsolete.

Hm...  I think this has the disadvantage that it both requires user
intervention, but in addition it'll force you into one usage pattern or
another at some point.

I think perhaps Stefan M's suggestion of adding a new user option to
control the behaviour here is the only practical way forward.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

>>>>> On Sun, 04 Sep 2022 12:52:47 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Stefan Kangas <stefankangas@gmail.com> writes:
    >> - Keep the old defcustom, and add a new multisession variable.
    >> 
    >> - If the old variable exists, use it.  Otherwise the new one.
    >> 
    >> - Add a new command that "upgrades" the variable into a multisession
    >> variable.  Make it say that it will break things on Emacs < 29.  Don't
    >> run it automatically.
    >> 
    >> - In Emacs version N (e.g., 5 major versions from now?), upgrade any
    >> user automatically, without prompting, and make the defcustom
    >> obsolete.

    Lars> Hm...  I think this has the disadvantage that it both requires user
    Lars> intervention, but in addition it'll force you into one usage pattern or
    Lars> another at some point.

    Lars> I think perhaps Stefan M's suggestion of adding a new user option to
    Lars> control the behaviour here is the only practical way forward.

Iʼm not convinced thatʼs necessary. I just did the following:

1. emacs-29+my patch, install some packages => no package-selected-packages
setting
2. emacs-28, run `package-list-packagesʼ => package-selected-packages
is populated with packages set in step 1, and saved in .emacs
3. emacs-28, install some packages
4. emacs-29+my-patch, `package-list-packagesʼ, error because it now
has a setting for package-selected-packages, which means we need to
call the multisession variable something different. I fixed that, and
the packages installed in step 3 show up [1]

So for the small number of people going backwards from emacs-29, the
problem fixes itself, and we can document that, and for people going
forwards to emacs-29 and staying there is no manual intervention
needed.

Robert

Footnotes:
[1]  We can now argue about what to call that variable. I went for
     `package-selected-packages-msʼ for now

-- 

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Iʼm not convinced thatʼs necessary. I just did the following:
>
> 1. emacs-29+my patch, install some packages => no package-selected-packages
> setting
> 2. emacs-28, run `package-list-packagesʼ => package-selected-packages
> is populated with packages set in step 1, and saved in .emacs

Oh, I didn't know that it was automatically recreated.  Then what's the
point of the variable?  According to the doc string:

---
Store here packages installed explicitly by user.
---

Does 2) there get that right, or does it contain all packages installed,
explicitly or not?

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

>>>>> On Mon, 05 Sep 2022 13:16:24 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >> Iʼm not convinced thatʼs necessary. I just did the following:
    >> 
    >> 1. emacs-29+my patch, install some packages => no package-selected-packages
    >> setting
    >> 2. emacs-28, run `package-list-packagesʼ => package-selected-packages
    >> is populated with packages set in step 1, and saved in .emacs

    Lars> Oh, I didn't know that it was automatically recreated.  Then what's the
    Lars> point of the variable?  According to the doc string:

I guess reading a variable is faster than recursing over the elpa
directory.

    Lars> ---
    Lars> Store here packages installed explicitly by user.
    Lars> ---

    Lars> Does 2) there get that right, or does it contain all packages installed,
    Lars> explicitly or not?

It contains the packages I installed using 'i' in list-packages, but
not their automatically installed dependencies.

Robert
-- 

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

>>>>> On Mon, 05 Sep 2022 13:51:00 +0200, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Mon, 05 Sep 2022 13:16:24 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:
    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >>> Iʼm not convinced thatʼs necessary. I just did the following:
    >>> 
    >>> 1. emacs-29+my patch, install some packages => no package-selected-packages
    >>> setting
    >>> 2. emacs-28, run `package-list-packagesʼ => package-selected-packages
    >>> is populated with packages set in step 1, and saved in .emacs

    Lars> Oh, I didn't know that it was automatically recreated.  Then what's the
    Lars> point of the variable?  According to the doc string:

Of course one use case that breaks is to do

emacs-29->emacs-28, *delete* some packages, and then go back to
emacs-29, which will pick up the old multi-session value.

I guess we could change emacs-29 to refresh the multi-session value
every time you run `list-packages'.

Robert
-- 

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

>     Lars> Oh, I didn't know that it was automatically recreated.  Then
>     Lars> what's the
>     Lars> point of the variable?  According to the doc string:

So it just caches data to speed things up, I guess?  

> Of course one use case that breaks is to do
>
> emacs-29->emacs-28, *delete* some packages, and then go back to
> emacs-29, which will pick up the old multi-session value.
>
> I guess we could change emacs-29 to refresh the multi-session value
> every time you run `list-packages'.

Hm.  Or...  refresh the value if the .emacs.d/elpa directory has fresher
contents than the multisession variable, but I don't know whether that's
faster.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

>>>>> On Mon, 05 Sep 2022 21:04:52 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:
    >> Of course one use case that breaks is to do
    >> 
    emacs-29-> emacs-28, *delete* some packages, and then go back to
    >> emacs-29, which will pick up the old multi-session value.
    >> 
    >> I guess we could change emacs-29 to refresh the multi-session value
    >> every time you run `list-packages'.

    Lars> Hm.  Or...  refresh the value if the .emacs.d/elpa directory has fresher
    Lars> contents than the multisession variable, but I don't know whether that's
    Lars> faster.

How do we define 'fresher'? Anyway, if in doubt, use brute force:

(define-multisession-variable package-selected-packages-ms
                              (package--find-non-dependencies))

works fine. We could even condition it on `package-selected-packages'
being non-nil.

Robert
-- 

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> How do we define 'fresher'?

I left that as an exercise for the reader.  🫠  

>  Anyway, if in doubt, use brute force:
>
> (define-multisession-variable package-selected-packages-ms
>                               (package--find-non-dependencies))
>
> works fine. We could even condition it on `package-selected-packages'
> being non-nil.

Hm...  I wonder whether we could just drop the variable altogether.
That is, checking for freshness sounds like as much work as
recalculating the variable.

What are the actual effects of the variable?  That is, what code is it
supposed to speed up?

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

>>>>> On Tue, 06 Sep 2022 12:34:00 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >> How do we define 'fresher'?

    Lars> I left that as an exercise for the reader.  🫠  

    >> Anyway, if in doubt, use brute force:
    >> 
    >> (define-multisession-variable package-selected-packages-ms
    >> (package--find-non-dependencies))
    >> 
    >> works fine. We could even condition it on `package-selected-packages'
    >> being non-nil.

    Lars> Hm...  I wonder whether we could just drop the variable altogether.
    Lars> That is, checking for freshness sounds like as much work as
    Lars> recalculating the variable.

On further testing, using package--find-non-dependencies doesnʼt seem
to work 100% correctly: installing a package in emacs-29 and then
going back to emacs-28 has it showing up as 'dependency', not
'installed', in `list-packages'.

    Lars> What are the actual effects of the variable?  That is, what code is it
    Lars> supposed to speed up?

Itʼs supposed to allow people to deactivate packages without
uninstalling them, I think, although Iʼve never used it like that, and
Iʼm not 100% sure it works, since `package--find-non-dependencies'
looks in `package-alist', which I presume contains all the packages
found in .emacs.d/elpa

Thierry, you added the defcustom, is there any light you can shed
here? From my perspective we can stop updating the defcustom, but
there are bound to be people using it for something.

Robert
-- 

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

>>>>> On Tue, 06 Sep 2022 15:49:29 +0200, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Tue, 06 Sep 2022 12:34:00 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:
    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >>> How do we define 'fresher'?

    Lars> I left that as an exercise for the reader.  🫠  

    >>> Anyway, if in doubt, use brute force:
    >>> 
    >>> (define-multisession-variable package-selected-packages-ms
    >>> (package--find-non-dependencies))
    >>> 
    >>> works fine. We could even condition it on `package-selected-packages'
    >>> being non-nil.

    Lars> Hm...  I wonder whether we could just drop the variable altogether.
    Lars> That is, checking for freshness sounds like as much work as
    Lars> recalculating the variable.

    Robert> On further testing, using package--find-non-dependencies doesnʼt seem
    Robert> to work 100% correctly: installing a package in emacs-29 and then
    Robert> going back to emacs-28 has it showing up as 'dependency', not
    Robert> 'installed', in `list-packages'.

OK, this is because emacs-28 doesnʼt always update
`package-selected-packages' the way I expected it to. I think weʼve
come full circle, and would have to have emacs-29 continue to update
it, unless itʼs explicitly set to some distinct value (Someone™
upthread suggested 'external).

Robert
-- 

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[jakanakaevangeli]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> How do we define 'fresher'?
>
> I left that as an exercise for the reader.  🫠  
>
>>  Anyway, if in doubt, use brute force:
>>
>> (define-multisession-variable package-selected-packages-ms
>>                               (package--find-non-dependencies))
>>
>> works fine. We could even condition it on `package-selected-packages'
>> being non-nil.
>
> Hm...  I wonder whether we could just drop the variable altogether.
> That is, checking for freshness sounds like as much work as
> recalculating the variable.
>
> What are the actual effects of the variable?  That is, what code is it
> supposed to speed up?

Isn't it used for distinguishing packages installed automatically as
dependencies from packages installed explicitly by the user? This
affects M-x package-autoremove.

And if my opinion counts, I like having the variable in my .emacs,
possibly under revision control. On a fresh machine, I can then simply
type M-x package-install-selected-packages to install all the packages
from the variable, and M-x package-autoremove to uninstall all packeges
not in listed the variable.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Stefan Kangas]

<jakanakaevangeli@chiru.no> writes:

> And if my opinion counts, I like having the variable in my .emacs,
> possibly under revision control. On a fresh machine, I can then simply
> type M-x package-install-selected-packages to install all the packages
> from the variable, and M-x package-autoremove to uninstall all packeges
> not in listed the variable.

Can't this use case be supported with a multisession variable too though?

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[jakanakaevangeli]

Stefan Kangas <stefankangas@gmail.com> writes:

> <jakanakaevangeli@chiru.no> writes:
>
>> And if my opinion counts, I like having the variable in my .emacs,
>> possibly under revision control. On a fresh machine, I can then simply
>> type M-x package-install-selected-packages to install all the packages
>> from the variable, and M-x package-autoremove to uninstall all packeges
>> not in listed the variable.
>
> Can't this use case be supported with a multisession variable too
> though?

Indeed it can, sorry. I was under the false impression that multisesion
variables were stored in a sqlite database by default, which isn't
suitable for revision control.

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Robert Pluim]

>>>>> On Wed, 7 Sep 2022 02:09:25 +0200, Stefan Kangas <stefankangas@gmail.com> said:

    Stefan> <jakanakaevangeli@chiru.no> writes:
    >> And if my opinion counts, I like having the variable in my .emacs,
    >> possibly under revision control. On a fresh machine, I can then simply
    >> type M-x package-install-selected-packages to install all the packages
    >> from the variable, and M-x package-autoremove to uninstall all packeges
    >> not in listed the variable.

    Stefan> Can't this use case be supported with a multisession variable too though?

Yes. The multisession variable would contain the same information, it
would just not be in the custom file. (I thought the whole reason for
this bug was that there was strong dislike for storing the selected
packages as a custom variable, precisely because it disturbed version
control)

Robert
-- 

bug#57503: 28.1.91; package-selected-packages should not be saved to custom-file

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Yes. The multisession variable would contain the same information, it
> would just not be in the custom file. (I thought the whole reason for
> this bug was that there was strong dislike for storing the selected
> packages as a custom variable, precisely because it disturbed version
> control)

Yes, that was the original complaint -- that a user may have different
installed packages on different machines, but want to use the same
.emacs file.  And in that case, package-selected-packages would be
wrong.

So there seems to be different use cases.